The Precision Revolution

The journey from A100 to B200 is a story of a relentless push into lower, more efficient numerical precisions. The A100's TF32 provided a "free" performance boost for FP32 workloads. The H100's Transformer Engine masterfully managed FP8 to accelerate LLMs. Now, Blackwell's second-gen engine introduces FP4, a game-changer for inference that can double performance and halve memory requirements compared to FP8.

Fueling the Engines

This immense compute power requires an equally impressive memory subsystem. Each generation has adopted the latest High Bandwidth Memory standard—from HBM2e on the A100, to HBM3 on the H100, and now HBM3e on the B200. This has resulted in an exponential increase in memory bandwidth, which is crucial for feeding the voracious Tensor Cores and handling the massive datasets and models of modern AI.

Purpose-Built for Different Scales

NVIDIA's data center GPUs are not one-size-fits-all. They come in two primary form factors designed for different server architectures and scalability needs:

• SXM: This is a mezzanine-style card that plugs directly into a specialized motherboard (like NVIDIA's HGX boards). It allows for the highest possible GPU density and the full bandwidth of NVLink between GPUs. This is the preferred form factor for building massive, scale-up AI supercomputers where inter-GPU communication is paramount.
• PCIe: This is the familiar double-width card that plugs into standard PCIe slots found in most servers. While offering broader compatibility, it typically has a lower power limit (TDP) and relies on the slower PCIe bus for GPU-to-GPU communication outside of an NVLink Bridge. It's ideal for scale-out workloads and deploying smaller numbers of GPUs in existing server infrastructure.

Beyond a Single GPU

Modern AI models are too large to fit on a single GPU. Training them requires a fleet of GPUs working in concert. The performance of this system is defined not by the speed of a single chip, but by the speed of the network connecting them. This is where NVLink and NVSwitch come in.

Each GPU generation has seen a corresponding leap in its dedicated interconnect fabric. A100's 600 GB/s NVLink was fast, but H100's 900 GB/s, combined with the external NVSwitch, allowed for 256 GPUs to be connected in a seamless, high-bandwidth domain. Blackwell's 5th-gen NVLink doubles the speed again to 1.8 TB/s per GPU, enabling staggering systems like the GB200 NVL72, which connects 72 GPUs as if they were one giant accelerator.

A Marriage of CPU and GPU

Recognizing that data movement between the CPU and GPU memory is a major bottleneck in HPC and massive AI, NVIDIA introduced the "Superchip" concept. This platform tightly integrates their high-performance Grace ARM CPU with a Hopper or Blackwell GPU over a high-speed, cache-coherent interconnect.

The GH200 Grace Hopper Superchip was the first of its kind, providing a massive, unified memory space for workloads that exceed the GPU's HBM capacity. The new GB200 Superchip takes this further by connecting two B200 GPUs to a single Grace CPU, creating a computational behemoth for the most demanding AI training and inference tasks. This system-level integration is a key part of NVIDIA's strategy to deliver performance that individual components alone cannot achieve.

AI Inference Throughput

For deployed AI services, inference performance is the critical business metric. Here, the architectural specializations for lower precision have a profound effect. The H100, with its FP8 support, delivered up to a 4.5x performance increase over the A100. Blackwell's introduction of FP4 is set to revolutionize inference economics again, showing up to a 4x improvement over the H100 on key LLM benchmarks, and a massive 30x gain in large, multi-GPU systems like the NVL72.

The Power Wall

The monumental performance gains have been accompanied by a dramatic increase in power consumption, with the B200's TDP reaching 1000W. This necessitates an industry-wide shift to liquid cooling for at-scale deployments. However, the more critical metric is performance-per-watt. Each generation has delivered significant improvements here, as the gains in computational performance have vastly outpaced the increases in power draw.

This dynamic trades higher instantaneous power for a massive reduction in time-to-solution, leading to a net improvement in total energy-to-solution. A single B200 can replace a rack of H100s for some inference workloads, leading to significant TCO savings in server count, networking, and rack space.