High Bandwidth Memory (HBM) is the main memory technology in modern data center GPUs, including NVIDIA's A100 and H100 and AMD's MI250. Unlike traditional DDR or GDDR memory, HBM is stacked right next to the GPU die and connected to it through a silicon interposer. That design gives you both large capacity and very high bandwidth, which are exactly the two things you need to train today's large models.
In the previous post, I gave a quick tour of GPU architecture. Here I want to look at HBM more closely, and especially at what its numbers actually mean when you read a spec sheet.

Figure 1: Components of the NVIDIA H100 GPU, including the GPU die, the compute cores, and the HBM.
Why HBM Matters for Machine Learning
Training a deep neural network means moving huge amounts of data around: the model's weights (its parameters), activations (the intermediate results from the forward pass), and gradients along with optimizer states (used during backpropagation). If the memory can't supply that data fast enough, the compute units stall, and much of the GPU's available FLOPs go to waste.
HBM solves this by moving data at terabytes per second (3,352 GB/s on the H100, as shown in Figure 1). That keeps the CUDA and tensor cores busy, and it's a big part of why training very large models is efficient at all.
How HBM Works
So what's actually different about it? HBM is made of several DRAM dies stacked vertically on top of each other (that's where the "stacked" part of the name comes from). The stack sits right next to the GPU die and connects to it through a silicon interposer, which is a layer of silicon with thousands of very fine wires running through it. That's what allows such dense communication between the memory and the GPU.
The interesting part is how it gets its speed. Instead of running at a very high clock frequency, HBM uses an extremely wide interface (thousands of bits wide) and moves a large amount of data every cycle. That gets you high bandwidth and good power efficiency at the same time. Each stack holds a few gigabytes, and GPUs put several stacks together to reach totals like the 80 GB on the NVIDIA H100.
The Two Numbers That Matter
When you look at a GPU spec sheet, HBM performance comes down to two numbers.
Capacity is how much model data fits directly on the GPU. More capacity means you can fit a bigger model or a larger batch without offloading to slower CPU memory or splitting the work across multiple GPUs.
Bandwidth is how fast data moves between HBM and the compute cores. The H100 provides about 3,350 GB/s, compared to roughly 100 GB/s for a high end CPU with DDR5. That's around a 30x difference, and it's the main reason a GPU can keep its tensor cores fed.
What This Means in Practice
More capacity lets larger models and batch sizes fit on a single GPU, so you don't need a complicated setup with several GPUs just to hold your data. More bandwidth keeps the matrix multiplication units busy instead of idle, which translates directly into faster training. And since HBM uses less energy per bit transferred than DDR or GDDR, data centers can get that throughput while staying within their power budgets.
Wrapping Up
HBM is stacked memory that sits right next to the GPU die and talks to it through a silicon interposer, which is how it reaches terabytes per second of bandwidth (far more than DDR or GDDR can offer). For machine learning, both of its numbers matter: capacity decides what fits on the GPU, and bandwidth decides how fast the computation can actually go. That's why data center GPUs like the A100, H100, and MI250 all rely on it.
Even at these speeds, though, HBM is still relatively far from the compute units. In the next post, I'll go over the caches and registers that sit between HBM and the cores.