In the last post, we saw that HBM gives GPUs an enormous amount of bandwidth. But in terms of latency, HBM is still relatively far away from the compute units. To close that gap, GPUs use a layered memory hierarchy: the L2 cache, the L1 cache (which doubles as shared memory), and registers. Each level trades capacity for speed, so the data being used most often stays as close to the cores as possible.
Why the Hierarchy Matters
Training and inference keep reading the same weights, activations, and intermediate results over and over. If every one of those reads went all the way to HBM, you'd saturate its bandwidth and slow everything down. By staging data through memories that get smaller and faster the closer they are to the cores, the GPU cuts down on both latency and energy, and keeps the tensor and CUDA cores fed.
Figure 1: The inside of the NVIDIA H100's compute die. You can see the 50 MB of L2 cache and the 132 streaming multiprocessors laid out next to each other.
L2 Cache: A Shared Buffer for Every SM
The L2 cache sits on the GPU die but outside the streaming multiprocessors (SMs), which you can see in Figure 1. It's shared across the whole GPU, so every SM can read from it. It's also fairly large: the A100 has 40 MB of L2, and the H100 bumps that up to 50 MB.
L2 is much faster than HBM (tens of nanoseconds versus hundreds), but still slower than L1. Its main job is to hold data that was recently pulled from HBM, so the GPU doesn't keep making trips to the same place. Say several SMs need the same weights. With L2 in the middle, only the first read has to go to HBM, and every read after that is served from the cache. This is especially useful with large batch sizes, or with layers where many SMs end up reading the same weights.
Figure 2: Inside a streaming multiprocessor (SM) of an H100, with the L1 instruction cache at the top.
L1 Cache and Shared Memory: Fast Storage Inside Each SM
The L1 cache lives inside each SM and is private to it. It's small (up to 256 KB per SM on the H100, and 192 KB on the A100) but very fast, with access times of just a few nanoseconds. In CUDA, this memory can be split between a regular cache and a region you manage yourself called shared memory, which lets you decide exactly how data gets reused.
That dual role is what makes L1 so useful. As a cache, it automatically keeps recently loaded data around so later reads don't have to go back to L2 or HBM. As shared memory, it lets all of the threads in a block work together: they load a piece of data once and reuse it many times. This is the whole idea behind tiled matrix multiplication. Instead of every thread fetching its own inputs, the block cooperates to load a tile into shared memory, and then everyone reads from there. Without it, matrix and convolution kernels would flood the memory bus with duplicate requests.
Registers: Each Thread's Working Memory
Registers are the fastest memory on the GPU, and reading one takes about a single clock cycle. They live inside each SM right next to the CUDA and tensor cores, and every thread gets its own private set that no other thread can see. The total amount of register storage is actually pretty large (256 KB per SM, or about 27 MB across an entire A100), but each individual thread can only use up to 255 registers.
Every arithmetic instruction the GPU runs works on values in registers. That includes the matrix elements being multiplied, loop counters, memory addresses, and the partial sums that build up during a computation. The catch is that if a thread needs more registers than it can get, the extra values get "spilled" out to slower memory, which hurts performance. There's also a tradeoff: the more registers each thread uses, the fewer threads can be active on the SM at once, which means less parallelism. Getting register usage right is a big part of writing fast GPU kernels.
Putting It All Together
Let's say you're training a transformer with billions of parameters. Here's how the data actually moves through the hierarchy.
First, the model weights, activations, and optimizer states are pulled from HBM, which is the big but relatively slow memory on the GPU. That data lands in the L2 cache, where every SM can reach it. If several SMs need the same weights (which is common), they read them from L2 instead of going back to HBM.
Next, inside an SM, warps copy tiles of the matrix data into shared memory in L1. The threads then reuse that shared copy to do their part of the matrix multiplication without fetching anything twice. Finally, each thread loads its own operands into registers, and the tensor or CUDA cores consume those values to do the actual multiply and accumulate. Intermediate results stay in registers until the computation finishes, and then everything flows back up the hierarchy: registers to L1, L1 to L2, and L2 back to HBM.
The way I think about it, every level of the hierarchy exists to save a trip to the level above it, and fast kernels are the ones that take advantage of all of them. In the next post, we'll look at the streaming multiprocessor, which is where all of this data finally gets used.