Skip to content

GPUs for Machine Learning · Part 4

Streaming Multiprocessors: Scheduling and Execution

What's inside a streaming multiprocessor, and how it keeps thousands of threads busy at once.

· 4 min read

Modern GPUs get their parallel performance from Streaming Multiprocessors (SMs). An SM is the basic compute block of a GPU, and it has everything needed to run thousands of threads in parallel: cores for arithmetic, fast local memory, registers, and schedulers. GPU makers scale performance by repeating the same SM design across the die, which is how a GPU goes from tens of gigaflops to multiple petaflops. NVIDIA's A100 has 108 SMs, for example, and the newer H100 has 132.

If you understand what an SM does, it becomes much clearer why GPUs are so good at deep learning. So in this post, I want to open one up and look at what's inside.

Why SMs Matter for Machine Learning

The SM is where the computation actually happens. Each one can keep up to 2,048 threads resident at once on the H100, contains both general purpose CUDA cores and tensor cores, and manages fast local storage through registers and shared memory. It also has schedulers that interleave all of those threads so the cores stay busy even while some threads are waiting on data. Without SMs, the GPU's cores would just be isolated units. The SM is what organizes them into something that can work in parallel.

Floorplan of an NVIDIA H100 streaming multiprocessor

Figure 1: H100 SM floorplan: 256 KB of L1 and shared memory, a 16K register file for each warp scheduler, 128 FP32 ALUs, and four fourth generation tensor cores that together deliver 4,096 mixed precision FLOPs per cycle.

Anatomy of an SM

I think the easiest way to understand what's inside an SM is to follow a single tile of a matrix multiplication through it.

First, the load/store units pull a tile of weights and activations in from L2 or HBM and write it into shared memory, which shares the same fast storage as the L1 cache. Shared memory is visible to every thread in a block, so the tile only has to be fetched once even though many threads are going to use it. This is the trick that makes tiled matrix multiplication work.

From there, each thread copies the elements it needs into its own registers. Registers are the fastest storage on the chip and are private to each thread, and every arithmetic instruction the SM runs reads from and writes to them.

Now the actual math happens. The tensor cores are built for exactly one kind of operation, a matrix multiply and accumulate of the form D=A×B+C\text{D} = \text{A} \times \text{B} + \text{C}. They work on small tiles in reduced precision (FP16, BF16, or FP8) and accumulate the results in FP32, which lets them do hundreds of floating point operations per cycle. If you're training a neural network, this is where most of the work is being done. The CUDA cores handle everything around it: they do general FP32, FP64, and integer math, so the indexing, loop counters, and control flow that keep the kernel running all happen on them.

Intermediate results stay in registers (or get parked in shared memory) until the tile is finished, and then the final values are written back out to L2 and eventually HBM.

The one piece we haven't seen yet is what keeps all of this busy, and that's the warp schedulers.

Parallel Execution in an SM

Threads on a GPU don't run one at a time. They're grouped into warps of 32, and warps are grouped into thread blocks. An SM can run several blocks at the same time, and each of its warp schedulers decides which warp gets to issue an instruction in a given cycle.

This is how GPUs hide latency. Fetching a tile from HBM takes hundreds of cycles, and a warp waiting on that data can't do anything useful. Instead of letting the cores sit idle, the scheduler just switches to another warp that's ready to go. Switching between warps is cheap, so as long as there are enough warps to choose from, the SM can keep its CUDA and tensor cores running close to peak throughput.

Put together, an SM is basically a self contained compute cluster. The GPU die is built by repeating it over and over, and that's what lets GPUs scale for everything from graphics to deep learning.

What This Means in Practice

More SMs means more parallel execution, so newer GPUs with higher SM counts can train larger models or get through more data in the same amount of time. But the SM count isn't everything. How many registers and how much shared memory each thread block uses, and how the warps get scheduled, all affect how many threads can be active on an SM at once (this is called occupancy). A badly tuned kernel can leave cores sitting idle even on a GPU that looks great on paper.

In the end, it comes down to balance. The interaction between compute and memory inside each SM decides whether a kernel is compute bound or memory bound, and getting that balance right is a big part of what makes GPU code fast.

That wraps up the single GPU side of this series. In the next post, we'll look at what happens when one GPU isn't enough and training has to be split across many of them.

References