If you've ever trained a deep learning model, you've probably watched a GPU turn weeks of training into hours. But what actually makes a GPU so good at machine learning? Having more cores is part of the answer, but the bigger reason is how its memory, compute units, and parallelism are balanced so that a huge number of operations can keep flowing at the same time.
At a high level, training a neural network boils down to matrix multiplications: multiplying and adding huge grids of numbers, millions or billions of times over. GPUs are built for exactly this. They give you thousands of compute units to do the math, specialized hardware for matrix operations (tensor cores), and a memory system (HBM and caches) that can feed all of those units fast enough to keep them busy.
So why care about the hardware at all? Once you know where GPUs shine and where they bottleneck, it gets much easier to see why training can slow down even on a powerful cluster, and why "more memory" or "more FLOPs" usually isn't the whole story.
In this article, I'll walk through the main pieces of a modern GPU and how they fit together. Most of them get their own deeper post later in the series, and I'll link to those along the way.
Memory: Feeding the Compute
No matter how powerful a GPU's cores are, they can only go as fast as the data reaching them. Training a large network means constantly moving weights, activations, and gradients around, and the memory system is what makes that possible.
High Bandwidth Memory (HBM)
Modern data center GPUs use High Bandwidth Memory (HBM) instead of the DDR or GDDR memory you'd find in a typical CPU or graphics card. HBM is physically stacked next to the GPU die and connected to it through a silicon interposer, which allows for thousands of connections between the two. The result is extremely high throughput without blowing up power consumption.
How high? That's what the "GB/s" number on a spec sheet tells you: the memory bandwidth, or the maximum rate at which data can move between HBM and the GPU cores. An NVIDIA H100 provides about 3.35 TB/s, compared to roughly 100 GB/s for a high end CPU. That gap is what keeps the compute units busy instead of sitting idle while they wait for data. (I go into more detail in the HBM post.)
Caches
Even with HBM, moving data is expensive in both time and energy. To cut down on that cost, GPUs use a cache hierarchy. The L2 cache is shared across all of the streaming multiprocessors (SMs), so the GPU doesn't have to go back to HBM every time it needs the same data. The L1 cache lives inside each SM and gives threads in the same block even faster access to nearby values. And registers are the fastest storage of all: they sit right next to the cores and hold the values being computed on at that moment. (More on this in the memory hierarchy post.)
Compute: Where the Math Happens
Memory supplies the data, but the compute units are what actually run the matrix operations that make up training. In modern GPUs, these units are organized into Streaming Multiprocessors (SMs), which are the basic building blocks of GPU computation.
Streaming Multiprocessors (SMs)
Each GPU die contains many SMs (108 in the NVIDIA A100 and 132 in the H100). An SM has everything it needs to run thousands of threads in parallel: CUDA cores for general floating point and integer math, tensor cores for matrix operations, registers for each thread's active variables, L1 cache and shared memory for fast local reuse, and schedulers that keep the cores busy even when some threads are stuck waiting on memory.
Because the same SM design is repeated across the whole chip, a GPU can run an enormous number of threads at once, which is exactly the kind of parallelism matrix multiplication needs. (See the SM post for what's inside one.)
CUDA Cores and Tensor Cores
Inside each SM there are two kinds of compute units. CUDA cores are the general purpose ones. They do standard single precision (FP32) math, and depending on the architecture they can also handle FP64 and integer operations, so they take care of everything from control flow and indexing to basic arithmetic.
Tensor cores are the specialists. They were introduced with NVIDIA's Volta architecture, and they're built for one thing: matrix multiply and accumulate (MMA) operations, which are the backbone of deep learning. Instead of working on one number at a time, a tensor core works on a small matrix tile (e.g., 16×16) in a single instruction, which adds up to hundreds of floating point operations per cycle.
They're optimized for lower precision formats like FP16, BF16, and FP8, and they usually accumulate results in FP32 to keep accuracy up. That's why, for workloads dominated by matrix math, tensor cores get orders of magnitude more throughput than CUDA cores.
Performance Metrics
Core counts and memory size alone don't tell you much about how a GPU will do on ML workloads. What actually matters is how much useful computation it can do every second, and whether the memory system can keep up with it.
FLOPs and FLOPs per Cycle
A FLOP is a floating point operation (a single addition or multiplication, for example). GPU datasheets usually report performance in teraFLOPs (TFLOPs), meaning trillions of floating point operations per second.
Total throughput comes down to three factors:
A cycle is one tick of the GPU's clock (clock speed is measured in GHz). FLOPs per cycle is how many operations a compute unit can do in each tick: tensor cores can do hundreds, while CUDA cores usually do one or two. The number of cores is how many CUDA and tensor cores you have across all of the SMs. Multiply the three together and you get the theoretical peak throughput of the GPU.
Memory Bound vs. Compute Bound
In practice, GPUs almost never hit that theoretical peak, and what holds them back depends on the workload. If a workload is compute bound, the math units are the bottleneck, so more FLOPs directly means more speed. If it's memory bound, the GPU spends its time waiting for data, and more bandwidth or bigger caches make the biggest difference. Deep learning workloads usually land somewhere in between, which is why spec sheets highlight both FLOPs and memory bandwidth.
Threads and Parallelism
The real strength of a GPU is that it can run tens of thousands of operations at once. So how is all of that work actually organized?
It starts with threads. Your training script runs as a process, and inside that process the work is split into threads, which are the smallest units of execution. Threads share the same memory space, but each one has its own registers and execution state, so they can work on different parts of a problem without stepping on each other. On NVIDIA GPUs, threads are grouped into sets of 32 called warps. A warp is the unit the GPU actually schedules: all 32 threads run the same instruction at the same time, each on its own piece of data. Warps are then grouped into thread blocks, and each block gets assigned to a specific SM.
This is where the schedulers come in. Each SM has several warp schedulers that decide which warps run in a given cycle, and when one warp stalls waiting on memory, the scheduler simply switches to another warp that's ready to go. That's how GPUs "hide" memory latency. With many SMs each running many blocks, a GPU can have tens of thousands of threads in flight at once, which is exactly what you want for deep learning, where the same operation gets applied over and over to large amounts of data.
If I had to sum it up, a GPU is fast at machine learning because it never lets its tensor cores wait for long: the memory system keeps data coming, and the schedulers always have another warp ready to run. The rest of this series goes one level deeper on each of these pieces, starting with High Bandwidth Memory.