When you train a large neural network, a single GPU often isn't enough. Data Parallelism (DP) is the simplest and most common way to spread training across multiple GPUs. The idea is pretty straightforward: give every GPU a copy of the model, split the data between them, compute gradients locally, and then average those gradients so every copy stays in sync.
PyTorch's DistributedDataParallel (DDP) and TensorFlow's MirroredStrategy both support it out of the box, which is why it's the default way to train on multiple GPUs in both research and production.

Figure 1: Data parallel training. Each GPU holds a copy of the model, processes its own minibatch, and computes local gradients . The gradients are then averaged across GPUs to get the global update, which keeps every copy of the model in sync.
How Data Parallelism Works
The whole process breaks down into six steps:
- Replicate the model: Each GPU keeps its own copy of the model and the optimizer state.
- Split the minibatch: If the global minibatch has size , it gets split evenly so that each of the GPUs processes a minibatch of size .
- Forward pass: Each GPU computes the activations and the loss for its own minibatch.
- Backward pass: Each GPU computes the gradients for its own minibatch.
- Gradient synchronization: The gradients are averaged across all of the GPUs, usually with an AllReduce operation.
- Parameter update: Each GPU applies the same averaged gradient, so all of the model copies stay identical.
In practice, steps 4 and 5 overlap. The backward pass produces gradients for the last layers first, so those can start syncing across GPUs while the backward pass is still working through the earlier layers.
The Math
On a single GPU, training with a minibatch of size minimizes the average loss:
where is the loss for a single sample.
With data parallelism across GPUs, the effective minibatch size becomes . Each GPU computes the loss on its own minibatch:
and its own local gradient:
The gradients are then averaged across GPUs:
so the update step becomes:
where is the learning rate.
Why It Works
So why is this the same as training on one giant GPU? It comes down to the fact that gradients are linear. The gradient of the average loss over all samples is exactly the average of the local gradients:
That means averaging gradients across GPUs gives you exactly the same update you'd get from training on a single GPU with a minibatch of size . Nothing about the training changes; it just runs faster.
Limitations
Data parallelism is simple, but it isn't free:
- Communication overhead: Every step needs an AllReduce across all of the GPUs to sync gradients, and that gets expensive at large scale.
- Memory duplication: Every GPU stores the entire model and its optimizer state, which just isn't possible for very large models.
- Diminishing returns: Past a certain number of GPUs, scaling efficiency drops because communication starts to take more time than computation.
In Practice
Data parallelism is the default for training models that fit in a single GPU's memory (think ResNet or BERT base). For very large models with billions of parameters, it isn't enough on its own, and it has to be combined with pipeline or tensor parallelism. There are also optimizer sharding techniques like ZeRO (Zero Redundancy Optimizer) in DeepSpeed, which extend data parallelism by splitting the optimizer states across GPUs to save memory.
When the Model Doesn't Fit
All of this works as long as the whole model and its optimizer state fit in one GPU's memory. For large models, that's exactly where things start to break down.
Here's a rough estimate. When training with Adam, each parameter needs four values: the weight itself, its gradient, and Adam's two optimizer states (the running estimates of the gradient's first and second moments). If each of those takes 2 bytes (FP16 or BF16), that's 8 bytes per parameter. So a model with 1 billion parameters already needs about 8 GB, and a 10 billion parameter model needs around 80 GB, which fills an entire 80 GB A100 by itself. (In real mixed precision training the optimizer states are usually kept in FP32, so the actual number is even higher.)
The fix is Fully Sharded Data Parallelism (FSDP). Instead of every GPU holding a full copy of the weights and optimizer states, they get split up across the GPUs. That's what I'll cover in Part 2.