In Part 1, we saw that Data Parallelism (DP) is simple and works well, but it assumes every GPU stores a full copy of the model weights, gradients, and optimizer states. Once you get to models with several billion parameters, that duplication stops being affordable.
Fully Sharded Data Parallelism (FSDP) fixes this by splitting the model state across GPUs. Instead of every GPU holding the whole model, the weights, gradients, and optimizer states are sharded so that each GPU only holds a piece of them. During training, a GPU only receives the parameters it needs right before it uses them, and throws them away afterward. This cuts memory usage dramatically, and the training itself is still mathematically the same as DP.
Why FSDP Matters
To see why this matters, think about what you have to store for every parameter when training with Adam: the weight itself, its gradient, and Adam's two optimizer states (the running estimates of the gradient's first and second moments). At 2 bytes per number, that's 8 bytes per parameter. For a model with 10 billion parameters, that comes out to about 80 GB of memory, which is enough to fill an entire NVIDIA A100 by itself.
How FSDP Works
In FSDP, each shard of the parameters has an owner GPU. The owner is responsible for storing that shard's weights, its gradients, and its optimizer states. Training then goes layer by layer.
Before the forward pass for a layer, the GPU that owns it broadcasts the weights to every other GPU. All of the GPUs run the forward pass for that layer with those weights, and then each one throws away its copy of to free up memory. There's no reason to hold onto weights that won't be needed again until backpropagation.
Before the backward pass, the owner broadcasts again. Each GPU computes its local gradients for that layer and then throws away the temporary copy of the weights. Next, the GPUs send their local gradients back to the owner with a ReduceScatter, and the owner combines them into the global gradient for . Finally, the owner uses its shard of the optimizer states to update its part of the parameters.
There's one more important optimization: overlapping communication with computation. While a GPU is running the forward pass with , it can already be fetching in the background. In the same way, while backpropagating through , the GPUs can be reducing the gradients for at the same time. Because of this overlap, communication doesn't end up dominating the training time.
One note: this "owner broadcasts its weights" picture is the simplified version, and it's the one shown in Figure 1. In PyTorch's actual FSDP implementation, every layer is split across all of the GPUs, and the full weights for a layer are rebuilt with an AllGather right before they're needed. The idea is the same though: each GPU only keeps its own shard, and the full weights only exist for as long as they're being used.

Figure 1: An example of how FSDP splits a model across GPUs. GPU 1 holds W₁ and W₂, while GPU 2 holds W₃ and W₄. Each GPU only computes gradients for its own shard and shares results as needed. The figure also shows a small optimization: the last weight from the forward pass isn't deleted right away, so it doesn't have to be sent again at the start of backpropagation.
Memory Efficiency
With regular data parallelism, every GPU stores the full model, so the memory per GPU is roughly:
With FSDP across GPUs, that footprint gets divided across the cluster, so the memory per GPU drops to:
plus the temporary full copy of whichever layer is currently being computed.
Advantages
The biggest win is memory. By getting rid of the duplicate copies of weights, gradients, and optimizer states, FSDP makes it practical to train models with hundreds of billions or even trillions of parameters, which you simply can't do with standard data parallelism. It also works well with the overlap tricks from earlier, so computation and communication can be pipelined to keep the GPUs busy.
Limitations
The tradeoff is communication. FSDP needs frequent broadcasts and ReduceScatter operations across GPUs, which adds overhead, and it's more complicated to implement and schedule than plain DP. For smaller models that fit comfortably on one GPU, that extra complexity usually isn't worth it, and standard data parallelism will be simpler and faster.
So here's how I'd think about choosing between them: if the model and its optimizer state fit on one GPU, use plain data parallelism. Once they don't, FSDP is usually the next thing to reach for, since it keeps the same training math while letting memory scale with the number of GPUs.