Distributed Training for Deep Learning Engineers
Distributed training spreads one training job across many GPUs or machines. Data parallelism copies the model and splits batches, tensor and pipeline parallelism split the model itself, and sharding approaches such as FSDP and DeepSpeed ZeRO divide parameters, gradients and optimiser state. Frameworks coordinate this with collective communication.
Stage: Training at Scale · Part of the Deep Learning Engineer roadmap · Last reviewed 19 November 2026
What to learn in Distributed Training
- Run data-parallel training with DistributedDataParallel
- Explain what all-reduce does to gradients across ranks
- Shard parameters and optimiser state with FSDP
- Compare tensor, pipeline and data parallelism trade-offs
- Choose a global batch size and scale the learning rate
- Launch a multi-node job with torchrun
Distributed Training interview questions
What is the difference between DDP and FSDP?
DDP keeps a full model copy on each GPU and averages gradients, so memory per GPU is the whole model. FSDP shards parameters, gradients and optimiser state across GPUs and gathers them when needed, allowing models too large for one GPU.
What does all-reduce do in data parallel training?
Each worker computes gradients on its own batch slice, and all-reduce sums them across workers so everyone ends with the same averaged gradient. This keeps the model replicas identical after each optimiser step.
Where Distributed Training fits in the Deep Learning Engineer roadmap
Distributed Training belongs to the Training at Scale stage (advanced level), which takes about 3 hours. Use many GPUs well.
Other units in this stage
Back to the full Deep Learning Engineer roadmap