Data Pipelines for Deep Learning Engineers
Data pipelines for deep learning move very large datasets to the GPU fast enough to keep it busy. This involves sharded file formats, streaming from object storage, parallel decoding and prefetching, shuffling at scale and caching. A slow pipeline leaves expensive accelerators idle, so it is often the first bottleneck to fix.
Stage: Training at Scale · Part of the Deep Learning Engineer roadmap · Last reviewed 19 November 2026
What to learn in Data Pipelines
- Store a dataset as shards in a format such as WebDataset or Parquet
- Stream data from object storage without copying it all locally
- Tune num_workers, prefetching and pinned memory
- Shuffle across shards while keeping epochs reproducible
- Resume a data stream correctly after a crash
- Move heavy augmentation to the GPU
Data Pipelines interview questions
How can you tell the data loader is the bottleneck?
GPU utilisation drops in periodic dips and profiling shows the training step waiting on the next batch. Increasing num_workers or prefetch improves throughput. Compare step time with synthetic in-memory data to confirm the pipeline is the limit.
Why store data in shards instead of millions of small files?
Small files cause heavy metadata and random-read overhead on disks and object stores. Larger shards read sequentially, stream well, distribute evenly across workers and are easier to version and move.
Where Data Pipelines fits in the Deep Learning Engineer roadmap
Data Pipelines belongs to the Training at Scale stage (advanced level), which takes about 3 hours. Use many GPUs well.
Other units in this stage
Back to the full Deep Learning Engineer roadmap