Training Craft for Deep Learning Engineers
Training craft is the set of techniques that make optimisation stable and models generalise: optimisers such as SGD with momentum and Adam, learning-rate schedules, careful weight initialisation, normalisation layers like batch norm and layer norm, and regularisation such as weight decay and dropout. Most training failures are fixed here.
Stage: Neural Network Fundamentals · Part of the Deep Learning Engineer roadmap · Last reviewed 19 November 2026
What to learn in Training Craft
- Compare SGD with momentum, Adam and AdamW and when to use each
- Set up a learning-rate warmup and cosine decay schedule
- Initialise weights with Xavier or He schemes for the activation used
- Explain what batch norm and layer norm normalise
- Apply dropout, weight decay and early stopping against overfitting
- Diagnose a loss that diverges or stalls
Training Craft interview questions
What is the difference between Adam and AdamW?
Adam adds L2 regularisation into the gradient, which then gets scaled by the adaptive terms. AdamW applies weight decay directly to the weights, decoupled from the gradient update, which regularises more predictably and is the usual choice for transformers.
Why do we use batch normalisation?
Batch norm normalises activations using batch statistics and then rescales them with learned parameters. This stabilises training, permits larger learning rates and reduces sensitivity to initialisation. It behaves differently at inference, where running averages are used.
Where Training Craft fits in the Deep Learning Engineer roadmap
Training Craft belongs to the Neural Network Fundamentals stage (advanced level), which takes about 4 hours. Build and train from first principles.
Other units in this stage
Back to the full Deep Learning Engineer roadmap