Skip to main content

Training Craft for Deep Learning Engineers

Training craft is the set of techniques that make optimisation stable and models generalise: optimisers such as SGD with momentum and Adam, learning-rate schedules, careful weight initialisation, normalisation layers like batch norm and layer norm, and regularisation such as weight decay and dropout. Most training failures are fixed here.

Stage: Neural Network Fundamentals · Part of the Deep Learning Engineer roadmap · Last reviewed 19 November 2026

What to learn in Training Craft

Training Craft interview questions

What is the difference between Adam and AdamW?

Adam adds L2 regularisation into the gradient, which then gets scaled by the adaptive terms. AdamW applies weight decay directly to the weights, decoupled from the gradient update, which regularises more predictably and is the usual choice for transformers.

Why do we use batch normalisation?

Batch norm normalises activations using batch statistics and then rescales them with learned parameters. This stabilises training, permits larger learning rates and reduces sensitivity to initialisation. It behaves differently at inference, where running averages are used.

Where Training Craft fits in the Deep Learning Engineer roadmap

Training Craft belongs to the Neural Network Fundamentals stage (advanced level), which takes about 4 hours. Build and train from first principles.

Back to the full Deep Learning Engineer roadmap