Skip to main content

Architecture for NLP Engineers

Architecture in this roadmap means the transformer: a network that uses self-attention to let every token weigh every other token in a sequence. Queries, keys and values compute the weights, multiple heads capture different relations, and positional information plus residual connections and layer normalisation make deep stacks trainable.

Stage: Transformers · Part of the NLP Engineer roadmap · Last reviewed 19 November 2026

What to learn in Architecture

Architecture interview questions

Why is attention scaled by the square root of the key dimension?

Dot products grow with vector length, which pushes softmax inputs into regions with tiny gradients. Dividing by the square root of the key dimension keeps the scores in a moderate range, so softmax stays smooth and training remains stable.

What does a causal mask do in a decoder transformer?

It blocks each position from attending to later tokens, usually by setting those attention scores to negative infinity before softmax. This makes training match generation, where the model predicts the next token using only the tokens already produced.

Where Architecture fits in the NLP Engineer roadmap

Architecture belongs to the Transformers stage (advanced level), which takes about 41 hours. The core architecture.

Back to the full NLP Engineer roadmap