Architecture for NLP Engineers
Architecture in this roadmap means the transformer: a network that uses self-attention to let every token weigh every other token in a sequence. Queries, keys and values compute the weights, multiple heads capture different relations, and positional information plus residual connections and layer normalisation make deep stacks trainable.
Stage: Transformers · Part of the NLP Engineer roadmap · Last reviewed 19 November 2026
What to learn in Architecture
- Compute scaled dot-product attention by hand for a short sequence
- Explain what multi-head attention adds over a single head
- Add positional encodings so the model knows token order
- Implement a causal mask for decoder-style language models
- Tell encoder-only, decoder-only and encoder-decoder transformers apart by use case
Architecture interview questions
Why is attention scaled by the square root of the key dimension?
Dot products grow with vector length, which pushes softmax inputs into regions with tiny gradients. Dividing by the square root of the key dimension keeps the scores in a moderate range, so softmax stays smooth and training remains stable.
What does a causal mask do in a decoder transformer?
It blocks each position from attending to later tokens, usually by setting those attention scores to negative infinity before softmax. This makes training match generation, where the model predicts the next token using only the tokens already produced.
Where Architecture fits in the NLP Engineer roadmap
Architecture belongs to the Transformers stage (advanced level), which takes about 41 hours. The core architecture.
Other units in this stage
Back to the full NLP Engineer roadmap