Neural Foundations for NLP Engineers
Neural Foundations are the deep learning building blocks used before transformers took over language modelling. They include feed-forward networks, backpropagation, recurrent networks such as LSTMs and GRUs, and sequence-to-sequence models with an encoder and decoder. These ideas explain why attention was invented.
Stage: Language Basics · Part of the NLP Engineer roadmap · Last reviewed 19 November 2026
What to learn in Neural Foundations
- Write a forward and backward pass for a small network in PyTorch
- Train a text classifier on top of averaged word embeddings
- Explain the vanishing gradient problem in recurrent networks
- Build an LSTM language model that predicts the next token
- Describe how an encoder-decoder passes information through a single context vector
Neural Foundations interview questions
Why do vanilla RNNs struggle with long sequences?
Gradients are multiplied through many time steps, so they shrink towards zero or blow up. The network then cannot learn dependencies far apart in the text. LSTMs and GRUs add gates that let information and gradients flow more steadily.
What is the bottleneck in a basic seq2seq model?
The encoder squeezes the whole input sentence into one fixed-size vector, and the decoder must produce everything from it. Long inputs lose detail. Attention fixed this by letting the decoder look back at every encoder state.
Where Neural Foundations fits in the NLP Engineer roadmap
Neural Foundations belongs to the Language Basics stage (intermediate level), which takes about 41 hours. Text as data.
Other units in this stage
Back to the full NLP Engineer roadmap