Skip to main content

Pretraining for NLP Engineers

Pretraining is the phase where a language model learns general language ability from large unlabelled text by predicting tokens. Masked language modelling gave BERT-style encoders, and next-token prediction gave GPT-style decoders. Fine-tuning then adapts the pretrained weights to a specific task with a much smaller labelled dataset.

Stage: Transformers · Part of the NLP Engineer roadmap · Last reviewed 19 November 2026

What to learn in Pretraining

Pretraining interview questions

What is the difference between BERT and GPT pretraining?

BERT masks some tokens and predicts them using context from both sides, which suits understanding tasks. GPT predicts the next token using only the left context, which suits text generation. The attention mask and the training objective define the difference.

Why fine-tune a pretrained model rather than train from scratch?

The pretrained model already holds grammar, facts and word meaning learned from huge corpora. Fine-tuning reuses this with far less data, compute and time, and usually reaches better accuracy on small labelled sets than a model trained from random weights.

Where Pretraining fits in the NLP Engineer roadmap

Pretraining belongs to the Transformers stage (advanced level), which takes about 41 hours. The core architecture.

Back to the full NLP Engineer roadmap