Open Models for LLM and RAG Engineers
Running open models means using LLMs whose weights you can download, such as Llama or Mistral families, on your own hardware or rented GPUs. It involves picking a model size, applying quantisation to reduce memory, and serving it with tools like Ollama, llama.cpp or vLLM instead of a hosted API.
Stage: LLM Foundations · Part of the LLM and RAG Engineer roadmap · Last reviewed 19 November 2026
What to learn in Open Models
- Run a quantised open model locally with Ollama or llama.cpp
- Estimate GPU memory needed for a given parameter count and precision
- Serve an open model behind an OpenAI compatible endpoint with vLLM
- Compare hosted and self-hosted options on cost, privacy and latency
- Check a model licence before using it in a product
Open Models interview questions
What is quantisation and what is the trade-off?
Quantisation stores model weights in fewer bits, such as 8 or 4, instead of 16. This cuts memory use and often speeds up inference, so larger models fit on smaller GPUs. The trade-off is a possible drop in output quality, usually small at moderate settings.
Why might a company self-host an open model?
Reasons include keeping sensitive data inside its own infrastructure, controlling cost at high volume, avoiding vendor limits and customising the model. The trade-offs are operating GPUs, handling scaling and updates, and usually lower capability than the strongest hosted models.
Where Open Models fits in the LLM and RAG Engineer roadmap
Open Models belongs to the LLM Foundations stage (intermediate level), which takes about 33 hours. How models behave.
Other units in this stage
Back to the full LLM and RAG Engineer roadmap