Skip to main content

Open Models for LLM and RAG Engineers

Running open models means using LLMs whose weights you can download, such as Llama or Mistral families, on your own hardware or rented GPUs. It involves picking a model size, applying quantisation to reduce memory, and serving it with tools like Ollama, llama.cpp or vLLM instead of a hosted API.

Stage: LLM Foundations · Part of the LLM and RAG Engineer roadmap · Last reviewed 19 November 2026

What to learn in Open Models

Open Models interview questions

What is quantisation and what is the trade-off?

Quantisation stores model weights in fewer bits, such as 8 or 4, instead of 16. This cuts memory use and often speeds up inference, so larger models fit on smaller GPUs. The trade-off is a possible drop in output quality, usually small at moderate settings.

Why might a company self-host an open model?

Reasons include keeping sensitive data inside its own infrastructure, controlling cost at high volume, avoiding vendor limits and customising the model. The trade-offs are operating GPUs, handling scaling and updates, and usually lower capability than the strongest hosted models.

Where Open Models fits in the LLM and RAG Engineer roadmap

Open Models belongs to the LLM Foundations stage (intermediate level), which takes about 33 hours. How models behave.

Back to the full LLM and RAG Engineer roadmap