Skip to main content

AI Red Teaming Roadmap 2026

Test AI systems adversarially: threat models, prompt injection, jailbreaks, data and model attacks, agents and reporting.

4 stages · 8 units · Advanced · about 60 hours · Last reviewed 19 November 2026

What is a ai red teaming?

AI red teaming is the practice of attacking AI systems on purpose, with permission, to find weaknesses before real adversaries do. Testers probe language models, retrieval pipelines and agents for prompt injection, jailbreaks, data leakage and tool abuse, then document findings and help teams fix them.

Step-by-step ai red teaming roadmap

  1. 1

    Foundations

    Threat landscape.

    • Concepts

      AI security concepts describe what makes AI systems attackable and how to reason about it. Threat modelling maps assets, entry points, attackers and impact for a model-backed system, so testing focuses on realistic risks such as untrusted input reaching a model that holds powerful tools.

    • Rules of Engagement

      Rules of engagement are the written agreement that defines what a red team may test, how, when and with what data. They set scope, authorisation, off-limits systems, reporting channels and ethical limits, protecting both the tester and the organisation from harm or legal trouble.

  2. 2

    Attacks

    Techniques.

    • Prompt Attacks

      Prompt attacks are inputs crafted to make a language model ignore its instructions or misuse its tools. They include direct and indirect prompt injection, jailbreaks that bypass safety training, and attempts to abuse model output or tool calls to reach other systems.

    • Data and Model

      Data and model risks are attacks aimed at what feeds and defines an AI system: poisoned training or retrieval data, leakage of private information, model extraction and weaknesses in RAG and agent pipelines. Red teamers test how data flows and what permissions the system holds.

  3. 3

    Testing Practice

    Tools and automation.

    • Tooling

      Red teaming tooling covers the open-source scanners and frameworks used to probe models, plus the manual methods that find what tools miss. Automation runs large sets of attack prompts and scores replies, while manual testing explores creative, multi-turn and context-specific attacks.

    • Evaluation

      Measuring risk in AI red teaming means turning attack results into numbers and judgements: attack success rate, severity, and coverage across risk categories. Graders, human review and fixed test sets let teams compare versions and decide whether a system is safe enough to release.

  4. 4

    Defence and Reporting

    Close the loop.

    • Defences

      Defences are the mitigations that reduce AI risk after weaknesses are found. They layer input and output filtering, hardened prompts, least-privilege tools, isolation of untrusted content, monitoring and human approval, since no single control stops every attack on a language model.

    • Reporting

      AI red team reporting is the write-up that turns test activity into action: clear findings with reproduction steps, evidence, severity and recommended fixes. It also feeds governance, helping organisations track AI risks, assign owners and show due diligence to leadership and auditors.

Frequently asked questions

Ready to practise for interviews?

Practise ai red teaming questions with an AI mock interviewer and get feedback on your answers.

More roadmaps