Prompt Attacks for AI Red Teamings
Prompt attacks are inputs crafted to make a language model ignore its instructions or misuse its tools. They include direct and indirect prompt injection, jailbreaks that bypass safety training, and attempts to abuse model output or tool calls to reach other systems.
Stage: Attacks · Part of the AI Red Teaming roadmap · Last reviewed 19 November 2026
What to learn in Prompt Attacks
- Craft a direct prompt injection that overrides a system prompt in a test app
- Hide an indirect injection inside a document the model retrieves
- Try common jailbreak styles such as role play and encoding tricks
- Show how unsanitised model output can lead to script or command injection
- Demonstrate a tool call being triggered by injected text
Prompt Attacks interview questions
What is the difference between a jailbreak and a prompt injection?
A jailbreak tries to get the model to break its own safety rules, usually by the user in the chat. A prompt injection plants instructions that override the application's intent, often through content the model reads from elsewhere.
Why is insecure output handling dangerous?
If an app passes model output straight into a browser, shell or database, an attacker can steer the model to emit script or commands that then run. Treat model output as untrusted input and encode or validate it.
Where Prompt Attacks fits in the AI Red Teaming roadmap
Prompt Attacks belongs to the Attacks stage (advanced level), which takes about 20 hours. Techniques.
Other units in this stage
Back to the full AI Red Teaming roadmap