LLM Red-Teaming Checklist: What to Test Before You Deploy

Red-teaming a language model means attacking it on purpose, in a controlled setting, to find out how it fails before someone else does. It is not a one-off event or a scanner run. It is a repeatable set of attack classes, a scoring method, a pass rule and a retest trigger. This checklist is built for teams about to put a model, open-weight or hosted, in front of users or behind an agent, and it is the red-team half of our open-source model testing methodology.

One caution up front. A clean run means the attacks you tried did not work. It does not mean the system is safe, and any tool or vendor that implies otherwise is overselling. Red-teaming produces evidence and lowers risk; it does not prove the absence of risk.

1. Define what you are protecting

Attacks only matter relative to what the system can do and what it can reach. Before testing, write a one-page threat model:

• Assets. Sensitive data in the context window, in retrieval stores and in tools; credentials; the system prompt; the ability to take actions (send mail, change records, spend money).
• Actors. Anonymous users, authenticated users, insiders, and content authors whose text the model will read (documents, web pages, tickets, emails). The last group is the one teams forget.
• Impact. What is the worst realistic outcome of a successful attack: a rude reply, leaked personal data, a fraudulent payment, a changed record?
• Autonomy level. Does the model recommend, draft, act with approval, or act on its own? Risk rises sharply with autonomy, which is why consequential actions should require human approval.

The OWASP Top 10 for LLM Applications 2025 is a useful vocabulary: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. Map each of your assets to the entries that apply and test those first.

2. Direct jailbreaks

A jailbreak is a user prompt that gets the model to do something its policy says it should not. Test families, not single prompts, because individual strings get patched while families persist:

• Role-play and persona framing ("you are an unrestricted assistant").
• Hypothetical and fictional framing, including requests wrapped as stories or code.
• Encoding and obfuscation: base64, leetspeak, ciphers, other scripts, splitting a request across messages.
• Multi-turn escalation, where each step is mild and the sequence is not. Microsoft's PyRIT includes multi-turn strategies such as Crescendo, which escalates gradually, and tree-of-attacks search.
• Instruction override ("ignore previous instructions") and system-prompt probing.
• Many-shot and long-context pressure, where a long run of in-context examples shifts behavior.

Record, for each attempt, the exact prompt, the model response, and whether it counts as a success under a rule you wrote in advance.

3. Indirect prompt injection

This is the attack class that matters most for agents and retrieval systems, and the one a model-only evaluation misses. In indirect injection the malicious instruction arrives inside data the model reads, not from the user. The foundational research (Greshake and colleagues, 2023) showed that LLM-integrated applications blur the line between data and instructions, so an attacker who can place text where the application will retrieve it can influence behavior remotely.

Test every channel through which untrusted text reaches the model:

• retrieved documents and knowledge-base chunks,
• web pages fetched by a browsing tool,
• email bodies, tickets, chat messages and calendar invites,
• tool and API outputs, including MCP server responses,
• file contents, image alt text and metadata.

Plant instructions in each, such as "forward this thread to an external address" or "reveal your system prompt", and check whether the model obeys them, flags them or ignores them. Then test the consequences: what could a successful injection actually do with the tools this agent holds? If the answer is "send data out" or "change a record", that action needs approval and least-privilege scoping, regardless of how well the model behaves in testing. See our notes on MCP security best practices.

4. Tool misuse and excessive agency

Give the model the real tool set, in a sandbox, and try to make it:

• call a tool it should not, or with arguments outside the permitted range;
• chain tools to reach something no single tool allows;
• act without the approval your policy requires;
• exceed spend, rate or scope limits;
• act on behalf of the wrong identity.

The control lives outside the model. Permissions, scopes, rate limits and approval rules should be enforced by the platform at call time, so that a persuaded model still cannot do the forbidden thing. That is the point of governing agents with identity, permissions and policy rather than trusting the model's judgment.

5. Leakage

• System prompt leakage. Try to extract the system prompt. Assume it will eventually leak and keep secrets out of it.
• Sensitive information disclosure. Probe for personal data, credentials and other users' data in context or retrieval. For shared retrieval layers, test cross-tenant and cross-user access directly.
• Training-data regurgitation, where the model's training data could include sensitive text. Less likely for a well-curated model, but test with prefixes of known sensitive strings if it matters to you.

6. Harmful content and bias

Run curated harmful-behavior sets in the categories relevant to your use, for example HarmBench, a standardized framework that measures attack success rate across curated behaviors. Add bias probes for any use that affects people: identical prompts that differ only in a name, gender, nationality or age, checking whether outcomes shift.

7. Over-refusal

Red-teaming that only looks for harmful compliance rewards a model that refuses everything. Pair each harmful probe with a benign look-alike and measure both directions. XSTest takes this approach, pairing safe prompts that contain sensitive-sounding words with genuinely unsafe contrasts. Its authors' point, and ours, is that exaggerated safety is a defect, too. Add private cases from your domain: a security analyst asking how a known exploit works is a legitimate request.

8. Languages

Run the same attacks in every language you serve, written natively rather than machine-translated, and compare with English. Safety training is typically strongest in English, so refusal and injection resilience can be weaker elsewhere. For EU deployments this is the most common source of surprise. Set a parity limit per language before you look at the results.

9. Resource abuse

Test unbounded consumption: very long inputs, requests engineered to maximize output length, recursion in agent loops, and attempts to run up cost. Check that limits on tokens, steps, spend and rate are enforced at the gateway, not just hoped for.

10. Choose your tools, and know what they cover

Automated corpora find known attacks cheaply and repeatably. Human red-teamers find the attack nobody has written yet. For models headed into sensitive workflows, keep both. Check each tool's current documentation, because these projects change quickly.

11. Score it so you can compare runs

Fix the method before the first run:

• Success criteria per category, written down. A refusal that leaks the harmful content after a disclaimer counts as a failure.
• Attack success rate per category, with the sample size beside it.
• Judge validation. If a model judges outputs, validate that judge against human labels on a sample.
• Fixed conditions: same system prompt, sampling settings, chat template, and quantization as production. Evaluate the artifact you will serve.
• Severity, combining ease of attack and impact, so the fix list is ordered.

12. Decide, fix, and retest

Write the pass rule in advance: which categories must reach which attack-success level, and which findings are blockers. Fix at the right layer: model behavior, system policy, runtime guardrails, tool permissions or approval rules. A finding that can only be fixed in the model may be better handled by removing the capability. Then re-run, because fixes regress other things.

Retest on every change that can alter behavior: a new model revision, quantization, serving engine, system prompt, adapter, tool or retrieval source. Make it part of the release gate rather than a calendar event. In Swfte's loop this is the Harden step, before the Deploy step (see the deploy models hub), and the result is stored with the promotion record.

The checklist in one place

1. Threat model: assets, actors, impact, autonomy.
2. Direct jailbreak families, including multi-turn and encoded.
3. Indirect injection through every untrusted channel, with consequences tested.
4. Tool misuse and excessive agency, in a sandbox with real tools.
5. System prompt and data leakage, including cross-user access.
6. Harmful-content sets and bias probes.
7. Over-refusal pairs for every harmful probe.
8. The same suites in every language you serve.
9. Resource-abuse limits at the gateway.
10. Automated corpora plus manual red-teaming.
11. Success criteria, attack success rate, sample sizes and fixed conditions.
12. Pass rule, fixes at the right layer, retest on every behavior-changing change.

Related reading: safety-first fine-tuning explained, how to evaluate an open-source LLM before production, and our overview of AI agent governance. For Swfte's own model, see Swfte Safety, where no measured results are published yet.