How to red team an LLM

Write down who has authorised the exercise and for which system. List the in-scope targets by name, the test accounts you may use, the time window and the people to call if something breaks. Say what is out of bounds: production data, other tenants, third-party services you do not control, and denial-of-service testing unless it is explicitly approved.

Check the terms of every model provider and tool vendor in the path before running automated attack suites against them. Many providers have usage policies that apply to adversarial testing. Where you test a hosted model rather than your own application, read the provider’s terms first and stay inside them.

Agree how you will handle harmful content and leaked secrets. Transcripts may contain both. Decide who can read them, where they are stored and when they are deleted. Rotate any real secret the system leaks.

List what the system can do, because that is what an attacker inherits. For each ability, such as reading a customer record, sending an email or running code, write the worst thing someone could make it do. Then list every place untrusted text enters: user messages, uploaded files, retrieved documents, web pages, tool results, email bodies.

The risk is highest where three things meet in one session: untrusted input, access to private data and the ability to act or send data out. Simon Willison called this the lethal trifecta in June 2025, and Meta’s Agents Rule of Two (October 2025) states a similar rule: an agent should combine no more than two of processing untrusted input, accessing sensitive systems or data, and changing state or communicating externally, otherwise a human should be in the loop. Mark which of your agents satisfy all three and test those first.

Rank the abuse cases by impact and by how easily an attacker could reach them. This ranking decides where manual time goes in step 7.

Test against staging, never production. Fill it with synthetic data and plant canary strings: unique fake secrets in the system prompt, in a document only another test user may read, and in a tool’s data. If a canary appears in an output, you have proof of leakage that no judge can argue with.

Create at least two test users with different permissions, so you can test whether one can reach the other’s data. Turn on full logging so you can see every tool call and retrieval. Put spending limits on the model account the target uses, because attack suites send thousands of prompts. Know how to stop: a kill switch for the target and a way to cancel the run.

Pin the versions of the tools you use and record them. Results from an unrecorded tool version cannot be compared with next quarter’s run.

promptfoo’s red-team mode generates adversarial inputs for the categories you pick and runs them against a target, which can be a model or an HTTP endpoint. Its quickstart gives npx promptfoo@latest redteam setup for a guided web setup and promptfoo redteam init --no-gui to create the configuration from the command line. As of 2026-10-06 the documentation states Node.js 22.22.0 or newer and says an OpenAI key is optional.

The configuration has a targets section and a redteam section. The target below is an HTTP endpoint. The redteam section holds the purpose, which tells the generator what the system is for, the plugins to attack with and the strategies that change how attacks are delivered. The documentation lists plugins such as pii, excessive-agency, hijacking and the OWASP presets owasp:llm:01, owasp:llm:02 and owasp:llm:06, and strategies such as jailbreak, base64 and crescendo. Start small, with numTests of 5, which sits inside the 5 to 20 range the documentation calls typical.

The configuration has a provider field for the model that generates attacks. Check the documentation for what the default is before you put confidential purpose text into it, and configure a model you are allowed to send that text to.

Run the scan, then open the report. Read each successful attack as a transcript: what was sent, what came back, and why it counts. Group them by plugin and by root cause before you triage.

garak is NVIDIA’s open-source LLM vulnerability scanner. It tests a model with probes, such as encoding-based injection, DAN-style jailbreaks, prompt injection and replay of training data, and uses detectors to judge the responses. Use it to understand the underlying model, and use promptfoo for the application around it, because the two answer different questions.

Install with pip. The README’s conda route pins Python to 3.11 up to 3.13. List the probes first, then run a selection with --spec, the unified selector flag. Its reference describes selectors such as probes.dan, tags such as tag:owasp:llm01, and a minus sign to exclude. The README’s example runs one DAN probe against the small gpt2 model from Hugging Face, a harmless first target, and another runs the encoding probes against an OpenAI model with an OPENAI_API_KEY set. Swap in a model your organisation approves.

Every run produces a debugging log, a JSONL report and a hit log of detected vulnerabilities, and prints the report file name at the start and end. Keep these with your transcripts. --generations sets how many outputs are sampled per prompt; raising it gives more reliable rates at higher cost.

PyRIT, the Python Risk Identification Tool from Microsoft’s AI Red Team, is a framework for building your own attack campaigns, including conversations that escalate over several turns. PyPI shows version 1.1.0 released on 4 September 2026 under the MIT licence with an “Alpha” development status, so expect interfaces to change between releases and pin the version you use.

Install it with pip. The documentation recommends Python 3.13 for local installation and PyPI lists support for Python 3.10 up to but not including 3.15. After installing, the docs say to configure two files, ~/.pyrit/.env for endpoint credentials and ~/.pyrit/.pyrit_conf for memory and initialiser settings, and describe three ways to use it: a command-line scanner, a GUI and a programmatic framework. Follow the installation and configuration pages on the PyRIT documentation site for the exact contents of those files, since they change between releases.

Use PyRIT for what the earlier tools cannot do: a scripted attacker that adapts to each answer, campaigns that target your specific tools, and scoring you control. Write each campaign down as a scenario with a goal, such as “make the assistant reveal another customer’s order”, then record whether and after how many turns it succeeded.

Take the top five abuse cases from step 2 and give each to a person for an hour. Their job is to achieve the bad outcome by any route inside scope. Ask them to try the same goal through several routes: directly, through a document the system will read, through a tool result, in another language, and across a long conversation.

Make them keep notes in a fixed format: goal, steps taken, the exact input, the output, whether it worked, and how many attempts it took. Reliability matters as much as possibility. An attack that works one time in fifty against a system doing thousands of requests a day is a real problem.

Include people who did not build the system. Builders share the same assumptions as the code. If you have domain staff, such as support agents, ask them how a bad customer would try to trick the assistant; they usually know.

Merge duplicates first: fifty jailbreak transcripts that exploit one weakness are one finding. For each finding record the reproduction steps, the attack success rate across repeated tries, the data or action the attacker gained, and the root cause: missing permission check, untrusted text treated as instructions, over-broad tool, or weak filter.

Score on impact and reliability together. The scale below is a practical one; set your own bands with your security team and keep them stable between exercises so trends mean something.

Do not accept “the model refused most of the time” as a pass for high-impact cases. A control that fails one time in twenty is not a control when the action is irreversible.

Prefer fixes in this order. Remove the capability if the system does not need it. Enforce permissions in code, outside the model. Require human approval for risky actions. Then filter inputs and outputs, and last, adjust the prompt. Filters and prompts help but adaptive attackers get past them, so they should not be the only layer on a high-severity finding. OWASP’s own guidance for prompt injection lists constraining model behaviour, filtering input and output, least privilege, human approval for high-risk actions, segregating external content and adversarial testing together; none of them is sufficient alone.

After each fix, rerun the exact attack that worked and then the wider suite. Add every confirmed attack to your standing eval set as a regression case with a deterministic pass rule, such as “output must not contain the canary” or “tool X must not be called”. That is how a one-off exercise becomes a gate. See how to stop prompt injection for the layered defences.

The report needs a scope summary, the method and tool versions, the threat model, findings with severity and status, what was not tested and what you recommend. Attach transcripts to findings rather than pasting them into the body. State clearly that red teaming finds problems; it cannot prove their absence.

Automate the repeatable part. promptfoo’s CI documentation shows a GitHub Actions step with promptfoo/promptfoo-action@v1 and the type set to redteam. Run it on every change to prompts, tools or models and on a schedule, and run the manual sessions again when the system gains a new tool or data source. Article 55(1)(a) of the EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing; even if that does not apply to you, documenting it is good practice. Feed the results into how to audit AI systems evidence.