What is a red team exercise?

A red team exercise is a planned attempt by a group to make a system fail, acting as an attacker would. For AI agents, the red team tries to push the agent beyond its bounds: getting it to act on a forged instruction, reveal data it should not, or take an action its role does not allow.

Why does a red team exercise matter?

An agent that passes its tests on ordinary cases may still behave badly when someone tries to mislead it. A supplier email with hidden instructions, a customer who claims to be a director, a document that looks sealed but is not: these are the cases an agent meets in the wild. A red team finds them before a fraudster does.

The exercise also trains people. Reviewers who take part learn what a manipulation attempt looks like on their own queue. After a red team has shown that an emailed invoice can carry a sentence telling the agent to change a bank account, nobody on that team treats an external document casually again.

How does it work in practice?

A red team exercise has a clear scope and a written plan. The usual stages:

1. Set the target: one agent, its capabilities and its depth, in a sandbox or a certified test environment.
2. List the attacks: prompt injection through documents, impersonation, requests just outside role scope, attempts to reach withheld actions, data extraction.
3. Run them: the red team tries each one and records what the agent did and which verdict it returned.
4. Score: for each attempt, did the agent refuse, hold, or comply? Did the verdict name the right bound?
5. Fix and retest: tighten the brief, sources or limits, then rerun.

The red-team test plan template gives a structure, and the guide on how to red-team an agent before go-live covers the full method.

An example

An accounts payable team prepares an agent that drafts supplier payments for approval at depth 2. The red team sends twelve test invoices. One contains a line of text addressed to the agent asking it to update the supplier's bank details. The agent flags the invoice as external, refuses the change because bank details sit outside its role scope, and records the refusal. Another invoice exceeds the customer's limit and is held for a named approver. Both outcomes are correct. A third, a duplicate with a changed reference, slips through to a draft; the team adds a duplicate check before go-live.

How is it different from a tabletop exercise?

A tabletop exercise is a discussion. People sit around a scenario, such as "the agent sent fifty wrong letters overnight", and talk through who does what. Nothing is actually attacked. A red team exercise acts: it sends real inputs to a real agent and records what happened.

The two answer different questions. A tabletop tests the people and the response plan. A red team tests the agent and its bounds, and teaches the people who watch it.

Which terms are related?

• Lab exercise: a guided practice task in a safe workspace, without an adversary.
• Red line: an action no reviewer may approve; red teams often target these. See what reviewers need to know about red lines.
• Prompt injection: hidden instructions in content the agent reads.

For how this fits into training, read training teams for AI agents.