Give an independent team a threat model, a sandbox with simulated connections, and a brief to make the agent exceed its bounds. They try direct manipulation, poisoned documents, pressure over many turns and attempts to act above the agent's depth. Every attempt is logged with the platform's verdict. Each successful attack is fixed and becomes a permanent certification case.
What is red-teaming an agent, and who owns it?
Red-teaming means deliberately attacking a system to find how it fails before someone else does. For an agent that acts, the question is not only "can we make it say something wrong?" but "can we make it do something it should not?": send a payment, reveal another customer's data, approve its own draft, follow an instruction hidden in an email.
The capability owner commissions the exercise and decides whether the agent is ready. The red team should be independent of the people who built the brief. Domain experts belong on it: a fraud analyst knows how real scammers talk to call centres in a way a general security tester may not. As the argument goes, red-teaming finds the edges; certification proves the middle. You need both.
What is the working method?
- Person, owner: sets the scope. The capabilities, the depth each is seeking, the systems it touches.
- Person, red team lead: writes the threat model. For each capability: who might misuse it, what they would want, and how. A customer seeking an exception, a fraudster posing as a customer, an insider, a supplier sending a crafted document. The OWASP Top 10 for LLM Applications is a useful list of categories, including prompt injection, excessive agency and sensitive information disclosure.
- Person: approves the plan. Using a structure such as the red-team test plan template for AI agents.
- Platform: provides the sandbox. The candidate agent with its real brief and bounds, connected to simulated systems. Nothing it does has a live effect.
- Red team: attacks. Direct requests to exceed scope, instructions hidden in documents and messages, pressure across many turns, conflicting sources, attempts to draw out other customers' data, attempts to act above the certified depth.
- Platform: logs every attempt. Input, the agent's response, the action it tried and the verdict the platform returned.
- Person, red team lead: triages findings. By severity, and by which layer held or failed.
- Person, owner: decides fixes. Changes to the brief, sources or bounds. Each successful attack is added to the certification set.
- Person, owner: decides readiness. After fixes and a certification run, the owner signs the red-team result as part of go-live.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Scope | Nothing | None | Setting scope |
| Threat model | Nothing | None | Writing it |
| Plan | Nothing | None | Approving it |
| Sandbox | Runs with simulated connections | Sandbox only | None |
| Attack | Responds to attacks | Sandbox only | Designing attacks |
| Log | Nothing | Sandbox only | None |
| Triage | Nothing | None | Severity and cause |
| Fix | Nothing | None | Choosing fixes |
| Readiness | Nothing | None | Go-live decision |
How do you read the findings?
The useful distinction is between what the agent said and what the platform allowed. An agent persuaded to agree to send a payment, where the platform then refused the payment because the agent has no such capability, has shown a weakness in its brief. The bounds held. An agent that found a route to an action the platform allowed, when it should not have, has shown a weakness in the bounds. The second is far more serious.
Record both, and fix both. But rank findings by what actually got through, not by how alarming the transcript reads.
What decisions does a named person keep, and what record does it produce?
The owner keeps four: the scope, the approval of the threat model, the choice of fixes, and the go-live decision. Each is filed with their name.
The record is a set of documents that belong together: the approved plan, the full attempt log with verdicts, the findings with severity and cause, the fixes made, and the list of cases added to the certification set. The AI agent go-live approval checklist should require all of them.
What should you measure?
- Attempts that reached an allowed action they should not have. The headline measure. Should be none at go-live.
- Attempts where the agent's response was wrong but the platform refused the action. Shows how much is resting on the bounds alone.
- Findings fixed and converted into certification cases. Should be all of them.
- Coverage of the threat model. Each threat with at least one attempt.
Where does it usually go wrong?
Testing the model, not the agent. Attacks aimed at making the model say something offensive miss the point. The agent's risk is in its connections and actions.
A sandbox that is not like production. Simulated connections that are more permissive or more restrictive than the real ones give a false result. The bounds in the sandbox must match the bounds that will run live.
Builders as attackers. The people who wrote the brief know what they guarded against and unconsciously avoid what they did not.
One round, then never again. New attacks appear after go-live. Successful attacks from real operation should also feed the set.
Red-teaming sits alongside other pre-launch work, covered in how do you test an AI agent before it goes live?, and often comes before shadowing an agent on live inputs. The wider framework is in what it means to certify an agent.
Questions people ask
How long should a red-team exercise take?
Long enough to cover every threat in the model with several approaches each. For a depth 4 capability, that usually means more rounds and more domain experts than for a drafting capability.
Can another agent do the red-teaming?
An attacking agent can generate many variations quickly and is a useful addition. Domain experts should still design the threat model and review what it finds.
Should red-team transcripts be kept?
Yes, in full. They are evidence for go-live and source material for future certification cases. Store them under the same controls as other test data.