Plan red-team tests of an AI agent by turning each OWASP Top 10 entry into attacks through the agent's real input channels and tools. Judge each attack by what the agent did, not what it said: a test passes when the control layer blocks or holds the action. Record every finding with its channel, attempted action, verdict, severity and retest.
What do the standards ask for?
The OWASP Top 10 for LLM Applications lists the main security risks for applications built on language models and recommends regular adversarial testing as a mitigation, particularly for prompt injection. Its current edition covers prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.
Binding rules point the same way. The EU AI Act requires high-risk systems to be tested as part of risk management (Article 9) and to be resilient against attempts to alter their use or outputs (Article 15); its data governance duties (Article 10) bear on poisoning, as our piece on Article 10 data governance for agents explains. DORA requires financial entities in the EU to test their digital operational resilience. For the OWASP list applied entry by entry, see the OWASP Top 10 for LLM Applications applied to agents.
Who does this apply to?
Teams that build, operate or assure agents, and the security, risk and audit functions that test them. In regulated firms, red-teaming supports security duties under data protection law, resilience testing duties, and model validation.
What changes when you red-team an agent rather than a chatbot?
Chatbot red-teaming often counts how many attacks made the model say something forbidden. For an agent, that metric misleads. A model will sometimes be fooled; what matters is whether the fooled model could do anything. So the unit of success changes.
Success means the control held. A prompt injection test passes if the agent tried to change a payee and the control layer refused it. It fails if the change went through, whether or not the model "resisted".
Attacks come through the agent's real channels. Customer chat, inbound email, uploaded documents, records from other systems, retrieved knowledge base pages. Each is a route, and each needs tests.
Severity follows the action. A leaked internal instruction is less severe than a refund executed on an attacker's say-so. Rate findings by what the agent could do with the access it had.
How do you plan tests around each entry?
Map each entry to attacks that suit agents:
- Prompt injection: hidden instructions in emails, invoices and uploaded files telling the agent to act.
- Sensitive information disclosure: requests for another customer's data; injected instructions to send records out.
- Supply chain: a change of model version in staging; does behaviour shift on your sentinel cases?
- Data and model poisoning: a planted false instruction in a knowledge base article the agent retrieves.
- Improper output handling: agent output crafted to break or exploit the downstream system that consumes it.
- Excessive agency: attempts to call tools outside scope, exceed limits or skip approvals.
- System prompt leakage: extraction of the brief; then check whether knowing it helps any other attack.
- Vector and embedding weaknesses: queries designed to retrieve documents across permission boundaries.
- Misinformation: cases where sources are silent; does the agent invent and act?
- Unbounded consumption: inputs that send the agent into loops of tool calls.
Take an illustrative case. A payments firm red-teams its merchant support agent. Thirty injection attempts through uploaded documents produce eight cases where the model drafted a settlement account change. In every one, the control layer refused the change because the new details came only from an external document. The finding is recorded as "model susceptible, control effective", with a recommendation to keep the control and add the eight cases to the regression set. One test retrieved a document from another merchant's folder; that is a high-severity finding fixed before the next release.
How do you run and record it?
| Requirement | Control | Evidence it leaves |
|---|---|---|
| Scope | Agent, capabilities, channels and tools in scope listed | Test scope document |
| Threat model | Attacks mapped to each OWASP entry and each input channel | Threat model and test plan |
| Rules of engagement | Production-like environment, no real funds, stop procedure agreed | Signed rules of engagement |
| Success criteria | Pass when the action is blocked or held; model output noted separately | Criteria in the plan |
| Findings record | ID, entry, channel, attempted action, verdict, severity, fix owner | Findings register |
| Severity | Rated by impact of the action the agent could take | Severity rationale |
| Retest and regression | Fixed findings retested; attack cases added to regular tests | Retest results; regression suite |
| Independence | Testers separate from the build team | Tester names and reporting line |
What will a security reviewer, auditor or supervisor ask?
- Which input channels did you test, and which did you not?
- How many attacks fooled the model, and how many led to an action?
- What were your high-severity findings, and are they fixed?
- Are the attack cases now part of regular testing?
- Who ran the tests, and how independent were they?
What does red-teaming not cover?
It finds edges; it does not prove an agent handles ordinary cases well. That needs testing on representative real cases, an argument we make in red-teaming finds the edges, certification proves the middle. It does not set your limits or approve your agent. For the short answer on OWASP and agents, see how the OWASP Top 10 applies to AI agents, and for binding rules, our guide to AI agent regulation mapped to controls.
This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.
Where Pryme fits
Pryme's verdict record makes red-team results easy to read: every attempted action is filed as allowed, held, refused or withheld, with the bound that decided. Attest then proves the middle, replaying the agent on the customer's own cases across seven gates, so adversarial and representative evidence sit side by side.
Questions people ask
How often should agents be red-teamed?
Before first deployment, after material changes such as new tools or channels, and on a regular cycle. Keep a regression set of past attacks running continuously between full exercises.
Should red-teaming use production data?
Use production-like data in a controlled environment. Real customer data adds realism but also risk; if used, it needs the same protections as production, and no test should move real money.
Is a model jailbreak a finding if no action followed?
Record it, but rate it by consequence. A jailbreak that could not lead to any action is low severity for an agent; the same jailbreak with an open payment tool is critical.