There is no single figure. An AI support agent must clear a floor you set for each capability, tested on your own past conversations; the floor rises with what the agent may do. An agent that only answers questions needs a high bar for grounded answers. One that issues refunds needs a higher one, plus near-perfect refusals and escalations.
Why is one accuracy number the wrong target?
Because "accuracy" hides several different things, and the ones that hurt customers are not the ones vendors usually quote.
An agent can answer most questions correctly and still be unfit to go live, if the few it gets wrong are refund promises or missed fraud reports. Another can answer fewer questions but hand over every one it cannot handle, and be perfectly safe. A single score averages these together. What you need is a set of separate floors, each of which must clear on its own:
- Grounded: every factual statement traces to an approved source.
- Refusal: the agent declines what is outside its bounds, every time it should.
- Escalation: it hands over the cases that need a person, including distress and complaints.
- Scope: it stays inside its role and does not drift into advice or sales.
- Adversarial: it holds its bounds when a customer pushes, and does not rush someone in difficulty.
A miss on refusal or escalation is worse than a miss on a routine answer, so those floors should sit higher.
What drives where you set each floor?
Depth. An agent at depth 1 (read) or depth 2 (draft, a person sends) has a person between its output and the customer for anything that matters. An agent at depth 3 acts on its own within bounds. The bar for acting alone is higher than the bar for drafting.
The cost of a miss. A wrong delivery estimate costs an apology. A wrong statement about a claim decision can cost a complaint and a regulator's interest. Set floors by consequence, capability by capability.
Your current baseline. Score a sample of your own team's past replies against the same tests. If the agent must beat your team, you need to know where your team stands. It is often lower than people assume, and that is useful information too.
How do you plan the test?
| Step | What happens | What it produces |
|---|---|---|
| Build the case set | Pull real past conversations per contact reason, including hard and adversarial ones | A case set that reflects your actual mix |
| Write the expected outcome | A person records what a good answer, refusal or handover looks like for each | The marking key |
| Replay | Run the agent against every case | Scored answers per test |
| Review misses | A person reads every failure and classifies it | A list of fixes to sources, brief or bounds |
| Shadow | Run the agent on live traffic without it replying | Evidence on live conversations, not archived ones |
| Go live by capability | Each capability goes live only when all its floors clear | A dated decision per capability |
The full test method is in how to test an AI support agent before launch, and shadowing in how to shadow an AI agent before it goes live.
An illustrative example
Take an online retailer with three contact reasons it wants to automate: order status, returns and refunds.
It sets the floor for grounded answers on order status at 196 out of 200 test cases, because a wrong answer is easy to correct. For refunds, where the agent would act, it sets the floor at 199 out of 200 for grounded answers and requires every refusal and every escalation case to pass. The agent clears order status and returns in the first replay. On refunds it misses two escalation cases where customers mentioned a bereavement. The retailer fixes the trigger list, replays, and still keeps refunds at draft depth until the next cycle shows a clean pass.
Order status goes live. Refunds wait. That split is the point: accuracy is decided capability by capability, not agent by agent.
What happens after go-live?
Accuracy before launch is a snapshot. Policies change, products change, customers find new phrasings. Keep replaying the case set on a schedule and after every policy change, and treat a fall below the floor as a reason to reduce what the agent may do the same day. The general principle is set out in how to test an AI agent before it goes live.
Where Pryme fits
Attest replays the agent against your own cases and scores every answer against a floor on seven gates: grounded, refuses, depth, escalates, scope, adversarial and stable. Each must clear on its own. Certificates lapse after 90 days, and a run that clears no depth drops the agent to depth 1 the same day. More is in the AI support agent that knows when to say no.
Questions people ask
Should we compare the agent to our human team?
Yes, using the same tests. It tells you whether the floor is realistic and where your team's own answers need work.
How many test cases are enough?
Enough to cover every contact reason and every refusal and escalation trigger more than once. A few hundred well-chosen cases beat thousands of near-duplicates.
Can we go live with a lower floor and monitor closely?
Only at draft depth, where a person sends every reply. Monitoring after the fact does not undo what an agent already did.