Measure it per capability, on real cases from your own operation, against answers a named person agreed before the agent ran. Accuracy is not one number. A correct answer, an answer grounded in the right source and a correct refusal are separate tests, and each needs its own floor, its own sample and its own filed record.
What does accuracy mean for an agent that acts?
For an agent that drafts, files or pays, "the answer was right" is too thin a test. An agent can reach the right outcome from the wrong source, or produce a fluent answer to a case it should have refused. Both pass a simple accuracy check and both will cost you later.
So split the measure into parts that match what the agent does. Was the output correct? Was every fact traceable to a permitted source? Did it refuse or escalate when the case was outside its role? Did it stay inside the capability it was certified for? A single headline score hides which of these is weak.
Which steps measure accuracy, and who does each?
- Process owner: defines correct. For each capability, write what a right answer looks like and which cases must be refused or escalated. Evidence: a signed definition, versioned.
- Operations analyst: builds the case set. Pull a few hundred past cases from your own records, including the awkward ones: missing documents, conflicting data, edge amounts. Evidence: the case list with its source and date range.
- Subject expert: sets the expected result for each case before the agent sees it. Evidence: an answer key that cannot be edited after the run.
- Agent, depth 1: runs the cases. It reads and proposes; nothing is sent or executed. Evidence: every proposal filed with the sources it used.
- Reviewer: scores disagreements. Where the agent and the key differ, a person decides whether the agent or the key was wrong. Evidence: each disputed case with a reason.
- Process owner: compares the scores to the floor and decides. Evidence: the decision and who made it.
Order matters: an answer key written after people see the agent's output drifts towards agreeing with it.
What does the agent do and what stays with a person?
| Part of the measurement | The agent | A person |
|---|---|---|
| Defining what counts as correct | No part | Process owner |
| Choosing the cases | No part | Analyst, with the owner |
| Producing answers to score | Yes, at depth 1 | No |
| Deciding disputed cases | No part | Named reviewer |
| Setting and applying the floor | No part | Process owner |
The agent is the thing being measured. It should never grade itself, and its own confidence score is not evidence of accuracy.
What does this look like for one agent?
Take a lender's servicing team testing an agent that answers questions about payoff amounts. The owner defines correct as the figure from the loan ledger on the requested date, and says any request involving a disputed fee must be escalated. The analyst pulls 400 past requests, 40 of them involving disputes.
The agent gets 352 of the 360 ordinary requests right, and all eight misses used a cached balance instead of the ledger. It escalates 31 of the 40 disputes. The headline looks strong. The split shows two separate problems: a sourcing fault, fixed by removing the cache from its permitted sources, and a weak escalation rule, fixed in the brief. The team reruns the whole set, because fixing one fault can move another.
Which controls keep the measure honest?
- Fixed answer keys. Lock them before the run and keep the original even if a reviewer overturns a case.
- Separate floors per test. A strong correctness score cannot buy back a weak refusal score.
- Fresh cases each cycle. Reusing the same set rewards an agent tuned to it.
- Production sampling. After go-live, a reviewer scores a weekly sample of live work. Our guide to measuring an agent's work in its first 90 days sets out a cadence.
- Rerun on change. A new model, new sources or a new brief means a new measurement.
Accuracy also has a legal side. Personal data an agent records must be accurate, which the piece on the GDPR accuracy principle and agent outputs covers, and high-risk systems under the EU AI Act carry their own accuracy and cybersecurity duties under Article 15. Accuracy is one input to measuring AI agent governance as a whole, and the wider sequence sits in our guide to deploying an AI agent.
Where Pryme fits
Attest does this replay on your own cases and scores every answer against a floor across seven gates: grounded, refuses, depth, escalates, scope, adversarial and stable. Each gate must clear on its own, so a high score on one cannot mask a low score on another. The result sets the depth the agent may hold, and the certificate lapses after 90 days.
Questions people ask
How many test cases are enough?
Enough to include every kind of case the capability will meet, with the rare ones represented on purpose. A few hundred is a common starting point for a narrow capability. If a category has only two examples, you have not tested it.
Can we use synthetic cases instead of real ones?
Use them to fill gaps, such as a fraud pattern you have rarely seen, but not as the core. Real cases carry the mess of your own data, and that mess is where agents fail.
Should accuracy be measured differently for drafts?
Yes. For draft work, track how much a reviewer changes before sending, as well as whether the draft was right. A draft that is correct but rewritten every time is not saving anyone effort.