Measure a compliance agent on what it gets wrong, not how much it gets through. The core measures are agreement with qualified reviewers, missed cases found by independent re-review, whether its claims are grounded in its sources, whether it stayed in scope, and whether any of these is drifting. Speed and volume matter, but only once the quality measures hold.
Why are speed and volume the wrong headline?
An agent that clears alerts twice as fast looks like a success. If it also misses one suspicious case in fifty, it is a failure the speed measure hides. In compliance, the costly errors are the quiet ones: the alert closed that should have been escalated, the breach not logged, the match discounted wrongly. Those errors do not show up in throughput figures. They show up later, in an investigation or an inspection.
So the measurement design starts from the errors that matter and works out how to see them. The same logic applies to customer-facing agents, though their key errors differ; see how do you measure an AI customer support agent?
Which measures matter, and who produces them?
| Measure | What it shows | Who acts | Evidence it leaves |
|---|---|---|---|
| Agreement rate | How often reviewers accept the agent's recommendation | Reviewers, as part of normal work | Decisions filed alongside recommendations |
| Override direction | Whether reviewers escalate more or less than the agent | Compliance lead analyses monthly | Override analysis |
| Missed-case rate | Cases the agent recommended closing that independent re-review would have escalated | A qualified person re-reviews a blind sample | Re-review working papers |
| Grounding | Whether every claim in the agent's output is in its cited sources | Sample check by a reviewer or a checking agent | Grounding check results |
| Scope | Attempts outside the agent's bounds, and whether they were refused | Automatic from the verdict record | Refused and held verdicts |
| Escalation quality | Whether the agent escalates what it should and states why | Reviewer sample | Sample results |
| Drift | Change in any of the above over time, or after a model or source change | Compliance lead | Trend reports |
| Cycle time and volume | Speed and throughput | Operations | System metrics |
The missed-case rate is the measure that most needs people. Agreement can be high because reviewers trust the agent too much. Only a blind re-review, where the reviewer does not see the agent's recommendation first, shows what the agent misses. For alert-generating rules, a similar method is covered in how to review transaction monitoring rule performance with an AI agent.
How do you run the measurement, step by step?
- Set the floor. The compliance owner sets minimum levels for each quality measure before go-live. Evidence: the approved measurement plan.
- Collect automatically. Agreement, overrides, scope verdicts and cycle time come from the record. Evidence: the extract.
- Re-review blind. Each month, a qualified person re-reviews a sample, weighted to closures and discounts. Evidence: re-review papers.
- Check grounding. A sample of outputs is checked claim by claim against sources. Evidence: grounding results.
- Review weekly. The agent's owner reviews the measures against the floor; see how to run a weekly AI agent performance review. Evidence: review notes and actions.
- Act on breaches. A measure below the floor reduces what the agent may do until it is fixed and re-certified. Evidence: the depth change and re-certification record.
- Report. Results go to the compliance committee with the rest of the second line's MI. Evidence: the committee pack.
This maps closely to the Measure function in the NIST AI Risk Management Framework; see NIST AI RMF Measure function for AI agents.
What does it look like in practice?
Take a firm using an agent to prepare PEP screening dispositions. In one month, reviewers accept the agent's recommendation in nearly all cases. That looks excellent.
The blind re-review of 100 discounted matches finds two that should have gone to a person: both were family members of PEPs where the link was in an adverse media article, not the list data. The agent had not read adverse media for family links. Grounding checks are clean. There are no scope breaches.
The compliance lead treats the two misses as a capability gap, not bad luck. The agent's discounting of family-member matches is moved back to held, pending a brief change to include adverse media, and the change is re-certified before discounting resumes.
Which controls make the measures trustworthy?
- Blind re-review, so agreement cannot be inflated by trust.
- Floors set before go-live, so they are not set to match results.
- Measures tied to depth. Performance below the floor changes what the agent may do, automatically.
- Governance measures sit alongside. Agent measures are one input to wider AI agent governance measurement.
The general principle is in our guide to AI compliance agents.
Where Pryme fits
In Pryme, Attest scores each capability against a floor on your own cases, across seven gates including grounded, escalates, scope and stable. A failed run drops the agent to read-only the same day. Certificates lapse after 90 days, so the measures above are re-proven, not assumed.
Questions people ask
What is a good agreement rate?
There is no universal number. A rate near total agreement is a reason to check reviewers' independence, not to celebrate.
Should the agent measure itself?
It can produce the raw data. The measures that matter, especially missed cases, need independent review.
How often should measures be reported?
Weekly to the owner, monthly to compliance management and quarterly to the committee.