Look at what reviewers do, not what they completed. Training worked if review decisions on held items are more accurate, reasons are clearer, red lines are not missed and reviewers stop approving drafts unchanged in seconds. Completion rates and test scores show that people sat the training; the record of their decisions shows whether it changed anything.
Why are completion rates not enough?
Completion tells you people opened the lessons. A test score tells you they could answer questions on the day. Neither tells you whether a reviewer, faced with a confident draft built on a declared fact at the end of a long shift, will stop and check.
The good news is that agent work leaves evidence. Every verdict is filed in an append-only record, including who approved each held action, what they changed and the reason they gave. That record is the best measure of training there is.
What should you measure?
| Measure | What it shows | How to get it |
|---|---|---|
| Review accuracy | Whether held items were decided correctly | Sample decisions and have a certified reviewer re-decide them |
| Missed red lines | Whether the most serious errors are caught | Count approvals that crossed a red line, from samples and incidents |
| Reason quality | Whether people can explain their decisions | Score a sample of reasons against a simple standard |
| Change rate | Whether people engage with drafts or rubber-stamp | Share of held items approved unchanged, by reviewer |
| Time to decision | Whether reviews are rushed | Distribution of time from opening to deciding, by reviewer |
| Calibration agreement | Whether the team decides alike | Agreement on shared cases in calibration sessions |
No single measure is enough. A reviewer who changes nothing may be reviewing excellent drafts; one who decides in seconds may be expert. Look at the measures together, and compare before and after training.
How do you set a baseline?
Measure before training, not only after. A readiness assessment gives a baseline on understanding, by concept, and shows where the gaps are; what a readiness assessment for AI agents measures covers this. For behaviour, take a sample of review decisions from the weeks before training and score them on the measures above. Then repeat after training, and again a few months later to see whether the change held.
The approach is set out more fully in how to measure the impact of AI agent training.
What does a measured result look like?
Take a customer service team of 40 reviewing an agent's held refund drafts. Before training, a sample shows most held refunds approved unchanged, many in under fifteen seconds, and reasons on declined drafts that read "not correct" or "see notes". Two approvals crossed a red line: refunds to accounts already flagged for fraud.
Three months after training, the change rate has risen, decision times have lengthened on complex items but not simple ones, and reasons name the source and the rule. No red line was missed in the sample. One reviewer still approves almost everything unchanged and fast; the team lead coaches them individually. Scoring reasons against a worked example of a good reason helps reviewers see the standard. For the agent's own performance, the measures differ; see how to measure an AI customer support agent.
What about customer-facing skills?
Some training is about conversations, not decisions: for example, how to answer a customer who asks whether a machine decided. Measure these through sampled calls or messages scored against a standard, as covered in how to train staff to answer customers who ask whether an AI decided.
Where the Academy fits
Pryme Intelligence Academy's Operator certification is itself a behavioural measure. The practicum requires two weeks of real review decisions with at least 95 in every 100 correct and no missed red line, signed off by a Pryme assessor. The admin view shows results and certificate expiry for the whole team. The topic guide on training teams for AI agents covers how to prove training to others.
Questions people ask
How large a sample is needed?
Enough to see patterns by reviewer, not just across the team. A few dozen decisions per reviewer over a period usually shows whether behaviour has changed.
Who should re-decide sampled items?
A certified reviewer who did not make the original decision. Ideally someone from outside the immediate team.
Should results be shared with reviewers?
Yes. Showing people their own accuracy and change rate is one of the most effective forms of coaching.