Calibration is the training nobody schedules

Calibration is the training that matters most once AI agents are live, and it is the one almost nobody puts in the diary. Reviewers who decide the same held cases and compare their answers learn faster than in any course, and the team stops drifting apart. It costs an hour away from the queue. Skipping it costs far more, but more quietly.

What is calibration, in an agent deployment?

A short, regular session in which reviewers look at the same set of real cases, decide each one independently, then compare and agree what the right decision was and why. The cases come from the queue: held refunds, refused requests, overrides, drafts that were edited heavily. The output is a shared standard, written down, and sometimes a change to the agent's brief or sources.

It is not a lecture or a policy refresher. Nobody presents. The material is the team's own decisions, and the discomfort of seeing a colleague decide the opposite way on the same case is the point. The definition and method are set out in what calibration is.

Why does review drift without it?

Because each reviewer learns from their own queue. One reviewer approves goodwill gestures freely because the customers they see seem reasonable. Another refuses almost all of them because they once approved one that turned out to be abuse. Both think they are applying policy. Customers get different outcomes depending on which shift they reached.

Agents make this worse in a specific way. The agent is consistent: it holds the same kinds of items every time. So the variation customers experience comes almost entirely from the people deciding the held items. Without calibration, the most consistent part of the system feeds the least consistent part, and nobody notices until a complaint compares two outcomes side by side. Every verdict and every human decision is filed in an append-only record, so the inconsistency will be visible to an auditor even if it is not visible to the team.

Why does nobody schedule it?

Three reasons, all understandable. First, it takes reviewers off the queue at the same time, which is exactly what operations managers are measured on avoiding. Second, it does not look like training: there is no course, no completion record, no certificate. Third, it surfaces disagreement, and many managers would rather not see it.

The trade-off is real. An hour of calibration for a team of eight is eight hours of review not done. Held items wait longer, and on a busy week that hurts. But the alternative is a team that decides inconsistently for months, an agent whose brief is never improved by what reviewers learn, and a training plan that ends at go-live. Teams that tried both rarely go back. The articles on why AI agent pilots fail when nobody is trained and pilots that fail because nobody was trained to say no show what the absence costs.

How do you run a calibration session that works?

Keep it short and concrete:

1. Pick five to eight cases from the past week where reviewers decided differently, or where the decision was close.
2. Decide independently first. Each reviewer records their decision before discussion, so nobody anchors on the loudest voice.
3. Reveal and compare. Look at the spread. Where everyone agreed, move on.
4. Discuss the disagreements. What did each person see? Did anyone check provenance, such as whether a key fact was declared by the customer or sealed from the system of record? Which bound held the item?
5. Agree and write it down. The agreed decision, the reason, and whether the agent's brief or sources need to change.
6. Track over time. Fewer disagreements on the same kind of case means the training worked.

Sector guides show how this looks in practice for claims handlers and wealth advisers.

What does calibration teach that courses cannot?

Judgement in context. A course can explain that a refusal means the action was outside the agent's bounds, and that overriding one needs authority and a reason. Only calibration shows a reviewer that their colleague overrides refusals twice as often, and why. A course can explain provenance; calibration shows that half the team never looks at it on refund cases.

Calibration also teaches the agent, indirectly. When the team agrees that a certain draft is always wrong in the same way, that finding goes back into the agent's brief or sources. That is qualification, not retraining: the agent's model is not changed, but what it is given and how it is bounded improve. Over time, fewer items need deciding, and the ones that remain are harder and more worth discussing.

What should you do differently?

Put calibration in the diary as a recurring, protected session, weekly for new deployments and at least monthly once things settle. Plan the queue around it rather than cancelling it when the queue is busy. Make a named person responsible for picking cases and writing down the outcome. Report the disagreement trend alongside throughput, so leaders see that consistency is being managed.

Treat it as part of the training plan, not an extra. The wider approach is in training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's Operator track covers human review and running the rhythm, the two modules where calibration lives. Certified Pryme Operator's practicum, two weeks of real review decisions signed off by a Pryme assessor, gives a team an external reference point to calibrate against.