How do you run calibration sessions for AI agent reviewers?

Give every reviewer the same set of real held cases, have each decide alone and write a reason, then compare the decisions as a group and discuss where they differ. Agree the standard, write it down and feed it into the agent's brief. Calibration is how a team of reviewers comes to decide the same case the same way.

Why do reviewers need calibrating?

Each reviewer builds their own sense of what is acceptable. One approves any refund the policy allows; another declines anything that looks like a repeat claim. Neither is wrong in isolation, but customers get different answers depending on who picked up their case, and the agent receives contradictory corrections.

Agents make this more visible and more important. The reasons reviewers give when they change or decline a draft are used to improve the agent's brief. If reviewers disagree, the agent learns noise. Calibration aligns the people so the agent can align with them.

How do you run a session?

1. Pick the cases. Choose eight to twelve real held items from the past fortnight, including some where reviewers disagreed and some near a red line. Remove anything that identifies the reviewer who decided them.
2. Decide alone. Each reviewer decides each case independently, before the session, and writes a short reason. No discussion yet.
3. Reveal the spread. In the session, show how many approved, changed or declined each case. Start with the cases with the widest spread.
4. Discuss reasons, not just outcomes. Ask reviewers to explain what they checked. Often the disagreement is about which source they trusted or which bound they thought applied.
5. Agree the standard. For each disputed case, agree the right decision and why. Where the policy is unclear, note it for the policy owner.
6. Write it down. Record the agreed standards in a short, shared note reviewers can consult.
7. Feed it back. Send agreed standards to the person who owns the agent's brief, so the agent's drafts reflect them, and to the champion or team lead who coaches reviewers.

A ready structure is in the calibration session agenda template.

Who should facilitate?

A certified reviewer who is respected by the team, often a champion or team lead, but not the person with the strongest opinions. The facilitator's job is to surface disagreements, not to win them. In larger organisations, it helps to have the agent's owner attend occasionally, so they hear how reviewers read the drafts.

What does a session look like?

Take a team of IT administrators reviewing an agent's held access requests. Ten cases are chosen. On one, a request for read access to a finance system by a new analyst, five reviewers approved and three declined. In discussion, the decliners explain that the manager's approval was in an email, not the request system; the approvers had not noticed. The team agrees that approvals must be in the request system, and that the agent should hold any request where they are not. The standard is recorded and the agent's brief updated. Role-specific versions of this session are described in how to run calibration sessions for IT administrators and for operations managers.

How often should you calibrate?

Every two to four weeks in the first months of a deployment, then monthly. Calibrate also after any change to the agent's depth, bounds or sources, and whenever sampling shows reviewers drifting apart. Teams often skip it under pressure, which is the argument in calibration is the training nobody schedules.

How do you know it is working?

Agreement on shared cases rises over successive sessions. Fewer sampled decisions are overturned. The agent's drafts are changed less often for the issues calibrated. And the written standards grow into a reference new reviewers can learn from. Certification programmes for reviewers, such as Pryme Intelligence Academy's Operator track, cover running this rhythm as part of the role. The topic guide on training teams for AI agents shows where calibration fits.