What is calibration?

Calibration is the practice of reviewers deciding the same set of cases independently, comparing their decisions and agreeing the standard. Its purpose is consistency: a held action should get the same answer whichever qualified reviewer picks it up. Without it, every reviewer slowly develops a private standard, and the agent is supervised by several different rulebooks at once.

Why does calibration matter once agents arrive?

Because reviewer decisions become the control. When an agent holds an action for a named person, that person's judgement is what stands between the draft and the customer. If two reviewers would decide the same held refund differently, the business does not have one control; it has a lottery.

Calibration matters for the agent too. Reviewers' corrections are the raw material for improving the agent. If reviewers disagree, their corrections point in different directions and the agent owner cannot tell which reason to bank.

How does a calibration session work?

A team lead picks eight to twelve recent cases, mixing routine items with hard ones: a borderline amount, a document of doubtful provenance, a case near a red line. Each reviewer decides them alone and writes a one-line reason. The group then compares. Where everyone agrees, the session moves on quickly. Where they split, the group works out why: a missing rule, an ambiguous policy, or one reviewer reading the facts differently.

Each disagreement ends in one of three outcomes. The standard is clarified and written down. The policy owner is asked to decide. Or the case shows an agent gap, and the reason goes to the agent owner. A ready-made structure is in the calibration session agenda template.

An example

Take a hypothetical finance team where an agent matches invoices and holds anything over the customer's limit. Four accounts payable clerks calibrate on ten held invoices. Nine produce agreement. On the tenth, two clerks approve and two decline: the supplier's bank details came from an emailed letter, which is external provenance. The lead confirms the standard that external evidence alone never supports a payment change, and the two who approved learn something they will use daily. How clerks treat those held items is covered in how accounts payable clerks handle an action an AI agent has held for them.

How is it different from model calibration?

The word has a second meaning that causes confusion. In statistics, a model is calibrated when its stated confidence matches how often it is right. That is a property of the model, measured by its builders.

Reviewer calibration is a property of people. It asks whether the humans overseeing the agent apply the same standard. An agent can be well tested and its reviewers badly calibrated, and the operation will still give inconsistent outcomes. Calibration also differs from training: training teaches the standard once, while calibration checks, on real cases, that people still apply it.

How often should it run?

Weekly for a new team or a new agent capability; fortnightly or monthly once agreement is steady. Run an extra session after any change to policy or to the agent's depth. Roles differ in what they calibrate on: KYC teams spend most of their time on provenance and identity mismatches, as in how to run calibration sessions for KYC analysts, while technical teams focus on access and connection changes, as in how to run calibration sessions for IT administrators. The reasons it so often slips off the calendar are set out in calibration is the training nobody schedules.

Related terms

Inter-reviewer agreement measures what calibration aims to improve. Banked corrections are often the output of a good session. The guide to AI agent skills by role places calibration within the team lead's work.