A calibration session agenda takes about 60 minutes: five minutes to set up, 15 for reviewers to decide the same cases alone, 25 to compare and discuss the differences, ten to agree decisions and changes, and five to assign follow-up. The template below gives each block, its owner and its output, ready to copy.
What is a calibration session for?
Calibration keeps a team of reviewers deciding alike. Each person decides the same small set of cases on their own, then the team compares answers and works out why they differed. Over time, the spread narrows and the team's standard becomes something people can describe, not just feel.
With AI agents, calibration matters more than usual. Reviewers decide held actions every day, and their approvals become part of the agent's record. If two reviewers would reach different verdicts on the same held refund, the agent's outcomes depend on who happens to be on shift. Calibration also feeds the agent: when reviewers agree a draft was wrong in a repeatable way, that becomes a change to the agent's brief or sources, made through its owner and recertified. The argument for doing this routinely is made in calibration is the training nobody schedules.
Hold a session every two to four weeks after go-live, and after any change in the agent's depth or sources. The guide to training teams for AI agents places calibration among the three methods that carry most of the learning.
How do you prepare the session?
The facilitator, usually a team lead, chooses five to eight cases from the agent's record since the last session. Pick for disagreement, not for volume:
- a held action approved by one reviewer and edited heavily by another on a similar case;
- a refusal that a customer or colleague challenged;
- an approval later overturned;
- a case where the agent relied on a declared or external source;
- one case that everyone handled well, as a reference point.
Strip names where they would bias judgement. Prepare a sheet for each reviewer with the case and blank fields for their verdict and reason. Teams with specialist work adapt the mix: the guides on how to run calibration sessions for internal auditors and how to run calibration sessions for AML investigators show two variations.
The agenda template
| Time | Block | Owner | What happens | Output |
|---|---|---|---|---|
| 0 to 5 min | Set-up | Facilitator | Purpose, ground rules, any changes to the agent since last time | Attendance list |
| 5 to 20 min | Silent decisions | Each reviewer | Decide every case alone: approve, edit, reject or escalate, with a one-line reason | Completed sheets |
| 20 to 45 min | Compare and discuss | Facilitator | Reveal decisions case by case; start with the widest split; ask what each person saw | Notes on each split |
| 45 to 55 min | Agree | Team, with the process owner if present | Agree the correct decision and the rule behind it | Decision log |
| 55 to 60 min | Follow-up | Facilitator | Assign actions: guidance updates, agent brief changes, coaching | Action list with owners and dates |
Keep the decision log in a fixed format: case reference, verdict each reviewer gave, agreed decision, the reason in one sentence, and whether it changes guidance, the agent's brief, or nothing.
Worked example
An illustrative accounts payable team of six meets for its fortnightly session. The facilitator brings six cases. In the silent round, four reviewers approve a held payment to a supplier whose bank details changed last week; two hold it. Discussion shows the change request came by email, an external source, and was never confirmed by phone. The team agrees: hold, verify, then approve. The guidance note is updated and the agent owner is asked to add a bound that holds any payment within 30 days of a bank detail change.
A second case shows full agreement on a refusal, which is noted as a reference. The session ends with three actions, each with an owner and a date before the next session. Sessions are sometimes recorded for absent reviewers; the trade-offs are explained in what a recorded session is.
Questions people ask
How is calibration different from a lunch-and-learn?
A lunch-and-learn teaches; calibration measures and aligns. One introduces ideas to a broad audience, the other tests a team's decisions on real cases. A lunch-and-learn session plan template on AI agents serves the first purpose.
What if reviewers keep disagreeing on the same type of case?
Then the rule is unclear, not the reviewers. Write the rule down, have the process owner sign it, and test it in the next session. If it involves the agent's sources or bounds, raise it with the agent owner.
Should calibration results count towards certification?
They are useful evidence of current competence, and a team lead should watch for one person who is consistently out of line. They do not replace a formal assessment.