How to run calibration sessions for operations managers

Operations managers calibrate on management decisions about agents, not individual held items: whether to pause an agent, whether to approve a request to raise a limit, whether a pattern counts as an incident, and how to staff a held queue. Give each manager the same short scenarios from the record, have them decide alone, compare, and agree a written standard.

What do operations managers calibrate on?

Reviewers calibrate on cases. Operations managers make a different kind of decision, about the agent and the team rather than a single item. Typical examples:

• Pausing. Volumes of held items jump; is that a reason to pause the agent to draft-only, or to add reviewers?
• Limit changes. A team lead asks to raise the agent's refund limit because too many routine items are held.
• Incidents. Three customer complaints mention the same wrong figure in an agent's reply; is that an incident?
• Capacity. A certificate is about to lapse; the agent will drop to depth 1 if its next run clears no depth. How many reviewers are needed on standby?
• Exceptions. A held action above every reviewer's limit needs a manager's decision.

If one manager pauses at the first sign of trouble and another waits a week, the business's risk depends on who is on duty. Calibration is how managers make those calls alike. The general method for reviewers is in how to run calibration sessions for AI agent reviewers.

The head of operations owns the sessions. A risk partner attends, since many of these decisions affect risk appetite.

How to build the scenarios

Use the record. Every verdict, limit change and pause is filed, so real episodes can be turned into short scenarios with names removed. Six scenarios fit a 90-minute session. Each should end with a decision point and the information the manager actually had at that moment, not what was learned later.

The session, step by step

1. Person (head of operations): writes six scenarios from the record, with names removed.
2. Person (each manager): decides each alone, recording the decision, the evidence relied on, and who they would inform.
3. Agent, depth 1: supplies the record extracts for each scenario: verdicts, the bounds that decided them, volumes of held items, certificate status.
4. Person (facilitator): reveals the spread.
5. Person (managers): explain their thresholds. Most disagreements are about thresholds: how many complaints, how large a backlog, how close to a lapse.
6. Person (head of operations, with the risk partner): agrees the standard, with explicit thresholds where possible.
7. Person (head of operations): publishes the standard to managers and team leads, and sends any change to limits or holds to the brief owner.
8. Person (head of operations): reviews the next real decisions of each kind against the standard.

What does a gold standard look like for managers?

Thresholds and actions, written down. For example: "Three or more customer reports of the same error in agent-sent replies within a day is an incident: pause the affected capability to draft-only and escalate. One report is investigated before any pause."

On limit changes: "A request to raise a limit is approved only with evidence from the agent's certification and a fortnight's sample of held items that would have been allowed. Volume of held items alone is not a reason."

On certificate lapses: "A week before a certificate lapses, the manager confirms enough reviewers to run the queue at depth 1 for three days in case the next run clears no depth."

What does a session look like?

Take a hypothetical operations team with five managers. On the limit-change scenario, three approved the increase because the held queue was long; two refused without evidence. The risk partner pointed out that a long queue says nothing about whether the agent would decide those items correctly. The standard above was agreed. In the following month, two limit requests came in; both were supported with a sample, and one was approved.

Which controls does calibration support, and what evidence does it leave?

Consistent pauses. Evidence: the incident threshold and real pause decisions against it.

Disciplined limit changes. Evidence: limit changes filed with the evidence required.

Lapse readiness. Evidence: staffing confirmations before each lapse date.

Escalation. Evidence: who was informed, by scenario and in real events.

What should the head of operations measure?

• Agreement across managers on scenarios.
• Real pauses and limit changes that followed the standard.
• Time from first signal to pause in real incidents.
• Lapses where the queue could not be staffed.

Where does it go wrong with operations managers?

Managers exempting themselves. They calibrate their teams and assume their own judgement is consistent.

Hindsight in scenarios. Scenarios written with the answer known make every manager look either reckless or slow.

No thresholds. "Use judgement" is not a standard. Write the number.

Calibrating only on bad days. Scenarios drawn only from incidents teach managers to pause. Include at least one where the right call was to keep the agent running and add reviewers, so the standard covers restraint as well as caution.

Risk and property managers calibrate on related management decisions; see calibration for risk managers and for property managers. For planning, see the AI agent training plan template for operations managers, and the topic guide on training teams for AI agents.