How to run calibration sessions for contact centre managers

Contact centre managers calibrate on two things advisers do not: how they score interactions, and which escalations they approve. Give every manager the same set of contacts where an agent drafted and an adviser sent, plus a few escalations above advisers' limits. Each scores and decides alone. Compare, agree a standard, update the scorecard and the agent's brief, then sample.

Why do managers need calibrating, not just advisers?

In a contact centre with an agent, managers do three things that shape everyone else's behaviour. They score interactions for quality. They approve escalations above advisers' limits, such as goodwill payments or complaint outcomes. And they decide when to pause the agent or narrow its bounds.

If managers score differently, advisers learn different lessons depending on whose team they are in. One manager marks an adviser down for sending an agent's draft unchanged; another marks the same thing as good practice. One approves goodwill on any repeat contact; another never does. Calibrating advisers without calibrating managers just moves the inconsistency up a level. The general method is in how to run calibration sessions for AI agent reviewers.

The head of the contact centre owns the sessions. The quality lead facilitates. Each team manager takes part.

Which contacts belong in the set?

Ten items from the past fortnight, with adviser and customer names removed:

• Four contacts where the agent drafted and an adviser sent, one of them with a subtle error the adviser missed.
• One contact where the adviser rewrote the agent's draft substantially, for better or worse.
• One contact the agent refused as outside its bounds, where the adviser treated the refusal as a fault and apologised for "the system".
• Two escalations above advisers' limits: a goodwill request after a third failed delivery, and a complaint asking for a refund outside policy.
• One contact where the customer asked whether they were talking to a person.
• One handover from agent to adviser mid-conversation, where the customer had to repeat everything.

The session, step by step

1. Person (quality lead): selects and anonymises the items.
2. Person (each manager): scores each contact alone on the scorecard and decides each escalation, with reasons.
3. Agent, depth 1: shows its original draft, verdict and the bound that decided for each contact, so managers can see what the adviser started from.
4. Person (quality lead): reveals the spread of scores and decisions.
5. Person (managers): explain their marks, item by item, starting with the widest spread.
6. Person (head of contact centre): agrees the standard for scoring and escalations.
7. Person (quality lead): updates the scorecard guidance, and sends any change to the agent's drafting or holds to the brief owner.
8. Person (quality lead): double-scores a sample of each manager's reviews over the next month.

What does a gold standard look like for managers?

For scoring: "An adviser who sends an agent's draft unchanged scores full marks on accuracy only if they checked the order and identity details; the scorecard asks them to record that check." And: "An adviser who explains a refusal as a system fault loses marks; the correct explanation is that the request needs a person, followed by handling it."

For escalations: "Goodwill after a third failed delivery is approved up to the agent's own limit; above it, managers approve where the record shows the failures were the business's fault."

For transparency: "When asked, advisers confirm whether a reply was drafted by an AI system, in one plain sentence."

Which controls does calibration support, and what evidence does it leave?

Consistent quality scoring. Evidence: spreads of scores by manager, narrowing over time.

Consistent escalations. Evidence: the escalation standard and sampled approvals.

Correct reading of verdicts. Evidence: scorecard guidance on refusals and holds.

Agent alignment. Evidence: brief changes and approvers.

What should the head of the contact centre measure?

• Score spread across managers on shared contacts.
• Double-scored reviews where the second score differs materially.
• Goodwill approvals by manager, which should become more similar.
• Contacts where advisers described a refusal as a fault, which should fall.

Where does it go wrong with managers?

Managers calibrating advisers but never themselves. Their scoring sets the standard advisers chase.

Scorecards that predate the agent. Old scorecards reward typing speed and personalisation, not checking the agent's draft against the record.

Treating escalations as discretion. Some discretion is right. Unwritten discretion means customers get different outcomes by team.

Skipping the handover case. Managers tend to score the adviser's reply and ignore whether the customer had to repeat themselves after the agent handed over. That is often the part customers complain about most, and it is a reading-the-workspace skill the scorecard should reward.

Risk and operations managers calibrate on wider decisions; see calibration for risk managers and for operations managers. For planning, see the AI agent training plan template for contact centre managers, and the topic guide on training teams for AI agents.

Where the Academy fits

Contact centre managers who run a live workspace every day are the audience for Pryme Intelligence Academy's Operator track, which covers reading the workspace, human review and running the rhythm. For teams on paid plans, an admin view shows each person's progress, results and certificate expiry, which helps managers see who on their team is certified to approve what.