Support staff calibrate on held tickets: the refunds, complaints and account changes the AI agent passed to a person. Each support agent decides the same ten tickets alone and edits the draft as they would send it. The team compares outcomes and wording, agrees a gold standard, and the brief owner updates the agent so its next drafts already match.
Why do support agents need calibration?
To keep the words apart: "support agents" here are the people; the AI is "the agent".
A support agent's day with an AI agent alongside is mostly reviewing. The agent sends routine answers at depth 3 within approved replies and the business's own limits, and drafts anything else at depth 2. Support agents approve, edit or reject those drafts, and handle tickets the agent refused.
Each support agent edits differently. One rewrites every draft in their own voice; another sends drafts unchanged. One approves a refund one unit over the limit; another escalates it. Customers get different answers, and the agent, learning from edits, gets contradictory signals. Calibration aligns the team. The general method is in how to run calibration sessions for AI agent reviewers.
A team lead owns the sessions. A champion, often an experienced support agent, facilitates.
Which tickets belong in the set?
Ten tickets from the past fortnight, with customer and staff names removed:
- A refund request just above the agent's limit, with a clear case for refunding.
- A repeat contact from a frustrated customer on their third attempt.
- A request to change the email address on an account, from an address not on file.
- A message that is both a question and a complaint.
- A draft that is accurate but cold in tone.
- A ticket the agent refused, such as a request to cancel another person's order.
- A draft with a small factual error a quick reader would miss.
The session, step by step
- Person (team lead): selects and anonymises the tickets.
- Person (each support agent): decides each ticket alone and writes the reply exactly as they would send it.
- Agent, depth 1: shows its original draft, its verdict and the bound that decided, so the team can see what each support agent changed.
- Person (champion): reveals the spread on outcomes, then puts two or three edited replies side by side.
- Person (support agents): explain their edits and decisions.
- Person (team lead): agrees the gold standard on outcome and wording.
- Person (team lead): sends the standard to the brief owner, with example replies, so the agent drafts closer to it.
- Person (team lead): samples sent replies for two weeks.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Select tickets | Nothing | 2 or 3 on live tickets | Choosing the set |
| Decide and write | Nothing | Unchanged | Outcome and reply |
| Agent's draft | Shows draft and deciding bound | 1 | Comparing edits |
| Discussion | Nothing | Unchanged | Reasons and wording |
| Gold standard | Nothing | Unchanged | Team lead |
| Update brief | Drafts per new standard | 2 or 3 | Brief owner |
| Sampling | Works live tickets | 2 or 3 | Team lead |
What does a gold standard look like for support?
It sets outcome and wording together. For example: "A refund up to a set amount above the agent's limit is approved by a support agent where the order record shows the item was not delivered. Above that, a team lead approves."
On identity: "A request to change account contact details from an unrecognised address is held. The support agent replies to the address on file, never the new one, and asks the customer to confirm through the signed-in account."
On tone: "Repeat contacts open by acknowledging the previous attempts in one sentence, then give the answer. No apology for 'the system' when the agent refused something; explain that it needs a person and handle it."
On edits: "Send the agent's draft unchanged when it is accurate and the tone fits. Rewriting for personal style is not required and makes replies inconsistent."
Which controls does calibration support, and what evidence does it leave?
Consistent outcomes. Evidence: spreads over successive sessions.
Identity checks. Evidence: the identity standard and sampled account changes.
Tone. Evidence: example replies agreed and sampled replies against them.
Agent alignment. Evidence: brief changes, with the example replies attached.
What should the team lead measure?
- Agreement on outcomes for shared tickets.
- Sampled replies that break the identity standard. The target is none.
- How heavily support agents edit the agent's drafts, which should fall as the brief catches up.
- Repeat contacts and complaints after calibration.
Where does it go wrong with support teams?
Calibrating outcomes only. Most of the variation in support is wording.
Rewarding heavy editing. If quality scores reward personalisation, support agents rewrite good drafts and introduce errors.
One shift only. Evening and weekend staff work with less supervision. Include them, or record the session for them.
Support team leads calibrate on escalations and pauses; see calibration for support team leads. Collections agents face a similar tone problem with more at stake; see calibration for collections agents. For planning, see the AI agent training plan template for customer support agents, and the topic guide on training teams for AI agents.
Where the Academy fits
Support agents who approve held replies every day are the people Pryme Intelligence Academy's Operator track is written for. Lessons can be read or listened to as audio, labs run in a free Build workspace, and module tests are set in the learner's own industry and role, so a support agent practises on tickets, not abstract examples.
Questions people ask
How often should support agents calibrate?
Every two weeks while the agent is new, then monthly, and after changes to refund limits or approved replies.
Should new starters take part?
Yes, from their second or third week. Calibration teaches the team's standard faster than shadowing alone.
What if the team prefers its own wording to the agent's?
Agree the preferred wording as the standard and send it to the brief owner. The agent should draft the way the team would write.