Four stages over about a month will get a support team reviewing well: learn how the agent decides and what each verdict means; calibrate on past tickets until reviewers agree; shadow live drafts and handovers beside a certified colleague; then sign off each person against a bar of correct decisions with no missed red line, before they approve anything alone.
What does a support reviewer decide?
A customer support agent can answer questions from the help centre and the customer's account, draft replies, issue refunds within a limit, update simple account details and hand conversations to people. Reviewers decide the rest: whether to send, edit or decline a draft; whether to release a refund held over the limit; how to pick up a handover; what to tell a customer the agent refused.
The handover is the decision most teams under-train. The workspace shows why a conversation was handed over: a refusal, a held action, low confidence, or a customer asking for a person. A reviewer who reads that reason in seconds picks up smoothly. One who does not makes the customer repeat everything.
The head of support owns the programme. Team leads with current certificates run calibration and shadowing.
What should reviewers learn first?
- Depth by capability. The agent may answer questions at depth 3 but only draft refunds at depth 2. Reviewers should know which is which for their agent.
- The four verdicts. Allowed, held, refused, withheld. A withheld action is a platform decision and cannot be released by anyone; reviewers explain it to the customer and move on.
- Provenance. An order status from the order system is sealed. A delivery date the customer typed is declared. A screenshot is external.
- Reasons. Every edit and decline needs one, written so a colleague and the builder can act on it.
The steps
- Person (head of support): lists decision types and red lines. Red lines might include: sending a reply that promises something outside policy; releasing a refund to a changed payment method without verification; ignoring a customer's request for a person; missing a vulnerability or safety signal.
- Person (reviewer): completes the learning above.
- Person (team lead): runs calibration on 60 anonymised past tickets over three sessions: drafts with subtle errors, refunds near the limit, handovers for each reason, and messages with a vulnerability signal.
- Agent, depth 2 and 3: handles live tickets in shadow mode. The trainee decides each draft, hold and handover in a shadow queue while a certified colleague decides for real.
- Agent, depth 1: logs both decisions, the handover reason and any red-line flag.
- Person (team lead): debriefs daily for fifteen minutes on mismatches.
- Person (head of support): signs off after two weeks at the bar.
- Person (workspace admin): grants approval rights by capability, starting with drafted replies and adding refunds later.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Help-centre question | Answers from approved content | 3 | Reviewer samples |
| Complaint reply | Drafts | 2 | Reviewer sends, edits or declines |
| Refund within limit | Issues and records | 3 | Reviewer samples |
| Refund over limit | Prepares and holds | held | Reviewer with rights decides |
| Handover | Hands over with the reason shown | 1 | Reviewer picks up without making the customer repeat |
| Request outside scope | Refuses and names the bound | refused | Reviewer explains and offers alternatives |
How should calibration handle handovers?
Give each reviewer the same ten handovers, one at a time, with two minutes each. Ask for the first message they would send. Then compare.
Good first messages show the reviewer read the reason and the history: "I can see our system could not change the delivery address because the parcel has already left the warehouse. I can arrange a redirect with the courier for you." Weak ones start from scratch: "How can I help today?" The difference is learned through repetition, not explanation.
Which controls matter, and what evidence do they leave?
- Handover reason shown to the reviewer. Evidence: the reason recorded with each handover.
- Approval rights by capability. Evidence: the admin view of who may release refunds, with certificate expiry.
- Independent shadow decisions. Evidence: trainee and colleague decisions filed separately.
- Sign-off against a written bar. Evidence: correct decisions out of the total and the red-line count, signed.
- Append-only record. Evidence: every verdict and decision, available for complaints.
What should you measure?
Agreement with the certified colleague by decision type. Red-line misses, which must be zero. Customers asked to repeat information after a handover, sampled weekly. Drafts sent unedited and later reopened. Time held refunds wait. And the share of declines with a reason a colleague could act on.
Where does it usually go wrong?
Teams train on drafts and forget handovers. Customers notice the handover first.
Reviewers approve drafts by reading the opening line. Errors usually sit in the middle: a wrong date, a policy misquoted.
Experienced agents from the pre-automation team are excused calibration. They often hold the strongest private standards, which is why calibration matters most for them.
Other teams follow the same arc with different stakes; compare training the risk team and training the IT team. For daily practice, see how customer support agents review an agent's decisions and how support team leads review an agent's decisions. The guide to training teams for AI agents covers sequencing.
Where the Academy fits
Pryme Intelligence Academy's Operator track is designed for reviewers like these, with modules on human review, teaching the agent and reading the workspace. Lessons can be read or heard as audio, with highlights and notes. Certified Pryme Operator requires a written exam of 40 questions in 60 minutes, with 32 of 40 needed to pass, and a two-week practicum signed off by a Pryme assessor.
Questions people ask
Should new joiners review from their first week?
Not alone. They can shadow from the first week and approve only after sign-off. Until then, their decisions are practice.
Do reviewers need to know how the agent is built?
No. They need to know how it decides, what each verdict means and where facts come from. Building is a separate role.
How do we keep standards consistent across shifts?
Rotate calibration so every shift takes part, and use the same case sets. Cold-reading each other's reasons across shifts also helps.