Train the operations team in four stages: learn how the agent decides, calibrate on the same past cases until reviewers agree, shadow live work while a certified reviewer decides for real, then sign each person off against a bar of correct decisions and no missed red line. Approval rights follow sign-off, not seniority or tenure.
What does reviewing agent decisions mean for an operations team?
An operations team handles the work that keeps a business running: account changes, order exceptions, refunds inside a limit, supplier queries, data corrections. Once an agent takes on part of that work, the team's job changes. Most of what they see is drafted or held by the agent, and the reviewer's decision is the last control before something reaches a customer or a system of record.
The head of operations owns the programme. A team lead with a current certificate runs calibration and shadowing. Each reviewer owns their own decisions and the reason they write for each one. Nobody gains approval rights by being on the rota; they gain them by passing sign-off.
What does each stage teach?
Learning covers how a governed agent reaches a verdict. Every action meets five bounds at once: role scope, certificate, sources, the business's own limits and the platform guardrails. The lowest one wins, and the verdict names which bound decided. A reviewer who cannot read that line will approve things for the wrong reason.
Calibration makes judgement consistent. Reviewers decide the same anonymised past cases alone, then compare. The point is not a right answer handed down; it is a shared standard the team can explain.
Shadowing puts that standard under live pressure without live risk. Sign-off checks it against a fixed bar.
The steps
- Person (head of operations): list the decision types and red lines. For example: approving an account change, releasing a refund held over the limit, confirming a data correction. Red lines might include approving a change of payee details from an emailed request, or releasing anything the agent refused.
- Person (trainee): complete the learning. Depth, the decision chain, the four verdicts, provenance of facts, and how to write a reason.
- Person (team lead): run three calibration sessions on 20 past cases each, mixed across decision types. Discuss every case where reviewers split.
- Agent, depth 2: drafts and holds live work in shadow mode. The trainee decides each item in a shadow queue; the certified reviewer decides it for real.
- Agent, depth 1: logs both decisions side by side, with the bound that decided the original verdict.
- Person (team lead): debriefs daily on every mismatch, using the trainee's written reason.
- Person (head of operations): signs off when the trainee meets the bar over at least two weeks of live shadow decisions.
- Person (workspace admin): grants approval rights tied to the reviewer's current certificate, capability by capability.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Account change | Prepares the change from a verified request | 2 | Reviewer approves or declines |
| Refund within limit | Executes and records | 3 | Reviewer samples |
| Refund over limit | Drafts and holds | held | Reviewer with rights decides |
| Data correction | Proposes the correction with its source | 2 | Reviewer confirms against the system of record |
| Out-of-scope request | Refuses and hands over | refused | Reviewer explains to the requester |
| Withheld action | Withholds at the platform floor | withheld | Nobody can release it; reviewer escalates |
What separates a good operations reviewer from a fast one?
Three habits, each of which should be practised in calibration rather than explained once.
The first is checking where each fact came from. A fact stamped sealed came from the system of record; one stamped external arrived from outside, such as an emailed form. An external claim cannot authorise a payment, and a reviewer who treats a tidy email as proof undoes the control.
The second is knowing the difference between refused and withheld. A refusal reflects the agent's bounds or the business's limits, and a change of limits might alter it later. A withheld action is a platform decision; no reviewer and no limit change will release it, so the reviewer stops looking for a workaround.
The third is writing a reason a colleague can follow. "Declined, customer's address differs from the ledger" is useful. "Not right" is not, and the agent cannot learn from it either.
Which controls matter, and what evidence do they leave?
- Independent shadow decisions. Evidence: the trainee's and the certified reviewer's decisions, filed separately before comparison.
- Calibration records. Evidence: each case, each reviewer's call and the agreed standard.
- Sign-off against a written bar. Evidence: the count of correct decisions out of the total and the red-line check, signed by the head of operations.
- Approval rights tied to a current certificate. Evidence: the admin record showing who may approve what, and when their certificate expires.
- Append-only decision record. Evidence: every verdict and every human decision, so any case can be reconstructed.
What should you measure?
Agreement between trainee and certified reviewer, by decision type. Red-line misses, which must be zero. Time from a held item appearing to a decision. The share of approvals with no edit; a reviewer who never declines or edits is often not reading. And the quality of reasons, scored on a sample each week.
Where does it usually go wrong?
Calibration is skipped because the team is busy. Reviewers then apply private standards, and the agent receives contradictory corrections.
The shadow period runs through a quiet fortnight with no refusals or held exceptions. Seed past cases so every decision type appears.
Senior staff are excused sign-off. Their approvals carry the most weight, so they need it most.
Specialist teams follow the same arc with different red lines; compare training the payments operations team and training the legal operations team. For the manager's day-to-day view, see how operations managers review an agent's decisions, and for who to train first across the business, the guide to training teams for AI agents.
Where the Academy fits
Pryme Intelligence Academy's Operator track is built for these reviewers. Its modules include how a governed agent decides, human review and reading the workspace, and module tests are set in the learner's own industry and role. Certified Pryme Operator requires a written exam and a two-week practicum of real review decisions, at least 95 in every 100 correct with no missed red line, signed off by a Pryme assessor.
Questions people ask
How many reviewers should we train before go-live?
Enough to cover every shift and absence with someone who has passed sign-off. Two certified reviewers per decision type is a sensible floor, so one person's leave does not stall the held queue.
Should team leads review differently from reviewers?
They review the same items to the same standard, and they also read patterns: who never declines, which bound decides most refusals, which held items wait longest. That second task is a separate skill worth training.
What happens when the agent's certificate lapses?
Agent certificates lapse after 90 days, and a run that clears no depth drops the agent to depth 1 the same day. Work that was executed then arrives as drafts, so trained reviewers must absorb a larger queue at short notice.