Coach reviewers from the record, one case at a time. Pull a sample of the held actions a reviewer decided, compare each decision with the right answer, name the specific gap, agree one change, and check the next sample for it. This template turns that into a plan per reviewer with cases, gaps, actions and follow-up dates.
Why coach from the record rather than from opinion?
A reviewer is the control on an AI agent's held work, not a courtesy check. The argument is made in the reviewer is the control, not the agent. If the control drifts, the agent's errors pass through untouched, and nobody notices until a customer or auditor does.
The record makes coaching concrete. Every verdict is filed: what the agent drafted, which sources it cited and their provenance, what the reviewer did, and when. A coach can open the exact case and ask, "What did you check before releasing this?" That is far more useful than a general sense that someone is "a bit quick". For a refresher on coaching as a discipline, see what is coaching?.
What gaps should the plan look for?
Most reviewer errors fall into a small number of patterns. Name them in the plan so coaching is consistent across coaches.
- Rubber-stamping: approving without opening the cited sources.
- Trusting the wrong provenance: treating an external claim, such as an emailed invoice or a forwarded letter, as if it were sealed or signed.
- Over-refusal: rejecting sound drafts out of caution, which pushes work back to people and teaches the team nothing.
- Editing into approval: rewriting a weak draft until it passes instead of refusing it with a reason the builder can use.
- Missing a red line: releasing something that should never have passed, the most serious gap.
- Thin reasons: refusing or approving without a reason that someone else could follow later.
The role itself is described in what is an AI agent reviewer? and, from a different angle, in what is an agent reviewer?.
How do you fill in the plan?
- Set the period and sample. For example, ten decisions a fortnight, chosen at random, plus every decision another reviewer later overturned.
- Record each case: reference, held action, what the reviewer did.
- Record the expected decision and who confirmed it, usually a senior reviewer or the team lead.
- Name the gap type from the list above, or "none".
- Agree one action per gap type, not per case. Three rubber-stamp cases need one action, not three.
- Set the check: the next sample will look for that gap specifically.
- Record the outcome at the follow-up: closed, continuing or escalated.
The template
Header
- Reviewer name and queue
- Coach name
- Period covered
- Sample size and how chosen
- Reviewer's certificate and expiry
Case log
| Case ref | Held action | Reviewer's decision | Expected decision | Confirmed by | Gap type | Note |
|---|---|---|---|---|---|---|
Actions
| Gap type | Agreed action | Check at next sample | Follow-up date | Outcome |
|---|---|---|---|---|
A worked example
A hypothetical contact centre's reviewer, Reviewer A, works the held refunds queue. The coach samples ten decisions over a fortnight. Seven are correct. Three are not.
| Case ref | Held action | Reviewer's decision | Expected decision | Confirmed by | Gap type | Note |
|---|---|---|---|---|---|---|
| R-104 | Refund of 180 | Approved | Refuse: proof of return was an external photo, not the returns system | Team lead | Wrong provenance | Did not open the returns source |
| R-117 | Refund of 60 | Approved | Approve | Team lead | None | |
| R-122 | Refund of 240 | Approved | Refuse: same as R-104 | Team lead | Wrong provenance | Same pattern |
| R-131 | Goodwill credit of 20 | Refused, no reason | Approve | Team lead | Thin reason, over-refusal | Customer re-contacted |
| Gap type | Agreed action | Check at next sample | Follow-up date | Outcome |
|---|---|---|---|---|
| Wrong provenance | Open the returns system before any refund; rehearse three cases with the coach | All refunds over 100 in the next sample | Two weeks | Closed: no repeats |
| Thin reason | Every refusal carries a reason a builder could act on | All refusals in the next sample | Two weeks | Continuing |
The provenance gap was the serious one: twice Reviewer A trusted a photo the customer sent over the returns system, which is the source of record. Coaching fixed it in one cycle because it was shown on real cases.
Coaching needs protected time, so build it into the reviewer rota for a contact centre rather than squeezing it between shifts. The broader map of what each role needs to know is in AI agent skills by role.
Questions people ask
Who should coach reviewers?
A senior reviewer or team lead who decides the same queue and holds a current certificate. Coaching from someone who has never worked the queue rarely lands.
How do we keep coaching from feeling punitive?
Sample everyone, including senior reviewers, and coach patterns rather than single mistakes. A missed red line still needs a firm response, but most gaps are habits, not misconduct.
What should happen after repeated gaps?
Narrow the reviewer's limit or move them to a lower-risk queue while coaching continues. Restore the limit once two clean samples show the gap is closed.