Hold it four to six weeks after go-live, once reviewers have decided enough real cases. Compare what people were taught with what the record shows they actually did: corrections, missed holds, slow decisions, questions they escalated. Agree what worked, what did not, and three changes for the next agent's training plan. Then assign owners and dates.
What is a training retrospective, and who owns it?
A training retrospective asks one question: did the training prepare people for the agent they actually got? It is a form of after-action review, focused on competence rather than on the agent's performance. The agent's own results are reviewed elsewhere, by its owner and through certification.
The learning lead for the deployment owns it. Attendees are a few reviewers, a team lead, the trainer or champion who delivered the training, and the agent's owner. Keep it to eight people or fewer.
Timing matters. Too early and reviewers have seen only easy cases. Too late and they have forgotten what training did not cover. The first month after go-live is the real training, so the retrospective comes just after it.
How do you run it?
- Agent, depth 1: compiles the evidence pack. Held items by type, decisions changed in sampling, corrections by cause, items that waited longest, and any refused action a reviewer tried to override.
- Learning lead: gathers reviewer input through a short form: which held cases surprised you, what you wished you had practised, what you looked up most.
- Group: compares training to reality (30 minutes). For each training module or session, did the cases reviewers met match what was taught?
- Group: examines three specific cases (30 minutes). One handled well, one handled badly, one that caused confusion. Trace each back to training.
- Group: names what worked and what did not (15 minutes). Specific and evidenced, not impressions.
- Learning lead: records three to five changes for the next agent's training, each with an owner and a date.
- Agent owner: notes any agent changes that came up, to be handled through the brief and certification, not through training.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Evidence pack | Summarises held, corrected and sampled items | 1 | Learning lead checks it |
| Reviewer input | Nothing | none | Reviewers |
| Compare | Nothing | none | Group |
| Trace cases | Retrieves the record for each | 1 | Group interprets |
| Name and record changes | Nothing | none | Learning lead |
| Agent issues | Nothing | none | Agent owner |
Which questions should the retrospective answer?
- Did people know what each verdict meant? If reviewers treated held items as errors, or tried to override refused actions, the lessons on the decision chain did not land.
- Did practice cases match real ones? If the training used tidy examples and real cases arrived with conflicting sources, practice needs messier material.
- Were people ready before the depth step? If the agent moved from drafting to executing routine work while reviewers were still learning to read held items, the schedule was wrong. Compare against how AI agent training should be scheduled around a go-live.
- Did calibration happen? And did its standards reach the agent's brief?
- Who was missing? Night shift, part-timers, a team that joined late.
- Were readiness criteria right? If go-live passed its readiness check but reviewers struggled anyway, the criteria measured the wrong things.
What does a good finding look like?
A useful finding names the gap, the evidence and the fix. For example: "Reviewers released four held invoices with changed bank details in week two. Training covered source holds in one slide and no practice. Next time, add a seeded exercise on source holds before depth 3." Compare that with "people needed more time on holds", which nobody can act on. Write every finding in the first form.
Which controls make the retrospective useful?
Evidence before opinion. The pack is circulated a day ahead. Evidence: the pack, filed with the notes.
No blame. Cases are discussed without reviewer names. Evidence: anonymised case references.
Changes have owners. Evidence: the action list with names and dates.
Agent issues routed out. Problems with the agent go to its owner and are fixed through the brief and re-certification. Evidence: the referral.
Follow-through checked. The next deployment's kickoff starts by reviewing these actions. Evidence: the kickoff notes.
What should you measure?
- Actions completed before the next go-live.
- Practicum or sign-off results for the next cohort compared with this one.
- Early sampling accuracy for the next agent's reviewers in their first month.
- Repeat findings. If the same gap appears in two retrospectives, the change did not work.
Where does it usually go wrong?
It becomes a project post-mortem. Scope, budget and vendor discussions crowd out training. Keep a separate project review.
Only managers attend. Reviewers know what training missed. Without them, the room guesses.
Findings are too general. "More practice" is not an action. "Add ten cases with conflicting sources to the calibration pack" is.
The agent gets blamed for training gaps. Some reviewer errors come from a confusing draft; most come from not knowing what to check. Be honest about which.
Reviewers feel judged. If people think the retrospective will be used against them, they stop admitting what confused them. Say at the start that findings are about the training, not individuals, and keep that promise when the notes are written up.
Nothing changes. The training retrospective template ends with an action table for that reason. For the wider programme, see training teams for AI agents.
Where the Academy fits
Pryme Intelligence Academy gives retrospectives harder evidence to work with. The admin view shows each person's progress, test results by module, certificates and expiry dates, so you can see whether reviewers who struggled had finished the human review module or skipped it. The Operator practicum produces two weeks of assessed decisions per person, which is a clear baseline for comparing cohorts.
Questions people ask
Should the vendor or implementation partner attend?
Only for the part about agent behaviour, if at all. The retrospective is about your people's readiness, and reviewers speak more freely without outsiders.
How long should it take?
Ninety minutes is enough with a good evidence pack. Without one, it will take longer and achieve less.
What if go-live went smoothly?
Run it anyway. Smooth go-lives hide lucky gaps, and the next agent may have a higher depth or messier sources.