Hold the retrospective four to six weeks after go-live, with the people who decided the agent's held actions in the room. This template sets the evidence to bring from the record, five questions to work through, and a decision log with owners. It ends with a sample retrospective for a lending team.
Why run a training retrospective, and when?
Training before go-live is a forecast. It predicts which cases people will meet and how they should decide them. The first weeks of live work test that forecast. Some cases arrive that nobody trained for; some red lines prove too loose or too tight; some people struggle in ways the practicum did not reveal. The case that this is when the real learning happens is made in the first month after go-live is the real training.
A retrospective captures those lessons while they are fresh, then feeds them into the course, the red lines, calibration and the agent's own brief. It is a form of after-action review: what was supposed to happen, what did happen, why the difference, and what to change. Run it four to six weeks after go-live, before habits set and before the next team starts training on the old material.
How do you prepare and run it?
The training owner and the agent owner prepare it together. Collect the evidence a week ahead: measures from the record, feedback forms, calibration logs and any incident reports. Invite reviewers, team leads, the agent owner, a builder and someone from risk. Allow 90 minutes.
Compare against what was planned. If the team used a go-live readiness check, bring it: the retrospective tests whether readiness was real. If training was scheduled tightly around go-live, as in how to schedule AI agent training around a go-live, ask whether the timing worked. The facilitation method is set out in how to run a training retrospective after an AI agent go-live.
The retrospective template
Part 1: Evidence (prepared in advance)
| Measure | Planned or expected | Actual |
|---|---|---|
| People on the approval rota with a current certificate | ||
| Held actions decided | ||
| Approvals later overturned | ||
| Missed red lines | ||
| Median time from hold to decision | ||
| Calibration agreement on shared cases | ||
| Incidents involving a human approval | ||
| Agent certificate events (lapse, failed run, depth change) |
Part 2: Five questions (in the session)
- Which cases did training prepare people well for?
- Which cases arrived that training did not cover?
- Which red lines worked, and which were unclear, too tight or too loose?
- Where did people feel least confident, and does the record agree?
- What happened when the agent stopped, lapsed or was held back, and did people know what to do?
Part 3: Decision log
| Finding | Decision | Change to (course / red lines / calibration / agent brief / rota) | Owner | Due |
|---|---|---|---|---|
Part 4: Close
- What the next team to train should be told
- Date of the next calibration session
- Date of the next retrospective, if needed
Worked example
An illustrative building society's mortgage lending team holds its retrospective five weeks after an agent began preparing completion packs at depth 2 and drafting disbursement instructions, all held.
- Evidence: 9 of 9 approvers certified; 420 held actions decided; 11 approvals overturned, 8 of them in the first fortnight; no missed red lines; median decision time of three hours, against two expected.
- Question 2: the course had no cases on drawdowns split between two solicitors' accounts, which arrived 14 times. Reviewers improvised, consistently but slowly.
- Question 3: the red line "never pay to changed details without verification" was clear; the red line on "conditions precedent met" was read differently by two reviewers, which explains most early overturns.
- Question 5: when the agent's certificate lapsed on day 19, team leads knew the procedure but reviewers did not; work stalled for half a day.
Decisions: add split-payment cases to the course and to the next calibration (owner: training lead, two weeks); rewrite the conditions red line with examples, signed by the head of lending (one week); a 20-minute lapse briefing for all reviewers (team lead, within a month); ask the agent owner whether split payments should always be held (agent owner, next review).
Where the Academy fits
Pryme Intelligence Academy can absorb some findings directly: module tests are set in the learner's own industry and role, and the Operator modules include what to do when something goes wrong. Findings about the business's own red lines and cases belong in its internal material. The guide to training teams for AI agents shows where each kind of change fits.
Questions people ask
Who should facilitate the retrospective?
Someone who did not design the training, so findings are not defended. A team lead from another area, or a risk partner, works well.
What if the evidence shows the training worked?
Say so, and still ask question 2. There are always cases the course missed; finding them early is cheaper than finding them in an incident.
Should the agent's own performance be discussed?
Only as it affects people's decisions. Agent behaviour belongs in its owner's review; note the issue and pass it on.