The Kirkpatrick model is a four-level framework for evaluating training: reaction (did people find it useful), learning (did they gain the knowledge and skill), behaviour (do they work differently) and results (did the business outcome change). For training on AI agent oversight, each level can be measured in the agent's own record.
Why does the Kirkpatrick model matter for agent training?
Most training evaluation stops at level one: a feedback form after the session. Some reaches level two: a test. Very little reaches levels three and four, because behaviour and results are hard to observe in ordinary work.
Agent oversight is unusual. Every held action a reviewer decides is filed in an append-only record, with the decision, the reason and the outcome. That means levels three and four, normally the hardest, can be measured directly.
How do the four levels apply to agent oversight?
| Level | Question | What to measure for agent reviewers |
|---|---|---|
| 1 Reaction | Did people find the training relevant? | Feedback on whether cases matched their real queue |
| 2 Learning | Did they gain the skill? | Exam and lab results on held items |
| 3 Behaviour | Do they work differently? | Review decisions in the record: reasons recorded, provenance checked, red lines caught |
| 4 Results | Did outcomes change? | Fewer reversed approvals, faster clearance of held items, fewer incidents |
Level three is where training proves itself. A reviewer who passed the exam but still approves held items without recording why has learned but not changed.
What does Kirkpatrick evaluation look like in practice?
Take a mid-sized insurer that trained 30 claims handlers to review settlements drafted by an agent. Level one feedback is good. Level two: most pass the reviewer test. Level three, from the record over the next month: about a third still approve held settlements with no reason recorded. Level four: reversals of approved settlements have not fallen.
The insurer adds a short coaching round focused on recording reasons, and checks the record again a month later. Without levels three and four, the training would have been reported as a success.
How is the Kirkpatrick model different from the 70-20-10 model?
The Kirkpatrick model evaluates training after the fact. The 70-20-10 model describes how people learn in the first place: mostly from experience, some from others, and a small part from formal courses. One is for designing learning; the other is for judging whether it worked. They are often used together.
Is this the same as model risk management?
No. The word "model" means something else there. Model risk management governs the statistical or AI models a bank relies on, and asks whether an agent counts as a model under bank model risk rules. In the UK, PRA SS1/23 sets model risk management principles for banks, and training model owners for agents under SS1/23 is a training question the Kirkpatrick model can help evaluate. The argument to govern the action, not the model applies to both.
Which related terms should you know?
- Training evaluation: the wider practice the Kirkpatrick model sits inside.
- Training transfer: whether learning shows up at work, which is level three.
- Learning analytics: the data used to measure each level.
For the full approach, see training teams for AI agents.
Where the Academy fits
Pryme Intelligence Academy's Operator certificate already reaches level three. The practicum is two weeks of real review decisions, signed off by a Pryme assessor, so the credential reflects behaviour on live work, not just a test score.
Questions people ask
Which Kirkpatrick level matters most for agent oversight?
Level three. It shows whether reviewers actually check sources and record reasons on live held actions, which is what protects the business.
How soon after training should level four be measured?
Allow at least one to three months, so there are enough decisions in the record to compare with the period before training.