What is a retention check?

A retention check is a short test given some time after training, weeks or months later, to find out what people still know and can apply. It differs from a test at the end of a course because it measures what survived. For teams overseeing AI agents, it shows which rules have faded before the gap reaches the review queue.

Why does a retention check matter?

Most training is measured on the day it ends, when knowledge is at its peak. That result says little about the reviewer facing a held payment three months later. Retention checks catch the decay that end-of-course tests miss.

Decay is uneven. People remember what they use every day and forget what is rare. A reviewer who sees hundreds of routine drafts will remember the verdicts. The same reviewer may have forgotten that an external document cannot authorise a payment, because the case has not come up since training. A retention check aimed at the rare cases tells you who needs a refresher on what.

How does it work in practice?

A retention check is small and targeted:

• Timing: a few weeks after training, then at intervals, or before a change in the agent's depth.
• Content: weighted towards rare but costly cases, such as withheld actions, red lines, provenance of sources and what happens when an agent's certificate lapses.
• Format: five to ten questions, set in the team's own cases, taking a few minutes.
• Use: results go to the team lead by topic, not as a ranking of people.

The output shapes the next refresher. If most of the team misses the question on lapsed certificates, that is next week's ten-minute session.

An example

A consumer lender trained its underwriting support team on an agent that gathers documents and drafts affordability summaries at depth 2. Six weeks later, the team takes an eight-question retention check. Scores are high on reading the summary and low on one question: whether a payslip uploaded by the applicant is declared or signed. The team lead runs a short session on provenance, and the next check shows the gap closed. The work itself is covered in whether an agent can check affordability for a loan.

How is a retention check different from a knowledge check?

A knowledge check is given during or just after learning, to confirm the point has landed. A retention check is given later, to confirm it has stayed. The questions can look alike. What differs is timing and purpose: a knowledge check guides learning in the moment; a retention check guides what to refresh.

The term is also used in other senses. It has nothing to do with checking how long records are kept, which is covered in applying record retention schedules with an agent, or with customer retention, the subject of a retention offer matrix.

Which terms are related?

• Knowledge check: the immediate version.
• Spaced repetition: revisiting material at intervals so it is retained.
• Refresher training: the response when a retention check shows gaps.

For the wider approach, read training teams for AI agents.