The rare decisions, and the habits that feel unnecessary on routine days. Within weeks, reviewers stop checking where each fact came from, forget red lines they have not met since training, decline without reasons, and lose the sequence for a lapsed agent certificate. A practicum, regular calibration and short drills are what hold those skills in place.
Why do some skills fade faster than others?
Skills fade when they are not used, and agent supervision has an awkward shape. Most held actions are routine, and most drafts are right. A reviewer can approve dozens of sound drafts in a row. The skills that matter most, catching an external instruction posing as authority or spotting a vulnerable customer the agent missed, are needed once a week or once a month. Those are exactly the skills with the least practice.
There is a second force. When an agent's drafts are usually good, people start to trust them. They read faster, check less and approve more. Nothing goes wrong for a long time, which feels like proof that checking was unnecessary. Then a rare case arrives and passes straight through.
What fades first?
| Skill | Why it fades | What holds it |
|---|---|---|
| Checking provenance on every draft | Most drafts rest on sealed records, so checking feels redundant | Calibration cases where the only problem is a declared or external source |
| Rare red lines | Met once a quarter, if that | Replaying one or two real red-line cases in each calibration |
| Declining with a reason | "No" is faster than "No, because…" | Team leads returning reasonless declines for a reason |
| Telling refused from withheld | Rarely matters on a normal day | A two-minute recap in each calibration |
| The lapse sequence | An agent's certificate lapse or a failed run is uncommon | A short drill every few months |
| Reading the record | Only needed when something is questioned | Asking each reviewer to reconstruct one decision per quarter |
The first row is the biggest risk. Provenance is the habit that protects against most fraud and most quiet errors, and it is the first to go because it is invisible when it works.
How does a practicum help?
A practicum puts real decisions straight after the formal learning, while the concepts are fresh, and has someone check each one. That does two things. It turns concepts into habits through repetition on real work. And it catches early drift: a reviewer who stops checking sources in the second week of a practicum is corrected the same day, not in a quarterly audit.
The practicum is not enough on its own. It sets the habit; something has to keep it.
How does calibration hold the skills?
Calibration is the main maintenance tool. Every four to six weeks, reviewers decide the same small set of held cases separately, then compare. A good set mixes routine cases with one or two rare ones: a held payment with only an external source, a customer showing signs of vulnerability, a refusal that someone might be tempted to work around. Where people disagree, the discussion relearns the rule.
Feed calibration with real material. Overturned approvals, complaints and near misses from the live queue are better than invented cases. An after-action review of any incident should end with at least one new calibration case.
When should you check what has faded?
At fixed points. The first month after go-live is when habits set, as the first month after go-live is the real training argues, so check then. Run a retrospective at around day 30, using the training retrospective template after an AI agent go-live, and ask specifically which checks people have stopped doing. The method is in how to run a training retrospective after an AI agent go-live. After that, look at calibration results each cycle: rising disagreement on a type of case is the first sign a skill is fading.
An example
Take a hypothetical accounts payable team of eight. After training, everyone checks the source of every supplier bank detail change. By week six, three reviewers are approving held payment drafts in under a minute. A calibration case, a payment whose only support is an emailed invoice with new bank details, splits the team: five decline, three approve. The discussion reminds all eight that an external claim cannot authorise a payment. Approval times rise slightly for a few weeks, which is the point.
Small teams feel this most, because each person covers more of the queue. How many people a single agent needs is covered in how many people need to be trained to run one AI agent.
Questions people ask
Does a yearly refresher stop forgetting?
No. A yearly renewal checks that someone still knows the rules; it does not keep the habit alive between renewals. Calibration does.
Should we test people without warning?
Some teams insert a known case into the live queue to see whether it is caught. If you do, tell people the practice exists, and use the results for coaching, not discipline.
Who should notice when skills fade?
Team leads, through calibration results and approval patterns. The guide to training teams for AI agents sets out their role.