On three clocks at once. Sample live work every week. Retest after any change to the model, sources, brief, connections or the process around it. And run a full recertification on fresh cases at least every 90 days, more often for agents that move money or act on customers. Higher depth and higher consequence mean tighter clocks.
Why is a single annual test not enough?
Because the things that shape an agent's behaviour move far more often than once a year. Providers update models. Policies are rewritten. A connected system adds a field. A new product brings cases nobody tested. An agent that passed in spring may be working on a different reality by summer, with nothing in its configuration having changed.
Annual testing suits software that does the same thing every time. Agents need evidence that is recent enough to describe the agent you are actually running.
What are the three clocks?
| Clock | What happens | Who runs it | What it catches |
|---|---|---|---|
| Weekly | A reviewer scores a fresh sample of live work against an answer key | Reviewer, with the process owner | Gradual drift, reviewer slippage, new case types |
| On change | The full case set is rerun before the change carries depth | Platform owner and process owner | Breakage from model, source, brief or connection changes |
| Every cycle | Full recertification on a refreshed case set | Process owner | Everything above, plus gaps in the old case set |
The weekly clock is light and continuous. The change clock is triggered by events. The cycle clock is a hard backstop: certificates lapse after 90 days, so an agent cannot coast on old evidence even if nothing seems to have changed.
What should shorten the interval?
- Higher depth. An agent that executes or moves money needs more frequent evidence than one that drafts.
- Higher consequence. Customer outcomes, regulatory decisions and payments warrant tighter clocks.
- Frequent change. Processes whose policies or products change monthly need more frequent retests.
- Warning signals. A rising refusal rate, a growing held queue or more complaints should trigger a retest now, not at the next scheduled date. Our piece on handling an agent that refuses too often covers one such signal.
Volume can lengthen the weekly clock's sample, but not the cycle clock's interval.
What does a retest calendar look like?
Say a consumer lender runs two agents. One drafts collections letters at depth 2. The other changes payment dates within the customer's limits at depth 3.
For the letters agent, a reviewer scores 30 letters a week, and full recertification runs every 90 days. For the payment-date agent, the weekly sample is larger and includes every case where a date moved by more than a month. Its recertification runs every 60 days, by the owner's choice, because a wrong date can cause a missed payment and a fee.
Midway through one cycle, the lender launches a new loan product. Both agents are retested on cases from the new product before they handle it, and the payment-date agent handles that product at depth 2 until the next full recertification includes it. One month, the held queue for payment-date changes doubles. The owner orders a retest that week rather than waiting, and finds a change in how the loan system reports arrears. The reasoning behind that kind of decision is in why the held queue tells you more than any dashboard.
Which controls make retesting dependable?
- A dated calendar for each agent, with recertification dates staggered across agents.
- A change log that automatically triggers a retest when model, sources, brief or connections change.
- A refreshed case set for each cycle, including recent cases, so the agent is not tested on the same material every time.
- Consequences that apply on their own. A run that clears no depth drops the agent to depth 1 the same day; a lapsed certificate does the same.
- Filed results for every retest, linked to the agent's version.
For the formal cycle in more detail, see how often AI agents should be recertified. The wider sequence is in our guide to deploying an AI agent.
Questions people ask
Is a weekly sample a retest?
It is a lighter check. It tells you whether live work is holding up, but it does not replace a full run against a fixed floor, which is what supports the agent's depth.
What if retests keep failing on the same gate?
Then the fix is in the brief, sources or bounds for that capability, or the capability should stay at a lower depth. Repeated failure is evidence too.
Can retests run automatically?
The run itself can be scheduled, and the agent can produce its outputs without a person. Writing answer keys and deciding disputed cases still needs people.