Shadowing exit criteria checklist

Shadowing ends when the evidence says so, not when the calendar does. This checklist lists the conditions an AI agent must meet before leaving shadow mode for certification: enough cases, every case type seen, no recent critical misses, correct refusals and holds, stable results, fixes re-tested and sign-off from the team it shadowed.

What is the checklist for?

It decides when shadowing has proved enough. Shadowing shows how the agent behaves on live work, as it arrives, with all its mess. That is something testing cannot show, as argued in what shadowing proves that testing cannot. But shadowing can go on forever if nobody defines the finish line, and it can end too early if the team is impatient.

Exit criteria fix the finish line in advance. They are tighter than pilot success criteria, because the next step is certification and, after that, the agent acting on its own.

When should you use it?

Agree it before shadowing starts. Check progress against it weekly, using each shadow run report, and score it formally when the team believes the agent is ready.

The same discipline applies at the other end of an agent's life. When an agent is retired, its successor shadows and exits on its own evidence; the AI agent decommissioning checklist covers that handover.

How do you fill in each criterion?

• Volume sets a minimum number of cases per capability. Set it high enough that rare case types appear.
• Coverage lists the case types that must each have been seen a minimum number of times. Rare but high-risk types matter most: customers in difficulty, disputed amounts, unusual documents.
• Critical misses must be zero over a recent window, not over the whole period. Early misses that were fixed are expected.
• Holds and refusals checks the agent stops where it should. Missed holds on high-impact work are critical.
• Stability checks the agreement rate has not been rising or falling sharply in the final weeks. A still-moving rate means the agent has not settled.
• Fixes re-tested confirms every change made during shadowing was re-run against the cases that prompted it.
• Team sign-off comes from the people whose work was shadowed. They know the cases best.

The checklist

A worked sample

A hypothetical bank's business account onboarding agent, document checking capability.

The agent met almost everything, but it had seen only 12 trust structures, and the team lead flagged them. The decision was to take trusts out of the agent's scope for certification, route them to people, and continue shadowing them separately. The agent exited for every other case type.

Some readers arrive here looking for criteria for exiting a customer relationship or a third-party provider. Those are different decisions; see how to prepare customer exit decisions with an AI agent and DORA exit strategies for AI agent platforms.

Where Pryme fits

In Pryme, shadowing is the stage between building the candidate and certifying it. Exiting shadowing leads to an Attest run on your own cases, scored across seven gates, and only then is any depth above read granted. See train, shadow, certify, widen.