Run the agent on the same live work your team handles, with every write and message blocked at the platform. Compare its output with what people actually did, without letting staff see the agent's answer first. A senior reviewer settles each disagreement, and the process owner decides go-live from the shadow report.
What does shadowing prove that testing does not?
Certification replays the agent against past cases, chosen and labelled. That shows it can do the job on the kinds of work you expected. Shadowing runs it on today's work, as it arrives, with all its mess: new case types, incomplete data, systems that respond slowly, customers who write in unexpected ways. It also shows how the agent's output compares with what your people decide on the same items, which past cases cannot.
Testing before go-live, in the wider sense, is covered in how to test an AI agent before it goes live. The concept itself is explained in what shadow mode is. This playbook is the working method.
The process owner owns the shadow run and the go-live decision. A senior reviewer, someone experienced in the work who did not build the agent, settles disagreements. The team does the work as normal.
What is the method?
- Agent operations: blocks all effects. The agent reads live systems but every write, message and file save is refused at the platform, not left to the agent's instructions. Test this before starting. Evidence: the configuration and the test showing refusal.
- Process owner: sets the scope and length. Which work types, how many items of each are needed, and the latest end date. Evidence: the shadow plan.
- Agent, depth 1: processes each live item and records what it would have done: the action, the draft, the verdict it would have reached and the bound that would have decided.
- Team: does the work as normal, without seeing the agent's output. Showing staff the agent's answer first contaminates the comparison: they will tend to agree with it.
- Agent, depth 1: pairs each of its outputs with the person's action once the person has acted.
- Analyst: classifies each pair: same outcome; different but both acceptable; agent wrong; person wrong; unclear.
- Senior reviewer: settles every "different" and "unclear" pair, and confirms a sample of the "same" pairs. Evidence: the adjudication record.
- Agent owner: feeds findings back into the brief, sources or bounds. Changes restart the count for affected work types.
- Agent, depth 2: drafts the shadow run report. Process owner reviews and decides: go live on a slice, extend the shadow, or stop. Evidence: the signed report, using a format like the shadow run report template.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Block effects | Nothing | none | Configuring, testing |
| Plan | Nothing | none | Setting scope and length |
| Process | Records intended action | 1 | Nothing |
| Do the work | Nothing | none | Team, blind |
| Pair | Matches outputs | 1 | Nothing |
| Classify | Nothing | none | Analyst |
| Adjudicate | Nothing | none | Senior reviewer |
| Feed back | Nothing | none | Agent owner |
| Report | Drafts | 2 | Process owner decides |
Why must the comparison be blind?
Because people anchor. A handler who sees that the agent proposed refusing a claim looks for reasons to refuse it. The shadow then shows close agreement, which is mostly the agent agreeing with itself through the person. Keep the agent's outputs out of the team's tools until they have acted, and pair afterwards.
What do you do when the person was wrong?
Shadowing regularly finds that people make mistakes the agent did not. That is useful to know, but it is not evidence for the agent's readiness on its own. Count "person wrong" separately, and look at it in two ways: as a quality finding for the team, and as a reminder that a high agreement rate with people is not the same as being right. The adjudicated outcome, not the person's action, is the benchmark.
How long should shadowing run?
Until each work type in scope has enough adjudicated items to support the decision, and through at least one peak, such as a month-end or a busy Monday. For high-volume work that may be two weeks; for rarer work, longer. Set a latest end date in the plan. If a work type does not reach enough items by then, it stays out of the first live slice.
More detail on the mechanics, including how to route outputs and what to block, is in how to run an AI agent in shadow mode.
What does the shadow record contain?
- The plan: scope, sample targets, end date.
- Proof that effects were blocked.
- Every pair: agent output, person's action, classification, adjudication.
- Changes made to the agent during the run, and which counts they reset.
- Results by work type, not only overall.
- The process owner's decision and its reasons.
This record becomes part of the agent's file, between certification and go-live in the sequence of train, shadow, certify, widen.
Where does it usually go wrong?
Effects leak. A connector was missed, and the agent sends a real email or writes a real note during shadow. Block at the platform and test the block.
Staff see the agent's output. Through a shared screen, a dashboard or curiosity. Agreement figures rise and mean nothing.
Only agreement is counted. "The agent agreed with staff on most items" ignores whether the disagreements were the serious ones. Look at what kind of items the agent got wrong.
Changes are made mid-run without resetting. The brief is improved on day eight, and days one to seven are still counted. Track versions.
Shadow never ends. It feels safe, so it continues. The team does double work and loses patience. Keep the end date.
Questions people ask
Does shadowing need customer consent?
The agent reads data the firm already processes for the same purpose, and takes no action. Check with your data protection team, especially if the agent's provider processes the data outside your own systems.
Should the agent shadow every work type at once?
Start with the work types you expect to go live first. Shadowing everything spreads review effort thin.
Can shadowing replace certification?
No. Certification tests the agent against known cases with known answers. Shadowing tests it against live work. You need both.