A prepared test shows how an agent handles the cases someone thought to include. Shadowing shows how it handles the cases that arrive, in their real mix, mess and volume, rare ones included. Running the agent alongside people on live work, without letting it act, is the only way to see that before it matters.
What is shadowing, precisely?
The agent receives the same live work as the team, does everything it would do, and stops short of acting. Its proposed outcome is recorded next to the outcome the person actually produced. Nothing the agent proposes reaches a customer or a system of record. At the end of each day, the two sets of outcomes are compared and every difference is examined.
This differs from A/B testing, where some customers receive one version and some another, and outcomes are compared. In A/B testing the new version acts on real customers. In shadowing it does not. That makes shadowing the safer way to gather live evidence before an agent has earned any authority.
What does testing prove well?
Testing, meaning a prepared set of cases with known correct answers, is good at specific questions. Does the agent refuse what it should? Does it stay within scope? Does it handle a deliberately tricky case correctly? Does its behaviour change after a model update? Tests can be designed to probe each bound precisely, and they can be rerun exactly, which makes them the right basis for certification and regression checks.
Their limit is that someone chose the cases. However carefully built, a test set reflects what its authors expected the work to contain.
What can only shadowing show?
Five things, each of which a test set tends to miss.
| What shadowing shows | Why a test set misses it |
|---|---|
| The real mix of cases | Test sets over-represent interesting cases and under-represent the dull majority |
| The real condition of inputs | Live documents are skewed scans, partial forms, forwarded chains; test inputs are cleaner |
| Rare cases nobody predicted | You cannot include a case you did not know existed |
| Behaviour at real volume and timing | Peaks, end-of-month rushes and overnight batches change what arrives |
| Disagreement with how people actually work | Tests encode the procedure; people follow practice, which may differ |
The last row is often the most revealing. When the agent and the person disagree, sometimes the agent is wrong. Sometimes the person is following an unwritten rule the procedure omits. Sometimes the person is wrong, or two people handle the same case differently. Shadowing surfaces all three, and the second and third are things you would never learn from a test.
What does shadowing reveal about exceptions?
It shows which cases the agent cannot process at all, and how many of them there are. A test set may include a handful of unusual cases; live work will show the full range and the real frequency. Some will be cases the agent should never take, which tells you where to set its scope. Some will be cases it could handle with a better source or a clearer procedure. Some will simply need a person.
Knowing the size and shape of that residue is essential for planning. If a quarter of the work turns out to be exceptions, the team that handles them needs to be sized for it. Our guide on handling the exceptions an AI agent cannot process covers how to route them.
How long should shadowing run?
Long enough to see the work's natural cycles and enough of its rare cases. For a process with a monthly rhythm, at least one full month, ideally two. For a process with seasonal peaks, ideally including one. Shadowing for a week during a quiet period proves very little.
Set exit criteria in advance, before anyone sees the results: the level of agreement required on each capability, the categories of disagreement that must be explained, the share of cases that must fall within the agent's scope, and how many rare cases must have been seen. A shadowing exit criteria checklist gives a structure. Without criteria set in advance, shadowing tends to end when the team feels confident, which is not the same thing.
How does shadowing relate to certification?
They answer different questions and both are needed. Certification replays the agent against real cases from the business's own operation and scores each answer against a floor, gate by gate. It produces a defensible pass or fail for each depth. Shadowing on live work checks that the cases used for certification really represent what arrives, and catches what the replay set did not contain.
Neither is the same as certifying an organisation's management system. An ISO/IEC 42001 certificate shows that an organisation runs a managed approach to AI; it does not prove how a particular agent behaves on particular work. Our piece on what ISO/IEC 42001 certification proves draws that line.
Shadowing also feeds customer outcome testing. Comparing agent and human outcomes on the same live cases is a direct way to check whether customers would have been treated well; our playbook on running customer outcome testing with an AI agent builds on this.
What is the trade-off?
Shadowing is expensive. People do all the work as before, and someone must compare outcomes and investigate every difference. It delays the point at which the agent saves any time. Teams under pressure to show results often cut it short or skip it.
What it buys is evidence that cannot be obtained any other way: proof that the agent works on your work, as it really arrives. Testing tells you the agent does what you designed it to do. Shadowing tells you whether what you designed matches reality. Before your next agent takes any action, run it in shadow on live work for at least one full cycle, with exit criteria written down first. The deployment sequence this belongs to is set out in our guide to deploying an AI agent: train, shadow, certify, widen.
Where Pryme fits
In Pryme, shadowing sits between building an agent and certifying it. Attest then replays the agent against real cases from the customer's own operation and scores every answer against a floor across seven gates: grounded, refuses, depth, escalates, scope, adversarial and stable. Depth is set by that evidence, capped by the role's ceiling.
Questions people ask
Can shadowing be done on historical cases instead of live ones?
Replaying historical cases is valuable and is how certification works, but it is not quite shadowing. Live shadowing shows current inputs, current sources and current volume, which historical replays may not reflect.
Does shadowing need customer consent?
Shadowing processes customer data, so data protection rules apply, including having a lawful basis and limiting use to what is needed. Because nothing the agent proposes reaches the customer, it does not usually change the customer's experience. Confirm the position with your data protection lead.
What should happen to disagreements found in shadowing?
Each one should be classified: agent wrong, procedure gap, person inconsistent, or a real judgement call. Agent errors feed into fixes and re-testing; procedure gaps go to the process owner; inconsistencies between people go to calibration.