A shadow run report summarises a period in which an AI agent worked on live cases alongside your team without its outputs being used. It compares what the agent would have done with what people actually did, classifies every difference by severity and cause, and recommends whether the agent is ready to certify, needs changes, or should keep shadowing.
What does a shadow run show?
How the agent handles the real mix of work: the odd cases, the incomplete forms, the customer who asks three things at once, the policy exception that only applies on Fridays. Test sets never capture all of it. Shadow mode does, because the agent sees exactly what your team sees, as it arrives.
Do not confuse this with shadow AI, which means staff using unapproved AI tools. A shadow run is approved, scoped and recorded. The method for setting one up is in how to run an AI agent in shadow mode.
When should you use it?
At the end of each shadow period, typically weekly during shadowing and in full at the end. The final report is the main input to certification. If certification later fails, the failed certification run report should refer back to it to see whether shadowing missed something.
How do you fill in each field?
- Cases counts every case the agent shadowed, by type. Exclusions are listed with reasons.
- Agreement is where the agent's proposed action matched the person's, within tolerance.
- Differences are classified by who was right: agent wrong, person wrong, both acceptable, unclear. Shadow runs regularly find human errors; record them, but do not let them flatter the agent.
- Severity uses a fixed scale. Critical means the agent's action, had it been taken, would have caused harm or breached a rule.
- Cause of each agent error points to a fix: brief, source, bound, data, or a case type the agent should not handle.
- Bound behaviour records whether the agent refused and held where it should have. An agent that matches people but never refuses is a warning, not a success.
A critical difference in shadowing is a near miss. Record it in the format of the AI agent incident report, short version.
The template
| No. | Section | What to write |
|---|---|---|
| 1 | Agent, capabilities, period | Name, ID, capabilities shadowed, dates |
| 2 | Cases shadowed | Count by type; exclusions |
| 3 | Agreement | Count and description of tolerance |
| 4 | Differences by who was right | Agent wrong, person wrong, both acceptable, unclear |
| 5 | Agent errors by severity | Critical, major, minor |
| 6 | Agent errors by cause | Brief, source, bound, data, out-of-scope type |
| 7 | Refusals and holds | Count; correct, missed, unnecessary |
| 8 | Critical cases | Each described, with near-miss reference |
| 9 | Changes made during period | Change IDs |
| 10 | Recommendation | Certify, change and continue, extend, stop |
| 11 | Sign-off | Agent owner, quality lead |
A filled example
A hypothetical insurer's motor claims triage agent, four-week shadow run.
| No. | Entry |
|---|---|
| 2 | 3,200 new claims; 40 excluded (commercial fleet, out of scope) |
| 3 | 2,880 matched the handler's triage category and priority |
| 4 | Agent wrong: 190. Handler wrong: 60. Both acceptable: 60. Unclear: 30 |
| 5 | Critical: 2. Major: 38. Minor: 150 |
| 6 | Brief: 120 (priority rules for glass claims). Source: 40 (policy system showed lapsed cover for renewed policies). Out-of-scope type: 30 (claims involving injury) |
| 7 | 210 holds, 200 correct, 10 unnecessary. Missed: 2 (both critical) |
| 8 | Both critical cases: injury mentioned in free text, agent triaged as damage only. Near-miss reports NM-14 and NM-15 |
| 9 | CHG-301 (glass priority rules), CHG-302 (injury keywords route to handler), data fix for renewal status |
| 10 | Change and continue for two more weeks, then certify |
The 60 handler errors are worth reading too. Several came from the same renewal-status data problem the agent hit. The shadow run found a fault that was affecting people before any agent arrived.
Where Pryme fits
In Pryme, a shadow run produces verdicts for every case without acting, so agreement, holds and refusals are counted from the record. The final report feeds Attest, which certifies each agent on your own cases before any depth is granted. See train, shadow, certify, widen.
Questions people ask
How many cases does a shadow run need?
Enough to see every case type the agent will handle, several times, including rare ones. For most processes that means several weeks of normal volume rather than a fixed number.
Should human errors found in shadowing be reported?
Yes, to the team manager, as process findings. They are valuable, and finding them builds trust in the agent. Keep them separate from the agent's own error count.
Can the agent's outputs be used during shadowing?
No. If anyone acts on the agent's output, it is no longer shadowing. Keep the agent's work invisible to customers until certification.