How do you run an AI agent in shadow mode?

Give the agent live work, read-only credentials and nowhere to send its output. It processes the same cases your people handle, at depth 1, and files what it would have done. People keep working without seeing its answers. Afterwards, a reviewer compares the two case by case and decides who was right where they differ.

What makes a shadow run useful rather than decorative?

Shadow mode, explained in our entry on what shadow mode is, is the only stage where you see the agent on your real, current work with no risk to customers. Its value depends on three conditions.

First, the agent must be technically unable to act. Read-only credentials, not a setting the agent is told to respect. Second, the people doing the work must not see the agent's proposals. Once they do, they anchor on them, and you are no longer comparing two independent judgements. Third, disagreements must be reviewed by someone who can say which answer was correct, because the person is not always the one who was right.

How do you set it up?

1. Platform owner: issues read-only access to the systems the agent needs, and no write or send rights. Evidence: the access grant, listing each system and its permission.
2. Process owner: picks the slice and the period. One case type, for at least one full business cycle. Evidence: the scope note.
3. Agent, depth 1: processes each case as it arrives and files its proposed outcome, its reasoning and the sources it used. Evidence: one filed proposal per case.
4. Analyst: matches proposals to what people did, using the case reference. Evidence: a paired list.
5. Reviewer: decides each disagreement. Agent right, person right, both acceptable, or both wrong. Evidence: a reason per disputed case.
6. Process owner: reads the report and decides whether to certify for depth 2, extend the shadow or stop. Evidence: the decision, filed with the shadow run report.

What should the agent do and what should people do?

What does a shadow run reveal in practice?

Say a property insurer shadows an agent that codes incoming invoices from repair contractors to the right claim and cost category. Over six weeks it sees 3,000 invoices that the claims payments team also handles.

The pair agree on most of them. The disputed cases split three ways. A cluster of contractors use their own job numbers instead of claim numbers, and the agent guesses the claim from the address, sometimes wrongly; the brief is changed so these are escalated. In a smaller group the agent was right and the team had coded storm repairs to the wrong category for months, which becomes a training point for staff. And a handful of invoices arrived as emailed attachments that could not be matched to any supplier record, which the agent correctly refused to code.

None of this would have shown up in a test set built from clean, closed claims. That is the point of the step covered in shadowing an AI agent before it goes live.

Which controls keep a shadow run safe?

• Technical, not instructed, read-only access. If the agent could send, the run is not a shadow run.
• Separate storage for proposals, so they do not leak into case notes.
• Data protection review in advance. A shadow agent still processes personal data, so the purpose and lawful basis must cover it.
• A fixed end date and exit criteria, so the shadow does not drift on for months.
• Filed proposals kept as the baseline for later drift checks.

The rest of the sequence, from shadow to certification and widening, is in our guide to deploying an AI agent.