Why do AI agent pilots fail when nobody is trained?

Because untrained reviewers break the one part of the pilot that produces evidence. Faced with held actions they do not know how to judge, people drift to one of two extremes: they approve everything, or they reject everything. Either way the pilot measures the reviewers' habits, not the agent's quality, and the decision to scale or stop rests on noise.

What happens to an untrained reviewer?

The first day is careful. By the end of the first week, the queue is long and most held actions look alike. A reviewer who has not been taught what to check falls back on instinct, and instinct goes one of two ways.

Some trust the agent. Its drafts are fluent, its reasoning sounds right, and most of what it holds is fine. They start approving in batches. Others distrust it. One bad draft on day two is enough, and from then on they reject or redo everything by hand. Both are understandable. Neither tells the business anything about whether the agent should run.

How do the two failure modes distort the result?

Each row is a training gap, not an agent defect. Read the record of a failed pilot and these patterns are usually visible.

Why does the rubber stamp do more damage?

Because it looks like success. A pilot where reviewers approved almost everything produces a confident recommendation to scale. The agent then runs at higher depth, with fewer holds, and the mistakes the reviewers waved through during the pilot start reaching customers at volume.

The blanket refusal is wasteful but safer: the business abandons an agent that might have been fine. The rubber stamp promotes one that was never properly tested. This is the argument of your AI agent pilot failed because nobody was trained to say no.

An example: an underwriting pilot

Take a consumer lender piloting an agent that drafts credit decisions for personal loans. The underwriting team gets an afternoon's introduction to the tool and nothing on how to review its work. For four weeks the agent holds every decision above a set amount for an underwriter.

Two underwriters approve nearly all of them; one rejects most. At the review meeting, the team's overall approval rate looks healthy and the pilot is judged a success. Nobody notices that one underwriter accepted several cases where the applicant's income came only from an uploaded statement, which is declared provenance, while the bank's verified data told a different story. Under the EU AI Act, creditworthiness assessment of natural persons is high-risk, so the lender has scaled a high-risk use on evidence it cannot defend. Training the team first is covered in how to train the underwriting team to review AI agent decisions, and the wider question in can lenders use AI agents before staff are trained.

What stops a pilot failing this way?

A short, specific preparation and a few controls during the pilot:

1. Teach the review before the pilot starts. What depth the agent has, which bound held each action, how to open the sources, and how to record a reason.
2. Make reasons mandatory. A decision without a reason is not evidence.
3. Calibrate weekly. Two reviewers decide the same sample of held actions and discuss every difference; see calibration is the training nobody schedules.
4. Watch each reviewer's approval rate. Very high or very low rates are a training signal, not a result.
5. Rehearse the fallback. Agent certificates lapse after 90 days and a run that clears no depth drops an agent to depth 1 the same day; the team should know what that looks like.

Pryme Intelligence Academy's free readiness assessment, for teams of up to 25, shows a pilot team's gaps by concept before the pilot starts.

Other causes of pilot failure, such as unclear scope or poor data, are covered in why AI agent pilots fail. The full approach is in training teams for AI agents.