Most failed AI agent pilots did not fail because the agent was weak. They failed because the pilot team approved nearly everything, or refused nearly everything, and left no reasons. A pilot exists to find where the agent's drafts break on your own cases. Only reviewers trained to refuse, and to say why, produce that evidence.
What is a pilot actually for?
Not to prove the agent works. A pilot is a structured test of how the agent performs on the business's real cases, so the owner can decide what depth it may earn and what must change before it does. The output is evidence: which drafts were right, which were edited, which were refused, and why.
That evidence comes from people. During a pilot the agent usually runs at depth 1 or 2, reading and drafting while a person approves, edits or refuses each action. Every decision is filed. When the pilot ends, the record is the answer to the question the pilot was asked. If the decisions in it are careless, the answer is worthless, however good the agent was.
Why do pilot teams fail to say no?
Four reasons, all about training rather than character.
They think success means approval. Pilots have sponsors, and sponsors want good news. A pilot cohort that has not been told otherwise assumes that a high approval rate is what everyone wants, and delivers one.
They do not know they are allowed. Many pilot reviewers have never been told that refusing a draft is a normal, expected outcome. They edit lightly and approve, because refusing feels like accusing the project of failing.
They do not know the grounds. Refusing well requires knowing what makes a draft wrong: a fact from an external source used as if it were sealed, an action outside the agent's role scope, a breach of the business's own limits, a red line. Untrained reviewers sense that something is off but cannot name it, so they let it go.
Refusal costs effort. A refusal with a written reason takes longer than an approval. Under queue pressure, the cheaper choice wins.
The detailed mechanics of untrained pilots are covered in why AI agent pilots fail when nobody is trained.
What does a pilot without refusals look like?
Excellent, at first. The report shows near-total approval and quick decisions. The sponsor approves wider rollout. The agent is put forward for certification.
Then the real cases arrive. Attest replays the agent against cases from the business's own operation and scores each answer against a floor, across seven gates that must each clear on their own, including refusal and escalation. The agent may fail gates the pilot never tested, because nobody in the pilot ever pushed back. Or it passes, reaches depth 3, and its first serious error in live work is one the pilot team saw, approved and did not mention. Either way, the business learns in production what it should have learned in the pilot.
Is refusing everything any better?
No, and it is the same failure. Some pilot teams, often in risk-averse functions, refuse almost every draft. The record then shows an agent that never gets anything right, and the pilot is abandoned.
But a refusal without a precise reason is as uninformative as an approval without a check. "Not comfortable" tells the builder nothing about what to fix. Trained reviewers refuse specifically: wrong source, wrong amount, outside scope, missing escalation. Those reasons point straight at a brief, a source or a bound that can change. That is how a pilot improves the agent instead of just judging it.
What is the trade-off?
A trained pilot team produces worse-looking numbers. Approval rates fall, refusal rates rise, and the pilot may take longer, because each refusal is investigated and the agent re-qualified. Sponsors used to green dashboards will be uncomfortable.
That discomfort is the pilot working. The choice is between finding the agent's weaknesses in a controlled test, with low depth and a person on every action, or finding them later with the agent acting on its own. Sponsors need to be told this before the pilot starts, with the refusal rate presented as a measure of rigour rather than failure.
What should you do differently?
Train the pilot cohort before the pilot starts, with refusal as the central skill. How to train a pilot team for an AI agent pilot sets out the steps, and what a pilot team should learn before an AI agent pilot lists the content. At minimum, every reviewer should be able to read a verdict, judge provenance, name the grounds for refusal, and write a reason the builder can act on.
Then run the pilot differently:
- Tell the cohort that refusals are success. Say it in writing, from the sponsor.
- Calibrate weekly. Reviewers decide the same small set of cases separately and compare; calibration is the training nobody schedules explains why this is where judgement converges.
- Review every refusal with the builder. Each one is a candidate change to the brief, sources or bounds.
- Report reasons, not just rates. The pilot report should show what the agent got wrong and what was changed.
- Gate the next step on evidence. Widen depth only when the record shows reviewers tested the agent and it held up.
The broader order for training people around agents is in the guide to training teams for AI agents.
Questions people ask
How many refusals should a good pilot have?
There is no target number; it depends on the agent and the process. What matters is that refusals happen where drafts are wrong and that each carries a specific reason. A pilot with no refusals at all should be treated as untested.
Should the pilot team include sceptics?
Yes. A sceptic trained to refuse with precise reasons is one of the most valuable people in a pilot. Pair them with enthusiasts and calibrate together, so the team's standard reflects both.
Can we rerun a pilot that approved everything?
Often you can reuse the cases. Have a trained cohort decide the same drafts again, with reasons, and compare. The differences show what the first pilot missed and give the builder something concrete to fix.