A pilot team's job is to produce evidence, not enthusiasm. Before the agent touches real work, the team should know how it decides, what each verdict means, how to decline with a reason, and what the pilot must prove. Then they practise on past cases, agree a shared standard and log every decision so the result can be trusted.
What is the pilot team's job, and who owns it?
A pilot asks one question: can this agent do this work for this business, under these bounds, with these people reviewing it? The answer comes from the record of real decisions. If the pilot team approves everything, the record says nothing. If each person applies a different standard, it says something misleading.
The pilot sponsor, usually the head of the team where the agent will run, owns the pilot. A pilot lead runs training and the daily rhythm. The builder who shaped the agent's brief attends the debriefs. Each team member owns their own decisions and reasons.
What should the team learn before day one?
- How the agent decides. Every action meets five bounds: role scope, certificate, sources, the business's limits and the platform guardrails. The lowest wins and the verdict names it.
- The four verdicts. Allowed, held, refused, withheld. Withheld actions are a platform decision and no feedback will change them, so the team should not log them as faults.
- Depth. Most pilots run at depth 2: the agent drafts and a person sends. Depth is earned on evidence from Attest, not configured to please a sponsor.
- How to decline. A decline with a clear reason is the most useful thing a pilot reviewer produces. It shows where the agent falls short and gives the builder something to fix.
- What the pilot must prove. The success criteria, written before the start, so nobody moves the goalposts in week three.
The steps
- Person (pilot sponsor): writes the success criteria and the red lines. For example: agreement between reviewers and the agent's drafts on at least an agreed count of cases out of every 100; no red-line breach; reasons recorded on every decline.
- Person (pilot lead): chooses the cohort: a mix of experienced and newer staff, including at least one known sceptic.
- Person (team): completes the learning above, in short sessions spread over a week.
- Person (pilot lead): runs two calibration sessions on 20 past cases each, until the team's decisions converge.
- Agent, depth 1: reads live work for a few days while the team watches its proposed actions without acting on them.
- Agent, depth 2: drafts live work; a pilot reviewer approves, edits or declines each draft with a reason.
- Agent, depth 1: logs every verdict and decision in the append-only record.
- Person (pilot lead): runs a daily fifteen-minute debrief on declines, disagreements and anything held.
- Person (pilot sponsor): reviews the evidence against the criteria at the end, with the builder.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Observation days | Proposes actions without acting | 1 | Team compares with what they would do |
| Routine drafts | Drafts replies and actions | 2 | Reviewer approves, edits or declines |
| Items over the business's limit | Prepares and holds | held | Named reviewer decides |
| Out-of-scope requests | Refuses and hands over | refused | Reviewer handles directly |
| Platform-floor actions | Withholds | withheld | No one; logged, not treated as a fault |
How do you stop the pilot team becoming a rubber stamp?
Reward declines that are well reasoned. In the daily debrief, read out the best decline of the day. Teams copy what gets attention.
Watch the edit and decline counts per reviewer. A reviewer with no edits and no declines across a week has either found a flawless agent or stopped reading. Ask them to talk through three of their approvals.
Include the sceptic. A pilot team made only of volunteers who want the agent to succeed produces evidence a board should doubt. The argument for this is made in why pilots fail when nobody is trained to say no.
Which controls matter, and what evidence do they leave?
- Written criteria set in advance. Evidence: the signed criteria document.
- Calibration before live work. Evidence: the case set and each reviewer's decisions.
- Reasons on every decline and edit. Evidence: the reason against each decision.
- Daily debriefs. Evidence: a short note of issues raised and actions taken.
- Append-only record. Evidence: every verdict, the deciding bound and the reviewer.
What should you measure?
Drafts approved unedited, edited and declined, by reviewer and by case type. Agreement between reviewers on calibration cases. Held items and their age. Refusals by deciding bound, which shows whether the business's limits are set sensibly. Red-line breaches. And the quality of reasons, scored on a weekly sample.
Where does it usually go wrong?
Training is a single launch session. A week later, people have forgotten what withheld means and are filing it as a bug.
Success criteria are written after the pilot. The team then argues about what the numbers mean instead of acting on them.
The pilot team is given extra time no production team will have. Run the pilot at real volume.
What the cohort should know is set out in what a pilot team should learn first, and choosing it in what a pilot cohort is. For running the pilot itself, see how to run a pilot of an AI agent in one team; for what comes next, moving an agent from pilot to production. The guide to training teams for AI agents covers who to train after the pilot.
Questions people ask
How large should a pilot team be?
Large enough to cover every shift and include a range of experience, small enough that the pilot lead can debrief everyone daily. Five to ten reviewers is common for one team's work.
Should the pilot team be certified before the pilot?
They should pass calibration before live work. Formal certification can follow during or after the pilot, using the pilot's own decisions as practice.
What if the pilot shows the agent is not ready?
That is a useful result. The declines and their reasons tell the builder what to change in the brief, sources or bounds before the agent is certified again.