Enough to produce evidence rather than anecdotes. Before a pilot starts, the team should know the agent's certified depth, the red lines, how to decide a held action and record a reason, how to log disagreements and refusals, and what to do if the agent drops to depth 1. The rest can be learned during the pilot.
Why does a pilot need trained people at all?
A pilot exists to answer one question: should this agent run here, at this depth, with these people? The only way to answer it is with decisions that can be counted and checked. If the team approves held actions without reading them, the pilot measures their speed, not the agent's quality. If they reject everything out of caution, it measures their nerves.
Untrained pilots end with a meeting where people describe how it felt. Trained pilots end with a record.
What is the minimum the team must know?
| Topic | What the team must be able to do | Why the pilot needs it |
|---|---|---|
| Depth | Say what the agent may do on its own and what it only drafts | So nobody expects actions its depth does not allow |
| Red lines | Name the cases that always go to a person | So the pilot never tests them on real customers |
| Held actions | Open the sources, check the bound that held it, decide, record a reason | So every decision becomes evidence |
| Refusals and withheld actions | Tell a correct refusal from a fault | So refusals are not counted as failures |
| Disagreement log | Note every case where they would have done it differently | So the pilot shows where the agent and the team diverge |
| Fallback | Run the work by hand if the agent drops to depth 1 | So a drop to depth 1 does not end the pilot |
That is half a day of teaching and a short practice set, not a course.
What should the team not be expected to learn yet?
How to design or retrain the agent. Pilot teams should not be editing its brief, sources or bounds while the pilot runs, because then nobody knows which version produced which results. Changes go into a list, and are made between pilot phases by whoever builds the agent.
Nor do they need a full certificate before the pilot starts. A pilot is a fair place to begin the practice that leads to one, as long as a certified person signs off the held actions that carry real risk.
An example: a collections pilot
Take a consumer lender piloting an agent that drafts arrears letters and proposes payment plans. The pilot team is six collections staff and their lead. In the first half day they learn that the agent is at depth 2 for letters and depth 3 for plans within set limits, that any customer flagged as vulnerable is a red line, and that every held plan needs an approve or reject with a one-line reason.
After four weeks the lead has several hundred recorded decisions. She can show how often the team changed the agent's plan and why, which cases it held correctly, and two cases where a reviewer approved something they should not have. That is evidence a credit committee can act on. A pilot without that half day would have produced a few stories and a vote.
What controls keep the pilot honest?
- A baseline before the start: how the team handles the same work without the agent, measured the same way.
- A fixed agent version: no changes to brief, sources or bounds mid-phase.
- Reasons on every decision: a decision without a reason is not evidence.
- A weekly calibration sample: two people decide the same held cases and discuss every difference.
- One named owner: who reads the record each week and calls a pause if needed.
Pryme Intelligence Academy's free readiness assessment, for teams of up to 25, gives a pilot team a scored starting point by concept before the pilot begins.
Where does this fit with the rest of the pilot?
For the training steps in detail, see how to train a pilot team for an AI agent pilot. For the pilot itself, see how to run a pilot of an AI agent in one team and the definition of a pilot cohort. The manager's preparation is covered in what managers should learn about AI agents before their team, and the cost of skipping this is argued in your AI agent pilot failed because nobody was trained to say no. The full approach is in training teams for AI agents.
Questions people ask
How big should a pilot team be?
Big enough to cover the work on every shift the pilot runs, and small enough that one lead can read the record each week. Six to twelve people suits most single-team pilots.
Should the pilot team include sceptics?
Yes. A sceptic who records reasons produces the most useful disagreements. Exclude only people who will not record their decisions.
What happens to the pilot team's training afterwards?
It becomes the start of their certification. Their recorded decisions are practice, and the strongest reviewers often become the trainers for the wider rollout.