Pick one team with measurable work and a lead who wants it, write the scope and success criteria before anything runs, take a baseline, then shadow before going live on a narrow slice. Set a fixed end date. On that date the team lead decides to stop, extend once, or move to production, and the pilot file records the evidence.
What is a pilot for?
A pilot answers one question: should this agent do this work for this organisation? It is not a demonstration that agents work in general, and it is not a soft launch. It should end in a decision, and the decision should be possible to make on the evidence the pilot produced.
The team lead owns the pilot, because the work is theirs and they will live with the result. A sponsor above them agrees the scope and clears obstacles. An analyst, part-time is fine, gathers the evidence. The team itself does the reviewing, which is why their readiness matters as much as the agent's; what skills a team needs to run AI agents is worth reading before choosing the team.
How do you choose the team?
Look for four things. Work that repeats often enough to produce evidence within weeks. Work whose quality can be judged by the team, against a written procedure. A lead who wants the pilot and has time for it. And a team that is stable: not mid-reorganisation, not under a backlog crisis, not losing its most experienced member.
Avoid the team with the biggest problem. A team drowning in backlog cannot spare time to review drafts, and any result will be muddied by the backlog itself.
What is the method?
- Team lead and sponsor: write the scope statement. Which work, which systems, which case types are in and out, which capabilities at what starting depth, and the end date. A ready format is in the pilot scope statement template.
- Team lead: writes the success criteria before anything runs: quality floor, review effort, turnaround, and what would end the pilot early. The pilot success criteria checklist covers what to include.
- Analyst: takes a baseline from the team's recent work: volume, time, errors from quality checks.
- Agent builder and team: build and certify the agent on the team's own past cases.
- Agent, depth 1 and 2: shadows for two to four weeks. It drafts what it would do; the team works as normal and compares.
- Team lead: decides whether to go live on a narrow slice, based on the shadow results.
- Agent, at certified depth: works the slice live, with held items going to named team members.
- Team lead and analyst: review weekly against the criteria, recording decisions.
- Team lead: makes the end decision on the end date: stop, extend once with a stated reason, or move to production.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Scope | Nothing | none | Writing and agreeing |
| Criteria | Nothing | none | Setting before the start |
| Baseline | Nothing | none | Measuring |
| Build and certify | Replays past cases | 1 | Setting floors |
| Shadow | Drafts | 1 to 2 | Comparing |
| Go-live | Nothing | none | Deciding |
| Live slice | Acts within bounds | certified depth | Held items |
| Weekly review | Produces figures | 1 | Decisions |
| End decision | Drafts the pilot report | 2 | Deciding |
Why must the criteria come before the results?
Because criteria written after the results tend to fit them. A pilot that misses its quality floor gets a new floor. A pilot that saves less time than hoped gets judged on something else. Writing criteria first, and having the sponsor sign them, makes the end decision honest and quick.
Criteria should include stop conditions, not only success conditions. "If the agent produces an error that reaches a customer and causes harm, the pilot stops and is reviewed." A stop condition protects customers and the team, and it shows the organisation takes the pilot seriously.
Why is extending only allowed once?
Pilots stall. They produce enough promise to continue but not enough evidence to decide, and they drift for months while the team does double work. Allowing one extension, with a written reason and a new end date, keeps the pressure to decide. If the second end date arrives without a decision, the default is to stop.
What record does the pilot produce?
- The scope statement and the signed criteria.
- The baseline.
- Certification results for the agent.
- Shadow comparison, case by case.
- Weekly review notes with decisions.
- Every error and incident, with follow-up.
- The pilot report and end decision, signed by the team lead.
If the decision is to move forward, this file is the starting evidence for production; the route is in how to move an AI agent from pilot to production. It also sits inside the agent's full record under train, shadow, certify, widen.
Where does it usually go wrong?
The scope grows. Halfway through, someone adds a case type because the agent is doing well. The evidence for the new case type is thin, and the comparison breaks. Changes to scope need the same sign-off as the original.
The team is not given time. Reviewing drafts is extra work. If it is squeezed into a full day, reviews get thin and the results are unreliable.
Success is measured on the demo cases. The agent is shown handling well-chosen examples and the pilot is judged a success. Judge it on the work the team actually received.
The second team is promised early. A neighbouring team is told it will be next before the first pilot ends. The pressure to declare success grows. Wait for the decision, then use the method in extending an AI agent to a second team.
Where Pryme fits
In Pryme, the pilot's agent is certified by Attest on the team's own past cases, and the agent's starting depth is the lower of its role's ceiling and the evidence. Every verdict in the pilot is filed, so the end decision rests on a complete record rather than on recollection.
Questions people ask
How long should a pilot run?
Long enough to cover the team's normal cycle, often six to twelve weeks including shadowing. Longer pilots rarely produce better decisions.
Should the pilot use production systems?
For shadowing, the agent should read production data, because test data hides the real variety. It should act only on the live slice, under its bounds.
What if the pilot shows the agent is not suitable?
Stop and record why. A clear negative result saves the organisation from a worse deployment later, and the record helps the next attempt.