PrymeIntelligence

Training teams for AI agents·Playbook

How to schedule AI agent training around a go-live

How to schedule AI agent training around a go-live: lessons before depth 1, practice and calibration before depth 2, sign-off before any depth 3 step.

Tie every training milestone to a depth step, not to a calendar date. Lessons finish before the agent starts reading live work. Practice and calibration finish before it drafts for people to send. Reviewer sign-off finishes before it executes anything routine. If training slips, the depth step waits. The schedule is set by competence.

Why schedule training against depth rather than dates?

Agents rarely go live at full depth. They earn it. A typical rollout starts at depth 1, where the agent reads and summarises; moves to depth 2, where it drafts and a person sends; and only later reaches depth 3, executing routine work within bounds, once certification evidence supports it. Each step changes what reviewers must do. At depth 1 they compare summaries. At depth 2 they approve every draft. At depth 3 they review held items and samples, which is a harder skill because most work no longer passes in front of them.

Training scheduled by calendar date drifts out of line with those steps. Project plans slip; training sessions do not move. People get trained for a depth that arrives a month late, or not trained for one that arrives early. Scheduling against depth fixes that.

The deployment lead owns the schedule. The learning lead owns the training content. The agent owner owns the depth steps, and agrees not to take one until reviewers are ready.

What does the schedule look like, step by step?

  1. Learning lead, before depth 1: runs a readiness baseline. A short test of what reviewers already understand about governed agents.
  2. Reviewers, during depth 1: complete core lessons. How the agent decides, the four verdicts, what each bound means. The agent is reading live work, so lessons can use real examples.
  3. Agent, depth 1: produces practice material. Its summaries and the sources it used become teaching cases.
  4. Reviewers, before depth 2: practise and calibrate. Two calibration sessions on drafted cases from a test environment or shadow mode.
  5. Agent, depth 2: drafts live; reviewers approve every draft. This is the practicum period. Team leads sample reviewer decisions.
  6. Assessor, before depth 3: signs off reviewers against an accuracy bar with no missed red line.
  7. Agent owner, at depth 3: takes the step only when sign-off is complete for enough reviewers to cover every shift.
  8. Learning lead, four to six weeks after: runs a retrospective.
Step The agent does Depth Stays with a person
Baseline and lessons Reads live work 1 Reviewers learn
Practice cases Summaries used as material 1 Learning lead selects
Calibration Drafts in shadow 2 Reviewers decide alone, then compare
Practicum Drafts live 2 Reviewers send every item
Sign-off Nothing none Assessor
Routine execution Executes within bounds, holds the rest 3 Reviewers handle held items and samples
Retrospective Compiles evidence 1 Learning lead and group

How do you set the gates between steps?

Write a short gate for each depth step, agreed in advance with the agent owner. A useful set:

  • Gate to depth 2: every reviewer has finished core lessons and attended one calibration session; agreement on shared cases is acceptable to the head of the function.
  • Gate to depth 3: enough reviewers signed off to cover all shifts, including absence; at least one calibration session held on live drafts; held-item procedures written.
  • Gate to depth 4, where the agent may move money or take hard-to-undo actions: a separate review, with named approvers for held items whose own certificates are current.

Treat the gates as part of go-live readiness. A depth step with no trained reviewers behind it is not ready, whatever certification says about the agent.

Which controls leave evidence?

Gate records. Each depth step has a signed gate. Evidence: the record, dated before the step.

Coverage by shift. Evidence: a rota showing signed-off reviewers on every shift.

Certificates checked at source. Evidence: certificate numbers and expiry dates, verified when the gate is signed.

Rollback rule. If a certification run that clears no depth drops the agent to depth 1, training does not restart, but held-item volumes change. Evidence: a note in the plan describing what reviewers do then.

What should you measure?

  • Slippage between training milestones and depth steps. Any depth step taken before its gate is a finding.
  • Sampling accuracy in the first two weeks after each step.
  • Held items waiting longest after the move to depth 3, a sign of thin coverage.
  • Reviewers still in progress at each gate.

Where does it usually go wrong?

The project date wins. Pressure to hit go-live pushes the depth step through without sign-off. Agree in advance that the gate is binding.

Training ends at depth 2. Teams train for approving drafts, then the agent moves to depth 3 and nobody teaches how to review samples and held items.

Shift workers trained last. Schedule them first; they have least slack.

No retrospective. The next agent repeats the same timing errors. Run one as described in how to run a training retrospective after an AI agent go-live, using the retrospective template. The short answer version is how should AI agent training be scheduled around a go-live, and the full programme is in training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's Operator track maps onto this schedule: lessons during depth 1, labs in a free Build workspace for practice, and a practicum of two weeks of real review decisions signed off by a Pryme assessor. The admin view shows who has finished what, so gate checks become a matter of reading progress and certificate expiry rather than chasing people.

Questions people ask

Can training start before the agent exists?

Foundations material can. Practice and calibration need the actual agent's drafts and held items, so they wait for depth 1.

What if a reviewer fails sign-off at the depth 3 gate?

They continue reviewing at depth 2 work, retake the practicum, and do not approve depth 3 held items until signed off.

Does every depth step need a new course?

No. Each needs a short top-up on what changes, plus calibration on the new kind of case.

Train the people who run your agents.

Pryme Intelligence Academy: free courses on the real product, audio lessons, labs in a free workspace, and certificates anyone can check on VerifyDoc.ai. Teams get seats, an admin view and a free readiness assessment.