Treat the first month after an AI agent goes live as the main training period, not the end of it. Courses before launch teach concepts; live work teaches judgement. Reviewers meet real held actions, real refusals and real customer pressure for the first time, and the habits they form in those weeks last. Plan the month deliberately, or it will train people by accident.
Why can't pre-launch training do the whole job?
Because the hardest cases cannot be predicted. Before go-live, a team learns what depth means, how verdicts work, how provenance labels sources and what is recorded. They practise on test cases. All of that matters; without it the first month is chaos.
But live work brings what test cases miss. The customer who writes in three languages. The policy exception everyone forgot. The supplier email that looks right and is not. The rush at month end when the queue doubles. Reviewers learn from these by deciding them, and in the first month they decide many for the first time. That is training, whether or not anyone calls it that.
Live work also changes the emotional weight of a decision. Approving a test case teaches the mechanics. Approving a real payment, knowing the customer will receive it and the record will name you, teaches care. Most reviewers say their first week live felt different from every exercise before it, and that is where the lessons stick.
What happens in the first month if nobody plans it?
Habits form by default. Reviewers under pressure learn that approving is faster than reading. They learn which colleague to ask, whether or not that colleague is right. They learn to work around refusals rather than understand them. A certificate lapses or a run that clears no depth drops the agent to depth 1 the same day, a flood of drafts arrives, and the team learns to approve in bulk.
By the end of the month these habits are the operation. Every verdict and every human decision is filed in an append-only record, so they are also evidence. An auditor reading the first month's record will see exactly how well the team was trained, and so will anyone investigating a complaint that started in week two.
How should the first month be structured?
As a training plan with the queue as its classroom:
- Week one: close supervision. Experienced reviewers or the people who led the shadow period sit with each reviewer. Daily debriefs of fifteen minutes cover the hardest cases of the day.
- Week two: sampling and feedback. Each reviewer's decisions are sampled and discussed individually. Patterns are noted for calibration.
- Week three: first calibration. The team decides the same contested cases independently, then compares. Disagreements become written guidance.
- Week four: retrospective. What did the team learn, what should change in the agent's brief or sources, and what should change in training for the next cohort?
The training retrospective template and the guide on how to run a training retrospective after go-live set out the last step. An after-action review is a useful format for individual incidents during the month.
What is the trade-off?
Throughput. A first month run as training is slower. Supervisors spend time coaching instead of reviewing. Daily debriefs and calibration take people off the queue. Leaders who bought the agent to save time see the opposite for four weeks.
That is the honest cost, and it should be stated up front. The alternative is a month of untrained habits that take far longer to undo, plus the risk of a serious error while nobody is watching. A team that is properly trained in the first month usually reaches steady throughput sooner than one that is not, because it stops making the same mistakes. Schedule the month with that in mind, and protect it from being cut when the queue grows. The guide on how to schedule AI agent training around a go-live shows how.
What should be measured in the first month?
Correctness, not speed. The share of sampled decisions that were right, by reviewer. The number of overrides of refusals, with reasons. Held items by age at the end of each day. The number of edits to agent drafts, by type, which shows where the agent's brief needs work.
Also measure readiness for the second month. Can each reviewer explain what held, refused and withheld mean? Can they read a provenance label and say whether a fact is sealed, signed, declared or external? Do they know what to do when a certificate lapses? These are the questions go-live readiness should have answered before launch; the first month confirms or corrects them.
What should leaders do differently?
Put the first month in the training plan, with named owners, daily and weekly sessions and a retrospective date. Budget the throughput cost explicitly. Keep the people who ran the shadow period close to the queue for those four weeks. Decide in advance what evidence would justify raising any capability's depth, and do not raise it during the first month without that evidence.
Above all, stop treating go-live as the finish line for training. It is the start of the part that counts. The broader approach is in training teams for AI agents. Pryme Intelligence Academy's Operator certificate follows the same logic: it requires a two-week practicum of real review decisions, not only an exam.
Questions people ask
Should the agent run at lower depth during the first month?
Often, yes. Running capabilities at depth 2 for the first weeks means people see every draft, which is the best training material available. Depth can rise later as evidence supports it.
Who should run the daily debriefs?
The person closest to the work who understands the agent best, often a team lead who ran the shadow period. It should not be someone who only sees reports.
What if the first month goes smoothly?
Keep the plan anyway. A quiet first month means fewer hard cases have arrived yet, not that the team is ready for them. Use the time to calibrate on the cases you have.