What should a team practise before an AI agent goes live?

Six scenarios, each run on realistic cases before customers are involved: clearing held actions at normal volume, spotting the held action that is a trap, explaining a refusal, standing firm on a withheld action, working by hand when the agent drops to depth 1, and handling a suspected incident. Reading about them is not enough.

Why practise rather than just train?

Training explains what a held action is. Practice shows whether someone opens the sources when the queue has eighty items in it and the phone is ringing. The gap between knowing and doing is where most early failures happen. A short rehearsal in the week before go-live closes much of it.

Practice also tests the set-up, not just the people. A drill often reveals that held actions reach the wrong queue, that nobody covers the late shift, or that the fallback process refers to a system the team no longer uses.

Which scenarios should every team rehearse?

1. The routine queue at real volume. Load a practice queue with a normal day's held actions from the team's own past work. Each reviewer clears their share, recording a decision and reason for each. What good looks like: steady pace, sources opened, no blanket approvals.
2. The trap inside the routine. Hide a few cases that should be rejected among the routine ones: a supplier invoice from an external email with changed bank details, a customer flagged as vulnerable, a draft that contradicts the source. An external claim cannot authorise a payment, and the reviewer should catch it.
3. The refusal a customer pushes back on. The agent correctly refused an action outside its bounds. A colleague plays an annoyed customer. The reviewer explains why, without blaming the system and without overriding it.
4. The withheld action someone asks to override. A senior colleague asks the team to make the agent do something withheld at the platform floor. The team should know that no setting changes it and that the action must be done another way, by a person, if it is allowed at all.
5. Depth 1 by noon. Tell the team at the start of a shift that the agent's last run cleared no depth and it is now at depth 1: it can read, nothing more. They run the work by hand for an hour. Certificates lapse after 90 days, so this will happen for real.
6. The suspected incident. The agent produces something odd: a reply quoting text that looks injected from a document, or a draft sent to the wrong customer. The team pauses the work, tells the named owner and preserves the record.

Who should take part?

Everyone who will approve held actions, their manager, and whoever owns the agent. The owner watches, because drills show where the agent's bounds, sources or routing need changing. Advisers who will only receive agent drafts can join scenarios 3 and 5.

An example: a mortgage servicing team

Take a building society's mortgage servicing team about to go live with an agent that drafts replies to payment queries and sets up routine payment date changes at depth 3. In the drill, scenario 2 includes a borrower asking to move their payment date who has also mentioned losing their job. The agent drafted the change and held it because of the society's own limit on hardship language.

Two of five reviewers approved it in the drill. That finding, a week before go-live, led the team lead to rerun the hardship cases with everyone and add a line to the team's checklist. On go-live day, the case would have reached a real borrower.

How do you know the team is ready?

Readiness is a judgement, but it should rest on the drills. Every reviewer has cleared a practice queue with reasons recorded. Nobody missed a red line in scenario 2. The team ran an hour at depth 1 without losing work. And the owner has a list of fixes, each one closed or accepted.

For the agent side of readiness, see how to test an AI agent before it goes live, how to shadow an AI agent before it goes live and go-live readiness. For timing, see how AI agent training should be scheduled around a go-live. The full approach is in training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's labs run in a free Build workspace, which is a safe place to rehearse these scenarios on a real product before go-live. Its Operator track includes modules on human review and on when something goes wrong, and ends with a practicum on real review decisions with no missed red line.