PrymeIntelligence

Training teams for AI agents·Playbook

How to run an AI agent incident drill for the support leadership

An AI agent incident drill for support leadership: compensation promises after an outage, pause to draft-only, decide, staff the queue, record and report.

Support leaders should drill decisions, not clicks. The scenario that tests them best is an agent promising customers something the policy does not offer during an outage. In a two-hour tabletop, leaders decide on the pause to draft-only, what to honour, how to staff a fully held queue, what customers hear and how to report upward.

What does a leadership drill rehearse, and who owns it?

Frontline staff practise spotting and escalating. Support leaders practise what happens after the escalation reaches them: weighing customer harm against cost, finding capacity when the agent stops sending, deciding what the business stands behind, and briefing executives who want a single number.

The support agents in scope typically send routine replies at depth 3 within approved answers and the business's own limits, issue small goodwill credits within a set amount, and draft everything else for a person at depth 2. Leaders own those limits and the decision to change them.

The head of customer support owns the drill. Someone outside the support line, often from risk or operations, facilitates so that leaders cannot steer the scenario. Team leads, the workforce planning lead, a communications contact and a finance partner take part. The support team drill is the frontline counterpart and works best run a week earlier, so leaders know what their teams will hand them.

The scenario

Wednesday, 18:30. A service outage lasting four hours ended an hour ago. Contacts are up fivefold. The support agent has been replying at depth 3 with an approved outage message. A team lead escalates: customers are quoting a line from the agent's replies promising "a full month's credit" to everyone affected.

The policy offers credit only on request and only for business customers. The line came from a draft outage statement that someone placed in the agent's sources during the outage, before it was approved. About 1,500 replies have gone out with the promise. Social media is quoting it.

The drill step by step

  1. Person (facilitator): reads the escalation and starts the clock. Executives will ask for an update in 30 minutes.
  2. Person (head of support): pauses the agent to draft-only. Agent, depth 2: every reply now waits for a person to send it, at the busiest moment of the quarter.
  3. Person (workforce planning lead): finds capacity for a queue where nothing sends itself: overtime, pulling trained reviewers from other queues, and deferring non-urgent work.
  4. Person (head of support): removes the unapproved statement from the agent's sources, and asks who added it and under what authority.
  5. Agent, depth 1: lists every reply containing the promise, from the record. Person (team lead): confirms the count.
  6. Person (head of support), with finance: decides what the business will honour. The options are to honour it in full, honour it for those who received it, or withdraw it with an apology. Each has a cost and a consequence.
  7. Agent, depth 2: drafts the follow-up message reflecting the decision. Person (team leads): approve the wording; the communications contact aligns it with any public statement.
  8. Person (head of support): reports to executives with the count, the decision, the cost and the cause, and records the decision with reasons.
  9. Person (facilitator): runs the debrief, including when and how depth 3 returns.
Step The agent does Depth Stays with a person
Escalation Has sent the promise in replies 3 Recognising it as an incident
Pause Drafts every reply Dropped to 2 The pause decision
Capacity Drafts at volume 2 Staffing a held queue
Source fix Uses corrected sources 2 Removing the draft statement
Scope Lists affected replies 1 Confirming the count
Honour or withdraw Nothing 2 The commercial decision
Follow-up Drafts the message 2 Approving and sending

What do leaders learn that frontline staff do not?

That a pause has a cost. Dropping to draft-only during an outage surge can double response times. Leaders should rehearse the trade-off in the room and make it consciously, not discover it at 22:00. The drill should end with a written view of how many trained reviewers the business needs to run the queue at depth 2 for a full day.

That the sources are a leadership responsibility. The promise did not come from the agent; it came from a document someone added under pressure. Leaders must decide who may change what the agent relies on during an incident, and make that a bound, not a habit.

That honouring a promise is a policy decision. Once 1,500 customers hold a written promise, the business has a fairness problem whichever way it decides. The drill makes leaders decide quickly and record why.

Which controls does the drill test, and what evidence do they leave?

The pause. Evidence: the limit change in the record, with the head of support's name and time.

Source change control. Evidence: who added the draft statement, who removed it, and when.

Scoping. Evidence: the list of affected replies drawn from the record.

Commercial decision. Evidence: the recorded decision, the reasons and the finance partner's agreement.

Return to depth 3. Evidence: the named approver and the agent's current certificate.

What should the head of support measure?

  • Time from escalation to pause, and from pause to the first held reply being sent by a person.
  • Response time during the pause, against the target leaders set in the drill.
  • Time to a decision on honouring the promise.
  • Whether executives received one consistent number throughout.

Where do leadership drills go wrong?

Leaders run the queue. Senior people start answering tickets. Nobody is left to decide, and the decisions arrive late.

No pre-agreed honour policy. Every incident becomes a fresh negotiation with finance. Agree a default for promises made in error before the next outage.

Restoring depth too fast. Under pressure, leaders lift the pause as soon as the source is fixed. Without a check of the agent's certificate and a sample of its replies, the next error goes out at full volume.

Fraud scenarios test leaders differently, because there is an adversary; the fraud team drill shows how. Leaders new to approving held work should complete the reviewer onboarding checklist for support leadership, and the cards for this drill can be built from the AI agent incident drill script template. The wider plan is in training teams for AI agents.

Where the Academy fits

Support leaders who run a live workspace day to day are the people Pryme Intelligence Academy's Operator track is written for; its modules on running the rhythm and on usage and billing cover the capacity and cost questions this drill raises. A business can also make the right to change an agent's sources or limits depend on a current certificate, which would have stopped the draft statement going in.

Questions people ask

Who should facilitate a leadership drill?

Someone senior enough to be listened to but outside the support line, so leaders cannot redesign the scenario as it runs.

Should executives take part?

Not in the room. Have the facilitator play the executive asking for updates, so leaders practise briefing without being overruled mid-drill.

How often should leaders drill?

Twice a year, and once before any expected peak such as a product launch or seasonal surge.

Train the people who run your agents.

Pryme Intelligence Academy: free courses on the real product, audio lessons, labs in a free workspace, and certificates anyone can check on VerifyDoc.ai. Teams get seats, an admin view and a free readiness assessment.