How to run an AI agent incident drill for the internal audit team

Internal audit's drill has two halves. First, auditors observe another team's incident drill without intervening. Then, a day later, they reconstruct that drill from the append-only record alone and test whether the pause, escalation, decisions and reporting can be evidenced. The exercise trains auditors to audit agent incidents, and tells the business whether its record would satisfy a supervisor.

Why does internal audit need its own drill?

Internal auditors do not run the queue or pause agents. They give independent assurance that controls work. When agents act, much of the evidence auditors need sits in the record: every verdict, the bound that decided it, every human decision, every limit change. Auditors who have never reconstructed an incident from that record will struggle to audit a real one.

The head of internal audit owns the drill. An audit manager plans it with the owner of the observed team. Auditors take part; the observed team runs its own drill as normal. For the general format, see what an incident drill is. A wider view of skills is in what a skills audit is.

Choosing the observed drill

Pick a team whose agents act at depth 3 or 4, so there is something meaningful to audit. A payments operations drill is a good choice, as in the payments operations team drill, or a clinic's practice team, as in the practice team drill. Agree with the team's owner that the drill will use a practice workspace or past data, and that auditors will see the resulting record.

The drill step by step

1. Person (audit manager): briefs auditors on the observed team's scenario, but not on the expected answers.
2. Person (auditors): observe the drill and note what they see: who paused, when, who was told, what was decided. They do not intervene.
3. Person (observed team): runs its drill as normal, including the pause to draft-only, escalation, recording and the reporting decision.
4. Next day. Person (auditors): reconstruct the drill from the record only, without their notes. For each step, they find the evidence: the limit change that paused the agent, the verdicts that followed and the bound each named, the decisions people recorded.
5. Person (auditors): compare the reconstruction with their observation notes. Every gap is a finding: something happened that the record does not show, or the record shows something nobody saw.
6. Person (auditors): test three controls in depth. Who has the right to pause, and is it limited? Were decisions recorded at the time, or written up later? Was the reporting decision reasoned and timed?
7. Person (audit manager): drafts findings for the observed team's owner.
8. Person (head of internal audit): debriefs with the observed team and decides whether any finding goes to the audit committee.

What does this drill teach auditors?

Three things specific to agents. First, the record is strong on what agents did and which bound decided, because every verdict is filed append-only. It is only as strong on human decisions as people make it. Auditors often find that the pause is perfectly evidenced and the decision to report is not.

Second, a verdict tells you which bound decided. Auditors can use that to test whether the business's own limits match approved policy: if a payment was held because of the business's limit, the limit's value should match what the board approved.

Third, timing. Auditors should check that records were made at the time. A decision recorded three hours after the event is weaker evidence than one recorded as it happened.

Auditors should also look at what the record does not need to hold. Conversations with customers, calls to suppliers and board briefings may live in other systems. The test is whether the record points to them, with a reference and a time, so an auditor can follow the trail without relying on memory.

Which controls does the drill test?

Pause rights. Evidence: who changed limits, and whether only authorised people can.

Contemporaneous recording. Evidence: timestamps on people's decisions against observed times.

Reporting. Evidence: a written decision with reasons and time.

Restoration. Evidence: who restored depth, on what basis, and whether a sample was checked first.

What should the head of internal audit measure?

• Gaps between observation and reconstruction, by type.
• Time auditors needed to reconstruct the incident.
• Findings accepted by the observed team, and remedies agreed.
• Whether auditors could explain the four verdicts and the five bounds without help.

Where do audit drills go wrong?

Auditors intervene. Helping the observed team destroys the test. Agree the rule in advance.

Reconstructing from notes. The whole point is to test the record. Put notes away until the comparison step.

Findings without remedies. Each gap should come with a practical fix the team can make before its next drill.

New auditors can prepare with the reviewer onboarding checklist for an internal audit team. The wider method is in training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's Foundations track covers the decision chain, depth and certification, and ownership, data and trust in about four hours, which gives auditors the vocabulary they need. Its badge is free and does not expire, and it is stamped with the product version, so auditors can see which version of the product their training reflects.