How to train the internal audit team to review AI agent decisions

Internal auditors should test agents the way they test any control: by reconstructing decisions from evidence. Start with a skills audit, teach the decision chain and how to read an agent's record, calibrate on whether sampled decisions are findings, shadow one engagement with every test reviewed, and sign off auditors who can trace any verdict to its bound.

What will internal audit be reviewing?

Internal audit looks at agent decisions from the third line. That means testing whether the controls around agents work, not reviewing day-to-day output. Typical questions:

• Was each agent operating at a depth its certification supported, within its role's ceiling?
• Were held items decided by people with the authority, and where approval rights depend on a current certificate, was it current?
• Did refused actions stay refused, or did people work around them?
• Is the record complete and append-only, so any decision can be reconstructed?
• Were changes to an agent's brief, sources or bounds re-certified before they took effect?
• Did the first and second lines review what they said they would?

Auditors may also use their own agent to pull samples and draft workpapers at depth 1 and 2. That is useful, and the audit team must apply to it the scepticism it applies to everyone else's.

The chief audit executive owns the programme. A senior auditor with technology audit experience leads delivery.

How does the programme run, step by step?

1. Chief audit executive: runs a skills audit of the team against the abilities needed below, to decide who trains first and how deep.
2. Learning, weeks 1 and 2: auditors complete lessons on the decision chain: the five bounds, the four verdicts, depth, certification and lapse, and the record.
3. Agent, depth 1: assembles a test pack of real decisions across several agents with their full record.
4. Calibration, week 3: auditors test the pack alone. For each decision: reconstruct it, judge whether the control worked, and decide whether there is a finding and how severe. Then compare.
5. Shadow, one engagement: each auditor runs tests on a live agent; the lead auditor reviews every test and finding before the report.
6. Agent, depth 1 and 2 in audit work: pulls samples, drafts workpapers. Auditors verify the sample was complete and the workpaper accurate.
7. Sign-off: chief audit executive assesses the shadow engagement.
8. Ongoing: annual calibration and refreshers when the organisation's agent estate changes materially.

What must auditors be able to do?

Reconstruct a decision. From the record alone: what the agent was asked, which sources it used and their provenance, which bounds applied, which one decided, the verdict, and what any person did next.

Test depth. Compare the agent's operating depth with its certification evidence and its role's ceiling. Check the certificate was current; agent certificates lapse after 90 days, and a run that clears no depth drops the agent to depth 1 the same day.

Test the human controls. Sample held items and check the decider's authority and competence at the time.

Look for workarounds. A refused action followed by a manual action of the same kind is a pattern worth testing.

Test change control. For each change to an agent, find the certification run that preceded it.

Grade findings. A missed hold on a small refund and a missed hold on a payment to a changed bank account are not the same finding.

Distrust their own tools. Check that the audit agent's sample was drawn as specified.

Take an illustrative test. An auditor samples 25 refunds an agent issued at depth 3. For one, the record shows the refund exceeded the customer's limit. Reconstruction reveals the limit was raised the same morning by a manager, and the agent acted on the new limit. Is that a finding? Only if the limit change itself bypassed approval. Calibration teaches auditors to follow the chain one step further before deciding.

Which controls leave evidence?

• Skills audit recorded. Evidence: results by auditor.
• Calibration results kept. Evidence: individual tests and the agreed answers.
• Every shadow test reviewed. Evidence: lead auditor's sign-off on workpapers.
• Sample completeness checked. Evidence: population and sample reconciliation.
• Sign-off recorded. Evidence: chief audit executive's assessment.

What should you measure?

• Calibration agreement on finding and severity.
• Time to reconstruct a sampled decision.
• Findings accepted by management without dispute on facts.
• Repeat findings across engagements.

Where does it usually go wrong?

Auditors test outcomes. They check customers got the right result and stop there. The control question is whether the right bound decided.

Record access is weak. If auditors cannot reach the record directly, testing depends on management's extracts. Secure direct read access.

Technology audit owns it alone. Agents act in business processes. Business auditors must be able to test them too.

Incident response untested. Practise with an AI agent incident drill for the internal audit team.

New starters use the reviewer onboarding checklist for the internal audit team, and individual technique is in how internal auditors review an AI agent's decisions. Procurement teams run first-line review of a different agent in how to train the procurement team to review AI agent decisions. The full programme is in training teams for AI agents.

Where the Academy fits

Auditors who test across many agents often benefit from Pryme Intelligence Academy's Architect track, designed for people who lead regulated, multi-agent deployments, ending in a capstone and a board review with two Pryme assessors. For checking first-line competence, the public register lists certified people by name, credential and expiry, and every certificate can be checked at source.