Internal auditors calibrate on two things: how they rate findings, and whether the evidence an agent gathered is enough to support them. Give the team the same draft findings and evidence packs, have each auditor rate and conclude alone, then compare. The head of internal audit sets the standard, which goes into the audit methodology and the agent's brief.
Why do internal auditors need calibration?
Audit teams use agents to pull samples, gather evidence from systems, test simple controls across whole populations, and draft findings at depth 2. Auditors review every finding and decide the rating and the wording. Nothing goes to management or the audit committee without an auditor's sign-off.
Ratings are where auditors diverge. One rates a missing approval on three payments as high; another as medium because the amounts were small. Management notices, and the most lenient auditor becomes the one every business unit wants. Agents add a second question: is the evidence the agent gathered sufficient and reliable? A screenshot pasted into an email is external; a report pulled directly from the system of record is sealed. Auditors who do not distinguish them will rate findings on evidence that would not survive challenge. The general method is in how to run calibration sessions for AI agent reviewers.
The head of internal audit owns the standard. An audit manager facilitates.
Which items belong in the set?
Six to eight draft findings from recent audits, with business units and auditor names removed, each with the agent's evidence pack:
- A control exception on a small number of transactions in a large population.
- A finding based on evidence the business provided by email, not extracted by the agent.
- A full-population test where the agent found a pattern a sample would have missed.
- A repeat finding from a prior audit, partly fixed.
- A finding about an AI agent's own controls, such as approvals at a depth the reviewer was not certified for.
- A draft finding the agent wrote that overstates what the evidence shows.
The session, step by step
- Person (audit manager): selects findings and anonymises them.
- Person (each auditor): rates each finding alone, decides whether the evidence supports it, and edits the wording.
- Agent, depth 1: shows how it gathered each piece of evidence and its provenance, and which test it ran.
- Person (facilitator): reveals the spread on rating and on evidence sufficiency.
- Person (auditors): explain their reasoning, especially on impact and likelihood.
- Person (head of internal audit): sets the gold standard.
- Person (head of internal audit): updates the methodology and sends evidence rules to the brief owner.
- Person (quality assurance reviewer): checks the next round of findings against the standard.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Select findings | Nothing | 1 to 2 on live audits | Anonymising |
| Rate alone | Nothing | Unchanged | Rating, sufficiency, wording |
| Evidence trail | Shows how evidence was gathered | 1 | Comparing |
| Discussion | Nothing | Unchanged | Impact and likelihood |
| Gold standard | Nothing | Unchanged | Head of internal audit |
| Update methodology | Gathers per new rules | 1 to 2 | Brief owner |
| QA review | Drafts findings | 2 | Quality reviewer |
What does a gold standard look like for internal audit?
It links rating to impact, not to how many exceptions were found. For example: "A missing approval is rated high where the control is the only safeguard on payments above a set amount, regardless of the number of exceptions; medium where a compensating control operated."
On evidence: "A finding may rest on evidence the agent extracted directly from the system of record. Evidence supplied by the business by email is corroborated before the finding is rated."
On agent controls: "Where the record shows a held action approved by someone without a current reviewer certificate for that depth, the finding is rated on the depth of the action, with depth 4 actions rated high."
On wording: "Findings state what was tested, what was found, and the risk. The agent's draft is edited to remove conclusions the evidence does not support."
What does auditing agents add?
As more operations run through agents, auditors will test the agents themselves. The record makes this possible: every verdict is filed, naming the bound that decided. Auditors should calibrate on reading it: whether a held action was approved by the right person, whether the agent's certificate was current when it acted, and whether a refusal was correct. Include one such finding in every session so the team's judgement on agent controls develops alongside its judgement on traditional ones.
Which controls does calibration support, and what evidence does it leave?
Consistent ratings. Evidence: spreads over sessions and the rating standard.
Evidence sufficiency. Evidence: the provenance rule in the methodology.
Audit of agents. Evidence: findings on agent controls rated against the standard.
Quality. Evidence: QA reviews of findings against the standard.
What should the head of internal audit measure?
- Rating agreement on shared findings.
- Findings challenged by management, and how many were downgraded.
- Findings resting on uncorroborated external evidence. The target is none.
- How often auditors rewrite the agent's draft findings substantially.
Where does it go wrong in internal audit?
Calibrating ratings without evidence. Teams agree a rating on a finding the evidence could not support.
Leaving agent-control findings out. The team is then least practised in the area growing fastest.
Letting the agent's wording stand. Agents can state findings more confidently than the evidence warrants. Auditors own the words.
Paralegals and caseworkers calibrate on similar mixes of evidence and judgement; see calibration for paralegals and for caseworkers. For planning, see the AI agent training plan template for internal auditors and how long it takes to train internal auditors to supervise AI agents. The topic guide is training teams for AI agents.
Questions people ask
How often should internal auditors calibrate?
Quarterly at minimum, and before each audit committee cycle, so ratings presented together were set to one standard.
Does using an agent affect auditor independence?
The agent gathers and drafts; the auditor concludes. Independence depends on who decides, and the record shows that.
Should the audit committee see calibration results?
The standards, yes. Individual spreads stay within the audit team.