How do you audit a compliance agent's recommendations?

Audit a compliance agent's recommendations by sampling them, reconstructing each one from the record, re-performing the work independently and comparing the result with the agent's recommendation and the person's decision. Then look at patterns: where people overrode the agent, where they did not but should have, and what the agent never looked at.

What exactly is being audited?

A compliance agent recommends: close this alert, discount this match, rate this finding medium, escalate this case. A person then decides. The audit question is not only "was the recommendation right?" It is three questions.

• Was the recommendation sound on the evidence the agent had?
• Did the agent have the right evidence, or did it miss sources it should have read?
• Did the people deciding apply real judgement, or did they accept recommendations without challenge?

The third question is often the most revealing. A team that accepts nearly every recommendation may be well served by a good agent, or may have stopped thinking. Only re-performance can tell which. Note that "recommendations" here means the agent's outputs, not the FATF standards of the same name; for those, see FATF Recommendations and AI agents in AML.

How do you run the audit, step by step?

1. Define the population. Internal audit or the second line lists every recommendation the agent made in the period, by type, from the agent's record. Evidence: the population extract with counts.
2. Stratify and sample. The auditor samples across types, outcomes and confidence levels, with extra weight on recommendations to close or discount, where errors hide. Evidence: the sampling method and selection.
3. Reconstruct each recommendation. For each item, the auditor retrieves what the agent read, the provenance of each source, the rule it cited, its reasoning and its output. Evidence: the reconstructed file. If an item cannot be reconstructed, that is a finding on its own.
4. Re-perform independently. A qualified person, not the original decision-maker, works the case from the raw sources without seeing the agent's recommendation first. Evidence: the re-performance working paper.
5. Compare three things. The agent's recommendation, the original person's decision and the re-performance. Evidence: a comparison table with each disagreement explained.
6. Analyse overrides. The auditor looks at all cases where people overrode the agent, not just the sample, and checks whether overrides were justified and whether they cluster. Evidence: override analysis.
7. Check what was not looked at. For sampled cases, the auditor checks whether the agent should have read sources it did not. Evidence: a list of missed sources.
8. Check bounds and changes. The auditor confirms the agent acted within its scope, its certification was current, and changes to its brief were approved. Evidence: certificate and change records.
9. Report. Findings, ratings and actions go to the audit committee. Evidence: the audit report.

The general method of reconstructing agent actions is in how do you audit what an AI agent did? and the record it depends on is described in what an audit trail is.

What might an audit of recommendations find?

Take an internal audit of an agent that recommends dispositions for sanctions screening alerts. The population is 30,000 recommendations over six months. Audit samples 150, weighted towards discounts.

Re-performance agrees with the agent on 144. Of the six disagreements, four are cases where the agent recommended discounting on a date-of-birth mismatch, but the customer record held two dates of birth and the agent used only the first. In all four, the analyst accepted the recommendation. None were true matches on re-review, but the pattern shows a data gap and an approval habit. The override analysis shows analysts overrode the agent in a tiny number of cases across the whole population, almost all in one direction: escalating what the agent proposed to discount.

Audit raises two findings: the agent's handling of multiple identifiers, and the depth of analyst review on discounts. Both have owners and dates. Preparing an agent so this kind of audit runs smoothly is covered in how to prepare AI agents for an audit.

Which controls make recommendations auditable?

• Every recommendation is filed with its sources, provenance, rule and reasoning at the time it was made.
• Human decisions are filed alongside, with reasons for overrides.
• Records are append-only, so the reconstruction matches what happened.
• Certification and change history are kept for each agent.
• Auditors have read access to the record without needing the agent's own permissions.

The general principle is in our guide to AI compliance agents.

Where Pryme fits

In Pryme, every recommendation and every human decision is filed in an append-only record with its sources, provenance class, the rule cited and the bound that decided the verdict. An auditor can reconstruct any item without the agent's permissions, and can see which certificate was in force when the recommendation was made.