How to run calibration sessions for AML investigators

Give AML investigators the same set of closed alerts, each with the agent's drafted disposition, and have them decide alone: close, escalate for a suspicious activity report, or request more information. Compare, discuss the splits, and have the money laundering reporting officer set the gold standard. Feed it into the agent's brief and sample the next month's dispositions.

Why does AML investigation need calibration?

An AML monitoring agent typically reads alerts, gathers account history and counterparties, and drafts a disposition with a narrative at depth 2. Investigators decide. In some firms, the agent may close low-risk alerts of defined types at depth 3 within the firm's own limits, with sampling. Escalation to a suspicious activity report is always a person's decision.

Investigators work under a risk-based approach, which the FATF Recommendations set as the international standard. Risk-based means judgement, and judgement varies. One investigator closes rapid in-and-out movement on a student account as normal; another escalates every instance. Both are defensible alone. Together they produce inconsistent outcomes and teach the agent noise. Calibration narrows that spread. The general method is in how to run calibration sessions for AI agent reviewers.

The MLRO or a deputy owns the standard. A senior investigator facilitates. Quality assurance attends.

What is different about calibrating AML work?

Confidentiality. Alerts that led to reports are highly sensitive, and nothing in the session must risk tipping off a customer. Use closed cases, restrict the material to the investigation team, keep it inside the case management system, and do not copy it into slides or chat.

Narratives matter as much as outcomes. Two investigators can both escalate and write very different reasons. Supervisors and law enforcement read the narrative. Calibrate the narrative as well as the decision.

The agent drafts the first narrative. If investigators accept weak drafts, the agent's narratives never improve. Calibration should include whether each investigator would have changed the agent's draft, and how.

Which alerts belong in the set?

Eight to ten closed alerts, mixed:

• Cash deposits just under a reporting threshold across several branches.
• A name match against a politically exposed persons list that turned out to be a different person.
• Payments from many unrelated third parties into a personal account.
• Funds moving in and out within a day with no clear purpose.
• An invoice-based payment where the goods described do not fit the business.
• One alert the agent closed at depth 3, to test whether investigators agree it should have been.

The session, step by step

1. Person (senior investigator): selects closed alerts, removes investigator names, and confirms with the MLRO that the set is safe to use.
2. Person (each investigator): decides each alert alone, writes a short narrative, and notes whether they would have changed the agent's draft.
3. Agent, depth 1: shows its drafted disposition, narrative and the sources relied on, with provenance: sealed ledger data, signed counterparty information, declared customer explanations, external documents.
4. Person (facilitator): reveals the spread on decisions, then on narratives.
5. Person (investigators): explain their reasoning, focusing on which facts and which provenance moved them.
6. Person (MLRO): sets the gold standard on each disputed alert and on what a good narrative contains.
7. Person (MLRO): sends the standard to the brief owner, including any change to which alert types the agent may close.
8. Person (quality assurance): samples dispositions for the next month against the standard.

What does a gold standard look like?

It names the facts that decide the case. For example: "Third-party credits from five or more unrelated payers in a month into a personal account, where the customer's declared occupation does not explain them, are escalated for review by a senior investigator, not closed on the customer's explanation alone. The customer's explanation is declared and is recorded as such in the narrative."

A narrative standard might read: "Every narrative states what was unusual, what was checked, the provenance of each fact relied on, and why the decision follows."

Which controls does calibration support, and what evidence does it leave?

Consistent dispositions. Evidence: the independent decisions and their spread.

Narrative quality. Evidence: the narrative standard and sampled narratives against it.

Scope of agent closure. Evidence: the MLRO's recorded decision on which alert types the agent may close at depth 3.

Confidentiality. Evidence: where the session materials were held and who accessed them.

What should the MLRO measure?

• Agreement on close or escalate, session by session.
• Sampled narratives meeting the standard.
• Agent-closed alerts that investigators would have escalated. Any is a reason to narrow the agent's closure scope.
• How often investigators rewrite the agent's narrative, and whether that falls as the brief improves.

Where does it go wrong in AML teams?

Calibrating outcomes only. Everyone agrees to escalate, and the narratives remain uneven.

Using live investigations. Never. Closed cases only.

Avoiding the agent's closures. If no agent-closed alert is ever in the set, nobody checks whether depth 3 closure is still justified.

Calibration for other reviewers looks different: collections agents calibrate on tone and affordability, and IT administrators on access risk. For planning, see the AI agent training plan template for AML investigators and the topic guide on training teams for AI agents.