Test it on your own closed cases, where you already know the right answer. Replay hundreds of past alerts through the agent, score every output against floors agreed with your money laundering reporting officer, count missed suspicion separately from false positives, then run it in shadow beside analysts before it touches a live queue at draft depth.
What exactly are you testing?
Not "the AML agent", but each capability it will use. Summarising an alert, gathering customer history, drafting a case narrative and recommending closure are different jobs with different failure modes. Each one is tested and approved on its own, at its own depth.
This matters because a single overall score hides the dangerous part. An agent can write excellent summaries and still recommend closing cases it should escalate. The structure of the work being tested is set out in how to prepare AML case investigations with an AI agent, and the second-line principle behind it in the guide to AI compliance agents.
How do you build the test set?
Use your own history, not a vendor's demonstration data. Your customers, products and typologies are what the agent will meet.
- Pull closed alerts across outcomes: ones closed as false positives, ones escalated, and ones that led to a suspicious activity report.
- Over-weight the hard cases: borderline decisions, cases later reopened by quality assurance, and cases where two analysts disagreed.
- Add cases with missing data, such as an incomplete customer file, so you can see whether the agent says what it could not check.
- Hold back a portion the agent's builders never see, so the result is not tuned to the test.
- Remove personal data you do not need for the test, and record the lawful basis for the rest.
What should every answer be scored on?
| Measure | What a fail looks like |
|---|---|
| Missed suspicion | Recommends closing a case your team escalated or reported |
| False positives | Escalates cases a reviewer would close in seconds |
| Grounding | States a fact that is not in the case file or the customer record |
| Completeness | Skips a check the procedure requires, without saying so |
| Recording gaps | Fails to note what it looked for and did not find |
| Disclosure | Puts suspicion wording anywhere a customer could see it |
Set the floor for missed suspicion separately and set it high. One missed reportable case is not offset by a hundred well-handled false positives. The trade-off on the other side is explained in what a false positive in AML is: an agent that escalates everything wastes analyst time but fails safe.
What does a test run look like in practice?
Take a mid-sized e-money firm planning an agent to prepare transaction monitoring cases. The financial crime team pulls 600 closed alerts: 450 closed as false positives, 120 escalated, 30 reported. Fifty borderline cases are held back.
The agent drafts a case narrative and a recommendation for each. It matches the team on most closures but recommends closing two of the reported cases, both involving funds routed through a newly opened business account. That is a fail on the missed-suspicion floor, whatever the other scores say. The team adds a required check on account age, reruns the full set and the held-back cases, and the agent clears. It then runs in shadow for a month, drafting cases that analysts work in parallel without seeing its output first.
Who signs off, and what happens after launch?
The money laundering reporting officer, or the person with equivalent responsibility, signs off the result and the depth. Testers should be independent of the people who built the agent. A first deployment sits at depth 2: the agent drafts, an analyst decides and files.
Testing does not stop at launch. Sample live cases, compare against analyst decisions, and rerun the full set when the model, the procedure or the typologies change. The same discipline applies to customer-facing agents, as in how to test an AI support agent before launch, though the stakes here are higher.
Where Pryme fits
Attest does this replay as a certification step: it runs the agent against real cases from your own operation and scores every answer against a floor on seven gates, each of which must clear on its own. Certificates lapse after 90 days, and a run that clears no depth drops the agent to depth 1 the same day.
Questions people ask
Does the EU AI Act require this kind of testing?
The Act's duties depend on how the system is classified and whether you are provider or deployer; the position for financial crime teams is set out in the EU AI Act and AI agents in compliance and AML teams. Testing on your own cases is good practice regardless.
Can another agent run the tests?
An agent can execute test runs and tabulate results, as covered in whether an AI agent can test compliance controls. Choosing the test set and passing the result stays with people.
How many cases is enough?
Enough to include every typology you care about several times, plus the hard cases. A large set of easy alerts proves very little.