How to review transaction monitoring rule performance with an AI agent

A monitoring rule earns its place by finding activity worth investigating, and most rules are never checked on that. The agent analyses each rule's alerts and outcomes, finds dead rules and uncovered typologies, and tests proposed changes against historical data in a copy of the system. The monitoring lead approves any change, and production rules change only through change control.

What is rule performance review, and who owns it?

Transaction monitoring systems run rules or models over account activity. Over time, rules decay: customer behaviour changes, products change, thresholds set years ago no longer fit. A rule that produces thousands of alerts and no cases wastes analyst time; a rule that produces none may be blind. Supervisors expect firms to tune and validate their monitoring and to show the evidence.

The transaction monitoring lead owns the rules, under the MLRO. A data or analytics team often runs the analysis, and model risk or independent validation reviews material changes where the firm treats monitoring rules as models. Whether an AI agent can do transaction monitoring discusses the boundary for the monitoring itself; this playbook is about the review of how well it works.

What does the agent do at each step, and at what depth?

1. Person, monitoring lead: sets the review scope. Rules in scope, period, and the outcome measures to use.
2. Agent, depth 1: extracts alert history. For each rule: alert volumes, dispositions, escalations to cases, cases leading to suspicious activity reports, and analyst time.
3. Agent, depth 1: calculates performance. Conversion from alert to case to report, by rule, segment and period, with trends.
4. Agent, depth 1: maps coverage. Rules against the typologies in the firm's risk assessment, showing typologies with weak or no coverage.
5. Agent, depth 1: analyses below-threshold activity. Samples activity just below each threshold to see whether it contains cases the rule misses.
6. Agent, depth 2: drafts tuning proposals. Threshold, segmentation or logic changes, each with the expected effect on volumes and the evidence for it.
7. Agent, depth 1: tests proposals in a copy. Re-runs the proposed rule over historical data in a test environment and compares the alerts produced with known cases. It has no access to production configuration.
8. Person, monitoring lead: decides which proposals to take forward.
9. Person, validation or model risk: reviews material changes, and change control applies them to production.

Which controls apply, and what evidence does each leave?

No production access. The agent tests in a copy. Evidence: the agent's identity has no write rights on production rules, confirmed in access reviews. The connection is described in connecting an agent to your transaction monitoring system.

Below-threshold testing done. Tuning that only looks at alerts can raise thresholds and never see what it stopped catching. Evidence: below-threshold samples and findings for each change.

Expected effects recorded before the change. Evidence: predicted volumes and conversion, compared with actual results after go-live.

Independent review of material changes. Evidence: validation sign-off.

Coverage map kept current. Evidence: the map linked to the business-wide risk assessment.

What should you measure?

• Alert-to-case and case-to-report conversion per rule, over time.
• Typologies with no effective coverage.
• Predicted versus actual effects of each change.
• Analyst time freed by retiring or tuning rules.
• Cases found below old thresholds after tuning, which show what was missed.

Where does it usually go wrong?

Tuning for volume. The easiest way to cut alerts is to raise thresholds. Without below-threshold testing, this quietly reduces detection.

Conversion as the only measure. A rule with few reports may still matter if it covers a rare, serious typology. Judge rules against the risks they cover, not only their hit rate.

Analyst behaviour mistaken for rule quality. If analysts close a rule's alerts quickly by habit, conversion looks poor because of the review, not the rule. Sample closed alerts before retiring a rule. Summaries from letting an agent summarise alerts can help show whether closures were reasoned.

Segments ignored. A threshold that suits retail customers may be useless for small businesses. Analyse by segment.

No feedback from investigations. Cases found by other means, such as referrals or law enforcement requests, show what monitoring missed. Feed them into the coverage map.

Test data that does not match production. A test copy with old customer data or missing fields gives tuning results that do not hold in production. The agent should report how the test data differs from production, such as its date and any masked or missing fields, so the lead can judge how far the results can be trusted.

How to measure the agent itself is covered in how to measure a compliance agent's performance. The overall stance is in AI compliance agents.

Where Pryme fits

In Pryme, the analysing agent holds depth 1 on alert history and the test copy, and depth 2 for tuning proposals. Production rule configuration is outside its role scope, so any write attempt is refused and the verdict names role scope as the bound that decided. Every analysis and proposal is filed, which gives validation a complete record.