Can an AI agent test compliance controls?

Yes, and often more thoroughly than a sample allows. An AI agent can select samples or test whole populations, re-perform the control, compare results with what the control owner recorded, and document every exception. Concluding whether a control is designed well and operating effectively, and rating the findings, stay with the compliance officer.

What does testing a control involve?

A compliance control is a step the firm relies on to meet an obligation: a four-eyes check on a payment, a suitability review before advice is given, a daily reconciliation, a screening step at onboarding. Testing asks two questions. Is the control designed to address the risk? And did it operate as designed, every time it should have, during the period?

The second question is where the hours go. A tester pulls a sample, finds the evidence for each item, re-performs the check and records the result. For a control that runs thousands of times, a sample of twenty-five tells you something, but not everything. An agent can often test the whole population, which changes what you can conclude.

What may the agent prepare?

The agent may:

• Read the control description and draft a test script: what evidence proves operation, and what counts as an exception.
• Select a sample under your methodology, or test the full population where the data allows.
• Re-perform the control on each item: re-run the check, compare with the recorded outcome, and note differences.
• Check evidence quality: whether the evidence was created at the time, by the right person, and is complete.
• Record each exception with the item, the evidence and why it fails.
• Draft the test results and a proposed rating.

The compliance officer decides:

• Whether the control's design addresses the risk.
• Whether exceptions are failures of the control or explained deviations.
• The rating of the control and of any findings.
• What the business must remediate and by when.

Can an agent test the controls that agents run?

Increasingly, the controls being tested are run by agents themselves: an agent screening customers, an agent checking payments before release. Testing those needs two things beyond output checking.

First, test the bounds, not only the outputs. An agent that produced correct results in the sample may still be able to act outside its scope. The test should try actions the agent should not be able to take and confirm they are refused. Our view is set out in compliance monitoring of agents: test the bounds, not just the outputs.

Second, keep the tester separate from the tested. An agent should not test a control it operates, or one operated by an agent with the same brief and sources. Pre-deployment testing of an AML agent is a related but different exercise, covered in how do you test an AML agent before using it?

What does a control test look like?

Take a building society testing its control that a second person approves every payment over a set limit before release.

The agent tests the full population for the quarter: 4,200 payments over the limit. It finds 4,188 approved by a second person before release. Of the twelve exceptions, nine were approved by the same person who created them, using a second login. Three were released before approval, all during a system outage, with retrospective approval the next day.

The draft rates the control as partly effective. The compliance officer agrees, but treats the two groups differently. The nine self-approvals are a serious design weakness: the system allowed one person to hold two logins. The three outage cases are an operating failure with a documented contingency, which she rates lower. She orders the logins removed and the outage procedure tightened. When the control is part of the financial reporting framework too, the same evidence serves both; see how do you evidence financial controls when an AI agent does the work?

Which controls govern the testing itself?

• Test scripts are approved. The officer approves what the agent will test and how before it runs.
• Full population where possible. Where data allows, test everything, and say so in the result.
• Evidence is re-performed, not just read. A recorded "approved" is checked against who approved and when.
• The tester is independent. The testing agent has its own brief, sources and record.
• Results are reproducible. Another tester, person or agent, can re-run the test and get the same exceptions.

The general principle is in our guide to AI compliance agents.

Where Pryme fits

In Pryme, a testing agent reads records and re-performs checks at read and draft depth; it cannot change the data it tests. When the control under test is run by another agent, the tester can also read that agent's filed verdicts, including refusals, so it sees both what the agent did and what it was stopped from doing.