Testing controls on agents that act is the skill compliance analysts need most. In retail banking, that means mapping a rule into the bounds an agent must respect, sampling the decision records the agent leaves behind, checking that held and refused verdicts happen where they should, and explaining to a supervisor how human oversight works in practice.
Why is this different from testing ordinary systems?
A rules engine does the same thing every time. An agent reads a case and reaches a judgement, so two similar cases can come out differently. Compliance can no longer test the rule once and sign off; it has to test outcomes continuously.
The good news is that a governed agent leaves better evidence than most human processes. Every verdict is filed in an append-only record that shows what the agent saw, which source each fact came from, what it proposed, which bound decided the outcome and who approved it. An analyst who can read that record has a ready-made audit trail.
What does the analyst check?
| Check | What you look for in the record |
|---|---|
| Scope | The agent only acted within its stated role |
| Depth | Actions match the depth the agent earned, not more |
| Sources | Decisions rest on strong sources; external claims never authorised a payment |
| Bank limits | Cases above the bank's limits were refused, and only drafted for a named person |
| Red lines | Nothing was allowed that should have been refused or withheld |
| Certificate | The agent's certificate was current; a run that cleared no depth dropped it to depth 1 the same day |
Remember the difference between refused and withheld. Refused means the action fell outside this agent's bounds. Withheld means the platform blocks it for every agent at every depth, whatever the bank configures. Analysts who mix them up write findings that cannot be fixed.
What does a control test look like?
Take a compliance analyst we will call Sofia at a retail bank. A complaints agent drafts final responses and may issue small redress payments within a limit. Sofia samples fifty verdicts from the month. Forty-eight are fine. One redress payment of 240 went through without a hold, although the customer had mentioned a bereavement; the bank's policy says vulnerability cases are held. One final response cited a fee table the agent pulled from an old web page rather than the bank's own tariff.
Her finding names two bound failures, not "the AI got it wrong": the vulnerability hold rule was missing from the agent's limits, and an external source was in its allowed list. Both are fixable, and her retest a fortnight later proves it.
Which skills should you build?
- Reading decision records fluently, including the line that names the deciding bound.
- Rule-to-bound mapping. Turn a policy into scope, limits and holds an agent must respect.
- Sampling and testing agent verdicts, with clear pass criteria.
- Source judgement: sealed, signed, declared and external, and which may support which decision.
- AI rule literacy. Depending on your market: EU AI Act duties on human oversight (Article 14), record-keeping (Article 12) and AI literacy (Article 4); GDPR Article 22 on solely automated decisions; in the UK, FCA Consumer Duty outcomes and named accountability under the Senior Managers and Certification Regime; model risk expectations such as PRA SS1/23 or SR 11-7.
- Writing findings that point to a fix, not just a failure.
Analysts who sit closer to financial crime should also read AI skills for a fraud analyst and AI skills for a KYC analyst.
How do you show it?
Describe the tests you have run and the evidence you used. The AI agent skills CV section for a compliance analyst offers wording, and rewriting a compliance analyst CV for AI operations jobs covers the full document. Titles vary: "AI assurance", "conduct risk", "controls testing" and "second line" adverts may all involve agents. A professional certificate that an employer can check at source, such as those issued through Pryme Intelligence Academy, shows you have reviewed agent decisions to a standard; it is not a regulatory qualification. More on proof in AI skills for job seekers.
Questions people ask
Do compliance analysts need to understand how models work?
Only at a high level. You need to know that an agent's output can vary and drift, and how the controls around it work. Testing outcomes and bounds matters more than the internals of a model.
Is AI oversight a first-line or second-line job?
Both. The first line reviews held decisions day to day; compliance tests whether that review and the agent's bounds work. Knowing both sides makes your testing sharper.
Will regulators accept agent decision records as evidence?
Regulators expect firms to evidence their controls, and an append-only record of every verdict helps. Whether it satisfies a particular requirement depends on the rule and the market; confirm with your firm's legal team.