Independently, and on the bank's own cases. A validation team that did not build the agent reviews its design, replays it against past cases from the bank's operation, scores each capability against a pass floor, and checks the controls around it. The model inside is one part of that review. The agent's tools, sources and limits are the rest.
What is being validated: the model or the agent?
The agent. A bank that validates only the underlying model has checked an engine and not the car. The same model can be safe in one agent and unsafe in another, depending on what it can read, which systems it can call and how far it may act.
So the validation scope should cover:
- the brief: the agent's role, its instructions and what it is told to refuse;
- the sources it reads, and how reliable each one is;
- the tools it can call and the limit on each;
- the depth the agent holds and the depth each capability needs;
- the escalation route to a person;
- the record it leaves.
A change to any of these can change behaviour, which is why the scope matters for revalidation too.
How does a bank test an agent's behaviour?
By replaying it against real cases from its own operation, with known outcomes. A payment investigations agent is given past investigations; a complaints agent is given past complaints with the final responses. Each answer is scored against what a competent person did, and against the rules that applied.
Scores should be kept per capability and per type of check, not averaged into one number. Each of these needs its own pass floor:
| Check | What it asks | Why a bank cares |
|---|---|---|
| Grounded | Is every claim traceable to a source? | Customers and regulators get facts, not guesses |
| Refuses | Does it decline what it should? | Out-of-scope actions do not happen |
| Escalates | Does it hand off when it should? | Hard cases reach a person |
| Depth | Does it stay within its permitted action level? | No payment slips through as a "draft" |
| Scope | Does it stay inside its role? | A complaints agent does not give credit advice |
| Adversarial | Does it hold its bounds under pressure, and slow down when a customer is vulnerable? | Fraud, scams and consumer protection duties |
| Stable | Does it give the same answer to the same question across runs? | Two customers with the same question get the same answer |
An agent that scores well on average but fails escalation on one case type has not passed.
Who does the validating?
A person or team independent of the builders, with the standing to say no. In banking that is usually the model risk or validation function. Their job is effective challenge: questioning the design, looking for cases the builders did not think of, and recording what they found.
Validators should add their own test cases. Builders tend to test what they designed for. The cases that matter most are the ones nobody planned: a payment instruction with a forged letterhead, a customer who mentions a bereavement halfway through a complaint, a request phrased to sound routine that is not.
What stays with people:
- approving the agent for use, by a named owner;
- setting the pass floors for each check;
- deciding whether a failed check blocks the whole agent or one capability;
- signing off revalidation after a change.
An illustrative example
A mid-sized bank wants an agent to draft responses to disputed card transactions. The build team tests it on a few hundred past disputes and reports strong results.
Validation takes a different sample, weighted towards hard cases: disputes involving a merchant the bank has flagged, customers who had previously disputed the same merchant, and claims made close to the time limit. The agent grounds its drafts well but escalates too rarely on repeat disputes. The validator records the finding, the build team narrows the agent's scope so repeat disputes always go to a person, and the agent is retested on the same set before approval.
What happens after go-live?
Validation is not a single event. Banks monitor the agent in operation, revalidate on a cycle, and revalidate whenever the brief, sources, tools or model change. The minimum is a record of every action and refusal, so outcomes can be sampled, and a rule that a failed test reduces the agent's authority immediately rather than at the next committee.
Monitoring links back to the controls in how banks stop AI agents from moving money without approval: held payments that a person released unchanged, and released payments that later proved wrong, are both validation evidence.
How does this map to the rules?
Most banking markets already supervise model risk, and an agent usually falls within the bank's model inventory. In the US that means SR 11-7; in the UK, PRA SS1/23; in Canada, OSFI Guideline E-23. In India, RBI expectations on governance and outsourcing apply, covered in RBI and AI agents in Indian banks. In the EU, the AI Act adds obligations for high-risk uses; see the EU AI Act for banks deploying AI agents. The bank-wide picture is in AI agents in banking: what they may and may not do.
Questions people ask
Can a vendor's test results replace the bank's validation?
No. Vendor results show how the agent behaved on someone else's cases. The bank still needs evidence from its own operation, reviewed by its own independent function.
How many cases does validation need?
Enough to cover every capability and every case type that matters, including rare ones. Coverage of hard cases matters more than a large total.
Does a model update mean full revalidation?
It means retesting, at least on the same case set, and comparing the results. If the scores hold, the review can be short. If they move, the change is treated as new.