What is model validation?

Model validation is the independent testing of a model to confirm that it is conceptually sound, works as intended and is fit for the specific use it is put to. It is done by people who did not build the model, before use and periodically after, and ends in a finding: approved, approved with conditions, or not approved.

Why does model validation matter once AI agents act?

Because validating the underlying model is not the same as validating the agent. A model provider's model may be capable in general and still produce an agent that fails on your work, because of its instructions, its sources, the tools it can call or the limits around it. What needs validating is this agent, with this brief, for this use.

Generic benchmark scores do not answer that question. A validator needs evidence from the bank's own cases. Supervisors expect validation to be done by people with the skill, independence and standing to change the outcome, not by the team that built the agent marking its own work. How this plays out under US guidance is covered in SR 11-7 model validation for AI agents, and under Canadian guidance in OSFI E-23 model lifecycle and AI agent validation.

How is an agent validated in practice?

Validation traditionally has three parts, and each translates to agents:

• Conceptual soundness: is the design fit for the purpose? For an agent, review the brief, the sources it may read, the actions it may take and the limits on each.
• Outcomes analysis: does it produce the right results? Replay the agent against past cases with known outcomes and score each answer.
• Process verification: do the controls work? Test that limits stop what they should, that the agent escalates when it should, and that every verdict is recorded.

Each part ends in findings, and each finding needs an owner and a date to close it.

What does that look like in an example?

A bank deploys an agent that drafts the narrative section of suspicious activity reports for its financial crime team. The validators review the brief and the typology documents it draws on. They replay several hundred past cases where the outcome is known and compare the agent's narratives with those the team filed. They test whether it ever presents a draft as final. They approve it with two conditions: a trained investigator reviews every narrative before filing, and any change to the typology documents triggers revalidation on the affected case types.

How is model validation different from drift monitoring?

Validation is a judgement at a point in time: is this model fit for use, now? Monitoring watches performance continuously after that judgement. Model drift is what monitoring looks for: a change in behaviour or accuracy since validation. Monitoring tells you when to validate again. Validation tells you whether the model was fit to begin with. Neither replaces the other.

Which related terms should you know?

• Model risk management: the wider discipline that validation sits within.
• Effective challenge: critical review by people able to change the outcome.

Whether an agent is a model for these purposes is the starting question, answered in is an AI agent a model under bank model risk rules?, and how this fits with other limits on agents in a bank is set out in AI agents in banking.