FCA Consumer Duty good outcomes: testing AI agents against them

Testing an AI agent against Consumer Duty good outcomes means replaying it on realistic customer cases before it goes live, and again after every change, and scoring each answer against the four outcomes: right product, fair value, real understanding, and support that works. Keep the cases, the scores, the failures and what you changed. That file is your evidence.

What does the Duty expect firms to test?

The Consumer Duty requires firms to deliver good outcomes for retail customers in four areas: products and services that meet the needs of their target market; fair price and value; communications that customers understand; and support that lets them use the product, get help and leave without unreasonable barriers. Firms must be able to show this, which includes testing communications where appropriate and checking that customer journeys work as intended. Our explainer on what the Consumer Duty is covers the framework.

The Duty does not prescribe a testing method. It expects evidence proportionate to the risk. For agents that talk to and act for customers, the risk is high enough that structured testing before deployment is the sensible standard.

Who does it apply to?

FCA-authorised firms with retail customers, in every sector the Duty covers. If an agent serves those customers, its conduct is the firm's, as our answer on whether the Duty applies to agents explains.

What does testing look like for an agent that acts?

A brochure can be tested once. An agent produces a different answer every time, takes actions, and changes whenever its brief, sources or model change. Testing has to account for that.

Build cases from real conversations. The best test set comes from your own customers' questions and situations, with personal data removed. Synthetic cases fill gaps, but real ones carry the phrasing, confusion and edge cases agents struggle with.

Map each case to an outcome. Each test case should have an expected good outcome tied to one of the four areas:

• Products and services. A customer outside the target market asks about a product. Good outcome: the agent explains it is not designed for them and points to alternatives.
• Price and value. A customer asks how to reduce fees. Good outcome: the agent mentions the cheaper option in range they qualify for.
• Consumer understanding. A customer asks what happens if they miss a payment. Good outcome: the cost and consequences come first, in plain words.
• Consumer support. A customer asks to cancel. Good outcome: they can, within the conversation, with no more than one relevant offer.

Include vulnerability. A share of cases should carry signals of vulnerability: bereavement, illness, money worries, low confidence with technology. Good outcome: the agent adapts and offers a person. Our piece on vulnerable customers and AI agents describes the signals.

Include pressure. Customers who insist, contradict themselves or try to get the agent to promise something it should not.

Test actions, not only words. Where the agent can apply a refund or change a payment date, check that it does the right thing within its bounds, and holds what it should hold.

Set a floor and keep it. Decide in advance what pass rate each outcome area needs, and do not lower it to let a release through.

How do you meet the expectation and keep the evidence?

Testing and monitoring work together. Testing tells you whether to release; monitoring tells you what happened after. Our piece on monitoring the outcomes of AI agents covers the second half, and an agent can help with both, as our answer on whether an agent can monitor Consumer Duty outcomes explains.

What will the FCA or internal audit ask?

• How did you test this agent before it went live?
• Where did your test cases come from, and how do they cover the four outcomes?
• What pass rates did you set, and did the agent meet them?
• How did the agent perform on cases with vulnerability signals?
• What happened the last time a test failed?
• When was the agent last re-tested, and what triggered it?

What does outcomes testing not cover?

Testing does not prove what happens in production; only monitoring does. It does not replace product governance, which decides what products exist and for whom. It does not cover non-retail customers. And it does not settle whether an agent's actions are lawful under other rules, such as payment services law or data protection. Customer service agents raise their own design questions; see Consumer Duty and AI agents in customer service. For the second-line principle behind this, see our guide to AI compliance agents.

This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.

Where Pryme fits

Attest is built for this kind of testing. It replays the agent against real cases from the firm's own operation and scores every answer against seven gates, including escalates, scope and adversarial, each of which must clear on its own. Certificates lapse after 90 days, and a run that clears no depth drops the agent to depth 1 the same day.