FCA Consumer Duty evidence: monitoring outcomes of AI agents

Consumer Duty monitoring of AI agents needs outcome data, not activity data. Record for each conversation what the customer wanted, whether they got it, how long it took, whether they came back or complained, and whether they reached a person. Break it down by customer group, including those with characteristics of vulnerability, and compare it with human channels.

What does the Duty require firms to monitor?

The Consumer Duty requires firms to monitor and regularly review the outcomes their customers experience, and to be able to evidence that they deliver good outcomes across the four areas: products and services, price and value, consumer understanding and consumer support. Firms must identify where outcomes differ for different groups, including customers with characteristics of vulnerability, and act on what they find. The board, or equivalent governing body, must review at least annually whether the firm is delivering good outcomes. Our explainer on what the Consumer Duty is sets out the framework.

Monitoring is not the same as testing before launch. Testing asks whether an agent should go live; monitoring asks what is happening now that it has. Our companion piece on testing agents against good outcomes covers the first.

Who does it apply to?

FCA-authorised firms serving retail customers. Any firm using an agent with retail customers needs the agent's outcomes in its monitoring; our short answer on whether the Duty applies to agents explains why.

What outcome data should an agent produce?

Agents have an advantage over human channels here: every conversation can be recorded and classified the same way. The trap is collecting what is easy (conversation counts, average handling time) rather than what shows outcomes. Collect these, per conversation:

1. Intent. What the customer was trying to do, from a fixed list of issue types.
2. Resolution. Whether the issue was resolved, and how: by the agent, by a person after hand-over, or not at all.
3. Hand-over. Whether the customer asked for or was offered a person, and how long it took to reach one.
4. Repeat contact. Whether the same customer came back about the same issue within a set window.
5. Complaint. Whether a complaint followed, linked to the conversation.
6. Drop-off. Whether the customer abandoned the conversation part way, and at what point.
7. Action outcome. For agents that act, what was done: a refund applied, a card frozen, a payment date changed, an action held for a person.
8. Vulnerability signals. Whether signals were detected and what followed, recorded with care for data protection, since health information is special category data.
9. Product outcome. Where relevant, what the customer ended up with: product taken, product cancelled, fee charged or waived.

Then compare. Outcomes for agent conversations against human ones for the same issue types. Outcomes for customers with vulnerability signals against those without. Outcomes before and after a change to the agent. A difference is not automatically a problem, but every difference needs an explanation.

How do you turn the data into evidence?

The last row matters. If an agent classifies its own conversations as resolved, you are trusting the agent to mark its own work. Check tagging with a person on a sample, and use independent signals such as repeat contacts and complaints. An agent can help analyse the data, and our answer on whether an agent can monitor outcomes for Consumer Duty covers that role, but conclusions and actions belong to people, in line with our guide to AI compliance agents.

Communications deserve their own measures, such as repeat questions within a conversation and complaints citing confusion. See Consumer Duty and AI agent communications.

What will the FCA or the board ask?

• What outcome data do you collect from agents, and how is it defined?
• How do outcomes for agent conversations compare with human ones?
• Do customers with characteristics of vulnerability fare differently with the agent?
• What changed after you found a poor outcome?
• How do you know the agent's own tagging is accurate?
• What did the last board review conclude about agents?

What does outcome monitoring not cover?

It does not replace testing before deployment or after change. It does not satisfy the complaints rules' own root cause analysis, though it should feed it. It does not cover non-retail customers. And collecting vulnerability data raises its own data protection duties under UK GDPR, which must be met separately.

This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.

Where Pryme fits

Pryme files every agent verdict, allowed, held, refused or withheld, in an append-only record with the action and the bound that decided. That record gives a firm reliable action outcome data, separate from the agent's own description of the conversation, and makes before-and-after comparisons simple to run when an agent changes.