How to run customer outcome testing with an AI agent

Let the agent read the files and a person call the result. It draws the sample, applies your test questions to each customer file, and drafts a finding with the evidence quoted. A named tester confirms every potential failure and checks a share of passes. Compliance owns the test design, and business owners decide the remediation.

What is customer outcome testing, and who owns it?

Outcome testing checks whether customers actually ended up where they should: the product suited them, the price was fair for what they got, they understood what they bought, and they were helped when they needed help. It differs from process testing, which asks whether staff followed the procedure. A file can pass every procedural check and still leave the customer worse off.

In the UK, the FCA Consumer Duty makes good outcomes for retail customers an explicit expectation, and firms test against it; testing AI agents against Consumer Duty good outcomes covers that frame. Firms elsewhere run the same kind of testing under their own conduct rules.

Compliance monitoring or a quality assurance team owns the testing. The business owns the outcomes. The agent sits in the second line as a reader, in line with AI compliance agents: second line, never the last word.

What does the agent prepare, and what does a person approve?

1. Person, compliance: writes the test script. The questions, what a pass and a fail look like for each, and which outcomes count as customer harm. The agent never writes its own pass criteria.
2. Person, compliance: sets the sample. Size, period, product, channel, and any targeted segments such as customers flagged as vulnerable.
3. Agent, depth 1: draws the sample from the case system to the defined rules, and records the selection so it can be repeated.
4. Agent, depth 1: reads each file. Call notes, letters, chat transcripts, product terms, fees charged, complaints raised later. Every item carries its source.
5. Agent, depth 2: answers each test question. A proposed pass, fail or "cannot tell", with the passage that supports it. "Cannot tell" is a valid answer and usually means the file is incomplete.
6. Agent, depth 2: drafts a harm note where a fail may have cost the customer money or left them without help.
7. Person, tester: reviews. Every proposed fail, every "cannot tell", and a set share of proposed passes. The tester's result is the result.
8. Person, compliance lead: signs off findings and themes, and sends them to the business owner.
9. Agent, depth 2: drafts the findings report from the signed-off results.

Which controls matter, and what evidence does each leave?

Fixed script, versioned. Results are only comparable if the questions do not drift. Evidence: the script version on every tested file.

Every answer quotes its basis. A proposed fail without the passage is rejected by the tester. Evidence: the quoted extract stored with each answer.

Passes are sampled too. If testers only look at fails, a lenient agent goes unnoticed. Evidence: the pass sample, with tester agreement recorded.

The agent does not test its own work. If an AI agent handled the customer, a different agent, certified separately, or a person tests the file. Evidence: the tested file's handler and the tester's identity, shown side by side.

Results are filed unchanged. The agent's proposal and the tester's decision are both kept. Evidence: the append-only record.

What should you measure?

• Tester agreement with proposed passes and fails, tracked by question. A question with low agreement is usually badly written.
• "Cannot tell" rate, by channel. It measures record-keeping as much as outcomes.
• Fails by theme, product and channel, over consecutive periods.
• Time from file selection to signed-off finding.
• Remediation actions raised and closed from findings.

Where does it usually go wrong?

The script tests the process. "Was the key facts document sent?" is a process question. "Did the customer understand the fee before paying it?" is an outcome question. Rewrite the script before blaming the agent.

Silence is read as a pass. No complaint does not mean a good outcome. Make "no evidence either way" a distinct answer.

The sample is too clean. Random samples miss small, high-harm segments. Add targeted samples, and treat comparisons between groups with care; outcome testing is not A/B testing, and differences between segments need a person to interpret.

Findings stall. A theme is found but nobody owns the fix. Route each signed-off theme as a remediation action with an owner and a date.

Agent-handled cases are excluded. These need testing most. A customer told a claim outcome by an agent, for example, should be tested against what they were told; see whether an AI agent can tell a customer the outcome of their claim.

Where Pryme fits

Attest certifies the testing agent itself on files your testers have already scored, so you know how often it agrees with them before it reads a live sample. Its scope stops at proposing results; the tester's decision is a separate, named act, and both are filed.