How to run quality assurance on support conversations with an AI agent

Let the agent score every conversation, not a small sample, against a scorecard your QA lead owns. It reads, scores each criterion with the passage that justifies the score, and flags conversations for review. A person calibrates its scoring every week and reviews what it flags. Feedback to staff and any judgement about their performance stay with team leaders.

What is support QA, and who owns it?

Quality assurance checks whether support conversations met your standard: accurate answers, correct identity checks, complaints recognised, vulnerable customers handled with care, promises kept. Traditional QA reviews a handful of conversations per person per month. The agent changes the coverage, not the ownership.

The QA lead owns the scorecard, the calibration and the review queue. Team leaders own feedback and coaching. If your support uses AI agents as well as staff, their conversations are scored on the same scorecard, but a person reviews those results, because an agent marking its own kind of work is not independent. The wider support position is in the AI support agent that knows when to say no.

How does the QA run work, step by step?

1. Person, QA lead: owns and versions the scorecard. Each criterion has a definition, examples of pass and fail, and a weight. A ready layout is in the support QA scorecard template.
2. Agent, depth 1: reads each closed conversation with its context: customer, account facts the staff member could see, policy in force at the time.
3. Agent, depth 2: scores each criterion and quotes the passage that justifies the score. A criterion it cannot judge from the conversation is marked "cannot assess", not guessed.
4. Agent, depth 2: flags conversations for review. Any fail on a critical criterion, such as a missed identity check, a missed complaint, or a missed request for a person. Also a random sample of passes, so reviewers see the good as well as the bad.
5. Person, QA reviewer: reviews flagged conversations. Confirms or overturns each score, with a reason.
6. Person, QA lead: calibrates weekly. Scores a fixed set of conversations independently and compares with the agent. Criteria where agreement drops are paused until the definition is fixed.
7. Person, team leader: gives feedback. Using confirmed scores only. The agent never contacts staff about their scores.
8. Agent, depth 2: drafts the period summary. Trends by criterion and team, for the QA lead to approve.

What stays with a person, and why?

Every consequence for a member of staff. QA scores feed coaching, performance reviews and sometimes pay. A score that affects someone's job should be confirmed by a person who can be asked to explain it. In some markets, monitoring staff and using automated scoring to assess them also raises data protection and employment questions; involve your data protection owner and, where relevant, staff representatives before you start.

Critical criteria also stay with people. If the agent says a staff member failed to escalate a vulnerable customer, a reviewer reads that conversation before anything else happens. The criterion about requests for a person deserves particular care, because it is the one customers remember; what good looks like is set out in what a support agent should do when a customer asks for a human.

Which controls matter, and what evidence do they leave?

Evidence quoted with every score. No score without the passage that supports it. Evidence: the quote stored with each criterion.

Calibration with a pause rule. Evidence: weekly agreement by criterion, and the criteria paused.

Context fixed to the moment. The conversation is judged against the policy and account facts as they were, not as they are now. Evidence: the policy version and timestamp used.

Overturns recorded. Every overturned score is kept with the reviewer's reason. Evidence: the overturn log, used to improve definitions.

Independent review of agent conversations. Evidence: the reviewer's name on each agent-conversation review.

What should you measure?

• Calibration agreement by criterion. The number that tells you whether to trust the rest.
• Overturn rate on flagged conversations.
• Critical fails per period, by criterion and team.
• Coverage: conversations scored as a share of conversations closed.
• "Cannot assess" rate. A high one means a criterion depends on information the conversation does not contain.

Where does it usually go wrong?

The scorecard rewards politeness over accuracy. Tone is easy to score, accuracy is hard. Weight the criteria that protect customers above the ones that are easy to measure.

Context is missing. The agent marks an answer wrong because it reads today's policy, not the one in force that day. Fix the version, not the score.

Bad source data skews scores. Missing account facts or broken transcripts make good conversations look bad. Handle those as a data issue, as described in how to handle data quality problems for AI agents, and exclude the affected conversations.

Full coverage becomes surveillance. Staff feel watched on every word. Use the coverage to find patterns and serious misses, and keep individual feedback to confirmed findings.

Where Pryme fits

In Pryme, the QA agent reads conversations at depth 1 and writes scores at depth 2, so it can never message staff or change records. Each score is filed with its quoted evidence and the scorecard version, and every overturn by a reviewer is kept alongside it.