Measure it on outcomes, not on how many conversations it closes. A useful scorecard asks whether answers were correct, whether refusals and handovers were right, whether the problem stayed solved, and what each resolved contact cost. Each measure has an owner and a source, and together they show whether the agent should keep, gain or lose what it may do.
Why is the obvious number misleading?
The number most teams start with is containment: the share of conversations that never reached a person. It is easy to count and it rewards the wrong thing. An agent can contain a conversation by giving a wrong answer confidently, by making it hard to ask for a person, or by ending the chat before the customer gives up. All three look like success.
The questions that matter are slower to answer. Was the customer's problem actually solved? Was the agent right to answer, or should it have handed over? Did a person have to clean up afterwards? Those need sampling and joining data across systems, which is why they are skipped.
What are the steps, and who does each?
- Support lead: define outcomes per contact reason. For "where's my order", the outcome is an accurate status and no recontact within a few days. For a complaint, it is a correct handover with a usable note. Evidence: a written outcome definition per reason.
- Quality team: sample and score. Each week, a person reads a sample of conversations across reasons and scores them against a fixed scorecard: correct, grounded, right verdict, right tone. Evidence: the scored sample, kept. A support agent promise checklist belongs in that scorecard.
- Analyst: join recontact data. Match each agent conversation to any later contact on the same topic, in any channel. Evidence: a recontact rate per contact reason, agent versus people.
- Quality team: audit refusals and handovers. Read a sample of refused and escalated conversations. Were they right? Were any refused that should have been answered? Evidence: a verdict accuracy score, split by verdict type, checked against your escalation matrix.
- Team leads: rate handovers. The person receiving each handover marks whether the note was enough to act on without asking the customer again. Evidence: handover quality per week.
- Finance or operations: cost per resolved contact. Total running cost divided by contacts resolved without recontact. Evidence: monthly figure with its inputs.
- Agent owner: decide. Once a month, review the scorecard and decide per capability: keep, expand or reduce. Evidence: a dated decision with its reasons.
What goes on the scorecard?
| Measure | What it tells you | Source |
|---|---|---|
| Answer correctness | Whether the agent says true things | Weekly scored sample |
| Verdict accuracy | Whether it answered, refused and escalated at the right times | Audit of each verdict type |
| Recontact within the window | Whether problems stayed solved | Joined contact data |
| Handover quality | Whether people could act on the note | Ratings by receiving staff |
| Customer effort | Whether customers had to repeat themselves | Sample and survey |
| Cost per resolved contact | What a real resolution costs | Finance and usage data |
Measuring accuracy on its own has its own method, set out in how to measure an AI agent's accuracy.
An illustrative example
Take a travel company whose support agent showed high containment on booking changes. The scorecard told a different story. Recontact on booking changes was higher for the agent than for people. The quality sample found why: the agent confirmed date changes before the booking system had accepted them, and some failed silently.
The agent owner reduced the booking-change capability to draft depth the same day, so a person confirmed each change. The integration was fixed to wait for the system's confirmation. Two weeks of clean recontact data later, the capability went back to acting on its own. Containment barely moved through all of this. Recontact did.
How does this fit wider governance?
Support measurement is one instance of a general discipline. The NIST AI Risk Management Framework names Measure as one of its four functions, alongside Govern, Map and Manage, and the way it applies to agents is set out in the NIST AI RMF Measure function for AI agents. For how support measures roll up into a governance view across all your agents, see how to measure AI agent governance.
Where Pryme fits
In Pryme, every verdict is filed with the bound that decided it, so verdict accuracy can be audited from the record rather than reconstructed. Attest replays your cases on a schedule, and a run that clears no depth drops the agent to depth 1 the same day. More is in the AI support agent that knows when to say no.
Questions people ask
How large should the weekly sample be?
Large enough to include every contact reason and every verdict type several times. Weight it towards higher-risk reasons rather than spreading it evenly.
Should the agent be measured against human staff?
Yes, on the same measures and contact reasons. It tells you where the agent is ready and where it is not.
Who should own the scorecard?
The support lead, with the quality team producing it. The agent owner uses it to decide what the agent may do.