Sample them every week and grade each one. A refusal review takes a set of the agent's refusals, both requests it declined and actions its bounds stopped, and asks three things: was refusing right, was the reason right, and did the person refused know what to do next. A named reviewer grades each, and the process owner acts on patterns.
What is a refusal review, and who owns it?
Agents refuse in two ways. They decline requests from people ("I can't change your address over chat"), and their own proposed actions are refused by a bound (a refund above the customer's limit). Both matter. A wrong refusal can harm a customer as surely as a wrong action: a hardship request turned away, a complaint not recorded, a legitimate payment left stuck.
The process owner owns the review and its decisions. A reviewer from the team that does the work, someone who knows what the right answer looks like, grades the sample. Compliance joins for customer-facing agents. Testing refusals before go-live is covered in how to test an AI agent's refusals; this playbook is about the ones that happen in production.
What are the steps?
- Agent, depth 1: draws the sample. A random selection of refusals from the week, plus every refusal on a capability flagged as sensitive, plus refusals the customer then complained about.
- Agent, depth 1: adds context to each. The request, the sources, the bound or rule that caused the refusal, the message the person received and what happened afterwards.
- Agent, depth 1: samples allowed actions that sit close to the edge of a bound, to look for refusals that should have happened and did not.
- Person, reviewer: grades each refusal: correct; correct but poorly explained; over-cautious; wrong reason; or should have been an escalation.
- Person, reviewer: grades the near-misses: should any allowed action have been refused?
- Agent, depth 2: drafts the pattern summary, grouping grades by capability, bound and channel.
- Person, process owner: decides actions: change the brief, change wording, change a limit, request access, or re-test.
- Person, process owner: decides customer remediation where a wrong refusal caused harm, such as contacting a customer whose request was wrongly turned away.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Sample | Selects refusals | 1 | Setting the sampling rules |
| Context | Assembles request, sources, bound | 1 | Nothing |
| Near-misses | Selects edge-of-bound allowed actions | 1 | Nothing |
| Grade refusals | Nothing | none | Judging each one |
| Grade near-misses | Nothing | none | Judging each one |
| Patterns | Drafts summary by capability | 2 | Nothing |
| Actions | Nothing | none | Changes to brief, limits, access |
| Remediation | Nothing | none | Putting customers right |
Which decisions does a named person keep?
Whether the refusal was right. The agent cannot review its own refusals. It can assemble them; a person who knows the work grades them.
Whether to widen anything. Over-cautious refusals tempt quick widening. Any change to scope, depth or limits goes through change control.
Whether a customer is owed anything. Wrong refusals can mean a firm failed a customer. The owner decides remediation, and legal or compliance advise on obligations. Who carries responsibility for an agent's error is set out in who is liable when an AI agent makes a mistake?
Whether a refusal pattern is an access problem. Many refusals at the sources bound mean the agent lacks data it needs, or is being asked for data it should never see. That decision belongs in the access review for AI agents, not in a quick fix.
What record does the review produce?
Each review leaves the sample with its selection rules, a grade and note per item, the pattern summary, the owner's decisions with reasons, and any remediation cases opened. Over time these records show whether the agent is getting better at refusing the right things. Under the EU AI Act, deployers are expected to monitor how high-risk systems operate; using an agent makes you a deployer, and a regular refusal review is part of showing you do.
What makes a good refusal message?
A refusal to a person should say plainly that the agent cannot do this, why in general terms, and what the person can do next: who to contact, which form to use, or that a colleague will follow up. It should not reveal internal limits in detail or sound like a final decision when it is not. Grading the message separately from the decision catches agents that refuse correctly but leave people stuck.
What should you measure?
- Over-cautious refusals per capability, which show where the agent is blocking good work.
- Wrong-reason refusals, where the outcome was right by luck.
- Near-misses found, which are more serious than over-caution.
- Refusals followed by a complaint.
- Time from pattern found to change made.
Where does it usually go wrong?
Only bound refusals are reviewed. Declined customer requests are left out, and the harm they cause goes unseen.
Refusals are treated as safe by default. A refusal feels cautious, so nobody checks it. For a customer with an urgent need, it can be the worst outcome.
Missed refusals are never looked for. Reviewing only what was refused tells you nothing about what should have been. Sample near the edge.
Fixes are wording only. Every problem is solved by rewriting the message, while the real cause, a misdrawn role or missing source, stays.
Where Pryme fits
In Pryme, every refusal is filed with the bound that decided, so the sample can be grouped by bound without guesswork. Attest's refusal gate tests on your own cases that the agent refuses what it should and only that, before depth is granted. See agent governance is a bound, not a model.
Questions people ask
How many refusals should we review each week?
Enough to see patterns per capability, which for most agents means dozens rather than hundreds, plus every refusal on sensitive capabilities. Increase the sample after any change to the agent.
Should customers be told a refusal came from an AI system?
They should know they are dealing with an AI system throughout. The refusal should make clear how to reach a person, so being refused by an agent never ends the customer's route to help.
Can the review be automated?
The sampling, context and summary can. The grading cannot, because it requires judgement about what the right answer was, and that judgement is what the review exists to supply.