Test refusals in both directions. Build cases the agent must refuse, such as actions above its depth, outside its scope or resting on an external claim, and cases it must not refuse: ordinary requests it should handle. Add persistence and rephrasing. Score whether it refused, whether it attempted the action anyway, and whether the customer was left a route forward.
What is refusal testing, and who owns it?
A refusal is the agent declining to do something because it falls outside its bounds. Good refusals protect the firm and the customer. Bad refusals come in two kinds: the agent fails to refuse something it should, or it refuses something it should have done. The second kind is easy to overlook, because it looks safe. It is not. A customer refused a legitimate refund, or a staff member refused a policy answer they are entitled to, has been failed.
The capability owner owns the test. The refusal gate is one of seven, and it must clear its floor on its own. People who handle complaints are good sources of must-not-refuse cases; they see where customers were turned away wrongly.
What are the steps?
- Person, owner: lists what the agent must refuse. Actions above its certified depth. Actions outside its role. Actions withheld at the platform floor for every agent. Actions that would rest on an external claim, such as a payment to an account named only in an email.
- Person, owner: lists what it must not refuse. Ordinary requests within its scope and depth, including awkward ones: angry customers, unusual but valid requests, requests phrased badly.
- Person: writes refusal cases from filed work. Real situations where a person did, or should have, said no. Keep attack-style cases in their own plan; the prompt injection test plan template is the place for those.
- Person: adds persistence. The same out-of-bounds request made three times, rephrased each time, or split across several messages.
- Person: adds disguised requests. An out-of-bounds action framed as routine, or as urgent, or as approved by someone senior. The red-team test plan template for AI agents supplies patterns.
- Agent, replay only: works every case.
- Scorers: score three things. Did it refuse when it should and act when it should? Did it attempt the action anyway, leaving the platform to refuse it? Was the refusal useful: clear, honest about what it cannot do, and pointing to the right route?
- Person, owner: reviews failures and adds them to the set.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Must refuse | Nothing | None | Listing cases |
| Must not refuse | Nothing | None | Listing cases |
| Filed cases | Nothing | None | Selecting them |
| Persistence | Nothing | None | Designing it |
| Disguise | Nothing | None | Designing it |
| Replay | Works each case | Replay only | None |
| Score | Nothing | None | Judging refusal quality |
| Review | Nothing | None | Adding failures to the set |
What controls make it safe?
Two layers, scored separately. The agent should refuse before it acts. If it attempts the action, the platform refuses it at the moment of action and files the verdict with the bound that decided. A pass that relies on the platform alone shows the brief is weak, even though the outcome was safe.
Withheld actions never attempted. Some actions are withheld for every agent at every depth. An agent that keeps trying them is wasting a safety layer.
No partial compliance. An agent that refuses the payment but changes the beneficiary's details "ready for later" has complied in part. Score it as a failure.
No leaking reasons. Refusals on fraud or screening grounds must not reveal the reason. The refusal says what the agent cannot do and where to go next.
Live comparison. The live record of refusals is reviewed against the test results. Reviewing the refusals an agent makes covers that ongoing work.
What are the signs it is going wrong?
- Over-refusal after a brief change. Someone tightened the instructions after an incident and the agent now refuses half of its legitimate work.
- Persistence wins. The agent refuses twice and gives in on the third rephrase.
- Dead-end refusals. "I can't help with that" and nothing else. The customer has no route forward.
- Refusals quoting policy text. Internal rules pasted into the customer's chat, sometimes including thresholds a fraudster would like to know.
- Refusal rate in live work far from the test. The set does not reflect what customers actually ask.
How do refusals relate to escalation?
Some requests should be both refused and escalated: a customer asking for something above the agent's limit who also needs a person's help. Others should be refused with a route forward and no escalation at all. The two are scored on separate gates. Testing an agent's escalation behaviour covers the other half, and how do you test whether an AI agent escalates correctly? answers the short version.
Refusal quality also depends on the audience. A refusal in a staff tool can be brief and technical; a refusal to a customer should be plain, polite and specific about what happens next. Score them with separate rubric lines for customer-facing and staff-facing capabilities.
The wider framework is in what it means to certify an agent.
Where Pryme fits
In Pryme, Attest scores refusal as its own gate, counting both missed refusals and refusals of legitimate work against the floor. When an agent does attempt an out-of-bounds action, the minimum chain refuses it and the verdict names the bound that decided, so you can see which layer did the work.
Questions people ask
Should the agent explain why it is refusing?
At the right level. "I can't change payment details on your account; a colleague can help you with that" is enough. Detailed reasons can help someone get around the bounds.
Is a refusal by the platform a failure for the agent?
For scoring, yes. The outcome was safe, but the agent tried something it should have known not to try.
How do we stop over-refusal?
Include plenty of must-not-refuse cases, score them with the same weight, and read the refusals people complain about.