Edge cases belong in the certification set, not in a footnote

Rare, awkward cases are where an agent's judgement is tested, so they belong inside the certification set and count towards the result. A set built only from tidy, common cases proves the agent can do the easy work. Listing the hard ones in a footnote as "known limitations" leaves them untested while the agent stays authorised to act on them.

What counts as an edge case in an agent's work?

Edge cases are not exotic. In most operations they are the cases that arrive every week but rarely on the same day: a joint account where only one holder is writing in, a customer who has died and a relative is asking, a refund request that is partly inside policy and partly outside it, two documents that disagree about the same fact, a message that contains two requests with different answers.

What they share is that the obvious response is wrong. The agent has to notice that something is unusual, apply a narrower rule or a different one, and often stop and hand over. That is exactly the judgement certification is supposed to measure. When you certify an agent, you are not asking whether it can answer the median question. You are asking whether it can be trusted with the work it will actually receive, including the parts that are hard.

Why do edge cases get left out?

There are understandable reasons, and they are worth naming because they recur:

• They are hard to find. Common cases fill any sample; rare ones have to be searched for in complaints, exception logs and the memories of experienced staff.
• They are expensive to label. Deciding the correct outcome for an awkward case often needs a senior person and sometimes a debate.
• They drag the score down. A team under pressure to launch notices that the agent passes comfortably without them.
• They feel like a later problem. The plan is to "handle them in phase two", and phase two never quite arrives.

Each reason is real. None of them changes what the agent will do on its first day live when one of those cases arrives.

What goes wrong when edge cases sit in a footnote?

A footnote is a statement of intent. It is not enforced at the moment of action. If the certification report says "not tested on bereavement cases" but the agent's bounds still let it reply to any account-holder query, the agent will reply to the relative of a deceased customer exactly as it replies to everyone else.

Take an insurer's claims support agent certified on a few hundred routine status questions. The report notes that claims involving a third party's solicitor were excluded. Live, a solicitor writes in on behalf of a claimant, and the agent, seeing a polite status question, shares claim details with someone it had no basis to treat as authorised. The footnote was accurate. It just was not connected to anything that stopped the action.

How should edge cases be scored?

The correct outcome for many edge cases is not an answer. It is a refusal, a hold for a named person or an escalation. That is fine, and it should be labelled that way. An agent that recognises a case is outside its competence and hands it over has passed that case.

Two scoring rules help:

1. Score edge cases against the gate they test. A bereavement case mostly tests escalation; a conflicting-documents case mostly tests grounding. Each gate must clear on its own, so a strong showing on routine accuracy cannot hide a weak showing on escalation.
2. Do not let volume dilute them. If a set has a thousand routine cases and twenty edge cases, a simple average makes the twenty almost invisible. Report them as their own slice with their own floor.

How many to include is a question of consequence, not proportion. The argument that certification set size should follow the cost of being wrong applies with extra force here: the rarer and more damaging a case type, the more deliberately it needs representing.

Where do you find the edge cases?

Your own operation already contains them. Good sources include:

• complaints and their root causes, especially ones upheld;
• items people overrode or corrected in the held queue;
• policy clauses that begin "except where" or "unless";
• cases routed to specialist or senior teams in the past year of work;
• questions experienced staff say new joiners always get wrong.

The method for drawing a representative set is covered in how to choose the cases for an AI agent's certification set, and the certification set selection checklist includes a specific check for awkward cases. Use real cases where you can, with personal data handled properly; invented edge cases tend to be tidier than the real ones.

What happens if the agent fails the edge cases?

Then you have learned something useful before a customer paid for it. Failing an edge case does not have to block the whole capability. It can narrow it.

If the agent cannot handle bereavement correctly, the bounds can be set so that any message suggesting a deceased account holder is refused by the agent and routed to a trained team. The capability is then certified for the work it was proven on, and the excluded case type is enforced as an exclusion rather than written down as a hope. That is the real difference between a footnote and a bound: one describes a limitation, the other applies it.

Narrowing has a cost. Every excluded case type is work that still lands on people, and too many exclusions can leave an agent with little left to do. That is a reasonable outcome to reach on evidence. It is better than an agent with a wide remit and a long list of things it was never tested on.

What should you do differently?

Before the next certification run, ask the people who handle exceptions which ten case types they would least like an agent to get wrong. Put real examples of each into the set, label them with the outcome a senior person would want, and report their results separately. If the agent fails them, encode the exclusion in its bounds the same day.