A certification set selection checklist makes sure the cases used to certify an AI agent reflect the work it will do: drawn from your own filed cases, covering every capability, including edge cases, refusals and escalation triggers, labelled by qualified people, kept separate from anything used to build the agent, and sized to the cost of being wrong.
What does a good certification set look like?
A certification is only as good as the cases behind it. An agent tested on a hundred tidy, typical cases will pass and then fail on the first awkward one in production. A good set looks like a sample of your real work, weighted toward the cases where mistakes hurt most.
Three properties matter. It is yours: cases from your own operation, with your products, customers and policies. It is hard in the right places: edge cases are in the set, not in a footnote, for the reasons argued in edge cases belong in the certification set, not in a footnote. And it is clean: no case was used to write the brief or tune the agent, or the result flatters it.
When should you use the checklist?
Use it for every first certification and every renewal. At renewal, the checklist also asks what has changed: new case types, recent incidents, held items people rejected. It pairs with the re-certification trigger checklist, which decides when a new set is needed outside the normal cycle.
How do you complete each field?
Work capability by capability. For each, the checklist asks for counts and a short note. Where a count is zero, say why.
- Routine cases come from a random sample of filed cases over a period long enough to include seasonal patterns.
- Edge cases come from people who do the work: ask them for the cases that took longest or caused arguments.
- Refusal cases are requests the agent must decline. If your filed cases have few, write some, and mark them as written.
- Escalation cases contain a trigger the agent must catch: distress, vulnerability, complaint, fraud.
- Size follows the cost of error. A read-only capability needs fewer cases than one that moves money; the reasoning is in certification set size should follow the cost of being wrong.
For the detailed method, see how to choose the cases for an AI agent's certification set.
The certification set selection checklist
| No. | Item | Entry |
|---|---|---|
| 1 | Capability and depth being certified | |
| 2 | Source of cases: system and period covered | |
| 3 | Routine cases: count and sampling method | |
| 4 | Edge cases: count and who nominated them | |
| 5 | Refusal cases: count; how many written rather than filed | |
| 6 | Escalation cases: count by trigger type | |
| 7 | Vulnerable or financially stretched customers: count | |
| 8 | Incident and complaint cases since the last run | |
| 9 | Held items people rejected or heavily edited | |
| 10 | Cases excluded, and why | |
| 11 | Confirmation no case was used to build or tune the agent | |
| 12 | Personal data: minimised, lawful basis recorded | |
| 13 | Labellers: names, roles, disagreement method | |
| 14 | Total size, and why it suits the cost of error | |
| 15 | Approved by |
A filled example: home insurance claims intake
A hypothetical home insurer certifies an agent to record first notice of loss at depth 3.
| No. | Entry |
|---|---|
| 2 | Claims system, 12 months of filed claims |
| 3 | 300, random sample stratified by peril: escape of water, storm, theft, accidental damage |
| 4 | 60, nominated by senior handlers: multiple perils in one claim, policy changed mid-term, claims during a weather event |
| 5 | 40; 25 written, including requests to backdate a claim |
| 6 | 50: injury mentioned, uninhabitable property, possible fraud indicators, complaint |
| 7 | 30: elderly policyholders living alone, customers in temporary accommodation |
| 8 | 12 |
| 9 | Not applicable: first certification |
| 10 | Commercial policies excluded; agent handles personal lines only |
| 11 | Confirmed: brief written from policy wording and process maps, not from these cases |
| 13 | Four senior handlers; disagreements to the claims manager |
| 14 | 492, larger than a read-only set because the agent writes to the claims record |
Row 4 found the hardest cases. The weather event cases mattered most, because they arrive in a surge and are exactly when the agent will be busiest.
What should never go into the set?
Cases used to write the brief or test the agent during development, duplicates, and cases whose correct answer nobody can agree on. Disputed cases are worth discussing, but in the set they make scores noisy. Note them and resolve the policy question first. Floors are set after the set is fixed, using the method in how to set the pass floor for each certification gate. The wider approach is in the guide on what it means to certify an agent.
Questions people ask
Can synthetic cases be used?
For refusals and rare triggers, written cases are useful and often necessary. Mark them, keep them a minority, and never let them replace filed cases for routine work.
How many cases do we need?
Enough that each gate's floor is meaningful for each capability. Small capabilities with low harm can use smaller sets; anything that moves money or writes to records needs more.
Who should pick edge cases?
The people who do the work today. They know which cases are hard, which the agent team rarely does.