The number of cases you use to certify an agent should depend on what it costs when it gets something wrong. An agent that answers branch opening hours can be certified on a modest set. One that releases payments needs many more cases, and harder ones, before it earns any depth. Using one set size for everything either wastes effort on safe capabilities or under-tests dangerous ones.
Why does set size matter at all?
A certification set is a sample. You are using a limited number of cases to infer how the agent will behave across all the cases it will meet. The larger the sample, the more confident you can be that a low error rate in testing reflects a low error rate in practice.
There is a simple rule of thumb that makes this concrete. If an agent makes no serious errors across a set of cases, the true rate of serious errors could still be as high as roughly three divided by the number of cases. Zero failures in 100 cases is consistent with up to about three failures in every 100 real ones. Zero failures in 1,000 cases narrows that to about three in every 1,000.
For a capability where an error is a mildly unhelpful answer, three in 100 may be tolerable while you gather more evidence. For a capability where an error is a payment to the wrong account, it is not.
How should cost of error set the size?
A practical approach is to tier capabilities by the cost of a wrong action and set a minimum set size and difficulty for each tier.
| Tier | Typical capabilities | Cost of a wrong action | Set size and difficulty |
|---|---|---|---|
| Low | Answering general questions, looking up public information | Inconvenience, quickly corrected | Modest set, mostly routine cases with some edge cases |
| Medium | Drafting customer letters, routine record updates, card freezes | Customer harm that can be put right | Larger set, a deliberate share of edge cases and refusals |
| High | Refunds, contact with customers in arrears, changes to payee details | Financial loss or harm that is hard to repair | Large set, heavy on edge cases, adversarial inputs and vulnerability cases |
| Highest | Releasing payments, actions that cannot be undone | Loss that cannot be recovered | Largest set, built to test every refusal and escalation path, with zero tolerance on harm |
The numbers behind each tier are a choice each firm makes, with risk and the business owner. What matters is that the choice follows the cost of error and is written down.
Is size more important than composition?
No. Composition matters at least as much. A thousand routine cases tell you less about a refund capability than two hundred cases that include excluded fees, repeat requests, vulnerable customers and attempts to manipulate the agent. The checklist in certification set selection covers what a balanced set needs, and the case for weighting towards awkward cases is made in edge cases belong in the certification set, not in a footnote.
As the cost of error rises, composition should shift. Higher tiers need a larger share of the cases that test the refusal, escalation and adversarial gates, because those are where a wrong answer is most expensive. How to pick cases for each gate is set out in how to choose the cases for an AI agent's certification set.
What does this cost?
Every case has to be labelled: someone with knowledge of the work has to say what the right answer is. That is the main expense of certification, and the reason labelling is the hidden cost of certification. Sizing sets by cost of error is partly a way to spend labelling effort where it matters.
The trade-off is direct. Large sets for every capability would be thorough and unaffordable. Small sets for every capability would be cheap and would leave high-risk capabilities under-tested. Tiering puts the labelling budget where a wrong action costs most.
Two further points keep costs in check. Cases can be reused across recertification runs, with fresh cases added to catch drift. And cases from real operations, especially ones a person corrected, are often already half-labelled by the correction itself.
How do set size and pass floors work together?
Set size and floors are linked. A strict floor on a small set is less meaningful than it looks: passing a zero-tolerance adversarial floor on a handful of cases proves little. A strict floor on a large, hard set is strong evidence. When you set the pass floor for each certification gate, consider the set size at the same time, so the floor can actually distinguish a safe agent from an unsafe one.
What should you change in your own certification?
List the capabilities your agent's certification will cover and place each in a tier by the cost of a wrong action. Set a minimum size and a target mix of edge, refusal and adversarial cases for each tier. Check that your highest-tier capabilities are tested on sets large enough that a clean result actually means something, and that your low-tier capabilities are not absorbing labelling effort they do not need. The topic guide what it means to certify an agent sets this inside the full approach.
Where Pryme fits
Attest replays the agent against real cases from your own operation and scores every answer against a floor across seven gates, each of which must clear on its own. Depth is granted per agent, and each action needs its own depth, so the cases for a high-cost capability can be larger and harder than those for low-cost ones.
Questions people ask
Is there a minimum number of cases for any capability?
There should be a floor for each tier, set by the firm. Even low-tier capabilities need enough cases to exercise their main paths and a few edge cases.
Should recertification use the same set?
Mostly, with fresh cases added each time. Reusing cases shows whether behaviour has drifted; new cases guard against the agent being tuned to a fixed set.
What if we cannot find enough high-risk cases?
Build them, from near misses, complaints, corrected actions and scenarios your experts write. Rare cases are rare in operations, which is why they have to be gathered deliberately.