How many test cases do you need to certify an AI agent?

There is no single number. You need enough cases to cover every case type for every capability, including the rare ones that matter most, with more where a failure would be costly. A read-only capability needs far fewer than one that moves money. Plan coverage per case type, then check the total is large enough to reveal rare failures.

What drives the number?

Five things, roughly in order of importance.

A capability that drafts replies for a person to check, with three case types, may be well covered by a modest set. A capability that issues refunds on its own across ten case types needs a much larger one.

Why does coverage matter more than the total?

Because a large set made mostly of routine cases tells you little about hard ones. If nine in ten of your past cases are simple, a random sample of a thousand contains only a handful of the cases where agents fail. The agent can score well overall and never meet the case that would have failed it.

So build the set by case type. List the types, including refusals and escalations, and give each enough cases to judge. How to pick them is covered in how to choose the cases for an AI agent's certification set.

How many cases does a rare failure need?

More than intuition suggests. If an agent passes every case in a set, that tells you its failure rate is probably low, but only down to a limit set by the size of the set. As a rough working rule, seeing no failures in a few hundred cases supports confidence that the failure rate is below about one in a hundred, not one in a thousand.

For most failures that is enough. For failures that cause real harm, such as disclosing another customer's data or moving money wrongly, it is not. Those need either many more targeted cases or, more practically, a control outside the agent that stops the harm regardless. That is why bounds enforced at the moment of action matter: they cover what testing cannot rule out.

How should refusal and escalation cases be counted?

Separately, and generously. These are the gates where agents most often fail under pressure, and each refusal reason needs several phrasings and several pressure tactics. A structure for these is in the refusal test cases template, and the method is in how to test an AI agent's refusals.

Count near misses too: legitimate cases that resemble a refusal or escalation trigger, to catch over-caution.

An illustrative plan

Take a lender certifying an agent for two capabilities: answering questions about loan balances at depth 3, and drafting payment plan proposals at depth 2.

Each case is run three times. The total is several hundred cases and well over a thousand runs. The numbers are illustrative; what matters is that no case type is left thin.

Where do the cases come from?

Your own closed cases, rebuilt as they stood at the time, are the best source; see how to test an AI agent against real cases. Hard and adversarial cases are often too rare in history, so subject experts add them. The full certification process is in how to certify an AI agent.

When should you add more?

• When a capability's depth is raised.
• When monitoring finds a live failure; it becomes a new case.
• When sources, the brief or the model change in a way that affects a case type.
• When a new case type appears in operation.

A certification set that grows over time is healthy. The reasoning behind certifying this way is in what it means to certify an agent.

Where Pryme fits

Attest scores each of seven gates separately on the customer's own cases, so thin coverage on one gate shows up rather than being hidden in an average. Certificates lapse after 90 days, and each renewal is a chance to add the cases operation has surfaced.