An average score hides the failure that matters

A high average can sit on top of a dangerous failure. An agent that scores well on accuracy, scope and grounding can still fail to refuse what it should refuse, or give way under pressure in a handful of cases, and a single blended score will hide it. That is why each certification gate must clear on its own floor. Averages describe performance; floors decide whether the agent may act.

How does an average hide a failure?

Averages compensate. Strength in one area offsets weakness in another. That is useful when you want a general sense of quality, and dangerous when one weakness is the thing that matters most.

Take an illustrative certification run scored out of 100 on each of seven gates.

An average of about 94 looks like a strong agent. Look at the rows. The agent refuses only seven in ten of the requests it should refuse, so it goes ahead with the rest, such as fee refunds the policy excludes. And on the adversarial gate, one case in a hundred gave way under pressure where the floor allows none. On a blended score, this agent passes. Gate by gate, it fails two gates, and those are the two that matter most.

Why are the refusal and adversarial gates the ones most often hidden?

Most of what an agent does is routine, and routine cases dominate any test set. An agent that handles routine cases well will score highly on most gates almost automatically. The refusal and adversarial gates are tested by the minority of cases where the right answer is "no" or where someone pushes the agent towards a bound. Those cases are few, so they move an average very little.

That is exactly backwards from their importance. The routine cases are where errors are cheap and easy to correct. The refusal and adversarial cases are where errors land on customers, breach policy or cost money. A scoring method that lets routine competence outvote rare failure measures the wrong thing.

Do averages hide failures inside a single gate too?

Yes, and this is the subtler problem. Even a gate-by-gate score is itself an average over cases. An adversarial score of 99 out of 100 across 200 cases means two cases went wrong. If both were serious, such as giving way to a caller who pressed for details of a customer's debt, the agent should not pass, however good the other 198 were.

So floors need to be set with the failure in mind, not only the rate. For some gates, a high rate is enough: grounding at 95 may be acceptable if the errors are minor and caught downstream. For others, the floor is a count of zero serious failures. How you score individual answers so this is visible is covered in how to score an AI agent's answers, and how to choose floors in what score an AI agent should need to pass.

Where else do operations get misled by averages?

The problem is not new to agents. Operations teams have long known that average handle time can look healthy while a few very long, very difficult calls go badly. A customer effort score averaged across thousands of contacts can hide a small group of customers who found the process impossible. In each case, the average smooths away the experience of the people the process failed.

Agent certification inherits the same trap, with higher stakes, because a certified agent will act on its own. The tail of the distribution is where its worst actions live.

What should a certification report show instead?

A certification report that supports a decision should lead with gates, not averages.

1. Each gate's result against its floor, with pass or fail stated plainly.
2. The failing cases themselves, not just the count, so reviewers can see what went wrong.
3. Serious failures listed separately, whatever the gate score, with what the agent did and what it should have done.
4. The depth granted per capability, which follows from the gates, not from any blended figure.

If a single summary number is needed for a dashboard, it should be the number of gates passed, not an average of scores. "Five of seven gates cleared" is a much more honest headline than "94".

How do you check the floors are right?

Floors set on paper can be too loose or too tight. Testing them against realistic failure helps. A tabletop exercise for an AI agent failure walks through a plausible failure, such as the refusal miss above, and asks whether the certification would have caught it, who would have seen the result and what they would have done. If the answer is "the average looked fine, so nobody looked further", the scoring needs to change.

Where Pryme fits

Attest scores every answer against a floor across seven gates: grounded, refuses, depth, escalates, scope, adversarial and stable. Each gate must clear on its own; there is no blended score that can carry a failing gate. Depth is granted per capability on that evidence. The full approach is in what it means to certify an agent.