Not one score, but a floor for each gate, set before testing starts. Severe failures, such as disclosing another customer's data or acting beyond permitted depth, should have zero tolerance. Other gates should have floors at least as demanding as the standard your own staff meet, and higher where the agent will act without a person checking.
Why not one overall pass mark?
Because an average lets strength in one area hide weakness in another. An agent that grounds every answer perfectly and misses a fifth of vulnerable customers can still post a high overall score. The failure that matters disappears into the average. The argument is made fully in an average score hides the failure that matters.
Separate floors force each gate to be cleared on its own. They also make the result easier to read: a reviewer can see exactly which gate failed and on which cases.
How should each floor be set?
Start from three questions for each gate:
- What does a failure cost? A slightly awkward phrase costs little. A wrong refund or a disclosure costs a great deal.
- Will a person catch it? At draft depth, a person reviews before anything happens, so some failures are caught. At execute depth, nobody does.
- What do staff achieve today? An agent should not be held to a standard your own people do not meet, but nor should it be allowed to fall below it.
| Gate | Typical approach to the floor |
|---|---|
| Adversarial | Zero tolerance for severe failures; giving way once on a serious bound fails the gate |
| Depth | Zero tolerance; acting beyond permitted depth fails the gate |
| Refuses | Very few failures allowed; none on high-risk refusals such as payment changes |
| Escalates | Very few failures allowed; none on vulnerability triggers |
| Grounded | High floor; any unsupported claim about money or rights counts heavily |
| Scope | High floor, with some tolerance for harmless drift in tone |
| Stable | Answers to the same question on repeated runs must agree on nearly every case |
Floors are best stated as counts, such as "no more than two failures in two hundred escalation cases, and none involving vulnerability", rather than abstract percentages. Counts are easier to reason about and harder to round up. A structured way to record them is the pass floor setting template for certification gates, and the method is in how to set the pass floor for each certification gate.
How should depth change the floors?
The higher the depth, the less room for error. A capability certified to draft replies a person sends can have slightly lower floors on grounding and scope, because a person reads every draft. The same capability certified to send replies on its own needs higher floors, because there is no second pair of eyes.
Adversarial and depth stay at zero tolerance at every level.
When should floors be set?
Before the first run, and recorded with the date. Floors set after seeing results tend to land just below the agent's score. That defeats the purpose and is easy for an auditor to spot.
Floors can be revised later, for example when you learn more about real failure costs. Revisions should be recorded, justified and reviewed by someone independent of the team that wants the agent to pass.
An illustrative example
Take a building society certifying an agent to answer savings account questions at depth 3. The process owner sets floors before testing: zero adversarial failures; zero depth failures; no more than one escalation failure in a hundred and fifty cases and none involving bereavement or hardship; no more than three grounding failures in three hundred, none involving interest rates or withdrawal terms.
The agent passes every gate but grounding, where it has two failures, both quoting an old bonus rate. The count is within the floor, but both involve interest rates, which the floor excludes. The gate fails. The source is fixed and the run repeated.
How is the score itself produced?
Answer by answer, gate by gate, against a written rubric. The method is in how to score an AI agent's answers. Do not confuse certification scores with customer measures such as a customer satisfaction score. A satisfied customer may have been told something wrong, and a correct answer may disappoint.
Where Pryme fits
Attest scores seven gates, each against its own floor, and each must clear on its own. The customer sets floors for its own operation within the platform's minimums, and the result names the gate that failed. The reasoning behind the approach is in what it means to certify an agent.
Questions people ask
Should floors be the same for every agent?
The approach should be consistent, but floors should reflect each capability's risk and depth. A refund capability needs tighter floors than a balance enquiry.
What if no agent can reach the floor?
Then the capability is not ready for that depth. Keep it lower, narrow its scope, or improve its sources. Lowering the floor to fit is the wrong answer.
Should the agent need to beat staff performance?
At high depth, usually yes, because nobody checks its work. At draft depth, matching staff is a reasonable bar.