Gather real cases from your own operation and replay the agent against them. Score each answer on separate gates, such as grounding, refusal and escalation, each against its own floor. Have someone independent review the results. Issue a certificate naming the agent, the depth allowed and the lapse date, then enforce that depth when the agent acts.
What are the steps?
- Name the role. Not "the claims agent" but "acknowledge new motor claims and request missing documents". A role is one job the agent does, in one process.
- Decide the depth you are asking for. Read, draft for a person to send, execute routine work within bounds, or move money and take hard-to-undo actions. The evidence needed rises with depth.
- Assemble the case set. Past cases from your operation with known outcomes. Include routine cases, hard cases, rare cases and cases where the right answer was to refuse or escalate.
- Set a floor for each gate. Grounded, refusal, depth, escalation, scope, adversarial and stable. Set floors before you see results, so they are not adjusted to fit.
- Run the agent, more than once. Repeated runs on the same cases show whether its behaviour is stable.
- Score every answer against what a competent person did and against the rules that applied.
- Review every failure. Each one either reveals a fix or shows the agent should stay at lower depth.
- Get independent review. A named person who did not build the agent accepts or rejects the results.
- Issue the certificate with its conditions and its lapse date.
- Enforce and monitor. The certificate sets what the agent may do at the moment of action. Live behaviour is watched for drift, and the certificate is renewed on fresh evidence.
What does each gate test?
| Gate | The question it asks | Typical failing case |
|---|---|---|
| Grounded | Does every claim trace to a source? | Agent states a fee that is not in the tariff |
| Refusal | Does it decline what is out of bounds? | Agent agrees to waive a charge it has no authority over |
| Depth | Does it stay within its permitted level? | Agent sends a message it was only allowed to draft |
| Escalation | Does it hand hard cases to a person? | Agent carries on after a customer mentions hardship |
| Scope | Does it stay within its role? | A complaints agent offers investment views |
| Adversarial | Does it resist attempts to manipulate it? | Agent discloses another customer's details |
| Stable | Is behaviour stable across runs? | Same case, different answers on different runs |
Each gate must clear on its own. A high average with one failed gate is a fail.
Why your own cases and not a benchmark?
Benchmarks show how a model or agent does on someone else's problems. Certification has to show how it does on yours, with your sources, your policies and your customers. The point is argued in benchmark scores do not certify an agent for your operation. It is also why a provider's own testing, however thorough, is not a certificate for you; see whether a vendor can certify its own AI agent.
If you have no history for a process, build realistic cases with your subject experts and start low. How to certify an AI agent without real data covers the method.
An illustrative example
A building society wants its agent to answer members' questions about their mortgage redemption figures at depth 2, drafting replies a person sends.
The team names the role, pulls a few hundred past redemption queries, and adds cases the team remembers as tricky: early repayment charges in the last month of a fixed rate, joint mortgages where only one borrower is asking, and members in arrears. Floors are set for each gate. Over three runs the agent grounds well and stays in scope, but it drafts full redemption figures for the joint mortgage cases without checking both borrowers' consent. That fails the escalation gate. The brief is changed so joint cases are escalated, the full set is rerun, and the agent passes. The risk manager accepts the certificate for depth 2, lapsing in 90 days.
What stays with a person?
Choosing the role and depth, approving the case set, setting the floors, accepting or rejecting the results, and deciding conditions. Narrow certificates are easier to defend; how to certify an AI agent with a single capability shows how to scope them.
What happens after certification?
The certificate is live evidence, not a trophy. A scheduled run that clears no depth drops the agent to read-only the same day. A change to the brief, sources, tools or model means a rerun; how to re-certify an AI agent after its brief changes covers the most common case. And the certificate lapses on its date unless renewed. The thinking behind all of this is in what it means to certify an agent.
Where Pryme fits
Attest runs this process: it replays the agent against the customer's own cases, scores the seven gates separately against their floors, and issues a certificate that lapses after 90 days. The depth it supports is then enforced at the moment of action, alongside the customer's own limits.
Questions people ask
How long does certification take?
Mostly as long as it takes to assemble a good case set and review the failures. Running the agent is the quick part.
Can you certify all of an agent's capabilities at once?
Yes. The agent is certified as a whole, to one depth: the highest its evidence proved. A run that falls short still certifies the highest depth it cleared.
Who should set the pass floors?
The process owner with risk input, before the results are seen. Floors set afterwards tend to fit the results.