How to handle a failed certification run

Once any gate falls below its floor, the agent drops that day to the highest depth the run cleared, or to depth 1 if it cleared none. Work above that depth moves to people. A named reviewer reads every failed case and classifies the cause. The fix goes into the brief, sources or connections, never into the model's weights. Then the full set is re-run, and depth returns only on a pass.

What counts as a failed run, and who owns the response?

A certification run fails when any one of the seven gates, grounded, refusal, depth, escalation, scope, adversarial or stable, scores below its floor. Each certification gate stands alone, so a run with six strong gates and one weak one has failed. That can happen at first certification, at renewal, or in a scheduled replay of a live agent.

The agent owner owns the response. Operations owns keeping the work going while the agent is reduced. The person who reviews the failures should be someone other than the author of the agent's brief; who should review an AI agent's certification results sets out why independence matters here.

What are the steps?

1. Platform: records the failure. The run, the gates that failed, the scores, the floors and every failed case.
2. Platform: drops depth the same day. For a live agent, depth falls to the highest depth the run cleared, or to depth 1 if it cleared none. At depth 1 it can read; it cannot draft, act or move money.
3. Platform: settles work in flight. Drafts already held stay with their reviewers. Actions queued for the agent that need more than its new depth are refused and returned to the queue for people.
4. Person, operations: switches to manual. Using the contingency plan agreed before go-live.
5. Person, reviewer: reads every failed case. And classifies the cause: the brief, a source, a connection, a model change, a wrong expected answer in the set, or a floor set wrongly.
6. Person, owner: decides the fix. Change the brief, correct or replace a source, fix the connection, or correct an expected answer with a second reviewer's agreement.
7. Platform: re-runs the full set. Not just the failed cases. A fix for one gate can break another.
8. Person, owner: approves restoration. If every gate clears, depth returns to the lower of the new evidence and the role's ceiling.
9. Person: files the report. What failed, why, what changed, and the re-run result.

Why drop depth the same day?

Because a failed run is evidence that the agent may not do what its certificate said. Waiting until the cause is understood means the agent keeps acting at a depth it can no longer show it has earned. A failed run should cost depth the same day, and should be automatic, so that nobody has to decide under pressure whether a failure is "real enough".

The cost is real: work moves to people, and queues grow. That cost is what makes the contingency plan necessary before go-live, not after the first failure.

What evidence does each step leave?

• Failure record: run identifier, gate scores, floors, failed cases.
• Depth change: a filed verdict showing the agent's depth before and after, with the time.
• Work in flight: refused verdicts for queued actions, naming the certificate as the bound that decided.
• Triage: each failed case with its cause and the reviewer's name.
• Fix: the change made to brief, source or connection, and who approved it.
• Corrected answers: the original expected answer, the correction, and both reviewers.
• Re-run: a full run record, like any certification.
• Report: the summary, filed with the certificate history. The failed certification run report template gives the fields.

What should you measure?

• Time from failure to depth drop. Should be the same day, every time.
• Time from failure to restored depth, and the work handled manually meanwhile.
• Causes of failure, by type. Repeated source failures point to poor document control; repeated model failures point to provider changes.
• Expected answers corrected after failure. A high number is a warning sign.

What are the signs it is going wrong?

Fixing the test, not the agent. After a failure, the quickest route to a pass is to decide the expected answers were wrong or the floor too high. Sometimes they were. That is why corrections need a second reviewer and floor changes need risk sign-off, and why both are filed.

Re-running only the failed cases. They pass, depth is restored, and a regression elsewhere goes unseen.

No contingency. If operations cannot run the work without the agent, there will be pressure to restore depth before the fix is proven.

Treating it like an audit finding. A failed run is not the same as a nonconformity under a management system standard, and AI agent certification is not the same as ISO certification. It is a live operational event with an immediate consequence.

The framework behind all of this is in what it means to certify an agent.