A failed certification run report records what happens when an AI agent misses the floor on one or more certification gates: which gates failed and by how much, the failing cases, the drop in depth the same day, whether the same behaviour reached production, the cause, the fix and the re-run plan. Complete it within a day.
What is the report for?
A failed run is not an incident in itself. Nothing may have gone wrong for any customer. But it is evidence that the agent cannot be trusted at its current depth, and the response has to be quick and recorded. A failed run keeps only the highest depth it cleared, and one that clears no depth drops the agent to depth 1 the same day; the argument for that rule is in a failed run should cost depth the same day.
The report explains that drop and plans the way back. It is read by the agent owner, risk, the operations team now handling more work by hand, and later by whoever reviews the next run.
When should you use it rather than an incident report?
Use this report when the failure was found in certification. If the look-back shows the same behaviour caused harm in production, open an incident as well, using the AI agent incident report template, and link the two. If the failure was found during a shadow run before go-live, the shadow run report template is the better fit, because no live depth is affected.
How do you complete each section?
- Run and agent. Run reference, agent, brief version, capabilities tested.
- Gates that failed. For each: floor, score, number of failing cases. List the gates that cleared too.
- Failing cases. Each one with a sentence on what the agent did and what it should have done.
- Depth change. The agent's depth before and after, and the time the change took effect.
- Operational effect. Volume of work now held for people, and who is handling it.
- Production look-back. Did the agent do the same thing in live work? How far back you checked, and what you found.
- Cause. Brief, source, connection, limit, model change, or case set problem. Say which, and the evidence.
- Fix. What changes, and who approves it.
- Re-run plan. Date, any cases added, who reviews.
- Approvals.
Section 7 should resist the easy answer. "The model got it wrong" is rarely the cause you can act on. Look first at what changed: a source update, a new case type, a limit edit. The full approach is in how to handle a failed certification run.
The failed certification run report template
| Section | Entry |
|---|---|
| Run reference, date, agent, brief version | |
| Capabilities tested | |
| Gates failed: floor, score, failing cases | |
| Gates cleared | |
| Failing cases summary | |
| Depth before and after; time of change | |
| Work now held for people; who covers it | |
| Production look-back: period, method, findings | |
| Incident opened? Reference | |
| Cause and evidence | |
| Fix and approver | |
| Re-run date and added cases | |
| Report owner; approvals |
A worked sample
A hypothetical bank's card servicing agent failed its renewal run.
| Section | Entry |
|---|---|
| Capabilities tested | Block card (depth 3); unblock card (depth 3); order replacement (depth 3) |
| Gates failed | Scope: floor of no more than 2 failures in 400 cases; 9 failures |
| Gates cleared | Grounded, refuses, depth, escalates, adversarial, stable |
| Failing cases | Agent unblocked cards that had been blocked by the fraud team, not by the customer |
| Depth change | Agent to depth 1 at 16:40 on the day of the run; the run cleared no depth |
| Work now held | Around 300 card requests a day; card operations team extended hours for one week |
| Look-back | 30 days of unblocks checked; 4 fraud-team blocks lifted by the agent; fraud team reviewed each, no losses |
| Incident | Opened and linked, because live unblocks occurred |
| Cause | Card system update merged two block reasons into one field; agent could no longer tell who placed the block |
| Fix | Integration restored block reason; brief now says unblock only when block reason is "customer" |
| Re-run | Five days later, with 40 added cases on fraud-team blocks |
Block and replacement capabilities had done nothing wrong, but all three dropped together because the agent is certified as a whole, to one depth. After the re-run passed, the agent's depth returned.
What do auditors look for in these reports?
They check that depth dropped the same day, not days later; that the look-back was real, with a period and a method; and that the re-run was on an enlarged set rather than the same cases. Where your platform provider supplies a SOC 2 report, auditors may also check that the change controls it describes cover certification failures; SOC 2 for AI agent platforms: what the report covers explains the scope. The wider frame is in what it means to certify an agent.
Questions people ask
Does a failure on one capability affect the others?
Yes. Depth drops for the agent, which is certified as a whole, to the highest depth the run cleared. The other capabilities come back with the agent's next passing run.
Who signs off the re-run?
The same independent reviewer who reviews normal runs, plus the risk owner who approved the depth drop. Record both in the report.
Should the failing cases stay in the set?
Yes. They become part of the set for the re-run and every renewal after, so the same failure is tested for every time.