Every certification run, pass or fail, should leave a certification evidence pack: the brief tested against, how cases were chosen and labelled, each gate's floor and why, the run configuration, per-case results, failures and who reviewed them. Keep it with the certificate, because it is how you defend the result later.
Who is the pack for?
Three readers. Internal reviewers, who decide whether to accept the result. Auditors and supervisors, who may ask months later why an agent was allowed to act. And the team running the next certification, who need to know what was tested last time.
The certificate states the outcome. The pack shows the working. Without it, a certificate can be questioned and not defended: "the agent passed" has no weight if you cannot show what it passed, against what bar, judged by whom. The broader idea of certification is in what it means to certify an agent.
When do you assemble it?
Build it as the run happens, not afterwards. The case set and floors are fixed before the run; results and review notes are added as they arrive. Assemble a pack for failed runs too. A run that falls short certifies only the highest depth it cleared, and the pack is the evidence for that change and for the fix. How to handle a failed certification run uses the pack as its starting point.
How do you fill in each section?
- Scope. Agent, brief version, capabilities and depths being tested.
- Case set. Size, where cases came from, the period covered, how they were selected, and what was deliberately included: edge cases, refusals, incidents.
- Labelling. Who decided the correct answer for each case, their role, and how disagreements were resolved.
- Gates and floors. For each of the seven gates, the certification gate definition used and the floor applied, with one sentence on why that floor. What a certification floor means for AI agents covers how floors relate to the cost of error.
- Run configuration. Model and version, source versions, connections, limits in force. Enough to repeat the run.
- Results. Per case and per gate. Summaries are not enough; reviewers sample individual cases.
- Failures. Each failed case with a short explanation.
- Review. Who reviewed, what they sampled, their conclusion. Who should review an AI agent's certification results sets out who is suited.
- Outcome. Certified depth, certificate ID, expiry.
The certification evidence pack template
| Section | Contents | Owner |
|---|---|---|
| 1. Index | List of documents, with versions | Certification lead |
| 2. Scope | Agent, brief version, capabilities, depths requested | Agent owner |
| 3. Case set | Size, source, period, selection rules, inclusions | Certification lead |
| 4. Labelling | Labellers, method, disagreement handling | Business owner |
| 5. Gates and floors | Seven gates, floor each, rationale | Risk |
| 6. Run configuration | Model version, sources, connections, limits | Platform team |
| 7. Results | Per case, per gate | Certification lead |
| 8. Failures | Each failed case explained | Certification lead |
| 9. Comparison | Scores against the previous run | Certification lead |
| 10. Review | Reviewers, sample checked, conclusion | Independent reviewer |
| 11. Outcome | Certified depth, certificate ID, expiry | Agent owner |
| 12. Record references | Links to the verdicts produced during the run | Platform team |
A sample index
For a hypothetical lender's arrears support agent:
| Section | Sample entry |
|---|---|
| 2. Scope | Arrears Support Agent, brief v1.6; answer arrears queries (depth 3); agree payment plan (depth 3 requested) |
| 3. Case set | 500 cases from the lender's own arrears conversations over six months; includes 60 customers in financial difficulty and 40 requests outside plan limits |
| 4. Labelling | Two senior collections officers labelled each case; 22 disagreements settled by the collections manager |
| 5. Gates and floors | Escalation floor set at no failures on cases involving financial difficulty, because one wrong plan there causes real hardship |
| 6. Run configuration | Model version as recorded; arrears policy v4; plan limits matrix v2 |
| 8. Failures | 3 escalation failures: agent offered a plan instead of referring a customer who mentioned a bereavement |
| 10. Review | Conduct risk reviewer sampled 50 cases, including all 3 failures; agreed with labels |
| 11. Outcome | Agent certified at depth 2, the highest depth it cleared, pending a fix to escalation |
Section 4 is where packs are most often thin. If the labels are wrong, the score means nothing, so the pack must show who labelled and how disputes were settled.
How long should a pack be kept?
At least as long as your record retention for decisions the agent made under that certificate, because the pack is what explains them. A certificate can also be lost quickly, as whether an AI agent can lose its certification overnight explains, and the pack for the next run will need the previous one for comparison.
Questions people ask
Does every run need a full pack?
Yes, but sections that have not changed can reference the previous pack rather than repeat it. Results, failures, review and outcome are always new.
Can the pack be generated automatically?
Run configuration, results and record references can be. Selection rationale, floor rationale and review conclusions must be written by people, because they record judgements.
Who should hold the pack?
The certification lead maintains it; it is stored with the certificate and the agent's inventory entry, where auditors will look for it.