Draw the cases from your own filed work for that capability, and choose them to cover every outcome the agent must get right: act, draft, refuse, escalate. Add the rare cases that cause the most harm, and deliberate attacks. Keep a hold-out set the brief was never tuned on. The capability owner approves the set, and it is versioned.
What is a certification set, and who owns it?
A certification set is the collection of cases an agent is replayed against before it is granted depth for a capability. Each case is a real piece of past work from the customer's own operation, such as a refund request, an alert or an application, together with the right answer: what a competent person should have done, and what the agent should do.
The set decides what "proven" means. If it is all easy cases, the certificate proves the agent can do easy work. The capability owner owns the set, with risk or compliance reviewing it. The person who wrote the agent's brief should not choose the cases alone, for the same reason that a student should not write their own exam.
What are the steps?
- Person, owner: restates the capability. One action at one depth. The set is built for that capability and nothing wider.
- Agent, depth 1: pulls the population. All filed cases for that capability over a period long enough to include seasonal and unusual work.
- Agent, depth 1: profiles it. By product, channel, customer type, complexity and outcome, so you can see what normal work looks like.
- Agent, depth 2: drafts a stratified sample. Enough cases from each group that a failure in any one would show.
- Person: adds the outcome classes. Cases where the right answer is to act, to draft for a person, to refuse, and to escalate. The refusal cases deserve special care; the refusal test cases template gives a structure.
- Person: adds edge cases. Rare cases where harm is high: vulnerable customers, disputed facts, unusual amounts, conflicting sources. Edge cases belong in the certification set, not in a footnote.
- Person: adds attack cases. Inputs that try to make the agent exceed its bounds, such as instructions hidden in documents or customers pressing for an exception.
- Person: checks expected answers. Filed outcomes are not always right. Each expected answer is reviewed, and corrected cases are marked as such.
- Person: splits off a hold-out set. Cases the brief is never tuned against, used only for the certification run.
- Person, owner: approves and versions the set.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Restate | Nothing | None | The capability |
| Population | Pulls filed cases | 1 | Period chosen |
| Profile | Groups cases | 1 | None |
| Sample | Drafts stratified sample | 2 | Accepting it |
| Outcome classes | Nothing | None | Choosing refusal and escalation cases |
| Edge cases | Nothing | None | Choosing them |
| Attacks | Nothing | None | Designing them |
| Expected answers | Nothing | None | Correcting them |
| Hold-out | Nothing | None | Keeping it apart |
| Approve | Nothing | None | Owner sign-off |
Note that the agent helping here is a separate assistant, not the agent being certified. The agent under test should never see the set before the run.
What controls make it safe?
Separation of builder and chooser. The brief's author can propose cases; the owner decides. Evidence: the approval record naming both.
Hold-out kept apart. The hold-out cases are stored where the brief's author cannot see them. Evidence: access logs and the set version used in each run.
Data minimisation. Cases contain personal data. Use only the fields the capability needs, mask what is not needed, and keep the set under the same controls as the live data.
Coverage report. For each approved set, a short report shows how many cases fall in each group and outcome class. Gaps are visible before the run, not after. The certification set selection checklist lists the groups to check.
Versioning. Every change to the set is a new version, and every certificate names the version it was earned on.
How do you size it?
Size depends on how finely you need to see failure. If one group is small in real life but high in harm, it needs enough cases that a single failure shows up against its floor. That is why the set and the floors are designed together; setting the pass floor for each certification gate covers the other half. A set of a few dozen cases can be enough for a narrow drafting capability. A depth 4 capability that moves money needs far more, and many more edge cases.
What are the signs it is going wrong?
- Every gate clears comfortably on the first run. Either the agent is excellent or the cases are easy. Check the edge cases first.
- The set looks like the demo. If the cases resemble the examples used to build the brief, the hold-out has leaked.
- No refusal or escalation cases fail, ever. Either the set lacks hard ones or the scoring is too lenient.
- The set has not changed in months. New products, new fraud patterns and new policies are not in it.
- Expected answers copied from filed outcomes without review. The agent is being certified to repeat past mistakes.
Where Pryme fits
In Pryme, Attest replays the agent against the approved set and scores every answer on all seven gates. The set version, its approver and its coverage report are filed with the certificate, so anyone reading the certificate can see what it was earned on. More on what certification means is in what it means to certify an agent.
Questions people ask
Can we use synthetic cases?
For a new capability with no history, yes, but the certificate should record that the set is constructed and the granted depth should reflect that. Replace them with real cases as they accumulate.
Should cases from other customers be used?
No. A certificate is per customer. Another firm's cases reflect their policies, customers and systems, not yours.
How often should the set be refreshed?
At least before each renewal, and whenever the capability, the policy or the customer base changes in a way the current set does not cover.