A filed case is a real piece of work from a customer's own operation that has already been decided, recorded and closed, such as a resolved complaint, a settled claim or a reviewed payment, used as evidence when certifying an AI agent. Its recorded outcome becomes the label the agent's answer is scored against.
Why do filed cases matter?
A benchmark tells you how an agent does on someone else's questions. A filed case tells you how it does on yours: your products, your policies, your customers' phrasing, your awkward exceptions. Certification on filed cases is evidence about the work the agent will actually do, which is the argument of scoring an agent against your own filed cases.
Filed cases also carry what a written test rarely does: the cases people found hard. A complaint that bounced between three teams before it was resolved is worth more as evidence than ten tidy examples someone wrote to illustrate a policy.
How are filed cases prepared for certification?
- Select cases from the system of record, such as a case management system, across every type the capability covers.
- Balance by type and difficulty, so common easy cases do not swamp rare hard ones.
- Minimise personal data to what the test needs.
- Review the label: the recorded outcome is what people did, which is not always what they should have done.
- Freeze the set as a version, so results can be reproduced.
- Replay the agent against each case and score it gate by gate.
Well-structured records make this far easier. Firms that keep investigations in a consistent format, such as the payment investigation case template, find their filed cases ready to use. An agent can even help build those files in the first place, as described in how to let an AI agent build case files in your case management system.
What does that look like in an example?
A payments firm builds a certification set for its investigations agent from 400 filed payment investigations across the past year. During label review, an analyst notices that one case was closed with a refund that policy did not allow. The recorded outcome was wrong. The reviewer corrects the label, notes why, and keeps the case, because it tests exactly the judgement the agent needs. Had the label stayed as filed, the agent would have been rewarded for repeating a mistake.
How is a filed case different from a synthetic test case?
A synthetic case is written to test something: an imagined customer, an invented scenario. A filed case happened. Synthetic cases are useful for rare situations your records do not yet contain, and for attacks such as attempts to talk the agent past its limits. But a certificate resting mostly on synthetic cases proves the agent handles what its testers imagined. Filed cases prove it handles what your business actually sees. Mark any synthetic cases in the set so readers of the results know which is which.
Which related terms should you know?
- Certification set: the collection of cases a run uses.
- Qualification: fitting an agent to a customer's work, ending in certification on filed cases.
- Recertification: a later run, ideally on freshly filed cases.
The full method is in what it means to certify an agent.
Where Pryme fits
Attest replays an agent against filed cases from the customer's own operation and scores every answer against a floor at each gate. Because every verdict Pryme issues is also filed in an append-only record, the agent's own past work, once reviewed, becomes material for the next certification set.
Questions people ask
Do filed cases contain personal data?
Usually. Using them is processing under data protection law, so it needs a lawful basis and should follow the GDPR Article 5 principles, including keeping only the data the test needs.
What if a case type has too few filed cases?
Supplement it with written cases, clearly marked, and read that gate's result with care. A floor tested on a handful of cases says less than it appears to.