Take closed cases from your own records, rebuild what was known when each arrived, and replay them to the agent in a sandbox where its actions go nowhere. Compare what it does with what a competent person decided, gate by gate. The hard part is rebuilding each case faithfully, so the agent cannot see the answer.
Why replay closed cases rather than live ones?
Because closed cases have known outcomes and carry no risk. A refund that was correctly refused, a complaint that was upheld, a payment that turned out to be fraud: each tells you what the right answer was. Running the agent against them shows whether it would have reached the same place, without any customer being affected.
Your own cases also contain the mess that test sets written from scratch lack: incomplete forms, contradictory notes, customers who change their request halfway through. That is where agents fail. The argument for this approach is set out in scoring an agent against your own filed cases.
What are the practical steps?
- Select cases across the full range: routine, hard, rare, and cases where the right outcome was to refuse or escalate.
- Strip the outcome. Remove the decision, the final reply and any later notes that reveal what happened.
- Rebuild the state at the time. The agent should see the customer's account, the policy version and the records as they were when the case arrived, not as they are now.
- Sandbox the tools. The agent's actions are captured, not executed. A refund it issues is recorded as an attempt.
- Run each case more than once to check consistency.
- Score against the outcome and the rules, gate by gate.
- Review disagreements one by one, because sometimes the agent is right and the original decision was wrong.
Why does rebuilding the state matter so much?
Because leakage makes the agent look better than it is. If the account record now shows a refund that was issued, the agent will "decide" to issue it. If the policy library has been updated since, the agent will apply rules that did not exist at the time.
Point-in-time reconstruction is the most labour-intensive part of real-case testing, and the part most often skipped. Teams that skip it get high scores that do not hold in production.
What should the case set include beyond normal work?
| Case type | Why include it |
|---|---|
| Routine cases | Show the agent handles the bulk of the work |
| Hard cases experts remember | Show it copes with ambiguity |
| Cases that should be refused | Test the refusal gate; a refusal test cases template helps structure them |
| Cases that should go to a person | Test escalation, especially vulnerability |
| Cases with planted instructions in documents | Test resistance to prompt injection |
| Cases where the original decision was later overturned | Check the agent does not repeat known mistakes |
Real inputs rarely contain attacks, so add them deliberately: an emailed invoice with hidden text telling the agent to change payee details, for example. How to test an AI agent against prompt injection covers this. How many of each type you need is discussed in how many test cases you need to certify an AI agent.
An illustrative example
Take a lender testing an agent that handles requests to change a payment date on a personal loan. The team pulls several hundred closed requests from the last two quarters.
For each, they rebuild the loan record as it was on the day of the request: arrears status, previous date changes, and the policy version then in force. They remove the handler's note and reply. The agent is run three times on each case in a sandbox. It matches the handler on most routine cases. On a cluster where the customer had changed their date twice already, it approved a third change; the policy allowed only two a year. The handlers had refused. The brief is fixed to read the change history, and the full set is rerun.
In a handful of other cases, the agent refused where the handler had approved. Review shows the handlers had made an exception without recording why. The policy owner decides those cases should go to a person in future, and the expected outcome is updated.
What controls keep real-case testing safe?
- Data protection review before using customer records for testing, with access limited to the test team.
- No live actions: every tool call is captured in the sandbox.
- No retention in the model: test cases change the agent's brief and sources, never model weights.
- A record of the case set, its version and the outcome agreed for each case.
When real cases do not exist yet, see how to certify an AI agent without real data. The framework this testing feeds is in what it means to certify an agent.
Where Pryme fits
Attest replays agents against real cases from the customer's own operation, with actions captured rather than executed, and scores seven gates separately. Pryme does not update model weights for a customer, so test cases inform the agent's brief, sources and bounds only.
Questions people ask
What if the original decision was wrong?
Have the process owner decide the correct outcome and record it. Testing often surfaces inconsistent past decisions, which is useful in itself.
Can real-case testing replace a pilot?
It reduces what the pilot needs to prove. A short period of live shadow running still catches things historical cases miss.
How often should the case set be refreshed?
Add new cases regularly, especially ones the agent got wrong in operation. A set that never changes tests yesterday's work.