With constructed cases, a low ceiling and a plan to replace them. Your subject experts write realistic cases from your policies and experience, including hard and rare ones. The agent is certified against those, but only to read or draft, with a shorter life than usual. Real cases are collected from day one and the agent is recertified on them.
Why would you have no real data?
Three common reasons. The process is new, such as a product launching next quarter. The data exists but cannot be used for testing, because consent, contracts or data protection rules prevent it. Or the history is too poor to score against: outcomes were never recorded, or the old process was so different the cases no longer apply.
Each calls for a slightly different response, but the principle is the same. Without your own cases, you have weaker evidence, so the agent gets less authority until better evidence exists.
How do you build cases that are good enough?
Start from the people who know the work. Constructed cases written by a general-purpose model, or taken from a public dataset, tend to be too clean. Experts know where the process actually breaks.
| Source of cases | Strength | Watch out for |
|---|---|---|
| Subject experts writing from experience | Includes real hard cases and edge cases | Biased towards memorable cases; slow to write |
| Cases derived from policy, clause by clause | Covers every rule at least once | Misses messy real-world inputs |
| Pseudonymised real cases | Close to real behaviour | Check the method meets your data protection duties |
| Cases from a similar existing process | Real inputs | Differences in rules can mislead |
| Adversarial cases written to break the agent | Tests refusal and harm | Can over-weight unlikely attacks |
A good set mixes these. Each case needs an expected outcome agreed by an expert before the agent is run, including what the agent should refuse and when it should hand off.
What depth can constructed evidence support?
Usually read or draft only. Constructed cases are a reasonable test of whether the agent follows rules and refuses what it should. They are a weak test of how it copes with the variety of real inputs. That makes them good enough for an agent whose output a person checks, and not good enough for one that acts alone.
This is where autonomy labels fall short. An agent may be technically able to act, and constructed cases may show it can, but the evidence does not support it yet. The point is argued in autonomy levels are not enough: certify the agent.
What are the steps?
- Narrow the scope to the one capability you need first; see how to certify an AI agent with a single capability.
- Have experts write cases with agreed expected outcomes, weighted towards hard cases.
- Run and score on every gate, exactly as you would with real cases. The method is in how to certify an AI agent.
- Certify at a low depth with a shorter lapse date and a condition that real cases replace constructed ones.
- Collect real cases from go-live, with the outcome a person decided.
- Rerun on real cases once there are enough to cover each case type, and certify again on that evidence.
An illustrative example
A lender is launching a new small-business overdraft. It wants an agent to answer applicants' questions about eligibility and documents, drafting replies for staff to send.
There are no past applications. The product manager and two experienced underwriters write a set of applicant questions from the product rules and their knowledge of similar products. They add questions the agent should refuse, such as "will I be approved?", and questions that should go to a person, such as a sole trader mentioning a recent bankruptcy. The agent passes and is certified at depth 2 for 30 days. Over the first month, real questions are logged with the replies staff sent. The agent is then rerun on those, fails scope on a group of questions about personal guarantees, gets a brief fix, and is recertified on real evidence.
What controls keep this safe?
- A depth ceiling of draft until real-case evidence exists.
- A short lapse, so constructed evidence cannot quietly become permanent.
- Every output reviewed by a person while the agent is on constructed evidence.
- A record of corrections, which becomes the next test set.
- A rerun trigger when real cases reach an agreed volume. When the agent's brief later changes, the same discipline applies; see how to re-certify an AI agent after its brief changes.
Testing on real cases once they exist is covered in how to test an AI agent against real cases. The wider reasoning is in what it means to certify an agent.
Questions people ask
Can synthetic data generated by a model stand in for expert cases?
It can add volume, but it tends to reflect what the model already finds easy. Use it alongside expert cases, not instead of them.
Is pseudonymised data safe to use for testing?
It reduces risk, but it is still personal data if individuals can be re-identified. Check the method with your data protection team before using it.
How many real cases do you need before recertifying?
Enough to cover each case type you care about, including rare ones. A target per case type works better than a single total.