Scoring an agent against your own filed cases

Rebuild each filed case as it stood when it arrived, hide the outcome, and let the agent work it in replay with every action simulated. Score each answer on all seven gates, using automated checks where a rule decides and trained scorers where judgement does. Compare each gate with its floor separately, then read the failures, not the averages.

What does scoring against filed cases mean?

A filed case is a piece of past work with its full record: what came in, what sources were available, what was done, and what the right outcome was. Replaying the agent against filed cases asks a precise question: given exactly what a person had, would the agent have done the right thing, at the right depth, within its bounds?

This is different from testing against invented prompts or public benchmarks. Your filed cases carry your policies, your customers, your systems and your edge cases. The broader question of how to test an AI agent against real cases covers when to use them; this piece covers how to score.

What are the steps?

1. Person, owner: provides the approved set. The cases, their expected answers and the floors for each gate.
2. Platform: rebuilds each case as it stood. Inputs, customer data and source documents as they were at the time. Policy documents updated since are replaced with the versions then in force.
3. Platform: hides the outcome. Fields that record what happened next, such as the final decision, the closure code or a later complaint, are removed. Otherwise the agent can read the answer.
4. Agent, replay only: works each case. Its connections run in replay mode. Every action it attempts is checked by the platform and given a verdict, but nothing reaches a live system.
5. Platform: captures everything. The agent's answer, each claim with its cited source, each attempted action with its verdict.
6. Scorers: score every answer on every gate. Automated checks where a rule decides, trained people where judgement does.
7. Scorers: double-score a sample. Two scorers on the same cases, to check they agree. Low agreement means the rubric is unclear.
8. Platform: compares each gate with its floor. Each gate clears or fails on its own.
9. Person, reviewer: reads the result. Failures by case type, not just totals.

How is each gate scored?

• Grounded. Every factual claim in the answer traces to a source available in the case. Largely automatable: check each citation exists and supports the claim.
• Refuses. On cases where the right answer is to refuse, did it refuse? On cases where it should act, did it avoid refusing? Both directions count.
• Depth. Did it attempt any action above its certified depth? The platform's verdicts show this directly.
• Escalates. On cases that should reach a person, did it escalate, and with enough information for the person to act?
• Scope. Did it stay within its role, or reach into work it was not asked to do?
• Adversarial. Did it hold its bounds when someone pushed, and slow down where the customer's state called for it? Usually scored by people, and usually with the lowest tolerance.
• Stable. Did the same case get the same answer across runs, and did it keep to what it was told earlier in the conversation?

The certification gate scoring rubric template lays out how to write the criteria for each.

How do you read the result?

Read gate by gate. A run where grounded clears easily and escalation misses its floor has failed, and the grounded score is no consolation. Then read by case type. A failure rate that looks small overall may be concentrated in one product, one channel or one customer group, and that concentration is what matters.

Then read the failed cases themselves. Some will show a fault in the brief. Some will show a source that was unclear. Some will show the expected answer was wrong, which needs a second reviewer to confirm before it changes. Attacks, such as hidden instructions in documents, deserve their own reading; testing an agent against prompt injection covers that set.

What controls make the score trustworthy?

Point-in-time reconstruction. Evidence: the source versions used for each case.

Outcome hiding. Evidence: the list of fields removed, per case type.

Scorer agreement. Evidence: the double-scored sample and the agreement result.

Floors set before the run. Evidence: the floor version's timestamp.

Every answer kept. The full run record is filed, so any score can be checked later.

Where does it usually go wrong?

Hindsight leakage. The case record includes a note written after the decision, and the agent reads it. Scores look excellent and mean nothing.

Scoring against what the person did. The person who handled the case may have been wrong. Expected answers must be reviewed, not copied.

Averages that hide. An overall score across gates is not a certification result. Never combine gates.

Too few cases in a group. If a high-harm group has only a handful of cases, one failure is invisible or decisive by chance. How many test cases you need to certify an agent addresses sizing.

The framework behind scoring is in what it means to certify an agent.