Run both versions on the same real cases, with the same sources and the same bounds, without either acting. Score each answer per capability, then line the two up case by case and have experts review every case where they differ. The new version replaces the old only if it does not fall below the floor anywhere, whatever its average.
Why is a fair comparison harder than it looks?
Because many things change at once if you let them. If the sources were updated between runs, you are comparing data, not versions. If the cases differ, you are comparing luck. If you look only at an overall score, a version that is better at easy cases and worse at refusing the dangerous ones can look like an improvement.
Models also do not always give the same answer twice. A case that flips between runs tells you about instability, not about the change you made.
What are the steps, and who does each?
- Change owner: state what changed and what should differ. A new source, a reworded brief, a different model. Write down which cases you expect to change and how. Evidence: the change record with the expected effect.
- Agent operations lead: freeze the case set and sources. Use the certification case set plus any cases the old version got wrong. Snapshot the sources so both versions see identical material. Evidence: case set version and source snapshot reference.
- Platform team: run both versions in parallel, without acting. Same cases, same bounds, same inputs. Run unstable cases more than once. Evidence: paired outputs for every case.
- Agent operations lead: score each capability. Per capability and per kind of check: grounded in sources, refused when it should, escalated when it should, stayed in scope. Evidence: a score table for each version.
- Subject experts: review the differences. Classify each changed case as improved, regressed or neutral. Review every regression and a sample of the rest. Evidence: the difference report with each expert's judgement.
- Process owner and risk: decide. Evidence: the decision, with reasons.
- Platform team: release and keep the old version ready. Evidence: the release record and rollback plan.
Lining up two sets of outputs and investigating every mismatch is the same discipline as reconciling records between two systems: match each item, then explain every break rather than accepting a net total.
How should differences be judged?
| Difference | How to treat it |
|---|---|
| Better answer on a case the old version got wrong | Improvement; confirm the reason matches the change |
| Worse answer on a case the old version got right | Regression; review every one |
| New version refuses or escalates less often | Treat as a regression until experts confirm each is correct |
| New version refuses or escalates more often | Check for over-caution, but it is the safer direction |
| Different wording, same substance | Neutral |
| Answer changes between repeated runs | Instability; investigate before release |
The third row matters most. A version that holds fewer items looks faster and cheaper. If even a few of those items should have been held, the comparison has found a problem the averages hide.
What does this look like in practice?
Take a broker's client onboarding agent. The team rewrites part of its brief to reduce unnecessary escalations on corporate clients. They run both versions on four hundred past onboarding cases.
The new version escalates fewer cases, as intended, and scores slightly higher overall. The difference report shows thirty changed cases. Experts confirm twenty-six are correct improvements. Four are regressions: corporate clients with a trust in the ownership chain, which the old version escalated and the new one did not. The brief change had loosened a rule it should not have touched. The team fixes the wording, re-runs, gets twenty-six improvements and no regressions, and releases.
When two versions give conflicting answers and the experts cannot agree which is right, treat it like any disagreement between agents: a person decides, and the case is added to the test set. See what happens when two AI agents disagree.
Which controls make the comparison trustworthy?
- Frozen inputs: same cases, same source snapshot, same bounds.
- Per-capability scoring, never only a single overall score.
- Every regression reviewed by a named expert.
- No regression below the floor on any check, especially refusal and escalation.
- Rollback ready for the first weeks after release.
Where Pryme fits
In Pryme, Attest replays any agent version against real cases from your operation and scores every answer against a floor across seven gates, each of which must clear on its own. That makes a version comparison a pair of Attest runs on the same case set. The certificate records which version passed. More in deploying an AI agent: train, shadow, certify, widen.
Questions people ask
Can we compare versions on live traffic instead?
Yes, by running the new version in shadow alongside the live one. It adds recent cases but cannot replace a fixed set of known hard ones.
How many cases are enough?
Enough to cover every capability and its awkward variants. The past failures matter more than the total count.
Does a model change need the same comparison as a brief change?
Yes. A model change can alter behaviour everywhere at once, so it needs the full comparison, not a sample.