It should, until the agent passes again. A certificate is evidence about the agent as tested, and the model is part of that agent. Any model update, even a minor version, can shift how it grounds answers, refuses, escalates and uses tools. Rerun the same cases, compare every gate, and restore the certificate only when each one clears.
Why does a "better" model still need retesting?
Because better on average is not the same as unchanged where it matters. A model update that improves general reasoning can also make the agent more willing to answer when it should hand off, more inclined to paraphrase rather than cite, or more literal about an instruction it used to apply sensibly. Those shifts rarely show in the provider's release notes, because the provider does not know your agent's brief.
Certificates are scored per gate for exactly this reason. An update can raise the grounded score and lower the escalation score in the same run.
Which model changes count?
| Change | Treat as voiding? | Notes |
|---|---|---|
| Moving to a new model family | Yes | Effectively a new agent |
| A new version of the same model | Yes | Behaviour can change in ways not described |
| A provider's silent update behind a stable name | Yes, if you can detect it | Pin versions where possible |
| Changing model settings, such as temperature or output length | Yes | Affects consistency and completeness |
| Switching to a smaller or cheaper model for cost | Yes | Often weaker on hard cases |
| Same model, different hosting region | Usually no | Unless the provider states behaviour differs |
The certificate should record the exact model and version, alongside the brief and sources, so a change is visible. The AI agent certificate template includes these fields.
How do you recertify after a model update?
- Pin the old version while testing, so production keeps running on certified behaviour if the provider allows it.
- Rerun the full certification set, not a sample. Rare cases are where regressions hide.
- Compare gate by gate against the previous run. Look at which individual cases changed outcome, not just the totals.
- Add cases if the update is meant to change something specific, such as handling longer documents.
- Have a named person review the comparison and accept or reject it.
- Switch in production only after acceptance, and record the date.
If the provider retires the old version before the rerun passes, the agent drops to read-only. That is uncomfortable, and it is the right outcome. Planning around providers' retirement dates avoids most of the discomfort.
What is different from a source change?
A source change usually affects specific capabilities: those that read the changed source. A model update affects every capability at once, because every capability runs on the model. That makes model updates the broadest single trigger for recertification. The narrower case is covered in whether changing an agent's sources voids its certificate.
It also means teams should not schedule a model update and a source change together. If the rerun fails, it is unclear which caused it.
An illustrative example
Take a payments firm whose agent drafts replies to merchants asking about delayed settlements, certified at depth 2. The model provider releases a new version with stronger reasoning.
The rerun shows better grounding: replies now cite the settlement record more precisely. But on cases where a merchant's settlement was held for a compliance review, the new version explains the hold in more detail than the approved wording allows. That fails the adversarial gate: a merchant asking why their money is held is pressing on a bound, and the reply risks telling them something they should not know about a review. The firm keeps the old version in production, tightens the brief on held settlements, and reruns. The second run passes and the new version goes live a week later.
What should the certificate say about the model?
The model name and exact version, the settings used, and the date of the run that certified it. When the certificate lapses because of a model change, the record should show which version replaced which. What certificate lapse means for AI agents covers the consequence; what an AI agent certificate covers and what an agent certificate is set out the scope. The wider framework is in what it means to certify an agent.
Where Pryme fits
Pryme treats a model change as a change to the agent. The certificate drops until Attest replays the agent on the customer's own cases and every gate clears again. Because Pryme does not retrain models for customers, the agent's brief, sources and bounds are what get adjusted when a new model needs tuning.
Questions people ask
Can we test a new model before the provider forces the switch?
Usually, yes. Most providers make new versions available ahead of retiring old ones. Use that window to run the certification set.
What if the provider updates the model without telling us?
Pin to a specific version where you can. Where you cannot, run a short daily check on a fixed set of cases, so a silent change shows up quickly.
Does a failed rerun mean the new model is worse?
Not necessarily. It means the agent's brief and the new model do not yet fit. Adjusting the brief often fixes it.