What is model drift?

Model drift is a change over time in how well an AI model performs, or in how it behaves, compared with when it was tested. For AI agents built on a provider's language model, it has two causes: the world moving away from the model, and the model itself being changed underneath you.

Why does model drift matter for agents?

An agent's certificate is evidence about one agent, built on one model, at one point. If the model's behaviour shifts, the certificate describes something that no longer exists. The agent may answer in the same tone and format while getting more cases wrong, refusing less often, or escalating differently.

Agents also face a cause that older models did not. A firm that trained its own fraud model controlled every version. A firm using a provider's model may find its behaviour changed after an update it did not choose, or that a model version it relied on is being retired.

How does model drift happen in practice?

There are three main routes:

• Provider change: the model version is updated or replaced, and outputs shift in style, length, refusals or accuracy.
• Environment change: the inputs or the right answers move, so a static model performs worse. This is really data drift or concept drift showing up as falling performance.
• Configuration creep: small changes to prompts, retrieval settings or tools accumulate until behaviour is noticeably different.

The symptoms are the same whichever route: reviewers editing more, holds or refusals moving, complaints rising, answers changing on test cases that used to pass.

What does an example look like?

A wealth manager's agent drafts summaries of client portfolios for advisers. Without any change by the firm, summaries start omitting the risk warning paragraph on some products. Investigation shows the provider has rolled out a new model version that tends to shorten outputs. The firm's test set, rerun weekly, catches the change; the omission rate on test cases has jumped. The operations lead drops the capability to depth 1 while the brief is adjusted, then recertifies on the new model before restoring drafting.

How is model drift different from data drift and concept drift?

Data drift is about inputs changing. Concept drift is about the right answer changing. Model drift, in the strict sense, is about the system's behaviour or performance changing. Many teams use "model drift" loosely for any decline in performance, which hides the cause. It is worth separating them, because the fixes differ: data drift calls for wider evidence, concept drift for updated sources, and a provider-driven model change for recertification on the new version.

How do you catch it?

1. Pin model versions where the provider allows, and treat any version change as a change needing approval.
2. Rerun a fixed test set on a schedule, so behaviour changes show up even when live metrics are noisy.
3. Watch live signals: reviewer edits, holds, refusals, complaints.
4. Recertify on a fixed cycle and after any model change.

Detecting drift in a certified AI agent gives the monitoring routine, and the NIST AI RMF guidance on monitoring and drift frames it as continuous measurement. The case for treating certification as perishable is made in drift is why certification is never finished. For where this sits in operations, see our guide to deploying AI agents.

Where Pryme fits

Attest's stable gate checks that an agent gives the same answer to the same question across runs. Certificates lapse after 90 days, so every agent is re-proven on recent cases at least that often, and a run that clears no depth drops the agent to depth 1 the same day, whether the cause was the model, the data or the rules.