Compare today's agent with the one you certified. Keep the certification run as a baseline, watch the live mix of verdicts, reviewer edits and escalations, and record every change to model version, sources and connected systems. Replay the certification set on a schedule and after any change. A failed replay cuts depth the same day: the agent keeps only the highest depth the replay cleared, or drops to depth 1 if it cleared none.
What is drift, and who owns detecting it?
Drift is a change in how the agent behaves, or in the world it works in, after it was certified. It comes from four places. The model provider updates the underlying model. Your sources change: a policy is rewritten, a product is added. The inputs change: new customer groups, new fraud patterns, a seasonal surge. The connected systems change: a field is renamed or a new status appears. None of these is a change to the agent's brief, which is why none of them prompts anyone to re-certify.
That is the reason drift is why certification is never finished. The capability owner owns the decision when drift is found. A model risk or AI governance function usually owns the monitoring. The platform does the comparing.
What are the steps?
- Platform: keeps the baseline. The certification run's gate scores, the mix of verdicts on the certification set, and the versions of model, sources and connectors at the time.
- Agent, live at its certified depth: works as normal. Every action is filed as a verdict.
- Platform: watches the live verdict mix. The share of actions allowed, held, refused and escalated, by capability, compared with what the certification run implied.
- Platform: watches reviewer edits. For drafting capabilities, how much people change the agent's drafts before sending. A rise is often the first visible sign.
- Platform: watches versions. Any change in the model version, a source document or a connector's schema is recorded and triggers a replay.
- Platform: watches the input mix. Live inputs compared with the certification set's profile. New kinds of input mean the set no longer represents the work, which is data drift in its plain sense.
- Platform: replays on schedule. The certification set, or an approved sample, is replayed and scored, and the result is compared with the baseline, gate by gate.
- Person, owner: reviews drift alerts. Accepts the change with a reason, orders a re-certification, or lowers depth.
- Platform: drops depth on a failed replay. Automatically, the same day, to the highest depth the replay cleared, or to depth 1 if it cleared none.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Baseline | Nothing | None | Approving the baseline |
| Live work | Acts within its certificate | Certified depth | Held actions |
| Verdict mix | Nothing new | Certified depth | Reading trends |
| Reviewer edits | Nothing new | Certified depth | None |
| Versions | Nothing new | Certified depth | Approving upstream changes |
| Input mix | Nothing new | Certified depth | Judging new input types |
| Replay | Answers the set, simulated | Replay only | None |
| Review | Nothing | Unchanged | Accept, re-certify, lower depth |
| Failure | Nothing | Drops to the highest depth cleared, or to 1 if none | Restoring depth through a new run |
What evidence does each step leave?
The baseline is filed with the certificate. Live verdicts are filed one by one in the append-only record, so the mix can be recomputed for any period. Version changes are filed with the time they took effect, which lets you line them up against changes in behaviour. Replay results are filed like any certification run: every case, every answer, every score. Drift alerts are filed with the owner's decision and reason.
When a supervisor asks how you know the agent still behaves as certified, the answer is a series of dated replays and a verdict history, not an assurance from the vendor.
Why is a certified model not enough?
A model can be evaluated, and a vendor may certify it. But a certified model does not make a certified agent. The agent is the model plus your brief, your sources, your connections and your bounds. Any of those can drift. And a model update can change behaviour in your cases while leaving the vendor's evaluation unchanged. That is why the replay uses your cases, and why the stable gate checks that the same question gets the same answer run after run: the only test that matters is your work.
What should you measure?
- Replay results against baseline, gate by gate.
- Verdict mix by week, for each capability.
- Reviewer edit rate on drafts.
- Escalation rate. A falling rate can be as worrying as a rising one, if the agent has started handling cases it used to pass on.
- Time from version change to replay result. Should be short; the window in between is unproven.
What are the signs it is going wrong?
- Model version changes nobody saw. The provider updated and no replay ran. Version watch exists for this.
- Alerts that are always accepted. If every drift alert is waved through with "within tolerance", the floors and the alerting are out of step.
- Replays on a stale set. The input mix has moved, but the replay uses the old set and passes. The set must be refreshed as the work changes.
- Drift blamed on the agent when the sources moved. A replay fails after a policy update, and the team starts rewriting the brief. Check the version log first. If a source changed, the fix may belong in the source, or the change may be right and the expected answers need review.
- Edit rates rising slowly. No single week looks bad. Look at the trend over months.
Whether an agent can be certified at all, given that it can drift, is taken up in can an AI agent be certified?. The wider framework is in what it means to certify an agent.
Questions people ask
How often should replays run?
On a schedule set by the capability's risk, and always after a version change. Depth 4 capabilities warrant more frequent replays than reading or drafting.
Can we pin the model version to avoid drift?
Pinning helps where the provider allows it, but sources, inputs and connected systems still change. Pinning reduces one source of drift; it does not remove the need to watch.
Does drift always mean the agent got worse?
No. Sometimes behaviour changes for the better, or the world changes and the agent copes. The owner still reviews it, because the certificate describes the agent as it was proven.