Drift is why certification is never finished

A certificate proves that an agent handled your cases well on the day it was tested. Drift is everything that changes after that day: the sources the agent reads, the customers who write to it, the model underneath it, the policies it applies. Each change weakens yesterday's evidence, which is why certification is repeated, never completed.

What changes after an agent is certified?

Certification is a measurement taken at a point in time. To certify an agent properly you replay it against real cases from your own operation and score each answer against a floor. The result tells you how that agent, with that brief, those sources and that model, performed on that set of cases. Change any of those inputs and the result describes a slightly different agent.

Four things move in almost every operation:

None of these is a failure. They are the normal life of a business. The point is that each one quietly changes the question the certificate answered.

Why does old evidence weaken even when nothing breaks?

The dangerous version of drift is the slow one. A sudden failure gets noticed: complaints arrive, the held queue fills, someone raises an incident. Gradual drift produces answers that still read fluently and still cite a source, while the share of live cases that resemble the certification set slowly falls.

Take a regional lender whose support agent was certified on questions about personal loans. Over a few months the lender starts offering a small business product, and customers begin asking the same agent about business repayment holidays. Nothing in the certification set covered those questions, so the certificate says nothing about them, yet the agent answers anyway, often well. The evidence has not been contradicted. It has simply stopped covering the work.

This is the idea behind data drift: the inputs a system sees move away from the inputs it was validated on. For an agent that acts, the consequence is not only lower accuracy. It is authority exercised on cases nobody tested.

Why do certificates lapse on a timetable at all?

If drift were always detectable, you could recertify only when it happened. It is not, so a fixed expiry acts as a backstop. In our model certificates lapse after 90 days. That is not a claim that agents decay at a particular rate; it is a limit on how long untested change can accumulate before the evidence must be renewed.

The timetable is the floor, not the plan. Most material changes should trigger recertification on their own, well before the date: a new model version, a rewritten policy, a new product line, a change in the customer's limits. A firm that recertifies only when the calendar says so is relying on luck for the weeks in between.

This is also where agent certification differs from management-system certification. An audit of how your organisation governs AI checks that processes exist and are followed; it does not replay the agent's answers. The difference is set out in whether AI agent certification is the same as ISO certification. Both have a place, but only one goes stale when your fee table changes.

How should drift be caught between certifications?

Catching drift is ongoing work, not an afterthought. At each run, Attest's stable gate checks something narrower: that the agent gives the same answer to the same question across runs and across a conversation. Between runs, the question is whether the agent's live behaviour still resembles the behaviour that was certified. Signals worth watching include:

• the mix of requests reaching the agent compared with the mix in the certification set;
• the rate of held and refused verdicts, in either direction, since a sudden drop in refusals can mean the agent is now answering things it used to decline;
• changes to any sealed source the agent relies on, such as the policy library or a product table;
• a new model version from the provider, even one described as minor;
• reviewer corrections on held drafts, which tend to rise before complaints do.

Detection is only useful if someone acts on it. A practical method for setting thresholds and owners is covered in how to detect drift in a certified AI agent.

What does recertification cost, and is it worth paying?

Recertification is not free. Someone has to refresh the certification set with recent cases, decide the correct outcome for the new ones and review the results. The temptation is to stretch the interval, or to keep reusing the original set because it already has agreed answers.

Reusing an old set is the cheaper mistake to make and the more expensive one to discover. A set that no longer reflects current work produces a passing score on yesterday's questions. The certification set selection checklist is worth running each time, specifically to add cases drawn from whatever has changed since the last run.

The trade-off is real, so scale the effort to the capability. A capability at depth 1, reading and summarising, can tolerate a lighter refresh than one at depth 4 that moves money. The more an agent can do without a person, the less stale evidence you can afford.

What should you do differently?

Treat a certificate as a dated statement, not a permanent property of the agent. Write down what it covered: the brief, the sources and their versions, the model, the case mix. Then make every change to any of those a recorded event that either triggers a new run or is explicitly judged not to need one, with a named person accepting that judgement.

Plan recertification as routine work, with time and reviewers budgeted, rather than as an exception that competes with launches. And when drift is detected, act on depth first and investigate second. An agent whose evidence no longer covers its work should lose authority until it is proven again.

Where Pryme fits

In Pryme, Attest certifies each agent against the customer's own cases, and one of its seven gates, stable, checks that the agent gives the same answer to the same question across runs. Certificates lapse after 90 days, and changes to the brief, sources or model prompt a new run. A run that clears no depth drops the agent to depth 1 the same day, and the verdict record shows which certificate was in force for every action.