What is data drift?

Data drift is a change in the inputs an AI system receives compared with the inputs it was built and tested on: a different mix of requests, new document formats, new languages, new products or a shift in who the customers are. The rules have not changed, but the agent is now seeing work its evidence does not cover.

Why does data drift matter for agents?

An agent's certificate is only as good as the cases behind it. If an agent was certified on a sample of last quarter's work and this quarter's work looks different, the evidence describes a job the agent is no longer doing. It may still perform well. You no longer know.

Agents are exposed to more kinds of data drift than older models because they read unstructured input. A new supplier sends invoices in a layout the agent has never seen. A marketing campaign brings in younger customers with different questions. A product launch adds a request type that did not exist when the agent was tested. Each one quietly moves the agent outside its proven ground.

How does it show up in practice?

Data drift is the easiest kind of drift to detect, because you can measure the inputs directly. Useful signals:

• the share of requests by type, channel, language or product;
• document formats and sources the agent reads;
• the length and complexity of cases;
• the share of cases the agent holds, refuses or cannot process;
• new values in fields the agent relies on.

Compare each against the certification set. A large shift is a prompt to look at quality closely, even before any error appears. Detecting drift in a certified AI agent sets out a monitoring routine.

What does an example look like?

An accounts payable agent was certified on invoices from a firm's main domestic suppliers. The procurement team signs up a group of overseas suppliers, whose invoices show tax and currency differently. Nothing breaks: the agent still extracts amounts and matches purchase orders. But the share of invoices from unfamiliar layouts climbs, and with it the share the agent holds for review. The operations lead adds examples of the new layouts to the case set and recertifies the capability before raising its depth on that supplier group.

How is data drift different from concept drift and model drift?

Data drift is a change in what comes in. Concept drift is a change in what the right answer is for the same input, such as a new policy. Model drift is a change in how the system itself behaves or performs. Data drift can occur without the answer changing: the overseas invoices above still have one correct total. That makes it more benign than concept drift, but also more common, and it is often the first warning that quality is about to slip.

What should you do when you find it?

Not every shift needs action. Ask whether the new inputs are covered by the agent's evidence. If they are a small, familiar variation, keep watching. If they are a new kind of case, add them to the case set and recertify the affected capability. If they are outside the agent's role altogether, route them to people. The point, made in drift is why certification is never finished, is that evidence ages as the work changes. The NIST AI RMF guidance on monitoring and drift for AI agents frames this as ongoing measurement rather than a launch check.

For the wider operating picture, see our guide to deploying AI agents.

Where Pryme fits

Attest certifies on cases from the customer's own operation, and certificates lapse after 90 days, so each renewal tests the agent on recent work. One of its seven gates, stable, checks that the agent gives the same answer to the same question across runs, and a run that clears no depth drops the agent to depth 1 the same day.