Providers of high-risk AI systems must run a post-market monitoring system: a documented plan to collect and analyse data on how the system performs once in use, so they can check it keeps meeting the EU AI Act's requirements. For AI agents, most of that data comes from deployers, in the form of what agents actually did.
What does post-market monitoring require?
In substance, a provider of a high-risk system must:
- establish and document a post-market monitoring system proportionate to the nature of the technology and its risks;
- actively and systematically collect, document and analyse relevant data on the system's performance throughout its lifetime, whether provided by deployers or gathered from other sources;
- use that data to evaluate the system's continuous compliance with the Act's requirements;
- where relevant, include analysis of the system's interaction with other AI systems;
- base the system on a post-market monitoring plan that forms part of the technical documentation.
Findings feed back into the provider's risk management under Article 9. Serious incidents found through monitoring must be reported to authorities. Financial institutions and others already subject to monitoring duties under sector law can integrate the AI duties into existing systems, provided they achieve an equivalent level of protection.
Who does what?
The provider designs and runs the monitoring system. The deployer, under Article 26, monitors the operation of the system on the basis of the instructions for use and informs the provider when relevant. If the deployer has reason to think the system presents a risk, it must inform the provider and the market surveillance authority and suspend use. Our explainer on who enforces the EU AI Act for AI agents identifies which authority that will be.
The two duties meet in the data. A provider cannot monitor how an agent behaves in a bank's credit process without the bank's records. A deployer cannot judge whether a pattern is a fault in the agent or in its own configuration without the provider's analysis across customers.
What should be monitored when the system is an agent?
The Act's mention of interaction with other AI systems is unusually relevant to agents. Agents call other agents, pass work between them, and read content that other AI systems wrote. A fault can arise in the hand-over, not in either agent.
For an agent, monitoring should track behaviour, not only accuracy:
- Actions by verdict. How many actions were allowed, held, refused, and why. A rise in refusals at a particular limit may mean the agent is trying to exceed its role.
- Human decisions on held actions. Approval, rejection and edit rates. Rising rejections mean the agent is getting worse or the work is changing.
- Escalations. Whether the agent hands over when it should. A falling escalation rate can be a warning, not an improvement.
- Source behaviour. Which sources the agent relies on, and whether it has started relying on less trusted ones.
- Outcomes for people. Complaints, reversals and corrections linked to agent actions, broken down by group where lawful.
- Drift after change. Behaviour before and after any change to model, brief or source.
This is the same discipline the voluntary US framework describes; see NIST AI RMF monitoring and drift for AI agents.
How do deployer evidence and provider monitoring fit together?
| Requirement | Control | Evidence it leaves |
|---|---|---|
| Provider collects performance data | Deployer shares aggregated action and outcome data on an agreed schedule | Data-sharing agreement; transfer log |
| Systematic analysis | Provider analyses across deployers; deployer analyses its own use monthly | Provider reports; deployer monitoring reports |
| Interaction with other AI systems | Hand-overs between agents logged as events, with both agents identified | Records of multi-agent runs |
| Feed risk management | Findings logged as risks with owners and actions | Risk register updates |
| Deployer monitors per instructions | Dashboards built from the action record, not from sampled transcripts | Dashboard exports |
| Inform provider of risks | Defined route and timescale for reporting concerns | Tickets to provider with dates |
| Suspend if risky | Stop control with a named decision-maker | Suspension log |
| Serious incidents | Incident procedure covering both parties | Incident records and notifications |
Data sharing needs care. Action records contain personal data. Share aggregates and pseudonymised samples where they serve the purpose, and record the lawful basis for anything more.
What will a supervisor or auditor ask?
- Show us the monitoring plan. What does it measure for this agent?
- What data do you receive from deployers, and how often?
- What has monitoring found, and what changed as a result?
- How do you monitor hand-overs between agents?
- When did a deployer last raise a concern, and what happened?
An illustration from a different area makes the point. A firm running an agent to triage alerts in market abuse surveillance needs to know if the agent starts closing alerts that analysts would have escalated. Only monitoring of escalation and override rates over time will show that; the question whether an AI agent can do market abuse surveillance turns on it.
What does post-market monitoring not cover?
It is a provider duty for high-risk systems only. It does not set metrics or thresholds. It does not replace the deployer's own monitoring under sector law, such as model performance monitoring in banking. And it does not resolve how personal data in agent records may be shared, which is a GDPR question. For prohibited uses that monitoring should never find, see whether the EU AI Act bans any uses of AI agents, and for the full map, AI agent regulation, mapped to controls.
This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.
Questions people ask
Do deployers have to share data with the provider?
The Act expects providers to collect data provided by deployers and expects deployers to inform providers of risks. The detail of what is shared should be agreed in the contract, within data protection limits.
How often should agent monitoring run?
Continuously for alerts on unusual patterns, and on a fixed reporting cycle for review. Run an extra review after every material change to the agent.
Is monitoring the same as re-testing?
No. Monitoring watches live behaviour; re-testing replays known cases under controlled conditions. You need both, because each catches failures the other misses.