People affected by a decision that a deployer bases on a high-risk AI system's output can ask for a clear and meaningful explanation of the system's role in the decision and its main elements. For an AI agent, a usable explanation names what the agent did, what it relied on, and what the person who decided weighed.
What does the Act require on explanations?
The EU AI Act gives a right to explanation in substance as follows. Where a deployer takes a decision based on the output of a high-risk system listed in Annex III, and that decision produces legal effects or similarly significantly affects a person's health, safety or fundamental rights, the person can obtain from the deployer clear and meaningful explanations of:
- the role the AI system played in the decision-making procedure; and
- the main elements of the decision taken.
The right applies only where other EU law does not already provide it. Two other parts of the Act support it. Article 13 requires high-risk systems to be transparent enough for deployers to interpret their output, so the deployer has the material to explain. Article 26 requires deployers to inform people that they are subject to the use of a listed high-risk system.
Explainability as a concept is broader than the legal right; our glossary entry on what explainability is sets out the difference.
Who can ask, and who must answer?
The person affected can ask. The deployer must answer, because the deployer took the decision. A lender using an agent to support credit decisions on individuals is the obvious case; so is an insurer pricing life or health cover, or a public body determining eligibility for a benefit.
The right sits beside GDPR, not on top of it. Where a decision is solely automated with legal or similarly significant effects, GDPR's Article 22 already gives the right to human intervention, to express a view and to contest, and GDPR requires meaningful information about the logic involved. Our piece on whether a customer can object to a decision made by an AI agent covers those routes.
Why is explaining an agent harder than explaining a model?
A score has features and weights. A deployer can say which factors pushed it down. An agent's contribution is a sequence of steps: it gathered documents, read some and not others, computed figures, summarised, and drafted a recommendation. The explanation a person needs is about that sequence.
Three things make it harder:
- The model cannot reliably explain itself. Asking the agent afterwards why it did something produces a plausible account, not a record. The explanation must come from the record of what happened, not from the agent's narration.
- The person deciding may have relied on parts of the agent's work. If an underwriter accepted the agent's income summary, the summary is part of the decision's main elements. If it dropped a source of income, that matters to the explanation.
- Reasons must be specific. "Our system assessed your application" is not meaningful. "Your stated monthly income could not be matched to the bank statements provided, and the underwriter declined on affordability" is.
Lenders in the US face a similar problem under adverse action rules; our piece on why adverse action reasons are harder when an agent helped makes the parallel.
How do you produce a usable explanation?
| Requirement | Control | Evidence it leaves |
|---|---|---|
| Explain the AI system's role | Action record showing each step the agent took in the case | Case-level record, retrievable on request |
| Explain the main elements | Decision record naming the person who decided and the factors they relied on | Decision note linked to the agent's record |
| Ground the explanation | Each fact in the agent's work carries its source | Source references in the record |
| Inform people of the use | Notice in application and decision communications | Templates and samples sent |
| Respond to requests | Process with owner, deadline and template | Request log and responses |
| Do not invent reasons | Explanations drafted from the record, checked by a person before sending | Approval of each explanation |
| Monitor quality | Sample explanations reviewed for specificity and accuracy | Review results |
An agent can draft the explanation from the record. A person should approve it before it goes out, because an explanation is a statement by the firm about its own decision.
What will a supervisor or auditor ask?
- Show us an explanation you sent. Can you rebuild it from your records?
- How did you separate the agent's work from the decision-maker's judgement?
- How quickly do you respond, and who approves the response?
- What do you do when the explanation reveals the agent made an error?
- How do these explanations feed back into monitoring?
The last question links explanations to the provider's monitoring loop, explained in EU AI Act post-market monitoring for AI agents.
What does the right not cover?
It does not require disclosure of the model's internals or trade secrets. It applies only to listed high-risk systems and only to decisions with legal or similarly significant effects. It does not apply where other EU law already provides an equivalent right. And it does not reach the UK, where explanation duties come through UK GDPR and the regulators' expectations instead; see EU AI Act vs the UK approach to AI agents. For the wider map, read AI agent regulation, mapped to controls.
This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.
Where Pryme fits
Pryme files every verdict in an append-only record that names the bound that decided, the sources used and their provenance class, and the person who approved a held action. An explanation can be drafted from that record rather than reconstructed from memory, and every fact in it traces to where it came from.
Questions people ask
Can an agent write the explanation to the customer?
It can draft one from the case record. A person should approve it before sending, because it is the firm's account of its own decision and errors in it compound the original harm.
What if no human was involved in the decision?
Then GDPR's rules on solely automated decisions are likely to apply as well, with rights to human intervention and to contest. Most regulated firms keep a person in the decision for listed high-risk uses.
Does every customer communication need an explanation?
No. The right applies on request, to decisions with significant effects based on high-risk systems. Routine service answers are covered by disclosure duties, not this right.