What is automation bias?

Automation bias is the tendency of people to accept what an automated system tells them, or proposes, without checking it as carefully as they would check a colleague's work. With AI agents, it turns a person's approval of a drafted action into a formality, so the control looks present but is not doing its job.

Why does automation bias matter once agents act?

Many agent deployments rely on a person approving the agent's drafts: the refund, the letter, the payment file. That approval is only a control if the person actually reviews. After a few hundred correct drafts, most reviewers start to skim, and some start to click. The agent's error rate has not changed; the review has quietly disappeared.

Regulators recognise this. Article 14 of the EU AI Act asks that people overseeing high-risk systems remain aware of the tendency to over-rely on their output, which is examined in Article 14, automation bias and the people who approve agent drafts.

Why are agents more prone to it than older automation?

Rule-based workflow tools produce output that is predictable: the same input gives the same result, and errors tend to be loud. An agent writes fluent, confident, plausible text, and its mistakes look like correct work. A wrong figure in a well-written letter is far easier to miss than a broken field in a form. That difference, explored in how an AI agent differs from workflow automation, is why reviewers of agent drafts need more support than reviewers of workflow output, not less.

How does it show up in practice?

Watch for these signs:

• approval times falling steadily while volumes stay the same;
• reviewers almost never editing or refusing a draft;
• the same reviewer approving long runs of items within seconds of each other;
• errors found later by customers rather than by reviewers.

None proves bias on its own, but together they suggest the review has become a click.

What does an example look like?

A finance team uses an agent to prepare supplier payment batches at depth 2; a payments officer releases each batch. For months the batches are clean. Then a supplier's bank details change through an emailed request, and the agent includes the new details in the next batch. The officer, used to perfect batches, releases it in under a minute. The fix is not to blame the officer. It is to make changed bank details stand out in the review, to require a second check on any payee changed in the last 30 days, and to stop external claims from changing payee details at all.

How do you design against it?

• Show reviewers what changed or is unusual, not just the finished draft.
• Show where each key fact came from and how strong its source is.
• Insert known-wrong test items occasionally and track whether they are caught.
• Rotate reviewers and cap queue lengths per sitting.
• Measure edits and refusals per reviewer, and talk about the numbers.
• Keep agents' depth low where review quality cannot be shown.

How is it different from over-reliance on a model?

Over-reliance is the broader organisational habit of trusting AI more than its evidence supports, such as granting depth without certification. Automation bias is the individual, moment-to-moment version: one person, one approval, not looking closely enough. Both are fixed with evidence, but automation bias also needs better review design. See our guide to deploying AI agents for how review fits into going live.