How do you collect feedback on an AI agent's work?

Capture it where people already touch the agent's work: when they approve, edit or reject a draft, sample its routine output, or hear back from customers and downstream teams. Ask for a short reason code each time, link it to the decision record, and route it to an owner. Feedback should never change the agent's behaviour directly.

Where does useful feedback come from?

The first two rows produce most of the volume and cost the least, because the person is already looking at the work. A held item in a work queue can carry the reason codes as a required field on rejection and an optional one on edit.

What are the practical steps?

1. Agree a short list of reason codes. Six to ten: wrong fact, wrong source, missed rule, wrong tone, incomplete, should have escalated, should not have escalated, other. Long lists go unused.
2. Make capture part of the action. Rejecting a draft requires a code. Editing records the difference automatically. Nobody should fill in a separate form.
3. Link every item to the decision record. Feedback without the agent's inputs, sources and verdict cannot be diagnosed.
4. Sample routine work. Work the agent executes without approval gets no feedback by default. A weekly sample, reviewed against a checklist, fills the gap.
5. Triage weekly. The operations lead groups feedback by reason and capability and decides which need a fix.
6. Route each fix to its owner. A wrong source goes to whoever owns the source. A missed rule goes to the process owner. A tone problem goes to the brief.
7. Add the failing cases to the test set, so the fix is proven and the problem cannot quietly return.

Building review time into the team's day matters as much as the tooling; see allocating work across an operations team with an AI agent.

Why should feedback not change the agent directly?

Because an unreviewed change is an untested change. If every edit an approver made fed straight into the agent's behaviour, one person's preference, or one mistake, would spread to every future case without anyone deciding it should. Feedback is evidence for a change; the change itself goes through change control and is re-tested on real cases before it reaches production.

This also protects the record. If behaviour shifted with each piece of feedback, you could not say which version of the agent made a given decision.

What does this look like in practice?

Take a utility's billing team. The agent drafts responses to billing queries, and handlers send them. Over a month, handlers reject a small number of drafts and edit many more.

The reason codes show a pattern: most edits on one query type are coded "wrong fact", and the differences show handlers replacing the agent's estimated reading date with the actual one. The operations lead traces it. The agent was reading the scheduled reading date from the billing system rather than the date the reading was taken. The process owner confirms which field is correct, the source mapping is fixed, the edited cases go into the test set, and the agent is re-tested before the change goes live. Edits on that query type drop away.

Routine, high-volume work generates this kind of signal quickly, which is one reason your first agent should do boring work.

Which controls make feedback trustworthy?

• Required reason codes on rejection, so rejections are never silent.
• Automatic capture of edits, so nobody has to remember.
• Sampling of unreviewed work, so executed actions are checked too.
• Named triage owner with a weekly routine.
• No direct learning. Changes go through change control and testing.
• Closed loop. Each fix is checked against later feedback to confirm it worked.

Feedback is what turns the widen stage of deploying an AI agent into a decision made on evidence rather than opinion.