How to measure how much reviewers change an AI agent's drafts

Keep two versions of every draft: what the agent produced and what the person sent. Compare them, then class each change as none, cosmetic, substantive or rewritten, and record why for substantive edits. Report by draft type and reviewer. Then check the unchanged drafts were actually read, because a low edit rate can mean good drafts or tired reviewers.

Why measure edits at all?

At depth 2 the agent drafts and a person sends. The edits that person makes are the most direct evidence you have about the agent's work: each one is a real reviewer, on a real case, saying "this was not right". The argument for treating this as the main quality measure for drafting work is made in reviewer edit rate is the honest measure of a draft. This playbook is the method.

The process owner keeps the decisions: what counts as substantive, what edit rate would justify moving a draft type to depth 3, and what to do about reviewers whose patterns look wrong. A quality analyst can run the numbers. The agent can compute most of them.

What are the steps?

1. Agent, depth 1: stores the draft as produced, with its draft type, sources and the time it was offered to the reviewer.
2. Agent, depth 1: stores the version sent, the reviewer's name and the time it was sent or discarded.
3. Agent, depth 1: computes the difference between the two at word level, ignoring whitespace and formatting changes your templates apply automatically.
4. Agent, depth 2: proposes an edit class for each draft: none, cosmetic (spelling, punctuation, word order with the same meaning), substantive (a fact, figure, commitment or required statement changed, added or removed), or rewritten (most of the draft replaced, or discarded).
5. Reviewer: confirms the class and gives a reason for substantive and rewritten edits, from a short fixed list: wrong fact, missing information, wrong tone, compliance wording, wrong customer or case, other.
6. Quality analyst: samples unchanged drafts each week and checks them against the procedure, to test whether "no edit" meant "no problem".
7. Agent, depth 2: drafts the weekly report: edit classes by draft type and by reviewer, the reasons, and the sample results.
8. Process owner: decides whether any draft type is ready for depth 3, needs more training cases, or should be withdrawn.

Why classify instead of counting characters?

A raw difference score treats a corrected interest rate and a changed greeting the same. A reviewer who fixes a figure has caught an error that would have reached a customer. A reviewer who swaps "Dear" for "Hello" has expressed a preference. Only the first tells you about the agent's quality.

The reason codes do the second half of the job. If most substantive edits are "missing information", the agent's sources are incomplete. If they are "compliance wording", its brief is missing a required statement. If they are "wrong customer", something is badly wrong upstream and the draft type should be stopped. Each reason points to a different fix.

How do you know unchanged drafts were read?

This is the hard part. When drafts are usually good, reviewers stop reading closely and approve what they see. The tendency is well known, and the EU AI Act's human oversight provision names it directly; the implications for people who approve drafts are covered in Article 14 automation bias and the people who approve agent drafts.

Three checks help. First, the weekly sample of unchanged drafts: if the sample finds errors that reviewers passed, the edit rate is understating the problem. Second, time spent: a reviewer who approves long drafts in a few seconds each is unlikely to be reading them. Third, known test cases: with the reviewers' agreement, include a small number of drafts with a known, deliberate flaw and see whether they are caught. These are removed before sending, and the results are used for coaching, not discipline.

What should the record contain?

For each draft: the draft text, the sent text, the difference, the class, the reason, the reviewer, the time taken, and whether it was in a quality sample. For each week: the report and the process owner's decision.

Over a quarter, this becomes the evidence for widening or narrowing depth. It also feeds the 90-day review set out in measuring an AI agent's work in its first 90 days, and the sampling results sit naturally beside the broader methods in measuring an AI agent's accuracy.

What should you report?

• Share of drafts sent unchanged, cosmetic, substantive and rewritten, by draft type.
• Substantive edit reasons, ranked.
• Errors found in the sample of unchanged drafts. This is the check on everything else.
• Edit patterns by reviewer. A reviewer who never edits, or always rewrites, needs a conversation.
• Review time per draft. Review is a real cost; what it costs to run an AI agent includes the people as well as the model.

Where does it usually go wrong?

Drafts are edited outside the system. Reviewers copy the text into an email and edit it there. The sent version is never captured. Make sending go through the tool that stores both.

Classification is left to the agent. The agent's proposed class is a starting point. Without reviewer confirmation, a confident misclassification turns substantive edits into cosmetic ones.

One reviewer dominates. If one person reviews most drafts, the edit rate measures them as much as the agent. Spread the work, or report by reviewer.

The target is a number. Teams set "under one in ten drafts substantively edited" and stop looking. The reasons matter more than the rate, and a single serious edit can outweigh a good month. Keep reading the reasons, as train, shadow, certify, widen asks of every widening decision.

Where Pryme fits

In Pryme, every held draft is filed with the agent's version and the version a person released, so the difference is part of the record rather than a separate exercise. Edit evidence feeds Attest's next certification run for that capability, alongside replayed cases.