Reviewer edit rate is the honest measure of a draft

An approval count says a person let a draft go, not whether they rewrote half of it first. How often reviewers change an agent's drafts, and what they change, is the most direct evidence of draft quality you have. A falling rate of substantive edits is a reason to widen; a rising one is an early warning.

Why do approval counts mislead?

Because "approved" covers very different events. A reviewer who reads a draft and sends it unchanged has approved it. So has a reviewer who corrects the refund amount, rewrites the explanation and removes a sentence that admitted liability, then sends it. Both count as one approval. Reported together, they suggest the agent is producing usable work when much of it needed a person to fix it.

Approval counts also rise when review weakens. A tired or overloaded reviewer approves more, not fewer. A team whose agent's approval figure is climbing may be seeing better drafts or worse review, and the count alone cannot say which.

The edit record separates these. It shows not just whether the draft was released, but how much of it survived.

What should you actually measure?

Three things, together.

How often drafts are edited at all. The simplest measure: of the drafts approved, how many were changed before release.

What kind of edit was made. Not all edits matter equally. Classify each one.

How large the edit was. A one-word change and a full rewrite are both edits. Measuring the size, even roughly, separates light touch-ups from drafts that were essentially replaced.

Our playbook on measuring how much reviewers change an AI agent's drafts covers how to collect these without making reviewers do extra work.

What does the edit pattern tell you?

Read it by capability, because each capability has its own quality. An agent may draft balance enquiries that nobody touches and complaint responses that are substantively rewritten every time. Averaging them hides both.

Substantive edits on a capability mean the agent should not be widened on it, however high its approval count. Factual edits clustering on one kind of fact point to a source problem. Removals clustering on one kind of statement point to a gap in the agent's brief or bounds. Cosmetic edits, unless very frequent, are mostly noise, and some may reflect individual reviewers' tastes rather than any fault in the draft.

Over time, the trend matters most. A capability whose substantive edit rate falls steadily over many weeks, on a stable mix of cases, is building evidence for widening. One whose rate jumps after a model change or a source update has regressed, and should be re-tested.

Why is edit rate better evidence than accuracy scores?

Accuracy scores, typically produced by testing the agent against cases with known answers, are essential for certification. But they measure the agent against the test, not against the people who actually use its work. Our guide on measuring an AI agent's accuracy covers how to build those tests.

Edit rate measures the agent against the reviewers' standard on live work, every day, at no extra cost. It catches problems the test set did not anticipate. It is also the measure that most directly reflects the cost of the agent: every substantive edit is reviewer time spent fixing work rather than checking it, which matters for any calculation of the ROI of an AI agent.

It is also more meaningful than measures borrowed from customer service. Containment rate, the share of contacts handled without a person, says nothing about whether the handled ones were handled well. Edit rate does.

When does edit rate itself mislead?

When reviewers stop reading. A reviewer who is rubber-stamping will make no edits, and a low edit rate will look like excellent drafts. Edit rate must therefore be read alongside review time and a regular second review of a sample of approved items. If drafts sail through unchanged and second reviewers find problems, the edit rate is measuring the reviewer, not the agent.

It can also mislead when reviewers edit to taste. If one reviewer rewrites every greeting, their cosmetic edits will swamp the signal. Classifying edits by type and looking at each reviewer's pattern helps separate preference from correction. Calibration sessions, where reviewers compare their edits on the same drafts, help reduce the variation.

How should edit rate be used in the first months?

As the main evidence for whether a drafting capability is ready to widen. In the first 90 days of an agent's life, most of its capabilities will be at draft, and edit rate is the richest signal you have. Our guide on measuring an AI agent's work in its first 90 days places it alongside the other early measures. A capability whose drafts are consistently released with only cosmetic edits, on a stable mix of real cases, confirmed by second review, has a strong case for moving to execution within bounds.

What is the trade-off?

Capturing edits properly needs the review tool to keep the agent's draft and the released version side by side, and some way of classifying the difference. That is more work than counting approvals. Reviewers may also feel watched, and the measure has to be framed as evidence about the agent, not about them.

Approval counts are easier and say much less. If your reporting shows how many drafts were approved, add how many were changed, what kind of change was made and how large it was, by capability. Decisions about widening will rest on firmer ground. The deployment approach this evidence feeds is set out in our guide to deploying an AI agent: train, shadow, certify, widen.

Where Pryme fits

In Pryme, when a named person approves a held draft, the agent's version and the released version are both filed in the append-only record, along with the time taken. That lets edit patterns be read per capability and set alongside Attest results when deciding whether a capability's depth should be widened.