Measure against a baseline taken before go-live, in three phases: daily checks in the first two weeks, weekly quality sampling to day 45, then outcome and cost measures to day 90. The point is a decision. On day 90 the process owner renews, widens, narrows or retires each capability, and the record shows the evidence behind that call.
Why 90 days?
Ninety days is long enough to see a full month-end, a peak and a quiet spell, and short enough that problems have not become habits. It also matches the life of a certificate: in our model, an agent's certification for a capability lapses after 90 days and must be renewed on fresh evidence. The first 90 days are the first full cycle, and what you measure in them becomes the case for the second.
The process owner keeps the decisions. That is the operations lead whose team the agent works in, not the project team that deployed it. The measurement work can be shared with an analyst or the agent itself, but the conclusions belong to the person who answers for the process.
What do you need before day one?
A baseline. For the same work done by people in the previous quarter: volume, time to complete, error rate from quality checks, rework, complaints, and cost per item. Without it, every number after go-live is unanchored.
A sampling plan. How many items a reviewer checks each week, chosen at random, plus every held and refused item.
The decision criteria. Write down, before the numbers exist, what would justify widening a capability and what would justify narrowing it. Criteria written afterwards tend to fit the numbers.
Which work it is. If the agent's first job is routine and high in volume, the measures are clear and the sample is large, which is one reason your first agent should do boring work.
What is the method, phase by phase?
- Days 1 to 14, person and agent: daily review. The process owner or a delegate reviews every refused and held item, and a daily random sample of allowed items. The first week has its own routine, set out in running an AI agent's first week in production.
- Agent, depth 1: produces the daily figures from its own verdict records: items handled, verdicts by type, deciding bounds, held queue size and age.
- Days 15 to 45, reviewer: weekly quality sample. A fixed number of allowed items is checked against the procedure. Errors are classified by severity.
- Agent, depth 1: compares the weekly figures with the baseline and flags any measure that has moved beyond the thresholds you set.
- Days 46 to 90, process owner: outcome measures. Complaints, rework downstream, customer contacts about the agent's work, and cost per item including review time.
- Agent, depth 2: drafts the 90-day report from the records. The process owner edits and signs it.
- Day 90, process owner: decides for each capability: renew at the same depth, widen, narrow, or retire.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Daily review | Produces figures | 1 | Reviewing held and refused items |
| Weekly sample | Draws the random sample | 1 | Judging quality |
| Baseline comparison | Flags movements | 1 | Interpreting them |
| Outcome measures | Gathers complaint and rework data | 1 | Linking cause |
| 90-day report | Drafts | 2 | Editing and signing |
| Decision | Nothing | none | Renew, widen, narrow, retire |
Which measures matter most?
Quality on allowed work. The error rate on items the agent completed without a person, from the random sample. This is the measure that decides depth. How to define an error and what accuracy means for different work is covered in measuring an AI agent's accuracy.
The held queue. Size, age, and how often the person approves unchanged. A queue that people approve without reading is a control that has stopped working.
Refusals by bound. If one bound decides most refusals, it may be set too tight, or the agent is being asked to do work outside its role.
Downstream effects. Rework in the next team, complaints, corrections. These arrive late, which is why they belong in the last phase.
Cost per item, including model use, platform, and the review time of people. The method for turning this into return is in measuring the ROI of an AI agent.
What record does the 90 days produce?
A single report per agent, signed by the process owner, containing: the baseline; the daily, weekly and outcome figures; every severe error and its follow-up; the decision for each capability and the reason. Attach the sample lists, so an auditor can re-check any figure.
This report is the evidence for renewing the certificate. It is also the document a new process owner reads to understand why the agent works the way it does. Treat it as part of the deployment record: train, shadow, certify, widen.
Where does it usually go wrong?
The baseline is missing or flattering. Teams often discover that no one measured human error rates before. Run a quick sample on past work before go-live, even if small.
Volume is mistaken for success. An agent that handles many items badly creates rework that shows up weeks later. Quality comes first.
Sampling stops after the first month. Quality sampling is tedious and is the first thing to go when the team is busy. Protect it in the reviewer's time.
The decision is never made. Day 90 passes, the agent keeps running, and no one renewed anything. Put the decision meeting in the calendar on day one.
Averages hide the worst case. An error rate that looks fine across all work can hide one item type where the agent is often wrong. Break the figures down by item type.
Questions people ask
What if the agent performs worse than people?
Then the decision on day 90 is to narrow or retire, and the report shows why. A worse result is useful evidence; the only waste is not acting on it.
Should the agent be measured on speed?
Speed matters only once quality is established. Fast wrong work costs more than slow right work.
Who should do the weekly sampling?
Someone who does the work, not someone who built the agent. They know what a good outcome looks like and have no stake in the agent looking good.