Compare the full cost of the work before the agent with the full cost after it, for one process, over the same volume. The usual mistake is counting the agent's output as saved time while ignoring the people who check, correct and redo it. Reviewer time, rework and incidents belong on the cost side.
What has to be in place before you can measure it?
A baseline. Before the agent touches the process, record how many cases arrive, how long each takes end to end, how many are reworked, and what errors cost when they slip through. Without this, every later number is a guess dressed as a result.
Second, a defined scope. ROI is measured per capability, not per agent. An agent that drafts replies and also classifies incoming mail is doing two jobs with different economics. Lumping them together lets a good capability carry a poor one.
Third, a fixed depth. The return from an agent at depth 2, where a person sends every draft, is different from the same agent at depth 3. Measure at the depth the agent actually holds, not the one you hope it will earn.
What goes on each side of the calculation?
| Benefits you can count | Costs you must count |
|---|---|
| Hours of handling time no longer spent by people | Model usage and platform fees |
| Shorter time from request to resolution | Integration and maintenance of connections |
| Fewer errors reaching customers or the ledger | Reviewer time on held and drafted work |
| Work done outside staffed hours | Rework when a person rewrites a draft |
| Capacity absorbed without new hires | Certification runs and recertification every cycle |
| Time spent on incidents and rollbacks |
The right column is where most business cases go thin. A reviewer who spends four minutes fixing a draft that saved six minutes of writing has turned a ten-minute task into a two-minute saving, not a ten-minute one. That is why the reviewer edit rate is the most honest measure of a draft: it shows how much of the agent's work survives contact with a person.
How do you run the measurement in practice?
- Operations lead: record the baseline for four to six weeks of normal volume.
- Agent, depth 1 or 2: run in shadow or draft mode. Time saved is not yet claimed; you are measuring quality.
- Analyst: log reviewer minutes per item and track how much reviewers change the agent's drafts.
- Finance partner: cost the agent's run, including the people around it, monthly.
- Process owner: compare against baseline at the same volume and mix, and decide whether to widen, hold or stop.
Volume and mix matter. If the agent handles easy cases and leaves hard ones for people, average handling time for people goes up. That is not a failure of the people; it is a sign that the saving is smaller than the headline.
What does an honest result look like?
Say a mid-sized insurer's claims team handles 2,000 first-notice emails a month at eight minutes each, about 270 hours. An agent drafts acknowledgements and classifies each claim at depth 2. Reviewers spend two minutes per item checking and sending, about 65 hours. Rework adds 10 hours. The agent's running costs and the share of a platform licence come to the equivalent of 30 hours of staff cost.
Net, the team releases roughly 165 hours a month. That is real, but well short of the 270 hours a vendor slide would show. It also tells the team where to look next: if the edit rate falls, reviewer time falls with it, and that is the lever worth pulling.
Which controls keep the ROI figure honest?
- Count only verified work. Output a person discarded is a cost, not a saving.
- Keep the depth in the label. Report "ROI at depth 2". It stops a figure from one depth being quoted for another.
- Include the failure cost. If a certification run that clears no depth drops the agent to depth 1 for a week, that week is part of the return.
- Re-baseline after process changes. If you redesigned the process to suit the agent, some of the gain belongs to the redesign.
ROI sits next to quality, not above it. Our guide to measuring an AI agent's accuracy covers the quality side, and the 90-day measurement plan shows when each number becomes reliable. The full deployment sequence is in deploying an AI agent.
Questions people ask
How soon can you claim a return?
Not before the agent has held a stable depth for a full cycle of normal volume, usually a couple of months. Early numbers are dominated by set-up effort and reviewer caution, which both fall over time.
Should we count avoided hires as savings?
Only where the volume growth is real and documented. Capacity you would not have needed anyway is not a saving. Record the assumption so the board can challenge it.
Does a higher depth always improve ROI?
Usually, because review time falls. But the cost of an error rises with depth, so the gain only holds if certification evidence supports the wider depth.