Inter-reviewer agreement is a measure of how often different reviewers reach the same decision on the same cases. If two reviewers each decide the same twenty held actions and match on eighteen, their agreement is high. If they match on twelve, it is low, and the outcome a customer receives depends on who happened to pick up their case.
Why does it matter for agent oversight?
Because human review is only a control if it is consistent. A held payment, refund or account change is waiting for a person precisely because the business does not want the agent to decide alone. If reviewers would decide it differently, the business has replaced one uncertain decision-maker with several.
Agreement also matters for the agent itself. Reviewer decisions are the evidence used to judge whether an agent is right and to improve it. If reviewers disagree, that evidence is noisy, and corrections banked from one reviewer may contradict another's. An agent tested against inconsistent reviewers cannot be fairly scored.
How is it measured?
The simplest way:
- Pick a set of cases. Ten to thirty recent held actions, including some hard ones.
- Have each reviewer decide them alone. Approve, correct or decline, with a reason.
- Count matches. For each pair of reviewers, how many cases did they decide the same way?
- Look at where they split. The cases with disagreement matter more than the overall score.
More formal statistics exist that adjust for agreement by chance, and they are useful for large teams. For most operations, a simple match count on a regular sample, tracked over time, tells the team lead what they need.
An example
Take a hypothetical payroll bureau. Three reviewers check held payroll changes the agent cannot make alone, such as salary changes above a set amount. The team lead gives all three the same fifteen cases. Two agree on fourteen; the third agrees with them on only ten. Her reasons show why: she approves backdated pay rises without checking for a signed approval from the client's HR contact, treating the client's email as enough. That is a coaching point with a clear answer, and the next sample shows her aligned. A structure for that conversation is in the reviewer coaching plan template.
How is it different from accuracy?
Agreement measures whether reviewers decide alike. Accuracy measures whether they decide correctly, against a gold standard or the policy owner's ruling. Reviewers can agree perfectly and all be wrong, for example if the whole team has misread a policy in the same way. Both are needed. Agreement is cheap to check often; accuracy needs an agreed right answer.
It is also not the same as a service level agreement, despite the shared word. An SLA is a commitment about service, such as how quickly held actions will be decided; the internal service level agreement template for AI agents covers that. Inter-reviewer agreement is about the consistency of the decisions themselves.
How do teams improve it?
Through calibration: regular sessions where reviewers decide the same cases, compare and agree the standard. Unclear policies get clarified, gaps get passed to the policy owner, and individual habits get coached. The role being measured is set out in what an AI agent reviewer is, and the guide to AI agent skills by role shows where reviewers fit.
Questions people ask
What level of agreement is good enough?
It depends on the stakes. For held payments and red-line cases, aim for near-complete agreement; for tone or wording, more variation is acceptable. Watch the trend as much as the level.
Should the agent be included as a reviewer?
You can compare the agent's proposals with reviewers' decisions, which shows where the agent and people differ. But the agent's verdicts are what reviewers are checking, so it should not be counted as one of them.