How to run calibration sessions for claims handlers

Pick ten claims the agent held for a handler, strip out who decided them, and have every claims handler decide each one alone: pay, decline, query or refer. Compare the spread, argue the disputed claims on cover and evidence, and agree a gold standard. Pass it to the agent's brief owner and sample settled claims for a month.

What does calibration fix in a claims team?

A claims agent usually takes first notification, gathers documents, checks cover against the policy wording, proposes a reserve, and drafts a settlement or a decline at depth 2. Some insurers let it settle small, clear claims of certified types at depth 4 within their own limits. Anything unusual is held for a handler.

Handlers decide those holds with a mix of policy wording, evidence and judgement about fairness. Two experienced handlers can read the same exclusion differently, or accept different evidence of loss. Customers then get different outcomes for the same facts, which is a fair treatment problem as much as a consistency one. And the agent learns from whichever handler corrected it last. The general method is in how to run calibration sessions for AI agent reviewers.

The claims team manager owns the sessions. A technical claims specialist facilitates. Someone from complaints attends every second session.

Which claims belong in the set?

Eight to twelve claims settled or declined in the past month:

• An ambiguous exclusion: water damage that might be gradual seepage or a sudden escape.
• Evidence of mixed provenance: the claimant's photos (external), a repairer's estimate from the insurer's network (signed), and the policy record (sealed).
• A fraud indicator: a claim shortly after cover started, with inconsistent dates.
• A vulnerable customer: a claimant who mentions a recent bereavement or illness.
• A claim just above the agent's settlement limit, to test whether handlers check the same things the agent would.
• A claim the agent settled at depth 4, to test whether handlers agree it should have been.

The session, step by step

1. Person (claims team manager): selects and anonymises the claims.
2. Person (each handler): decides every claim alone, with outcome, reserve, reason and the evidence relied on.
3. Agent, depth 1: shows its proposed outcome and reserve, the policy clauses it applied and the provenance of each piece of evidence.
4. Person (facilitator): reveals the spread, first on outcome, then on reserve.
5. Person (handlers): explain their reasoning, especially on wording and evidence.
6. Person (claims team manager, with the technical specialist): agrees the gold standard. Where the wording itself is unclear, it goes to the product owner.
7. Person (claims team manager): sends the standard to the brief owner, including any change to which claim types the agent may settle.
8. Person (quality assurance): samples handled and agent-settled claims for the next month.

What does a gold standard look like for claims?

It ties a decision to wording and evidence. For example: "Where a water damage claim relies only on the claimant's photos and description, it is held for a handler, who requests a repairer's or surveyor's report before deciding between sudden escape and gradual seepage. The claimant's photos are external evidence and are not enough to decline or to settle on."

A fairness standard might read: "Where a claimant mentions bereavement, illness or financial hardship, the handler calls rather than writes, and deadlines for documents are extended."

Write a reserving standard too, because outcomes can agree while reserves do not. For example: "For escape of water claims held for a surveyor's report, the initial reserve is the repairer's estimate where one exists, and the agent's proposed reserve otherwise, reviewed when the report arrives." A standard like this gives the agent a rule it can apply when it proposes a reserve, and gives handlers a reason to change the proposal only when the evidence differs. Without it, every handler adjusts the agent's reserve by their own feel, and the record shows a pattern of corrections nobody can explain.

Which controls does calibration support, and what evidence does it leave?

Consistent outcomes. Evidence: the spread of independent decisions over successive sessions.

Reserving. Evidence: the spread of reserves and the agreed approach.

Fair treatment. Evidence: standards for vulnerable customers and complaints input.

Agent scope. Evidence: the recorded decision on which claim types the agent may settle at depth 4.

What should the claims team manager measure?

• Agreement on outcome and reserve by session.
• Complaints upheld on claims decided after calibration, compared with before.
• Agent-settled claims that handlers would have decided differently. Any is a reason to review the agent's scope.
• How often handlers change the agent's proposed reserve, and by how much.

Where does it go wrong in claims teams?

Calibrating outcomes, not reserves. Reserves drive the insurer's financial picture. Handlers who agree on outcome can still reserve very differently.

Excluding agent-settled claims. If nobody checks them in calibration, depth 4 settlement drifts without scrutiny.

Arguing wording without the product owner. Handlers can agree on a reading the wording does not support. Send unclear wording to the owner.

Other reviewers calibrate on different questions: collections agents on affordability and tone, IT administrators on access risk. For planning, see the AI agent training plan template for claims handlers and how long it takes to train claims handlers to supervise AI agents. The topic guide is training teams for AI agents.