How to run an AI agent readiness assessment in an insurer

Run it on claims or underwriting, not both at once. Claims handlers and underwriters meet agents in different ways, and a blended result hides both teams' gaps. Each cohort takes a test set in insurance on held actions, depth, provenance of evidence and the record. The report then decides the first agent, its depth, and what to train before go-live.

What is the assessment for in an insurer?

Insurers are adopting agents in two main places. In claims, agents take first notification of loss, summarise documents, draft reserve recommendations and, within limits, settle small claims. In underwriting, they gather risk information, quote standard risks and refer anything outside appetite. In both, a trained person reviews what the agent holds.

The assessment measures whether that person understands what they are reviewing. The head of claims or the underwriting manager owns it, depending on where the agent starts. The conduct or compliance partner should see the result, because how claims are handled and how risks are priced both affect fair treatment of customers. For the concept, see what a readiness assessment is.

Who should take it?

One team of up to 25, chosen by where the first agent will work.

For a claims agent: claims handlers on the line of business in scope, team leaders who hold payment authority above the handlers', and one or two people from the complaints team.

For an underwriting agent: underwriters and underwriting assistants for the class in scope, the referral desk, and someone who manages broker relationships.

Do not mix the two. Claims handlers will score well on payment questions and poorly on appetite; underwriters the reverse. Averaging them tells you nothing about either.

The steps

1. Person (head of claims or underwriting manager): chooses the team and the candidate agent, and agrees with compliance what the agent will never decide alone.
2. Person (owner): asks for the test set in insurance, with questions about claims, evidence and referrals.
3. Person (each member): sits the test alone.
4. Person (owner and compliance partner): read the report by concept.
5. Person (owner): chooses the first agent and its starting depth. A claims agent that pays is depth 4; most insurers start at depth 2.
6. Person (team leaders): train the weakest concept before the agent drafts live work.
7. Agent, depth 1: reads past closed claims or quotes in shadow and proposes the verdict it would give, for comparison with what the team decided.
8. Person (cohort): reassesses before live drafting.

What does the test usually find in insurers?

Evidence and provenance. Claims run on evidence of mixed quality. The policy administration system is sealed. A repairer's estimate from an approved network is signed. A claimant's description of events is declared. Photos sent by the claimant are external. Handlers know instinctively that these differ; the test shows whether they read the agent's provenance stamps and weigh them correctly, or approve anything that looks complete.

Payment authority and depth. Handlers understand authority limits. What they need to learn is that an agent's depth is earned per agent: an agent may be certified to settle glass claims at depth 4 and only draft on water damage. Teams that assume one depth for the whole agent will misread holds.

Pricing rules. For underwriting cohorts, the test often shows that people do not know which uses of AI attract stricter rules. Under the EU AI Act, risk assessment and pricing for life and health insurance are listed as high-risk. This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.

What does the report change?

It changes the first capability, not just the first agent. A claims team weak on provenance should start with an agent summarising first notifications at depth 2, not one settling claims on photographic evidence.

It changes authority. If team leaders score below handlers, the insurer should not give team leaders sole approval of held payments until they train. Some insurers adopt a rule that approval rights depend on a current reviewer certificate, such as Pryme Intelligence Academy's Certified Pryme Operator.

It changes the referral desk. If referral staff misread refusals, every refused quote will be escalated as a fault.

What evidence should the insurer keep?

The never-decide list agreed with compliance.

The report by concept for each cohort.

The decision on the first agent and depth, with reasons.

The reassessment before live work.

What should the owner measure?

• Scores by concept, by cohort.
• During shadow, cases where the agent's proposed verdict differed from the team's, and which was right on review.
• After go-live, held payments approved without any change, checked by sampling.

Where does it go wrong in insurers?

Blending claims and underwriting. The result looks reasonable and helps nobody.

Assuming experience equals readiness. Senior handlers often score lowest on provenance, because they rely on judgement built over years and skim the labels.

Forgetting delegated authority. Where brokers or third parties handle claims or bind risks, their staff may also review agent output. They need assessing too.

Banks and fintechs face the same evidence questions in different forms; see the bank version and the fintech version. The readiness assessment report template and what a readiness assessment measures cover the report itself. The topic guide is training teams for AI agents.