How to run calibration sessions for collections agents

Collections agents calibrate on three things at once: the arrangement they would offer, the words they would use, and when they would stop collecting altogether. Give the team ten arrears cases the AI agent held, have each person decide alone, then compare and agree a written standard. Send it to the brief owner and sample live accounts for a month.

Why does collections need calibration?

A note on words first. In this article, "collections agents" are the people who work arrears; the AI agent is "the agent".

In most collections teams the agent sends routine reminders at depth 3 within the lender's or supplier's own limits, proposes payment arrangements inside a standard affordability range, and drafts letters for anything else at depth 2. It holds cases with signs of hardship, disputes, or vulnerability for a person. Collections agents decide those.

Those are exactly the cases where judgement varies most. One collections agent offers a three-month pause to anyone who mentions losing a job; another asks for evidence first. One writes warmly; another quotes the contract. Customers get different treatment by luck, and the agent learns contradictory corrections. Calibration is how the team converges. The general method is in how to run calibration sessions for AI agent reviewers.

The collections team leader owns calibration. A senior collector facilitates. Someone from the vulnerable customers or hardship team attends every session.

Which cases belong in the set?

Eight to ten accounts held in the past fortnight, anonymised:

• A customer who says they have lost their job and offers a token payment.
• A customer disputing the debt, where collection activity should pause until the dispute is resolved.
• A customer who mentions a serious illness in passing while asking for more time.
• A broken arrangement on its second failure.
• A third party calling on the customer's behalf, whose authority is declared, not verified.
• A case the agent refused to contact because a vulnerability flag was set, to check collectors read the refusal correctly.

The session, step by step

1. Person (team leader): selects and anonymises the cases.
2. Person (each collector): decides each case alone: the arrangement, the next contact, the message, and whether to pause collection, with a reason.
3. Agent, depth 1: shows its proposed action and the bound that held the case, for example the hardship trigger in the business's own limits.
4. Person (facilitator): reveals the spread on the decision, then reads two or three of the messages aloud.
5. Person (collectors): explain their reasoning, especially about affordability evidence and tone.
6. Person (team leader, with the hardship lead): agrees the gold standard for decision and tone.
7. Person (team leader): sends the standard to the brief owner, including changes to the agent's letter templates and hardship triggers.
8. Person (quality assurance): samples live accounts for a month.

What does a gold standard look like in collections?

It covers the action and the words. For example: "Where a customer reports job loss, offer a 30-day pause on contact and a call to complete an income and expenditure review. Do not ask for documents in the first contact. The customer's account of their situation is declared and is recorded as such."

On tone: "Messages name the amount owed once, state the options plainly, and include how to get free debt advice. No message mentions further action in the same paragraph as an offer of help."

On stopping: "A dispute pauses all collection activity until a person resolves it. The agent's bounds hold any account flagged as disputed."

What does a session look like in practice?

Take a hypothetical team of nine collectors at a consumer lender. On the job-loss case, four offered a 30-day pause, three asked for payslips before agreeing anything, and two proposed a reduced payment of 20 a month. In discussion, the three who asked for payslips explained that a previous customer had falsely claimed redundancy. The hardship lead pointed out that asking for documents in the first contact puts many honest customers off replying at all. The team agreed the pause-then-review standard above, the brief owner changed the agent's hardship template the same week, and the next session showed seven of nine collectors deciding a similar case the same way.

Which controls does calibration support, and what evidence does it leave?

Consistent treatment. Evidence: the spread of decisions over successive sessions.

Hardship handling. Evidence: the standard approved by the hardship lead.

Tone. Evidence: the agreed wording and sampled messages against it.

Agent alignment. Evidence: changes to templates and triggers, with approvers.

What should the team leader measure?

• Agreement on decisions, session by session.
• Sampled messages meeting the tone standard.
• Accounts with a hardship or dispute flag that received collection contact. The target is none.
• Complaints about collections contact, before and after calibration.

Where does it go wrong in collections teams?

Calibrating arrangements but not words. Customers remember how they were spoken to.

Leaving the hardship team out. Collectors agree on a standard the hardship team would not accept.

Using calibration to raise collection rates. It is about consistency and fair treatment. Performance targets belong elsewhere, or people will hide their real decisions.

Underwriters and risk managers calibrate on different kinds of judgement; compare calibration for underwriters and for risk managers. For planning, see the AI agent training plan template for collections agents, and the topic guide on training teams for AI agents.