How to run calibration sessions for credit analysts

Credit analysts calibrate by deciding the same referred applications alone, each with a decision and the reasons they would give the applicant. Use cases where evidence is mixed or the policy leaves room. Compare decisions and reasons, agree a gold standard with the credit policy owner, feed it into the agent's brief, and sample decisions over the following month.

Why do credit analysts need calibration?

A credit agent typically gathers application data, checks it against lending policy, and drafts a credit summary with a recommendation at depth 2. It refers to an analyst anything outside policy, anything with weak evidence, and anything near a threshold. The analyst decides.

Those referred cases are where analysts' own habits show. One accepts two months of bank statements for a self-employed applicant; another wants two years of accounts. One declines at a debt-to-income ratio just over the line; another approves with a lower limit. Applicants with the same facts get different answers. And because creditworthiness assessment of individuals is treated as high-risk under the EU AI Act, and declines must carry specific reasons in many markets, inconsistency is a governance problem, not just a fairness one. Calibration addresses it directly. The general method is in how to run calibration sessions for AI agent reviewers.

The head of credit or the credit policy owner sets the standard. A senior analyst facilitates.

Which applications belong in the set?

Eight to ten referred applications from the past month, anonymised:

• Self-employed income supported only by bank statements the applicant uploaded.
• A thin credit file with strong current income.
• A ratio just over the policy threshold with a strong repayment history.
• Income declared on the form that differs from the figure in data received directly from the applicant's bank.
• A policy exception request from a relationship manager for a valued customer.
• A decline the agent drafted, to test whether analysts agree with the reasons it gave.

The session, step by step

1. Person (senior analyst): selects and anonymises the applications.
2. Person (each analyst): decides each alone, recording the decision, any conditions, and the reasons they would give the applicant.
3. Agent, depth 1: shows its recommendation, the policy rules applied and the provenance of each figure: declared on the form, signed by a known data source, external as an uploaded document.
4. Person (facilitator): reveals the spread on decisions, then on reasons.
5. Person (analysts): explain their reasoning, naming the evidence each relied on.
6. Person (credit policy owner): sets the gold standard, or refers a policy gap to the credit committee.
7. Person (credit policy owner): sends the standard to the brief owner, including how the agent should word reasons.
8. Person (quality assurance): samples referred decisions for a month.

What does a gold standard look like for credit?

It sets the evidence and the reasons. For example: "Self-employed income supported only by uploaded bank statements is treated as external evidence. Analysts may approve up to a set limit with twelve months of statements; above it, filed accounts are required."

On reasons: "A decline states the specific factor that failed, in plain words, such as 'monthly commitments are too high relative to verified income', not a code. The agent drafts reasons in this form, and the analyst confirms each is true from the record."

On mismatches: "Where declared income differs from income in bank data by more than a set margin, the bank data is used and the difference is noted."

Which controls does calibration support, and what evidence does it leave?

Consistent decisions. Evidence: spreads over successive sessions.

Specific, true reasons. Evidence: the reasons standard and sampled declines against it.

Evidence rules. Evidence: the standard on what each provenance class supports.

Agent alignment. Evidence: brief changes and approvers.

What should the head of credit measure?

• Agreement on decisions for shared applications.
• Sampled declines whose reasons could not be traced to the record.
• Approval rates on referred cases by analyst, which should become more similar.
• How often analysts change the agent's drafted reasons.

Where does it go wrong in credit teams?

Calibrating decisions but not reasons. Two analysts decline for different stated reasons. The applicant, and any reviewer, sees inconsistency.

Exceptions without a standard. Policy exceptions for valued customers are where drift starts. Put one in every session.

Not changing the agent. Analysts agree a standard and the agent keeps drafting reasons in the old form. Every analyst then rewrites them, differently.

Ignoring provenance in the discussion. Analysts argue about whether an applicant can afford the loan when the real disagreement is about which income figure to trust. Ask each analyst to name the source of every figure they used; the split usually becomes obvious within minutes.

Credit controllers and fraud analysts calibrate on related but different calls; see calibration for credit controllers and for fraud analysts. For planning, see the AI agent training plan template for credit analysts, and the topic guide on training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's Operator track covers human review and teaching the agent, which is where a calibrated standard becomes a change to the agent's brief. Its written exam is 40 questions in 60 minutes, with a pass mark of 32 of 40, and the practicum is two weeks of real review decisions. A lender can make approval rights on referred cases depend on a current certificate.