How to run calibration sessions for KYC analysts

KYC analysts calibrate on the onboarding files an agent held: documents of doubtful provenance, name mismatches, ownership chains and higher-risk customers. Each analyst decides the same files alone, approve, request more, or escalate, and names the deciding evidence. Compare, have the head of financial crime set the gold standard, and update the agent's rules on what it may clear.

Why do KYC analysts need calibration?

A KYC agent typically reads identity documents, checks them against registries and screening lists, and drafts a customer due diligence decision. Many firms let it clear low-risk individual customers at depth 3 where every check passes on strong evidence. Everything else is held for an analyst.

Due diligence under the FATF Recommendations is risk-based, which means analysts exercise judgement. One accepts a mobile phone bill as proof of address; another wants a bank statement. One treats a transliterated name as a match; another wants a second document. Customers wait longer with some analysts than others, and the agent's corrections pull in different directions. The general method is in how to run calibration sessions for AI agent reviewers.

The head of financial crime or the onboarding manager owns the standard. A senior analyst facilitates.

Which files belong in the set?

Eight to ten onboarding files from the past fortnight, with customer details masked where possible:

• Proof of address as an uploaded utility bill with a slightly different address format.
• A name mismatch between passport and application: a transliterated surname, or a married name.
• An identity document expiring next month.
• A small company with two layers of holding companies between the applicant and the ultimate owner.
• A customer from a higher-risk jurisdiction with a clean screening result.
• A screening hit on a common name with a different date of birth.
• A file the agent cleared at depth 3, to test whether analysts agree.

The session, step by step

1. Person (senior analyst): selects files and masks unnecessary personal data.
2. Person (each analyst): decides each file alone: approve, request more evidence, or escalate for enhanced due diligence, naming the evidence relied on.
3. Agent, depth 1: shows its checks and the provenance of each fact: a company registry extract (sealed or signed, depending on the source), a verified identity check (signed), the applicant's stated details (declared), an uploaded document (external).
4. Person (facilitator): reveals the spread, starting with ownership and name cases.
5. Person (analysts): explain their reasoning.
6. Person (head of financial crime): sets the gold standard.
7. Person (head of financial crime): sends the standard to the brief owner, including any change to what the agent may clear at depth 3.
8. Person (quality assurance): samples approved and agent-cleared files for a month.

What does a gold standard look like for KYC?

It sets the evidence and the threshold. For example: "An uploaded utility bill is external evidence. It is accepted as proof of address only where the identity check is signed by a verified provider and the address matches after standard formatting. Otherwise, request a second document."

On names: "A transliterated surname is accepted where date of birth and document number match the verified identity check. A married name needs a supporting document."

On ownership: "Every ownership layer is traced to a natural person through a registry extract. The applicant's own ownership chart is declared and does not substitute for registry evidence."

On screening hits: "A hit with a different date of birth is discounted only with the reason recorded; the agent may not discount hits itself."

What does a session look like?

Take a hypothetical team of ten analysts at a payments firm. On the two-layer company, six approved on the applicant's ownership chart, four traced it through registry extracts. One of the holding companies was registered in a jurisdiction where the registry did not show owners. The head of financial crime set the standard that an unverifiable layer leads to enhanced due diligence. The brief owner changed the agent so it holds any company file where a layer cannot be traced to a registry, instead of drafting an approval.

Which controls does calibration support, and what evidence does it leave?

Consistent due diligence. Evidence: spreads over sessions.

Evidence rules by provenance. Evidence: the standard, in the onboarding procedures.

Agent clearance scope. Evidence: the recorded decision on what the agent may clear.

Screening decisions. Evidence: recorded reasons for every discounted hit.

What should the head of financial crime measure?

• Agreement on shared files.
• Agent-cleared files that analysts would have held. Any is a reason to narrow its scope.
• Time to onboard by analyst, which should become more similar.
• Discounted screening hits without a recorded reason. The target is none.

Where does it go wrong in KYC teams?

Only individual customers in the set. Company files are harder and drift more.

Calibrating speed, not evidence. Faster onboarding is a result of consistent standards, not the aim of the session.

Never testing the agent's clears. Depth 3 clearance needs regular scrutiny.

Credit and fraud analysts calibrate on related evidence questions; see calibration for credit analysts and for fraud analysts. For planning, see the AI agent training plan template for KYC analysts and how long it takes to train KYC analysts to supervise AI agents. The topic guide is training teams for AI agents.