How to run calibration sessions for compliance officers

Compliance officers calibrate by reviewing the same items an agent held, deciding each alone, and arguing out the ones where they split. Marketing approvals, breach assessments and complaint classifications make good material. The head of compliance sets the gold standard, which goes into the agent's brief, and sampling over the next month shows whether officers now decide alike.

Why do compliance officers need calibrating?

Compliance teams increasingly use agents to do the first pass on routine reviews. A typical agent reads a draft marketing piece against the firm's rules and drafts comments at depth 2. It reads incoming complaints and proposes a classification. It reads incident reports and drafts an assessment of whether a breach may be reportable. Officers decide every one.

Second-line judgement varies more than people admit. One officer passes a promotion that says "low risk"; another requires the risk warning to be moved up. One classifies a complaint about a delay as a service issue; another as a potential conduct issue. Business teams learn which officer to send things to. And the agent, corrected by all of them, cannot converge. The general method is in how to run calibration sessions for AI agent reviewers.

The head of compliance owns the standard. A senior officer facilitates. Legal attends when a case turns on interpretation of a rule.

Which items belong in the set?

Eight to ten items from the past month, with business and officer names removed:

• A marketing piece with a benefit stated prominently and a risk stated in small print.
• A social media post a business team drafted quickly, with no risk wording.
• A complaint that mentions both a delay and a misleading statement.
• An incident report about a short system outage that delayed some customer payments.
• A policy exception request from a business team under time pressure.
• An item the agent refused because it fell outside its bounds, such as a request to approve a promotion for a product it has no source rules for.

The session, step by step

1. Person (senior officer): selects items and removes names.
2. Person (each officer): decides every item alone, with the outcome, the rule relied on and any required change.
3. Agent, depth 1: shows its drafted comments or classification, the rules it relied on and their provenance, for example the firm's own policy library (sealed) against a business team's description of the product (declared).
4. Person (facilitator): reveals the spread, item by item.
5. Person (officers): explain their reasoning, naming the rule each relied on.
6. Person (head of compliance): sets the gold standard. Points of interpretation go to legal.
7. Person (head of compliance): sends the standard to the brief owner and to business teams that submit work.
8. Person (quality assurance): samples decisions for a month.

What does a gold standard look like for compliance?

It names the rule and the line. For example: "Any promotion that states a benefit in the headline must state the main risk with equal prominence, in the same view, without scrolling." Or: "A complaint that alleges the customer was told something untrue is classified as a potential conduct issue, whatever else it mentions, and is held for an officer."

A breach standard might read: "Any incident that delayed customer payments is assessed by an officer within one working day for whether it must be reported. The agent's draft assessment lists facts and their provenance only; it does not conclude."

How is calibrating second-line staff different?

Compliance officers set standards for others, so inconsistency among them spreads further than inconsistency in a single queue. A promotion approved by one officer becomes the template business teams copy.

They are also less used to being calibrated. Senior officers may resist having their reasoning compared. The facilitator should present spreads without names, and the head of compliance should take part as a participant before setting the standard. Linking calibration to compliance training more broadly helps it feel like professional practice rather than an audit of the team.

Which controls does calibration support, and what evidence does it leave?

Consistent second-line decisions. Evidence: spreads over successive sessions.

Interpretation. Evidence: the standard, with legal's view where sought.

Business guidance. Evidence: the standard circulated to business teams.

Agent alignment. Evidence: brief changes and approvers.

What should the head of compliance measure?

• Agreement on outcomes, session by session.
• Business teams resubmitting work to a different officer after a rejection, which should fall.
• Sampled decisions that break a standard.
• How often officers rewrite the agent's draft comments.

Where does it go wrong in compliance teams?

The head of compliance sitting out. If the person setting the standard never decides the cases alone, the team never learns where they differ from the head.

Calibrating only marketing. Promotions are easy material. Breach assessments are harder and matter more.

Not telling the business. A standard only compliance knows about leads to more rejections, not better submissions.

Loan officers and procurement officers calibrate on decisions close to the business; see calibration for loan officers and for procurement officers. For planning, see the AI agent training plan template for compliance officers, and the topic guide on training teams for AI agents.

Where the Academy fits

Compliance officers who oversee agents across several regulated business lines may be candidates for Pryme Intelligence Academy's Architect track, which ends in a board review with two Pryme assessors and is valid for 24 months. Officers who review held items daily fit the Operator track, which covers running the rhythm of review, calibration included.