How to run calibration sessions for risk managers

Risk managers calibrate by rating the same cases independently and then explaining their scores. Choose eight to twelve items the agent drafted or held, such as incident severities, control exceptions and risk acceptances. Collect each manager's rating and reason beforehand, discuss the largest gaps, agree a gold standard, and change the agent's brief to match.

What are risk managers calibrating, exactly?

Risk work runs on ratings. Likelihood and impact, severity of an incident, whether a control exception is tolerable, whether a key risk indicator breach needs escalation. Two experienced risk managers can look at the same event and score it a 2 and a 4, both with good reasons. Once an agent starts drafting those ratings, the gap matters more: the agent receives contradictory corrections, and the risk register fills with scores that depend on who reviewed them.

A risk agent typically reads incident reports, control test results and indicator feeds, drafts a classification and a summary, proposes a rating against the risk appetite statement, and routes anything above a threshold to a named manager. It works at depth 1 and 2 almost entirely. Risk acceptance is a decision that stays with an accountable person, so calibration is about how people use the agent's drafts, not about letting it decide.

The chief risk officer, or the head of operational risk, owns calibration. The facilitator should be someone respected for consistency rather than seniority.

How do you run it?

1. Agent, depth 1: compiles candidate cases from the period, each with its draft rating, the sources used and the reviewer's final rating.
2. Facilitator: selects the pack. Include at least one near-miss incident, one indicator breach, one control exception and one third-party issue. Remove reviewer names.
3. Each manager: rates alone before the session, giving likelihood, impact and a short reason, and naming the evidence relied on.
4. Agent, depth 1: plots the spread for each case.
5. Group: discusses the widest spreads first. Ask which part of the appetite statement each person applied and which evidence moved them.
6. Head of risk: confirms a gold standard for each disputed case. If the appetite statement is ambiguous, the gap goes to the risk committee secretary for the next committee.
7. Agent owner and builder: update the brief with the reasoning, add the cases to the certification set, and re-certify.

Which cases should risk managers calibrate on?

Near misses. An erroneous payment was caught before release. Managers often rate impact as zero because nothing was lost; others rate the potential impact. The gold standard should say which the register records and why.

Indicator breaches with an explanation attached. A breach arrives with a business-line note saying it is a one-off. That note is declared, not verified. Managers must agree how much weight a declared explanation carries against sealed data from the system of record.

Control exceptions. A reconciliation was not performed for two days during a system change. Tolerable, or a finding? The answer depends on compensating controls, which managers should name.

Third-party incidents. A supplier reports an outage. Is it your incident, theirs, or both? Agree the classification rule.

Agent-originated events. The agent refused an action it should have taken, or a held item waited too long. Risk managers need a shared view of how to rate the agent's own failures, including a certification run that cleared no depth and dropped the agent to depth 1.

Which controls make the practice credible?

• Independent ratings first. Evidence: submissions timestamped before discussion.
• Anonymised spread. Evidence: case numbers only in the session pack.
• Appetite gaps escalated, not resolved in the room. Evidence: the referral to committee and its outcome.
• Agent changes re-certified before they apply. Evidence: the certification run.
• Gold standards versioned. Evidence: each entry carries the confirmer and the appetite statement version it rests on.

An internal auditor should be able to pick any rating in the register and trace it to a gold standard, or to a documented reason for departing from one.

What should you measure?

• Spread on shared cases, which should narrow across sessions.
• Re-ratings in sampling. A second manager re-rates a weekly sample; fewer changes means the standard is holding.
• Agent draft acceptance by risk type. If managers keep changing the agent's incident severity drafts but not its control exception drafts, you know where the brief needs work.
• Escalation timeliness. Calibrated managers escalate faster on the cases that should go up and slower on the ones that should not.

Where does it usually go wrong?

Calibration becomes a debate about the framework. The session is for applying the appetite statement, not rewriting it. Park framework questions for committee.

Seniority settles the score. The head of risk speaks last, not first.

Only incidents get calibrated. Teams calibrate severity and forget acceptances and exceptions, where judgement varies most.

The agent is left behind. Without the agent owner present, the standard never reaches the brief.

Our explainer on what calibration is covers the underlying idea, and the general method for any reviewer is in how to run calibration sessions for AI agent reviewers. For a contrasting case mix, see calibration sessions for operations managers. The wider programme is set out in training teams for AI agents. Pryme Intelligence Academy's Operator track teaches calibration as part of running the rhythm of a live workspace.