Fraud analysts calibrate on held transactions: release, block, or contact the customer. Give each analyst the same set of cases, timed at real working pace, and have them decide alone. Compare decisions and the signals each relied on, agree a gold standard with the fraud manager, update the agent's bounds, and sample the next month's releases.
Why does fraud need its own approach to calibration?
A fraud agent typically scores transactions, releases low-risk ones at depth 3 within the business's own limits, and holds the rest with a summary for an analyst. Analysts decide whether to release, block, or contact the customer, and whether to restrict the account.
Three things make fraud calibration different. Decisions are fast: an analyst may have a minute per case, so calibration should test decisions at realistic pace, not after leisurely debate. There is an adversary: patterns change, so a standard that was right last quarter can be wrong now. And the cost of error runs both ways: releasing fraud loses money; blocking a legitimate payment loses a customer's trust and sometimes their rent payment. The general method is in how to run calibration sessions for AI agent reviewers.
The fraud manager owns the sessions. A senior analyst facilitates. Someone from customer operations attends quarterly, since they field calls from blocked customers.
Which cases belong in the set?
Ten closed cases from the past fortnight, outcome known, analyst names removed:
- A first payment to a new payee, large for the customer, at an unusual hour.
- Several small card payments just below the agent's release limit at the same merchant category.
- Incoming payments from many senders followed by an immediate transfer out, a possible mule pattern.
- A payment the customer confirmed by phone, where the call itself may have been coached by a fraudster.
- A false positive that was blocked and caused a complaint.
- A transaction the agent released at depth 3 that later turned out to be fraud.
The session, step by step
- Person (senior analyst): selects closed cases with known outcomes and removes analyst names.
- Person (each analyst): decides each case alone with a time limit of two minutes, recording the decision and the one or two signals that decided it.
- Agent, depth 1: shows its score, summary and the bound that held each case, with the provenance of each signal: sealed account history, signed device data, declared customer statements.
- Person (facilitator): reveals the spread, then the actual outcome.
- Person (analysts): explain the signals they relied on, especially on cases they got wrong.
- Person (fraud manager): agrees the gold standard, including any change to the business's own limits.
- Person (fraud manager): sends the standard to the brief owner, with any bound changes, such as a cap on releases per merchant category per hour.
- Person (quality assurance): samples releases and blocks for a month.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Select cases | Nothing | 3 on releases | Choosing cases with known outcomes |
| Timed decisions | Nothing | 3 | Release, block or contact |
| Agent's summary | Shows score, signals, provenance | 1 | Comparing |
| Reveal outcomes | Nothing | 3 | Learning from errors |
| Gold standard | Nothing | 3 | Fraud manager |
| Update bounds | Applies new limits | 3 | Brief owner |
| Sampling | Releases and holds live traffic | 3 | QA |
Why reveal the real outcome?
Most calibration agrees a standard where the right answer is a matter of policy. In fraud, many cases have a known answer: it was fraud or it was not. Revealing it after the timed decisions shows analysts which signals they over- or under-weight. An analyst who blocked the false positive learns which signal misled them; one who released the mule pattern learns what they missed.
Be careful not to let hindsight become the standard. A case that turned out to be legitimate may still have deserved a hold given what was known. The standard should describe what the analyst should do with the information available at the time.
What does a gold standard look like?
For example: "A first payment to a new payee above a set amount, made at an unusual hour for the customer, is held for a call to the customer on a number already on file. Analysts ask open questions about the payee, not 'did you make this payment?', because a coached customer will say yes."
On mule patterns: "Several incoming payments from unrelated senders followed by an outgoing transfer within an hour leads to a hold on the outgoing payment and a referral to financial crime."
Which controls does calibration support, and what evidence does it leave?
Consistent decisions at pace. Evidence: timed decisions and spreads over sessions.
Learning from outcomes. Evidence: the discussion notes on missed and false-positive cases.
Bound changes. Evidence: changes to the business's own limits, filed with approver.
Customer impact. Evidence: complaints linked to blocks, reviewed quarterly.
What should the fraud manager measure?
- Agreement on timed decisions across analysts.
- Fraud released by the agent at depth 3 that analysts would have held.
- False-positive blocks per analyst, which should become more similar.
- Time from a new pattern appearing in calibration to a bound change.
Where does it go wrong in fraud teams?
Untimed sessions. Analysts decide well with ten minutes and poorly with one. Calibrate at working pace.
Only fraud cases in the set. Without false positives, analysts learn to block more and customers pay for it.
Standards that never expire. Revisit each standard every quarter. The adversary will have moved.
Credit and KYC analysts calibrate on slower decisions with different evidence; see calibration for credit analysts and for KYC analysts. For planning, see the AI agent training plan template for fraud analysts, and the topic guide on training teams for AI agents.
Questions people ask
How often should fraud analysts calibrate?
Every two weeks, because patterns change quickly. Short sessions of 30 to 45 minutes work better than long monthly ones.
Should the agent's release limit be discussed openly?
Within the team, yes. Analysts need to know it to spot clustering just below it.
Can calibration lower the agent's depth?
It can lead the fraud manager to narrow the agent's limits, which is a bound change. A fall in depth itself follows from a failed certification run, not from a session.