How to teach fraud analysts to bank corrections that improve an AI agent

Fraud analysts bank a correction by tagging why they overturned the agent's alert decision, then routing that reason to the agent owner as a change to the agent's brief, sources or bounds. The rule that matters most: corrections that tighten the agent move quickly, while corrections that loosen it need a second analyst and full re-certification.

Why is banking corrections different in a fraud team?

Because the other side is adapting. In most teams a correction fixes a fixed mistake: the agent misread a policy, and the policy will be the same next month. In fraud, the pattern the analyst corrected today may be gone next week, replaced by a variant built to slip past whatever the team just learned. That changes both speed and caution.

Speed, because a new typology, say a run of mule accounts opened with the same device fingerprint, needs to reach the agent within days. Caution, because every correction that teaches the agent to release something is also, potentially, a map of what fraudsters can now get through. An analyst who clears twenty false positives on a merchant category and asks for the agent to stop flagging it has made a reasonable request and created an opening at once.

The improvement never comes from retraining the model. It comes from the agent's brief, the watchlists and device-risk sources it reads, its bounds, and a fresh certificate. The broader map of who needs which skill is in AI agent skills by role.

What are the steps?

1. Agent, depth 2: scores the alert and drafts a recommendation to hold, release or escalate, with the signals and sources it relied on. A person decides.
2. Person (fraud analyst): overturns or confirms, and on every overturn picks a reason: missed signal, false signal, stale list, unclear rule, or one-off.
3. Person (analyst): marks the direction. Does banking this make the agent stricter or more permissive?
4. Agent, depth 1: clusters overturns daily by reason, direction and pattern, so a new typology surfaces within a shift or two.
5. Person (senior analyst): approves the cluster for banking. Tightening changes need one approver; loosening changes need two, one of whom did not make the original corrections.
6. Person (agent owner): changes the brief, source or bound. A new mule indicator goes into the brief; an updated high-risk merchant list goes into sources; a lower release threshold goes into the customer's limits.
7. Attest replays the agent against your own historic alerts, including confirmed fraud, before the change goes live.

What must a fraud analyst learn?

First, the difference between a false signal and a missed signal. Analysts spend much of the day clearing false positives, so their corrections lean towards loosening. A team that only banks those corrections will drift towards an agent that flags less and less. Teach analysts to record the missed signals too, the cases they caught that the agent scored low, because those are the corrections that keep the agent sharp.

Second, provenance. A tip from a merchant by email is an external claim; it can prompt an investigation but cannot by itself go into the agent's sources as a confirmed indicator. A confirmed chargeback from the card scheme record is a different matter. Analysts should know which class each fact sits in before proposing it as a source.

Third, the limits of banking. Some decisions should never become a rule: an analyst's call to release a payment because they spoke to the customer and recognised the voice is judgement, not a pattern. These stay as one-offs. The review habits underneath are covered in how fraud analysts review an AI agent's decisions.

Which controls matter, and what evidence do they leave?

• Asymmetric approval. Loosening needs two approvers, tightening needs one. Evidence: approver names and direction on each banked change.
• Replay against confirmed fraud. Every change is replayed on known fraud cases as well as false positives, so a loosening that would have released real fraud fails the harm or escalation gate. Evidence: gate scores per case.
• Sources stamped by class. New lists enter as sealed or signed only. Evidence: provenance on every fact.
• A visible depth. If re-certification after a change clears no depth, the agent drops to depth 1 and every alert goes to a person. Evidence: the depth change in the record.

How do you know it is working?

Measure the overturn rate per reason code before and after each banked change. Measure the time from first sighting of a pattern to a live change; in fraud, this is the number that matters most. Measure missed-signal corrections separately from false-signal ones, and watch the ratio: if the team stops recording missed signals, it has stopped looking for them. And sample released alerts monthly to see whether any later turned into confirmed fraud.

Where does it usually go wrong?

Loosening by accumulation is the classic failure. No single change is reckless, but ten small permissive changes over a quarter add up to an agent that waves through a whole category. Review the net direction of banked changes each month.

The second failure is banking a typology after it has moved on. If the change takes three weeks, fraudsters may have switched devices or merchants. Short paths for tightening changes fix this.

The third is analysts using free text instead of reason codes, which makes clustering impossible. Keep the code list short and make it mandatory. Payments teams in markets with fast rails feel this pressure acutely, as fraud analyst training in payments in the Netherlands describes, while KYC teams banking similar corrections at onboarding follow a slower rhythm, covered in teaching KYC analysts to bank corrections.

Where the Academy fits

The Operator track at Pryme Intelligence Academy teaches the review and correction habits fraud analysts need, including modules on teaching the agent and on when something goes wrong. Tests are set in the learner's own industry and role. Certified Pryme Operator includes a two-week practicum of real review decisions where a single missed red line fails the practicum, which suits a team whose worst errors are releases.