Fraud analysts learn to supervise an AI agent best through scenarios: the payment the agent held, the customer insisting it be released, the alert it closed too quickly. This plan pairs a short learning path with a scenario log that every analyst must work through in shadow and practicum, then sets sign-off, limits and yearly renewal.
What is this plan for?
Fraud agents score transactions, hold suspicious payments, block cards when patterns match known fraud, draft messages asking customers to confirm activity, and prepare cases for investigation. Blocking a card is protective and reversible, so it often runs at depth 3. Releasing a held payment moves money and is depth 4, held for an analyst. Closing a case as genuine customer activity may run at depth 3 for narrow alert types once proven.
The hardest calls are scams where the customer authorises the payment themselves. The agent holds it; the customer calls, upset, insisting it is genuine; the analyst must decide. That decision cannot be taught from slides. It needs practice on real and realistic cases, which is why this plan is scenario-led. How long that takes is covered in how long it takes to train fraud analysts to supervise AI agents.
How do you fill it in?
Part A is the learning path. Part B is the scenario log: each analyst must meet every scenario type in shadow before the practicum, and the assessor checks them again during it. Add scenario types from your own record; the best ones are cases your team got wrong.
The template
Part A: learning path
| Step | Content | Hours or weeks | Evidence | Signed off by |
|---|---|---|---|---|
| 1 | ||||
| 2 | ||||
| 3 | ||||
| 4 | ||||
| 5 |
Part B: scenario log
| Scenario type | What the agent does | Decision expected from the analyst | Met in shadow (date) | Met in practicum (date) |
|---|---|---|---|---|
A worked example
A hypothetical digital bank's fraud operations team. Values are samples.
Part A
| Step | Content | Hours or weeks | Evidence | Signed off by |
|---|---|---|---|---|
| 1 | Foundations | About 4 hours | Foundations badge | Fraud team lead |
| 2 | Operator modules: how a governed agent decides, human review, when something goes wrong | About 14 hours over three weeks | Module tests in banking fraud | Fraud team lead |
| 3 | Shadow on held payments and alert closures | Two weeks | Scenario log Part B complete | Senior analyst |
| 4 | Written exam and practicum | Two weeks | Exam pass of at least 32 of 40; practicum at least 95 in every 100 decisions correct, no missed red line | Pryme assessor |
| 5 | Sign-off and limit | One day | Certified Pryme Operator; release limit set at 10,000 | Head of fraud |
Part B
| Scenario type | What the agent does | Decision expected |
|---|---|---|
| Customer paying a new payee after a call from "their bank" | Holds the payment; flags new payee and unusual amount | Keep held; call the customer on the number on file; ask open questions; do not release on reassurance alone |
| Customer calls repeatedly demanding release | Records each contact | Recognise coaching by a scammer; escalate to a senior analyst; release only on clear evidence |
| Card blocked after overseas transactions; customer is travelling | Blocks at depth 3; drafts confirmation message | Confirm through the app; unblock with a note |
| Alert closed by the agent as genuine | Closes at depth 3 within bounds | In weekly sampling, re-check the evidence the closure relied on |
| Payee name does not match the account | Holds and shows the mismatch | Treat as high risk; contact the customer before any release |
The second scenario is the one teams fail. A scammer often coaches the victim on what to say, and the victim then pressures the analyst. An analyst who has met this pattern in shadow knows to slow down, ask open questions and escalate. One who has not will release the payment to end an uncomfortable call. Add more practice through how to train fraud analysts to spot prompt injection and pressure, and keep the team aligned on borderline releases with calibration sessions for fraud analysts.
Fraud analysts often work beside credit and onboarding teams, whose plans are the training plan for credit analysts and the training plan for KYC analysts. The team-wide order is in training teams for AI agents. Renewal is yearly, through an update module and a short quiz, plus new scenarios from the past year's record.
Questions people ask
Should the agent ever release a payment it held for fraud?
No. The agent held it because something did not fit. Release is a depth 4 decision for a certified analyst, with the reason recorded.
How do we cover held payments overnight?
Name an on-call certified analyst with a defined response target. Never let a held payment release itself because nobody decided in time.
How often should scenarios be refreshed?
Every quarter, from the record. New fraud patterns show up first as held items your analysts found hard; those become the next scenarios.