Fraud analysts review an agent's decisions in both directions: what it stopped, to see whether genuine customers were blocked, and what it let through, to see whether fraud slipped past. For each decision they read the record, find the deciding bound, test the signals against the firm's own data, and decide with a reason.
Why review both directions?
Fraud work has two kinds of error, and an agent tuned against one tends to drift into the other. If reviewers only look at held items, they see blocks and releases but never the fraud the agent allowed. If they only look at confirmed fraud, they never see the customers whose rent payments were stopped for no reason. Both cost money and trust. A review routine that covers both keeps the agent honest.
What are the steps?
- Read the verdict and the bound. Allowed, held, refused or withheld. For holds, which bound decided: the firm's own limits, a source that was unavailable, or the agent's role scope.
- List the signals. Device, location, payee history, amount, velocity, session behaviour. Note their provenance: the firm's own systems are sealed; third-party risk data may be signed; customer statements are declared.
- Test the signals. Recompute what you can from the source systems. A signal the agent saw but misread is a different problem from one it never saw.
- Check the outcome. For blocks: was the customer genuine? For allowed transactions: was fraud later reported?
- Decide open items. Release, amend, block or escalate.
- Record the reason, naming the signal.
- Log patterns for the agent's owner and the rules team.
What does a look-back review add?
| Look-back | What it finds | What to do |
|---|---|---|
| Allowed transactions later reported as fraud | Signals the agent missed or weighted too low | Report to the agent's owner; consider narrowing its depth 3 scope |
| Blocks later confirmed genuine | Over-sensitive signals | Review limits for that pattern |
| Holds released by analysts that became fraud | Analyst judgement or scam coaching | Retrain; tighten release scripts |
| Holds declined by analysts that were genuine | Over-caution | Review guidance |
| Holds caused by unavailable sources | Data outages | Fix resilience with the source owner |
What may the agent do, and what stays with the analyst?
The agent may score, enrich, block high-confidence card fraud at depth 3, and draft cases and messages. The analyst decides releases of held payments, confirms fraud, unblocks, refunds, and anything that links customers into a case. The agent never decides a customer is a fraudster; it finds evidence for a person to weigh.
An example: a pattern in the allowed transactions
Take an e-commerce marketplace. A fraud analyst runs a weekly look-back on orders the agent allowed that later became chargebacks. Most are scattered. One cluster stands out: orders for gift cards using newly created accounts with delivery to a parcel locker, each just under the amount the agent holds.
The record shows the agent saw every signal and scored each order below its threshold. The amounts were deliberately chosen. The analyst declines the open orders matching the pattern, records the reason, and raises it with the agent's owner, who adds the combination as a held pattern and re-certifies the agent. Practice in two markets is in AI agent training for fraud analysts in e-commerce in India and in Australia.
Which controls make review reliable?
- Look-back reviews run weekly on both allowed and blocked decisions.
- Release decisions are sampled daily by a senior analyst.
- Reasons name the signal and the verification.
- Patterns go to the agent's owner through change control, not ad hoc edits.
Training a team to review this way is covered in how to train the fraud team to review AI agent decisions. Neighbouring roles review differently; see how KYC analysts review an AI agent's decisions and how credit analysts do. The role map is in AI agent skills by role.
Where the Academy fits
Pryme Intelligence Academy's Operator track covers how a governed agent decides, human review and when something goes wrong, with tests set in the learner's own industry and role. A business can make release rights on held payments depend on a current Certified Pryme Operator certificate, so analysts whose certificate lapses prepare cases until they renew.
Questions people ask
Who should run the look-back review?
A senior analyst or a fraud quality team, independent of the analysts whose releases are being reviewed.
How do we stop fraudsters learning the agent's thresholds?
Vary limits where policy allows, watch for amounts clustering just below them, and treat that clustering as a signal.
Should analysts tune the agent themselves?
No. They provide evidence. Changes go through the agent's owner and re-certification, so the record shows what changed and why.