Proof of real work beats a multiple-choice score

A multiple-choice score shows a person can recognise the right answer when it is printed in front of them. Proof of real work shows they can find it when nothing is printed. For people who approve what AI agents prepare, the second matters. Exams have a place, but certification should rest on observed decisions in a live workspace.

What does a multiple-choice test actually measure?

A well-written multiple-choice exam measures recall and recognition across a wide range of topics. It can check that someone knows what each depth means, can name the four verdicts, understands provenance classes and knows what the record contains. It is cheap to run, consistent across candidates and easy to mark. For breadth, it is hard to beat.

What it cannot measure is the moment of judgement. On a test, the question frames the problem: "An agent has held a payment because the bank details came from an email. What should the reviewer do?" The candidate picks the option that says "verify through a trusted channel". In real work, nobody frames the problem. The held payment sits in a queue with eleven others, the reason is shown in a field the reviewer may or may not read, and the customer has phoned twice. The skill is noticing, not choosing.

That is why high exam scores and poor review decisions coexist so often. The exam tested whether the person knew; the job tests whether they look.

What counts as proof of real work?

Work-based assessment means judging a person on decisions they made in the job, or in conditions close enough to it that the same skills apply. For reviewers of agent work, that means a practicum: a defined period of real review decisions in a live workspace, scored by an assessor against a standard.

Three features make it strong evidence:

1. It is unframed. The person decides without being told what kind of problem they face.
2. It includes red lines. Some errors are not averaged out. Approving a payment that a refusal should have stopped, or closing an alert without checking a missing source, fails the practicum regardless of everything else.
3. It is signed off by someone independent. An assessor who is not the person's manager checks the decisions and records the outcome.

The record helps here. Because every verdict and every approval is filed, an assessor can reconstruct exactly what the person saw and what they did.

Where does the difference matter most?

In work where the right answer depends on context the test cannot supply. Financial crime is the clearest case. Under the FATF risk-based approach, good AML work depends on asking the right questions about a particular customer's risk; training AML staff who work with AI agents is about building that habit. A multiple-choice exam can test whether an investigator knows the red flags. Only observed casework shows whether they spot the missing counterparty in an agent's tidy draft.

That is the core of the case for certifying AML investigators and the parallel case for certifying fraud analysts. In both roles, the agent's draft looks complete, and the person's value is in finding what it lacks.

It also matters as deployments grow. When several agents hand work to each other, a reviewer may see an item that one agent prepared and another passed on. Understanding how multiple agents work together safely is something you can test in theory; tracing a real handoff to find where an assumption crept in is something you can only prove in practice.

Why not rely on practice alone?

Because practice has gaps of its own. In two weeks of real work, a reviewer may never meet some important situations. A rare type of withheld action, an unusual escalation route or a failure mode that occurs once a quarter may simply not come up. A written exam covers those systematically.

The strongest certification uses both. The exam checks breadth: does the person know everything they should? The practicum checks depth: do they apply it when it counts? Neither alone is enough, but if you had to choose one for people who approve agent actions, choose the practicum.

What is the trade-off?

Proof of real work is expensive. It needs assessors, time and real cases. It is harder to standardise: two assessors may score the same borderline decision differently. And it is slower to scale than an online exam that thousands can sit in a week.

There is also a fairness question. A candidate whose practicum falls in a quiet fortnight sees fewer hard cases than one whose falls during a surge.

These are real costs, and they can be reduced. Clear scoring rules, with a defined pass threshold and explicit red lines, make assessment more consistent. Assessors can sample rather than review every decision. Practicum periods can be long enough to see a reasonable spread of cases. What cannot be reduced away is the core trade: a cheaper test that measures less, against a dearer one that measures what you actually need to know.

What should you do differently?

If you certify, or rely on certificates for, people who approve agent work:

• Ask how the certificate was earned. If the answer is a multiple-choice test alone, treat it as evidence of knowledge, not of judgement.
• Require observed practice before granting approval rights at depth 3 or 4.
• Insist on red lines that fail a candidate outright, rather than averages that hide them.
• Keep the practicum record as evidence, alongside the exam result.
• Use the exam for breadth, the practicum for proof.

The guide to AI literacy and certification for people who work with agents explains how the two fit together. The Operator certificate from Pryme Intelligence Academy, for example, requires both a written exam and a two-week practicum signed off by an assessor.