Decisions, not definitions. A good AI agent exam puts candidates in front of realistic cases, an agent's held action with its record, and asks what they would do and why. It tests whether they find the deciding bound, check where facts came from, give a usable reason, and never miss a red line.
Why do definition questions fail?
A candidate can know that depth 4 means moving money or taking actions that are hard to undo, and still approve a held payment built on an emailed invoice. Knowing the term and applying it under pressure are different skills. Employers tie approval rights to the exam result, so the exam has to predict the thing they are relying on: good review decisions.
Definition questions also age badly. They reward memorising the course. Scenario questions reward judgement, which carries across product changes.
What should the exam cover?
| Area | What a question looks like | What it reveals |
|---|---|---|
| Reading the record | Here is a held action and its record. Which bound held it? | Whether the candidate can find the cause before judging |
| Provenance | The agent relied on these four facts. Which could support approval? | Whether they know a sealed fact from an external claim |
| Verdicts | The agent refused this request. Should anyone override it? | Whether they know refused from withheld, and who can change what |
| Approve, edit or decline | Here is a draft. Choose an action and write the reason | Whether their reason would stand up in a complaint |
| Red lines | A case that must never be approved, hidden in a routine batch | Whether they catch it every time |
| Escalation | A case beyond the candidate's authority | Whether they know when to stop |
| Change and drift | The agent's behaviour shifted after a source change | Whether they notice and report |
Red-line items deserve special treatment. Most exams weight them like any other question. For agent reviewers, missing a red line should fail the candidate outright, because in live work that miss is the incident.
How should questions be written?
Set them in the candidate's industry and role. A fraud analyst should see card disputes and held payment releases; a claims handler should see settlement offers and reserve changes. The same principle tested in an unfamiliar setting tests reading speed, not judgement.
Include the record, not a summary of it. Real reviewers read real records, with noise and missing fields. Give them that.
Ask for the reason as well as the choice. A correct verdict with a weak reason is a near miss. Where the exam is marked automatically, offer reasons as options and mark the pair.
Plan the mix with an exam blueprint, a table of areas, weights and question counts. The internal AI agent exam blueprint template gives a starting layout, and how to design an internal AI agent exam covers the build.
An example: one question, done well
Take a question for accounts payable reviewers. The candidate sees a held supplier payment. The record shows the invoice arrived by email, the bank details differ from those on the supplier master file, and the agent held the payment under the customer's own limits for any bank detail change. The options are: approve, because the invoice matches the purchase order; edit the amount; decline and request verification through the supplier contact on file; or approve and flag for later review.
Only the third is correct, and the reason matters: the email is an external fact and cannot authorise a payment to new details. A candidate who chooses the first has shown a red-line gap, whatever else they score.
Is a written exam enough?
No. A written exam shows judgement on prepared cases in a quiet room. Review in a live queue adds volume, interruptions and ambiguity. A practical period of real decisions, assessed by someone else, closes that gap. If stakes are high, supervise the written part as a proctored exam. Auditors check both parts, as described in how auditors test that AI agent reviewers are qualified. The wider structure is in the guide to AI literacy and certification for people who work with agents.
Where the Academy fits
The Certified Pryme Operator assessment in Pryme Intelligence Academy follows this shape: a written exam of 40 questions in 60 minutes, then a two-week practicum of real review decisions where a single missed red line fails the candidate, signed off by a Pryme assessor. Module tests along the way are set in the learner's own industry and role.
Questions people ask
How many questions should an agent exam have?
Enough to cover each area in the blueprint at least twice and include several red-line items. For a reviewer exam, 30 to 60 questions is a sensible range.
Should candidates use the real workspace during the exam?
For the practical part, yes. For the written part, give them realistic record extracts so the test is fair and repeatable.
How often should the question bank change?
Whenever the product or the agent's scope changes, and on a regular cycle so questions do not circulate.