Write it from your own cases. A good internal AI agent exam asks people to make the decisions they will make at work: clear or return a held action, tell a correct refusal from a wrong one, name the bound that decided a verdict. Fix the blueprint first, set the pass mark before anyone sits it, and keep a rotating question bank.
What is the exam for, and who owns it?
An internal exam answers one question: can this person be trusted with this decision, at this depth, on this team? It is not a test of whether people know what a model is. It sits between general literacy and the approval rights the business grants.
Ownership splits in two. A subject owner, usually the operations or risk lead for the agent's area, owns the content and signs off every question. Learning and development owns the mechanics: scheduling, invigilation, marking and records. Nobody who writes questions should mark their own candidates' papers.
The output is evidence. A result that cannot be linked to a named blueprint, a question version and a pass mark will not satisfy an auditor, so treat the paperwork as part of the design, not an afterthought.
The steps to design it
- Person: write the blueprint. An exam blueprint lists the areas to be tested and how many questions each gets. For a reviewer exam, weight it towards held actions, refusals and reading the record. The internal AI agent exam blueprint template gives a starting grid.
- Agent, depth 1: pull candidate cases. An agent reads the record and lists recent verdicts by type: allowed, held, refused and withheld, with the bound that decided each.
- Agent, depth 2: draft scenario questions. From those cases the agent drafts scenarios with the customer's data removed and amounts rounded. Each draft is a candidate, not a question.
- Person: edit, verify and sign off each question. The subject owner checks that the correct answer is defensible and that every wrong option is wrong for a reason a reviewer should know.
- Person: set the pass mark and conditions. Decide marks, time and any question that is an automatic fail, such as approving an action that crosses a red line.
- Person: pilot. Run the exam with three or four people already trusted with the work. If they fail a question, the question is usually the problem.
- Person: build the bank. Write at least three versions of every scenario so candidates cannot pass on memory. Retire questions that leak.
- Person: review every 90 days. Agent certificates lapse after 90 days and depths change, so a question about what an agent may do can go out of date.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Blueprint | Nothing | none | Choosing what matters and its weight |
| Case selection | Lists verdicts and deciding bounds from the record | 1 | Choosing representative cases |
| Drafting | Drafts anonymised scenarios and options | 2 | Writing the final question |
| Sign-off | Nothing | none | Approving each question and answer |
| Marking | Scores multiple-choice answers | 1 | Marking written answers and appeals |
| Bank refresh | Flags questions tied to changed depths | 1 | Rewriting and retiring questions |
What should the questions test?
The detail is in what an AI agent exam should test, but four kinds of scenario carry most of the weight.
Held actions. Show the draft, the deciding bound and the context, and ask whether to release, edit or return it. The skill being tested is noticing what the agent could not see.
Refusals and withheld actions. Ask candidates why the agent refused. A refusal because the action sat outside the agent's bounds is fixed by a builder, not overridden by a reviewer. A withheld action is a platform decision for every agent at every depth, and a good candidate knows not to look for a workaround.
Provenance. Give a fact with its source class, sealed, signed, declared or external, and ask what it can support. An emailed instruction is external and cannot authorise a payment.
Depth and certificates. Ask what happens when an agent's run fails. The answer is that it drops to depth 1 the same day and its work arrives as drafts.
Which controls make the result trustworthy?
- Invigilation. For approval rights at depth 3 and above, use a proctored exam. Evidence: the invigilation log and identity check.
- Version control. Every paper has a version tied to the blueprint. Evidence: the paper version on each result.
- Fixed pass mark. Set before the first sitting and recorded. Evidence: the signed blueprint.
- Second marker. Written answers marked twice for a sample. Evidence: both marks on file.
- Practical follow-up. A written pass alone should not grant approval rights at depth 4. Pair it with supervised decisions on live work. Evidence: a practicum log.
How do you set the pass mark?
Set it from the consequence of a wrong answer, not from what feels fair. For reviewers who clear money movement, a high written bar plus a zero-tolerance rule on red lines is reasonable. One sensible pattern: 40 questions in 60 minutes, pass at 32 of 40, with any red-line error failing the paper outright.
Then check the mark against the pilot. If trusted reviewers cluster just above it, it is about right. If they all score full marks, the paper is too easy to tell anyone apart.
Where does it usually go wrong?
Most internal exams test vocabulary. Candidates learn definitions and pass, then freeze at their first refused action. Scenarios fix this.
The second failure is a bank that never changes. After a few cycles the answers circulate, and the pass rate rises while decisions in the record do not improve.
The third is a mismatch between the exam and the approval rights it grants. If the paper covers depth 2 drafting and the business uses it to grant depth 4 approval, the exam is the wrong evidence. Who answers for an agent's action is a choice the business makes, as who answers for an AI agent is a design decision explains, and the exam has to match that choice.
Where the Academy fits
Pryme Intelligence Academy sets module tests in the learner's own industry and role, so a team can use them as a base before adding internal scenarios. The Certified Pryme Operator exam follows the pattern above: 40 questions, 60 minutes, pass at 32 of 40, then a two-week practicum of real review decisions signed off by a Pryme assessor. The wider context is in the guide to AI literacy and certification for people who work with agents.
Questions people ask
Should the exam be open book?
For reviewer exams, partly. People at work can read the policy, so test judgement rather than recall, and allow the policy library while keeping the time limit firm.
How long should an internal exam take?
Long enough for scenarios to be read properly. An hour for 40 questions works for multiple-choice scenarios; written cases need more.
Can a person resit straight away?
Allow a resit after a short gap and some targeted study, with a different version from the bank. An immediate resit on the same paper tests memory, not judgement.