An exam blueprint fixes, before anyone writes a question, what the exam covers, how many questions each topic gets, what type those questions are, and what counts as a pass. For an internal AI agent exam, add one rule most blueprints lack: a short set of red-line questions that must all be answered correctly, whatever the overall score.
What is the blueprint for?
It stops the exam drifting towards whatever is easiest to ask. Left alone, question writers produce recall items ("what does depth 2 mean?") because they are quick to write. Reviewers of held actions need judgement items ("here is a held refund, its deciding bound and its sources; do you release it?"). The blueprint fixes the mix in advance. The general idea is in what an exam blueprint is.
It also makes the exam defensible. If an auditor asks why someone was allowed to approve held payments, you can show the blueprint, the version of the exam built from it, and the person's score against it.
When should you write one?
Before the first question is drafted, and again whenever an agent's depth changes or a new agent reaches the team. If a new capability means reviewers now release payments rather than read drafts, the weights must move. For the wider design process, see how to design an internal AI agent exam.
How do you fill it in?
- Topics. Start from what the role does with agents, not from a syllabus. For guidance, see what an AI agent exam should test.
- Questions per topic. Weight by consequence: the topics closest to money or hard-to-undo actions get the most questions.
- Question types. Single-answer, multiple-answer, scenario with a case file, and short written reasoning.
- Cognitive level. Recall, apply or judge. Aim for most questions at apply or judge.
- Must-pass items. Mark the questions where a wrong answer would mean a missed red line in live work.
- Pass rule. State it as a count and include the must-pass rule.
- Conditions. Time, open or closed book, and whether it is a proctored exam.
The blueprint template
| Topic | Questions | Question types | Level | Must-pass items | Source material |
|---|---|---|---|---|---|
| How a governed agent decides: depth and the five bounds | |||||
| Reading a verdict: allowed, held, refused, withheld | |||||
| Provenance: sealed, signed, declared, external | |||||
| Releasing or returning held actions | |||||
| Refusals and withheld actions: what not to override | |||||
| Certificates: lapse, failed runs, depth drops | |||||
| Teaching the agent: corrections and knowledge updates | |||||
| When something goes wrong: escalation and incidents | |||||
| Our own limits and red lines | |||||
| Total |
Exam conditions
| Field | Entry |
|---|---|
| Number of questions | |
| Time allowed | |
| Pass rule | Score as a count, plus all must-pass items correct |
| Conditions | Open or closed book; proctored or not |
| Retake rule | Waiting period; new question set |
| Blueprint version and owner | |
| Review date |
A worked example
Take a sample payments team whose agent repairs failed payments at depth 3 and releases held payments at depth 4 for a named person. Values are illustrative.
| Topic | Questions | Types | Level | Must-pass |
|---|---|---|---|---|
| Depth and the five bounds | 3 | Single-answer | Recall, apply | 0 |
| Reading a verdict | 4 | Scenario | Apply | 1 |
| Provenance | 5 | Scenario | Judge | 2 |
| Releasing held payments | 7 | Scenario with case file | Judge | 3 |
| Refusals and withheld actions | 3 | Scenario | Apply | 1 |
| Certificates and depth drops | 2 | Single-answer | Recall | 0 |
| Teaching the agent | 2 | Short written | Apply | 0 |
| Escalation and incidents | 2 | Scenario | Judge | 1 |
| Our limits and red lines | 2 | Multiple-answer | Apply | 2 |
| Total | 30 | 10 |
Conditions: 30 questions, 45 minutes, closed book, invigilated in the office. Pass rule: 24 of 30 or better, and all 10 must-pass items correct. A candidate scoring 27 of 30 who releases a payment to beneficiary details changed by email fails.
Provenance gets five questions here because, in this team, an external claim such as an emailed change of bank details is the most likely route to a loss. The exam must also be re-checked when the payment agent's own certificate is renewed every 90 days, as the auditing angle in keeping internal auditors ready for an agent's 90-day recertification shows.
Where the Academy fits
The Certified Pryme Operator exam from Pryme Intelligence Academy is a written exam of 40 questions in 60 minutes with a pass mark of 32 of 40, followed by a two-week practicum. Module tests are set in the learner's own industry and role. Many teams use an internal exam like the one above for local limits and red lines, alongside an external certificate for the general skill. See AI literacy and certification for people who work with agents.
Questions people ask
How many questions should an internal exam have?
Enough to cover each topic at least twice and every red line at least once. For one team and one or two agents, 25 to 40 questions is usually enough.
Should the pass mark be the same for every role?
The count can be similar, but the must-pass items should differ. A reviewer of drafted letters and an approver of payments face different red lines.
How often should the question bank change?
Rotate questions at every sitting and rewrite the bank when the agent's depth, limits or sources change. A blueprint version number makes the link visible.