An exam blueprint is the written plan for an exam: which topics it covers, how many questions or marks each topic gets, what kind of task tests each one, and at what level of difficulty. For exams on AI agent oversight, the blueprint decides whether the test measures recall of terms or the decisions reviewers actually make.
Why does an exam blueprint matter?
Without one, exams drift towards what is easy to write. Definitions are easy to test, so exams fill up with "what does depth 3 mean?" and run short on "this payment was held; what do you check first?". People pass, and the business learns little about whether they can review.
A blueprint fixes the weights before anyone writes a question. It also makes the exam defensible: when a regulator or auditor asks why a credential means anything, the blueprint shows what was tested and why.
What does an exam blueprint contain?
A working blueprint for an agent oversight exam has these columns:
| Area | Weight | Task type | Level |
|---|---|---|---|
| How a governed agent decides (depth, bounds, verdicts) | 20 marks of 100 | Multiple choice | Recall and apply |
| Reviewing held actions | 35 marks | Scenario: approve, edit, decline or pass up | Apply and judge |
| Provenance of sources | 15 marks | Scenario: which fact can be relied on | Judge |
| Incidents and stopping an agent | 15 marks | Short sequence task | Apply |
| Reading the record | 15 marks | Find the bound that decided | Apply |
The blueprint also sets the pass mark, the time limit, whether the exam is open-book, and any red-line items that fail the candidate regardless of score.
What does a blueprint look like in practice?
Take a mid-sized insurer writing an internal exam for claims reviewers. The first draft has 30 questions, 22 of them definitions. The blueprint forces a rebalance: at least half the marks must come from scenarios using held settlements from the insurer's own record, and one red-line item, approving a payment based on an external claim, fails the exam outright.
An internal exam blueprint template gives a starting layout, and how to design an internal agent exam covers writing the items.
How is an exam blueprint different from a syllabus?
A syllabus lists what is taught. A blueprint lists what is tested and how heavily. They should match, but often do not: a syllabus can spend hours on background that the blueprint gives few marks, which is fine, and the blueprint can weight decisions heavily that the syllabus covers too briefly, which is a problem.
Reading the two side by side is a quick check on any course, internal or bought in.
Which related terms should you know?
- Open-book exam: a format where candidates may use reference material, which suits tests of judgement.
- Time-boxed exam: an exam with a fixed time limit, which tests fluency under pressure.
- Item writing: the craft of writing each question so it tests what the blueprint says.
For what belongs in the blueprint in the first place, see what an agent exam should test and the topic guide on training teams for AI agents.
Questions people ask
Should an exam blueprint be shared with candidates?
Yes, at the level of areas and weights. It helps people prepare for the right things. Keep the items themselves private.
How often should the blueprint change?
When the product, the agent's depth or the type of decisions reviewers make changes. A blueprint written for depth 2 drafting will under-test the judgement needed at depth 4.