An open-book exam is an assessment in which candidates may consult reference material, such as policies, notes or product documentation, while they answer. For people who oversee AI agents, it suits the real work, because reviewers always have the policy and the record to hand; what matters is whether they can apply them to a held action under time pressure.
Why does an open-book exam suit agent oversight?
A closed-book exam tests memory. Reviewers of agent decisions rarely need to recall a rule from memory; they need to find the relevant fact, read the verdict line, check provenance and decide. An open-book exam tests that.
It also removes a false signal. A candidate who memorises the definitions of the four verdicts can pass a closed-book quiz and still freeze on a real item. In an open-book exam, the definitions are freely available, so the only way to score is to use them correctly.
How does an open-book exam work in practice?
The design is different from a closed-book test:
- Scenario items. Each question presents a held action, with its draft, sources and verdict line.
- Realistic materials. Candidates get the same policy, approver rule and job aid they would have at work.
- A time limit. Long enough to look things up, short enough that someone who has to look everything up runs out.
- Judgement over lookup. Questions ask what to do and why, not what a page says.
- Red-line items. A wrong answer on one fails the exam whatever the score.
What does an open-book exam look like in a real team?
Take a wealth manager's operations team reviewing client instructions drafted by an agent. The exam gives each candidate the firm's instruction policy and twelve held items in 45 minutes. One item: a client asks to move funds to a new account by email. The policy is on the candidate's desk. The test is whether they find the rule on new payees and see that an emailed instruction is external, so cannot authorise the payment alone.
What an agent exam should test and how to design an internal agent exam cover the wider choices.
How is an open-book exam different from a closed-book exam?
A closed-book exam tests what candidates can recall. An open-book exam tests what they can do with what is in front of them. Closed-book suits core vocabulary that must be instant, such as recognising a red line. Open-book suits judgement on realistic cases. Many good assessments combine a short closed-book section with a longer open-book one.
An open-book exam is not easier. Without a tight time limit and scenario items, it can be; with them, it is often harder, because it exposes candidates who know the words but not the work.
Which related terms should you know?
- Exam blueprint: the plan that sets whether each section is open or closed book.
- Time-boxed exam: an exam with a fixed limit, which every open-book exam should be.
- Practicum: assessment on live work, which goes beyond any exam.
For the full approach, see training teams for AI agents.
Questions people ask
Can candidates use the agent itself during an open-book exam?
Only if the exam is testing how they use it. For a reviewer exam, give them the materials a reviewer has, not a tool that answers the question for them.
How long should an open-book exam be?
Long enough for a good reviewer to finish with a little spare time. Pilot it on experienced reviewers and set the limit from their times.