What is a question bank?

A question bank is the pool of questions from which tests and exams are assembled, each question tagged by topic, difficulty, role and sometimes industry. Rather than writing one fixed paper, the issuer draws a different set for each sitting according to a plan. A good bank is larger than any single exam and is maintained like a product.

Why does a question bank matter?

A fixed exam leaks. Once candidates have seen it, they share it, and the pass rate measures memory of the paper rather than knowledge of the subject. A bank lets each candidate face a different but equivalent set, so the result means the same thing for everyone.

For AI agent oversight, the bank also has to change as the work does. When an agent gains a capability or a new rule applies, new questions go into the bank and outdated ones come out. An exam assembled from a stale bank certifies knowledge of a system that no longer exists.

How does a question bank work in practice?

Building and running one follows a cycle:

1. Plan: list the topics and the share of the exam each should take, often in an exam blueprint.
2. Write: draft several questions per topic, at different difficulties, with clear correct answers and the reason.
3. Tag: topic, difficulty, role, industry, product version.
4. Trial: test with real candidates; drop or fix questions that everyone gets right, or that strong candidates get wrong.
5. Draw: assemble each sitting from the bank to match the plan.
6. Retire and refresh: remove exposed or outdated questions on a schedule.

Drawing different questions for each sitting is question rotation, and it only works if the bank is deep enough.

What is an example?

A bank builds an internal exam for staff who review an agent's account-change drafts. Its question bank has questions on provenance, held actions, limits and escalation, each written in three versions: for branch staff, for the contact centre and for operations. A separate set for directors covers oversight rather than review, since what a board needs to know is different; see what AI agent training a bank board needs and how the EU AI Act made AI literacy a board question. When the agent gains a new capability, the team adds twelve questions and retires six.

How is a question bank different from an exam?

An exam is one assembled test, sat once by a candidate. A question bank is the source it is drawn from. Two candidates sitting the same exam on the same day may answer entirely different questions from the bank, as long as both sets follow the same plan. The exam is what the candidate sees; the bank is what the issuer manages.

Which terms are related?

• Exam blueprint: the plan that says how many questions come from each topic.
• Pass mark: the threshold applied to the drawn set.
• Question rotation: varying the questions between sittings.

For the wider topic, read AI literacy and certification for people who work with agents.