What is question rotation?

Question rotation is the practice of giving each candidate, or each sitting, a different set of questions drawn from a larger pool, while keeping every set equivalent in topic and difficulty. It stops a test turning into a memory exercise once its questions circulate, so a pass still means the person understands how the agents they oversee decide.

Why does question rotation matter for agent oversight?

Internal tests on AI agents are usually sat by a whole team within a few weeks. By the third sitting, the questions are in a shared document. Without rotation, the scores rise and the knowledge does not. The organisation then grants approval rights on the strength of a result that measures recall of a paper.

The stakes are specific. A reviewer who has memorised that "the answer to question 12 is held" has not learned why an agent holds a payment above the customer's limit, or why an emailed invoice cannot authorise one. Those are the judgements that matter on a live queue, and rotation is what forces the test to check them.

How does it work?

Rotation sits on top of a question bank. The issuer writes several questions for each point that matters, tags them by topic and difficulty, and then assembles each sitting according to a fixed plan. Two candidates may answer different questions, but each faces, say, four on held actions, three on source provenance and two on refusals.

Three rules keep it fair:

1. Equivalence: questions that can replace each other must test the same point at the same difficulty.
2. Depth of pool: each topic needs enough questions that a candidate who resits is unlikely to see the same set.
3. Retirement: questions that leak, or that describe a version of the agent that no longer exists, come out of the pool.

An example from an insurer

A regional insurer trains claims handlers to review an agent that drafts settlement letters at depth 2. The internal test has 20 questions. The insurer writes three versions of each, varying the claim type and the amount, so a handler who resits after a fail sees new cases testing the same idea. One version asks about a motor claim with a garage estimate sent by email; another asks about a home claim with a contractor's quote uploaded by the policyholder. Both test whether the reviewer knows a declared or external figure must be checked before a person sends the letter. Our piece on how insurers should train staff before deploying agents puts this test in context.

How is question rotation different from shuffling?

Shuffling changes the order of the same questions. Every candidate still sees the same paper, so once it is shared it is spent. Rotation changes which questions appear. Shuffling deters copying in the room; rotation protects the test over months.

Rotation also differs from adaptive testing, where the next question depends on the last answer. Rotation keeps the shape of the test fixed and varies only the content within it.

Which terms are related?

• Question bank: the pool rotation draws from.
• Proficiency scale: the levels a score is mapped to; see what a proficiency scale is.
• Exam blueprint: the plan that fixes how many questions come from each topic.

Boards increasingly ask how literacy is tested, a point made in the EU AI Act made AI literacy a board question. For the full approach to training, see training teams for AI agents.