What is item writing?

Item writing is the craft of writing individual exam questions, called items, so that each one tests exactly the skill it is meant to and cannot be answered by guessing, wording clues or general knowledge. For exams on AI agent oversight, good items test decisions on held actions, not recall of definitions.

Why does item writing matter?

An exam is only as good as its items. A blueprint can give half the marks to judgement, but if the items written for that half can be answered by spotting the longest option, the exam measures test-taking, not judgement.

For agent oversight the stakes are concrete. A business that ties approval rights to an exam result is trusting the items to separate people who can review safely from people who cannot. Weak items let the wrong people through.

What makes a good item for agent oversight?

Most good items share five features:

1. A realistic stem. A held action, with its draft, sources and verdict line, taken or adapted from real work.
2. One clearly best answer, which an experienced reviewer would agree on.
3. Plausible wrong answers. Each one reflects a real mistake reviewers make, such as trusting an external claim.
4. No clues. Options of similar length, no "always" or "never" that give the game away.
5. A tag linking the item to one area of the blueprint and one competency level.

Items should be reviewed by a second writer and tried on a small group before use.

What does item writing look like in practice?

Take a weak item: "What does a held verdict mean? (a) The action was refused (b) The action waits for a named person (c) The action was allowed (d) The agent failed." Anyone who skimmed a lesson can answer it.

A stronger item for the same concept: "The agent has drafted a refund of 400. The verdict line says held: customer's own limit. The customer's request came by email. What do you check before approving?" The options then test whether the reviewer knows the limit is the customer's setting, and that the email is external provenance.

The strongest items come from edge cases in the team's own record, rewritten to remove personal details.

How is item writing different from policy writing?

Both need precise language, but they aim at opposite things. A policy is written so everyone reads it the same way and acts the same way. An item is written so that people who understand the policy answer one way and people who do not answer another. A good AI policy that covers agents, such as one written for an ISO/IEC 42001 management system, gives the item writer clear rules to test against.

Item writing also differs from writing a lesson. A lesson explains; an item checks without explaining.

Which related terms should you know?

• Exam blueprint: the plan that tells item writers what to write and how many.
• Distractor: a wrong answer option, written to catch a specific misunderstanding.
• Item analysis: checking after use which items most people got wrong, and why.

Items also suit customer-facing skills, such as answering customers who ask whether an AI decided. When buying training, ask to see sample items; it is one of the fastest ways to evaluate an agent training provider. See also training teams for AI agents.

Where the Academy fits

The Certified Pryme Operator written exam in Pryme Intelligence Academy has 40 questions in 60 minutes, and the pass mark is 32 of 40. Module tests are set in the learner's own industry and role, so the stem a claims handler sees is a held claim, not a generic example.