What is blind preference?

Blind preference is a way of comparing two versions of the same piece of work without telling the judge which is which. The judge reads both, picks the better one and says why. Teams use it to test whether a change to an agent improved its output, and to check whether reviewers judge agent work fairly.

Why does it matter for teams that oversee agents?

People are biased by authorship. Some reviewers trust anything the agent drafts; others mark it down on sight. Both habits distort the evidence a business relies on when it decides whether a new brief is better than the old one, or whether the agent's replies are as good as an experienced person's. Hiding the source removes that bias from the comparison.

It also exposes something about the reviewers themselves. If a team lead finds that reviewers prefer the agent's draft when it is unlabelled but edit it heavily when it is labelled, the edits are about the label, not the content. That is a coaching point, not an agent fault.

How does it work in practice?

1. Choose the question. For example: is the agent's reply clearer than a person's, or is version two of the brief better than version one?
2. Build pairs. Take the same real cases and produce both versions for each. Strip anything that gives the source away, such as signature lines or formatting habits.
3. Shuffle the order. The agent's version should appear first in some pairs and second in others.
4. Ask for a choice and a reason. "A", "B" or "no difference", with one sentence on why.
5. Read the reasons, not just the count. A version preferred for warmth but criticised for a wrong date has a problem the tally hides.

Thirty to fifty pairs, judged by two or three people, is usually enough to see a clear pattern for one type of work.

An example

Take a hypothetical online retailer's support team. The agent owner has rewritten the agent's brief for delivery complaints. The team lead builds forty pairs: the same complaints answered by the old brief and the new one, unlabelled and shuffled. Three reviewers judge them. The new brief wins most pairs on clarity, but two reviewers flag that it promises refunds the agent cannot authorise. The tone change ships; the refund wording goes back to the owner. That kind of weekly quality work is part of AI agent training for support team leads in e-commerce in South Africa.

How does it differ from a gold standard?

A gold standard is a fixed set of cases with answers agreed to be correct. Work is scored against it, one answer at a time, so it measures accuracy. Blind preference is relative: it says which of two versions is better, not whether either is right. Both can be wrong, and the preferred one can still breach a red line.

So use them for different jobs. Gold standards decide correctness: the right amount, the right rule, the right refusal. Blind preference decides quality where several answers are acceptable: tone, clarity, structure. It also sits near inter-reviewer agreement, which measures whether judges agree with each other; low agreement on a blind test means the question was unclear.

Who uses it?

Agent owners use it before changing a brief. Team leads use it to check reviewer bias. Internal auditors can use it to test whether oversight is real, a skill set out in which AI agent skills matter most for internal auditors. The guide to AI agent skills by role shows where each role fits.

Where the Academy fits

Pryme Intelligence Academy does not teach blind preference as a separate subject. It sits inside the Operator track's work on human review and reading the workspace, where reviewers learn to judge the output in front of them on the facts and their provenance, not on who produced it.