Every case in a certification set needs its correct outcome decided before the agent is scored, and deciding it takes an expert's time. That labelling work is the cost most teams forget to budget. Collecting cases is quick; agreeing what the right answer was, and why, is slow and skilled, so it should be planned like any other expert work.
Why is labelling harder than collecting cases?
Collecting cases is mostly retrieval: export a sample of tickets, claims or payment requests from the last few months and remove what should not be shared. Labelling is judgement, and it is the part of the effort to certify an agent that cannot be automated away. For each case, someone who knows the process has to say what the agent should have done, and that turns out to be harder than it sounds.
Three things make it hard. First, the historical outcome is not always the right one. The person who handled the case may have made a mistake, applied an old policy or made an exception that was never meant to be repeated. Second, many cases have more than one acceptable answer, and the label has to say which variations are fine. Third, for a large share of cases the right outcome is not an answer at all but a refusal, a hold or an escalation, and the labeller has to recognise those.
None of this can be delegated to the agent being tested, and little of it can be delegated to junior staff. The people who can label well are the same people the business relies on for its hardest cases.
What does a good label contain?
A label is more than "correct answer: yes". To be useful in scoring and defensible in review, each label should record:
| Field | Why it is needed |
|---|---|
| Expected outcome | Answer, draft, refusal, hold or escalation |
| Acceptable variations | So reasonable differences in wording or approach are not scored as failures |
| Required sources | The documents a correct answer must rest on, for checking grounding |
| Gate tested | Which gate the case mainly exercises, so results can be reported by gate |
| Rationale | One or two sentences on why this is the right outcome |
| Labeller and date | Who decided, and against which version of the rules |
The rationale is the field most often skipped and the one that saves the most time later. When a case is disputed, or a policy changes, the rationale tells you whether the label still holds without re-deciding it from scratch.
Who should do the labelling?
The process owner's team, not the vendor and not the people who built the agent. Labels define what "correct" means, and that definition belongs to whoever answers for the process. For capabilities with the most authority, two labellers working independently on each case, with disagreements resolved by a senior person, catch more errors than one labeller working alone.
Disagreements are worth recording rather than smoothing over. If two experienced people cannot agree on the right outcome, the case is ambiguous, and an agent should probably not be deciding it on its own. Those cases often become hold or escalation labels.
How much labelling does a set need?
That depends on the set's size, which should depend on what is at stake. The argument that certification set size should follow the cost of being wrong means a depth 4 payment capability needs far more labelled cases than a depth 1 summarising capability.
A rough way to plan: estimate the cases needed per capability, estimate the minutes an expert needs per case including rationale, and add time for disagreements. For example, a capability needing 400 cases at around six minutes each is roughly 40 hours of expert time before any second labeller. That is a week of a senior person's attention, and it should be scheduled as one, not squeezed around their day job.
Why does the cost keep coming back?
Certification is not a one-off, so neither is labelling. Certificates lapse, and drift is why certification is never finished: sources change, products change, customers change. Each recertification needs new cases drawn from current work, and every new case needs a label.
Existing labels also decay. When a policy changes, every case whose label depended on the old wording needs re-checking. The re-certification trigger checklist is a useful prompt here, because the same events that trigger a new run usually invalidate some old labels.
This recurring cost is also part of why an AI agent can lose its certification overnight without anyone having done anything wrong. If the labels behind a certificate no longer reflect the rules, the certificate is resting on an outdated definition of correct.
How do you keep the cost under control without cutting corners?
There are real savings available. Reuse labels across runs, but check each one against the current policy version before reuse, which the rationale and date fields make quick. Label held-queue decisions as you go: when a person approves, edits or rejects an agent's draft, that decision is most of a label already. Prioritise labelling effort towards the gates and capabilities that carry the most authority. And keep a small group of trained labellers, since consistency improves with practice.
What should not be cut is the expert review itself. A set labelled quickly by people who do not know the process will produce a certificate that measures agreement with guesses. The certification gate is only as strong as the labels it scores against.
What should you do differently?
Put labelling in the project plan as a named activity, with named people and hours, before the first run is scheduled. Agree the label fields in advance. Budget for recertification labelling as a recurring cost, not a launch cost. If the business cannot spare the expert time, that is a signal about how much authority the agent should have, not a reason to label faster.
Questions people ask
Can the agent help label its own certification set?
It should not decide labels, since that would mean the agent defines its own standard. It can help prepare cases, for example by gathering the relevant documents, as long as a person decides the outcome.
What if historical outcomes were wrong?
Label the case with the outcome that should have happened under the rules in force, and note the discrepancy. Those cases are valuable because they show where people, and probably agents, go wrong.
How should ambiguous cases be labelled?
If experts cannot agree, the safe label is usually a hold or an escalation. An agent should not be certified to make decisions that experienced people still dispute.