What is a quality check?

A quality check is an examination of finished work against a stated standard, to confirm it meets it and find where it falls short. It usually happens after the work is done, often on a sample rather than every item. In AI agent operations, quality checks look at both the agent's output and the human decisions made on it.

Why do quality checks matter in AI agent work?

Live review catches problems one case at a time. Quality checks catch patterns. A reviewer approving drafts in a queue sees each case once; a quality checker sampling fifty cases from last week can see that the agent has started citing an old policy, or that one reviewer approves faster than the rest with more errors. Those findings feed back into the agent's sources, brief and bounds, and into coaching for people.

This is also a job in its own right. Checking quality, rather than doing first-line review, is often the next step for an experienced reviewer. What the role involves is set out in what an AI quality reviewer does.

How does a quality check work in practice?

A useful quality check has five parts:

1. A standard: what good looks like, written down before checking starts.
2. A sample: chosen at random, with extra weight on higher-risk cases.
3. A scorecard: the same questions asked of each case, such as was the evidence checked, was the rule right, was the reason clear, was it routed correctly.
4. Findings: what was wrong, how often and why.
5. Action: a change to the agent, the process or someone's training, and a later check that it worked.

The checker should be independent of the work. Checking your own decisions is useful practice but not a quality check.

What is an example?

A quality reviewer we will call Lena samples forty decisions a week from a team reviewing agent-drafted insurance renewal letters. One week she finds that six letters quoted a premium without the discount the customer was entitled to. All six were approved by reviewers. Tracing back, she finds the discount rule was added to the policy library after the agent's sources were last refreshed. She reports it; the agent's sources are updated, and reviewers are reminded to check discounts. Her next sample confirms the fix.

How is a quality check different from a sanity check?

A sanity check is a quick test of whether a result is plausible at all, usually done by the person producing the work before moving on: does this total look about right? It is explained in what a sanity check is. A quality check is a structured examination against a standard, usually by someone else, after the fact. One catches the obvious slip in the moment; the other finds systematic problems over time.

A quality check also looks at more than whether a decision was right. Decision quality is one thing it examines; others include format, tone and completeness of records.

Which terms are related?

• Sampling: choosing which cases to check.
• Calibration: aligning several checkers so they score the same case the same way.
• Root cause: the underlying reason a pattern of errors occurred.

The skills a checker needs are set out in what skills an AI quality reviewer needs. For the wider set employers test, read AI skills for job seekers.