AI evaluation is the practice of measuring how an AI system behaves against a stated standard, using test cases, scoring rules and human review. For a chatbot it answers "are the answers good?" For an AI agent that acts, it must also answer "does it stop, refuse and hand over when it should?", because its costly failures are actions.
Why does evaluation matter once an agent acts?
A model that writes a poor summary wastes a reader's time. An agent that posts a wrong journal, sends an unapproved message or releases a payment creates a loss that someone has to unwind. Evaluation is how you find those failures before customers do.
It is also how you answer the people who will ask. Risk committees, auditors and supervisors want to know what was tested, on which cases, against what standard, and what happened when the agent failed. The NIST AI Risk Management Framework names this work in its Measure function, and our guide to NIST AI RMF testing, evaluation and validation of AI agents maps it to agent controls.
What does AI evaluation cover for an agent?
Answer quality is only one part. A useful evaluation of an agent looks at several behaviours separately:
- Grounding: is every claim traceable to a source the agent was allowed to use?
- Refusal: does it decline requests outside its role?
- Escalation: does it pass a case to a named person when the rules say so?
- Scope: does it stay inside the systems and actions in its brief?
- Harm: does it avoid outputs that could hurt a customer or the business?
- Drift: does its behaviour hold steady as models, sources and data change?
Averaging these into one number hides the one that fails. Each needs its own floor.
How does evaluation work in practice?
Teams build a set of cases, ideally drawn from their own operation, with an agreed correct outcome for each. The agent runs every case. Its outputs are scored by rules where the answer is exact, by people where judgement is needed, and sometimes by a second model checked against human scores. Results are compared to a floor, and failures are read one by one rather than summarised away.
The work repeats. A new model version, a new policy or a new data source can change behaviour, so evaluation runs on every material change and on a fixed cycle, not only at launch. Evals for AI agents covers how individual test sets are built.
An example from a claims team
Take a regional insurer's claims team testing an agent that drafts claim acknowledgements. The evaluation set holds 300 past claims. The agent drafts well in most of them, but in a handful it states a settlement timescale the policy wording does not support. On an averaged score it looks strong; on a per-behaviour floor, the grounding check fails and the agent stays at read-only until the brief and sources are fixed.
How is AI evaluation different from certification?
Evaluation produces evidence. Certification is a decision made on that evidence: this agent may perform this capability, at this depth, for this customer, until a stated date. You can evaluate an agent endlessly without ever deciding what it may do. Agent certification is where the evidence becomes authority, and where authority lapses if the evidence goes stale.
An AI management system, such as one built to ISO/IEC 42001, sits a level above both: it governs how an organisation runs evaluation and certification as repeatable processes.
Questions people ask
Who should design the evaluation cases?
The people who own the process the agent works in, with help from risk and engineering. They know which cases are routine, which are traps and what a good outcome looks like. Engineering makes the cases runnable; it should not decide what correct means.
Is evaluation the same as testing?
Close, but not identical. Software testing checks that code does what it was written to do. Evaluation checks whether a system that reasons and chooses behaves acceptably across cases nobody fully scripted, which is why it relies on floors and human judgement rather than pass or fail on fixed outputs alone.