A/B testing is a method of comparing two versions of something, A and B, by giving each to a separate share of real users or cases at the same time and measuring which performs better. With AI agents, it can compare two briefs, two models or two sources, but only where both versions are already safe to act.
Why does A/B testing matter for agents?
Agents change often: a reworded brief, a new source, a different model. Offline tests show whether a change is acceptable on past cases. They cannot always show which version customers respond to better, or which one reviewers edit less, in live conditions. A/B testing answers that by putting both versions in front of real work and comparing outcomes.
It has a sharp limit. Every case routed to version B is a real case. If B makes worse decisions, real customers bear them. That is why A/B testing suits choices between two good options, not proving that a new option is good in the first place.
How does it work in practice?
- State the question. "Does the shorter reply style lead to fewer follow-up contacts?" not "is B better?"
- Certify both versions first. Each must clear certification at the depth it will act.
- Split the work randomly, by case, and keep the split fixed for the test.
- Choose the measure in advance: reviewer edit rate, follow-up contacts, resolution time, complaint rate.
- Run long enough to see a stable difference across case types.
- Decide and record which version stays, and why.
The measures should reflect customer outcomes, not only efficiency; running customer outcome testing with an AI agent covers how to choose them.
What does an example look like?
An energy supplier's support agent drafts replies to billing questions at depth 2. The team wants to know whether replies that lead with the answer, then explain, need fewer edits than replies that explain first. Both versions pass certification. For four weeks, cases are split evenly. Version B, answer first, is edited less and gets fewer follow-up questions, with no change in complaints. The team adopts it and files the result with the brief change.
How is A/B testing different from shadow mode?
In shadow mode, the new version runs alongside the live one on the same cases, but its outputs are not used; they are only compared. Nobody is affected by its mistakes. In A/B testing, both versions act on real cases. Shadow mode is how you find out whether a new version is safe; A/B testing is how you choose between versions that already are. What shadowing proves that testing cannot explains why shadowing comes first for anything new.
It also differs from formal testing and evaluation, which measures a version against fixed criteria rather than against another version. The NIST AI RMF view of testing, evaluation and validation for AI agents describes that discipline.
Which terms sit close to it?
Canary release: sending a small share of traffic to a new version to check it is safe before rolling out. Shadow mode: running without acting. Multi-armed bandit: a variant that shifts traffic towards the better version as results come in. For where testing sits in going live, see our guide to deploying AI agents.
Questions people ask
Can you A/B test a change to an agent's depth?
Not safely. Depth is earned through evidence and set per agent; giving half your cases a higher depth to see what happens is exactly the risk certification exists to avoid.
Do customers need to be told they are in an A/B test?
They need to be told when they deal with an AI system, whichever version they get. Whether the test itself needs disclosure depends on what differs and your obligations; take advice if the versions treat customers differently in substance.