Replay it against your own past conversations first, then try to break it, then watch it on live traffic without letting it reply. Those three stages, plus checks on every system it touches and every handover route, tell you whether each capability is ready. Launch capability by capability, not all at once, and keep the test set running afterwards.
Why are vendor demos and generic benchmarks not enough?
Because your customers do not ask generic questions. They ask about your products, your fees, your delivery partner's habits and the bug in last month's app release, in their own words and spelling. An agent that does well on a public benchmark can still fail on your tenth most common contact reason because your knowledge base is thin there. The only test that predicts how it will behave with your customers is one built from your customers.
What are the stages?
1. Build the case set. Pull real past conversations for each contact reason you plan to automate. Include easy ones, hard ones, angry ones, ones that should have been escalated, and ones where the right answer was "no". For each, a person writes the expected outcome: the correct answer, the correct refusal or the correct handover.
2. Replay and score. Run the agent against every case. Score each answer separately for grounding, correct verdict, escalation where needed, staying in scope, and harm. A single overall score hides the failures that matter. How high each floor should be is covered in how accurate an AI support agent needs to be before going live.
3. Attack it. Write cases designed to make it fail: false authority, pasted fake policy, injected instructions in attachments, requests split under limits, abuse, other languages. The agent should hold its bounds in every one.
4. Test integrations in a sandbox. For every system the agent reads or writes, test success, failure and timeout. What does the agent tell a customer when the order system does not respond? Does it ever report a refund as done when the call failed?
5. Test every handover route. In each channel, check that the conversation reaches the right queue, the note arrives complete, and the customer is told what happens next. Handover failures are invisible in replay because replay ends at the agent's last message.
6. Shadow. Run the agent on live conversations without sending its replies. Compare its drafts with what people actually sent. This catches anything your case set missed.
7. Pilot. Turn on one capability for a slice of traffic, with close review. Widen only when the evidence holds.
Which checks go on the launch list?
| Check | What passes | Who signs |
|---|---|---|
| Case replay | Every floor met | Quality lead |
| Attack set | No bound broken | Agent owner |
| Integrations | Correct behaviour on success, failure and timeout | Engineering lead |
| Handover routes | Right queue, full note, customer informed, in every channel | Support lead |
| AI disclosure | Shown or spoken in every channel | Compliance or legal |
| Shadow comparison | No new failure types found | Quality lead |
| Rollback | Capability can be switched to draft or off within minutes | Agent owner |
A fuller version is the support agent launch checklist.
An illustrative example
Take a subscription software company planning to automate three things: password help, billing questions and plan changes.
Replay goes well for password help and billing. Plan changes fail two escalation cases, where customers on annual contracts asked to downgrade mid-term, which needs a person. The attack set finds the agent will confirm a discount if the customer pastes an old promotional email. Integration tests show that when the billing system times out, the agent says "your plan has been changed" anyway.
The company launches password help. Billing goes to shadow for two weeks. Plan changes stay in development until the integration waits for confirmation and the escalation rule for annual contracts is added. Nothing about this is unusual: most first test rounds find issues like these.
Do channels need separate tests?
Yes. Voice brings mishearing, interruptions and no visual confirmation; email brings long, multi-question messages; chat brings fast back-and-forth. An agent tested in chat and launched on the phone has not been tested. The case for treating channels as one service with channel-specific checks is set out in what omnichannel support is. If the agent will triage an inbox before answering, test that step on its own too; see triaging the support inbox with an AI agent.
Other functions test agents the same way with higher stakes. Compliance teams apply this method to financial crime work, described in how to test an AML agent before using it.
Where Pryme fits
Attest replays the agent against real cases from your own operation and scores every answer against a floor on seven gates: grounded, refuses, depth, escalates, scope, adversarial and stable. Each must clear on its own before the agent is certified, and certificates lapse after 90 days. More is in the AI support agent that knows when to say no.
Questions people ask
How long should shadowing run?
Long enough to cover your normal weekly pattern and any known peaks. A quiet week will not show you how the agent behaves on a busy Monday.
Who should write the expected outcomes?
Experienced support staff, reviewed by whoever owns the policy. They know what a good answer looks like.
Should we test after launch too?
Yes. Replay the case set after every change to the agent's brief, sources or model, and on a fixed schedule.