Not for your operation. A vendor can test its own agent and publish the results, which is useful. But a certificate that decides how far the agent may act in your business needs evidence from your own cases and review by someone who does not profit from a pass. Self-certification answers the vendor's question, not yours.
What is wrong with a vendor certifying itself?
Two things, and they are separate.
The cases are wrong. A vendor tests on data it has: synthetic cases, public examples, or cases from other clients. None of these show how the agent behaves with your sources, your policies, your customers and your limits. An agent that passes on the vendor's set can fail on yours for reasons the vendor never saw.
The reviewer is conflicted. The vendor chooses the cases, sets the floors and reads the results, and benefits when the agent passes. Even with good intentions, that arrangement tends to test what the agent does well. The argument is made at length in a vendor's certification is not yours.
What vendor evidence is still worth having?
Plenty, as long as it is read for what it is.
| Vendor evidence | What it tells you | What it does not |
|---|---|---|
| Pre-release test results | The agent can pass on some set of cases | How it behaves on yours |
| Test method documentation | How the vendor thinks about failure | Whether the method fits your risks |
| SOC 2 report | Whether the vendor's controls over security, availability and similar areas were operating | Whether the agent's answers are correct or safe |
| ISO/IEC 42001 certificate | The vendor runs an AI management system | Anything about a specific agent's behaviour |
| Change and release notes | What changed and when | Whether the change broke anything for you |
A SOC 2 report is especially easy to misread as agent assurance. Reading an AI agent vendor's SOC 2 report explains what it covers.
Who should certify the agent instead?
The organisation that will use it, with an independent reviewer. In practice:
- the cases come from the customer's own operation;
- the floors for each gate are set by the customer, reflecting its own risk appetite;
- the run can be operated by the vendor, the customer or a third party, but the method is visible;
- a named person at the customer, not involved in buying or building the agent, reviews the results and accepts or rejects the certificate.
That last point is the one that most often slips. A procurement team keen to close a deal, or a project team keen to go live, is not independent either. The steps are set out in how to certify an AI agent.
What if you have no real cases yet?
New processes and new products sometimes have no history. Then the customer builds a test set from realistic constructed cases, reviewed by its own subject experts, and starts the agent at low depth while real cases accumulate. How to certify an AI agent without real data covers the approach. Even then, the customer owns the set; the vendor's generic set does not stand in for it.
An illustrative example
An insurer is offered a claims intake agent. The vendor presents a report showing the agent passed every test on a large set of claims, with high scores across the board.
The insurer's risk team asks three questions. Whose claims were these? Mostly synthetic, plus a few from another client. Who set the pass marks? The vendor. Were escalation cases included? Some, but none involving the insurer's own vulnerability policy. The insurer runs the agent against several hundred of its own past claims. It passes most gates, but fails escalation on claims that mention a recent death. The insurer narrows the agent's scope to one capability, acknowledging claims, certifies the agent at depth 3, and leaves the rest out of scope until the brief is fixed. Narrowing the scope like this is covered in how to certify an AI agent with a single capability.
What stays with the customer?
- choosing and approving the test cases;
- setting the floor for each gate;
- the decision to accept a certificate and raise depth;
- the decision to drop depth when a run fails;
- responsibility to customers and regulators for what the agent does.
None of these can be delegated to the vendor. The vendor can help with all of them. The wider framework is in what it means to certify an agent.
Questions people ask
Can an independent third party certify the agent instead of the customer?
A third party can run the tests and add independence, which is valuable. The cases should still come from the customer's operation, and the customer still decides whether to rely on the result.
Should vendors stop publishing their own test results?
No. Published results help buyers compare products and show how a vendor thinks about failure. They should be labelled as vendor testing, not certification.
What should the contract say about certification?
That the customer may test the agent on its own cases before go-live and after every change, and that the vendor will notify the customer of changes that could affect behaviour.