A vendor's certification shows how its agent performed on the vendor's cases, under the vendor's assumptions. That is useful context. It is not evidence about what the agent may do in your operation, with your customers, your policies and your limits. Only testing on your own cases can decide how much authority the agent holds for you.
What does a vendor's certification actually cover?
Vendors test their products, and the better ones test thoroughly. A typical vendor package might include benchmark results on public datasets, internal evaluations on synthetic or anonymised cases from several customers, a red-team summary, and attestations about the vendor's own controls.
All of that answers questions about the product in general. Is it accurate on common tasks? Does it resist well-known attacks? Does the vendor run its infrastructure securely? Those are fair questions, and a vendor that cannot answer them is a poor choice. But they are questions about the vendor, asked on the vendor's terms. What it means to certify an agent for your own use is a different exercise.
What can it not tell you?
Everything that makes your operation yours. Specifically:
- Your sources. The vendor tested against its own documents or other customers' documents. Your policy library, fee tables and product terms are different, and an agent grounded on the wrong ones is confidently wrong.
- Your cases. Your customers ask in their own way, about your products, with your edge cases. A set that represents the vendor's customer base in general represents nobody in particular.
- Your limits. The vendor cannot know which actions your firm wants held for approval, which amounts are routine, or which teams must see what.
- Your configuration. The agent you run has your brief, your connections and possibly a different model version from the one the vendor tested.
- The date. Vendor results describe the product when it was tested. Your evidence has to describe it now.
This is why certification is per customer, not per vendor. The same product, deployed at two firms, is two different agents for the purpose of deciding what it may do.
How should you use vendor evidence?
As a starting point and a filter, not as a conclusion. It helps you decide whether a product is worth certifying at all, and it tells you where to look harder.
| Vendor evidence | What it tells you | What you still need |
|---|---|---|
| Benchmark scores | General capability on standard tasks | Scores on your own cases and labels |
| Internal evaluation reports | How the vendor tests and what it found | Your own run, on your sources, gate by gate |
| Red-team summary | Which attacks the vendor tried | Testing against your connections and inputs |
| A SOC 2 report | Whether the vendor's controls operated as described | Controls over what the agent may do for you |
| ISO/IEC 42001 certificate | The vendor runs an AI management system | Evidence about this agent's behaviour |
On the last row, the question of whether AI agent certification is the same as ISO certification is worth reading in full. A management-system certificate is a statement about how an organisation works. It does not describe what any particular agent will do with your customers.
Isn't the vendor best placed to test its own product?
It is best placed to test the product's general behaviour, and it should. But there is a structural problem with relying on that testing to authorise the agent. The question of whether a vendor can certify its own AI agent comes down to independence and accountability: the vendor benefits from a pass and does not answer for your outcomes.
None of this implies bad faith. It is the same reason firms do not let suppliers approve their own invoices. The control is designed so that it does not depend on anyone's good intentions.
What should you ask a vendor for instead?
Ask for what helps you produce your own evidence. That means the ability to run the agent against your own cases before it acts on anything live, results reported gate by gate, a clear account of what changed between versions so you know when to retest, and a way to export the record of what the agent did.
Ask, too, how the vendor's product behaves when it is refused. An agent that is built to work inside someone else's bounds, and to explain itself when it hits one, is far easier to certify than one that assumes it decides its own permissions.
Finally, make sure your contract does not treat the vendor's certification as satisfying your obligations. Your process owner should accept the evidence, as argued in the process owner should sign off certification, not the vendor.
What does this cost you?
Running your own certification takes time and expert attention that relying on the vendor would save. That is the trade-off, and it is worth stating plainly. The saving from skipping it is real but short-lived: the first time the agent gets your edge case wrong, you will have no evidence to show it was ever tested on it.
Where Pryme fits
In Pryme, Attest certifies each agent for each customer, on that customer's own cases, whatever the agent's origin. Pre-Trained Agents earn depth at a new customer only once they are certified on that customer's cases. Vendor and provider testing is kept as context in the record, not counted as evidence for depth.
Questions people ask
Is vendor testing worthless, then?
No. It is a sensible way to shortlist products and to find where to focus your own tests. It just cannot decide what the agent may do in your operation.
Do we need to recertify if the vendor releases a new version?
You need to decide whether the change is material, and record that decision. A new model or a change to how the agent reasons usually is; a cosmetic interface change usually is not.
Can we share certification sets with other firms to save effort?
You can share methods and templates. The cases and labels should be your own, because they encode your policies and customers, and often contain data you cannot share.