Choose on your own cases, not demonstrations. Pick the process first, build a set of real past cases from it, and run each shortlisted agent against them. Alongside, test whether the vendor's controls sit outside the model, whether every decision is recorded, and whether you can stop the agent. Keep the evidence from each step; it becomes your approval file.
Why do demonstrations mislead?
Because a demonstration uses cases the vendor chose. They are clean, typical and rehearsed. Your operation has the awkward ones: the joint account where one holder has died, the invoice in two currencies, the complaint that is also a fraud report. How an agent handles those decides whether it is useful, and no demonstration shows it.
Vendor certifications have the same limit. A vendor can show that its agent passed its own tests, but that is not evidence it works in your process; a vendor's certification is not yours.
What are the steps, and who does each?
- Process owner: choose the process. Pick the job before the vendor, so you are comparing agents on the same work. Evidence: a one-page process description. Our guide on choosing the first process to give an AI agent helps here.
- Process owner and subject experts: build a case set. Several hundred past cases, including the awkward ones, each with the outcome your team reached. Evidence: the case set and how it was selected; see choosing cases for a certification set.
- Procurement and risk: shortlist on control requirements. Ask the questions in the table below in writing. Evidence: each vendor's written answers.
- Process owner and experts: trial on your cases. Run each shortlisted agent against the case set without it acting, and score every answer. Evidence: scored results per vendor, per capability.
- Security and third-party risk: due diligence. Information security, data handling, resilience, sub-processors. In the EU, financial entities also have ICT third-party risk duties under DORA. Evidence: the due diligence file.
- Legal: negotiate terms. Data use, incident notification, audit rights, exit and data return. Evidence: the contract and a summary of key terms.
- Accountable executive: decide. On the evidence from steps 2 to 6, not the demonstration. Evidence: the signed decision.
What should you ask every vendor?
| Question | A good answer |
|---|---|
| Where are limits enforced? | At the moment of action, outside the model |
| Is authority set per capability? | Yes, each action has its own level and evidence |
| Does each agent have its own identity? | Yes, visible in your access management |
| Is every decision recorded, including refusals? | Yes, in a record that cannot be edited afterwards |
| Can we reconstruct a single decision? | Yes, inputs, sources, checks and verdict |
| Does every fact carry where it came from? | Yes, and weak sources cannot authorise payments |
| Can we stop an agent at once? | Yes, without contacting the vendor |
| Is our data used to train models? | No |
| Can we test on our own cases before buying? | Yes |
A vendor that answers "the model is instructed not to" to the first question is describing a request, not a control. Written assurances of this kind still need checking, because vendor assurances are not your governance.
What does this look like in practice?
Take a specialist lender choosing a vendor for an agent that reviews broker-submitted mortgage applications for missing documents. It shortlists three vendors and runs each against three hundred past applications.
Vendor A scores highest on finding missing documents but cannot show which document each finding was based on. Vendor B scores slightly lower, records every source, and enforces its limits outside the model. Vendor C performs well on standard applications and poorly on self-employed applicants, a large part of the lender's book. The lender chooses B and asks for the self-employed cases to be part of ongoing testing.
Where Pryme fits
Pryme is one of the options you would assess this way, and we would expect to be. Attest replays an agent against real cases from your own operation and scores every answer against a floor, so the trial in step 4 is how certification works in Pryme anyway. Limits are enforced in the minimum chain at the moment of action, and every verdict is filed. More in deploying an AI agent: train, shadow, certify, widen.
Questions people ask
How many vendors should we trial?
Two or three. More than that spreads your experts' time too thinly to judge the results carefully.
Can a vendor certify its own agent for us?
It can show its own testing, which is useful input. Certification for your process should rest on your cases and your judgement; see whether a vendor can certify its own AI agent.
What if no vendor passes on the awkward cases?
Then start with the agent drafting and people deciding on those cases, and keep them in the test set so you can see if that changes.