How to evaluate an AI agent training provider

Evaluate an AI agent training provider on four things: whether learners practise on a real governed product rather than slides; whether skill is proved on real decisions, not only a quiz; whether certificates expire and renew; and whether anyone can check a certificate at source. A provider that cannot show all four is selling awareness, not competence.

What are you buying, and who should own the evaluation?

You are buying evidence that the people who approve, correct and build your agents can do it. That is different from buying general AI education. The question to keep in front of every provider is: after this, which of our people will be able to approve which agent actions, and how will we prove it?

The evaluation is usually led by learning and development, with a risk or compliance owner and an operations lead on the panel. Operations knows what reviewers actually do; risk knows what a supervisor will ask. The broader programme the provider must fit into is set out in training teams for AI agents.

How do you run the evaluation, step by step?

1. Person (L&D lead): writes the requirement from your agent register: which roles, which depths, how many people.
2. Agent, depth 1: reads provider materials you have collected and lists, for each provider, what they say about practice, assessment, expiry and verification, quoting the source document.
3. Agent, depth 2: drafts a comparison table for the panel, marking where a claim is unsupported.
4. Person (panel): asks each provider the questions below, and requires evidence, not descriptions.
5. Person (operations lead): runs a trial with two or three reviewers on one module and its assessment.
6. Person (risk owner): checks that certificates can be verified independently and that lapses show.
7. Person (panel): decides, recording the reasons.

What should you ask about practice?

Do learners work on the real product? Reviewing an agent's held action on a slide is not the same as doing it in a workspace where the verdict, the bounds that decided it and the record are all there. Ask to see a lab.

Is practice set in our industry and role? A claims handler learns more from claims cases than from generic customer service examples. Ask whether tests are set in the learner's own industry and role.

Does the content teach how a governed agent decides? Look for depth, the five bounds, the four verdicts (allowed, held, refused, withheld), and the record. A provider that teaches prompt writing but not held actions is preparing people for a different job.

What should you ask about proof of skill?

Is there an assessment on real work? The strongest proof is a period of real review decisions, scored for accuracy, with missed red lines treated as failures on their own. A quiz on concepts is useful but not enough for people who approve high-depth actions.

Who signs off? An assessor independent of the learner's manager adds weight.

What is the pass mark, and is it stated? Vague pass criteria are a warning sign. Ask for the number of questions, time allowed and the mark required.

Can a learner pay to pass? They should pay, if at all, only to be issued a certificate, never to pass.

What should you ask about expiry and checking?

Do certificates expire? Agents change, products change, and people forget. A certificate that never expires says nothing about competence today. Ask how long each level lasts and what renewal involves.

Can anyone check a certificate at source? An auditor, a supervisor or a client should be able to verify a named person's certificate by number or code, and see whether it is current, lapsed or revoked, without asking the provider by email.

Is there a register? A public register of certified people, by name, credential and expiry, makes checking easy.

These questions matter more for outsourcing providers, whose clients will want to check their staff. Who to train first in that setting is covered in who in an outsourcing provider needs AI agent training first, and the board-level view in what AI agent training an outsourcing provider board needs.

What about readiness and team administration?

Ask whether the provider can assess your team before you buy, so you know the gaps. A short readiness assessment, scored by concept, tells you more than a sales call. Running an AI agent readiness assessment in an outsourcing provider shows what a useful one looks like. Ask too for an admin view: progress, results, certificates and expiry dates by person, so you can tie approval rights to current certificates.

Note the vocabulary trap: under the EU AI Act a "provider" is the party that develops an AI system and places it on the market, a different meaning entirely. What a provider is under the EU AI Act explains it; do not confuse the training provider with your agent's provider in contracts.

Which evidence should the evaluation leave?

A requirement document; the comparison table with sources; trial results from real reviewers; a verified certificate check; and the panel's recorded decision. If a regulator later asks why you chose this provider, that file answers.

What should you measure after choosing?

Whether trained people's review decisions improved in the record: accuracy, overturned approvals, red-line misses. Whether certificates are renewed before they lapse. And how long new reviewers take to reach a standard you would trust on a high-depth agent.

Where does it usually go wrong?

Choosing on content volume. More hours is not better if none of them are spent on real decisions.

Accepting certificates that cannot be checked. A PDF with a logo proves little.

Skipping the trial. Ten minutes of a reviewer using a lab tells you more than an hour of presentation. Pryme Intelligence Academy, for example, runs every lab in a free Build workspace, which makes a trial easy to arrange with any provider that works this way.