Completion rates measure attendance, not competence. Knowing that every reviewer finished the AI agent course says nothing about whether they can tell a held action from a refusal, read a provenance label or decide a borderline case correctly. Organisations that report completion as proof of readiness are measuring the easy thing and missing the one that matters.
Why are completion rates so popular?
Because they are easy to collect and easy to report. A learning platform produces them automatically. They go up in a satisfying way as deadlines approach. They fit into a board pack as a single figure. And they satisfy a certain kind of compliance check: "all relevant staff have completed AI training". The definition and its proper uses are covered in what a training completion rate is.
Completion has a place. It tells you who has not started, which is useful for chasing. It is a reasonable input to a training plan. The problem begins when it is treated as the outcome.
What do completion rates hide?
Almost everything that decides whether an agent deployment is safe:
- Whether people understood. A course can be completed with the audio playing in the background and the quiz answered by guessing.
- Whether understanding survives the queue. A reviewer who could explain depth in a test may approve held items in seconds when the queue is long.
- Whether people apply it to their own work. Generic examples do not transfer automatically to a payments repair or a claims settlement.
- Whether it lasts. Certificates lapse after 90 days, and a run that clears no depth drops the agent to depth 1 the same day. A team that completed training months ago may have no idea what that looks like.
A team can show complete training and still override refusals routinely, trust external claims and miss red lines. The completion rate will not change.
Completion also hides who is missing from the picture. A rate calculated over "relevant staff" depends on who counted as relevant. Stand-in reviewers who cover the queue during leave, temporary staff brought in for a peak, and managers who approve the occasional large item are often outside the denominator. They are also the people most likely to approve a held action without understanding it. A competence measure drawn from the record covers everyone who actually made a decision, because the record names them.
How should competence be assessed in practice?
In two layers. The first is an assessment set in the learner's own work: a credit analyst is tested on affordability cases, a payments analyst on repairs and recalls, a support agent on refunds and complaints. Questions test judgement, such as which bound held this action or whether this fact can be trusted, rather than recall of definitions.
The second is sampling of live decisions. A qualified person reviews a small set of each reviewer's real decisions after the fact and records whether each was right. Over time this gives a far truer picture than any course result. It also gives each reviewer specific feedback, which is itself training.
What should you measure instead?
Measures that show judgement in real work. The record makes this possible: every verdict and every human decision is filed in an append-only record, so review behaviour can be sampled and scored.
| Instead of | Measure |
|---|---|
| Course completed | Score on assessment set in the learner's own industry and role |
| Quiz passed | Share of sampled live review decisions later judged correct |
| Hours of training | Time to competence for new reviewers |
| Attendance at refreshers | Overrides of refusals, with reasons, over time |
| Certificate held | Certificate current, and decisions since issue still sound |
The same logic applies to agents. Resolution rate alone hides whether a support agent's resolutions were right; measuring a support agent beyond resolution rate makes that case for the agent side. Measure people the same way: by the quality of what they decide, not the quantity of what they completed.
What is the trade-off?
Effort and discomfort. Sampling decisions takes time from someone qualified to judge them. Assessments set in the learner's real work take more effort to build than a generic quiz. And competence measures produce uncomfortable results: some people who completed training will turn out not to decide well, and someone has to act on that.
Completion rates are comfortable precisely because they rarely produce bad news. That comfort is the cost. An organisation that learns about weak review from its auditors or a regulator, rather than from its own measures, pays much more than the effort of sampling.
How does this connect to shadow AI and governance?
Weak training measures have a side effect. When staff are "trained" but not competent, they often fall back on tools they understand better, including unapproved ones. How to stop shadow AI agents addresses that risk directly. Competence measures show early where staff are struggling with the governed tools, which is where shadow AI tends to start.
Governance committees should ask for competence measures alongside completion. A single slide showing sampled decision accuracy by team, overrides of refusals and current certificates says far more about readiness than a completion percentage.
What should you do differently?
Keep completion as an administrative measure for chasing. Stop presenting it as evidence of readiness. Replace it in reports with two or three competence measures drawn from assessment and from sampled live decisions. Set assessments in the learner's own work. Track certificates as current or lapsed, not just held. And when a measure shows a gap, act on it with targeted practice rather than another course. The wider approach is in training teams for AI agents.
Where the Academy fits
Pryme Intelligence Academy's module tests are set in the learner's own industry and role, and Certified Pryme Operator requires a practicum of real review decisions with no missed red line, not just a completed course. For teams, an admin view shows progress, results, certificates and expiry dates together, so competence and currency are visible alongside completion.
Questions people ask
Do regulators accept completion rates as evidence of training?
They may accept them as evidence that training happened. Supervisors increasingly ask whether people are competent, which needs assessment results and evidence of decisions, not attendance.
How many decisions should be sampled per reviewer?
Enough to see a pattern, weighted towards higher-risk categories. Small regular samples work better than occasional large ones.
What if competence scores are low after training?
Treat it as information about the training, not only the learners. Look at which concepts are weak, adjust the material, and give targeted practice on the real queue.