Competency-based assessment judges whether a person can perform defined tasks to a stated standard, rather than ranking them against others or counting the hours they studied. The question is binary for each competency: can they do it reliably, or not yet? Learners pass by showing every required competency, not by averaging strong areas against weak ones.
Why does it suit AI agent oversight?
The job of overseeing an agent breaks down into specific things a person must get right every time. A reviewer who is excellent at editing drafts but cannot tell an unverified invoice from a verified one is not safe to approve payments, however high their overall score. Competency-based assessment stops strengths in one area hiding a gap in another.
It also makes training decisions clearer. If a team fails the same competency, the fix is training on that one point, not a repeat of the whole course.
How does it work in practice?
The work starts with a list of competencies, each written as an observable action with a standard. For a reviewer of a governed agent, that list might be:
| Competency | Standard |
|---|---|
| Read a verdict | Names the bound that decided it, every time |
| Judge evidence | Rejects external or declared evidence where sealed or signed is required |
| Respect a refusal | Does not work around a refused or withheld action |
| Escalate | Passes on items beyond their authority, without being prompted |
| Edit precisely | Changes only what is wrong and records why |
Each competency is then assessed with evidence that fits it: some by scenario questions, some by observed decisions on real work. A needs assessment before training shows which competencies a team already has.
What is an example?
An anti-money laundering team is preparing to review an agent's screening alerts. The competency list includes "identifies when a match rests on a declared name only". In a scenario test, most analysts pass. One misses it twice. Rather than repeating the whole course, the lead gives that analyst focused practice on provenance and reassesses that competency alone. This matches the FATF risk-based approach, where effort follows risk; see training AML staff who work with agents.
How is it different from work-based assessment?
Competency-based assessment is about what is judged: defined competencies against a standard. Work-based assessment is about where the evidence comes from: the person's real work in their real role. They combine well. A competency like "never misses a red line" is best proven on real decisions over time, so it is assessed competency-based and work-based at once.
It also differs from an AI risk assessment, which judges a system in a use rather than a person.
Which terms are related?
- Pass mark: the threshold in score-based tests; competency-based tests set a standard per competency instead.
- Red line: an error that fails the assessment on its own.
- Evidence of competence: the record the assessment produces.
For the full topic, read AI literacy and certification for people who work with agents.
Where the Academy fits
Pryme Intelligence Academy combines both approaches for Certified Pryme Operator. The written exam covers knowledge; the two-week practicum requires real review decisions at least 95 in every 100 correct and no missed red line, so a single serious error fails it regardless of the overall score. Module tests are set in the learner's own industry and role.
Questions people ask
How many competencies should a role have?
Few enough that each can be assessed properly: often five to ten for a reviewer. If the list runs to dozens, group them or separate the role.
Can competencies change over time?
Yes. When the agent gains a capability or a rule changes, add or revise competencies and reassess the people affected.