Start with roles, not people. An AI literacy needs assessment maps every role to how it touches your agents, tests what the people in that role know today, and records the gap against the level the role requires. The template below has three parts: a role map, a short knowledge check, and a gap summary that feeds the training plan.
What does the assessment tell you?
It tells you who needs what, in what order. A board member who reads an agent's monthly report needs a different grasp of agents from a reviewer who releases held refunds all day. Treating them the same wastes the board's time and leaves the reviewer under-prepared.
It also gives you a defensible starting point. The EU AI Act's Article 4 asks providers and deployers to take measures to ensure a sufficient level of AI literacy among the staff who deal with AI systems, taking into account their knowledge, experience and context. A needs assessment is how you show you considered all three. For who falls in scope, see who needs AI literacy under the EU AI Act.
Keep it separate from an AI risk assessment. The risk assessment asks what could go wrong with the agent; the needs assessment asks whether the people around it can spot and stop that.
How do you run it?
- List every role, including contractors, that reads, reviews, approves, builds or answers for an agent's work.
- Classify each role's contact using the five levels in Part A.
- Set the target each contact level requires (Part A, last column).
- Run the knowledge check (Part B) with everyone in scope. Ten scenario questions set in their own work beat forty generic ones.
- Score by concept, not just in total, so the gaps point to specific lessons.
- Complete the gap summary (Part C) and hand it to whoever owns the AI literacy policy.
Self-ratings are useful only next to a test. People who have never seen a held action cannot tell you whether they understand one.
The template
Part A: role map
| Role | Headcount | Contact with agents | Highest depth they touch | Target level |
|---|---|---|---|---|
| None / reads outputs / reviews held items / approves depth 4 / builds or certifies | 1 to 5 | Aware / practising / certified |
Part B: knowledge check (one row per concept)
| Concept | Sample scenario question | Score (0 to 2) | Self-rating (1 to 5) |
|---|---|---|---|
| Depth | The agent drafted a reply but did not send it. What depth is that? | ||
| Decision chain | Which bound decided this refusal, and where is that shown? | ||
| Verdicts | What is the difference between held, refused and withheld? | ||
| Provenance | An invoice arrived by email. Can it authorise a payment? | ||
| Human review | What do you check before releasing a held action? | ||
| Certificates | What happens to the agent the day a certification run fails? | ||
| The record | Where would you find why this decision was made? | ||
| Escalation | Who do you call when the agent does something unexpected? |
Part C: gap summary
| Role | Target level | Average score by concept | Largest gaps | Priority (1 to 3) | Proposed route |
|---|---|---|---|---|---|
A worked example
Take a sample motor insurer's claims operation of 24 people. Values are illustrative.
Part A shows four roles: 16 claims handlers who review held settlement offers (depth 4 touched, target "certified"); 4 team leads who approve offers above handler limits (target "certified"); 3 fraud referrers who read agent summaries (depth 1, target "practising"); 1 head of claims who answers for the deployment (target "aware" plus board-level briefing).
Part B, run as ten scenario questions, shows handlers scoring well on depth and verdicts but poorly on provenance: most would accept a repair estimate emailed by a third party as if it came from the insurer's own system. Team leads score well everywhere except certificates; few knew a run that clears no depth drops the agent to depth 1 the same day.
Part C ranks provenance for handlers as priority 1, because it sits directly on the money path. The route for handlers is scenario training followed by a supervised period on live work, which a work-based assessment can confirm. Team leads get a short module on certificates and lapse. For insurance-specific detail on Article 4, see AI literacy for AI agents in insurance.
Where the Academy fits
Pryme Intelligence Academy offers a free readiness assessment for teams of up to 25 people. The team takes the Foundations test set in its own industry, and the report shows scores by concept, the gaps, and the first agent to start with. That covers Parts B and C for a single team. The wider context is in AI literacy and certification for people who work with agents.
Questions people ask
How often should the assessment be repeated?
Repeat it when you deploy a new agent, raise an agent's depth, or reorganise the teams around it, and at least once a year. Each of those changes who touches what.
Should results be anonymous?
Report them anonymously by role, but keep named results for the people who will approve held actions. Their training record has to show they reached the target.
What if a role scores well but the work goes wrong?
Then the test measured recall, not judgement. Add scenario questions drawn from the errors and move that role to supervised live work before relying on the score.