PrymeIntelligence

Training teams for AI agents·Playbook

How to run an AI agent readiness assessment in a government agency

Run an AI agent readiness assessment in a government agency: test caseworkers and decision makers on explaining decisions, then pick the first agent.

Assess caseworkers and decision makers in one directorate before an agent touches their casework. The test, set in public service, checks whether people understand held actions, refusals and the record well enough to explain any decision to the public. The report decides the first agent, its depth, and which concepts to train before anything reaches a citizen.

Why does a public body need a readiness assessment?

A government agency's decisions affect people's rights, money and access to services. People can complain, appeal or ask for the reasons behind a decision. Public bodies also face additional transparency and fairness duties, and under the EU AI Act some public uses of AI are classed as high-risk. An agent that drafts a decision letter is therefore drafting something the agency must be able to explain, line by line.

The assessment asks a simple question: does the team that will review the agent's work understand it well enough to stand behind it? It is owned by the senior responsible owner for the programme, with the directorate's operations lead running it. For the general idea, see what a readiness assessment is.

Who should take it?

One directorate or service at a time, up to 25 people. Good first cohorts are teams that handle high volumes of similar cases: licensing applications, grant claims, correspondence units, or case triage.

Include:

  • Caseworkers who will approve or amend the agent's drafts.
  • Decision makers who sign off outcomes with legal effect.
  • Team leaders who allocate work and handle escalations.
  • The information rights or complaints lead, who will need to reconstruct decisions from the record.

Where staff representatives are consulted on new ways of working, brief them before the sitting. The results are about readiness, not individual performance, and saying so in advance makes the answers more honest. Choosing who goes first is covered in who in a government agency needs AI agent training first.

The steps

  1. Person (senior responsible owner): chooses the directorate and agrees what decisions the agent will never make.
  2. Person (operations lead): asks for the test set in public service, with questions on casework, correspondence and decisions.
  3. Person (each member): sits the test alone.
  4. Person (operations lead): reads the report by concept and by role, separating caseworkers from decision makers.
  5. Person (senior responsible owner): chooses the first agent and its depth. In most agencies that is an agent drafting routine correspondence at depth 2.
  6. Person (operations lead): trains the weakest concept, starting with decision makers if they score lowest.
  7. Agent, depth 1: reads past closed cases in shadow and proposes the letter it would have drafted, which caseworkers compare with what was sent.
  8. Person (cohort): reassesses before any draft goes to a member of the public.
Step The agent does Depth Stays with a person
Choose directorate Nothing yet None Scope and never-decide list
Sit the test Nothing yet None Individual answers
Read by role Nothing yet None Caseworkers versus decision makers
First agent Named as candidate None Agent, depth, sign-off
Train the gap Runs in practice labs Practice Learning
Shadow Proposes letters on closed cases 1 Comparing with what was sent
Reassess Nothing new 1 then 2 Go-live decision

What does the test usually find?

Refusals are well understood in principle. Public servants are used to rules that say no. The gap is in telling a refusal from a held action: a held decision waits for a named person, a refused one was outside the agent's bounds altogether. Teams that confuse them will treat every held case as an error.

The record is the second gap. Caseworkers rarely think about how a decision will be reconstructed months later for an appeal or a request for information. They need to know that every verdict is filed in an append-only record, and that their own approval, with their name, is part of it.

Decisions with significant effects are the third. Under GDPR Article 22, people have rights around decisions based solely on automated processing that have legal or similarly significant effects. Teams should understand why their review must be real, not a rubber stamp. This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.

What does the report change?

It narrows the first agent. A team weak on the record should not start with an agent that drafts decision letters. Routine correspondence, such as acknowledgements and requests for missing documents, is the safer start.

It changes who trains first. Decision makers who score below caseworkers must train before the agent drafts anything they will sign.

It gives the agency evidence. The report, the decision it led to and the reassessment form part of the record of how the agency prepared its people.

What should the operations lead measure?

  • Scores by concept for each role.
  • During shadow, how often the agent's proposed letter differed from what was sent, and why.
  • After go-live, how many held drafts were approved unchanged, which should not be all of them.

Where does it go wrong in public bodies?

Assessing after procurement. The agency buys an agent for eligibility decisions, then finds its staff are ready only for correspondence.

Leaving out decision makers. Senior staff assume the test is for caseworkers. They sign the letters.

No never-decide list. Without one written down, the scope of the first agent creeps.

A wider rollout follows in how to roll out AI agent training across a government agency, and the agency's board needs its own preparation, covered in what AI agent training a government agency board needs. The topic guide is training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy offers a free readiness assessment for teams of up to 25, based on the Foundations test set in the team's own industry. The report shows scores by concept, the gaps, and the first agent to start with. Certificates are issued through VerifyDoc.ai and can be checked at source, which helps an agency show who was trained to approve what.

Questions people ask

Can the agent make eligibility decisions?

It can draft them for a decision maker if the agency allows that. Whether it should depends on the readiness of the people reviewing and on the rules that apply.

Should the public be told an agent drafted their letter?

Many public bodies tell people when AI is involved, and some rules require it. Decide the agency's position before go-live.

How often should we reassess?

Before each new agent, before any increase in depth, and after organisational changes that bring new staff into the casework.

Train the people who run your agents.

Pryme Intelligence Academy: free courses on the real product, audio lessons, labs in a free workspace, and certificates anyone can check on VerifyDoc.ai. Teams get seats, an admin view and a free readiness assessment.