Caseworkers calibrate by deciding the same held cases independently, comparing outcomes and reasons, and agreeing a written standard where they split. Choose cases where evidence or discretion is in question, include one where the agent's draft letter was wrong, and send the standard to the agent's brief owner. Then sample decisions across offices to check the standard holds.
Why does casework need calibration?
Caseworkers in public bodies and charities decide applications, claims and requests: grants, housing applications, licences, support payments. An agent typically reads the case, checks it against the rules, and drafts a recommendation and a decision letter at depth 2 for a caseworker to approve. Decisions with legal effect stay with a person.
Public bodies are expected to treat like cases alike, and people can challenge decisions that look arbitrary. Without calibration, outcomes depend on which caseworker or which office picked up the file. With an agent in the loop, inconsistency also feeds back into the agent's brief: if caseworkers amend its drafts in different directions, it cannot settle. The general method is in how to run calibration sessions for AI agent reviewers.
A senior caseworker or team manager owns the sessions. A policy adviser attends whenever the standard touches interpretation of the rules.
Which cases belong in the set?
Eight to ten cases closed in the past month, anonymised, covering:
- Evidence sufficiency: a self-declared income with no supporting documents.
- Discretion: an application just outside a rule where discretion is allowed.
- Vulnerability: an applicant whose circumstances call for a different approach to deadlines or contact.
- Conflicting evidence: a third-party letter that contradicts the applicant's account.
- A draft letter with a flaw: the agent's letter states the right outcome with an unclear reason.
- A refusal: a case the agent refused as outside its bounds, to check caseworkers pick it up correctly.
The session, step by step
- Person (team manager): selects and anonymises the cases, drawing from more than one office if the service has several.
- Person (each caseworker): decides every case alone, records the outcome, the reason, and what they would change in the agent's draft letter.
- Agent, depth 1: shows its recommendation, the rules it applied and the provenance of each fact: declared by the applicant, signed by a named third party, sealed in the agency's own records, or external.
- Person (facilitator): reveals the spread, case by case.
- Person (caseworkers): explain their reasoning, especially on evidence and discretion.
- Person (team manager, with the policy adviser): agrees the standard for each disputed case.
- Person (team manager): sends the standard to the brief owner, including changes to the agent's draft letters.
- Person (team manager): samples decisions across offices for the next month.
| Step | The agent does | Depth | Stays with a person |
|---|---|---|---|
| Select cases | Nothing | 2 on live cases | Choosing across offices |
| Decide alone | Nothing | 2 | Outcome, reason, letter changes |
| Agent's recommendation | Shows rules and provenance | 1 | Comparing |
| Discussion | Nothing | 2 | Evidence and discretion |
| Standard | Nothing | 2 | Manager and policy adviser |
| Update brief | Drafts per new standard | 2 | Brief owner |
| Sampling | Drafts live cases | 2 | Cross-office checks |
What does a gold standard look like for casework?
It states what evidence is enough and when discretion applies. For example: "Where income is declared without documents and the amount affects eligibility, the case is held and the applicant asked for evidence, with a 14-day deadline extended to 28 days where a vulnerability flag is present." And for the letter: "Every refusal letter names the rule, the fact that failed it, and how to challenge the decision."
The second sentence matters as much as the first. People are entitled to understand a decision about them. A letter that reaches the right outcome for a reason the applicant cannot follow will be challenged.
Which controls does calibration support, and what evidence does it leave?
Like cases treated alike. Evidence: the spread of independent decisions and its reduction over time.
Reasoned decisions. Evidence: the letter standard and sampled letters against it.
Policy alignment. Evidence: the policy adviser's input on any standard that interprets the rules.
Agent alignment. Evidence: the brief change and who approved it.
What should the team manager measure?
- Agreement on shared cases by office and over time.
- Sampled decisions where the reason does not meet the letter standard.
- Cases the agent refused that caseworkers then decided without escalation where escalation was required.
- How often caseworkers amend the agent's letters, and in which direction.
Where does it go wrong in casework teams?
Calibrating within one office. The biggest inconsistencies are usually between offices.
Leaving policy out. Caseworkers agree among themselves on a reading of the rules that policy would not support. The standard spreads it.
Ignoring the letters. Teams agree outcomes and leave the agent's letters as they are. Reasons are where challenges start.
Choosing comfortable cases. Facilitators sometimes pick cases they expect the team to agree on, so the session feels productive. It teaches nothing. The most useful case is the one where two experienced caseworkers in different offices reached opposite outcomes, and the record shows both were approved.
Paralegals and compliance officers calibrate on similar mixes of evidence and judgement; see calibration for paralegals and calibration for compliance officers. For planning, see the AI agent training plan template for caseworkers and the topic guide on training teams for AI agents.
Where the Academy fits
Pryme Intelligence Academy's Operator track is built for people who review an agent's held work daily, and its module tests are set in the learner's own industry and role. Its practicum is two weeks of real review decisions signed off by a Pryme assessor, which for caseworkers means real held cases, not practice ones.
Questions people ask
How often should caseworkers calibrate?
Every two weeks in the first months of an agent's use, then monthly, and after any rule change.
Should applicants' names be visible in the session?
No. Anonymise every case. Caseworkers need the facts and the evidence, not the identity.
What if caseworkers and the policy adviser disagree?
The policy owner decides, and the reasoning goes into the standard. If the rule itself is unclear, the agent's bounds should hold those cases until it is clarified.