How to run calibration sessions for IT administrators

IT administrators calibrate on the tickets an agent held because they carry access risk: privileged access, multi-factor resets, movers keeping old rights, and emergency requests. Each administrator decides the same set alone, with the check they would make. The team compares, the security lead sets the gold standard, and the agent's bounds change to match.

Why do IT administrators need calibration?

An IT service agent typically resolves routine tickets at depth 3 within bounds, such as password reset links to a verified device, standard software from an approved list, and access to shared folders with a recorded manager approval. It holds anything that raises access risk for an administrator.

Those held tickets are where social engineering lands. One administrator resets multi-factor authentication for a caller who sounds like a senior manager; another insists on a video call. One grants temporary admin rights for an urgent fix; another refuses without a change ticket. Attackers find the lenient administrator. And the agent, corrected inconsistently, holds and releases inconsistently. Calibration closes the gap. The general method, with an access-request example, is in how to run calibration sessions for AI agent reviewers.

The IT operations manager owns the sessions. The security lead sets the standard on anything touching identity or privilege.

Which tickets belong in the set?

Eight to ten held tickets from the past month:

• A multi-factor reset request by phone from someone claiming to be travelling and locked out.
• A request for local admin rights to install a tool, from a developer, with no change ticket.
• A mover: an employee who changed department and asks to keep access to the old team's finance folder.
• A leaver's account still active a week after the leaving date, with a manager asking to keep it open for handover.
• A request to create a service account with broad permissions for a new integration.
• Emergency access to a production system outside hours.
• A ticket the agent refused, such as a request to disable logging on a server.

The session, step by step

1. Person (IT operations manager): selects tickets and removes requester and administrator names.
2. Person (each administrator): decides each ticket alone, recording the decision and the verification step they would take.
3. Agent, depth 1: shows its verdict, the bound that held it and the provenance of the request: a ticket raised from a signed-in session is signed; a phone call is declared; an email from outside is external.
4. Person (facilitator): reveals the spread, starting with identity cases.
5. Person (administrators): explain their checks.
6. Person (security lead): sets the gold standard for identity and privilege cases; the operations manager for the rest.
7. Person (IT operations manager): sends the standard to the brief owner, with any change to what the agent resolves at depth 3.
8. Person (security lead): samples resolved and held tickets for a month.

What does a gold standard look like for IT administration?

Specific about verification. For example: "Multi-factor resets are never done on a phone call alone. The administrator calls back on the number in the HR system and confirms one fact not visible on social media, or arranges a video call with the user's manager present."

On movers: "Access from a previous role is removed on the move date. Exceptions need the new and old managers' approval in the request system, for a set period, after which the agent removes the access automatically."

On privilege: "Local admin rights require a change ticket and are granted for 24 hours. Service accounts with broad permissions need security lead approval."

On refusals: "Requests to disable logging or security controls are refused by the agent and escalated to the security lead."

What does a session look like?

Take a hypothetical team of eight administrators. On the travelling caller, five would have reset after asking for the user's employee number and manager's name, both easy to find. Three insisted on a call-back. The security lead set the call-back standard, and the brief owner changed the agent so any multi-factor reset request that does not come from a signed-in session is held with a note reminding the administrator of the call-back step. In the next session, all eight chose the call-back on a similar case.

Which controls does calibration support, and what evidence does it leave?

Identity verification. Evidence: the verification standard and sampled resets.

Least privilege. Evidence: standards on movers, admin rights and service accounts.

Security refusals. Evidence: the agent's refusals and escalations.

Agent alignment. Evidence: bound changes and approvers.

What should the IT operations manager measure?

• Agreement on identity cases. The target is full agreement.
• Resets done without the call-back step, found by sampling.
• Movers with old access beyond the agreed period.
• Time from a new social engineering pattern to a standard and bound change.

Where does it go wrong in IT teams?

Treating it as a service-desk exercise. Without the security lead, the standard favours speed.

Only real requests in the set. Add one simulated social engineering ticket each session.

Not touching the agent's routine scope. If a routine category produced disagreement, it should move from depth 3 to held until the standard is set.

Payroll administrators and underwriters calibrate on different risks; see calibration for payroll administrators and for underwriters. For planning, see the AI agent training plan template for IT administrators, and the topic guide on training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's Operator track includes a module on when something goes wrong and one on teaching the agent, which covers changing its bounds after a session like this. A business can make approval rights on held access tickets depend on a current Certified Pryme Operator certificate, which is valid for 12 months.