How to run calibration sessions for payroll administrators

Payroll administrators calibrate on the pay changes an agent held before the payroll run: late overtime claims, backdated adjustments, leavers' final pay and bank detail changes. Each administrator decides the same anonymised set alone, with the evidence they would require. Compare, agree a gold standard with the payroll manager, and change the agent's bounds before the next cut-off.

Why do payroll administrators need calibration?

A payroll agent typically reads timesheets, starter and leaver forms and change requests, drafts pay changes at depth 2, and prepares the run for approval. Paying people moves money, which is depth 4; most organisations keep the final run with the payroll manager and hold unusual changes for an administrator.

Payroll errors land on individual employees, who notice immediately. Administrators differ in small but painful ways. One accepts an overtime claim submitted after cut-off if the manager approved by email; another defers it a month. One processes a backdated pay rise on the manager's word; another waits for HR's confirmation. One updates bank details from an email that looks like it came from the employee. Employees in different teams are treated differently, and the agent learns no consistent rule. The general method is in how to run calibration sessions for AI agent reviewers.

The payroll manager owns the sessions. A senior administrator facilitates. HR attends when standards touch contracts or policy.

Which changes belong in the set?

Eight to ten held items from the last two runs, anonymised:

• An overtime claim submitted after cut-off, approved by the manager by email.
• A backdated pay rise where the manager's request and the HR record differ.
• A leaver's final pay with untaken holiday and a company loan outstanding.
• A bank detail change sent by email from a personal address. The email is external.
• A sickness absence that may move the employee to a lower rate of sick pay.
• A deduction an employee disputes.
• A change the agent refused, such as a request to pay an employee in cash outside the run.

The session, step by step

1. Person (payroll manager): selects and anonymises items.
2. Person (each administrator): decides each item alone: process now, defer, or query, naming the evidence relied on.
3. Agent, depth 1: shows its drafted change and the provenance of each input: the HR system record (sealed), a manager's approval in the HR system (signed), an email (external), the employee's own statement (declared).
4. Person (facilitator): reveals the spread.
5. Person (administrators): explain their reasoning.
6. Person (payroll manager, with HR where needed): agrees the gold standard.
7. Person (payroll manager): sends the standard to the brief owner, including changes to what the agent holds.
8. Person (payroll manager): samples changes in the next two runs.

What does a gold standard look like for payroll?

Tied to the source of each input. For example: "Overtime submitted after cut-off is paid in the next run unless approved in the time system before cut-off. Email approval is external and does not bring a claim into the current run."

On backdating: "A backdated pay change is processed only when the HR system record shows the new rate and effective date. A manager's request alone is declared and is held."

On bank details: "Bank detail changes are accepted only through the employee self-service portal, where the employee is signed in. Any change by email is refused by the agent, and the administrator tells the employee how to make the change themselves, using contact details already on file."

On leavers: "Final pay including deductions for loans is checked against the signed loan agreement; disputed deductions are held for the payroll manager."

What does a session look like?

Take a hypothetical payroll team of five administrators serving 1,500 employees. On the emailed bank detail change, two would have processed it after replying to the same email to confirm. The payroll manager pointed out that replying to the sender confirms nothing if the mailbox is compromised. The standard above was agreed, and the brief owner changed the agent so emailed bank changes are refused outright, with a drafted reply directing the employee to the portal.

Which controls does calibration support, and what evidence does it leave?

Consistent treatment of employees. Evidence: spreads over sessions.

Bank detail fraud. Evidence: the portal-only standard and the agent's refusals.

Backdating. Evidence: changes processed only with an HR record.

Run approval. Evidence: the payroll manager's approval of each run.

What should the payroll manager measure?

• Agreement on shared items.
• Bank detail changes processed outside the portal. The target is none.
• Corrections needed after each run, which should fall.
• Employee queries about pay, before and after calibration.

Where does it go wrong in payroll teams?

Calibrating in run week. Nobody has time. Use the week after.

HR not involved. Payroll agrees a backdating rule HR's records cannot support.

Being kind to the employee in front of you. A late claim paid for one person and deferred for another is the inconsistency calibration exists to remove.

IT administrators and collections agents calibrate on other risks with the same method; see calibration for IT administrators and for collections agents. For planning, see the AI agent training plan template for payroll administrators and how long it takes to train payroll administrators to supervise AI agents. The topic guide is training teams for AI agents.