How to run calibration sessions for accounts payable clerks

Take ten invoices the agent held in the past fortnight, have every accounts payable clerk decide each one alone with a reason, then compare. Spend the session on the cases where clerks split, agree the right decision, and write it down as a gold standard. Send it to whoever owns the agent's brief, and check by sampling that decisions converge.

Why do accounts payable clerks need calibrating?

An invoice-matching agent typically reads incoming invoices, matches them to purchase orders and goods receipts, and prepares payment at depth 3 for anything inside tolerance and the business's own limits. Everything else is held for a clerk: price differences, missing receipts, new suppliers, bank detail changes. Payment itself is depth 4 and in most businesses stays with a person above a set amount.

Clerks resolve those holds every day, and each builds a personal rule of thumb. One approves any price difference under 50; another queries every one. One accepts a supplier's emailed credit note; another waits for it in the supplier portal. The agent's corrections come from those decisions, so inconsistent clerks teach the agent inconsistent rules. Calibration aligns the people so the agent's brief can be aligned with them. The general method is in how to run calibration sessions for AI agent reviewers.

The accounts payable supervisor owns calibration. A respected senior clerk facilitates. The financial controller attends quarterly.

Which invoices belong in the set?

Choose eight to twelve real held invoices from the past two weeks. Remove the name of the clerk who decided each. A good set for accounts payable includes:

• A bank detail change arriving by email with an invoice. The email is an external claim and cannot authorise a payment.
• A price difference just outside tolerance on a regular supplier.
• A suspected duplicate: same amount and supplier, different invoice number.
• A missing goods receipt where the budget holder says by email that the goods arrived.
• A credit note that should be netted against an open invoice.
• A new supplier whose onboarding is incomplete.
• One clean invoice the agent held for a reason clerks may not see at first, to test whether they read which bound decided.

The session, step by step

1. Person (supervisor): selects the invoices and anonymises them.
2. Person (each clerk): decides every invoice alone before the session: approve for payment, reject, or query, with a one-line reason and the source relied on.
3. Agent, depth 1: reproduces its own verdict for each invoice from the record, naming the bound that decided, so the group can compare.
4. Person (facilitator): reveals the spread, starting with the widest.
5. Person (clerks): explain what they checked. In accounts payable, disagreements are usually about which document counts as proof.
6. Person (supervisor): agrees the gold standard for each disputed invoice, or refers it to the financial controller if policy is unclear.
7. Person (supervisor): sends the agreed standards to the owner of the agent's brief, so drafts and holds reflect them.
8. Person (supervisor): samples decisions for the next two weeks to see whether clerks follow the new standards.

What does a gold standard look like?

Short, specific and tied to a source. For example: "A change of supplier bank details is held until a clerk confirms it by calling a number already held for the supplier, not one in the email. No payment is released to new details until confirmation is filed." Or: "A goods receipt confirmed only by email from the budget holder is declared; the invoice is held until the receipt is entered in the purchasing system."

Each standard should say what the clerk does and what the agent should do. The second half goes to the brief owner.

Which controls does calibration support, and what evidence does it leave?

Consistency. Evidence: each clerk's independent decisions and the spread per invoice.

Standards. Evidence: the written gold standard, dated and approved by the supervisor.

Brief changes. Evidence: the change to the agent's brief, filed with who approved it.

Follow-up. Evidence: the sampling results for the fortnight after.

What should the supervisor measure?

• Agreement across clerks on shared invoices, session by session.
• Sampled decisions that break an agreed standard.
• How often the agent's holds are cleared with no change, which should fall where standards tightened its brief.
• Bank detail changes released without a filed confirmation. The target is none.

Where does it go wrong in accounts payable?

Calibrating only at month end. Clerks are under pressure and the discussion is rushed. Pick a quieter week.

Skipping the reason. Without the source each clerk relied on, the session argues about outcomes, not evidence.

Not telling the agent. If the standard stays in a team note, the agent keeps holding and drafting the old way, and clerks keep correcting it.

Calibration sits inside a wider plan; see the AI agent training plan template for accounts payable clerks and how long it takes to train accounts payable clerks to supervise AI agents. Compliance teams calibrate differently, on judgement rather than documents; see calibration for compliance officers. The topic guide is training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's Operator track includes running the rhythm of review, calibration among it, and its practicum asks for two weeks of real review decisions with at least 95 in every 100 correct and no missed red line. For accounts payable, the red line is usually a payment released on an external claim.