How to run calibration sessions for clinic practice managers

Bring practice managers from several clinics together, give them the same held items the agent escalated, and have each decide alone. Refund disputes, complaints and records requests are the usual material. Compare, discuss where managers split, and agree a gold standard that applies across every site. Send it to the agent's brief owner and check by sampling that sites converge.

Why do practice managers need calibrating?

In a group of dental, physiotherapy or medical clinics, each site has a practice manager who handles what the front desk cannot: disputed cancellation fees, complaints, requests for records, unusual bookings and refunds. When an agent handles routine front-desk work, it holds those items for the practice manager.

Each manager has their own instincts. One waives every late cancellation fee for a regular patient; another never does. One treats a complaint about waiting times as administrative; another passes it to the lead clinician because the patient mentions pain. Patients in the same group get different answers by site, and the agent receives conflicting corrections from different managers. Calibration brings managers to one standard. The general method is in how to run calibration sessions for AI agent reviewers.

The group operations manager owns the sessions. One experienced practice manager facilitates. A clinical lead attends for any case with a clinical element.

Which items belong in the set?

Eight to ten items the agent held across sites in the past month, anonymised:

• A late cancellation fee dispute from a patient with a reason that may or may not qualify.
• A refund request for a course of treatment stopped partway.
• A complaint that mixes waiting time with a mention of worsening symptoms.
• A request for records from a patient, and another from a relative.
• A booking request for a treatment the clinic needs a clinician to approve first.
• A payment plan request outside the group's standard terms.

The session, step by step

1. Person (group operations manager): selects items from several sites and anonymises patients and sites.
2. Person (each practice manager): decides every item alone, with the reason and what they would tell the patient.
3. Agent, depth 1: shows its verdict for each item and which bound decided it, for example the group's own refund limit, or a red line that holds any message mentioning symptoms.
4. Person (facilitator): reveals the spread and starts with the widest.
5. Person (practice managers): explain their reasoning. Differences often come from local habit rather than policy.
6. Person (group operations manager, with the clinical lead where needed): agrees the standard.
7. Person (group operations manager): sends the standard to the brief owner, so the agent's holds and drafts change across all sites at once.
8. Person (group operations manager): samples decisions at each site for the next month.

What does a gold standard look like?

Short and usable at the desk. For example: "Any complaint or message that mentions new or worsening symptoms is passed to the treating clinician the same day, whatever else it is about. The practice manager handles the administrative part separately." Or: "A late cancellation fee is waived once in twelve months for any patient, and always where the patient gives a medical reason; the agent drafts the waiver and the practice manager approves it."

Records requests need care. A patient's own request is handled under data protection rules; a relative's request is declared and needs the patient's authority or another lawful basis. The standard should say who checks that, and the agent's bounds should hold every records request for a person.

Which controls does calibration support, and what evidence does it leave?

Consistency across sites. Evidence: the spread by site and its narrowing over time.

Clinical routing. Evidence: the red-line standard approved by the clinical lead.

Fees and refunds. Evidence: the standard and the group's own limits applied to the agent.

Agent alignment. Evidence: the brief change and its approver.

What should the group operations manager measure?

• Agreement across practice managers, session by session.
• Complaints with a clinical element that reached a clinician the same day.
• Fee waivers and refunds by site, which should become more similar.
• How often practice managers change the agent's drafted patient replies.

Where does it go wrong in clinic groups?

Each site calibrating alone. The point is consistency across the group. One site's calibration hardens its own habits.

No clinical voice. Practice managers agree on handling a complaint the clinical lead would route differently.

Calibrating fees but not words. What the patient is told matters as much as the decision. Include the reply in the session.

Managers who also run operations or risk calibrate on wider questions; see calibration for operations managers and calibration for risk managers. For day-to-day review, see how clinic practice managers review an AI agent's decisions, and for planning, the AI agent training plan template for clinic practice managers. The topic guide is training teams for AI agents.

Where the Academy fits

Practice managers who run a live workspace and decide what the agent holds are the people Pryme Intelligence Academy's Operator track is for, and its module tests are set in the learner's own industry and role. Certified Pryme Operator is valid for 12 months, with renewal through an update module and a short quiz.