How to run an AI agent readiness assessment in a hospital

Begin with the administrative teams an agent will join first: patient access, booking, medical secretaries and clinical coding. Test them on governed-agent concepts set in hospital work, with clinical red lines built into the questions. The report shows the gaps by concept and points to a first agent that stays well clear of clinical judgement.

What is the assessment for in a hospital?

Hospitals are likely to introduce agents in administration long before anything near the bedside: rebooking appointments, drafting clinic letters from dictation, preparing referral paperwork, and suggesting codes for a coder to confirm. Every one of these touches health data, which is special-category data under GDPR and protected health information under HIPAA in the US. Every one can also contain a clinical signal that must reach a clinician.

The assessment checks whether the admin team understands what the agent may do, what it will hold, and what it must never decide. The head of patient access or the operations manager for outpatients usually owns it, with the information governance lead involved from the start. For the concept, see what a readiness assessment is.

Who should take it?

One administrative service of up to 25 people. Outpatient booking is the usual choice, because volumes are high and the work is rule-based. The cohort should include:

• Booking and patient access staff across day, evening and weekend rotas.
• Medical secretaries who will approve drafted letters.
• Team leaders who handle escalations and complaints.
• One clinical lead as an observer, to confirm the questions on red lines match practice.

Clinicians take their own sitting later, if at all at this stage. Mixing them in changes how admin staff answer. The question of order is covered in who in a hospital needs AI agent training first.

The steps

1. Person (head of patient access): chooses the service and cohort, and lists the clinical red lines the agent must route to a person.
2. Person (head of patient access): asks for the test set in healthcare, with questions on appointments, letters and patient messages.
3. Person (each member): sits the test alone, with sittings on every rota.
4. Person (head of patient access and information governance lead): read the report by concept.
5. Person (head of patient access): chooses the first agent and its depth.
6. Person (team leaders): train the weakest concept in short sessions that fit around clinics.
7. Agent, depth 1: reads past rebooking requests in shadow and proposes the action it would take, including which it would hold for a clinician.
8. Person (cohort): reassesses before the agent drafts anything a patient will see.

What does the test usually find in hospitals?

Red lines are understood but not applied. Staff know that symptoms belong with clinicians. When the question is a rebooking request that mentions, in passing, that the patient's pain is worse, many answer as if it were a booking problem. The agent will hold such messages if its bounds say so; the reviewer must then route them, not book them.

Declared versus sealed. The patient administration system is the system of record. A patient's message saying their GP has changed is declared. A letter forwarded by a relative is external. Admin staff often treat all three as equally reliable.

The record. Few admin staff think about how an appointment change will be reconstructed after a complaint. The test shows whether they know their approval is part of the filed record.

What does the report change?

It decides the first agent. A team that misses clinical signals in the test should start with an agent that drafts appointment confirmation letters, where content is predictable, rather than one that reads free-text messages from patients.

It sets the bounds. The red-line list written in step one becomes part of the agent's bounds, so messages containing clinical signals are held for a person whatever else they contain.

It informs rota cover. If weekend staff score lowest, the agent should not draft at weekends until they are trained, because fewer senior staff are on site to catch errors.

What evidence should the hospital keep?

The red-line list, approved by the clinical lead.

The report by concept, with information governance sign-off.

The decision on the first agent and its bounds.

The reassessment before patient-facing drafts.

What should the head of patient access measure?

• Scores by concept, by rota.
• During shadow, how many messages with a clinical signal the agent held, and how many the team would have missed.
• After go-live, time from a held clinical-signal message to a clinician seeing it.

Where does it go wrong in hospitals?

Starting with clinicians. Their time is scarce and their questions differ. Start where the agent starts.

Skipping evening and weekend rotas. Those are the hours with least supervision.

Treating red lines as training content only. A red line that lives in a slide deck, not in the agent's bounds, will eventually be missed by a tired person.

Contact centres face similar shift and volume problems; the contact centre version is a useful comparison, as is the bank version for its approach to second-line oversight. The readiness assessment report template gives a format, and the topic guide is training teams for AI agents.

Where the Academy fits

Pryme Intelligence Academy's free readiness assessment covers teams of up to 25. The team takes the Foundations test set in its own industry, so hospital staff answer questions about appointments, letters and patient messages, and the report shows the gaps and the first agent to start with.