An AI agent runbook is the operating manual for one agent in production. It lists the routine checks and who does them, the alerts and the response to each, how to suspend and restore the agent, who to call, and where its records live. Anyone on the rota should be able to run the agent from it.
What should a runbook hold that a design document does not?
A design document explains how the agent was built. A runbook explains how to keep it running safely today. It is written for the person on shift, often not a specialist, who needs to know what normal looks like, what abnormal looks like, and what to do next. The basics of the form are in what a runbook is.
For agents the runbook has three sections a system runbook usually lacks: the review of held and refused work, certificate expiry, and changes to the agent's sources. Each can change behaviour without any code changing. A fuller list of contents is in what an AI agent runbook should contain.
When do you use it?
Every day the agent runs. Write it before go-live, test it during shadowing by having operations staff use it, and update it with every change. If an incident reveals a missing step, the incident report should list the runbook update among its changes.
The service levels the agent works to sit in a separate document, the internal service level agreement for AI agents. The runbook says how to meet them.
How do you fill in each field?
- Normal ranges give the expected volumes and rates: items per hour, share held, share refused. Without these, nobody can spot abnormal.
- Routine tasks are listed by frequency with an owner and the evidence they leave.
- Alerts each get a response: what it means, first check, action, escalation. Keep one line per alert.
- Source changes list the documents and feeds the agent relies on and who tells operations when they change.
- Certificate expiry gives the date and the lead time for renewal.
- Contacts name roles and people, including out of hours.
The template
| No. | Section | Content |
|---|---|---|
| 1 | Agent summary | Name, ID, purpose, owner, capabilities and depth |
| 2 | Normal ranges | Volumes, held and refused rates, pickup times |
| 3 | Daily tasks | Task, owner, time, evidence |
| 4 | Weekly tasks | Task, owner, day, evidence |
| 5 | Monthly tasks | Task, owner, date, evidence |
| 6 | Alerts and responses | Alert, meaning, first check, action, escalation |
| 7 | Suspend and restore | Steps, who may do it, link to suspension procedure |
| 8 | Sources and dependencies | Systems and documents, change notifier |
| 9 | Certificate | Current expiry, renewal lead time |
| 10 | Contacts | Roles, names, hours, out-of-hours |
| 11 | Records | Where verdicts, changes and incidents are kept |
| 12 | Version | Runbook version and last test |
A filled example
A hypothetical payments firm's dispute intake agent, sections 2, 3 and 6.
2. Normal ranges. 400 to 600 disputes a day. Roughly one in five held for an analyst. Refusals under 30 a day. Held items picked up within two hours in business hours.
3. Daily tasks.
| Task | Owner | Time | Evidence |
|---|---|---|---|
| Review overnight held queue | Disputes team lead | 8.30am | Queue report signed |
| Sample 20 allowed items for QA | QA analyst | 11am | Scorecards |
| Check refusals for patterns | Operations analyst | 3pm | Refusal summary |
6. Alerts and responses.
| Alert | Meaning | First check | Action | Escalate to |
|---|---|---|---|---|
| Held rate above twice normal | Agent unsure or a source is wrong | Recent change log; card scheme feed status | If feed fault, suspend intake capability | Agent owner |
| Refusals spike | Brief or scope issue, or unusual requests | Sample 10 refusals | Log finding; no action if the requests really are out of scope | Agent owner |
| Certificate expires in 14 days | Renewal due | Renewal scheduled? | Book certification run | Agent owner |
| Verdict record write failure | Actions may not be recorded | Platform status | Suspend agent immediately | IT on call and agent owner |
The last alert has the strongest response for a reason: an agent whose actions are not being recorded should not act at all.
Where Pryme fits
In Pryme, the counts in normal ranges and alerts come from filed verdicts, certificates carry their expiry, and lowering a capability's depth is the suspend step. The deployment guide, train, shadow, certify, widen, explains how the runbook changes as depth widens.
Questions people ask
Who should own the runbook?
The operations lead for the team the agent works in. The technical team contributes the alert and recovery sections, but ownership sits where the agent's work sits.
How long should a runbook be?
A few pages. If it grows beyond that, move detailed procedures into separate documents and link to them.
How do we know the runbook works?
Test it. Have someone who did not write it run a day of operations and a simulated alert from it, then fix what they stumbled on.