How to write runbooks for AI agent operations

Write one short runbook per operational event, not one long manual. Each names its trigger, who may invoke it, numbered steps saying who acts, the evidence each leaves and how to confirm it worked. Cover the events that will happen: lowering depth, taking work back, rollback, certificate expiry, approver absence and provider outage. Test each one before you need it.

What is a runbook, and how is it different from a brief or a policy?

Three documents are easy to confuse. The brief tells the agent what its job is, which sources it may use and when to stop; it is written for the agent, and how to write a brief for an AI agent covers it. The policy sets the organisation's rules for using agents at all: who may approve one, what must be recorded, which uses are off limits; see how to write an AI agent policy. The runbook tells people what to do when something specific happens to an agent in production.

Runbooks are for the moment when someone is under pressure and needs to act correctly without thinking from first principles. They should be short enough to follow on a phone during an incident.

The head of agent operations owns the runbook library. Each runbook has an owner who keeps it current. Agent owners approve runbooks for their agents. Risk reviews those that affect customers or money.

Which runbooks do you need?

Start with the events that will certainly happen:

• Lower a capability's depth, immediately, for any reason.
• Take work back from the agent to people.
• Roll back a change to the brief, sources, bounds, connector or model.
• Certificate about to lapse, and what happens if renewal is late.
• Failed drift check, which lowers depth automatically; the runbook covers what people do next.
• Approver absent, and how cover works.
• Provider outage: the model or platform is unavailable.
• Source system unavailable: the agent cannot read what it needs.
• Suspected manipulation: an input appears designed to make the agent misbehave.
• Widen depth for one capability.
• Retire the agent.

What goes in each runbook?

1. Trigger. The exact event or signal that starts it. "A drift check fails on any gate" is a trigger; "the agent seems off" is not.
2. Who may invoke it. By role, with deputies.
3. Steps, numbered. Each says who acts, what they do, and in which system. Where the agent acts, say at what depth: "Agent, depth 1: lists all items in flight."
4. Evidence. What record each step leaves, so the response can be reconstructed.
5. Verification. How to confirm the step worked: "Check the agent's bounds show depth 1 for the capability, and the next action is refused or held."
6. Communication. Who to tell, with a template where useful.
7. Exit. What ends the runbook and who decides.
8. Owner and version. Who maintains it and when it was last tested.

What is the method for writing them?

1. Head of agent operations: lists the events from the set above and from the agent's own risk assessment.
2. Runbook owner: drafts each runbook with the people who will follow it, not alone.
3. Agent, depth 2: drafts the factual parts: the systems involved, the bounds to change, the records to check, from its own configuration.
4. Agent owner: approves the runbook for their agent; risk reviews where customers or money are affected.
5. Runbook owner: tests it, in a test environment or a tabletop session, timing each step.
6. Head of agent operations: publishes it in the library, versioned.
7. Runbook owner: reviews it after every real use, after every change to the agent, and at least yearly.

What decisions stay with a named person?

Runbooks guide action; they do not remove judgement. The person who invokes a runbook decides that its trigger has occurred. The agent owner decides when to exit a runbook that lowered depth. Communication to customers and regulators is approved by the people the runbook names. And anyone may depart from a runbook if following it would cause harm, provided they record why. Write that permission into the library's introduction, so no one feels trapped by a step that no longer fits.

What record do runbooks produce?

• The library, versioned, with owners and test dates.
• Test records with timings and issues found.
• Each invocation: trigger, who invoked, steps taken with times, evidence, departures and why.
• Reviews after use, with changes made.

These records show a supervisor or auditor that the organisation can operate its agents under stress, not just in normal conditions.

Where does it usually go wrong?

One long manual. Nobody can find the right section during an incident. Keep runbooks short and separate.

Written by builders only. The steps make sense to engineers but not to the operations staff who will follow them. Write with the people who will act.

Never tested. The first use is the real incident, and a step turns out to need access no one has. Test every runbook.

Out of date. The agent's bounds change, the runbook still names the old ones. Tie runbook reviews to the agent's change process, as part of train, shadow, certify, widen.

Verification missing. Someone lowers depth, assumes it worked, and the agent keeps acting through a path no one checked. Every step that changes the agent needs a check.