How to run a weekly AI agent performance review

Hold a short weekly meeting with the agent owners, the head of agent operations and the certification lead. Work from a fixed pack the agent compiles from its own records: quality samples, held queue, refusals by bound, drift checks, expiring certificates and incidents. End every review with recorded decisions, even "no change". A review without decisions is a status meeting.

What is the weekly review for?

Agents change without anyone touching them. Their inputs shift, their sources age, the case mix moves, and approvers' habits drift. The weekly review is the routine where someone looks at the evidence and decides whether each capability still deserves its depth. It is also where small problems are caught before they become incidents.

The head of agent operations chairs. Each agent owner attends for their agents and owns the decisions about them. The certification lead brings drift results and upcoming renewals. A risk partner attends periodically and for any capability at depth 3 or above that changed in the week.

What are the steps?

1. Agent, depth 1: compiles the pack from verdict records, sampling results and monitoring, in the same layout every week.
2. Agent, depth 1: flags movements beyond thresholds the owners have set, so the meeting starts with what changed.
3. Chair: runs the agenda in the same order every week (below).
4. Agent owners: decide for each flagged capability: no change, investigate, narrow, or propose widening for certification.
5. Certification lead: confirms renewals due and any capability whose drift checks failed.
6. Scribe or agent, depth 2: drafts the decision log, which the chair approves the same day.
7. Agent operations: applies depth changes decided in the meeting, and the agent's records show the change from the moment it takes effect.

What is on the agenda?

1. Incidents and stop events. Anything that went wrong, what was done, and whether the response is complete. A full investigation is a separate exercise, the post-incident review; the weekly meeting checks that it is happening.

2. Quality. Errors found in the week's random samples, by capability and case type, not just in total.

3. Held queue. Size, age, and the share of held items approved unchanged. A very high unchanged share can mean the agent is ready for more depth, or that approvers have stopped reading.

4. Refusals. By deciding bound. A sudden change in refusals by one bound is a signal, whether up or down.

5. Drift. Results from rerunning certification cases against the live agent. Any failed gate drops that capability's depth the same day, before the meeting; the meeting confirms the follow-up. Techniques for catching drift early are in how to stop an AI agent's performance from drifting.

6. Certificates expiring in the next month, and the plan for each.

7. Changes planned: new sources, brief edits, connector updates, and whether each needs recertification.

8. Decisions. Read back and recorded.

Which controls make the review safe?

A fixed pack. The same measures every week, so movements are visible and nothing can quietly disappear. Evidence: the pack, archived each week.

Figures from records, not from memory. Every number traces to the verdict records or the sample log. Evidence: the query behind each figure.

Breakdown below the average. Measures by capability and case type, not just per agent. An agent that is fine overall can be failing on one kind of work.

Decisions recorded. Evidence: the decision log with owner and reason.

Depth changes take effect from a recorded time. Evidence: the depth change in the agent's bounds and records.

For agents doing compliance work, the measures need more care: a missed alert matters more than a slow one. The specific measures are covered in how to measure a compliance agent's performance.

What are the signs the review is going wrong?

It is skipped. Once, because people are busy; then every other week. Agents are never "fine enough" to stop looking.

No decisions are made. Weeks pass with "noted" against every item. Depth should change sometimes, in both directions.

Nothing is ever narrowed. If the only depth changes are widenings, the review is acting as a promotion panel, not a control.

The same item recurs. A problem raised four weeks running without an owner is a problem the meeting has accepted.

Approval rates near total. Approvers who approve almost every held item unchanged may not be reviewing. Test with a sample.

The pack grows. New measures are added and none removed, until the meeting cannot read it. Keep the pack to what drives decisions.

Access is never discussed. The weekly review looks at what agents did, not what they could do. Run a separate periodic access review for AI agents and bring its findings here.

How long should it take?

For a handful of agents, 30 minutes. For many, split by business area, with a short combined session for cross-cutting issues. If it routinely runs over, the pack is too long or decisions are being deferred. The review is one of the routines that keeps an agent within the sequence in train, shadow, certify, widen; it should feel like part of running the operation, not an extra.

Where Pryme fits

In Pryme, the pack is drawn from the append-only record of every verdict, broken down by capability and deciding bound. Drift checks rerun Attest's gates against the live agent, and a failed gate lowers depth the same day, so the meeting reviews a change already made rather than deciding one under pressure.