Run the weekly AI agent review from the record, not from impressions. The agenda below takes 45 minutes: certificate status first, then the week's verdicts by type, overturned approvals, refusals customers queried, missed response targets, people's certificates, and actions with owners. Every item starts from a number or a case someone can open.
What is the meeting for?
It is where the team that runs an agent decides what to change. The agent does not learn from its mistakes by itself; we change its brief, sources, bounds and certification. People do learn, but only if someone points to the cases. The weekly meeting is where both happen.
The meeting fails in two common ways. It turns into a status update where everyone says things went fine, or it becomes a complaint session about the agent with no cases on the table. Building the agenda from the record prevents both. A wider guide to the session is how to run a weekly review of AI agent decisions with a team, and the performance-focused variant is how to run a weekly AI agent performance review.
Who attends and what do they bring?
- Chair: the operations manager or team lead who owns the agent's day-to-day running.
- Reviewers: one or two who worked the review queue that week, bringing cases they found hard.
- Builder: brings anything that changed in the brief, sources or bounds, and the latest certification results.
- Risk or compliance: attends monthly, or weekly for agents at depth 4.
Each attendee pulls their own section from the record before the meeting. Nobody presents slides.
How do you fill in the agenda?
Use the same order each week so trends are visible. For each item, fill the "From the record" column before the meeting and the "Decision" column in it.
The template
| No. | Item | Minutes | From the record | Question to answer | Decision and owner |
|---|---|---|---|---|---|
| 1 | Agent certificate status | 3 | Current certified depth, certificate expiry, any failed runs | Is any certificate due to lapse within 30 days? | |
| 2 | Verdict counts | 5 | Allowed, held, refused, withheld this week and last | Did any count move sharply, and why? | |
| 3 | Overturned approvals | 10 | Approvals later reversed or corrected, by reviewer | Was it the agent's draft, the reviewer's check, or both? | |
| 4 | Refusals queried | 7 | Refusals a customer or colleague challenged | Was the refusal right? If so, is the explanation clear? | |
| 5 | Held items over target | 5 | Items past their response target, by class and shift | Rota problem or volume problem? | |
| 6 | Source and provenance issues | 5 | Holds caused by conflicting or external sources | Does a source need fixing or reclassifying? | |
| 7 | People | 5 | Reviewer certificates expiring, new joiners in shadow, coaching outcomes | Who renews, who moves out of shadow? | |
| 8 | Actions | 5 | Last week's actions | Closed, continuing or escalated? |
Withheld counts belong in item 2 even though nobody can release them. A rise means the agent, or people prompting it, are attempting actions the platform floor blocks, which is worth understanding.
A worked example
A hypothetical KYC team reviews its onboarding agent. Values are samples.
| No. | Item | From the record | Decision and owner |
|---|---|---|---|
| 1 | Certificate status | Certified at depth 3; risk rating drafted at depth 2; certificate expires in 24 days | Builder books the recertification run this week |
| 2 | Verdict counts | Held rose from 40 to 95; refused steady at 12; withheld 0 | Investigate under item 6 |
| 3 | Overturned approvals | 2: both approved proof of address from a utility bill image | Team lead coaches both reviewers; add case to calibration set |
| 4 | Refusals queried | 1: refused a company director's passport photo as unreadable | Refusal right; reviewer to improve the customer message template |
| 5 | Over target | 6 items, all Friday evening | Ops manager adds Friday cover |
| 6 | Source issues | Company registry feed down for two days; agent held every business case | Builder confirms the feed is back; no change to bounds |
| 7 | People | Reviewer C's certificate expires next month; Reviewer F ready to leave shadow | Reviewer C books renewal; team lead signs off F |
| 8 | Actions | 3 of 4 closed | One carried to next week |
Item 6 explains item 2. Held items doubled because a sealed source went down, so the agent did exactly what it should: it held rather than relying on declared information. Without the record, the team might have concluded the agent had become overcautious and loosened its bounds. KYC review in more depth is covered in how KYC analysts review an AI agent's decisions, and a property sector view in how property managers review an AI agent's decisions. For who needs which skills to take part, see AI agent skills by role.
Questions people ask
How long should the meeting be?
Forty-five minutes for one agent; add fifteen for each further agent the same team runs. If it regularly overruns, split the agents across two meetings.
Should the meeting be minuted?
Record decisions and owners, not discussion. File them with the agent's records so a later review can see why a bound or source changed.
What if nothing interesting happened that week?
Spend the time on calibration: pick three held cases and have each reviewer decide them separately, then compare. A quiet week is the best time to check consistency.