Support staff who review an AI agent's work keep this log for a two-week practicum. Every held reply, refund, account change or handover they decide gets a row: what the AI agent drafted, what the person changed and why, what they decided, the team lead's independent decision, and whether an identity, authority or complaint red line was involved. Template below.
Why log edits as well as decisions?
Because most of a support reviewer's work is editing, not approving or rejecting. The AI agent drafts a reply at depth 2; the reviewer sends it as it is, edits it, or rewrites it. A log that only records "sent" or "not sent" misses whether the reviewer improved the reply or made it worse.
So this log has a column for edits, classified simply: none, wording, facts, or outcome. A wording edit changes tone. A facts edit corrects something the AI agent got wrong. An outcome edit changes what the customer receives, such as a larger refund or a different remedy. Outcome edits are the ones the team lead should look at closely.
What reaches a support reviewer?
In a typical deployment, the AI agent resolves simple contacts end to end within bounds at depth 3, and holds or hands over the rest:
- drafted replies on topics outside its routine set;
- refunds and replacements above its limit, held for a named person;
- account changes, such as a new email address or delivery address, held until identity is verified;
- complaints, which go to a person by rule;
- contacts the AI agent refuses, where the customer then asks for a person.
Handovers deserve their own entries. When the AI agent passes a conversation to a person, the reviewer inherits its summary, and a poor summary leads to a poor decision. The log records whether the summary was accurate.
The practicum log template
Header: reviewer; team lead with a current certificate; queues; practicum refund limit; dates; bar of at least 95 correct in every 100 and no missed red line.
Red lines
- Changing account details without completing identity verification.
- Approving a refund or replacement above the reviewer's limit.
- Treating a complaint or a vulnerability signal as a routine query.
- Sending a reply that promises something outside policy.
| Column | Entry |
|---|---|
| No. and date | |
| Ticket and channel | |
| Contact reason | |
| What the AI agent did | Drafted, held, refused or handed over; deciding bound |
| Handover summary accurate? | Yes / partly / no / not a handover |
| Reviewer's decision | Send, edit and send, approve, decline, escalate |
| Edit type | None / wording / facts / outcome |
| Reason | |
| Team lead's decision | Recorded before seeing the reviewer's |
| Match | Yes / no / defensible |
| Red line | None / caught / missed |
| Note |
A worked example
Take a sample meal-kit subscription company. Values are illustrative.
| No. | Contact | What the AI agent did | Reviewer | Edit | Team lead | Match | Red line |
|---|---|---|---|---|---|---|---|
| 140 | Box arrived warm; customer asks for a refund of 45 | Refund held, above the AI agent's limit of 30 | Approve 45 | None | Approve | Yes | None |
| 141 | Customer asks to change delivery address and email in one message | Change held pending verification | Approve both changes; message came from the account email | Outcome | Hold; verify by code to the phone on file, because the email address itself is being changed | No | Missed |
| 142 | Third late delivery; customer says they will "go to the papers" | Handover with summary "late delivery" | Treat as complaint; log, apologise, offer credit within policy | Facts | Same | Yes | Caught |
| 143 | Customer asks if a recipe is safe for someone with a nut allergy | Draft says "this recipe contains no nuts" | Edit to say the box is packed where nuts are handled and point to the allergen page | Facts | Same | Yes | None |
Row 141 fails the bar. The request looked safe because it came from the account's own email address, but when the email is the thing being changed, it cannot also be the proof of identity. Row 143 shows a facts edit that mattered: the AI agent's draft was accurate about the recipe and wrong about the risk.
How do you read it at the end?
Count outcome edits and check each one. A reviewer who changes many outcomes may be generous with refunds, or the AI agent's limits may be set too low; the team lead should know which. Check the handover column too: if summaries are often marked "partly", that is feedback for whoever maintains the AI agent's brief.
Team leads keep a related log of their own, with coaching in place of edits; see the practicum log for support team leads. IT administrators share the identity red line from a different angle; compare the log for IT administrators. Background on the method is in what a practicum is and what a practicum means in AI agent training, and the setup in how to run a two-week practicum for customer support agents. For the wider picture, see AI literacy and certification for people who work with agents.
Questions people ask
Do contacts the AI agent resolved alone belong in the log?
No, only decisions the reviewer makes. The team lead can sample resolved contacts separately to check quality.
How many entries should a fortnight produce?
In a busy queue, 300 or more. If volume is low, aim for at least 200 so a bar of 95 in every 100 is meaningful.
What if the reviewer is right and the team lead is wrong?
Mark it defensible, discuss it, and record the reason. A log that rewards only agreement teaches people to guess.