Mostly skills your operations team already has, applied in new ways. Running agents well depends on people who can define a correct outcome, review work without rubber-stamping it, write a clear brief, read test evidence and act on it, and follow a runbook under pressure. A few technical people cover connections and the platform. Data science is rarely the bottleneck.
Why are the key skills operational rather than technical?
Because the hard questions are about the work, not the model. What counts as a correct answer to this query? Which cases must a person see? Is this source trustworthy enough to act on? When should the agent be given more authority? Those are judgements about a process, and the people best placed to make them are the ones who run it.
Technical skills still matter, but they are concentrated in fewer people. The case is made in more detail in whether you need data scientists to deploy AI agents.
Which skills does the team need, and who holds them?
| Skill | What it looks like in practice | Usually held by |
|---|---|---|
| Defining correct | Writing answer keys and escalation rules for real cases | Senior practitioners, process owner |
| Reviewing with judgement | Checking sources, not just reading the draft; resisting the pull to approve | Reviewers and approvers |
| Writing briefs | Plain, specific instructions with good examples | Practitioners, with help |
| Reading evidence | Interpreting certification scores, shadow reports and drift signals | Process owner, analyst |
| Making depth decisions | Widening or narrowing on evidence, not confidence | Process owner |
| Operating | Following runbooks, lowering depth, routing work, handling incidents | Team leads, on-call staff |
| Connecting systems safely | Credentials, read-only access, integration upkeep | Platform engineers |
| Data protection and risk | Screening new uses, mapping rules to bounds | Risk and privacy specialists |
Two rows are often underestimated. Reviewing with judgement is a skill, not a role description: a reviewer facing a queue of plausible drafts needs to know what the agent tends to get wrong and how to check it quickly. Reading evidence is what lets an owner decide on depth; someone who cannot read a shadow run report will decide on gut feel.
How do you build these skills?
- Start with one team. A pilot is as much a training ground as a test, which is the logic of running a pilot of an AI agent in one team. Evidence: named people who have done each task once.
- Train reviewers on the agent's failure modes, using real examples from shadow and certification. Evidence: training records per reviewer.
- Pair practitioners with someone experienced for their first brief. Evidence: the brief's revision history.
- Have owners present depth decisions with evidence to a peer before acting. Evidence: filed decisions.
- Drill the runbook with on-call staff. Evidence: drill results.
What does a team look like in practice?
Say an insurer's claims operation runs three agents: intake, document chasing and payment preparation. The skills sit with about a dozen people, most of whom were already in claims.
The claims operations manager owns all three agents and makes depth decisions with the risk partner. Two senior handlers write and maintain the briefs and answer keys. Eight handlers review drafts and approve held payments as part of their normal day, after a half-day of training on what the agents get wrong. An analyst shared with another team reads the weekly signals and prepares the review. One platform engineer looks after connections for these and two other teams' agents. Nobody on the team is a data scientist, and nobody needs to be.
How these people are organised is covered in organising an agent operations team, and the day-to-day management of mixed work in managing a team of AI agents and people.
Which controls make skills count?
- Training records for everyone who approves an agent's work, refreshed when the agent changes.
- Calibration sessions so reviewers apply the same standard.
- Written depth decisions, so evidence-reading can be checked.
- Rotation through on-call, so more than one person can follow the runbook.
The full sequence is in our guide to deploying an AI agent.
Questions people ask
Do reviewers need to understand how models work?
They need to understand how this agent fails: what it tends to invent, miss or misread. A general explanation of models helps less than ten real examples of the agent's mistakes.
Should we hire specialists or train existing staff?
Train existing staff for the operational skills, since process knowledge is the hard part to hire. Hire or borrow for platform engineering and, if needed, model risk.
Who trains the team?
The process owner, with the people who ran the pilot and shadow. Vendor training can cover the platform, but not your agents' behaviour on your work.