How to hire for AI operations roles

Test judgement on real agent output, not enthusiasm for AI. Give reviewer candidates a held action with mixed sources and ask what they would send, correct or pass up. Give agent owner candidates a run of verdicts and ask what they would change in the brief, sources or limits. Hire the people who know when to stop.

What are AI operations roles, and who owns hiring for them?

AI operations roles are the people who run agents day to day. Reviewers work the review queue: they check what an agent drafted or held, correct it and send it, or pass it up. Agent owners answer for one agent's results and bounds. Approvers release held actions that move money or are hard to undo. Our guide to AI agent skills by role sets out what each role needs to know.

The hiring manager is usually the process owner whose team the agent serves. They should write the role from the role sheet, not from an old job description. If you have not yet written that sheet, start with defining AI agent roles and responsibilities in a team, because you cannot interview for a role nobody has described.

What are the steps?

1. Hiring manager: writes the role from the agent's capabilities. Which depths will this person review or approve? Which queues will they cover? Which refusals will land on them?
2. Hiring manager: builds two work samples from anonymised past cases: one held action and one short run of verdicts. Use real formats from your own workspace with names and amounts changed to plain sample values.
3. Candidate: reviews the held action. The agent, at depth 2, has drafted a supplier payment instruction. One source is a sealed ledger entry; another is an emailed invoice, which is external. The candidate must decide what to do and say why.
4. Candidate: reads the verdict run. Ten verdicts: allowed, held, refused and one withheld. The candidate explains each one and names what they would ask the agent owner.
5. Panel: interviews for stopping. Ask about a time the candidate refused to sign something off, and what happened next.
6. Second line: joins one interview for approver roles. Compliance or risk tests whether the candidate understands separation of duties.
7. Hiring manager: checks credentials at source. Where a candidate claims a certificate, check its number with the issuer, not the paper.

What should the work samples test?

For reviewers, three things. First, provenance: do they notice that an emailed invoice is external and cannot authorise a payment, however official it looks? Second, scope: do they see when the agent's draft promises something outside its bounds, such as a refund above the limit? Third, the pass-up: do they write a reason a busy approver can act on?

For agent owners, the test is pattern reading. Give them ten verdicts in which the same refusal appears three times for the same reason. A good candidate sees that the refusal is either correct and the request should never reach the agent, or that a limit is set too tight and should go to the process owner. A weak candidate wants to "turn the agent up". Depth is earned through certification on your own cases, not configured, and candidates should know that or learn it quickly.

For approvers, add a held depth 4 action where everything looks right except that the approver themselves set the limit it was checked against. The right answer is to pass it to someone else.

Which controls keep hiring fair, and what evidence do they leave?

Same samples for every candidate. Evidence: the sample pack, versioned, with each candidate's written answers.

Scored against a written key. Two scorers mark independently before discussing. Evidence: the two score sheets.

Credentials checked at source. Evidence: the certificate number and the check result on the candidate file.

No sample built from live customer data. Evidence: the anonymisation note on the pack.

What should you measure?

• Practicum outcomes of new hires in their first months: how many review decisions are correct and whether any red line was missed.
• Time to first unsupervised approval.
• Pass-ups with no reason from new reviewers, which should fall quickly.
• Work-sample score against later performance, to see whether your samples predict anything.

Sector pages on which roles in retail banking need AI agent training and which roles in consumer lending need AI agent training help tailor the samples to regulated work.

Where does it usually go wrong?

Hiring for prompt skill. Candidates who write clever prompts are not necessarily good reviewers. The job is to judge output and stop bad actions, not to coax the model.

Hiring only from the old team. Experienced staff know the process, which is valuable, but test them on agent output too. Knowing the work is not the same as knowing when an agent has got it slightly wrong.

No sample at all. A conversation about AI tells you little. Twenty minutes on a held action tells you a lot.

Ignoring the onboarding. A good hire still needs time on your agents. Treat their first weeks the way you would onboard an AI agent like a new hire: narrow scope first, more as evidence builds.

Where the Academy fits

Pryme Intelligence Academy certificates are checkable at source: anyone can look up a certificate through VerifyDoc.ai by number or QR code, and a lapsed or revoked one shows as such. A public register lists certified people by name, credential and expiry. For hiring managers this turns a claimed credential into a fact in seconds, and the Certified Pryme Operator practicum means the holder has already made real review decisions under assessment.