How to test whether an AI agent stays in scope

Scope testing starts with a written boundary: which customers, topics, records, systems and actions the agent is for. Test the edges: requests just outside the role, extra tasks slipped into ordinary ones, and tools that could reach more than the task needs. Score what it said, what it did and what it opened. The owner decides where the boundary sits.

What does staying in scope mean, and who owns it?

Scope is the agent's role: the work it exists to do. An agent that handles card disputes is in scope when it gathers transaction evidence. It is out of scope when it answers a question about mortgage rates, updates the customer's address while it is there, or reads the customer's savings account because the data was in reach. None of these may look harmful. Each is work nobody certified.

Scope failures matter more as depth rises. A reading agent that strays wastes time; an agent at depth 3 or 4 that strays takes actions nobody checked. The scope gate is one of the seven, and it must clear its floor on its own.

The capability owner owns the scope. Risk or compliance should review it where straying could become regulated activity, such as giving advice.

What is the working method?

1. Person, owner: writes the scope statement. In the brief: customers and products covered, topics the agent may discuss, records it may read, systems and actions it may use, and an explicit out-of-scope list.
2. Person, owner: decides the response. For out-of-scope requests, whether the agent refuses with a route forward or escalates.
3. Person: builds core cases. Ordinary in-scope work, to confirm the agent does not shrink its own scope by refusing what it should do.
4. Person: builds boundary cases. Requests just outside the role: a neighbouring product, a related topic, a question that edges into advice.
5. Person: builds "while you're at it" cases. An in-scope task with an extra request tacked on: "and can you also change my phone number?"
6. Person: builds reach cases. Tasks where the agent's tools could open records it does not need. The test is whether it opens them.
7. Agent, replay only: works every case.
8. Scorers: score three things. What it said: did it answer out-of-scope questions? What it did: did it take or attempt actions outside its role? What it opened: did tool logs show access to records the task did not need?
9. Person, owner: reviews boundary results and decides whether the scope statement needs changing. Widening scope means new cases and a new certification.

What decisions does a named person keep?

Where the boundary sits. The agent cannot decide its own role. If cases show the boundary is unclear, the owner clarifies it in the scope statement, not by letting the agent use its judgement.

What happens at the boundary. Refuse or escalate, and with what wording. Testing how the agent declines is shared with testing the agent's refusals.

Whether to widen scope. Sometimes testing shows customers keep asking for something just outside the role. Adding it is a decision for the owner, followed by new cases and a new run. It is never something the agent grows into.

What record does it produce?

• The scope statement, versioned with the brief.
• The case set, labelled core, boundary, extra request and reach.
• Per-case results for speech, action and access, with tool access logs attached.
• Boundary decisions by the owner, with reasons.
• Platform verdicts for any out-of-scope action attempted, each naming role scope as the bound that decided.

What should you measure?

• Out-of-scope answers, by type. Advice-like answers matter most.
• Out-of-scope actions attempted. Each one was stopped by the platform; each one also shows a weakness in the brief.
• Records opened beyond the task's need. Often the least visible failure and a data protection concern.
• In-scope requests refused. Over-narrow scope is a failure too.

Where does it usually go wrong?

Helpfulness. Models are trained to be helpful, and adjacent tasks feel like good service. The scope statement must say plainly that adjacent tasks are handed off, not done.

Broad tool access. The agent was given read access to the whole customer record because that was easier to connect. Scope testing should push for the narrowest access the task needs.

Scope that lives only in the instructions. If the boundary is described in the brief but the agent's connections reach far beyond it, the instructions are doing all the work. The minimum chain enforces role scope at the moment of action, and every out-of-scope attempt is filed as refused, naming role scope as the bound that decided. Test that this layer works, as well as testing what the agent says.

Scope confused with grounding. An agent can stay in scope and still make unsupported claims, or stick to its sources while straying from its role. They are separate tests: see testing whether an agent's answers are grounded in sources and how to test whether an agent stays grounded in its sources.

The framework is in what it means to certify an agent.