Tie every answer to one versioned policy source, and test against it whenever it changes. The agent answers only from approved, dated policy; a named owner publishes changes; a test set is replayed before each change goes live; and a person samples live replies. Each step leaves evidence, so you can show which policy version an answer came from.
Why do agents drift from policy?
Usually because there is more than one version of the policy. The help centre says returns within 30 days. An internal guide says 28 days for sale items. An old email from a team lead says sale items cannot be returned at all. A person learns which to trust. A model, given all three, blends them, and the answer depends on which text it happened to retrieve.
The second cause is the model's general knowledge. Asked about a cooling-off period, an agent with no policy text will answer from what is common in the industry, which may not be your rule. Both causes have the same fix: one source, and nothing outside it. The related problem of invented answers is covered in how to stop a support agent from hallucinating answers.
What are the steps, and who does each?
- Policy owner: consolidate. One person per policy area gathers every version, settles the conflicts and publishes a single text. Evidence: the published policy, with its version number and the conflicts resolved.
- Policy owner: mark what the agent may say. Some policy is for staff only, such as discretion thresholds. Mark those passages so the agent can use them for routing but never quote them. Evidence: tagged sections.
- Agent owner: restrict sources. The agent's brief points only to the approved policy library. Old help articles, email threads and shared drives are removed from its retrieval. Evidence: the source list in the brief, with versions.
- Agent, depth 1: cite. Every policy answer carries the reference of the clause it used, visible to reviewers if not to the customer. Evidence: the citation in each filed answer.
- Agent: refuse when there is no clause. If no clause covers the question, the agent says it cannot confirm and hands over, rather than reasoning from general knowledge. Evidence: a log of uncovered questions.
- Policy owner: fill gaps. The uncovered-questions log is reviewed on a set cadence and the policy is extended where needed. Evidence: dated changes linked to the gaps that prompted them.
- Quality team: replay on change. Before any policy change goes live, the test set for that area is replayed, with expected answers updated. Evidence: the replay result, signed off.
- Quality team: sample live replies. A person checks a sample of live answers against the current policy each week. Evidence: the sample, scored.
The pattern holds where several agents share the same policy facts; keeping their common memory aligned is the subject of how to keep agents' shared memory consistent.
What does a policy change look like in practice?
Take a regional insurer changing its excess for windscreen claims from 100 to 75.
The policy owner updates the clause and publishes version 4.2 with an effective date. The quality team replays the windscreen test cases against the new version; three expected answers are updated and all pass. On the effective date the agent's source switches to 4.2. A week later, the weekly sample includes a customer whose policy started before the change. The agent quoted 75, but their contract still carries 100 until renewal.
That is a policy gap, not an agent error: the clause did not say which customers it applied to. The owner adds the condition, the test set gains a case for pre-change customers, and the replay is run again. Insurers see the same issue at every renewal, as covered in handling policy renewal questions with an AI agent.
Which checks catch what slips through?
- Output check for commitments. A rule that scans replies for promises, figures and dates and confirms each against the source. The method is in checking support replies for promises the business cannot keep.
- Consistency tests. The same question asked several ways should get the same answer. Paraphrase sets in the test suite catch retrieval that depends on wording.
- Version stamps. Every filed answer shows the policy version it used, so a complaint can be checked against the rule in force at the time.
When an inconsistent answer does reach a customer, the version stamp tells you whose fix it is: the policy owner's if the text was wrong, the agent owner's if the agent strayed from it. That division is explored in who answers for an AI agent's mistake.
Where Pryme fits
In Pryme, the policy library arrives through the Pryme Context Protocol as a sealed source, so the agent can tell an approved clause from a forwarded email. Attest replays your cases on each change, and its grounded and stable gates check that each answer comes from the approved source and that the same question gets the same answer. More is in the AI support agent that knows when to say no.
Questions people ask
Should the agent tell customers which policy clause it used?
For complex answers, yes: a reference to the published terms helps the customer check. For simple ones, it adds noise.
What if two policies conflict and the owner has not resolved it?
The agent should not choose. It refuses to confirm and hands over, and the conflict goes on the owner's list.
How often should the test set be replayed?
On every policy change in its area, and on a fixed schedule even when nothing changes, because model updates and retrieval changes can shift answers too.