Confabulation is the NIST Generative AI Profile's word for confident output that is false or unsupported. For an AI agent, the remedy is grounding: it may state or act on a fact only if that fact traces to a given source, and it stops when sources are silent. Provenance then decides how far each source may be trusted.
What does the profile say about fabricated output?
NIST AI 600-1, the Generative AI Profile, is a companion to the NIST AI Risk Management Framework 1.0. It lists confabulation among the risks that generative AI creates or worsens: models can produce plausible content that is wrong, including invented citations, and can present it with unwarranted confidence. It links this to a second risk, information integrity, where it becomes hard to tell fact from fabrication or to trace where content came from.
The profile's suggested actions, in substance, include reviewing and verifying the sources and citations in generated output, testing systems for confabulation before deployment, tracking the provenance of content, and keeping people able to check what the system relies on. It is voluntary and does not prescribe methods. For the profile's wider scope, see the Generative AI Profile and AI agents.
Who does it apply to?
Organisations that build or deploy generative AI systems and choose to follow the NIST framework. For agents, confabulation sits mostly with the deployer: the model provider cannot know which facts your agent should rely on, so only you can define and enforce what counts as grounded.
Why is grounding harder for an agent than for a chatbot?
Grounding means tying output to supplied sources rather than to whatever the model absorbed in training. Our glossary entry on what grounding is covers the basic idea. Three things make it harder, and more important, for agents.
Agents act on facts, not just state them. A chatbot that confabulates a fee gives a wrong answer that a customer may question. An agent that confabulates a fee may charge it. Grounding has to apply to the fact that justifies an action, not only to the words in a reply.
Grounded is not the same as true. An agent can faithfully cite a document that is itself wrong, stale or planted. An emailed invoice and the general ledger are both "sources", but they deserve very different weight. Grounding without provenance gives a false sense of safety.
Silence must stop the agent. The most dangerous confabulations fill gaps. When no source answers the question, a well-grounded agent says it does not know and hands the case to a person. Testing has to include cases where the right answer is "I cannot tell from what I have".
What does grounding look like in practice?
Grounding works at the level of the claim. Every factual statement in an answer, and every fact on which an action rests, should carry a pointer to its source. Each source should carry a label for where it came from and how far it can be trusted. And certain actions should require facts of a certain strength.
Take an illustrative case. A commercial bank's agent helps relationship managers prepare covenant reviews. A client sends a spreadsheet claiming its leverage has fallen below the covenant limit. The agent's draft quotes the figure but labels it as declared by the client, and sets it beside the last audited figure held in the bank's own records. It does not mark the covenant as met. The relationship manager sees both, with their sources, and decides. Without labels, the draft would have stated the client's figure as fact.
The firm's tests then score grounding directly: what share of claims in each answer trace to a source, how often the agent declines when sources are silent, and whether it ever relies on a weak source for an action that needs a strong one. Any unsupported claim in an action is a failure, whatever the average.
How do you meet the profile's concerns with grounding?
| Requirement | Control | Evidence it leaves |
|---|---|---|
| Verify sources in output | Every claim and every action-justifying fact cites a supplied source | Claim-level citations in answer and action records |
| Content provenance | Each source labelled by origin and trust level | Provenance label on each fact used |
| Confabulation testing | Pre-deployment tests on the firm's own cases, including unanswerable ones | Test results with grounding and abstention scores |
| Information integrity | Weak sources cannot justify high-impact actions | Refused or held verdicts citing source strength |
| Human review | Reviewers see the facts relied on and their sources | Review records showing source evidence |
| Ongoing monitoring | Unsupported claims tracked in production | Grounding failure reports |
What will a supervisor, auditor or customer ask?
- For this decision, show us each fact the agent relied on and where it came from.
- What does the agent do when its sources do not answer the question?
- Can a document from outside the firm cause the agent to act?
- How do you test for invented facts, and what was the last failure you found?
- How would a reviewer know that a figure came from the customer rather than your records?
In UK banking, these questions also arise under model risk rules; our piece on PRA SS1/23 and generative AI agents covers that angle.
What does the profile not cover?
It does not define a grounding standard, set an acceptable rate of unsupported claims, or say which sources are strong enough for which actions. It does not require provenance labels in any particular form. Those choices are yours, and binding rules such as consumer protection and model risk guidance will judge the outcome. For the binding side, see our guide to AI agent regulation mapped to controls.
This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.
Where Pryme fits
The Pryme Context Protocol stamps every fact reaching an agent with where it came from: sealed, signed, declared or external, strongest first. An external claim cannot authorise a payment. Attest's grounded gate scores whether answers trace to sources, and its refusal and escalation gates score what the agent does when they run out.
Questions people ask
Does retrieval-augmented generation solve confabulation?
It helps but does not solve it. Retrieval gives the model relevant sources, but the model can still ignore them, misread them or fill gaps. You still need claim-level citation, abstention when sources are silent, and testing.
Should an agent ever answer from general knowledge?
For low-stakes explanations, sometimes, if it says so. For anything that affects a customer's money, rights or records, no. Those answers should rest only on the firm's sources.
How do you test whether an agent abstains correctly?
Include cases in your test set where the sources do not contain the answer. Score the agent on whether it says so and escalates, rather than on whether it produces something plausible.