NIST AI 600-1, the Generative AI Profile, applies the NIST AI RMF to systems built on generative models, which includes almost every AI agent in use. It names risks that generative AI creates or worsens and suggests actions for each. For agents, confabulation, human-AI configuration, information security and value chain risks matter most, because an agent acts on its output.
What does the Generative AI Profile set out?
The profile is a companion to the NIST AI Risk Management Framework 1.0. It keeps the four functions, Govern, Map, Measure and Manage, and adds a catalogue of risks that are unique to generative AI or made worse by it. Among them:
- confabulation: confidently stated output that is false or unsupported;
- data privacy: leakage or inference of personal information;
- information security: new attack routes, such as instructions hidden in content the system reads;
- information integrity: content that blurs fact and fabrication, and the difficulty of tracing where content came from;
- human-AI configuration: people relying too much or too little on the system, or treating it as a person;
- harmful bias and homogenisation: unfair outputs and a narrowing of views;
- value chain and component integration: risks from third-party models, data and components that are hard to inspect;
- intellectual property, harmful content, and environmental impacts.
For each, the profile offers suggested actions mapped to the framework's functions. It also highlights four themes that cut across them: governance, content provenance, pre-deployment testing and incident disclosure. Like the framework, the profile is voluntary. Our glossary entry on the NIST AI Risk Management Framework gives the background.
Who does it apply to?
Organisations that build or use generative AI and choose to follow the framework. It is written for both developers of models and organisations that deploy systems built on them. A firm running agents on a third-party model is a deployer of a generative system and the profile applies to its use.
Which of its risks change when the system acts?
The profile was written mostly with content generation in mind. An agent turns generated content into actions, which sharpens four of the risks.
Confabulation becomes a wrong action. A chatbot that invents a policy term gives a wrong answer. An agent that invents one may decline a claim on the strength of it. The profile's actions on verifying outputs against sources become, for an agent, a requirement that every fact it acts on can be traced. Our companion piece on the Generative AI Profile and grounding agent answers goes deeper.
Information security becomes control of tools. Instructions hidden in a document or email are a nuisance for a chatbot. For an agent with access to a payment tool, they are a route to fraud. The profile's security actions have to reach the boundary where the agent calls a tool.
Human-AI configuration becomes the design of oversight. The profile warns of automation bias. Agents make it concrete: reviewers of held actions start approving in bulk. The answer is to design who reviews what, with what information, and to measure whether reviews change outcomes.
Value chain risk becomes a moving target. An agent depends on a model provider that can change the model. The profile's call to monitor third-party components means re-testing the agent after each change, not trusting a release note.
Which actions does the profile point to for agents?
Take an illustrative case. A health insurer runs an agent that answers members' questions about cover and drafts pre-authorisation letters. Using the profile, the team finds confabulation risk in cover answers, privacy risk because the agent sees health data, and information security risk because members can attach documents. The actions it takes: every cover answer must cite the policy wording it relies on; the agent cannot see more health data than the case in hand; attached documents are treated as untrusted and can never trigger an approval; and the agent is re-tested after every model change before it drafts letters again.
| Requirement | Control | Evidence it leaves |
|---|---|---|
| Govern: generative AI policy | Policy covering agent authority, model changes and acceptable sources | Approved policy |
| Confabulation | Agent may act only on facts traceable to a provided source | Source citations per answer and action |
| Content provenance | Each input labelled with its origin and trust level | Provenance labels in action records |
| Information security | Untrusted inputs cannot authorise actions; tool calls checked against limits | Refused and held verdict records |
| Data privacy | Agent sees only data needed for the case | Access scope per capability |
| Human-AI configuration | Reviewers see facts and sources; override rates tracked | Review records; override statistics |
| Pre-deployment testing | Testing on the firm's own cases before the agent acts | Test reports by capability |
| Value chain | Model change triggers re-testing | Change notices tied to results |
| Incident disclosure | Defined process for recording and reporting agent incidents | Incident log |
What will a supervisor, auditor or customer ask?
- How do you know the facts your agent acts on are real?
- What stops an instruction hidden in a customer document from triggering an action?
- How do reviewers avoid simply approving whatever the agent drafts?
- What happens when your model provider updates the model?
- How do you record and disclose agent incidents?
In banking, a further question is whether the agent counts as a model; our piece on PRA SS1/23 and generative AI agents addresses it.
What does the profile not cover?
It does not set thresholds, mandate controls or certify anything. It does not address agent-specific questions in detail, such as how much authority each capability should have or how to enforce limits at the moment of action. It does not replace binding law. For how the framework's four functions apply to agents, read how the NIST AI RMF applies to AI agents; for the binding rules, see our guide to AI agent regulation mapped to controls.
This is an explanation of the rules, not legal advice; confirm how they apply to you with counsel.
Questions people ask
Does the Generative AI Profile replace the NIST AI RMF?
No. It is a companion that applies the framework to generative AI. You still use the four functions; the profile adds generative-specific risks and suggested actions within them.
Is the profile written for model developers or for deployers?
Both. Many suggested actions fall on whoever builds the model, but a large share apply to organisations that deploy systems on top of third-party models, which is where most agent users sit.
Which risk in the profile is most often underestimated for agents?
Human-AI configuration. Firms plan for the agent to be wrong, but not for people to stop checking it. The profile's warning about over-reliance becomes, for agents, the most common route to unsupervised action.