A canary release is a way of rolling out a new version by sending it a small share of real work first, watching closely, and widening its share step by step only while it behaves well. For an AI agent, the canary should be certified before it starts and begin where a mistake is easiest to catch.
Why does a canary release matter for agents?
An agent change can look fine on past cases and still behave differently on live work: new phrasings, new document types, edge cases the test set missed. A canary limits how many real cases a bad version can touch before someone notices. If the new version goes wrong on a tenth of the work for an hour, the damage is a tenth of what a full switch would have caused.
The name comes from the canary once taken into mines as an early warning.
How does it work in practice?
- Certify the new version at the capabilities and depths it will hold. A canary is a check on live behaviour, not a substitute for certification.
- Choose the slice. Start with low-value, reversible cases, or with cases where a person reviews every action anyway.
- Set the stop conditions before you start: hold rate, refusal rate, reviewer edits, errors, complaints.
- Widen in steps, for example from a twentieth to a quarter to half to all of the work, pausing at each step.
- Stop and roll back at once if a stop condition is met.
- Record each step, the measures seen and who approved widening.
Where should a canary start?
Start where a person still stands between the agent and the outcome. A practical rule: run the canary first on a capability at depth 2, where every action is drafted and a named person releases it. The canary's mistakes then surface as reviewer edits or refusals, not as customer harm.
Payment work shows this well. Suppose a finance team updates the agent that prepares bank payment files for release. The new version is canaried on a share of daily files. Because a person reviews and releases each file, using a high-value payment release checklist for the larger ones, a canary error is caught at the release step. Only once the canary has run clean does the new version take all files.
What does an example look like?
A lender changes the brief of the agent that drafts arrears letters. The new version is certified, then given one branch's letters for a week, all still approved by an arrears officer before sending. Officers edit the new drafts no more than the old ones, and no complaints arise. The share rises to a third of branches, then all. At one step, edits on letters for customers in hardship rise; the team pauses, fixes the brief, recertifies and resumes.
How is a canary release different from blue-green deployment?
Blue-green switches all work to the new version at once and relies on a quick switch back. A canary moves work across gradually and relies on catching problems while exposure is small. A canary is slower but touches fewer cases if something is wrong. Many teams canary first and use a blue-green switch for the final step.
It also differs from shadow mode, where the new version acts on nothing at all and its outputs are only compared. Shadowing comes before a canary; a canary is the first time the new version's work counts.
Which terms sit close to it?
Rollback: the route back if the canary fails. Feature flag: the switch that often controls the canary's share. A/B testing: comparing two safe versions on live work. Change management: the approval around each widening step. For the wider process, see our guide to deploying AI agents.
Questions people ask
How long should a canary run?
Long enough to see the full range of case types the agent handles, including weekly or month-end patterns. For many processes that means days, not hours.
Can a canary run at full depth?
It can if the new version is certified for that depth, but starting where a person reviews each action gives you a cheaper early warning.