How to Plan a Staged Rollout With AI
Turn a release into a staged rollout plan you can defend — cohorts built from your risk rather than round numbers, gates that widen on evidence instead of a timer, and a written difference between pausing and rolling back.
Get the Staged Rollout PromptThe flag has been at ten percent since March
Nobody decided to stop there. The first two stages went fine, the numbers were unremarkable, and then a quarter happened — an incident somewhere else, a reorg, a launch that mattered more — and widening the rollout stopped being anybody's Tuesday. Ask the team and the answer is honest and slightly embarrassed: it's fine, we just haven't got back to it.
Meanwhile a tenth of your users are on one product and the rest are on another. Support answers questions about both and has learned to ask which one you're on. Every feature built since has to work with either. The flag is now load-bearing, and the branch it guards has quietly become a permanent fork that somebody will be reasoning about in eighteen months. This is the most common way a staged rollout fails, and it never looks like failure — the stages that ran, ran fine. What was missing was a sentence saying what finished meant, and somebody whose job it was to notice it hadn't happened.
Exposure is a separate decision from deployment
The deploy is one event: the code reaches an environment, and either it's there or it isn't. Exposure is a schedule — who meets the new behavior, in what order, and what has to be true before the next group does. Different documents, because they fail differently and have different owners. Conflating them produces the plan everyone has seen, which is a deploy procedure with "start with a canary" at the top and nothing underneath about how the canary gets read.
The reading is the whole difficulty. A deployment step verifies something deterministic: the health check returns 200, the row count is zero, the version string matches. A rollout stage asks a statistical question — is this group doing worse than they were? — and the honest answer is often that you cannot tell yet, because the metric that would tell you needs a week to arrive and this cohort is too small to move it. A plan that can't say "we don't know yet" will instead say "looks fine," which is the same sentence with the uncertainty deleted.
A model will draft one of these from what you give it — the change, the population, the segments you already know are strange, the signals you already collect, and what you cannot undo. That is real help, and a draft is where it stops. It has never seen your traffic, does not know your enterprise accounts have contractual guarantees about result order, and cannot know your outcome metric is attributed on a seven-day window. NewPrompt's part is narrower still: it builds the prompt, in your browser. It doesn't deploy, doesn't touch a feature flag or a traffic router, doesn't pick which users are in a cohort, doesn't change anyone's exposure, doesn't read your telemetry or analytics, doesn't watch a rollout, and doesn't advance, pause, or reverse anything. Cohorts, percentages and bake times that come back in a draft are guesses in a confident font. You run the prompt in your own AI tool, and the release owner checks every number against real data and real system behavior before one extra person sees the change.
Step 1: Draw the line around what you are exposing
Write what changes from the user's side, not the repo's. "The ranker behind job search" is a rollout subject. "PR #4412" is not — a stage boundary has to be something you can describe to support in two sentences.
Then, before any cohort: who is eligible, and who is carved out. The exclusions are where the work is, and they are rarely technical. Accounts with contractual guarantees about how your product behaves. Users already inside another experiment, because two changes on one population produce two results that mean nothing. Internal accounts doing QA, whose searches are not demand. Each exclusion needs a reason and an owner — and one thing more that almost every plan misses: an end. An exclusion is a promise that somebody keeps making, usually with the same flag you're trying to remove, and an exclusion with no resolution path is how a rollout acquires a permanent branch while looking complete.
One sentence more, and it shapes everything: what do you actually believe might go wrong? Not "it might be worse" — the specific asymmetry. A better order for the median searcher can be a worse order for a narrow one, and the people who can't find anything won't file a ticket, they'll leave. That sentence determines which signal you watch and which cohort goes first. Written down, it also survives the meeting where somebody suggests starting at fifty percent.
Step 2: Build cohorts from your risk, not from round numbers
One percent, ten, fifty, a hundred. Everybody writes it; almost nobody derives it. It isn't wrong so much as unearned: those cuts encode a claim about how much exposure your signals can pay for, and that arithmetic usually hasn't happened. A percentage nobody derived is still a percentage people will defend in a meeting.
So build the sequence from what each stage is for. The first group exists to find out whether the thing works at all — usually internal, and worth being clear-eyed about, because staff are not your users and a cohort that can't produce your outcome signal can't advance on it. The second exists to find out whether it's better, which means a real population, big enough to move the metric inside the bake, ideally one that complains loudly. Later stages exist to find out whether it survives scale and the segments you were nervous about, which is why the ones you're nervous about go last rather than never.
When the model hands you a size and you don't have the traffic number to check it against, that's not a gap to fill with a plausible one. Write TBD, name the owner, and say what would settle it — not just "the segment's share" but the arithmetic underneath: the base rate, the smallest change worth catching, the variance. A pilot whose size you can't justify has a specific failure mode: too small to read, so the stage bakes, produces nothing, and advances on the absence of bad news.
Step 3: Say what each stage has to prove before the next one opens
Three gates per stage, and they're different objects:
Two rules the three-gate structure needs, and most plans leave both implicit. First, precedence: gates fire on independent signals, so two can be true at once — latency bad, outcome good — and the plan has to already know which wins. Rollback beats pause beats advance, written once, at the top, because the alternative is discovering you disagree during the thing the gates existed to prevent. Second, the advance gate is the one that gets written as a duration and shouldn't be. "Bake for three days" is not a criterion, it's a wait; a stage widens on evidence, and time is only how long the evidence takes to arrive.
And a fix does not inherit the stage it broke. If a stage pauses, gets a fix, and resumes, that stage bakes again from zero — the version that baked is not the version you're now running, and the temptation to skip the second bake is strongest exactly when the fix felt small.
- Entry — what must already be true before this stage opens at all. Usually the previous stage advanced, plus something specific: the baseline exists, the exclusion is confirmed, last time's fix baked.
- Advance — what this stage must prove to earn the next one. Evidence, not elapsed time.
- Pause and rollback — the two ways it stops, and they are not the same thing. Step 5.
Step 4: Pick signals that can decide, and know when they cannot
Four kinds, doing four jobs. An outcome signal — did the thing get better for users. Limits — the lines that are not traded against the outcome, whatever the outcome does, and there are usually two or three rather than one. And an operational signal, which is normally your support queue and is normally the first to know. Four signals that all move together aren't four signals; they're one, with three witnesses.
Then the part that gets skipped, and it decides whether any of this works: write down what your metric can't do. When is it attributed — and do the arithmetic rather than adding a few days, because a bake of ten days against a seven-day attribution window contains three days of readable data, which is exactly the number you rejected, wearing a bigger one. How much traffic does it need — a cohort under some size produces a flat line whether the change is good or catastrophic. And what's the baseline, precisely: the four weeks before this stage on the same weekday mix, not "last month," because your traffic is seasonal and January is not November. Note those two constraints interact — a sample too short to contain a weekday mix can't be compared to a weekday-matched baseline at all.
You are reading signals that already exist. Deciding which signals should exist, what they should page at, and who gets woken up is monitoring design — a different job with a different document. Where a gate draws its line on a signal you already have is this one's. If a gate needs a signal you don't collect, that's a finding for that other document and a reason this stage isn't ready.
Step 5: Separate holding from undoing
Most rollout plans have one stop, and it's the rollback. That's one state too few, and the missing one is the state most incidents actually want.
A pause stops the widening. The people already on the new version stay on it, nothing reverses, the measurement keeps running, and you've bought time to think. Latency regressed and you know why? Pause. The fix is a day away? Pause. Nobody is being hurt and the number is merely bad? Pause, and go and look properly. A rollback returns exposure to zero, and it costs: you lose the cohort's measurement, you may reverse data, and you pay the point-of-no-return tax if you've passed one. Reaching for it when a pause would do is how a fixable regression becomes a lost fortnight.
So write both, with different triggers, and note that a rollback condition almost always satisfies the pause condition too — which means the rollback line has to name a level pause has already failed to hold, or the two are the same gate with different names. They answer different questions: a pause asks "should this go further?", a rollback asks "should this exist right now?" Users can't complete the core job, or the change is costing something you can't get back: that's a rollback. And what a rollback actually does — the reverse steps, the data, the point of no return, how you confirm it worked, who executes it — is not this document. It's the rollback plan, it has its own owner, and this plan's job is to say when it fires, not how.
Step 6: Write down what finished looks like
One hundred percent is not finished. One hundred percent with the flag still in the code is a permanent branch, a second implementation you now maintain, and a conditional every future change has to reason about. Finished is: everyone eligible has it, it held for a bake, the flag is out, the old path is deleted, and the dashboard is retired.
Which is where Step 1's exclusions come back with a bill. One hundred percent of eligible is not one hundred percent, and the difference is whoever you carved out — and if the flag is the thing enforcing that carve-out, you cannot remove the flag. So a rollout with a permanent exclusion has a permanent flag, and every clause after "held for a bake" is unreachable. That's a legitimate outcome, but only if somebody decides it: this ends in a scoped flag with an owner and a review date. Nobody deciding it produces the same artifact and calls it finished.
Then exposure needs somewhere to live that isn't a Slack thread — which stage, which cohorts, since when, readable on a Monday by someone who wasn't in the room. Rollouts don't stall loudly. They stall by everyone assuming somebody else is watching, and the tell is always the same: the person who could tell you what percentage it's at is on holiday. And look again after the watching stops. The fortnight when everyone is staring at the dashboard is not a representative fortnight.
Common mistakes
The pattern in most of these: the plan produces a decision it hasn't earned the right to make.
- The internal cohort standing in for users. Staff don't behave like customers, don't share their incentives, and often can't generate the outcome metric at all. Internal answers "does it work", never "is it better", and a stage that advanced on staff enthusiasm advanced on nothing.
- The stage that advances because nothing bad happened. Absence of a signal isn't evidence when the cohort was too small to produce one, or the metric hadn't landed yet. Silence and safety look identical on a dashboard.
- Trading a limit against an outcome. Conversion is up and latency doubled, so it widens. A limit that can be outvoted by good news isn't a limit, it's a preference — and writing it down beforehand matters precisely because in the moment, with a green outcome metric on screen, it will feel eminently tradeable.
- The gate with no margin. "Above baseline" fires half the time by construction, because a baseline is an average. Without a margin and a sustained window you haven't written a gate, you've written a coin flip that pages somebody.
- One dashboard for the whole rollout. The cohort's numbers are diluted by everyone not in it, so a real problem for the ten percent shows up as a rounding error in the aggregate and the stage advances.
- Reading the plan as a rollout. It's a candidate somebody has to check against real data — NewPrompt doesn't move traffic, read your metrics, or decide anything about who sees what.
Rolling out a new search ranking model
A jobs marketplace replaces the ranker behind search. The code is deployed already, dark, behind a flag at zero — this document is only about who meets it. Interesting because its outcome metric is slow and noisy, which is the ordinary case and the one where rollout plans quietly fail. Everything below is fictional.
One flag at zero, opened across four stages and an exclusion nobody had scheduled: a bake length that had to be recomputed twice, an internal cohort whose outcome panel reads 0.0%, a limit and an outcome that can never appear on screen together, six numbers left at TBD with names against them, and a definition of finished the exclusions make unreachableSTAGED ROLLOUT — new search ranking model, jobs marketplace
plan v2, nothing exposed yet
WHAT IS CHANGING
The ranker behind job search. Same query box, same filters, different order.
Deployed behind a flag that is already live at 0%. The deploy is done; this
document is about who meets it.
RISK HYPOTHESIS (the sentence the whole plan is built to test)
A better order for the median searcher can be a worse order for a narrow one.
If it is wrong, the people who cannot find jobs will not tell us — they leave.
ELIGIBLE All job seekers using search. ~1.1M searches/week.
EXCLUDED - Employer accounts previewing their own listings. Their
searches are QA, not demand; they would skew every
number below.
- The 9 enterprise accounts with contractual result-order
guarantees. Legal, not technical. See S4 — this one does
not just sit in a list.
- Anyone in the active onboarding experiment. Two changes
on one population and neither result means anything.
──────────────────────────────────────────────────────────────
BASELINE The 4 weeks before S0, same weekday mix. Not 'last month' —
hiring is seasonal and January is not November.
SIGNALS Four, doing four jobs. Not four ways of saying one thing.
outcome apply-through rate: searches that end in an application
limit p95 search latency <- not traded against the outcome
limit zero-result rate <- not traded against the outcome
operational "can't find" support contacts per 1k searches
DATA FRESHNESS apply-through is attributed on a 7-day window. An
application today lands in the metric next Tuesday.
^ this one line rewrote the plan twice. First draft: 3-day bakes. But a
3-day bake cannot read apply-through at all — only the two limits,
which are instant. Second draft: 10 days. Also wrong, and wrong in a
way worth sitting with: a bake of B days contains only B minus 7 days
of fully attributed searches, so 10 days is 7 of lag and 3 of readable
data — the same 3 we had just thrown out, wearing a bigger number.
And 3 days has no weekday mix, so it cannot be compared to the
weekday-matched baseline above, which is the comparison this plan
mandates. Floor: 7 exposed + 7 to land = 14 days for any stage gated
on the outcome. Nobody caught any of this until the window got written
down, and then it had to be caught a second time.
GATE PRECEDENCE rollback beats pause beats advance. If two fire, the
stronger one wins and it is not a discussion. Write this
once, here, or you will have it at 3pm instead.
──────────────────────────────────────────────────────────────
S0 INTERNAL 40 staff, opt-in. Bake: 3 days.
Entry Flag live at 0%. Baseline captured. Dana has confirmed the
enterprise exclusion holds.
Advance No crash. p95 within its ceiling (below). Two people who do this
job say the results look sane.
Reads latency, zero-result. NOT apply-through.
^ staff do not apply to jobs, so the apply-through panel at S0 reads
0.0% — and it would mean nothing even if it moved, twice over: the
cohort cannot produce the signal, and a 3-day bake sits inside the
attribution window anyway. This is the false positive the plan exists
to pre-empt. Without the Reads line, somebody stares at an empty panel
at 9am Thursday and calls it a regression. An internal cohort tests
whether the thing works. It cannot tell you whether it is better,
because staff are not the users.
S1 PILOT SEGMENT size TBD. Bake: 14 days minimum.
Who Candidate: searches using the remote-only filter. Self-contained
query mix, no employer-brand exposure, and a population that
complains loudly, which is a feature here.
Size TBD — Priya, with Analytics. Needs the segment's share of weekly
searches AND the arithmetic underneath it: the apply-through base
rate, the smallest drop worth catching, and the variance. If the
segment cannot reach a readable sample in 14 days, the segment is
the wrong pilot — not the bake the wrong length.
Entry S0 advanced. The segment's own 4-week baseline exists.
Advance apply-through within a band of segment baseline. Band TBD — Priya.
It is the largest drop we are willing to not notice.
Pause p95 above its ceiling for a sustained window. Ceiling TBD, window
TBD — Ana. Same class of decision as the band: the ceiling is the
slowest we are willing to ship, and the window is long enough that
a deploy blip does not trip it and short enough that a real
regression does not bake.
Zero-result rate above baseline by a margin, sustained. Margin TBD
— Ana. It must be a margin: baseline is an average, and a metric
sits above its average half the time by construction. 'Above
baseline' with no margin is a gate that fires on Tuesday, forever.
Rollback Zero-result rate above a level strictly worse than the pause
margin — level TBD, Ana. The ordering is the point: every rollback
condition already satisfies the pause condition, so rollback has
to name a level that pause has demonstrably failed to hold.
Or: any result-order complaint from an excluded account, which
would mean the exclusion leaked.
^ six TBDs on this stage and not one of them is coyness. Every one is a
number somebody has to derive from data this document does not contain,
and each has a name against it. Note what is NOT TBD: that latency is a
limit rather than an input to a trade. That part is a decision, not a
measurement, and it is made here while it is cheap.
^ THE CASE THIS GATE EXISTS FOR: p95 goes 180ms -> 850ms in the first
hour. The pause fires on latency alone, and here is the thing — the
outcome metric has no opinion yet and will not have one for two weeks.
When it finally lands at +6%, the pause is eleven days old. Those two
numbers are never on screen together, and that is the only reason this
plan is not tempted by them. Put them side by side and +6% is a
genuinely persuasive argument for shipping 850ms to everybody. The
limit is written down precisely because the moment it matters is the
moment it will feel tradeable.
S2 BROAD share TBD. Bake: 14 days.
Entry S1 advanced. If S1 paused, the cause is closed and the fix has
baked through S1 again from zero. A fix does not inherit the
stage it broke.
Advance / Pause / Rollback As S1.
S3 ALL ELIGIBLE Bake: 14 days.
S4 EXCLUSION RESOLUTION No exposure change. Owner: Dana.
The 9 enterprise accounts are the only thing standing between this rollout
and a deleted flag. Either the guarantee gets renegotiated, or it is
confirmed permanent — in which case COMPLETE below is a lie and this plan
ends in a scoped flag with an owner and an annual review. That is a decision
somebody makes on purpose. The alternative is the one every stalled rollout
arrives at by not deciding.
^ on the numbers generally: 1% -> 10% -> 50% -> 100% is a folk sequence.
Not wrong — unearned. Those cuts encode a claim about how much exposure
your signals can pay for, and nobody here has done that arithmetic yet.
A percentage you did not derive is a number you will defend anyway.
──────────────────────────────────────────────────────────────
PAUSE IS NOT ROLLBACK
Pause exposure stops widening. The people on it stay on it. Nothing is
reversed, the measurement keeps running, and you have bought time.
Rollback exposure returns to zero. Reverse steps, verification, the point of
no return and who executes it live in the rollback plan, not here.
This document says WHEN. That one says HOW, and it has its own owner.
^ most of what goes wrong in a rollout wants a pause. Reaching for the
rollback is how a fixable latency regression turns into a day of
reverse-migration and a lost fortnight of measurement.
OWNERSHIP
Rollout owner Priya. Advances stages. Calls pauses.
Rollback call Ana (on-call). Does not need Priya, and should not wait.
Exclusions Dana. It is a legal boundary and S4 is hers.
COMMUNICATION
Support gets the S1 date, the segment, and the two sentences describing what
changed — before S1, not with it. They are the operational signal; a signal
you did not brief arrives as a surprise ticket.
EXPOSURE TRACKING
Which stage, which cohorts, since when, in one place somebody reads on a
Monday. Every rollout that stalls, stalls quietly.
COMPLETE means
100% of eligible, held 14 days. AND S4 closed. AND the flag is removed
from the code. AND the old ranker is deleted. AND this dashboard is
retired.
^ 100% of eligible is not 100%, and the gap is nine accounts with a
contract. The flag is what enforces that exclusion — so without S4,
this plan executed perfectly still terminates in a permanent flag
guarding a permanent second ranker, which is the exact thing the last
three lines exist to prevent. A rollout can be run flawlessly and still
never finish, and it will not feel like failure while it is happening.
POST-ROLLOUT Two weeks after COMPLETE: did the outcome hold once the
novelty and the watching stopped.
This plan changes nobody's exposure. It is a document about who meets what and
when you stop. The flag, the traffic, the numbers and every decision above
belong to the people whose names are on them. Fictional throughout.Where this fits in NewPrompt
The Staged & Canary Rollout Prompt is the closest thing to a starting point: a copy-paste prompt that takes your release and asks for stages with cohorts and percentages, the gating metrics per stage, the bake time, the halt condition, and the segments to exclude or save for last. It also carries the two rules that separate a canary from a countdown — each stage widens only on evidence and never on a timer alone, and the halt condition must be objective and agreed before it fires. What it doesn't carry is the arithmetic underneath: baselines, how long your metric takes to attribute, how much traffic it needs before it can say anything, the difference between holding and undoing, and what finished means. Those are Steps 3 through 6, and they're yours.
Markdown Output Builder assembles a prompt like that in your browser, and it pays to know precisely what it does before you click through. It ships a real staged-rollout preset — not an empty form — and a good one: five sections mirroring the prompt's own items, tables required, and a strict instruction that the model use every heading in the given order. Two gaps, though. It emits a document contract with no input slot: what comes out says in as many words that there is no source text to summarize, so the release has to go into its goal field or get pasted in afterwards. And it has no rules channel of its own — the rule blocks it emits are markdown, table and consistency rules from fixed lists, none of which can hold a claim about your rollout, and the preset's goal doesn't carry the two either. You can type them into the goal field yourself. Do that deliberately rather than intending to: evidence-not-timer is exactly the rule that erodes under launch pressure, and a rule you have to remember to paste back is a rule that won't survive the third rollout.
Two neighbours, and the seams matter because "deployment" is a crowded word. The Rollback Plan Prompt owns the undo — the trigger, the reverse of every forward step, the data, the point of no return, who makes the call, and how you confirm it worked. This plan says when a rollback fires; that one says what happens next. And the Monitoring & Alerting Plan Prompt is where signals get designed: which exist, what they page at, who they wake. A rollout gate reads signals that already exist and draws its own line on them. If yours needs a signal that doesn't exist, that's a finding for that document and a reason this stage isn't ready.
Zooming out, this plan lives inside a larger arc that the AI Deployment & Release Workflow lays out — readiness, then the deploy and its rollback, then the monitoring and launch checks. Read its shape honestly: there's no staged-rollout step in it, which is the gap this guide fills, and it's worth noticing rather than glossing. NewPrompt supplies the prompts and the order; you run them.
The reason to write the stop conditions down while the flag is still at zero is that you will not agree with yourself later. At 3pm on a Wednesday, with conversion up six percent and latency doubled and a launch date behind you, the case for widening will be genuinely persuasive, and the person making it will be you. A plan written in advance isn't smarter than that person. It just hasn't met the launch date yet, and it's the only version of you that gets to decide what a limit means before there's something to lose by meaning it.