How to Plan a Zero-Downtime Deployment With AI
"We do rolling deploys, so we have zero downtime" is true about the mechanism and often false about the change. Here is how to plan a zero-downtime deployment with AI: work out whether the old and new versions can actually run at once, across the database, the queue, and the traffic switch, before you ship.
Get the Zero-Downtime Deployment PromptThe rolling deploy that spreads the outage instead of removing it
The plan, as it usually exists, is one sentence: "we do rolling deploys, so there is no downtime." And the sentence is true about the mechanism. A rolling deploy really does replace instances a few at a time without ever taking the whole fleet down. What it says nothing about is whether the change riding on it is safe to have half-deployed — and that is the part that actually decides whether users notice.
Because for the length of that rollout, two versions of your code are live at the same moment, serving the same users, reading the same database, pulling from the same queue. If those two versions disagree about the shape of a row or a message, the rolling deploy has not removed the outage. It has spread it across the whole window — quieter than a hard crash, so nobody gets paged, and longer, so more requests fall into it. A steady trickle of errors for ten minutes is not zero downtime. It is downtime that chose not to announce itself.
So a zero-downtime plan is not a choice of mechanism. Rolling, blue-green, canary — those are how the switch happens. The plan is the harder question underneath: can the old version and the new version coexist for the minutes or hours it takes to cross over, everywhere they touch shared state? This guide is about answering that before you ship, not discovering it during.
Zero downtime is the whole compatibility matrix, not the last cell
The instinct is to check that the new version works. But during the crossover the new version working is only one of four things that all have to be true at once, and it is the one you were always going to test. Lay them out as a grid: old code against old data, old code against new data, new code against old data, new code against new data. "Data" here means everything the two versions share — rows, message payloads, cache entries, session shapes.
Cell four — new code, new data — is the target, and it is the cell everyone verifies. Cell one — old code, old data — is the status quo, already fine. The two that decide whether you have downtime are the diagonal ones nobody demos. New code reading old data is usually manageable by design: the new version knows the old shape exists and can treat a missing field as unknown. The dangerous one is almost always old code meeting new data — the still-running old version handed a row or a message written by the new version, in a shape it was never taught to expect. Zero downtime lives or dies in that cell, and it is exactly the cell a "does the new version work?" test skips.
This is the frame for everything below. Each surface the two versions share — the schema, the queue, the cache, the session store — gets walked through all four cells, and the plan's real content is what you do about the cells that do not hold on their own.
What the AI plans, and what only your infrastructure can answer
You bring the parts only you have: the architecture, the deployment mechanism you actually use, the state and schema and jobs involved, and the compatibility constraints of your real system. From those, an AI can draft a candidate plan — the sequence, the matrix, the questions each surface raises. What it produces is a plan to review, not a deployment to trust.
None of the operational reality is something the AI touches. It has no connection to your CI/CD, your orchestrator, or your load balancer, so it starts nothing, shifts no traffic, drains no instance, runs no migration, and performs no rollback — and it reads none of your health checks or telemetry, which is why a readiness gate it describes is one you have to build and wire before it means anything. A plan is only ever a candidate here. Whether the deployment actually holds up is decided by your real infrastructure and confirmed by the people who own it, not promised by the document.
That boundary is not a caveat on the plan — it is why the plan is written the way it is. Because the AI cannot watch the deploy happen, the plan has to make the compatibility reasoning explicit up front, where a human can check it against a system the model has never seen.
Step 1: Name the surfaces the two versions share
Start by listing every place the old and new versions touch the same state, because each one is a place the matrix has to hold. The database is the obvious one, but it is rarely the only one: a message queue where a payload written by one version is read by another, a shared cache whose entries outlive a single request, a session store, in-memory state on a sticky connection, static assets a browser mixes with a backend that moved on. Miss a surface and you have not planned a zero-downtime deploy; you have planned three of them and left the fourth to chance.
For each surface, name the change concretely: what shape it has now, what shape the new version gives it, and whether the change is additive (a new optional field nobody old has to know about) or a modification (something old code reads or writes is moving or disappearing). Additive changes are the ones that can coexist; modifications are the ones that force a sequence. The distinction is the single most useful thing you can establish before writing any steps.
And state the availability objective plainly, including its honest limits. Zero downtime is not achievable for every change on every architecture — some genuinely need a brief window, and a plan that pretends otherwise is worse than one that schedules the window openly. If a maintenance pause is the real answer for one surface, that belongs in the plan, not hidden behind a rolling deploy that quietly degrades instead.
- List every shared surface: database, queue, cache, sessions, sticky state, static assets. Each must hold the matrix.
- Classify each change as additive or a modification. Additive coexists; a modification forces an ordering.
- A brief maintenance window, scheduled openly, beats a rolling deploy that degrades and calls itself zero-downtime.
- Low error rate during a deploy is not proof of coexistence — it can just mean the broken cell has not been hit yet.
Step 2: Order the deploy so the reader is ready before the writer
Most zero-downtime failures are ordering failures, and they follow one rule: whatever reads a new shape has to be deployed before whatever writes it. Put the schema change that adds a column before the code that populates it. Put the consumer that can handle a new message field before the producer that starts sending it. Ship the writer first and you have manufactured the dangerous cell on purpose — new data arriving at old readers that were never taught it.
The queue is where this bites hardest and gets skipped most, because a message is a contract between two services that deploy separately. Walk it explicitly: can the currently-running consumer read a payload the new producer would send? Can the new consumer still read the old messages already sitting in the queue? What happens to an old message that gets retried after the consumer has moved on? Often the cleanest answer is that consumers tolerate unknown fields, so an additive payload change needs no version bump at all — but that is a property you confirm about your real consumers, not one you assume. If they strict-parse and reject unknowns, the ordering is forced: every consumer that can read the new field ships and is confirmed before the producer emits it.
For the schema, keep this step thin and additive on purpose. Add the new column as nullable so old code that ignores an unknown column is unaffected — but "ignores it" is the same kind of claim as the queue one and deserves the same check, not a shrug, because old code that does a SELECT * into a strict mapper is handed the new column whether it asked for it or not. The heavier machinery — backfilling old rows, tightening the column to required later — is a data-migration job with its own sequence and its own risks, and it is deliberately not this plan's work. This plan's job is the deployment ordering; the migration's job is the data.
- The rule: readers before writers. Schema before the code that fills it; consumers before the producer that emits.
- Walk the queue in both directions — old consumer/new message and new consumer/old message — plus retried old messages.
- Unknown-field tolerance can remove the need for a version bump — but confirm it about your consumers, do not assume it.
- Keep the schema step additive and nullable; hand backfill and the eventual not-null to the migration plan.
Step 3: Gate traffic on readiness, and drain what is in flight
A new instance being alive is not the same as it being ready, and traffic has to key on the second, not the first. Liveness asks whether the process is running. Readiness asks whether it can actually serve a request — startup finished, dependencies reachable, any warmup done. An instance that takes traffic in the gap between those two states serves errors for exactly as long as the gap lasts, and the failure looks like a flaky deploy rather than the ordering bug it is.
So readiness is a gate, per instance, before the load balancer routes to it — not a check you run after rolling everything out. "Deploy all the new instances, then verify" has the order backwards: verification that happens after traffic is already flowing is a post-mortem, not a gate. Each instance earns its place in the pool by passing readiness first. This is a check you have to build and the platform has to honor; it is not something that appears on its own because you configured a rolling deploy.
Draining is the same discipline on the way out. When an old instance is being retired, the load balancer stops sending it new requests while its in-flight ones finish, so no active request is cut mid-flight. Workers stop pulling new jobs and finish or safely re-queue the one in hand. Every part of this has a timeout, and the timeout is a real number you derive rather than invent — from your actual p99 request duration, your longest-running job, the behavior of any streaming or long-lived connection. A drain timeout shorter than your slowest legitimate request is a plan to drop it; a made-up round number is how that happens.
- Traffic keys on readiness, not liveness. Alive and not-yet-ready is a real state, and it serves errors if it takes traffic.
- Readiness gates each instance before the pool — not a check after the whole fleet is deployed.
- On the way out: stop new work, let in-flight finish or safely re-queue, then remove the instance.
- Derive drain timeouts from measured request and job durations. A round number pulled from the air drops the slow ones.
Step 4: Check whether rollback is still an option once new data exists
A rollback plan tells you the reverse steps; this asks a different question — whether reversing is still safe at all, given what the new version has already written. The moment matters, because rollback is usually free at the start of a deploy and quietly stops being free once the new version has produced data or messages in a shape the old version cannot read.
Additive changes are kind here. A new nullable column is readable by old code that tolerates it — the property you confirmed in the ordering step — so a code rollback lands back in the already-handled cell and costs nothing extra. The queue is where it can turn one-way without anyone deciding it should. Once the new producer has enqueued messages carrying the new field, rolling the consumers back to a version that rejects unknown fields means those already-queued messages now fail on the old code you just restored. The rollback that was supposed to be your safety net becomes a second incident. Worker rollback is only safe if the version you roll back to still tolerates the new shape — the same tolerance question from the ordering step, asked in reverse.
So the plan names the point where reversing stops being clean, honestly, before the deploy rather than during a bad one. Often the good news is that a well-ordered additive deploy has no hard point of no return in its own window at all — the only genuinely irreversible step is a later, separate contract release that removes the old shape, and that one ships alone, after every old instance is gone. Naming that keeps the reversible window reversible and stops the two from being confused under pressure.
- Rollback safety is a question about written state, not just a list of reverse commands.
- Additive schema rolls back cleanly; a queue full of new-shape messages may not, if the old consumer rejects them.
- Worker rollback is safe only if the target version tolerates the new shape — the ordering question, reversed.
- Separate the reversible coexistence window from the later, one-way contract step. Do not let them blur together.
Step 5: Verify coexistence, and leave the contract step for later
Verification here is not "are errors low" — a low error rate can just mean the broken cell has not been exercised yet. It is a direct check that the cells you were worried about actually hold: old code reading a row the new version wrote, the new consumer handling a message the old producer sent, an old message retried against a new consumer. And a check that the readiness gate did its job — that a not-ready instance was actually held out of the pool rather than waved in. If you cannot point at a real request exercising the dangerous cell, you have not verified coexistence; you have verified that nothing tested it.
Then, deliberately, stop. The clean-up — dropping the old column, retiring the old message shape, deleting the tolerance shims that let the two versions coexist — is a separate release for a later day, run once you are certain no old instance is left. Folding it into this deploy is how a zero-downtime plan reintroduces the exact incompatibility it spent all this effort avoiding: you remove the old shape while something old is still reading it. The coexistence window ends when you decide it ends, not when the deploy finishes.
What this plan is not is the operational script. The exact commands, the per-step checks, the reverse procedures and their triggers, the data backfill — those live in a deployment runbook, a rollback plan, and a migration plan respectively. This plan is the compatibility reasoning that those documents assume you have already done. It answers whether the crossover is possible; they carry out the crossover it describes.
- Verify the dangerous cells directly — old-reads-new, new-consumer/old-message — not just the aggregate error rate.
- Confirm the readiness gate actually excluded a not-ready instance. An untested gate is not a gate.
- The contract step (drop the old shape, remove the shims) is a later, standalone release — never folded into this deploy.
- This plan is the compatibility reasoning; the runbook, rollback plan, and migration plan are the execution.
Common mistakes
The assumptions that turn a rolling deploy into a slow, quiet outage:
- Calling the mechanism the plan. "We use rolling deploys" describes how instances swap; it says nothing about whether the half-deployed change is safe.
- Testing only that the new version works. That is cell four. The cell that breaks is old code meeting new data, and it is the one no demo covers.
- Deploying the writer before the reader. A producer emitting a shape its consumers cannot yet read is the ordering bug that causes most of these outages.
- Starting with a rename or a drop. Removing or moving something the still-running old code depends on breaks it the instant the change lands, before the new code is even there.
- Routing traffic on liveness. A process that is alive but not ready serves errors until it warms up; readiness, per instance, is the gate.
- Inventing a drain timeout. A round number shorter than your slowest real request drops that request; derive it from measured durations instead.
- Assuming rollback is always safe. Once the new version has written data or queued messages the old version cannot read, reversing can become a second incident.
- Reading a low error rate as proof. During a deploy it often means the incompatible path has not been hit yet, not that there is not one.
A worked example: adding a field to an inventory reservation service
Here is a change small enough to sound trivial — add one optional field — worked through to the point where the coexistence questions are visible. The system has API instances, background workers that expire reservations off a queue, and one shared database, which is exactly the mix where a "simple" field touches three surfaces at once.
Read the shape rather than the prose: a four-cell matrix where cell two is the whole job, a sequence whose every step is survivable by whatever is running when it lands, a rollback that is free on the database and conditional on the queue, and two decisions left open with owners because they are facts about a real system, not things a plan gets to invent. No timeout and no duration is filled in — where a number belongs, it says who measures it.
One optional field across three shared surfaces: the four-cell matrix, a reader-before-writer sequence, a rollback that is free on the database and conditional on the queue, and two decisions left open with owners rather than guessedTHE SYSTEM (inventory reservation service):
- API instances behind a load balancer
- background workers that expire stale reservations, pulling from a queue
- one shared database, table `reservations`
- messages flow API -> queue -> workers
THE CHANGE:
reservations gain a new optional `channel` field (web, mobile, partner).
- the NEW api writes `channel` on the row and includes it in the queue message
- the NEW worker reads `channel` and expires by channel-specific rules
- the OLD api and OLD worker keep running through the whole window
THE FOUR CELLS (zero downtime means ALL of them hold, not just the last):
1. old code + old row/message the status quo. Works.
2. old code + NEW row/message old api reads a row that now has a `channel`
column; old worker pulls a message that now
has a `channel` field. THIS is the cell that
breaks first, and cell 2 is the whole job.
3. new code + old row/message new code sees no `channel`; treats it as
unknown. Fine by design.
4. new code + NEW row/message the target. Works.
THE SEQUENCE (each step survivable by whatever is running when it lands):
STEP A -- expand the schema, additively
add `channel` as a NULLABLE column. Additive-and-nullable is the safe
DATA-side shape -- but "old code ignores a new column" is a property to
CONFIRM, not assume, exactly like the message field below. Code that
does SELECT * into a strict deserializer, or an ORM frozen at the old
shape, is handed the new column whether it wanted it or not, and can
break on it. Confirm the old readers tolerate an unknown column (this
is the data-side twin of D1).
(Making it NOT NULL, and backfilling old rows, is a later job and a
different guide -- the migration plan owns the data mechanics. Here
it stays additive and nullable.)
STEP B -- settle cell 2 on the MESSAGE side BEFORE anyone emits it
the crux. A new message with a `channel` field reaches whatever worker
is running -- and during the window that includes OLD workers. So the
reader has to be ready before the writer speaks:
- if old workers already ignore unknown fields, they tolerate the new
message and you can proceed;
- if old workers reject unknown fields, the producer CANNOT emit the
new field until the workers that can read it are fully deployed.
Deploy the consumers first, confirm the fleet is updated, THEN let the
producer emit. Writer-before-reader is the classic way to break this.
STEP C -- roll the new instances in, one at a time, gated on readiness
each new api instance must pass a readiness check -- process up AND
dependencies reachable AND startup finished -- BEFORE the load balancer
sends it traffic. Each new worker must be ready before it pulls a job.
Not "deploy them all, then check": an instance that takes traffic
before it is ready serves errors for the gap. Readiness is per instance,
and it is the gate, not a step after the gate.
(Liveness -- is the process alive -- is a different question. A process
can be alive and not yet ready. Traffic keys on ready, not alive.)
STEP D -- drain the old instances as they leave
the load balancer stops sending NEW requests to an old instance, and
its in-flight requests are allowed to finish before it stops. Workers
stop taking NEW jobs and finish (or safely re-queue) the one in hand.
The drain timeout is a real number and it is NOT invented here: derive
it from the p99 request duration and the longest expiry job. TBD, owner
named below. Long-lived connections (streaming, websockets) need their
own answer -- also open.
ROLLBACK -- and where it stops being free:
the DATA side rolls back cleanly: `channel` is additive and nullable, so
old code reading those rows is cell 2 -- safe as long as STEP A's check
held and the old readers really do tolerate the extra column.
the QUEUE side is where rollback can bite. Once the new api has enqueued
messages carrying `channel`, rolling the WORKERS back to a version that
rejects unknown fields means those already-queued messages now fail. So
worker rollback is only safe if the rolled-back version still tolerates
the field -- which is the same tolerance question as STEP B, asked in
reverse. Rollback is not automatically safe just because the deploy was.
POINT OF NO RETURN:
there is NO hard one-way step in this window -- that is what makes it a
zero-downtime change and not a migration. The only irreversible step is a
LATER, separate contract release: make `channel` required, retire the old
message shape, remove the tolerance shims. It ships alone, after every
old instance is gone, and its data side goes back to the migration guide.
TWO DECISIONS STILL OPEN (named, with owners, not guessed):
D1 do the workers running RIGHT NOW tolerate an unknown message field,
or strict-deserialize and reject it? owner: reservations/worker team
-> decides STEP B ordering AND whether worker rollback is safe. The
single fact the whole plan turns on. Answer it first.
D2 drain timeout (api) and in-flight grace (worker). owner: SRE / deploy
-> derive from measured p99 request time and longest expiry job.
No number until those are read. Do not paste in "30 seconds".
VERIFY (that both versions are actually coexisting, not that errors are low):
- old api reads new rows without error; new api reads old rows
- a new-format message is handled by whatever worker version pulls it
- the readiness gate actually held a not-ready instance out of the pool
low error rate alone does not prove this -- it can also mean the broken
path just has not been hit yet.
LEFT TO OTHER GUIDES ON PURPOSE:
the exact deploy commands and per-step checks -> deployment runbook
the reverse steps and the triggers to pull them -> rollback plan
the backfill and the eventual NOT NULL -> database migration plan
which users see channel-specific behavior first -> staged rolloutWhere this fits in NewPrompt
The Zero-Downtime Deployment Prompt is the honest starting point, because it states the constraint before you touch a tool: two versions will run at once, and every compatibility rule has to hold for that whole window. It lays out the shape of a strategy — the pattern, the traffic shift, how in-flight work drains, the version-compatibility rules, the health gating, the stateful concerns — and its own note sends the data specifics to the migration prompt, which is the boundary this guide keeps. It gives you the frame; the matrix walk, the queue ordering, and the rollback-after-writes reasoning are what you fill in on top of it.
To turn that frame into a prompt you run, the Markdown Output Builder assembles it in that six-section shape — in your browser, running nothing. One honest limit to know going in: it builds a document to a structure and has no dedicated slot for your input, so your architecture, mechanism, and constraints go into the goal you give it, and what comes back is prompt text you run in your own AI assistant. The tool shapes the strategy; it does not see your system.
The neighbours own the parts this plan deliberately hands off, and naming them is half of keeping this plan focused. The Database Migration Plan Prompt owns the data mechanics — the backfill, the dual-write, tightening a column to required — that Step 2 kept additive on purpose. The Rollback Plan Prompt owns the reverse steps and the triggers to pull them, where this guide only asked whether reversing is still safe. And the Deployment Runbook Prompt owns the command-level execution of the sequence — the plan here decides that the crossover is possible; the runbook is how it actually runs.
Across the whole release, the AI Deployment Release Workflow is the broader arc — readiness review, then planning the deploy and its rollback, then watching it land. It has no step dedicated to zero-downtime compatibility, which is the gap this guide fills: the workflow decides to deploy and how to undo it, and this is the reasoning about whether the two versions can share a running system while that happens. The plan is a candidate the owners check against real infrastructure — reading it is what turns "we do rolling deploys" from a hope into a claim you can actually stand behind.