Zero-Downtime Deploys of Stateful Engines

Before you start — this chapter leans on a handful of primer ideas:

FIX sessions and sequence numbers — the venue-side session state that makes takeover hard: ch00f

Feed snapshot + delta resync — what a reconnecting consumer must do to rebuild a book: ch00f

The determinism contract and state hashes — how a new binary proves it agrees with the old one: chapter 13

N/N+1 schema compatibility — why the new version must not write what the old can’t read: chapter 14

Read those first — 20 minutes there saves an hour here.

Blue-green deployment is a solved problem for stateless web services: run two fleets, flip the load balancer, done. Fowler wrote it up two decades ago. Web blue-green works because state lives in the database and sessions live in a cookie — a load-balancer flip moves nothing that matters. The reason this is an interview topic for trading infrastructure is that your engine is the opposite of stateless, in three compounding ways:

Derived state — books and positions that took hours of message-by-message accumulation to build up in memory. A load-balancer flip transfers none of it.
Open orders — orders resting at venues, which keep existing whether or not your process does. This is state held at a third party.
Live sessions — FIX connections with sequence numbers (ch00f), authenticated WebSockets: stateful protocols whose other half the counterparty owns — and the counterparty didn’t agree to your deploy.

Any deploy story that doesn’t address all three is a web-app answer wearing a trading costume — interviewers are listening for exactly that tell.

The escape hatch from all three is the event log. Because state is a deterministic fold over the log (the fold of ch13: state = events.reduce(apply, empty) — feed the same events in, get the same state out), “transfer state to the new version” reduces to “let the new version consume the log.” Every pattern below is a variation on that move.

Pattern A: hot-standby cutover (the flagship)

This is the pattern you’ve actually run, so it’s your anchor story. The choreography:

Start the new binary (green) as a standby. It loads the latest snapshot, replays the tail, then consumes the live sequenced stream, staying at the head. Those three steps on one picture of the log:
```
log:   [ events the snapshot already summarizes ][ tail ]──► head (live, still growing)
green:  1. load snapshot ═══════════════════════► 2. replay tail ► 3. follow the head
```
It is a full engine with outputs disabled — same fold, sink discarded (the no-I/O replay mode of ch13 doing production work).
Health-verify while both run. Green publishes rolling state hashes; they must match blue’s at the same sequence numbers. This is the payoff of the determinism contract: you get a proof the new version agrees with the old on live traffic before it owns anything. If the release intentionally changes behavior, you instead pre-verify via replay-diff in CI (ch17) and accept expected-diff categories; unexplained divergence aborts the deploy.

Preview: replay-regression testing (ch17). Replay recorded production days through the candidate build and diff its decisions — orders, cancels, prices — against what production actually did. Sort every diff into two piles: intended by this change (predicted in advance, in roughly the predicted amounts) and anything else. Anything else fails the gate. That’s all this chapter means by “replay-diff” and “expected-diff categories”; the change-management chapter builds the full machinery.
Cutover. Quiesce briefly (go quiet) at a sequence boundary: blue stops accepting new inputs (or the sequencer marks a LeadershipTransfer event in the log — cleaner, because the handover point is itself logged and replayable), green confirms it has applied through that sequence, then green’s outputs go live and inputs route to it. With both processes hot, the gap is the routing flip — milliseconds.
Fence blue. The old primary must be unable to act after the flip — never rely on “we told it to stop.” The mechanism is the epoch (also called a term or generation number): every leadership transfer bumps it by one, and every log append and outbound order carries the writer’s epoch stamp; anything downstream rejects stale epochs. Concretely: blue, still running because a shutdown script hung, emits an order stamped epoch 42; the gateway’s current epoch is 43; the order is rejected at the door. It’s a rotated API key — the old process can still make requests, but its credential no longer opens anything. Deploys are just voluntary failovers with the same fencing, and saying that sentence in an interview reframes your whole standby experience as a deployment skill.
Blue drains and lingers. Keep it running (outputs fenced) as the instant rollback target for the bake period (the watch-and-wait window after cutover, before you fully trust the new version).

The whole choreography on one timeline:

 time ──►
 BLUE (v42)   live: applying log, outputs ON ─────────────────╳ fenced: epoch 42
                                                              │ rejected downstream;
                                                              │ stays hot as rollback
 log (seq)   ─1000──1001──1002──[LeadershipTransfer{epoch:43}]─┬─1003──1004──►
                                                              │
 GREEN (v43)  snapshot ► tail replay ► live tail, outputs SINKED
              └── rolling state-hash vs BLUE: must match ──┘  └► outputs ON,
                 at same seq numbers (gate)                       owns sessions

The session takeover problem

The hard residue is step 3’s unstated assumption: green can talk to the venues. Sessions don’t transfer through the log.

FIX (tradfi): a FIX session is (SenderCompID, TargetCompID, inbound/outbound sequence numbers) over TCP. Three takeover options, in ascending sophistication.

Option (a): re-logon. Blue logs out, green logs on. You must persist and hand over the sequence numbers — they’re session state, so put them in the log/snapshot like everything else. If the seqnums don’t line up on logon, the two sides negotiate a resend/gap-fill. In miniature:

Green logs on: “my next outbound is 5001, and I expect your 8200.”
Venue: “I only got through your 4990 — resend 4991–5000.”
Green resends them (or gap-fills the ones that are now stale), and the session is level again.

The classic incident is the naive seqnum reset: green comes up claiming sequence 1 while the venue’s counter says 12000. One side now believes 11999 messages went missing and asks for all of them — the venue replays, or rejects, what looks like a full day of traffic. Even done right, re-logon costs seconds of session downtime; resting orders at the venue survive (they live in the venue’s book), but you’re blind and can’t cancel during the gap.

Option (b): session handover via a FIX gateway tier. The venue-facing TCP session lives in a thin, rarely-deployed gateway process; engines behind it come and go without the venue ever seeing a logout. This separation — long-lived dumb edge, frequently-deployed smart core — is the architectural answer interviewers want, and it’s the same isolation move as the feed handlers of ch14.

Option (c): TCP handoff / connection-migration tricks. These exist but are exotic; name them only to dismiss them.

Crypto (your world): WebSocket sessions with auth tokens; there is no seqnum continuity contract, which is simpler and worse. Deploy = green must re-authenticate and resubscribe N venues × M streams, which raises six concerns at once:

Rate limits — the auth burst hits per-connection and per-endpoint limits (you have felt this).
Resubscription storm — N × M subscribe messages, all at once.
Book resync — a snapshot-plus-buffered-deltas window per venue before its book is trustworthy again (your daily bread).
Pre-warming — open green’s connections before cutover where venues allow duplicate sessions; many do.
Session-kill quirks — some venues kill the older session on new auth. Know it per venue; that per-venue quirks table is itself an asset worth mentioning.
Reconciliation as a gate — open orders rest at the venue under venue-assigned IDs, so green needs the ID map (in the log) and must reconcile — compare its own open-order picture against the venue’s, line by line (ch15) — before it may trade. A gate, not a nicety.

The structural mitigation is the same gateway-tier answer as FIX: keep the connection owners out of the deploy blast radius.

Pattern B: drain-and-replace (for gateways and order-path services)

For services where in-flight work is short-lived — order gateways, risk checkers, REST/API frontends — you don’t need state transfer at all:

Mark the instance draining: stop routing new orders/requests to it.
Let in-flight work complete or time-bound it: wait for outstanding acks, cancel-on-timeout stragglers.
When quiesced, kill and replace; new instance registers for traffic.

The design prerequisites are the interview substance:

An upstream router that can exclude an instance — or venue-session multiplexing, so a draining gateway’s sessions move elsewhere.
Idempotent order handling — client-order-ID dedupe, so a retry through the new instance can’t double-submit.
A hard drain deadline — so one stuck order can’t wedge the deploy.

In crypto, “drain” also means “stop the strategy quoting through this gateway and let its resting orders be cancelled or adopted by another gateway.” Adoption is less mystical than it sounds: the orders never move — they rest at the venue the whole time. Gateway 2 loads the venue-order-ID map from the log and takes over cancel/replace duty for them.

Pattern C: shadow deployment (decision diffing)

Run the candidate against reality with zero risk: green consumes the live production feed and order flow, runs its full logic, and its outbound orders go to a sink that records instead of sends. Then diff green’s decisions against blue’s actual decisions, streamed, with a triage UI or even just a diff log.

This is strictly stronger than replay-based testing for one reason: it exercises today’s regime (the market’s current personality: its volatility, volumes, and quirks) — including inputs your recorded days don’t contain. It’s the cheapest high-fidelity test in the industry if and only if you have determinism and log-consumption as primitives, which you do; shadow mode is literally your standby with a recording sink. Limits to volunteer, because they’re the senior half of the answer: shadow can’t see market impact (its phantom fills come from a fill simulator against the live book), can’t test venue interaction (rejects, rate-limit behavior, partial-fill sequencing), and decision-diffs need noise discipline (an intended pricing change diffs everywhere; you need expected-vs-unexpected diff classification before the signal is usable — ch17). The fill simulator gets dishonest about queue position: the shadow’s order never actually stood in the price level’s line, so “would it have filled” is a guess about where in the queue it would have sat. Those guesses skew optimistic — the simulator awards fills that a real order, waiting behind everyone who arrived first, would not have gotten.

Pattern D: rolling by shard / venue (bounding blast radius)

How it works. If the system is sharded — per-venue feed handlers and gateways, per-symbol-group engines — deploy one shard at a time. A canary is the web-deploy idea unchanged: ship to one small, low-risk slice and watch it before the rest; here the slice is a small, liquid, forgiving venue — not your biggest P&L venue. So: canary venue first, bake, proceed in waves, halt-and-rollback on any regression.

Prerequisites. Shards genuinely independent (a shared risk service or cross-venue arb strategy couples them — know your coupling before claiming independence); per-shard health metrics with automatic gate checks between waves; and N/N+1 message compatibility (ch14), since mixed versions now coexist for hours, not minutes, on the bus.

When it applies — and when it doesn’t. For your 20-venue reality this is the default deploy mode for handler/gateway changes: venue-by-venue is both blast-radius control and a natural fit to per-venue protocol quirks. Engine-core changes, by contrast, are usually all-or-nothing per engine instance — which is why Pattern A exists.

Maintenance windows: 24/7 crypto vs. tradfi

Tradfi hands you a nightly maintenance window and a weekend; a huge fraction of “zero-downtime” pressure evaporates: you deploy at 5:30pm after the close, with the whole evening to verify and roll back. Session-close rituals (EOD snapshots, seqnum resets on many FIX venues at start-of-day) even give you natural state boundaries. Crypto gives you nothing: markets never close, weekends are often the highest-volatility periods, and there is no moment when open orders and positions are flat by nature. Consequences you should state as lived experience: every deploy is a market-hours deploy, so the Patterns above aren’t aspirational — they’re the only way to ship; you create synthetic windows by choosing low-activity hours (and your desk knows its venue-local quiet hours) and by flattening or reducing exposure pre-deploy as a policy decision with a real P&L cost (missed volume) that engineering must justify; and venue-side maintenance (exchanges restart their own matching engines, announced or not) doubles as your chaos testing. Tradfi interviewers enjoy hearing that last inversion: crypto infra people get failover drills for free because the venues perform them on you.

Rollback discipline

Forward-fix vs. roll back — the decision tree. Default is roll back: the old binary is a known-good artifact, the new one is a hypothesis you just falsified. Forward-fix only when: (a) rollback is unsafe because the new version has already written state the old can’t read — which you architect to avoid, next paragraph; (b) the defect predates the deploy (rolling back changes nothing); or (c) the fix is truly trivial and the bake/verify pipeline can validate it faster than a rollback — rarer than 3am-you believes. The discipline that makes the tree usable: decide the criteria before the deploy, in the runbook, because judgment during an incident is the worst judgment you own; and the moment rollback is on the table, kill switches (ch17) flatten risk first — you can think clearly about binaries once you’re not bleeding.

State compatibility with the N-1 binary is what makes rollback real. Rollback = the old binary must consume what the new one wrote: log events, snapshots, config. Rules: the new version must not emit new event/snapshot versions until it has baked past the rollback horizon — the point in time after which you’d no longer roll back, so N-1 compatibility can finally be relaxed. (This is the readers-first choreography of ch14: ship the ability to read the new format everywhere first, and only later let anything write it — here, v2-write capability flips on after bake, via config.) If the new version already wrote v2 events, rollback targets a patched N-1 that can at least skip-or-parse v2 (length-prefixed records and unknown-type tolerance make this survivable), or you accept replaying from the pre-deploy snapshot and reconciling — an incident, not a rollback. Test it: CI replays new-binary-written logs through the previous release. A rollback path that was never exercised is the untested recovery path of ch13 with worse timing.

The worked runbook: engine deploy at 3pm, market open

The interview set-piece. Deploying matching-engine build v42 → v43 (perf work + one intended behavior change in cancel handling), markets live. Three gates recur below; the legend:

Gate 1 — replay-regression clean: only the intended diffs, in roughly the predicted amounts.
Gate 2 — N-1 compatibility: the old binary can read what the new one writes.
Gate 3 — live agreement: rolling state hashes match while both versions run on live traffic.

Narrate it in phases:

T-1 day — pre-verification. The day before, you prove the change is exactly what you think it is — nothing more. Replay-regression: last 5 prod days through v43; decision-diff classified — only expected cancel-path diffs, counts within predicted bounds (gate 1). Determinism dual-replay green. N-1 check: v42 replays a v43-written staging log (gate 2). Runbook reviewed; rollback criteria written down: any unexplained decision divergence, p99.9 order-path latency +20%, recon mismatch, or venue session instability → roll back, no debate.

T-30 min. Half an hour out, you freeze the world and warn the humans. Freeze other changes. Page-out to the desk: deploy window, what changes, abort authority (desk can veto). Verify snapshots current, standby healthy, kill switches tested today. Reduce exposure per policy: widen quotes / cut size on the canary scope.

T-15 min. Now the new version starts running for real — with its outputs still off. Start v43 as shadow/standby: snapshot load + tail replay + live consumption (outputs sinked). Watch it reach head; rolling state-hash comparison against v42 running — matching on all paths except the expected cancel diffs, each one auto-classified (gate 3). Pre-warm v43’s venue connections where duplicate sessions are allowed.

T-0 — cutover. The flip itself is one logged event and a routing change. Sequencer writes LeadershipTransfer{epoch: 43}; v42 stops emitting at that boundary (fenced by epoch on every downstream); v43 confirms applied-through-boundary, enables outputs, takes the sessions (gateway-tier handover, or scripted re-logon per venue in dependency order). Total order-path gap target: <1s; measured and recorded.

T+0 to T+15 — verification gates. The first fifteen minutes are verification, not celebration. Open-order reconciliation vs. every venue (hard gate — trading stays reduced until clean). Latency histograms vs. baseline. Fill/reject/cancel rates per venue vs. same-hour baseline. First N decision spot-checks on the changed path. v42 stays hot, fenced, at head — instant rollback.

T+15 to T+2h — bake. Then you hand back the full keys, slowly. Restore full size stepwise. v43 still writing v1-compatible events (write-flip comes tomorrow, after the rollback horizon). Desk sign-off closes the deploy; v42 stays resident until end of day.

Rollback branch (rehearsed). If any tripwire fires, the path back is already rehearsed: kill switch to cancel-only on affected scope → LeadershipTransfer{epoch:44} back to v42 (it’s hot, at head) → sessions back → recon gate → resume → post-mortem with the shadow-period logs, which — because everything is in the log — reproduce the divergence exactly.

Two sentences of meta land well after the walkthrough: every gate is mechanical (a number and a threshold decided in advance), and the whole procedure is only possible because state transfer, verification, and rollback all reduce to log operations — the deployment story is the event-sourcing story.

Plain-English recap

Why blue-green fails here: the engine is a stateful WebSocket server with money attached. Web blue-green assumes state lives in the DB and sessions in a cookie; the engine holds hours of accumulated in-memory state, open orders resting at third parties, and live authenticated connections whose other half the counterparty owns. A load-balancer flip transfers none of that.
Hot-standby cutover is promoting a Postgres replica — with a checksum gate. Green syncs from the ledger (the log), catches up to zero lag, and must prove byte-identical state (matching hashes at the same sequence number) before it’s allowed to lead. Determinism turns “we think it’s ready” into a mechanical gate.
Session takeover is the PSP-connection problem. Your OAuth sessions and webhook registrations with a PSP don’t move because you redeployed. The gateway tier is the same cure you already use: keep a thin, stable edge (API gateway/proxy) that owns the external connections, and redeploy the smart backends behind it freely.
Drain-and-replace is a k8s rolling deploy plus idempotency keys. Stop routing new work, let in-flight work finish with a hard deadline, and rely on client-order-ID dedupe — your idempotency-key reflex — so a retry through the new instance can’t double-submit an order.
Shadow deployment is traffic mirroring / a dark launch. Run the candidate on live traffic with its outputs recorded instead of sent, then diff decisions — GitHub’s “Scientist” pattern with a P&L. Its blind spot is the same one mirroring has: the world never responded to the shadow’s actions, so market impact and venue reactions are unmeasured.
Rollback discipline is “new app version, old DB schema” until bake ends. The new binary must not write formats the old binary can’t read until the rollback horizon passes — the same reason you don’t run destructive migrations in the same deploy as the code that needs them. And criteria are written down before the deploy, like auto-rollback thresholds in a pipeline, because 3am judgment is the worst judgment you own.

Interviewer will ask

Q1: “Why can’t you just blue-green a matching engine like a web service?” Three kinds of state a load-balancer flip ignores: derived in-memory state (hours of book/position accumulation — solved by log replay + live catch-up), open orders resting at venues (survive your process; need ID mapping and reconciliation before green may act), and stateful venue sessions (FIX seqnums / WS auth — the counterparty holds half the state). Then: every workable pattern is “new version consumes the log,” which is why event sourcing is a deployment primitive.

Q2: “How do you know the new version is safe before it takes over?” Layered: replay-regression over recorded prod days with classified decision-diffs (pre-deploy); live shadow consumption with rolling state-hash comparison against the incumbent (during deploy); recon and metric gates with pre-committed thresholds (post-cutover). Emphasize that determinism turns “hope” into “proof of agreement on live traffic,” and that intended changes need a diff-classification story — an intended change makes hashes diverge by design, so without classified diffs you can’t tell planned divergence from a bug.

Q3: “Walk me through FIX session continuity across an engine restart.” Seqnums are session state, so they live in the log and snapshot like all other state. Re-logon then negotiates from the persisted numbers: each side says where its counters stand, and they gap-fill the difference. Name the naive-reset incident — coming up claiming sequence 1 against a venue counter at 12000, so one side demands what looks like a full day of traffic back. Even done right, resting orders survive at the venue during the gap, but you’re blind and can’t cancel.

Better architecture: a gateway tier owns the venue-facing TCP session, and engines deploy behind it. The venue never sees a logout — long-lived dumb edge, frequently-deployed smart core.

Then the crypto contrast, in its own breath: WebSockets have no seqnum contract at all — less to manage, more that breaks silently. A deploy means re-auth bursts into rate limits, resubscription storms, and a book-resync window per venue before its data is trustworthy. Different failure surface, same cure: the thin, long-lived edge.

Q4: “Design a shadow deployment. What does it not tell you?” Candidate consumes the live feed and order flow, runs full logic into a recording sink; stream-diff its decisions against production’s, classified expected vs. unexpected. Then the blind spots, unprompted — and derive the big one from the queue picture: the shadow’s order never actually stood in the price level’s line. So “would it have filled” is a guess about queue position, and those guesses skew optimistic — the simulator awards fills a real order, waiting behind everyone who arrived first, would not have gotten. That’s why shadow can’t measure market impact. It can’t test venue interaction either — rejects, rate limits, partial-fill sequencing — because nothing was ever sent. And an intended change diffs everywhere, so without classification the noise drowns real regressions. Land: shadow complements replay — today’s regime versus curated hard days — it doesn’t replace it.

Q5: “Deploy went out, metrics look bad. Roll back or fix forward?” Default roll back — old binary is known-good, new is a falsified hypothesis; but first, kill switch to flatten/limit risk so the decision isn’t made while bleeding. Forward-fix only if rollback is state-unsafe (new-format events already written), the bug predates the deploy, or the fix genuinely validates faster than rollback. The deciding factor: criteria were written in the runbook before the deploy, and rollback works because we hold N-1 compatibility (write-flips after bake) and test old-binary-reads-new-log in CI.

Q6: “When can the old binary NOT read what the new one wrote, and what then?” When the new version emitted new event/snapshot schema versions before the rollback horizon — a self-inflicted wound the readers-first/write-later choreography exists to prevent. If it happens anyway: patched N-1 that skips unknown record types (possible because records are length-prefixed and type-tagged), or restore from pre-deploy snapshot and reconcile against venues — which you classify as an incident with an RTO, not a rollback.

Q7: “How do you deploy across 20 venues without betting the firm?” Rolling by venue: canary on a small forgiving venue, mechanical health gates between waves (fills/rejects/latency/recon per venue), halt-and-rollback on regression, N/N+1 bus compatibility because mixed versions coexist for hours. Note the coupling caveat — shared risk or cross-venue strategies mean shards aren’t as independent as the deploy plan assumes — and that per-venue rollout doubles as per-venue-quirk testing.

Q8: “There’s no maintenance window in crypto. How does that change your engineering?” It removes the escape hatch that lets tradfi defer this whole chapter to 5:30pm: every deploy is market-hours, so hot-cutover, shadow, and rolling-by-venue are the baseline, synthetic windows are created by policy (quiet hours, pre-deploy exposure reduction with acknowledged P&L cost), and venue-side restarts function as involuntary failover drills. Then flip it: this is why crypto infra experience transfers up to tradfi — you’ve been doing the hard version daily.

The HFT Gap Closer