Question Bank: State & Deployment

25 questions, easy to brutal, covering chapters 13–17. This is the bank where your matching-engine and hot-standby experience does the most work — nearly every answer can be grounded in something you’ve built or run (engine, standby, pgbouncer/Postgres personal platform, 20-venue integrations). Speak in first person; cite the concrete thing; name the gaps rather than paper over them.

1. What does it mean for your engine to be deterministic, and why do you care? Same event log in, same state out — the engine is a pure fold over a totally ordered input stream, with time, randomness, and config changes arriving as events rather than ambient inputs. I care because everything expensive falls out of it: the hot standby is just a second consumer of the log, recovery is snapshot-plus-tail replay, testing is replaying production days through candidate builds, and incident forensics is exact re-execution. Lose determinism and all four break at once, usually silently.

2. Name the classic determinism violations and how you catch them. Wall-clock reads in the fold, RNG, HashMap iteration order (Rust randomizes it per process — the violation that passes every test and diverges in prod weeks later), floats with build- or hardware-dependent behavior (use integer ticks/lots), branching on I/O readiness or batch boundaries, and runtime config reads. Catching: dependency discipline on the core crate (no std::time or rand in its graph), dual-process replay in CI comparing fixed-seed state hashes — separate processes specifically to expose iteration-order bugs — and in production, continuous rolling hash comparison between primary and standby, which turns the replication pair I already run into a live determinism monitor.

3. How does snapshotting work without stalling the writer? The hot path never snapshots. A secondary replayer — in my case the standby, which already maintains identical state from the log — writes snapshots on its own schedule, tagged with the last applied sequence number; its pauses cost nothing. Alternatives I’d name: COW fork (Redis RDB style — works, but the fork stall and page-fault jitter land on the parent) and persistent data structures (pay per-operation forever to make snapshots cheap — usually the wrong trade for an engine). Durability details unprompted: write tmp, fsync, atomic rename, fsync the directory, checksum the blob, keep K generations so a corrupt latest snapshot isn’t fatal.

4. You replay an old log through a new binary. When is that valid? Valid when the new code is semantically identical on every event type in the log — refactors, perf work, additive features — and I prove it: CI replays recorded prod logs and requires state-hash equality. Invalid when the change intentionally alters decisions: replay then produces a history that never happened, while real fills went out under the old logic. Escapes, in preference order: log decisions (fills) not just inputs, so replay applies recorded outcomes verbatim; logic-version epoch events in the log so old segments replay with old semantics (the bitemporal move of ch13); or a snapshot fence — new code only replays from the cutover snapshot, with archived old binaries owning older segments.

5. How do you version events in a log retained for years? Never mutate a published version — v2 is a new type beside v1; every record carries (type, version) in its header; upcasters (pure v1→v2→v3 chains applied at read time) translate so the engine only ever sees the current model, with defaults that reproduce old behavior exactly; golden-file tests freeze the old decoders; the log is never rewritten. I’ve implemented exactly this shape — in my lab version the entire v2 upcast is one if ver >= 2 line in the codec, and that locality is deliberate: version sprawl lives in one file, not in the engine.

6. What’s the N/N+1 compatibility guarantee and why does rolling deploy need it? During any deploy, versions N and N+1 coexist on the same streams, so every schema change must be readable in both directions across one version step: new readers accept old messages via defaults, old readers accept new messages via unknown-field skip or extension semantics. Choreography: ship readers first, flip writers later — by config, after bake — and the classic outage is doing it in the other order. Note the asymmetry with the log: live traffic needs N/N+1, but replay needs N back to the oldest retained segment, which is what the upcaster chain is for.

7. Compare protobuf and SBE evolution rules in two breaths. Protobuf: tag-length-value, unknown fields skipped (and retained on re-serialize in proto3), evolution by adding fields, cardinal sin is reusing a field id — reserved exists for that; cost is varint and dynamic decode, fine for control plane. SBE: fixed offsets, decode is a pointer cast, evolution only by appending fields or claiming pre-reserved padding with zero-as-legacy-default, schema version in the header so old readers read their known prefix; cost is rigidity, and inserting a field mid-message silently corrupts every old record. Choose by tier: SBE where latency is the product, protobuf where cross-team evolution matters more than nanoseconds.

8. A venue announces a breaking protocol change. Walk me through your process. I’ve lived this repeatedly across 20+ integrations. The venue’s protocol exists only inside its feed handler and gateway — the normalize layer is the isolation boundary — so: capture raw traffic on the new API while the old still runs; build the new handler as a parallel module, never in-place edits; shadow it, diffing normalized output against the current handler live, which catches the undocumented changes (units, side conventions, snapshot depth); cut over that one venue with instant fallback; retain the old decoder as long as old captures exist. Downstream systems see zero change unless I choose an additive internal-schema update, which then follows the readers-first rollout.

9. Why doesn’t the hot path touch a database, and where does data actually live? Anything with a query planner, lock manager, or network hop is orders of magnitude off a microsecond budget — and that includes Redis, not just Postgres. State lives in process memory; durability is the append-only event log with a replicated standby; every database is a derived projection consuming the log: a columnar tick store for time-series analytics, Postgres for reference data, accounts, and compliance, Redis for ephemeral-by-policy coordination. The log is the system of record; databases are views. I run this split personally — Postgres 17 behind pgbouncer plus a Redis tier on my own platform, with Redis explicitly allowed to lose data.

10. Walk me through expand-migrate-contract for changing a column type on a live 500M-row table. In-place ALTER TYPE rewrites the table under ACCESS EXCLUSIVE — hours of downtime — so instead: expand — ADD COLUMN qty_v2 bigint (metadata-only, instant, with lock_timeout set); dual-write via app code or a sync trigger; migrate — backfill in primary-key-range batches, small idempotent transactions (IS DISTINCT FROM guard), throttled and resumable, watching replication lag and bloat; verify counts; enforce with ADD CONSTRAINT ... CHECK (qty_v2 IS NOT NULL) NOT VALID then VALIDATE CONSTRAINT (only SHARE UPDATE EXCLUSIVE) then SET NOT NULL (instant on PG12+ because the CHECK proves it); switch reads; contract — days later, a separate deploy drops the trigger and old column. The invariant that makes it safe: every intermediate schema works with app versions N and N+1, and each step rolls back independently.

11. Which ALTER TABLE operations are traps, and what’s the meta-trap? Traps: ALTER COLUMN TYPE (full rewrite), ADD COLUMN DEFAULT volatile-fn (rewrite — constant defaults are instant since PG11), SET NOT NULL pre-PG12 without the CHECK trick (full scan under exclusive lock), plain CREATE INDEX (blocks writes), VACUUM FULL (exclusive for its whole run — use pg_repack instead). The meta-trap is lock queuing: even an instant ALTER needs ACCESS EXCLUSIVE, waits behind one long-running query, and every subsequent statement — including plain SELECTs — queues behind it, so production freezes while your migration does literally nothing. Defense: SET lock_timeout = '2s' plus retry built into the migration runner, and a lint layer (squawk / strong_migrations style) in code review.

12. CREATE INDEX CONCURRENTLY failed halfway. What state are you in? An INVALID index: it’s left behind, maintained on every write (pure overhead) but unusable for reads — you must DROP INDEX CONCURRENTLY and retry. Other failure modes to volunteer: it waits out every older transaction, so one idle-in-transaction session (a wedged pooler client, a forgotten psql) can stall it indefinitely; it can’t run inside a transaction block, so your migration tool needs a non-transactional mode; and unique builds can fail late on duplicates. Knowing “INVALID index” cold is the tell that you’ve actually run this, not read about it.

13. Streaming vs logical replication — pick for three scenarios: HA, major version upgrade, feeding ClickHouse. HA: streaming — physical WAL bytes, exact replica, sync or async per your durability need, with Patroni automating failover. Major upgrade: logical — decoded row events are version-independent, so you replicate into the new-version cluster and cut over near-zero-downtime. Feeding ClickHouse: logical decoding as CDC, Debezium-style — the database becoming an event producer, which is the mirror image of my engine’s log-projection pattern. Teeth to mention: logical replication doesn’t carry DDL, sequences don’t replicate, and an abandoned replication slot pins WAL until the primary’s disk fills — the classic 3am incident.

14. What does Patroni actually solve, and what are the concepts underneath? It automates leader election and failover: an agent beside each Postgres, a leader lease in etcd or Consul (a key with a TTL that only the leader keeps renewing — lose the renewal, lose the crown), promotion of the least-lagged replica on lease loss, client rerouting via proxy or health endpoints. Underneath are exactly my engine’s hot-standby problems wearing DB clothes: fencing (the demoted primary must not accept writes, or you get split-brain and forked timelines), bounded data loss (the async lag window, maximum_lag_on_failover, or synchronous replication to zero it), and divergence repair (pg_rewind to rejoin the old primary as a replica). I make that mapping explicitly, because it’s true. I run single-node Postgres with my own failover being “restore from backup” — Patroni I know as architecture, not as scar tissue, and I’d say so.

15. Why is pgbouncer necessary and what breaks under transaction pooling? Postgres backends are processes with real per-connection memory; connection storms from fleets of app instances collapse a server that’s perfectly happy at 50 active backends, so pgbouncer multiplexes thousands of client connections onto tens of server connections. Transaction pooling — the production default — breaks session state: named prepared statements (protocol-level support only in recent pgbouncer), persistent SETs, session advisory locks, LISTEN/NOTIFY. I run one in production on my own platform: host-native pgbouncer on the public port routing by database name to a loopback Postgres 17, roughly 25 backend connections per tenant DB under a 500-client ceiling — and I can also describe the isolation trade-off I consciously accepted in that stack — a cache tier that bypasses per-tenant auth (ch15).

16. How do you deploy a new matching engine version during market hours? Hot-standby cutover, run as a runbook with mechanical gates, in six phases:

Pre-verify — replay-regression with classified decision-diffs, plus an N-1 read-back check.
Follower start — the new binary comes up as a follower: snapshot load, tail replay, live consumption with outputs sinked.
Compare — rolling state-hash comparison against the incumbent while both run.
Cut — at a logged sequence boundary (a LeadershipTransfer event), fencing the old primary by epoch number on every log append and outbound order.
Reconcile — open orders checked against every venue before restoring full size.
Bake — the old binary stays hot and fenced as the instant rollback target through the bake period, and the new binary keeps writing old-format events until past the rollback horizon.

Every gate is a pre-committed number, not a judgment call at T-0.

17. How do venue sessions survive an engine restart? Mostly they don’t — that’s the hard part. FIX: sequence numbers are session state, so they belong in the log/snapshot like all state; re-logon negotiates gap-fill from the persisted seqnums (a naive reset triggers replay-or-reject storms — the peer expects seq N, sees seq 1, and either demands a resend of everything or rejects the session outright); resting orders survive at the venue, but you’re blind and can’t cancel during the gap. Crypto, my daily world: no seqnum contract — instead you get re-auth bursts into rate limits, resubscription storms, and per-venue book resyncs, mitigated by pre-warming connections where venues tolerate duplicate sessions, which is a per-venue quirk I literally keep a table of. The architectural answer for both worlds: a thin, rarely-deployed gateway tier owns the venue sessions so the frequently-deployed engine restarts behind an unbroken connection — the same isolation move as the feed-handler normalize layer.

18. Design a shadow deployment and tell me its blind spots. The candidate consumes the live feed and order flow, runs full logic, and its outputs go to a recording sink; a comparator stream-diffs its decisions against production’s, with expected-vs-unexpected classification — an intended change diffs everywhere, and without classification the real regressions drown. Shadow tests today’s regime, which recorded replays can’t. Blind spots I volunteer before being asked: market impact (its fills are simulated, and queue-position modeling is where fill simulators lie), venue interaction (rejects, rate limits, partial-fill sequencing), and anything triggered by its own orders’ effects on the market. That’s why the ladder is replay → shadow → capped canary: each rung covers the previous rung’s blindness, and only the canary — real money under hard risk-layer caps — sees impact.

19. Design the kill-switch system for a multi-venue trading firm. Taxonomy by blast radius: per-strategy, per-venue (the tier I’d use most across 20 venues), per-symbol and per-account, global cancel-everything, plus a tiered flat-position action — passive-flatten with a deadline, then aggressive, because naive market-order flattening into a dislocated book realizes the worst possible price. Engineering: enforced at the minimal-dependency edge (gateway/risk layer) so it works even when the engine is wedged; state persisted so a restarting engine comes up stopped if the switch was pulled; pull-cheap/reset-expensive authority — anyone on the desk pulls, seniority plus a checklist un-pulls; every pull logged with who/when/why; and scheduled production drills with measured time-to-stopped, where a failed drill is a P1. Close with: regulators mandate kill functionality anyway (RTS 6 flavor), so build it once, properly, and let the drill records double as compliance evidence.

20. Runtime feature flags in the hot path — argue both sides, then pick. For: instant enable/disable without a deploy, gradual rollout, operational flexibility — real virtues in web systems. Against, and decisive on a deterministic path: every flag is a branch and possibly a shared-state load; 2^N flag combinations of which you tested three; and a runtime flip changes behavior without passing replay-regression — a bypass around the entire verification pipeline, which is how Knight Capital died (a repurposed flag plus a partial deploy). My position: config-at-startup — behavior toggles read once into immutable config, so every change is a deploy through the gates, with hot-path variants monomorphized at init; the only runtime-mutable controls are the enumerated risk plane (kill switches, limits, throttles), which is deliberately not a feature system. And if a toggle must affect the fold mid-session, its changes enter as logged events so replay stays truthful.

21. Your standby diverged from primary — how do you find out, and what do you do? Find out by construction, not by luck: both sides publish rolling state hashes keyed by sequence number every N events; a comparator alerts on first mismatch, so detection latency is bounded and it pages me long before a failover would need that standby. Immediate action: the divergent standby is disqualified as a failover target — a standby with wrong state is worse than none — while the primary keeps trading and I spin a replacement standby from snapshot-plus-replay. Diagnosis: offline, replay the log from the last matching snapshot on both binaries and bisect to the first diverging sequence number, then inspect how that event was applied — the culprit is almost always a determinism violation (unordered iteration, config skew between hosts, a float path) or, rarer, torn log shipping, which per-record checksums distinguish. Then the fix becomes a permanent CI determinism test, so the class of bug dies, not just the instance.

22. How do you reconcile engine state against the venues and against your own DB? Two loops. Venue recon — the one that costs money: diff open orders, fills, and balances against drop-copy or execution feeds plus REST snapshots, continuously at low rate and mandatorily on any restart before trading resumes; in crypto the REST view is rate-limited and eventually consistent with the venue’s own WS stream, so the logic needs tolerance windows, not equality asserts — I’ve been burned by treating a venue’s REST snapshot as instantaneous truth. DB recon: compare aggregates — positions, open quantity, cash — between an engine snapshot at sequence N and the projection’s view as of N; sequence numbers make “the same instant” well-defined, and without them recon chases its own tail. Doctrine: recon detects, runbooks decide, and every correction is applied as a new logged event (ManualAdjustment { reason, ticket }) — never by mutating state or rows in place, or you’ve corrupted replay and created drift the next recon can’t explain.

23. Rollback vs forward-fix at 3am — give me the decision tree and its prerequisites. Kill switch first — flatten or cap the risk so the decision isn’t made while bleeding — then default to rollback: the old binary is a known-good artifact, the new one is a hypothesis you just falsified. Forward-fix only if rollback is state-unsafe (new-format events already written past what N-1 can read — which the write-flip-after-bake discipline exists to prevent), the bug predates the deploy so rollback changes nothing, or the fix genuinely validates faster than the rollback — rarer than it feels at 3am. Prerequisites that make the tree real: the criteria were written in the runbook before the deploy; N-1 compatibility with everything the new binary wrote (events, snapshots, config) is tested in CI by replaying new-written logs through the previous release; and the old binary is still resident, fenced, and at the head of the log. An unrehearsed rollback isn’t a rollback — it’s a second incident.

24. Brutal: a bad fill report from a venue corrupted your position state six hours ago and you’ve been trading on it since. Unwind the situation. Stop the bleeding at the right scope first: kill switch for the affected strategies or venue, establish true exposure by reconciling against every venue’s authoritative view — their fills, not my state — and flatten to safe bounds if the divergence is material. Then the event-sourced unwind: the corrupting input is in the log with a sequence number, so I can replay to the moment of corruption, quantify exactly how state diverged, and correct forward by applying adjustment events (or a corrected interpretation of the venue’s amended report) — while the actual log stays immutable as the audit record of what the system genuinely believed and did, which both compliance and the post-mortem need. The six hours of decisions were made on false state, but those orders are real and stand; corrections are forward-looking events, never retroactive mutation. Close the loop: the bad report becomes a permanent replay-regression fixture, and the recon cadence that let it live for six hours gets shortened — the six hours was the real failure, not the bad message.

25. Brutal: “You’ve never worked in tradfi. Why should we trust you with our deployment and state architecture?” Because the hard version of this problem is the one I already operate: crypto has no maintenance windows — every deploy I’ve ever done was during market hours; venues restart their matching engines under me routinely, which amounts to involuntary failover drills; and 20+ live integrations mean protocol evolution, session chaos, and reconciliation are my daily reality, not a quarterly event. The architecture I run is the one your engineers respect: a deterministic event-sourced core, sequenced log as the record, hot standby by log consumption, replay-based verification — and I can defend each choice down to why HashMap iteration order breaks replay and which upcast defaults preserve history. What I’d be new to is specific and bounded: FIX session mechanics at your particular venues, your regulatory evidence chain, colo-grade tooling — named gaps with learning plans, not conceptual gaps. And the meta-point I’d actually say: someone who can tell you precisely what they haven’t run is safer around your production state than someone who can’t.

The HFT Gap Closer

Question Bank: State & Deployment

Further reading

Keyboard shortcuts

The HFT Gap Closer

Question Bank: State & Deployment

Further reading