Event Sourcing as a Deployment Primitive

Before you start — this chapter leans on a handful of primer ideas:

WAL, snapshots, and replay in databases — Postgres does internally exactly what this chapter does architecturally; seeing it there first makes everything here familiar: ch00f

Exchange feed architecture and sequence numbers — the ITCH-lineage “sequenced stream” pattern this chapter generalizes: ch00f

Memory pages and copy-on-write — needed only for the fork-based snapshot pattern: ch00b

Tick-to-trade and the trading-system vocabulary (fills, books, positions, venues): ch00f

Read those first — 20 minutes there saves an hour here.

You already built an event-sourced matching engine with snapshot/replay and a hot standby. Good — you know the shape. This chapter is about the parts that only show up in production and in interviews:

The determinism contract as an auditable property, not a hope.
Snapshotting without stalling the writer.
Replay speed as an engineered number, not whatever you get.
Replaying old logs through new code — the versioning trap.
The sequencer (the component that assigns a global order — sequence numbers — to every input event; full section below) as the architectural center of the system.

By the end, “event sourcing” should stop meaning “a pattern I used” and start meaning “the mechanism that makes deployment, recovery, testing, and compliance all fall out of one design decision.”

Why the event log is a deployment primitive, not a persistence trick

Most engineers meet event sourcing as a persistence pattern: instead of storing state, store the deltas. In a trading engine it’s more than that. The log is:

The database of record. The book, positions, open orders — all derived state. If it’s not in the log, it didn’t happen.
The replication protocol. Your hot standby is just a second consumer of the same log. You already know this; hold onto it, because it generalizes.
The deployment mechanism. A new binary that can replay the log to identical state can take over from the old binary. That’s the zero-downtime chapter (ch16).
The test oracle. Replay yesterday through the candidate build, diff decisions. That’s the change-management chapter (ch17).
The compliance artifact. Regulators asking “why did you send this order at 14:32:07.123456” get an exact answer, not a log-grep guess.

One design decision buys all five — if you hold the determinism contract. Lose determinism and you lose all five at once, usually silently.

The determinism contract

The contract: same log in, same state out, on any conforming binary, any machine, any time. Formally, your engine is a pure function fold(initial_state, events) -> state. If you know Array.reduce or a Redux reducer, you already know fold — same shape: (state, event) => newState, applied to every event in order. And the contract is the pure-reducer rule. Redux bans Date.now() and Math.random() inside reducers for the same reason: anything impure goes into the action (the event), so replaying the events always rebuilds the same state. Everything that violates purity must be pushed out of the fold and into the log itself.

The forbidden inputs

Wall clock. SystemTime::now() inside the state machine is the classic sin. Time must arrive as an event field, stamped by the sequencer when the event was ordered. If your engine needs “current time” (for order expiry, session close), it consumes timer events that are themselves in the log. Replay then sees the exact same timestamps.
Randomness. No rand::thread_rng() in the fold. If you genuinely need randomness (randomized order queue priority on some venues, jittered internal IDs), the seed or the drawn value goes into the log as part of the input event.
Iteration order of unordered collections. HashMap iteration order in Rust is randomized per-process (SipHash with a random key). If you ever iterate a HashMap and the iteration order affects an output — say, cancelling all orders for a client and the sequence of cancel events matters — you’ve broken determinism. Use BTreeMap, IndexMap (insertion-ordered), or sort before iterating. This is the one that passes every unit test and fails in production three weeks later.
Floats — with nuance. IEEE 754 arithmetic is actually deterministic for the same operations in the same order on the same settings. The dangers are: (a) compiler reassociation under -ffast-math-style flags (Rust doesn’t do this by default — a point worth stating in an interview), (b) different builds rounding intermediate results differently — e.g. one build computes a*b+c as a fused multiply-add with one rounding while another rounds after the multiply and again after the add, so the last bit of the result differs (the historical version of this is x87 hardware keeping 80-bit intermediates), (c) accumulation order changing when you refactor. The professional answer: use fixed-point integers for prices and quantities (price in ticks, quantity in lots, i64 everywhere). Floats are for analytics, never for the state machine.
External I/O and channel timing. Any if socket.ready() branch, any “batch until the queue is empty” logic where batch boundaries affect state, any cross-thread race. Concretely: if a fee discount applies per batch and live processing happened to drain 10 events as two batches of 5, a replay that drains them as one batch of 10 computes a different fee — state diverges even though the events are identical. The fold must be single-threaded over a totally ordered input.
Config read at runtime. If a config value affects decisions, either it’s constant for the life of the log segment, or config changes are themselves events in the log (ConfigUpdated { key, value }). Otherwise replaying with today’s config against last month’s log produces different state.

Auditing for determinism

You will be asked “how do you know it’s deterministic?” Weak answer: “we’re careful.” Strong answers, layered:

Structural audit: make the forbidden inputs unimportable — the same idea as an ESLint rule banning Date.now inside reducers, enforced at the dependency level. The state machine crate has #![no_std]-adjacent discipline — no std::time, no rand, no I/O in its dependency graph — enforced with clippy lints, a cargo deny rule on the crate’s dependencies, and code review convention: the engine-core crate takes events in, emits events out, nothing else.
Dual-run in CI: replay the same log twice in two separate processes (fresh state each time), compare a state hash. Cheap, and it catches HashMap-iteration bugs precisely because hash randomization differs per process.
Cross-run in production: your primary and standby are already dual-running live. Continuously compare rolling state hashes (e.g., every 10k events, both sides publish hash(state) keyed by sequence number). Divergence pages you before failover would have hurt. This is the tie-in to your hot-standby: you built the replication; the hash comparison is the cheap upgrade that turns it into a determinism monitor.
Nightly replay: replay today’s full log on a different machine class, compare final hash against the primary’s end-of-day hash.

A practical state hash: fold a structural hash over the book (per level: price, total qty, order count) plus positions plus sequence number. Don’t hash incidental fields (internal pointer-ish IDs, capacity of vectors).

#![allow(unused)]
fn main() {
fn state_hash(book: &Book, seq: u64) -> u64 {
    use std::hash::{Hash, Hasher};
    let mut h = twox_hash::XxHash64::with_seed(0); // fixed seed!
    seq.hash(&mut h);
    for (price, level) in book.bids.iter() {        // BTreeMap: ordered
        price.hash(&mut h);
        level.total_qty.hash(&mut h);
        for o in &level.orders { o.id.hash(&mut h); o.qty.hash(&mut h); }
    }
    // ... asks, positions ...
    h.finish()
}
}

Fixed seed matters: DefaultHasher with a random key gives you a hash that can’t be compared across processes.

Snapshot mechanics: don’t stall the writer

Naive snapshotting: pause the engine, serialize state, resume. At 1M events/sec with a book that serializes in 200ms, that’s a 200ms hole in your latency distribution — unacceptable. Three production patterns:

1. Secondary replayer snapshots (the usual answer)

The hot path never snapshots. A separate process (or your standby!) consumes the same log, maintains the same state, and snapshots its copy at leisure. Its pauses cost nothing. Snapshot = (state_blob, last_applied_seq). Recovery = load blob, replay log from seq+1.

This is the pattern to lead with in interviews because you’ve effectively already run it: your hot standby is a secondary replayer; giving it a “write a snapshot every N minutes” job is a small delta. It also means snapshots are implicitly determinism-checked — if the standby’s snapshot replays to a different hash than the primary’s live state, you’ve caught divergence.

2. Copy-on-write fork

fork() the engine process; the child inherits a copy-on-write (COW) view of memory frozen at the fork instant — parent and child share pages until one of them writes (ch00b) — and serializes it while the parent keeps trading. Redis does exactly this for RDB saves. Costs: fork itself is not free (page-table copy — hundreds of µs to ms for big heaps, and it stalls the parent for that duration), COW page faults add jitter to the parent as it writes, and memory can transiently double under heavy write load. Viable, used in practice, but the page-fault jitter is why latency-sensitive shops prefer pattern 1.

3. Immutable / persistent data structures

Book built on persistent structures (e.g., an immutable tree per side); snapshot = grab the current root pointer, serialize from it on another thread while the writer keeps producing new versions. The root pointer works like a git commit: grabbing it pins an immutable view of the whole tree at that instant, while new “commits” build on top without disturbing it. Bounded jitter, but you pay per-operation allocation/indirection cost on the hot path forever to make occasional snapshots cheap. Usually the wrong trade for a matching engine; worth naming to show you know the space.

Snapshot atomicity details that interviewers probe: write to snapshot.tmp, fsync, rename to snapshot.{seq} (rename is atomic on POSIX), fsync the directory — the rename lives in the directory’s own data, so skipping that fsync means a crash can forget the file was ever renamed. Keep the last K snapshots — a corrupt latest snapshot must not be fatal; fall back to the previous one and replay a longer tail. Checksum the blob (xxhash/crc32c) and validate on load. Record the exact seq inside the blob, not just the filename.

Log compaction and retention economics

The log grows without bound; state doesn’t. Retention policy is snapshots + tail:

Operational tier: you need latest_snapshot + tail to recover. Keep several snapshot generations and the log back to the oldest one you’d trust. On fast NVMe.
Regulatory/analytical tier: full raw log, compressed, shipped to object storage. Order/quote data compresses extremely well (delta-encoded integers, repetitive symbols) — 10–20x is normal. Multi-year retention is a compliance requirement in tradfi (order record-keeping obligations), and it’s cheap: even 100GB/day raw is single-digit GB/day compressed, dollars per month in S3-class storage.
Compaction in the Kafka sense (keep last value per key) is mostly wrong for a trading log — you need the history. Compaction applies to derived/reference topics (latest config, latest position snapshot), not the order-flow log.

Do the arithmetic out loud in interviews: 500k msgs/sec × 64 bytes ≈ 32 MB/s ≈ 115 GB per 8h session raw; ~10 GB/day compressed; three years ≈ 11 TB — one cheap object-storage bucket. Retention is never the technical bottleneck; the bottleneck is replay time, next section.

Replay speed engineering

Recovery time = snapshot load + tail replay. Tail replay speed also gates your nightly regression replays (ch17). Target: replay an 8-hour day in minutes. That means replaying at 50–200x real time. How:

No-I/O replay mode. In live mode, applying an event emits outbound messages (acks, fills, market data) to sockets. In replay mode, outputs are either discarded or written to a buffer for diffing — never sent. Gate this with a mode enum, not scattered ifs: the fold returns Vec<OutboundEvent> (or writes into a caller-supplied sink), and the caller decides live-publish vs. discard vs. record. The state machine itself doesn’t know which mode it’s in — that’s what keeps replay honest.
Sequential, batched reads. The log is append-only, so replay is a pure sequential scan. Read in large chunks (4–64 MB), decode in-place with zero-copy framing. NVMe gives you multiple GB/s; decode is usually the bottleneck, not the disk.
Cheap decode. Fixed-layout binary events (ch14) decode at memory bandwidth. If your replay is slow, the usual culprits are per-event allocation, serde-style dynamic deserialization, or logging. Replay profile should show ~all time in the fold itself.
Bound the tail. With a snapshot every N minutes, tail replay is bounded regardless of day length. Snapshot cadence is a knob: snapshot interval × replay speed = worst-case tail time. If you replay at 5M events/sec and snapshot every 50M events, tail replay ≤ 10s.
Parallelism — careful. The fold itself is inherently sequential (that’s the contract). You can parallelize decode/checksum ahead of the fold (pipeline: reader thread → decode threads → single apply thread), and you can replay independent symbols/shards in parallel only if they’re truly independent partitions with independent logs. Cross-symbol state (margin, or self-match prevention across books — blocking your own buy order from trading against your own sell) breaks that; know which side of the line your engine is on.

Numbers to have ready: a clean Rust fold applying simple book events does 5–20M events/sec/core. An 8h day of 500k/sec = ~14.4B events… which is why nobody replays whole days from genesis: 14.4B ÷ 10M/sec ≈ 24 minutes — fine for nightly regression, too slow for recovery. Hence snapshots: recovery replays minutes of tail, not hours.

Replaying old logs through new code: state-machine versioning

Event sourcing meets deployment here, and it’s where interviewers separate people who’ve read the blog posts from people who’ve operated the thing.

The question: binary v2 replays a log written (and originally applied) by binary v1. When is the result valid?

Valid when: v2’s fold is semantically identical on all event types that appear in the log. Adding new event types, adding fields with defaults (via upcasters — ch14), refactoring internals, performance work — all fine. The state hash after replaying the v1 log through v2 must equal v1’s hash. This is precisely your deploy gate: no behavior change intended → hashes must match, and you verify that in CI by replaying recorded prod logs.

Dangerous when the change is a semantic fix. Suppose v1 had a bug: it matched against a stale level in some edge case. v2 fixes it. Now replaying the v1 log through v2 produces a different book than production actually had — but production’s history really happened; real fills were sent to real counterparties. You cannot retroactively “fix” the past. Options, in order of preference:

Log outputs, not just inputs. If fills/executions are themselves events in the log (the sequencer logs the engine’s decisions, not just requests), replay applies recorded fills verbatim and you sidestep the problem: old segments replay old decisions, new events get new logic. Many real engines log decisions for exactly this reason — it decouples “reconstruct state” from “re-run logic.”
Effective-version epochs. Log a LogicVersionChanged{v2} event at deploy time. The fold dispatches on the active logic version: events before the marker replay with v1 semantics, after with v2. You’re keeping the old code path alive — bitemporal in spirit (what we knew/did then vs. what we’d do now). Costs code retention; prune once segments age past retention.
Snapshot fence. Deploy v2 with a fresh snapshot taken at cutover; declare logs before the fence non-replayable through v2 (only through archived v1 binaries — keep them!). Simple, common, and what most shops actually do. The compliance archive still has the raw log plus the v1 binary artifact if a regulator asks.

Say the word “bitemporal” and then explain it plainly: two time axes — when it happened vs. what logic/knowledge applied — and your log must let you reconstruct along the first axis without contamination from the second.

Sequencer patterns: who owns the order of events

Determinism requires a total order of inputs. Something must impose it. That thing is the sequencer, and it is the real single point of design in every serious engine. The whole architecture in one picture:

 orders ────────────┐
 market data ───────┤     ┌──────────────────┐    sequenced log
 timer events ──────┴───► │    SEQUENCER     │──► ①②③④⑤...
                          │ stamps each input│         │
                          │ with 1, 2, 3, …  │         ├──► matching engine
                          └──────────────────┘         ├──► risk
                                                       ├──► hot standby
                                                       └──► drop-copy

Everything left of the sequencer is unordered chaos arriving on many wires; everything right of it consumes one identical, totally ordered stream. Once order is imposed exactly once, every consumer — engine, risk, standby, drop-copy — is just a deterministic fold over the same log.

Single-sequencer architecture (the classic tradfi pattern). One process receives all inputs (orders, market data callbacks, timers), assigns monotonically increasing sequence numbers, writes the log, and multicasts/streams the sequenced log to every consumer — matching engine, risk, drop-copy (the duplicate feed of your own executions kept for reconciliation — ch00f), standby. Everything downstream is a deterministic function of the sequenced stream. This is the design behind exchange architectures in the NASDAQ/ITCH lineage and most prop-shop internal buses. (ITCH is NASDAQ’s sequenced multicast market-data feed — ch00f.) The payoff: replication, recovery, and fan-out are all “consume the log.” Its cost: the sequencer is a SPOF (single point of failure) and the latency floor (everything crosses it).
Sequencer failover is then the hard sub-problem: a standby sequencer must take over without gapping or double-assigning sequence numbers. Options: shared reliable log the standby resumes from; or consensus.
Raft / Aeron Cluster. (Raft: the standard consensus algorithm for getting several nodes to agree on one log; Aeron: a low-latency messaging and clustering library from the LMAX/Real Logic lineage — the same people as the Disruptor.) Aeron Cluster is the production-grade off-the-shelf version of “replicated deterministic state machine”: Raft consensus orders the input log across 3–5 nodes, each node runs your deterministic service (clustered service model), snapshots and log replay are built into the framework, and leadership transfer is the failover story. Used in real tradfi matching engines and post-trade systems. The trade: consensus adds a quorum round-trip to every input — a majority of the 3–5 nodes must acknowledge each event’s position in the log before it counts (~tens of µs on a good LAN with kernel bypass) — versus a naive single sequencer, in exchange for principled failover. Know both designs and the trade; interviewers love “single sequencer vs. Raft — when and why.”
Your world: your primary/standby pair with log shipping is the single-sequencer pattern with a manually-managed standby. The interview upgrade is being able to say what you’d need for automatic failover: fencing (old primary must be unable to write after takeover — every append carries the leader’s epoch number, and the log and its consumers reject appends stamped with an old epoch, so a deposed primary’s writes simply bounce), gap-free handover (standby confirms it has the full log to seq N before claiming N+1), and split-brain prevention (leases/quorum, never “ping timed out so I’m leader”).

Recovery drills: RTO from snapshot + tail

An untested recovery path is a rumor. Components of RTO (recovery time objective):

Detect failure (health checks, watchdog): target seconds.
Load latest snapshot: size / disk bandwidth — a 5 GB snapshot on NVMe ≈ 2–3s plus deserialize.
Replay tail: bounded by snapshot cadence (see arithmetic above — engineer this to seconds).
Re-establish venue sessions and reconcile open orders (often the longest pole — the zero-downtime chapter (ch16) covers session takeover).
Resume, initially in a safe mode (cancel-only or reduced limits) until reconciliation confirms state matches the venues’ view.

Drill it: monthly, kill the primary for real in production-like conditions (staging with recorded feed at minimum; the brave do game-days in prod with tiny limits). Measure each stage. The number you quote in interviews should sound measured, not aspirational: “our snapshot+tail recovery was ~X seconds; the venue re-logon dominated at Y; here’s what we did about Y.” You have hot-standby failover experience — mine it for one concrete story with numbers before interview day.

Plain-English recap

The log is a double-entry ledger. You never edit a posted entry; you append. Balances (books, positions) are derived by summing the entries, and any “correction” is a new entry with a reason — the accounting discipline you already trust with money, applied to all state.
The determinism contract is the pure-reducer rule. Impurity belongs in the action, not the reducer — push clocks, randomness, and config into the logged event and replay can never disagree with itself. The HashMap-iteration-order trap is just the sneakiest impurity.
State-hash comparison is reconciliation with a statement date. Comparing primary and standby hashes at the same sequence number is exactly comparing your ledger balance to the PSP settlement report at a common cutoff — without the common cutoff, recon chases its own tail.
Snapshot + tail replay is backup + WAL. Load the checkpoint, replay everything after it. Snapshot cadence × replay speed = worst-case recovery time, a knob you engineer, not a hope.
The semantic-fix trap is “you can’t retroactively re-price settled payments.” A bug fix changes what the engine would have done, but real fills went to real counterparties — history happened. Logging decisions (not just inputs) is the ledger answer: old entries replay verbatim, new logic applies only going forward.
The sequencer is your single Kafka partition / single Postgres primary. Total order has to come from somewhere; one process assigning sequence numbers is the cheapest way, and then replication is “everyone consumes the same partition.” The failover problems (fencing, split-brain) are the same ones Patroni — Postgres’s automated-failover agent — solves in the databases chapter (ch15).

Interviewer will ask

Q1: “How do you guarantee your replay is deterministic?” Same picture as the Redux reducer: the fold must be pure, and determinism breaks wherever the same log could produce two different states. So you hunt the leak sources one by one — anything the fold reads that isn’t in the log. Wall clock: time enters as an event, so replay sees identical timestamps. Randomness: the seed or drawn value is itself logged. Float rounding: fixed-point integers, so no build can round differently. Map iteration order: ordered collections wherever iteration touches output. Then don’t promise it — audit it: the core crate can’t even import time or rand, CI replays the same log in two separate processes and diffs state hashes, and production compares primary/standby hashes continuously. The caveat: HashMap iteration passes every single-process test, because the random hash key only differs across processes — exactly why the CI dual-replay uses two processes.

Q2: “How do you snapshot a live engine without pausing it?” Name the problem first: pausing to serialize a big book is a 200ms hole in the latency distribution, so the hot path must never snapshot. The standby already consumes the same log and holds the same state, so it snapshots its copy at leisure — a state blob tagged with the last applied sequence number. Mention COW-fork (Redis RDB) as the alternative and why it loses: the fork itself stalls the parent, and copy-on-write page faults add jitter afterward. Then the durability details, each with its why: tmp file plus atomic rename, because a crash mid-write must never leave a half-snapshot under the real name; fsync the directory, because the rename lives in the directory’s own data; checksum the blob, because corruption must fail loudly at load, not silently at failover; keep K generations, because a corrupt latest must not be fatal — fall back one and replay a longer tail.

Q3: “Your standby’s state hash diverged from primary. What now?” First: the divergent side stops being a failover candidate immediately — a standby with wrong state is worse than none. Then bisect: replay the log from the last matching snapshot on both binaries offline, find the first event where hashes diverge, inspect. Usual suspects: unordered iteration, a float sneaking in, or config skew between the two hosts. It’s almost never “cosmic rays”; it’s almost always a determinism-contract violation that CI’s dual-replay didn’t cover.

Q4: “You fixed a matching bug. What happens to replay of old logs?” Show you see the trap: replaying old logs through fixed code produces state that never existed — real fills went to real counterparties under the old logic, and history can’t be re-run. Three ways out, each with its mechanism. Log decisions, not just inputs: replay then applies recorded fills verbatim, so old segments never re-run any logic at all — the best answer. Logic-version epochs: a marker event in the log, so the fold applies v1 semantics before it and v2 after — bitemporal, at the cost of keeping old code alive. Snapshot fence: fresh snapshot at cutover, older logs replayable only through the archived v1 binary — the pragmatic answer for most shops. Then say which you’d pick, and why, for the system at hand.

Q5: “How fast can you recover, and how do you know?” Run the clock on a concrete config instead of reciting a formula. Say snapshots every 30 seconds and a peak log rate of 100k events/sec. Second 0: the primary dies; the watchdog misses a few 1kHz heartbeats and declares it dead inside a second. Seconds 1–3: load the last snapshot — a few GB, mostly sequential read and deserialize. Seconds 3–5: replay the tail — worst case 30 seconds of log, ~3M events, and because replay is the pure fold with no network waits it runs at millions of events/sec, so ~1–2 seconds. State is now current: books, positions, sequence numbers. But nobody can trade yet — venue sessions still have to come back: re-logon, sequence negotiation, resend processing, 10–20 seconds per venue in parallel, and reconciliation against venue state is the gate before orders flow. So: state in ~5 seconds, trading in tens of seconds — and the tuning knob is snapshot cadence, because halving the interval halves the worst-case tail. How do I know? We drilled it by killing the process, and the drill is what exposed that session re-logon dominated, not replay — recovery numbers you haven’t measured by killing the process are fiction.

Q6: “Why a single sequencer? Isn’t that a SPOF?” Yes, deliberately: a total order must be imposed somewhere, and one process assigning sequence numbers is the lowest-latency way to do it; everything else becomes a deterministic consumer, which makes replication and recovery trivial. The SPOF is then handled by standby-with-fencing or by paying a quorum round-trip for Raft (Aeron Cluster) when you need automatic failover. The wrong answer is distributing ordering ad hoc — then nothing agrees on history.

Q7: “Kafka is an event log. Why not build the engine on Kafka?” Anchor in the sequencer picture: the engine needs one totally ordered stream, and something must impose that order. Kafka gives total order only within a single partition, so the engine would use exactly one partition — the parallelism Kafka exists to provide buys you nothing. And every event would cross the network to a broker and back before the fold sees it — a hop the microsecond budget can’t pay, versus an in-process append to a memory-mapped log. So: right shape, wrong tier. The caveat that shows judgment: Kafka still belongs in the architecture — downstream, shipping the sequenced log to analytics, risk, and archive, where its fan-out is exactly right.

Q8: “What’s in your snapshot, exactly — and what’s deliberately not?” Everything needed to resume the fold: books, orders, positions, session-level counters, active timers, active config, and the sequence number it corresponds to. Not in it: anything derivable that’s cheaper to rebuild than to store, and anything non-deterministic (socket state, wall-clock). Also: versioned snapshot format with its own compatibility story — a snapshot is just a big event, and it needs the same never-mutate versioning the log itself needs (next chapter).

The HFT Gap Closer