Lab III: Live-Upgrading an Event-Sourced Book

Before you start — this lab compresses Part III; have these fresh:

The determinism contract and state hashes — every PASS in this lab is that contract, executed: chapter 13

Upcasters and wire versioning — the one-line if ver >= 2 the whole lab turns on: chapter 14

Hot-standby cutover and fencing — what phase 4 is a miniature of: chapter 16

What a limit order book is (bids, asks, price levels, resting orders): ch00f

Read those first — 20 minutes there saves an hour here.

Chapters 13–16 in one runnable file. You will build a tiny event-sourced limit-order-book, log it to disk in a v1 wire format, snapshot it, then perform the exercise that is the interview: ship a v2 schema (adds an order source tag), write the upcaster (the read-time version translator of ch14), and prove two things with state hashes: (a) the v2 binary replays the v1 log to bit-identical state, and (b) a v2 “process” (a second engine instance in the same binary; Extension 1 makes it real) can tail a live v1 writer, take over mid-stream, and continue the log in v2 — after which a cold replay of the mixed log reproduces the leader’s state. Zero dependencies, so nothing hides the mechanics. This code compiles and the output below is real (rustc 1.92).

The five phases on one timeline:

 phase 1   A (v1 binary) writes 1000 v1 records ─────────────────► log
 phase 2   B (v2 binary) cold-replays that log  ► hash == A's?   PASS
 phase 3   B snapshots, reloads the snapshot    ► hash == B's?   PASS
 phase 4   A writes on; B tails to head ► hash match ► A fenced,
           B leads — and now writes v2 records
 phase 5   fresh cold replay of the MIXED v1+v2 log ► hash == B's? PASS

Setup

cargo new lab-upgrade && cd lab-upgrade

Cargo.toml:

[package]
name = "lab-upgrade"
version = "0.1.0"
edition = "2021"

[dependencies]
# none — the lab is dependency-free so nothing hides the mechanics

The code

src/main.rs, complete. How to read it: skim main() first — it’s at the bottom, and it is just the five phases from the timeline above, a few lines each. Then read the helpers it calls, in file order: the event model and wire codec (encode/decode — decode is the entire upcaster), the Book fold, the log/snapshot I/O, and the seeded workload generator. Two deliberate details — the torn-tail break and the unknown-id no-op — are explained after the run.

// Lab III: live-upgrading an event-sourced limit order book.
// Zero dependencies. One run walks all five phases; every PASS is an assert.

use std::collections::BTreeMap;
use std::fs;
use std::io::{Read, Seek, SeekFrom, Write};
use std::path::Path;

// ---------------- current (v2) in-memory event model ----------------
// The engine ONLY knows this. Old wire versions exist solely in the codec.
#[derive(Clone, Copy, Debug, PartialEq)] // derive = the compiler writes the boilerplate (copy, print, ==) for you
pub enum Side { Buy, Sell }

#[derive(Clone, Debug, PartialEq)]
pub enum Event {
    Place { id: u64, side: Side, price: i64, qty: u64, source: u8 }, // source: NEW in v2
    Cancel { id: u64 },
    Execute { id: u64, qty: u64 },
}

pub const SRC_UNKNOWN: u8 = 0; // upcast default: MUST reproduce v1 behavior

// ---------------- wire codec: [len u32][type u16][ver u16][payload] ----------------
// Picture one record as a strip of bytes; the log is these strips laid nose to tail:
//
//   byte:     0  1  2  3    4  5     6  7     8 ...............
//           [ len: u32    ][ type  ][ ver   ][ payload: len bytes ]
//             "payload is    "which   "which    fields packed at fixed
//              N bytes"       event"   dialect"  positions — no names
//
// Unlike JSON, no field names travel with the data — the positions ARE the
// names. That is why a shipped layout can never be rearranged: v2 may only
// append bytes at the end.
const T_PLACE: u16 = 1;
const T_CANCEL: u16 = 2;
const T_EXECUTE: u16 = 3;

// A u16 is 2 bytes; to_le_bytes lays them out least-significant byte first:
// 0x0102 is stored as [0x02, 0x01]. That fixed order IS the wire format: any
// machine reading the bytes back in that order rebuilds the same number.
fn put_u16(b: &mut Vec<u8>, v: u16) { b.extend_from_slice(&v.to_le_bytes()); }
// Same idea, wider: a u64 becomes its 8 bytes, smallest-first, appended to the buffer.
fn put_u64(b: &mut Vec<u8>, v: u64) { b.extend_from_slice(&v.to_le_bytes()); }
// And the signed flavor: i64 as 8 bytes. Negatives ride along fine — two's
// complement is just the agreed-on bit pattern for "below zero", and both ends agree.
fn put_i64(b: &mut Vec<u8>, v: i64) { b.extend_from_slice(&v.to_le_bytes()); }
// The mirror image: grab the 2 bytes sitting at offset `o`, rebuild the u16.
// (`try_into().unwrap()` turns the slice into the fixed-size array from_le_bytes
// wants; the unwrap can't fail — the slice length always matches.)
fn get_u16(b: &[u8], o: usize) -> u16 { u16::from_le_bytes(b[o..o + 2].try_into().unwrap()) }
// Same read-back for u64: the 8 bytes at `o` become the original number again.
fn get_u64(b: &[u8], o: usize) -> u64 { u64::from_le_bytes(b[o..o + 8].try_into().unwrap()) }
// And for i64.
fn get_i64(b: &[u8], o: usize) -> i64 { i64::from_le_bytes(b[o..o + 8].try_into().unwrap()) }

/// Encode = lay one strip down: each field lands at a known offset,
/// in write order, no names. `schema_ver` is what THIS binary writes: the v1
/// process writes ver=1 (no source byte); the v2 process writes ver=2.
pub fn encode(ev: &Event, schema_ver: u16) -> Vec<u8> {
    let mut p = Vec::with_capacity(32); // payload buffer; with_capacity pre-reserves space — a perf hint, not format
    let ty = match ev {
        Event::Place { id, side, price, qty, source } => {
            put_u64(&mut p, *id);
            p.push(if *side == Side::Buy { 0 } else { 1 }); // side as one byte: 0=buy, 1=sell
            put_i64(&mut p, *price);
            put_u64(&mut p, *qty);
            if schema_ver >= 2 { p.push(*source); } // the entire v2 schema change: one byte, appended last
            T_PLACE
        }
        Event::Cancel { id } => { put_u64(&mut p, *id); T_CANCEL }
        Event::Execute { id, qty } => { put_u64(&mut p, *id); put_u64(&mut p, *qty); T_EXECUTE }
    };
    // Assemble the strip from the diagram above: 8-byte header, then the payload.
    let mut rec = Vec::with_capacity(p.len() + 8);
    rec.extend_from_slice(&(p.len() as u32).to_le_bytes()); // bytes 0-3: payload length
    put_u16(&mut rec, ty);                                  // bytes 4-5: event type
    put_u16(&mut rec, schema_ver);                          // bytes 6-7: wire version
    rec.extend_from_slice(&p);
    rec
}

/// Decode = read the strip back at the known positions — and UPCAST at the
/// boundary: whatever wire version comes in, a current-model Event comes out.
/// This function is the entire v1->v2 upcaster.
pub fn decode(ty: u16, ver: u16, p: &[u8]) -> Event {
    // These offsets mirror encode's write order byte for byte. With no names on
    // the wire, the positions are the schema: shift one offset and every record
    // ever written silently decodes to garbage.
    match ty {
        T_PLACE => Event::Place {
            id: get_u64(p, 0),                                    // payload bytes 0-7
            side: if p[8] == 0 { Side::Buy } else { Side::Sell }, // byte 8
            price: get_i64(p, 9),                                 // bytes 9-16
            qty: get_u64(p, 17),                                  // bytes 17-24
            source: if ver >= 2 { p[25] } else { SRC_UNKNOWN }, // <-- the upcast
        },
        T_CANCEL => Event::Cancel { id: get_u64(p, 0) },
        T_EXECUTE => Event::Execute { id: get_u64(p, 0), qty: get_u64(p, 8) },
        _ => panic!("unknown event type {ty}"),
    }
}

// ---------------- the book: a pure fold over events ----------------
// "Fold" = replaying a bank statement: start at zero, apply every transaction
// in order, arrive at the balance. Same statement, same order -> same balance,
// on any machine. The log is the statement; the Book is the running balance.
#[derive(Default, Clone)]
pub struct Book {
    pub seq: u64,
    // id -> (side, price, remaining qty, source). BTreeMap iterates in sorted
    // key order, so the book reads out identically on any machine. (HashMap's
    // iteration order is arbitrary and differs per process, so hashes would
    // differ; the ch13 contract.)
    pub orders: BTreeMap<u64, (Side, i64, u64, u8)>,
    pub bids: BTreeMap<i64, u64>, // price -> total qty
    pub asks: BTreeMap<i64, u64>,
}

impl Book {
    // Pick the price-level map for a side: bids for buys, asks for sells.
    fn level(&mut self, side: Side) -> &mut BTreeMap<i64, u64> {
        match side { Side::Buy => &mut self.bids, Side::Sell => &mut self.asks }
    }
    // Shrink or remove order `id`: Some(n) = execute up to n, None = cancel it all.
    fn reduce(&mut self, id: u64, by_qty: Option<u64>) {
        // `if let Some(...)` reads "only if the id existed": remove() hands back the
        // order's fields (unpacked right in the pattern), and an absent id skips the
        // whole block — no null check needed, the shape of the code is the check.
        if let Some((side, price, qty, src)) = self.orders.remove(&id) {
            let take = by_qty.unwrap_or(qty).min(qty);
            let lv = self.level(side);
            let left = lv[&price] - take;
            if left == 0 { lv.remove(&price); } else { *lv.get_mut(&price).unwrap() = left; }
            if take < qty { self.orders.insert(id, (side, price, qty - take, src)); } // partial fill: re-insert the remainder
        } // unknown id: deterministic no-op (idempotent replay)
    }
    /// Fold one event into the book: bump the sequence number, then mutate state.
    pub fn apply(&mut self, ev: &Event) {
        self.seq += 1;
        match *ev {
            Event::Place { id, side, price, qty, source } => {
                self.orders.insert(id, (side, price, qty, source));
                *self.level(side).entry(price).or_insert(0) += qty; // entry/or_insert: fetch the level, creating it at 0 if new
            }
            Event::Cancel { id } => self.reduce(id, None),
            Event::Execute { id, qty } => self.reduce(id, Some(qty)),
        }
    }
    /// Canonical state hash: a fingerprint of the entire book. Two processes
    /// showing the same fingerprint hold the same state — that comparison is
    /// every PASS in this lab.
    pub fn hash(&self) -> u64 {
        // FNV-1a: xor a byte in, multiply, repeat. Same recipe over the same bytes
        // gives the same fingerprint on any machine — and that is all it promises.
        // It detects divergence; it is NOT security (nobody here is forging books).
        let (mut h, prime) = (0xcbf29ce484222325u64, 0x100000001b3u64);
        // `mix` is a closure that captures `h`: each call feeds one value's 8 bytes
        // into the running fingerprint. wrapping_mul lets the multiply overflow and
        // wrap around on purpose — the wrap is part of the recipe, not a bug.
        let mut mix = |v: u64| { for b in v.to_le_bytes() { h = (h ^ b as u64).wrapping_mul(prime); } };
        mix(self.seq);
        for (id, (side, price, qty, src)) in &self.orders {
            mix(*id); mix(*side as u64); mix(*price as u64); mix(*qty); mix(*src as u64);
        }
        for m in [&self.bids, &self.asks] {
            for (price, qty) in m { mix(*price as u64); mix(*qty); }
        }
        h
    }
}

// ---------------- log + snapshot I/O ----------------
/// Encode one event and append its bytes at the end of the log file.
pub fn append(log: &mut fs::File, ev: &Event, ver: u16) {
    log.write_all(&encode(ev, ver)).unwrap();
}

/// Read every complete record from `pos` onward, apply each to the book, return
/// the new position. Picture tailing a log file someone may STILL be writing:
/// take whole records only, and if the last one is half-written, stop and
/// remember where you got to. This one loop is cold replay, standby catch-up,
/// AND live tailing — the only difference is whether the writer has finished.
pub fn replay_from(path: &Path, book: &mut Book, mut pos: u64) -> u64 {
    let mut f = fs::File::open(path).unwrap();
    f.seek(SeekFrom::Start(pos)).unwrap(); // jump the file cursor to `pos` bytes from the start
    let mut buf = Vec::new();
    f.read_to_end(&mut buf).unwrap(); // slurp cursor-to-end into memory in one go
    let mut o = 0usize; // read offset within buf
    while buf.len() - o >= 8 { // is at least one full 8-byte header left?
        let len = u32::from_le_bytes(buf[o..o + 4].try_into().unwrap()) as usize; // header bytes 0-3: payload length
        // Torn tail: the header promises more bytes than the file has yet — a
        // half-written record. Not an error: park here; the next call resumes at `pos`.
        if buf.len() - o < 8 + len { break; }
        let (ty, ver) = (get_u16(&buf, o + 4), get_u16(&buf, o + 6)); // header bytes 4-5, 6-7
        book.apply(&decode(ty, ver, &buf[o + 8..o + 8 + len]));
        o += 8 + len;
        pos += (8 + len) as u64;
    }
    pos
}

/// Serialize the book to one file — a JSON.stringify of the whole state, except
/// positional bytes instead of named text: seq, log position, then every open
/// order in id order (BTreeMap order, so the bytes come out identical every time).
pub fn write_snapshot(path: &Path, book: &Book, log_pos: u64) {
    let mut b = Vec::new();
    put_u64(&mut b, book.seq);
    put_u64(&mut b, log_pos);
    put_u64(&mut b, book.orders.len() as u64);
    for (id, (side, price, qty, src)) in &book.orders {
        put_u64(&mut b, *id);
        b.push(if *side == Side::Buy { 0 } else { 1 });
        put_i64(&mut b, *price);
        put_u64(&mut b, *qty);
        b.push(*src);
    }
    fs::write(path, &b).unwrap(); // prod: write tmp, fsync, rename over — a reader sees old or new, never half (Q3)
}

/// Read a snapshot back: rebuild the book, return it plus the log position to resume from.
pub fn load_snapshot(path: &Path) -> (Book, u64) {
    let b = fs::read(path).unwrap();
    let mut book = Book { seq: get_u64(&b, 0), ..Default::default() }; // like { ...emptyBook, seq }: every field not named starts empty
    let (log_pos, n) = (get_u64(&b, 8), get_u64(&b, 16));
    let mut o = 24usize;
    for _ in 0..n {
        // A snapshot entry is laid out byte-for-byte like a v2 Place payload
        // (26 bytes), so the existing codec reads it for free — one layout, two files.
        let ev = decode(T_PLACE, 2, &b[o..o + 26]);
        if let Event::Place { id, side, price, qty, source } = ev {
            book.orders.insert(id, (side, price, qty, source));
            *book.level(side).entry(price).or_insert(0) += qty;
        }
        o += 26;
    }
    (book, log_pos)
}

// ------- deterministic workload -------
// Lcg is a seeded pseudo-random generator: one u64 of state, scrambled on
// each call. Same seed, same sequence — seed it with 42 and it emits the same
// "random" workload on every machine, every run, which is
// why your output below will match this book's byte for byte. (LCG = linear
// congruential generator; no rand crate needed.)
struct Lcg(u64);
impl Lcg {
    // Next pseudo-random u64. wrapping_mul/_add let the math overflow and wrap
    // mod 2^64 on purpose — the wrap IS the scramble; `>> 33` keeps the
    // better-mixed high bits.
    fn next(&mut self) -> u64 { self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407); self.0 >> 33 }
}
// Roll one random event: ~60% places, ~20% cancels, ~20% executes (cancels and
// executes may name ids that no longer exist — fine, reduce() shrugs those off).
fn gen_event(rng: &mut Lcg, next_id: &mut u64, source: u8) -> Event {
    match rng.next() % 10 { // picks 0-9; `0..=5` is a range pattern — matches 0 through 5
        0..=5 => { let id = *next_id; *next_id += 1; Event::Place {
            id, side: if rng.next() % 2 == 0 { Side::Buy } else { Side::Sell },
            price: 10_000 + (rng.next() % 200) as i64 - 100, qty: 1 + rng.next() % 50, source } }
        6..=7 => Event::Cancel { id: rng.next() % (*next_id).max(1) },
        _ => Event::Execute { id: rng.next() % (*next_id).max(1), qty: 1 + rng.next() % 10 },
    }
}

// ---------------- the five phases ----------------
// Runs the five phases in order; every PASS is an assert_eq! that aborts on mismatch.
fn main() {
    let dir = std::env::temp_dir().join("lab-upgrade");
    fs::create_dir_all(&dir).unwrap();
    let log_path = dir.join("book.evlog");
    let snap_path = dir.join("book.snap");
    let _ = fs::remove_file(&log_path); // start clean; `let _` ignores "file didn't exist"
    // Open in append mode, creating if missing — fs.createWriteStream(path, {flags: "a"}):
    // every write lands at the current end of the file, after whatever is already there.
    let mut log = fs::OpenOptions::new().create(true).append(true).open(&log_path).unwrap();

    let mut rng = Lcg(42);
    let mut next_id = 1u64;

    // Phase 1: process A (v1 binary) writes 1000 v1 records, folding as it goes.
    let mut a = Book::default();
    for _ in 0..1000 {
        let ev = gen_event(&mut rng, &mut next_id, SRC_UNKNOWN); // v1 has no source concept
        append(&mut log, &ev, 1);
        a.apply(&ev);
    }
    // flush() on a raw File is actually a no-op: there is no userspace buffer to
    // drain (that's a BufWriter thing) — every write() above already went to the
    // OS, so readers already see the bytes. Surviving power loss is a separate
    // promise: that needs sync_all() (fsync), deliberately skipped here — ch13
    // covers when the log must be synced.
    log.flush().unwrap();
    println!("phase 1: v1 writer   seq={:>4}  hash={:016x}", a.seq, a.hash());

    // Phase 2: v2 binary cold-replays the v1 log through the upcaster.
    let mut b = Book::default();
    let mut b_pos = replay_from(&log_path, &mut b, 0);
    println!("phase 2: v2 replay   seq={:>4}  hash={:016x}", b.seq, b.hash());
    assert_eq!(a.hash(), b.hash());
    println!("phase 2: DETERMINISM CHECK PASS (v2 binary == v1 state on v1 log)");

    // Phase 3: snapshot B, reload, verify roundtrip.
    write_snapshot(&snap_path, &b, b_pos);
    let (c, c_pos) = load_snapshot(&snap_path);
    assert_eq!(b.hash(), c.hash());
    assert_eq!(b_pos, c_pos);
    println!("phase 3: SNAPSHOT ROUNDTRIP PASS  (snap at seq={} pos={})", c.seq, c_pos);

    // Phase 4: hot cutover. A keeps writing v1; B tails the log live; at the
    // handover boundary the hashes must match, then B takes over writing v2.
    for i in 0..500 {
        let ev = gen_event(&mut rng, &mut next_id, SRC_UNKNOWN);
        append(&mut log, &ev, 1);
        a.apply(&ev);
        if i % 100 == 99 { log.flush().unwrap(); b_pos = replay_from(&log_path, &mut b, b_pos); }
    }
    log.flush().unwrap();
    let _ = replay_from(&log_path, &mut b, b_pos); // B reaches head
    assert_eq!(a.hash(), b.hash());
    println!("phase 4: HANDOVER PASS at seq={}  hash={:016x}  (A fenced, B leads)", b.seq, b.hash());
    for _ in 0..250 { // B is primary now, writing v2 records with a real source tag
        let ev = gen_event(&mut rng, &mut next_id, 2);
        append(&mut log, &ev, 2);
        b.apply(&ev);
    }
    log.flush().unwrap();
    println!("phase 4: v2 leader   seq={:>4}  hash={:016x}", b.seq, b.hash());

    // Phase 5: fresh cold replay of the MIXED v1+v2 log must equal B exactly.
    let mut d = Book::default();
    replay_from(&log_path, &mut d, 0);
    assert_eq!(b.hash(), d.hash());
    println!("phase 5: MIXED-LOG REPLAY PASS  seq={}  hash={:016x}", d.seq, d.hash());
    println!("all phases PASS  (log: {})", log_path.display());
}

Run it

cargo run --release

Expected output (byte-for-byte reproducible — LCG seed 42, fixed-seed FNV hash):

phase 1: v1 writer   seq=1000  hash=3067adf006947f53
phase 2: v2 replay   seq=1000  hash=3067adf006947f53
phase 2: DETERMINISM CHECK PASS (v2 binary == v1 state on v1 log)
phase 3: SNAPSHOT ROUNDTRIP PASS  (snap at seq=1000 pos=27389)
phase 4: HANDOVER PASS at seq=1500  hash=5d27565fba06ac86  (A fenced, B leads)
phase 4: v2 leader   seq=1750  hash=daf16d78feb8e7ad
phase 5: MIXED-LOG REPLAY PASS  seq=1750  hash=daf16d78feb8e7ad
all phases PASS  (log: /tmp/lab-upgrade/book.evlog)

Your hashes will match these because every source of nondeterminism was designed out — seeded LCG instead of thread_rng, BTreeMap instead of HashMap, fixed-seed FNV instead of DefaultHasher, integers instead of floats, time absent entirely. That exact reproducibility is the lesson the exercise exists to teach.

What each phase proves, and where it maps to production

Phase 1 — one event, two destinations. Log first; state is derived, never authoritative.

 ENGINE A (v1 writer)         │ DISK
                              │ ◄─ disk (fs write) boundary
   gen_event ──► Event        │
       │           │          │
       │           └─(1) encode(ev,1) ─► [len|ty|ver=1|payload] strip
       │ (3) a.apply(&ev)     │               │ (2) append
       ▼                      │               ▼
   A's in-memory Book         │   book.evlog  [v1][v1][v1]…
       │ (4) a.hash()         │
       ▼                      │
   3067adf006947f53           │

Phase 2 — the determinism check.

 DISK                  │ ENGINE B (fresh v2 instance)
 ◄─ disk (fs read)     │ ◄─ upcast boundary: v1 bytes cross
    boundary           │    into the v2 model inside decode
 book.evlog            │
  [v1][v1][v1]… ──(1) replay_from(path, &mut b, 0)
      each strip ──────►(2) decode(ty, ver=1, payload)
                       │      │  ver < 2 → source = SRC_UNKNOWN
                       │      ▼
                       │  Event { …, source: 0 }
                       │      │ (3) b.apply(&ev)
                       │      ▼
                       │  B's Book ──(4) b.hash()
 (5) assert_eq!(a.hash(), b.hash())  →  DETERMINISM CHECK PASS

The PASS holds because the upcaster (decode, one if ver >= 2 line) defaults the new field to a value that reproduces v1 behavior, and the fold never branches on anything outside the event stream. Production equivalent: the pre-deploy replay gate of ch13/ch17. Note where the versions live: the engine model has no V1 type — v1 exists for one line of the codec, at the marked boundary.

Phase 3 — snapshot = state + cursor.

 ENGINE B                 │ DISK
                          │ ◄─ disk (fs write) boundary
 B's Book ──(1) write_snapshot ─►  book.snap
   + b_pos (cursor)       │   [seq][log_pos][orders…]
                          │       │ cursor: pairs the state
                          │       │ with a byte offset
                          │       ▼
                          │   book.evlog [v1]…[v1]▌◄ pos 27389 (head)
 fresh Book c, c_pos ◄──(2) load_snapshot ── book.snap
       │
      (3) assert c.hash()==b.hash() && c_pos==b_pos → ROUNDTRIP PASS
 recovery = load_snapshot, then replay_from(c_pos) for the rest

The comment about tmp+fsync+rename in write_snapshot is the production delta (ch13).

Phase 4 — hot cutover.

 ENGINE A (v1, leader) │ DISK: book.evlog │ ENGINE B (v2, follower)
 ─── process boundary (in this lab: two instances, one binary) ───
 (1) append(ev,1) ───► [v1]              │           time ↓
     a.apply(&ev)      [v1]              │
        │              [v1] ◄──(2) replay_from(b_pos): tail;
        │              [v1]     cursor advances strip by strip;
        ▼              [v1]     torn tail at head → break, park
 … 500 v1 records … ─► [v1]              │
                       [v1] ◄──(3) replay_from: B reaches head
                       │                 │
 (4) hash gate: assert_eq!(a.hash(), b.hash()) → HANDOVER PASS
 (5) A FENCED ✕        │                 │
     lane ends — A     │                 │  B leads now:
     never writes      [v2] ◄──(6) append(ev,2), b.apply(&ev)
     again             [v2]              │
                       [v2] …            │
 the WRITE ROLE crossed the process boundary: same file, new writer,
 new wire version — B could read v2 (phase 2) before anyone wrote it

The handover assert is the go/no-go gate from the runbook of ch16: leadership flips only when the follower’s hash matches the leader’s at the boundary sequence. Then B writes v2 into the same log — the readers-first choreography of ch14 compressed into one file.

Phase 5 — mixed-log replay.

 DISK: book.evlog (two dialects)   │ ENGINE D (fresh Book)
                                   │
 [v1][v1]… 1500 v1 strips …[v1]║[v2]… 250 v2 strips …[v2]
             version boundary ─┘ (ver flips 1→2 mid-file)
      │                            │
     (1) replay_from(path, &mut d, 0) — one pass, no mode switch
      └──── every strip ──────────►(2) decode:
                                   │  ver 1 → upcast source=0
                                   │  ver 2 → read source byte
                                   │      │ (3) d.apply(&ev)
                                   │      ▼
                                   │  D's Book ──(4) d.hash()
 (5) assert_eq!(b.hash(), d.hash()) → MIXED-LOG REPLAY PASS

This is the property that makes rollback horizons and long retention livable: any conforming binary, at any later date, can rebuild any historical state from the heterogeneous log.

Two deliberate details worth noticing: replay_from’s torn-tail break (a half-written record at the file end is waited on, not an error — that’s what makes the same loop serve live tailing), and reduce’s unknown-id no-op (cancels/executes for absent orders are deterministic no-ops, which keeps the generated workload — and real-world duplicate-message replay — idempotent).

Extensions if you have another hour

Make it actually two processes: split into a lib plus two bins (writer, follower) sharing the file; the follower polls replay_from in a loop and prints its hash; kill the writer and promote for real. The code already supports it.
Break determinism on purpose: swap orders to a HashMap and iterate it in hash(). Watch phase 2 pass in-process but fail across two separate processes (per-process SipHash keys) — the exact reason ch13 insists dual-replay tests run in separate processes.
Add a v3 that widens qty semantics or adds a second field via the reserved-byte trick from ch14, chaining upcasters v1→v2→v3.
Add a crc32 to the record header and corrupt a byte mid-file with dd; make replay_from stop at the corruption and report the last good (seq, pos).

Plain-English recap

The lab is a mini ledger plus a read model, with a versioned serialization format. Events are journal entries; the Book is the derived balance; the wire codec is your API’s payload schema.
Phase 2 is “new service, old archive, identical balances” — the replay-regression gate from ch17, in miniature.
Phase 3’s (snapshot, log_pos) pair is Kafka consumer-offset checkpointing — materialized state plus its cursor, the same reason every stream consumer persists its offset.
Phase 4 is promoting a replica after lag reaches zero and checksums match. Only after promotion does the new format flow — the readers-first choreography of ch14 in one file (Q6 shows where the lab deliberately cheats).
Phase 5 is a webhook archive with old and new payload shapes both parseable forever — the property that makes long retention, audit replay, and rollback horizons livable.
The two “deliberate details” are your at-least-once instincts. The torn-tail break treats a half-written record as “wait for more,” not an error; the unknown-id no-op makes duplicate or stale cancels/executes idempotent — the idempotency-key reflex, applied to a book.

Interview narration

How you tell this in an interview, sixty seconds, first person: “I keep the engine as a pure fold over a length-and-version-prefixed record log. When I ship a schema change I never touch the old decoder — the new field is appended on the wire, and a one-line upcaster defaults it so old records reproduce old behavior exactly. My deploy gate is mechanical: the candidate binary replays the production log and must hash-match the running engine’s state at the same sequence number — same fixed-seed structural hash both sides. For the cutover itself, the new version runs as a follower tailing the live log; when it’s at head and hashes match at a boundary sequence, leadership flips, the old process is fenced, and the new one continues writing the new version into the same log. And because replay tolerates mixed-version logs, rollback and audit replay keep working across the upgrade — I’ve got a 250-line toy that demonstrates the whole cycle end-to-end, and the same shape scaled to my production engine with its hot standby.” Every clause of that paragraph is a line of code you just ran.

Interviewer will ask

Q1: “Why default the new field to SRC_UNKNOWN = 0 instead of something meaningful?” Because the upcast default must make v2 semantics degenerate to v1 semantics — replaying old logs must reproduce the state that actually existed, and any “smart” default (inferring source from order id ranges, say) is a silent history rewrite. Zero-as-legacy also composes with fixed-layout formats where old writers emit zeroed padding (the reserved-byte trick of ch14).

Q2: “Your handover compares hashes once at the boundary. Is that enough?” For the toy, yes; in production you compare rolling hashes continuously during the whole shadow period (every N sequence numbers), because a single end-point match can mask transient divergence that happened to cancel out, and because you want divergence to page you hours before a cutover, not fail the gate at T-0. You’d also gate on the follower being at head within a lag bound, not just matched at some past boundary.

Q3: “What’s missing from this snapshot for a real engine?” Atomic write, unpacked: write to a tmp file, fsync it, then rename over the real name — rename is atomic on POSIX, so a reader sees old or new, never half. Then fsync the directory too, because the rename lives in the directory’s own data — skip it and a crash can forget the file was ever renamed (ch13). Plus a checksum, a format version field, and K retained generations so a corrupt latest falls back to the previous one. And content beyond open orders: session-level state like venue sequence numbers, active timers, in-force config — anything the fold reads. Also note this snapshot cheats pleasantly: it rebuilds price levels from orders on load, trading load time for snapshot simplicity — a legitimate trade you should be able to defend either way.

Q4: “What happens if the process dies mid-append?” The log ends in a torn record; replay_from refuses to apply it (length check) and reports the last durable position — recovery resumes from complete records. In production you’d add a per-record checksum so a torn or bit-rotted record is distinguishable from a clean tail, and the writer would fsync on a policy (per event, per batch, or rely on the replicated standby as the durability story — a latency/durability trade you should name explicitly).

Q5: “Why is the hash function hand-rolled FNV instead of DefaultHasher?” DefaultHasher is SipHash with a per-process random key — hashes are incomparable across processes, and comparing across processes is exactly what this hash is for (primary vs standby, old binary vs new). Any fixed-seed hash works; the requirements are canonical field order (hence BTreeMap traversal), fixed serialization, and covering all state the fold can produce while excluding incidentals like capacity.

Q6: “The v2 leader starts writing v2 records immediately after taking over. Is that safe?” In the lab, yes, because there’s no rollback consumer; in production it violates the rollback horizon — the fenced v1 binary can’t parse v2 records, so if you rolled back you’d strand the tail. The discipline of ch16: new leader keeps writing v1-compatible records through the bake period and flips write-format later by config. Being able to point at the exact line of the lab that’s “wrong on purpose” is a strong interview move.

The HFT Gap Closer