Publishing Market Data: Building the Feed

Before you start. This chapter assumes the sequencer + deterministic-consumer layout (the feed is derived from the same log as the fills, ch23), UDP multicast vs TCP fanout (one-packet-many-receivers vs one-connection-per-client, ch02), feed-consumer mechanics — gap detection, snapshots, book building — from the client side (ch00f, ch00d), event sourcing / snapshot + replay (ch13), and queueing under bursts (ch08). If any are new, read those first.

You spent years on the receiving end of market-data feeds — 20+ venues’ worth of gap detection, snapshot recovery, and book building in your ingestion pipeline. Every quirk you handled defensively (missed sequence numbers, snapshots inconsistent with increments, venues that silently dropped you under load) was a producer-side design decision, made well or badly. This chapter is where you make those decisions. The matching engine decides what happened, but the publisher has to tell ten thousand people at once — and at crypto scale, the fanout tier dwarfs the matching engine.

The publisher is just another consumer

                    ┌──────────────────────────► [ matching engine ] ──► fills
   [ SEQUENCER ] ───┤  same sequenced stream
   (one ordered     └──────────────────────────► [ FEED PUBLISHER ]
    event log)                                        │
                                                      │ maintains its own book replica
                                                      │ (deterministic → bit-identical
                                                      │  to the engine's book, by determinism)
                                                      ▼
                                       ┌──────────────┬──────────────┐
                                       │  L1 channel  │  L2 channel  │  L3 channel
                                       │ (top of book)│   (depth)    │ (every order)
                                       └──────────────┴──────────────┘
                                                      │
                                              fanout tier (this
                                              chapter's real topic)

Start with where the feed comes from: the feed is not a report from the matching engine — it’s an independent derivation from the same sequenced log. The publisher consumes the sequenced stream, maintains its own book replica, and emits deltas. Because everything downstream of the sequencer is deterministic (ch23), the publisher’s book is bit-identical to the engine’s without ever talking to it. No RPC from the engine’s hot path, no “publish” call that can slow matching, no possibility of feed-vs-fills divergence that isn’t a determinism bug. Your event-sourced engine had exactly this property — you could hang any number of read-model projections off the event log without touching the writer. A market-data feed is a read-model projection with ten thousand subscribers and a latency SLA.

Product tiers: L1, L2, L3

Tier	What it is	Your-world analogy	Who buys it
L1 / top of book	Best bid, best ask, last trade — the one-line summary	The webhook that says “payment settled” without the line items	Retail platforms, charting, anyone who needs a price, not the book
L2 / depth	Aggregated quantity per price level, top N levels or full depth — the book as a histogram	An API response with the pagination depth you choose	Most algorithmic traders; your old SOR (queue-depth-aware routing needs it)
L3 / order-by-order (ITCH-style full feed)	Every individual order’s add, cancel, replace, execute, with order IDs — the raw event stream itself, minimally disguised	Not the API response but the change-data-capture stream off the database	HFT firms who rebuild the book themselves and mine it for microstructure signal — signals hiding in the order flow itself, not the price. Queue-position estimation needs this tier; an L2 consumer literally cannot compute what an L3 consumer can

Why the venue tiers them, beyond bandwidth: market data is a product line, often rivaling trading fees as a revenue source. Tiering is price discrimination by information content — same instinct as your API product tiers (webhook granularity levels, basic vs firehose), except here the premium tier’s customers can measure its value in basis points (hundredths of a percent, the unit execution quality is scored in). Note the derivation direction: L3 → L2 → L1 are each computable from the previous, so the publisher builds once from the sequenced stream and projects three ways. One book replica, three serializers.

The incremental + snapshot architecture

This is the contract you coded against for years. Now specify it from the producer side:

   CHANNEL A: incremental (every change, seq-numbered)
   ──► seq 1001: bid 64999.5 qty 3.2 (level update)
   ──► seq 1002: ask 65000.0 qty 0   (level delete)
   ──► seq 1003: trade 65000.5 qty 0.4
        ... continuous, low-latency, the "real" feed ...

   CHANNEL B: snapshot (periodic full book state)
   ──► snapshot { as_of_seq: 1000, bids: [...], asks: [...] }   every N sec
   ──► snapshot { as_of_seq: 1450, ... }

   consumer recovery = your old client-side dance:
   buffer increments ► fetch snapshot(as_of_seq=S) ► drop increments ≤ S
   ► apply buffered > S ► live

The producer-side book-building contract — the invariants your consumers’ correctness rests on. You know what breaking each one does to a client, because you handled the breakage; now you’re the one who must never break them:

Gap-free, monotonic sequence numbers within a channel. The consumer’s only loss-detection mechanism is “I saw N, next must be N+1.” Implication for you: per-channel sequence assignment is sacred, and any internal publisher failover must resume without gaps or dups — which the sequenced log makes possible (replay from last published seq) and ad-hoc designs get wrong.
Every snapshot carries an exact consistency point (as_of_seq), and applying increments from that point onward yields the true book. The classic producer bug — snapshot generated from a book mid-mutation, or stamped with a fuzzy sequence — gives consumers permanently corrupt books that look plausible. You debugged venues with exactly this bug; it costs your consumers days of “our book drifts from reality” forensics.
Deterministic emission: same log → same feed messages. This makes the feed itself replayable for your own testing (ch13) and lets you regression-test the publisher the way you regression-tested your engine.
Documented conflation semantics (below) — if the feed may skip states, consumers must know which states can vanish and what is never skipped (trades, in any sane design).

The snapshot channel is cheap insurance: a snapshot every 1–60s costs little and bounds every consumer’s recovery time. As a client you cursed venues with 60s snapshot intervals during volatile gaps; as the producer, snapshot frequency is a knob trading your bandwidth against their worst-case recovery — put it in the spec and make it burst-aware.

Tradfi fanout: multicast, and why it’s a different universe

   TRADFI (UDP multicast)                      one send, N deliveries
                          ┌─► subscriber 1
   publisher ──1 packet──►│switch│──► subscriber 2     the SWITCH replicates
   (one send() total)     └─► subscriber N             in hardware, ~300ns

   A/B feeds: two independent multicast groups, disjoint network paths,
   same payload — consumers arbitrate (first-arrival wins per seq),
   loss on one path masked by the other  [you knew these as a consumer;
   see the TCP/UDP chapter]

Multicast fanout: the publisher sends each packet once, to a group address, and the switches replicate it to every subscribed port in hardware. It’s a CDN edge duplicating your origin’s single stream to every viewer — except the “CDN” is L2 switching silicon and adds nanoseconds.

The consequence: publisher cost is O(1) in subscriber count.

ITCH-class feeds peak at millions of messages/sec (NASDAQ TotalView peaks in the tens of millions market-wide). The price: it’s UDP, so no delivery guarantee — which is why the A/B dual-feed pattern and the retransmission/snapshot infrastructure exist, and why the seq-number contract above is the load-bearing wall.

The recovery back-office: retransmission and replay services

Multicast’s fire-and-forget speed pushes reliability to dedicated side services — infrastructure you interacted with as a consumer without necessarily naming it:

   ┌───────────────┐   lost pkts 1001-1005?   ┌──────────────────────┐
   │   consumer    │─────request (TCP/UDP)───►│ RETRANSMISSION server │  small gaps:
   │ (gap detected)│◄────those packets────────│ (recent-history cache)│  re-request
   └───────────────┘                          └──────────────────────┘
          │            too far behind / too big a gap?
          └──────────────────────────────────────────► SNAPSHOT/REPLAY service
                                                       (start over from a
                                                        consistency point)

NASDAQ’s MoldUDP64 layer is the canonical example: sequenced UDP packets on the multicast group, plus a re-request server that serves recent history to consumers who name a sequence range. Design decisions you now own as the producer:

The retransmission window is deliberately small (seconds of history, bounded memory). It exists for microbursts and single-packet drops, not for consumers that went to lunch. Past the window, the answer is the snapshot channel — this two-tier split (tiny fast re-request, big slow resync) bounds the cost of your recovery infrastructure no matter how broken the consumer.
Re-request capacity is itself rate-limited per consumer, or one badly-written feed handler in a loop becomes a DoS on the recovery path during exactly the burst that caused everyone’s gaps. (You saw venues throttle recovery endpoints; now you know why.)
A/B arbitration reduces re-requests to nearly zero in practice — two independent paths rarely drop the same packet — which is why the dual-feed pattern is cheaper than it looks: you pay 2× bandwidth to almost never touch the recovery path.

The crypto equivalent is coarser: the REST depth-snapshot endpoint plus WS resubscribe is the recovery service, and its rate limits during volatile periods — which you cursed — are the same “protect the recovery path from stampedes” logic, minus the fast small-gap tier that would have made your life easier.

Crypto reality: per-client TCP fanout, and the arithmetic of pain

No multicast over the public internet. Every consumer is a WebSocket — a private TCP connection with its own send queue, its own congestion state, its own slowness:

                       ┌─► [queue][TLS][TCP] ─► client 1     (fast, fine)
   publisher ──►(copy)─┼─► [queue][TLS][TCP] ─► client 2     (fast, fine)
   every message,      ┼─► [queue][TLS][TCP] ─► client 3     (SLOW ◄── problem)
   every client        └─► ... × 10,000

The arithmetic: 10,000 connected clients × 1,000 msgs/sec of book updates = 10,000,000 sends/sec — each a userspace copy, TLS encryption, and TCP transmission. Meanwhile the matching engine that generated those 1,000 msgs/sec is loafing on one core. The fanout tier dwarfs the matching engine — at a crypto venue, market-data distribution is commonly the largest compute fleet in the building, an inversion that surprises people who assume matching is the expensive part. Human scale: the engine’s day is a single busy Postgres writer; the fanout tier is a CDN origin under permanent load. This is why crypto venues shard fanout fleets by symbol and subscription tier, and why their engineering blogs are full of “how we rewrote our WS distribution layer” posts.

The slow consumer problem

This is the chapter’s central engineering problem. On TCP, a slow reader backpressures its connection: their receive window fills, your socket buffer fills, your per-client queue grows. The one thing you must never do is let one slow client backpressure the publisher — the feed is shared fate, and the market does not slow down because someone’s book-builder is GC-pausing. Your options, worst to best:

Unbounded per-client queues: memory grows until the fanout host OOMs. One bad client kills service for everyone on that box. Never.
Disconnect policy: bounded queue; on overflow, cut the client. Simple, predictable, and standard as the backstop — but as the only tool it’s harsh: on a volatile spike, everyone’s queues spike, and you’d mass-disconnect exactly when clients most need data (and their reconnect-plus-snapshot stampede hits you at the worst time).
Conflation — with disconnect as the backstop.

Conflation: when a client falls behind, don’t queue every tick — keep only the latest state per price level and send that when the connection drains. It’s the webhook consumer that can’t keep up: you don’t slow the producer or buffer a million events, you skip to current state and let them re-sync.

You’ve built this before, pointed the other way. Your pipeline did fan-in conflation — 20 venues feeding your strategies, keep-latest per book level when your consumers lagged. This is the same data structure as fan-out: one keep-latest map per slow client:

   per-client conflation map (bounded by book size, NOT by message rate):

   updates while     {bid 64999.5 → qty 3.2}      later updates to the same
   client is slow:   {bid 64999.5 → qty 1.1}  ──► level OVERWRITE in place:
                     {ask 65000.0 → qty 0  }      map holds ONE entry per level
   client drains ──► send current map contents (+ seq jump marker)

What makes conflation cheap: the conflation map’s size is bounded by book width, not message rate — a client can be behind by a million messages and owe you only a few thousand level-states. The costs, which you must document as feed semantics: the client loses tick-by-tick history (intermediate book states vanish — fine, because the current state supersedes them), and the client must be told (a seq discontinuity or explicit conflation flag) so they know their view skipped states.

Trades are facts and are never conflated. A missed book state is harmless — the latest state replaces it. A missed trade is a lost fact: trades must be queued faithfully or recovered via the snapshot/recovery channels, never collapsed away.

Venues run this as product tiers: full-rate feed for those who keep up, conflated feeds (e.g., 100ms-interval book states) as an explicitly throttled cheaper product — your API-tier instinct again, and tradfi vendors sell exactly this split.

Timestamps in the feed: the producer side of “when”

Your feed handlers compared venue timestamps to local receive time for years (ch07, ch08). Now you’re the one stamping, and each message wants several times, because consumers use them for different jobs:

   one feed message, three producer times:

   ┌───────────────────────────────────────────────────────────────┐
   │ event_time    : when the sequencer sequenced the cause        │ ◄ market truth
   │ (a.k.a. transact/match time — same for every consumer,        │   (use for
   │  replay-stable, tied to the seq number)                       │    research/backtests)
   │ send_time     : when THIS publisher put it on the wire        │ ◄ measures the
   │                                                               │    venue's own lag
   │ [consumer adds] recv_time : their NIC timestamp               │ ◄ measures the path
   └───────────────────────────────────────────────────────────────┘

   send_time − event_time  = publisher lag  (the venue's problem — publish it honestly)
   recv_time − send_time   = network path   (the consumer's problem)

The design rules:

event_time must come from the sequenced log, never from the publisher’s wall clock. It’s part of the deterministic output (same log → same event_times on replay), it’s what backtests and surveillance key on, and it’s identical across L1/L2/L3 so a consumer can join tiers.
send_time is diagnostic, not truth. It differs across A/B feeds and across publisher restarts; its whole value is letting sophisticated consumers decompose “the data was late” into “the venue was slow” versus “my path was slow.” You did exactly that decomposition from the outside, usually with worse tools; publishing an honest send_time is the producer-side courtesy that makes it tractable.
Never flatter send_time. Venues get caught stamping it early to improve their published latency, because colo consumers with hardware timestamps (ch07) can measure the lie.
One clock for everything. Gateways, sequencer, and publishers PTP-synced to the same grandmaster (the one reference clock every machine in the building disciplines to), or your own timestamps can’t be compared across components — the venue-internal version of the multi-venue clock problem your pipeline fought.

Fairness, again: everyone “at the same time”

Tradfi multicast makes simultaneity credible: one packet, hardware replication with nanosecond skew, measured cable lengths from switch to every colo cage (ch24) — the venue can defend “all subscribers were sent the data at the same instant” as physical fact.

Per-client TCP fanout cannot be perfectly fair, and you should be able to say why precisely: sends are serialized (a loop over sockets — client #1 in iteration order beats client #10,000 by whole microseconds every single time), each connection’s TLS/TCP state differs, and kernel scheduling adds jitter.

Mitigations, not cures:

Randomize send order per tick — no client is systematically first; a structural advantage becomes zero-mean noise, which is what fairness means in practice.
Shard clients evenly across fanout hosts — no host’s send loop gets disproportionately long.
Keep per-tick fanout loops tight — so the first-to-last spread stays small.

As a client you suspected some venues’ WS feeds had favorites; as the producer, randomized send order is how you make that accusation false — and provably so, because you can show the shuffle in code.

The reconnect stampede

The failure mode that couples everything in this chapter together: something blips — a fanout host dies, a network path flaps, or you mass-disconnect slow consumers during a volatility spike — and now thousands of clients simultaneously run the recovery dance: reconnect, re-auth, request snapshot, replay increments.

   t=0   fanout host dies (2,000 clients)
   t+1s  2,000 reconnects hit surviving hosts        ◄─ TLS handshakes: CPU spike
   t+2s  2,000 snapshot requests                     ◄─ snapshot service: 100×
         ... during the same volatile burst              normal load, worst moment
         that caused the disconnects ...

This is a thundering herd with a cruel correlation: recovery load peaks exactly when live load peaks, because volatility causes both the disconnects and the message-rate spike. Standard mitigations, all of which you’d recognize from web-scale work but must re-derive under microsecond-adjacent constraints:

Jittered reconnect backoff, enforced server-side — clients won’t do it voluntarily; you didn’t, when reconnect speed was money.
Pre-generated snapshots served from memory, never computed per-request — the snapshot at as_of_seq=S is identical for every requester, so build once per interval and serve many: your CDN-cache instinct exactly.
Connection-accept rate limiting — surviving hosts degrade gracefully instead of collapsing under the TLS-handshake spike.
Capacity math for N−1 hosts during a burst — steady-state sizing is the wrong question, because the stampede arrives mid-burst by construction.

A venue that mass-disconnects on a spike and then can’t absorb the re-entry has converted a slow-consumer policy into a full outage — this coupling is why the disconnect threshold and the recovery capacity have to be designed as one system, not two settings owned by two teams.

Testing the publisher

The determinism dividend again: because the feed is a pure function of the sequenced log, the publisher is about as testable as a stateful component gets — provided you build the harness:

Golden-feed regression: replay a captured production log through the candidate publisher; diff emitted bytes against the previous version’s output. Any unexplained diff is a bug or an intentional (and therefore documented) format change. This is exactly your engine’s replay-based regression testing pointed at a different output.
Invariant checking in CI and prod: run a reference consumer that does what your clients do — build books from snapshot + increments — and continuously assert it matches a directly-derived book replica. This catches the consistency-point bugs (invariant 2) that plague real venues, before clients do the catching.
Adversarial consumer simulation: a load harness of deliberately slow, gappy, reconnect-happy fake clients hammering the fanout tier — because the slow-consumer and stampede machinery above is exactly the code that never gets exercised until the worst day of the year, and “tested only in production during incidents” is how feed reputations die.

The other end of the wire: building the book as a consumer

Everything above is you publishing. Flip to the seat you’ve actually sat in — broker, trader, anyone consuming this feed — and two questions every market-data interview asks: how do you build the book in the first place, and how do you get the book as of some time N?

Building the live book

The algorithm depends on which product tier (above) you bought.

Level-based feed (L2/MBP — most crypto websockets). Events are absolute level updates: “bid 10001 now 400.” Your book is two sorted maps px → qty per side; apply each update, delete the level when qty is 0. A trivial fold — if you start from a correct base. Establishing that base is the interview question: the bootstrap ordering.

Subscribe to the diff stream first. Apply nothing — you have no book yet. Buffer the deltas.
Then fetch the snapshot. It carries a sequence number, say S.
Discard buffered deltas with seq ≤ S — they’re already inside the snapshot. Verify the first surviving delta is S+1; a hole here means your book would be silently wrong until the next reconnect, so re-bootstrap instead. Apply the buffered tail, then go live.
Any sequence gap later → the book is stale: mark it, stop trading on it, re-bootstrap. (The router rule from the broker chapter: a book you can’t trust is a book you don’t act on.)

Subscribe-then-snapshot is the load-bearing choice. Reverse it and every update that arrived between the snapshot fetch and the stream connect is lost forever — you carry a corrupt level for hours with no error anywhere. Some venues also publish book checksums with each update; apply, compare, and a mismatch is your corruption detector firing — re-bootstrap.

Order-by-order feed (L3/MBO — ITCH-style). Events are Add(order_id, side, px, qty), Execute(order_id, qty), Cancel(order_id), Replace(…). Your state is a hashmap order_id → (side, px, qty) plus per-price aggregates maintained as orders arrive and leave; the L2 ladder is now derived. Start from the venue’s start-of-day reset or its snapshot channel, apply in venue-sequence order. The reward for the extra work is queue position: you know exactly which orders rest ahead of yours at the touch — which is why L3 costs more (product tiers, above).

A production feed handler runs four checks continuously: sequence continuity; checksum where offered; the book never crossed against itself; and a staleness heartbeat — no update and no heartbeat for X ms means the feed is lying by silence.

The book as of time N

The move: the book at time N is not stored anywhere — it is derived. Nearest snapshot at-or-before N, plus a replay of deltas up to N. The same snapshot-plus-tail move as the venue’s audit tool (ch23) — except as a consumer you must manufacture the raw material yourself:

Capture the feed as it arrives. Gold standard: a passive tap with hardware timestamps (ch05). Practical minimum: journal every normalized event with its venue sequence number and your receive timestamp. Add periodic snapshots of your built book (hourly, or every M events) so replay cost stays bounded.
Storage shape: snapshot files plus compressed delta segments, keyed (symbol, day), time-indexed. Recognize it — it’s a WAL plus checkpoints, the same design making its fourth appearance in this book.
Reconstruction: binary-search snapshots for the last one ≤ N, load it, replay the segment’s deltas while ts ≤ N, stop. Cost is bounded by snapshot cadence — which is why cadence is a decision, not an afterthought.
Which clock is “time N”? Venue event-time and your receive-time differ by transit plus your own lag (ch07’s whole lesson). For TCA and best-execution the correct clock is your receive time — the question is “what could we have known when we routed,” not “what had objectively happened.” For market research, venue time. An answer that doesn’t state its clock isn’t an answer.
Why your own capture beats vendor data: a best-ex dispute asks “what did we see” — a vendor archive is someone else’s clock and someone else’s gaps. Vendor data is fine for research; evidence needs your wire.

Who uses it: TCA’s arrival-mid (the book at parent arrival), markout computation, backtests, and the 2am dispute where someone claims your fill was off-market — you load the snapshot, replay to 14:32:07 on your clock, and read the answer off the screen.

Numbers to hold

Quantity	Value	Human scale
ITCH-class full-feed peak	millions–tens of millions msgs/sec market-wide	a message per ~100ns at peak — hardware-timestamp territory (ch07)
Multicast publisher cost	O(1) in subscribers	1,000th subscriber is free
Crypto WS fanout	10k clients × 1k msg/s = 10M sends/sec	fanout fleet ≫ matching engine
Per-send cost (copy+TLS+TCP)	~1–5 µs of CPU	10M sends/sec ≈ tens of cores just moving bytes
Snapshot interval	1–60 s typical	bounds every consumer’s worst-case recovery
Conflation map bound	book width (≈10²–10⁴ levels)	behind by 1M msgs, owe only the current book
Retransmission window	seconds of history, in memory	microburst repair only — beyond it, snapshot resync
Multicast replication skew	nanoseconds (switch hardware)	vs whole microseconds first-to-last in a TCP send loop

Plain-English recap

Feed messages carry two producer times with different jobs: event_time from the sequenced log (replay-stable market truth for backtests and surveillance) and send_time from the wire (diagnostic — it splits “data was late” into the venue’s lag vs the consumer’s path).
The feed isn’t the engine reporting out — it’s a second, independent projection of the same event log, like a read model hanging off your event store. Determinism is what lets the fills and the feed never disagree.
One book replica, three serializers: L3 → L2 → L1 are successive projections, so the publisher builds state once from the log and prices the projections as products.
L1/L2/L3 are the same data at three information densities — summary webhook, paginated API, raw CDC stream — and they’re a revenue line, tiered like any API product.
Incremental + snapshot is a contract, and you already know every clause from the consumer side: gap-free monotonic seq per channel, snapshots with exact as_of_seq consistency points, documented conflation semantics. Producer bugs here cost every consumer days of book-drift forensics.
Tradfi multicast = the switch is your CDN: one send, hardware replication, O(1) in subscribers, nanosecond skew. Crypto reality = one TCP/WS connection per client, so 10k clients × 1k msg/s = 10M sends/sec and the fanout tier dwarfs the matching engine.
Slow consumers: never backpressure the producer. Bounded per-client queues, conflate book state to keep-latest-per-level (bounded by book width, not message rate), never conflate trades, disconnect as the backstop. It’s your fan-in conflation flipped to fan-out.
Recovery is two-tier: a small in-memory retransmission window for microbursts (MoldUDP64-style, itself rate-limited), and the snapshot channel for everyone else. A/B dual feeds exist so the recovery path almost never gets touched.
The reconnect stampede is the coupling failure: recovery load peaks exactly when live load peaks. Server-enforced jittered backoff, pre-built snapshots served from memory, and N−1 capacity math during a burst — or a slow-consumer policy becomes a full outage.
Per-TCP fanout can’t be perfectly fair — someone is always first in the send loop — so you randomize send order per tick to turn a systematic edge into noise, and you keep the receipts.
Determinism makes the publisher highly testable: golden-feed byte-diffs against replayed prod logs, a reference consumer continuously asserting snapshot+increments equals truth, and adversarial slow/gappy client simulations for the code that otherwise only runs on the worst day of the year.

Interviewer will ask

“You consumed 20 venues’ feeds. Design the feed you wished they’d built.” “Two seq-numbered channels per product tier — incremental and snapshot — with three invariants I’ll never break because I paid for every venue that broke them: gap-free monotonic seq per channel, every snapshot stamped with an exact as_of_seq so the buffer-snapshot-replay recovery dance is deterministic, and trades never conflated even when book updates are. Publisher is a deterministic consumer of the sequenced log — no coupling to the engine, feed replayable for regression tests. Documented conflation semantics and an explicit flag when a client’s view has skipped states — the worst venues were the ones where I had to discover their conflation behavior empirically during a volatile open.”

“Why is the feed derived from the log rather than emitted by the matching engine?” “Decoupling and provable consistency. If the engine publishes directly, the fanout tier’s problems — slow consumers, TLS costs, reconnect stampedes — are one backpressure bug away from the matching path, and any engine/feed mismatch becomes an unanswerable reconciliation question. As a log consumer, the publisher can’t slow matching, scales independently — which matters since fanout is the bigger fleet — and determinism guarantees its book is bit-identical to the engine’s. It’s exactly the read-model projection pattern from my event-sourced engine: writers never wait for projections.”

“One client on your 10k-client WS fanout reads at half rate. What happens, and what do you do?” “Their TCP receive window fills, my socket send buffer fills, their per-client queue grows — and the design requirement is that this is completely invisible to the other 9,999 and to the publisher. Bounded per-client queue; on threshold, switch that client to conflation: collapse queued book updates to latest-state-per-level, so their debt is bounded by book width instead of message rate; queue trades faithfully since those are facts, not states. Mark the seq discontinuity so their book-builder knows to treat it as a partial resync. Hard overflow past that: disconnect and let them re-enter through snapshot recovery. I built the mirror image of this — fan-in conflation across 20 venues when my downstream lagged — so I’d also insist the conflation semantics be in the public spec, because as a consumer I had to reverse-engineer them.”

“Why can’t a TCP fanout be fair, and does it matter?” “Sends are serialized — some client is first in the loop, and iterating in a fixed order hands client #1 a systematic multi-microsecond edge over client #10,000, every tick, which sophisticated clients will detect and either exploit or complain about. Randomizing send order per tick converts the systematic edge into zero-mean noise — that’s the honest definition of fairness available on TCP. Contrast tradfi: one multicast packet, switch replicates in hardware, cable lengths equalized — simultaneity is a physical claim there. On WS it can only ever be a statistical claim, and the venue should be able to demonstrate the shuffle.”

“Your incremental feed and your snapshot service disagree during an incident. How?” “If both are deterministic consumers of the same sequenced log, disagreement is a determinism bug or a consistency-point bug — my first suspect is the snapshot’s as_of_seq: a snapshot cut from a live book without a coherent sequence point, or off-by-one on which increments it includes. That bug ships consumers a book that’s subtly wrong forever after recovery — I’ve debugged it from the client side against a real venue and it presents as slow book drift, which is why I’d build the snapshot service as a replay-from-log at an exact sequence number, never a read of live mutable state, and continuously verify snapshot-plus-increments against a reference book replica in CI and in prod.”

“What timestamps go in a feed message and where do they come from?” “Two from the producer: event_time — when the sequencer sequenced the cause — which must be derived from the log, never the publisher’s wall clock, because it’s part of the deterministic output, identical across tiers and across replays, and it’s what backtests and surveillance key on. And send_time — when this publisher hit the wire — which is purely diagnostic: send minus event is the venue’s own lag, honestly published; consumer receive minus send is their path. I spent years doing that decomposition from the outside, sometimes against venues whose timestamps were flattering rather than true. The producer-side lesson from that: colo consumers with hardware timestamps can measure the lie, so flattery always gets caught. Which means the only defensible posture is one clock — gateways, sequencer, and publishers PTP-synced to the same grandmaster, so my own timestamps are comparable across components. And once the numbers are honest, publish the lag distribution yourself — better clients read it from your docs than discover it in their receive logs.”

“ITCH peaks at millions of messages a second. What does that force on the publisher?” “At tens of millions market-wide, a message arrives every ~100ns at peak. Hold that against costs you know: a single syscall or a single allocation costs more than that entire per-message budget, so anything the publisher does per message has to be a plain memory operation. That forces the pipeline’s shape — messages pre-serialized into fixed binary layouts so emission is a copy, sends batched so one syscall amortizes across dozens of messages, and the multicast path on kernel bypass (ch04) so there’s no per-packet kernel toll at all. But the deeper implication is for the retransmission and recovery infrastructure: at that rate a 100ms consumer glitch is a million-message gap, so gap recovery must come from snapshots and dedicated replay services, never from ‘please resend the increments’ — which is exactly why the incremental+snapshot split exists rather than a reliable-delivery protocol. Reliability is pushed to the edges; the feed itself stays fire-and-forget fast.”

“You connect to a venue’s L2 feed. Walk me through getting a correct book — and what goes wrong if you snapshot first.” “Subscribe to the diff stream first and buffer — I have no book yet, so there’s nothing to apply to. Then fetch the snapshot; it carries sequence S. Discard buffered deltas at or below S — the snapshot already contains them — check the first survivor is exactly S+1, apply the tail, go live. If I snapshot first instead, every update that lands between my snapshot fetch and my stream connect is simply gone: no gap, no error, just a level that’s wrong until the next reconnect — I’ve debugged exactly that as hours of slow book drift. And the standing rule afterward: any sequence gap makes the book stale — stop trading on it, re-bootstrap — because a book that might be wrong is worse than no book. Where the venue publishes checksums, I apply them per update; a mismatch is my corruption detector firing early.”

“Show me the book as it was at 14:32:07 last Tuesday. How?” “The book at a time isn’t stored anywhere — it’s derived: nearest captured snapshot at or before that moment, then replay my journaled deltas up to it. That presumes I built the capture: every normalized event journaled with venue sequence and my receive timestamp, plus periodic snapshots of my built book so the replay is bounded — a WAL plus checkpoints, same design as everywhere else in this stack. The senior half of the answer is the clock: 14:32:07 on whose clock? For best-execution and TCA it’s my receive time — the question is what I could have known when I routed, not what had objectively happened at the venue. And it has to be my own capture, not a vendor’s: a dispute asks what we saw, and a vendor archive is someone else’s clock with someone else’s gaps.”

“How do you know the rebuilt book is right — that you didn’t lose anything?” “Three layers. Completeness is sequence continuity: every delta carries the venue’s sequence number, and I gap-check twice — at capture, where a gap permanently marks that window degraded, and again at replay, where a gap means the archive itself is damaged. Unbroken sequence from the snapshot to the target is the proof nothing is missing — that’s what venue sequence numbers exist for. Correctness is hashes: where the venue publishes book checksums I recompute them during replay — the counterparty certifying my rebuild — and my own snapshots store a state hash the rebuild must reproduce bit-for-bit, which integer-tick prices make possible (the event-sourcing chapter’s determinism contract paying off again). And the systemic layer: adjacent snapshots verify the deltas between them — replay snap-14:00 plus its segment and it must hash-equal the independently-cut snap-15:00; run that over the whole archive nightly and the archive audits itself, with any failure localized to one segment of one symbol. Anything that fails any layer is served as degraded, never silently — same stale-book discipline as the live path.”

The HFT Gap Closer