Introduction: How to Use This Book
You are a senior software engineer with a production trading-systems background — matching engine, order routing, market-data pipelines, built on crypto/web-stack infrastructure. The hardware floor under those systems is the gap, and three parts of it stand between you and holding your own against an experienced HFT engineer:
- Kernel bypass & networking — you’ve optimized your code; HFT engineers also optimize the path between the wire and your code.
- Performance measurement — you know your system does ~10ms; they will ask which microsecond went where, and how do you know your measurement isn’t lying.
- State upgradability & deployment — how do you change the schema, the protocol, or the binary of a system that must not stop.
Each part ends in a runnable lab. Read a chapter, run its lab, then answer its “Interviewer will ask” box out loud before moving on. Part IV is pure drilling — question banks and spoken scripts to rehearse, no new material.
Running the labs
- Labs are Rust unless stated; each lab chapter has a
cargoscaffold to copy. - Networking labs that need Linux (io_uring, AF_XDP,
perf) are marked [Linux]; run them on a throwaway cloud VM or any Linux box — 2 vCPU is enough. macOS-safe alternatives are given where they exist. - Nothing here needs special hardware. Where real HFT uses exotic gear (solarflare NICs, FPGAs, PTP grandmasters), the chapter tells you what to say about it, since the interview tests understanding, not ownership.
The standing rule
Every claim you make in an interview must be one of: something you built, something you measured, or something you can derive on the whiteboard. This book fills the third category and gives you the vocabulary for the first two.
The Machine: What’s Actually in a Server
You’ve deployed to hundreds of servers without ever caring what’s inside the box. That was the correct level of abstraction for your work — a server was “CPU + RAM + network, Docker figures out the rest.” HFT (high-frequency trading) work happens below that abstraction, so this chapter builds the floor. One read of this and the words NIC, PCIe, NUMA, and cache line stop being noise.
Here is the whole machine. Everything in this chapter is a zoom-in on one part of this picture.
┌─────────────────────────────── THE SERVER ───────────────────────────────┐
│ │
│ ┌───────────────── CPU PACKAGE (one "socket") ─────────────────┐ │
│ │ │ │
│ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │
│ │ │ Core 0 │ │ Core 1 │ │ Core 2 │ │ Core 3 │ ... (8–64x) │ │
│ │ │ L1 L2 │ │ L1 L2 │ │ L1 L2 │ │ L1 L2 │ private caches │ │
│ │ └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ │ │
│ │ └──────────┴────┬─────┴──────────┘ │ │
│ │ ┌───────┴────────┐ │ │
│ │ │ L3 cache │ shared by all cores │ │
│ │ │ (~32–256 MB) │ │ │
│ │ └───┬────────┬───┘ │ │
│ │ ┌─────────┴──┐ ┌──┴──────────┐ │ │
│ │ │ Memory │ │ PCIe root │ │ │
│ │ │ controller │ │ complex │ │ │
│ └────────┴─────┬──────┴──┴──────┬──────┴───────────────────────┘ │
│ │ │ PCIe lanes ("the highway") │
│ ┌───────┴──────┐ ┌─────┴──────┬─────────────┐ │
│ │ RAM │ │ │ │ │
│ │ (DIMM sticks│ ┌─┴────┐ ┌────┴────┐ ┌─────┴────┐ │
│ │ 64–512 GB) │ │ NIC │ │ NVMe │ │ GPU / │ │
│ └──────────────┘ │ │ │ SSD │ │ FPGA │ │
│ └──┬───┘ └─────────┘ └──────────┘ │
│ │ │
└─────────────────────────────┼────────────────────────────────────────────┘
│
══════╪══════ the wire (fiber/copper to the switch)
Keep this map in your head. The core question of all low-latency work is: a market-data packet arrives on the wire at the bottom — how many of these boxes does it pass through, and how long does each one take, before your strategy code sees it?
The CPU: cores, threads, clocks
┌──────────────── ONE CORE ────────────────┐
│ │
│ Hyperthread 0 Hyperthread 1 │ ← two "logical CPUs"
│ (register set A) (register set B) │ the OS schedules onto
│ └──────┬───────────────┘ │
│ ┌──────┴────────────────┐ │
│ │ ONE set of execution │ │ ← but only one physical
│ │ units (ALUs, load/ │ │ engine underneath
│ │ store, branch pred.) │ │
│ └───────────────────────┘ │
│ L1i 32KB │ L1d 32KB │ L2 ~1MB │ ← this core's private caches
└──────────────────────────────────────────┘
CPU core — one independent instruction-executing engine; a 32-core chip is genuinely 32 little computers sharing a memory system. A trading system cares because latency work is about giving one core one job and never letting anything else touch it.
Clock speed — how many instruction-steps a core takes per second; 3 GHz (gigahertz, billions of cycles per second) means one cycle every 0.33 nanoseconds. Trading systems care because every latency number in this book is secretly a cycle count: “100 ns” means “this core could have executed ~300 instructions in that time.” Adopt this human-scale conversion and keep it for the whole book: if one cycle were one second, 1 ns is ~3 seconds, and 100 ns is ~5 minutes.
Hyperthread (SMT, Simultaneous Multi-Threading) — one physical core pretending to be two logical CPUs by keeping two threads’ register state loaded and interleaving them whenever one stalls waiting on memory. Analogy: one Express server handling two requests “concurrently” — it’s not parallelism, it’s interleaving during I/O waits; throughput up, per-request latency jittery. A trading system cares because a hyperthread sibling competes for the same execution units and L1/L2 cache as your hot thread — so HFT boxes typically disable SMT or leave the sibling of every hot core empty. Free throughput is precisely what you don’t want; you want deterministic latency.
Pinning (CPU affinity) — telling the OS scheduler “this thread runs on core N and nowhere else,” via taskset or pthread_setaffinity_np. Physically it means: this core’s L1/L2 caches fill up with your data and stay warm, because no other thread is ever scheduled there to evict them. Analogy: sticky sessions. A load balancer that can bounce a user between app servers loses every server-local cache; pin the user to one server and its cache stays hot. Trading systems pin every latency-critical thread, always — an unpinned thread that migrates cores arrives at the new core with every cache cold, which is a multi-microsecond tax paid at a random moment.
The cache hierarchy: the latency ladder
core issues a load: "give me the 8 bytes at address X"
│
┌───── hit? ────▼──────┐
│ L1 ~1 ns 32 KB │ per-core, checked first
└───── miss ────┬──────┘
┌───── hit? ────▼──────┐
│ L2 ~4 ns ~1 MB │ per-core
└───── miss ────┬──────┘
┌───── hit? ────▼──────┐
│ L3 ~15 ns ~64 MB │ shared by all cores
└───── miss ────┬──────┘
┌───────────────▼──────┐
│ RAM ~80 ns 256 GB │ "main memory" — the actual DIMMs
└──────────────────────┘
Cache hierarchy — a stack of progressively bigger, slower memories between the core and RAM (Random Access Memory), where every load checks L1, then L2, then L3, then finally RAM. Analogy: it is exactly your caching stack — in-process object cache → Redis → Postgres → S3 — except in silicon and managed automatically by the hardware. A trading system cares because the difference between “hot path fits in L1/L2” and “hot path chases pointers through RAM” is the difference between a 500 ns and a 5 µs tick-to-trade — same algorithm, same big-O.
The ladder, with the rest of the machine included and translated to human scale (1 cycle = 1 second):
| Where the data was | Latency | Human scale | Your-stack equivalent |
|---|---|---|---|
| L1 cache | ~1 ns | ~3 seconds | reading a local variable |
| L2 cache | ~4 ns | ~12 seconds | in-process LRU cache |
| L3 cache | ~15 ns | ~45 seconds | same-box Redis |
| RAM | ~80 ns | ~4 minutes | Postgres on localhost |
| NVMe SSD read | ~100 µs | ~3.5 days | S3 GET |
| Same-datacenter network hop (cross-service call, full kernel stack both ends) | ~50–500 µs | days to weeks | cross-service HTTP call |
| Internet round trip | ~1–100 ms | weeks to years | third-party API call |
Two lessons hiding in this table. First: RAM is not fast. RAM is your “database” — an 80 ns miss is a 4-minute stall in human scale, and code that misses cache on every access is doing the equivalent of one DB query per variable read. Second: the gap between RAM and network (~80 ns → ~50 µs) is three orders of magnitude, which is why the entire HFT game is “never leave the box, and inside the box, never leave the cache.”
Cache line — the unit in which memory actually moves: 64 bytes, always; ask for 1 byte and the hardware fetches the surrounding 64. Analogy: Postgres never reads you one column — it reads the whole 8 KB page containing the row, because the expensive part is the trip, not the payload. Same logic: the trip to RAM costs ~80 ns whether you take 1 byte or 64, so the hardware always takes 64. A trading system cares in two directions: (a) pack your hot struct so everything you touch per tick sits in one or two lines — one fetch instead of five (this is why order-book entries in HFT code are laid out by hand, like designing a table around its access pattern instead of normalizing it); (b) never let two cores write to different variables in the same line — the line ping-pongs between their caches, a pathology called false sharing you’ll meet in the concurrency chapters.
NUMA: the box is secretly two boxes
┌── Socket 0 ─────────────┐ ┌── Socket 1 ─────────────┐
│ cores 0–15 │ inter- │ cores 16–31 │
│ L3 cache │◄─socket─► L3 cache │
│ memory controller │ link │ memory controller │
└──────┬──────────────────┘ (UPI) └──────┬──────────────────┘
│ │
┌───▼──────┐ ┌───▼──────┐
│ RAM bank │ local: ~80 ns │ RAM bank │
│ 128 GB │ ← from socket 1: │ 128 GB │
└──────────┘ ~130–200 ns └──────────┘
▲
┌───┴───┐
│ NIC │ ← the NIC is also plugged into ONE socket's PCIe
└───────┘
NUMA (Non-Uniform Memory Access) — on a two-socket server, each CPU package has its own RAM banks; a core reading its own socket’s RAM pays ~80 ns, but reading the other socket’s RAM crosses the inter-socket link and pays roughly double. Analogy: primary database plus a read replica in another region — same data model, same API, but one of them silently costs you an extra round trip, and the ORM won’t warn you. A trading system cares because the NIC hangs off one socket’s PCIe lanes: if your hot thread is pinned to a core on the other socket, every packet takes the cross-socket detour before your code sees it — a permanent, invisible ~50–100 ns tax on every single message. The standard HFT layout: find which socket owns the NIC (/sys/class/net/eth0/device/numa_node), pin the entire hot path — threads and their memory allocations — to that socket, and treat the other socket as a separate machine that runs logging and cron jobs.
PCIe: the network inside the box
CPU package
┌───────────────────┐
│ PCIe root complex│ "root complex" = the CPU-side
└──┬─────┬───────┬──┘ terminus of all PCIe traffic
│x16 │x8 │x4 ← lane counts: like link bandwidth tiers
┌──▼──┐ ┌▼────┐ ┌▼─────┐
│ GPU │ │ NIC │ │ NVMe │
└─────┘ └─────┘ └──────┘
each lane ≈ 2–8 GB/s each way (gen 3→5); a x16 slot = 16 lanes
one device round trip (CPU → device register → CPU): ~500 ns – 1 µs
PCIe (Peripheral Component Interconnect Express) — the point-to-point packet-switched network inside the box connecting the CPU/RAM complex to every plug-in card: NIC, NVMe (Non-Volatile Memory Express, i.e. modern SSDs), GPU, FPGA. It is literally a network — devices exchange addressed packets over serial links, with lanes bundled for bandwidth like link aggregation. Analogy: it’s the network between your app server and your database, except it lives inside the chassis and its round trip is ~500 ns–1 µs instead of ~500 µs. A trading system cares because PCIe is an unavoidable segment of the packet’s journey — the wire delivers bits to the NIC, but they only reach RAM by crossing PCIe — so on a well-tuned box PCIe becomes one of the largest remaining line items in the latency budget. In human scale: a 1 µs PCIe device round trip is ~50 minutes. This is also the door to the endgame you’ll meet later: FPGAs (Field-Programmable Gate Arrays) win partly by responding from the card itself, skipping the trip to the CPU entirely.
The NIC: a sidecar computer that owns the network
the wire (light in fiber / voltage on copper)
│
┌────────────────── NIC ────────────▼────────────────────────────┐
│ ┌──────┐ analog signal ⇄ digital bits │
│ │ PHY │ (clock recovery, encoding — pure electronics) │
│ └──┬───┘ │
│ ┌──▼───┐ bits ⇄ Ethernet frames │
│ │ MAC │ (framing, addresses, CRC checksum, drop bad frames)│
│ └──┬───┘ │
│ ┌──▼──────────────────────────────┐ │
│ │ on-card processor + SRAM buffers│ queues, filtering, │
│ │ + DMA engines │ timestamping, offloads │
│ └──┬──────────────────────────────┘ │
└─────┼──────────────────────────────────────────────────────────┘
│ PCIe
▼
host RAM ← the NIC *writes packet bytes into RAM by itself*
NIC (Network Interface Card) — the hardware whose job is turning signals on a wire into bytes sitting in RAM, and back. A NIC is not a dumb port — it is a small independent computer with its own processor, own memory, and own direct access to your RAM, living on the PCIe bus. Analogy: a sidecar container that owns all network I/O for the pod — your app never touches the wire, it exchanges messages with the sidecar through shared memory; the sidecar has its own CPU budget, its own config, its own failure modes. A trading system cares because everything in Part I is really about your relationship with this device: how it signals you (interrupts vs polling), where it writes packets (kernel buffers vs your process’s memory — the entire “kernel bypass” story), and what work you delegate to it (checksums, timestamping, filtering). Premium NICs (Solarflare/Xilinx, Mellanox/NVIDIA, Exablaze) are HFT’s most-tuned components after the strategy itself.
PHY (physical layer transceiver — pronounced “fie”) — the analog electronics at the NIC’s edge that turn light pulses or voltage wiggles into clean digital bits: modulation, clock recovery, encoding. Analogy: the terminating proxy that turns raw TLS byte-noise into clean HTTP before your app sees anything — except this one terminates physics. Trading cares mostly at the extremes: PHY choices add fixed tens-of-nanoseconds, and exotic setups (hollow-core fiber, microwave links between exchanges) are competitions at this layer.
MAC (Media Access Control layer) — the digital logic one step up from the PHY that groups bits into Ethernet frames, reads source/destination hardware addresses, verifies the CRC (Cyclic Redundancy Check — a checksum), and silently drops corrupt frames. Analogy: body-parsing + signature-verification middleware — it turns a byte stream into discrete validated messages and rejects garbage before anything downstream runs. Trading cares because “the MAC” is where hardware timestamping hooks in (stamping a packet the instant it crosses this boundary — the most honest arrival time you can get), and because frames the MAC drops are invisible to software — you’ll learn to check NIC hardware counters when the sequence numbers in your feed have gaps.
C-states and frequency scaling: the CPU is trying to nap
C0 ───────── running wake cost
C1 ───────── halted, clocks gated ~1 µs
C3 ───────── core caches flushed ~tens of µs DEEPER
C6 ───────── core powered off, state ~100+ µs SLEEP
parked in shared cache
plus: frequency scaling — an idle core drops from 3.5 GHz to 1.2 GHz
and takes real time to spin back up when work arrives
C-states — numbered depths of CPU sleep (C0 = running, C6 = core effectively powered off) that idle cores drop into automatically to save power, each deeper level cheaper to sit in and more expensive to wake from. Analogy: serverless cold starts, in miniature and in silicon — scale-to-zero saves money and murders p99. Frequency scaling (P-states) is the same instinct on a running core: clock down when load looks light. A trading system cares because market data is bursty — the moment that matters most (a news spike after a quiet spell) is precisely the moment the governor has decided your core should be asleep and slow, adding tens of microseconds of wake-up latency to exactly the packet you care about. So HFT boxes disable C-states beyond C1, pin the clock at maximum (idle=poll, performance governor), and burn watts for determinism. It’s also half the reason hot loops busy-spin instead of blocking: a spinning core is a core that can never be caught napping. Provisioned concurrency, but for silicon.
The whole-machine latency map
Every number from this chapter on one picture. This is the terrain map for the rest of the book — later chapters are campaigns to shorten or eliminate specific arrows.
┌──────── CORE (3 GHz: 1 cycle = 0.33 ns) ────────┐
│ register: 0 ns │ branch mispredict: ~5 ns │
└───┬───────────────────────────────────┬─────────┘
~1 ns │ L1 │
~4 ns │ L2 │ cross-core /
~15 ns │ L3 ◄───────────────────────────►│ false-sharing
▼ ping-pong:
┌───────┐ cross-socket (NUMA): ~40–100 ns
│ RAM │ +50–100 ns on top
│ ~80ns │
└───┬───┘
│ PCIe round trip: ~500 ns – 1 µs
┌───▼───┐
│ NIC │ wire→RAM via DMA: ~1 µs-ish
└───┬───┘
│ serialization, 100B @ 10 Gbps: ~80 ns
══════╪══════ the wire
│ propagation: ~5 ns per metre of fiber
▼
same-rack switch hop: ~0.5–5 µs · cross-country: ~30 ms
human scale (1 cycle = 1 s): L1 = 3 s · RAM = 4 min · PCIe = 50 min
SSD = 3.5 days · 1 ms of network = 5 weeks
What you can now read
You now have the vocabulary for the physical machine. This unlocks:
- The Kernel, Your Code, and the Wall Between Them (next) — what the software between the NIC and your code does with these components, and what it costs.
- Part I’s network path and kernel-bypass chapters — “the NIC DMAs into a ring on the NUMA node local to the pinned core” is now a sentence, not a hash of acronyms.
- Part I’s CPU isolation and tuning material — isolcpus, C-state and governor settings, and SMT policy are now just “stop the nap, stop the migration, stop the sibling.”
- The cache-aware data structures and false-sharing chapters — the latency ladder and the 64-byte cache line are their entire foundation.
The Kernel, Your Code, and the Wall Between Them
The previous chapter was the hardware. This chapter is the software layer that sits between that hardware and every program you’ve ever written — and what it costs to cross into it. HFT’s most famous move, “kernel bypass,” is just refusing to pay that cost. You can’t appreciate the bypass until you’ve priced the wall.
┌─────────────────────────────────────────────────────────────────┐
│ RING 3 — userspace (unprivileged) │
│ │
│ your strategy binary · node · postgres · nginx · everything │
│ you have ever deployed │
│ │
│ allowed: arithmetic, branches, reading/writing ITS OWN memory │
│ forbidden: touching hardware, other processes' memory, the NIC│
├══════════════════ THE WALL (CPU-enforced) ══════════════════════┤
│ RING 0 — kernel (privileged) │
│ │
│ scheduler · TCP/IP stack · filesystems · device drivers · │
│ memory management │
│ │
│ allowed: everything — all memory, all devices, all state │
├─────────────────────────────────────────────────────────────────┤
│ HARDWARE — CPU, RAM, NIC, disks │
└─────────────────────────────────────────────────────────────────┘
Kernel vs userspace — the CPU itself has privilege modes (“rings”); the kernel runs in ring 0 with full hardware access, your code runs in ring 3 where instructions that touch hardware or other processes’ memory physically fault, and the only doorway between them is a controlled trap. Analogy: the kernel is the framework/runtime and your code is the route handlers — Express owns the listener, the parsing, the socket lifecycle; your handler receives a cooked req and returns a cooked res, never touching the raw TCP stream. That division of labor is exactly kernel vs userspace, and it has the same trade: enormous convenience, generality, safety — paid for in per-request overhead you normally never itemize. A trading system cares because at microsecond scale that overhead is the product: the kernel’s network path costs ~2–10 µs per packet, and firms discovered they could rewrite the parts they need in userspace and get the same packet for ~1 µs. The wall is enforced by the CPU’s privilege bits, which is why crossing it has an irreducible hardware cost.
Syscalls: knocking on the wall
your code (ring 3) kernel (ring 0)
────────────────────── ────────────────────
recv(fd, buf, len)
│ args into registers
▼
`syscall` instruction ──── CPU switches privilege mode,
jumps to kernel entry point ──► dispatch table
→ tcp_recvmsg()
→ copy data into
your buf
resume after the ◄────── `sysret`: mode flips back ◄──┘
call, ~50–100 ns+ later (plus the work itself)
Syscall (system call) — the one legal way userspace asks the kernel to do privileged work: your code puts arguments in registers and executes a special CPU instruction (syscall) that flips the privilege mode and jumps to a fixed kernel entry point; the kernel does the work and flips back. Analogy: an RPC (remote procedure call) to a service you don’t own — except the “network” is a privilege-mode flip. Every read, write, send, recv, epoll_wait, every console.log bottoms out here. The mechanical crossing alone costs ~50–100 ns before any useful work — in the human scale from the last chapter, ~5 minutes of overhead per knock — and the flip also disturbs branch predictors and cache state, so the true cost bleeds past the instruction itself. A trading system cares because one syscall in a per-packet hot loop is like an HTTP round-trip inside a render loop: individually invisible, structurally fatal. Watch for the pattern all low-latency APIs share — amortize or eliminate the crossing: epoll (one syscall reports many sockets), io_uring (queue requests in shared memory, syscall rarely or never), and kernel bypass (never knock again).
Mode switch vs context switch — a mode switch is the syscall crossing above: same process, same address space, just a privilege flip (~50–100 ns). A context switch is the scheduler replacing which process/thread owns the core: save all registers, swap the address-space mappings, restore another thread’s state — ~1–5 µs, and the direct cost is the small part. The real bill is cache pollution: the incoming thread evicts your L1/L2/TLB, so when your thread returns it runs at RAM-speed until its working set re-warms. Analogy: a context switch is a cold start — your thread got descheduled, and on resume the warm cache it was relying on is gone. A trading system cares because a context switch at the wrong moment adds 1–10+ µs at a time you don’t choose — hence the standard prescription assembled from last chapter’s parts: one pinned thread per hot core, that core isolated from the scheduler (isolcpus), and the thread never blocks, so the scheduler never has a reason to touch it. A hot thread that calls a blocking syscall has volunteered for a context switch.
Interrupts: the hardware’s webhook
┌───────────────┐
packet lands in NIC ───► │ NIC raises │
│ interrupt │ ── electrical signal over
└──────┬────────┘ PCIe to a specific core
▼
Core 2: mid-instruction in SOMEBODY'S code
│ drop everything, save state
▼
run the driver's interrupt handler (in kernel, ring 0)
│ "packet queue 3 has data"
▼
resume the interrupted code — which just mysteriously lost ~1–2 µs
Interrupt (IRQ, Interrupt Request) — a hardware signal from a device that forces a core to suspend whatever it’s running, mid-instruction-stream, and execute that device’s kernel handler right now. Analogy: webhook vs polling — instead of the CPU asking the NIC “anything yet?” a billion times a second, the NIC pushes a notification when there’s data. The default for all of computing. A trading system cares for two opposite reasons: interrupts add ~1–2 µs of delivery latency to the packet (the “webhook delivery time”), and they victimize whichever thread was running on the receiving core (your pinned strategy thread, unless you steer IRQs away from it — that’s irq affinity — page one of every HFT tuning guide). HFT’s resolution is blunt: on the hot path, don’t use interrupts at all — poll. Push notifications are for machines that are allowed to sleep.
MSI-X (Message Signaled Interrupts, eXtended) — the modern interrupt mechanism where a device signals by writing a message over PCIe, and one device can own many independent interrupt vectors, each targeted at a chosen core. Analogy: instead of one shared webhook endpoint for all events, the NIC gets many endpoints — one per queue — each with its own consumer. A trading system cares because MSI-X is what makes traffic-to-core steering possible at all: the NIC can be told “market-data queue interrupts core 3, order-entry queue interrupts core 5, and nothing interrupts cores 1–2, where strategy threads spin.”
hardirq, softirq, NAPI: splitting the webhook handler
NIC interrupt fires
│
▼
┌─ HARDIRQ (top half) ────────────────────────────┐
│ microseconds matter; other interrupts blocked │
│ do ALMOST NOTHING: ack the device, │
│ schedule the bottom half. return. │ ← "return 200 fast"
└───────────────┬─────────────────────────────────┘
▼
┌─ SOFTIRQ (bottom half) ─────────────────────────┐
│ runs soon after, interruptible │
│ the bulk work: drain the NIC ring, build │ ← "the queue worker"
│ sk_buffs, run the TCP/IP stack, wake sockets │
└─────────────────────────────────────────────────┘
NAPI, under load: interrupt fires ONCE →
"interrupts OFF for this queue; I'll POLL the ring in a loop,
draining batches, until it's empty — then interrupts back ON"
hardirq / softirq — the kernel splits interrupt handling into a minimal urgent part (hardirq: acknowledge the device, schedule follow-up) and a deferred bulk part (softirq: actually process the packets), because while a hardirq runs, other interrupts are blocked and the whole core is hostage. Analogy: a webhook handler that validates the signature, enqueues a job, and returns 200 in two milliseconds — while a worker pool drains the queue. Same pressure, same shape, same reason: keep the front door clear. A trading system cares because “my packet arrived” and “my packet was processed” are now two events at two times — softirq work can be deferred, migrated, or batched, and it’s a classic source of latency jitter that shows up in your p99.9 with no smoking gun in your own code.
NAPI (New API — genuinely its name) — the kernel’s adaptive strategy for high packet rates: after one interrupt, disable that queue’s interrupts and switch to polling the NIC in a loop, draining packets in batches, re-enabling interrupts only when the queue runs dry. Analogy: your Redis pub-sub consumer melting under per-message callbacks, so you switch to batch-draining the queue in a loop under load and only re-subscribe to notifications when it’s empty. When the doorbell never stops ringing, stop answering the door and just keep it open. A trading system cares because NAPI is the intellectual bridge to HFT’s signature move: the kernel itself concedes that under load, polling beats interrupts. HFT extends the logic to its limit — if polling wins under load, and you care only about moments of load, then poll always, from userspace, with a dedicated spinning core, and never take an interrupt at all.
DMA and the descriptor ring: how packets skip the CPU
┌───── NIC ─────┐ ┌────────── RAM ─────────┐
│ DMA engine │ ── writes bytes ─────► │ packet buffer #17 │
└───────┬───────┘ over PCIe, │ packet buffer #18 │
│ NO CPU INVOLVED └────────────────────────┘
│
│ then updates a slot in the shared ring:
▼
RX DESCRIPTOR RING (circular array in RAM, shared NIC ⇄ driver)
┌──────┬──────┬──────┬──────┬──────┬──────┐
│ #15 │ #16 │ #17 │ #18 │empty │empty │
│ done │ done │ FULL │ FULL │ ▲ │ │
└──────┴──────┴──▲───┴──────┴─┼────┴──────┘
driver reads──┘ └── NIC writes next here
from here (driver pre-posted these free
buffers for the NIC to fill)
DMA (Direct Memory Access) — hardware devices writing to (and reading from) RAM directly over PCIe, without the CPU copying a single byte; the CPU only finds out afterwards. Analogy: presigned S3 uploads — the client writes the 5 GB file straight to storage and your API server just receives a small “upload complete” notification, instead of proxying every byte through its own process. A trading system cares because DMA is why packet arrival is nearly free for the CPU — the expensive part of the kernel path is everything that happens after the bytes are already sitting in RAM. It’s also the enabling trick of kernel bypass: DMA doesn’t care whose memory it targets, so point the NIC’s DMA engine at buffers your process owns, and packets materialize directly inside your application. Same trick as the S3 upload: cut out the middleman that was only ever copying bytes.
Descriptor ring — a fixed-size circular array in RAM, shared between driver and NIC, where each slot (“descriptor”) is a small record — pointer to a buffer, length, status flags — through which the two sides coordinate: the driver pre-posts empty buffers, the NIC fills them and flips status bits, the driver harvests and re-posts. Analogy: a bounded SPSC (single-producer single-consumer) work queue where the consumer pre-posts empty envelopes and the producer fills them — backpressure by construction, allocation-free by construction. A trading system cares for three reasons: if the ring fills because software drains too slowly, the NIC drops packets on the floor (that’s rx_missed in NIC stats — check it when your feed gaps); ring size is a latency/loss dial (a big ring absorbs bursts but lets queued packets go stale; a small ring keeps everything fresh or drops it); and the ring is exactly the interface DPDK-style bypass frameworks map into your process — you’ll spend real time with these rings later, so learn the shape now.
sk_buff (socket buffer) — the kernel’s per-packet metadata object: a struct wrapping the packet’s bytes plus headroom, protocol headers, timestamps, routing verdicts, and bookkeeping, allocated for every packet on arrival and freed on delivery. Analogy: Express’s req — one object per request, created by the framework, progressively annotated by each middleware (parsed body, auth context, route params) as it moves down the chain; sk_buffs move through the netfilter/IP/TCP layers accumulating annotations the same way. A trading system cares because the sk_buff is the itemized overhead of the kernel path: allocation, initialization, per-layer bookkeeping, and freeing cost hundreds of nanoseconds per packet — for a market-data payload that might be 40 bytes. Kernel-bypass frameworks’ packet objects are, by design, almost nothing: a pointer and a length. No req object, just the bytes.
Pages, page faults, TLB, hugepages
virtual address (what your pointers hold)
│
▼
┌───────────┐ hit (~free)
│ TLB │ ────────────► physical RAM address, done
│ (a cache) │
└─────┬─────┘
│ miss
▼
page-table walk: up to 4 dependent memory reads ← ~100 ns
│
▼
mapping not present at all? → PAGE FAULT → kernel takes over ← ~1+ µs
hugepages: map 2 MB (or 1 GB) per page instead of 4 KB
→ 512× fewer mappings → TLB almost never misses
Page — the granularity of the virtual-memory illusion: memory is managed in 4 KB chunks, and every address your process uses is a virtual address translated to a physical RAM location through per-process page tables maintained by the kernel. Analogy: the pointer you hold is a DNS name, not an IP — every access resolves through a mapping layer you normally never see. Trading cares because the mapping layer has costs, and they hide in tails.
Page fault — the CPU trap taken when code touches a page with no valid mapping: the kernel intervenes — allocating a zeroed page on first touch, or (catastrophically) reading from disk if the page was swapped out. Analogy: lazy loading — the ORM handed you an object stub, and touching a field fires a query. Costs a microsecond-plus, at first touch — a moment you didn’t schedule. A trading system cares because faults are silent tail-latency landmines inside innocent memory accesses; hence the HFT idiom: allocate everything up front, touch every page once during warm-up, and mlock the lot so the kernel can never page it out. Eager-load the entire object graph at boot; never lazy-load in the hot path.
TLB (Translation Lookaside Buffer) — the tiny hardware cache of recent virtual→physical translations (~1,500 entries) that makes paging affordable; a miss forces a page-table walk of up to four dependent memory reads (~100 ns) before your actual memory access even starts. Analogy: a route cache in front of a slow resolver; hits are free, misses pay the full lookup, and 1,500 entries × 4 KB covers only ~6 MB of hot data. A trading system cares because a large working set (a full order book, say) can thrash the TLB and quietly double effective memory latency — you’re paying two lookups per access and neither shows up in your code.
Hugepages — mapping memory in 2 MB or 1 GB pages instead of 4 KB, so each TLB entry covers 512×–262,144× more memory and misses effectively vanish. One big long-lived mapping replaces thousands of tiny ones, and the same fixed-size TLB now reaches your whole working set. A trading system cares because hugepages are free tail-latency insurance for large in-memory state, and — practical note — DPDK and friends flatly require them for their packet buffer pools; the first line of every DPDK setup guide is reserving hugepages, and now you know why it’s there.
The whole path: a packet arrives — who does what, in order
This is the entire chapter in one diagram, with the bill attached. The next chapter walks this same path from the top down; Part I of the book is about deleting steps 4–9.
1. bits on the wire PHY/MAC ~100 ns
2. NIC validates frame, picks an RX queue NIC ~100 ns
3. DMA writes packet into a ring buffer in RAM NIC/PCIe ~500 ns
4. NIC raises MSI-X interrupt at the queue's core hardware ┐
5. HARDIRQ: ack, schedule NAPI, return kernel ├ ~1–2 µs
6. SOFTIRQ/NAPI: drain ring, allocate sk_buff, kernel ┘
─ then per packet:
7. IP + TCP/UDP processing, socket lookup, kernel ~1–2 µs
enqueue on socket receive queue
8. wake the blocked/epoll-ing process; kernel ~1–5 µs
scheduler gets it back onto a core
9. recv() syscall: mode switch + copy bytes kernel→you ~0.5–1 µs
from sk_buff into your buffer
10. YOUR CODE finally sees the packet ═══════
total: ~ 2–10 µs
(kernel bypass: steps 4–9 replaced by
"your spinning thread reads the ring": ~1 µs total)
What you can now read
- From an Express Handler to the Wire (next) — this same stack, walked top-down from code you already own, with the microsecond bill itemized per layer.
- Part I’s kernel-bypass chapters (DPDK, Onload, ef_vi, AF_XDP) — every one of them is a different answer to “which of steps 4–9 do we delete, and who does the ring-draining instead?”
- The OS tuning chapters — isolcpus, IRQ affinity, NAPI/busy-poll settings, hugepage reservation: each knob now maps to a specific box in the diagrams above.
- The jitter and tail-latency material — softirq scheduling, context switches, page faults, and TLB misses are the standard suspect list, and you’ve now met each one with a price tag.
From an Express Handler to the Wire
This is the bridge chapter. Take the thing you know best — a request arriving at a handler — and ride it downward through every layer from the last two chapters, until we hit photons. At each level: what you always thought happened, what actually happens, what it costs, and what an HFT engineer does about it.
The full elevator shaft first. We’ll stop at every floor.
LEVEL 1 app.post('/order', handler) ← your mental model ends here
LEVEL 2 event loop / epoll ← the runtime's secret life
LEVEL 3 socket API + syscalls ← the wall from the kernel chapter
LEVEL 4 kernel TCP/IP stack ← sk_buffs, protocol layers
LEVEL 5 driver + NIC descriptor rings ← the shared queue
LEVEL 6 PCIe + DMA ← the highway from the machine chapter
LEVEL 7 the wire ← physics
A note before we descend: everything below Level 1 is symmetric. A request arriving climbs up 7→1; your response leaving rides down 1→7. We’ll narrate the receive direction because that’s where trading systems bleed.
Level 1 — your handler
app.post('/order', async (req, res) => {
const order = validate(req.body); // ← "my code starts here"
await book.execute(order);
res.json({ status: 'accepted' }); // ← "my code ends here"
});
What you always thought: requests arrive, my function runs, latency is my function’s fault. Which was true at your scale — when the handler does 2 ms of database work, the transport is rounding error.
What actually happens: by the time your handler’s first line runs, the request has already crossed six layers, been copied 2–3 times, triggered an interrupt, woken a process, and passed through a scheduler. Your function is the visible tenth of the iceberg.
What it costs: the sub-basement adds ~2–10 µs before line one of your code — invisible under a 2 ms handler, unacceptable when the whole job must finish in 5 µs.
What HFT does: inverts the ratio. The business logic (parse tick → update book → decide → emit order) is often only ~1 µs of real work, so the transport layers dominate, and that’s why this entire book exists. The handler stops being “where latency lives” and becomes the cheap part.
Level 2 — the event loop and epoll
┌────────────────── node / tokio runtime ─────────────────┐
│ │
│ loop { │
│ events = epoll_wait(epfd, ...); ← ONE syscall, │
│ │ sleeps until │
│ │ any socket fires│
│ for ev in events { │
│ read the socket, parse HTTP, │
│ invoke YOUR callback ← level 1 lives │
│ } inside this loop│
│ } │
└─────────────────────────────────────────────────────────┘
What you always thought: “Node is event-driven; libuv handles the sockets; callbacks fire when data arrives.” True — and you correctly never asked how a sleeping process finds out data arrived.
What actually happens: the runtime parks in epoll — a Linux facility where you register thousands of sockets once, then make a single epoll_wait syscall that blocks until any of them has data, returning a batch of ready ones. This is the entire trick behind “Node scales to 10k connections on one thread”: one blocked thread, one syscall, N sockets. (kqueue on macOS, IOCP on Windows — same idea.) The part nobody told you: blocks until means your process is descheduled — off the CPU entirely — and when a packet finally arrives, the kernel must mark the socket ready, find your process, and get the scheduler to put it back on a core.
What it costs: that wake-up-and-reschedule dance is ~1–5 µs, plus — remember the kernel chapter — your thread comes back to a possibly cold cache and possibly a different core. Cheap per event amortized across 10k idle connections; brutal as a fixed tax on the one connection you care about.
What HFT does: refuses to sleep. The hot thread never parks in epoll_wait — it spins, checking for data in a tight loop, burning a whole core at 100% forever so there is nothing to wake up. Scale-to-zero versus provisioned concurrency, taken to the physical limit. More on this trade at the end of the chapter.
Level 3 — the socket API: crossing the wall
runtime calls: read(fd, buf, 65536)
│
▼ ring 3 ──► ring 0 mode switch (~50–100 ns)
═════╪═══════════ THE WALL ═══════════════════════
▼
kernel: find socket for fd
copy waiting bytes: kernel sk_buff ──► your buf (~0.2–1 µs)
free the sk_buff
│
▼ ring 0 ──► ring 3
returns: "here are 143 bytes"
What you always thought: nothing — socket.read() was a function like any other.
What actually happens: every socket operation is a syscall through the wall from the kernel chapter, and — the detail that matters — the data is copied: packet bytes already sitting in kernel memory (inside an sk_buff) are duplicated into your process’s buffer, because the wall means kernel memory and your memory are disjoint worlds.
What it costs: ~50–100 ns for the crossing plus ~0.2–1 µs for the copy and socket bookkeeping. Per packet — and market data is millions of packets.
What HFT does: attacks both halves. The copy → zero-copy techniques: arrange for packet buffers to be readable by your process directly, so you get a pointer to where the bytes already are instead of a duplicate (same instinct as sendfile, or streaming a request body instead of buffering it). The crossing → amortize it (io_uring: submit and reap I/O through queues in shared memory, syscalls optional) or delete it outright (kernel bypass, next levels).
Level 4 — the kernel TCP/IP stack
sk_buff climbing the stack (receive direction):
driver hands over raw frame
│
▼
┌ Ethernet ┐ is this MAC address mine? strip header
▼
┌ IP ──────┐ is this IP mine? checksum, firewall/netfilter
▼ hooks, routing decision, reassembly if fragmented
┌ TCP ─────┐ which connection? (hash lookup) · in order?
▼ · update windows · schedule ACK · or:
┌ UDP ─────┐ which socket? checksum. done. (this is why
▼ market data ships over UDP)
append to socket receive queue ──► wake epoll (level 2)
What you always thought: “TCP (Transmission Control Protocol) guarantees ordered, reliable delivery” — a property of the network, roughly like gravity.
What actually happens: TCP is not a thing the network does; it’s a large kernel program that runs per packet on your CPU. Reliability means the kernel keeps a copy of everything sent until acknowledged, tracks sequence numbers, detects gaps, retransmits, manages congestion windows — a full bookkeeping department, run in softirq context (kernel chapter), with state lookups and locks per packet. UDP (User Datagram Protocol) by contrast is barely a protocol: checksum, find socket, deliver — which is why exchanges ship market data over UDP multicast and let applications handle gaps (you’ll meet gap-fill and recovery channels in the market-data chapters; the reliability bookkeeping moves up a layer — it never disappears).
What it costs: ~1–2 µs per packet through the full TCP path; UDP a fraction of that.
What HFT does: UDP wherever the venue allows; where TCP is mandatory (order entry at many venues), a stripped userspace TCP stack — Onload and friends reimplement TCP outside the kernel with none of the general-purpose baggage — plus knobs like TCP_NODELAY (disable Nagle’s batching of small writes; batching is throughput’s friend and latency’s enemy — the same reason you flush a webhook immediately instead of coalescing).
Level 5 — driver and NIC rings
┌── NIC ──────────────┐ RX descriptor ring (RAM)
│ frame arrives, │ ┌────┬────┬────┬────┬────┐
│ RSS hash picks a │──DMA──►│full│full│full│free│free│
│ queue │ └──▲─┴────┴──▲─┴────┴────┘
└─────────────────────┘ driver─┘ └─NIC's write cursor
harvests filled slots (in softirq/NAPI),
builds sk_buffs, re-posts free buffers
What you always thought: the driver shuttles frames between the NIC and the kernel — true, but you never had reason to watch how.
What actually happens: the descriptor ring from the kernel chapter, live. The driver pre-posts empty buffers; the NIC fills them and advances its cursor; NAPI polling harvests them in batches. One detail is new and important: RSS (Receive Side Scaling) — the NIC hashes each packet’s addresses/ports to pick among multiple RX rings, each ring owned by a different core. That’s consistent-hash sharding of inbound traffic across consumers, in silicon — one flow always lands on the same core (ordering preserved per flow), load spreads across cores.
What it costs: the harvest-and-rebuild machinery is part of the ~1–2 µs softirq bill from level 4; the dangerous cost is drops — ring overflows when draining lags a burst, and market data bursts precisely when it matters.
What HFT does: steering, then seizure. First steering: use RSS/flow-steering rules so the feed you care about lands on a dedicated ring whose interrupts target a dedicated core — traffic-shaped, isolated, like a priority queue with its own worker pool. Then seizure: kernel-bypass frameworks map the rings directly into your process — your spinning thread (level 2’s refusal to sleep) polls the descriptor ring itself, no driver, no sk_buff, no syscall. Levels 2 through 5 collapse into “read the next slot of a shared-memory SPSC queue,” which — note — is a thing you already know how to do.
Level 6 — PCIe and DMA
NIC PCIe RAM
┌─────────┐ write burst: packet bytes ┌──────────────────┐
│ frame in │ ════════════════════════════► │ posted buffer │
│ SRAM │ write: descriptor update │ descriptor slot │
└─────────┘ ════════════════════════════► └──────────────────┘
(~500 ns – 1 µs for the round trip;
writes can land ahead of the doorbell — ordering rules
on this bus are their own dark art)
What you always thought: the network card is “attached to” the computer, prepositionally.
What actually happens: the NIC is a peer on the PCIe network (machine chapter) doing DMA writes into RAM — packet bytes first, then the descriptor status flip that announces them. The CPU is not involved; the first CPU-visible evidence of the packet is a changed word in RAM.
What it costs: ~500 ns–1 µs of PCIe transit — and here’s the sobering part: this is the floor. Once you’ve deleted the kernel (levels 3–5) and the sleep (level 2), PCIe transit is most of what remains — and unlike the rows above it, there is no software trick that deletes it.
What HFT does: shaves and relocates. Shaving: cache-coherent tricks so polling the descriptor is cheap, write-combining for the transmit direction, NUMA-local buffers (machine chapter) so DMA lands next to the consuming core. Relocating: if PCIe transit to the CPU is the floor, move the decision onto the card — FPGA NICs that parse the feed and fire the response from the device itself, never crossing PCIe at all. That’s the endgame chapter, and now you know exactly which cost it exists to delete.
Level 7 — the wire
NIC serializes the frame: 100 bytes = 800 bits
at 10 Gbps: one bit every 0.1 ns → 80 ns to put the frame on the wire
then propagation: light in fiber ≈ 5 ns per metre
across the colo hall: ~50–200 ns
Chicago → New York: ~4,000,000 ns one way. no appeal.
What you always thought: the network is “fast” and its speed is someone else’s SLA (service-level agreement).
What actually happens: two separate physical costs. Serialization — payload size ÷ link speed; the time to spell out the bits. Propagation — distance ÷ two-thirds the speed of light (light travels slower in glass); the time for the first bit to arrive. Nothing negotiates with either.
What it costs: serialization: 80 ns per 100 bytes at 10 Gbps (halved at 25 Gbps — this is why link speed matters even at tiny message sizes). Propagation: 5 ns per metre, forever.
What HFT does: buys distance. Colocation — your server in the exchange’s own data center; cross-connects — the literal shortest fiber the facility will sell between your cage and the matching engine, priced per metre because metres are nanoseconds; microwave links between cities (air beats glass — light in air is ~50% faster than in fiber). It’s CDN edge logic driven to its physical conclusion: when you can’t make the machine faster, move the machine. And per-message: smaller encodings serialize faster — one reason binary formats like SBE beat text protocols (a message half as long hits the wire in half the time).
The bill, itemized
One 100-byte market-data packet, 10 Gbps link, well-configured stock Linux, process parked in epoll. Where the microseconds actually go:
| # | Step (level) | Cost | Running total |
|---|---|---|---|
| 1 | serialization onto the wire — 800 bits @ 10 Gbps (L7) | 0.08 µs | 0.08 µs |
| 2 | propagation across the colo hall (L7) | 0.1 µs | 0.2 µs |
| 3 | PHY + MAC: signal → validated frame (L6, machine chapter) | 0.2 µs | 0.4 µs |
| 4 | DMA: frame + descriptor into RAM over PCIe (L6) | 0.5–1 µs | ~1 µs |
| 5 | MSI-X interrupt + hardirq on the queue’s core (L5) | 0.5–1 µs | ~2 µs |
| 6 | softirq/NAPI: harvest ring, allocate + build sk_buff (L5) | 0.5–1 µs | ~3 µs |
| 7 | IP + UDP/TCP processing, socket lookup, enqueue (L4) | 0.5–2 µs | ~4 µs |
| 8 | mark socket ready → wake process → scheduler places it on a core, caches cold (L2) | 1–4 µs | ~6 µs |
| 9 | recvmsg syscall: mode switch + copy sk_buff → your buffer (L3) | 0.3–1 µs | ~7 µs |
| 10 | your code sees byte zero | — | ~2–10 µs |
That range — spanning good and unlucky runs — is the “kernel path costs ~2–10 µs per packet” figure Part I asserts on its first page. Now it’s not an assertion; it’s rows 5–9 of a table you can audit. And notice the shape of the fix: rows 5–9 are all software convention, not physics. Kernel bypass replaces them with “spinning thread reads the descriptor ring”: the same packet reaches your code in ~1–1.5 µs, i.e. rows 1–4 plus a ring read. Rows 1–4 are physics and PCIe — that’s the floor FPGAs exist to dig under.
epoll vs blocking recv vs busy-polling
Three ways for code to consume a socket — you’ve lived the first two without naming them. In event-loop terms:
BLOCKING recv one thread per connection; each thread sleeps in
recv() until its socket has data
= classic thread-per-request servers (pre-Node)
wake cost per event: ~1–5 µs · burns: a thread each
EPOLL one thread sleeps in epoll_wait() on N sockets;
kernel wakes it with a batch of ready ones
= the Node/libuv/tokio model — YOUR model
wake cost per event: ~1–5 µs, amortized · burns: ~nothing
BUSY-POLL one PINNED thread, never sleeps:
loop { if (ring/socket has data) handle() }
= while(true) queue.tryPop() — no subscription,
no notification, no wake-up, ever
cost per event: ~0.05–0.5 µs · burns: 100% of a core
The progression is a single trade surfacing three times: notification is cheap when idle and expensive when it fires; polling is expensive always and cheap when it matters. Blocking recv and epoll are push — webhook-style — and pay the wake-up tax (scheduler + cold cache, the largest single row in the table above) on exactly the packet you care about. Busy-polling is pull at maximum aggression: the core is always on, always warm, already looking. Your world optimizes for efficiency at rest because idle servers are wasted money; HFT optimizes for readiness at the burst because a core held spinning in reserve is ammunition — the fight arrives with no warning and is over in microseconds. (Linux offers a halfway house, SO_BUSY_POLL — the kernel busy-polls the driver ring for you while you wait — but serious hot paths go all the way: pinned core, spinning loop, kernel evicted.)
What you can now read
- Part I in full, starting from its opening premise — the ~2–10 µs kernel-path figure is now row-by-row auditable, and each Part I chapter is a named attack on specific rows: kernel bypass (rows 5–9), zero-copy (row 9), busy-polling (row 8), NIC/IRQ tuning (rows 5–6).
- The kernel-bypass framework chapters (DPDK, Onload/ef_vi, AF_XDP) — you can now place each on the elevator shaft: which levels it deletes, which it keeps, who drains the ring.
- The market-data chapters — UDP multicast feeds, gap detection, recovery channels: level 4 explained why reliability moved out of the kernel and into your feed handler.
- The colo and time-sync chapters — level 7’s serialization/propagation split is the vocabulary they assume.
- The Jargon Decoder (next) — every term this part introduced, plus the rest of the book’s vocabulary, in one lookup table.
How Computers Measure Themselves
This chapter is the vocabulary and mental model for everything in Part II — clocks through the measurement lab. The performance chapters assume you already know what a cycle is, what perf reads, why an average latency number is worthless, and what a flamegraph’s x-axis means. None of that is hard — it’s just never explained. Here it is, explained.
A recurring device in this chapter: the cycle-as-second scale you adopted in the machine chapter — one cycle ≈ 0.33 nanoseconds at ~3 GHz, and we pretend one cycle takes one second:
real time cycles human scale
───────── ────── ───────────
1 ns ~3 3 seconds
100 ns ~300 5 minutes
1 µs ~3,000 50 minutes
10 µs ~30,000 8 hours
1 ms ~3,000,000 5 weeks
35 ms (SG→Tokyo) ~105,000,000 3.3 years
Keep this table in your head. When the profiling chapter (ch09) says a syscall costs ~100 nanoseconds before it does any useful work, that’s the CPU taking a five-minute coffee break in the middle of your hot path.
1. How a computer knows what time it is
┌──────────────┐ ticks ┌──────────────────────┐
│ crystal │ ───────────► │ counters built on │
│ oscillator │ (a fixed │ the tick stream │
│ (a quartz │ frequency, │ │
│ tuning │ e.g. some │ ┌────────────────┐ │
│ fork) │ MHz base │ │ TSC: cycles │ │
└──────────────┘ clock) │ │ since boot │ │
│ └────────────────┘ │
│ ┌────────────────┐ │
│ │ wall clock: │ │
│ │ TSC + offset + │ │
│ │ NTP correction │ │
│ └────────────────┘ │
└──────────────────────┘
There is no magical “time” inside a computer. There is a crystal oscillator — a sliver of quartz that vibrates at a fixed frequency when you run current through it, exactly like the crystal in a quartz watch — and everything else is counting those vibrations. Every clock your code has ever read is an integer counter plus arithmetic.
-
TSC (Time Stamp Counter) — a 64-bit register inside the CPU that increments once per cycle since boot. — It’s the odometer of the CPU: it only goes up, it doesn’t know what year it is, and it’s absurdly cheap to read. — The instruction that reads it,
rdtsc, costs roughly 20 cycles — 20 seconds on our human scale, versus the five-minute coffee break of a syscall. — Trading systems timestamp with the TSC because when your whole budget is a few microseconds, you cannot spend hundreds of nanoseconds asking what time it is; the latency-methodology and observability chapters (ch08, ch11) assume TSC timestamping as the default. -
Monotonic clock vs wall clock — a monotonic clock only moves forward and measures durations; a wall clock tells you the calendar time and can jump backwards or forwards when it’s corrected. — This is exactly
performance.now()vsDate.now()in the browser/Node: you already know never to compute a duration from twoDate.now()calls, because NTP (below) can step the clock mid-measurement and give you a negative latency. — Same rule, one layer down: durations come from the TSC, timestamps for the audit log come from the wall clock, and confusing the two produces impossible latency numbers. -
vDSO (virtual Dynamic Shared Object) — a small page of kernel code mapped into every process so that “syscalls” like
clock_gettimerun as ordinary function calls, no kernel transition. — When you callDate.now()in Node orInstant::now()in Rust, you end up in the vDSO, which reads the TSC and applies the kernel’s calibration (cycles → nanoseconds, plus the NTP-corrected offset). No mode switch, ~20–30 ns. — This matters because it means timestamping can be cheap — and because profiling output in the profiling chapter (ch09) will show[vdso]frames and you should know they’re clock reads, not a bug.
So the stack is: quartz vibrates → CPU counts vibrations in the TSC → kernel calibrates TSC-ticks-per-nanosecond → vDSO exposes that as clock_gettime → your language runtime wraps that as Instant::now() / Date.now(). Every timestamp you’ve ever logged was this pipeline.
2. Why clocks disagree across machines
machine A machine B
┌─────────────┐ ┌─────────────┐
│ crystal: │ │ crystal: │
│ 3.000000 GHz│ │ 2.999994 GHz│ ← manufacturing spread +
│ (nominal) │ │ (2 ppm slow)│ temperature
└──────┬──────┘ └──────┬──────┘
│ drift apart ~2 µs every second │
▼ ▼
"event at 09:00:00.000100" "event at 09:00:00.002300"
│ │
└── which happened first? unknowable ──┘
Two crystals are never identical. Drift is measured in ppm (parts per million) — a 10 ppm crystal gains or loses 10 µs every second, which is ~1 second per day. Temperature changes the vibration frequency, so drift isn’t even constant. Left alone, two servers’ clocks walk away from each other indefinitely. Three technologies fight this, at three price points:
-
NTP (Network Time Protocol) — the standard internet protocol where your machine periodically asks time servers what time it is and slews its clock toward the answer. — It’s the default on every Linux box you’ve ever deployed; it’s what keeps your GCP VMs roughly on time. — Accuracy: milliseconds, sometimes tens of milliseconds, because the correction rides over ordinary software networking and assumes the path is symmetric. — For trading, NTP is not measurement-grade: a millisecond of clock error is a thousand times larger than the latencies you’re trying to measure.
-
PTP (Precision Time Protocol) — a time-sync protocol where the network hardware itself stamps sync packets at the moment they touch the wire, removing all the software-stack noise from the measurement. — Think of the difference between measuring an API’s latency from your app code (includes your event-loop lag, GC pauses, scheduler noise) versus having the load balancer stamp packets at ingress: PTP is the load-balancer version. — Accuracy: sub-microsecond, often tens of nanoseconds with good hardware. — Exchanges and regulators (MiFID II in Europe, for one) require trading timestamps synced at this grade; the clocks chapter (ch07) covers deploying it.
-
PHC (PTP Hardware Clock) — a real clock that lives on the network card itself, which PTP disciplines directly. — The NIC keeps its own clock instead of asking the CPU. — This is what makes hardware packet timestamping possible: the NIC stamps a packet with its clock at wire-touch time, and the clocks and latency-methodology chapters (ch07, ch08) lean on this for honest one-way latency numbers.
Why you care, in payments terms: you have debugged an incident by lining up your logs against Stripe’s dashboard timestamps, and you know the pain — if the two providers’ servers disagree by even a second, you cannot order the events; you can’t tell whether the webhook arrived before or after your retry fired. Now shrink that to trading: “our tick-to-trade is 200 µs” computed from a timestamp on machine A minus a timestamp on machine B is pure fiction if those machines’ clocks differ by 2 ms — the clock error is 10× the thing being measured. Cross-machine latency numbers are only as good as the clock sync underneath them, which is why an entire chapter about clocks (ch07) exists before any chapter about measuring latency.
3. What a profiler actually does
SAMPLING INSTRUMENTATION
───────── ───────────────
timer fires 999×/sec every function wrapped:
│ fn foo() {
▼ let t = now(); ← added
┌───────────────┐ ...real work...
│ interrupt! │ record(now()-t); ← added
│ what's on the │ }
│ stack RIGHT │
│ NOW? write it │ exact counts & times,
│ down. resume. │ but the measuring
└───────────────┘ changes the measured
statistical picture,
~zero distortion
-
Sampling profiler — a profiler that interrupts the program N times per second (say 999 Hz), records the current call stack, and resumes; after a few seconds, the statistics of those snapshots tell you where time goes. — This is exactly what the Chrome DevTools Performance tab does when you hit record, and exactly what Datadog’s continuous profiler does to your Node services in production: nobody rewrote your code, they just photographed the stack a few thousand times. — If
parse_orderappears in 40% of samples,parse_orderconsumes ~40% of CPU time. It’s polling, not tracing — statistically true, individually meaningless. -
Instrumentation — modifying the code (manually or automatically) so every function entry/exit records a timestamp. — This is you sprinkling
console.time()/console.timeEnd(), or an APM agent monkey-patching every HTTP and database call. — You get exact call counts and per-call durations… but the recording itself costs time, and for functions that run in nanoseconds, the measurement can cost more than the function. Heisenberg with a stopwatch.
Why sampling wins in production, and especially in trading: the observer effect. Wrapping a 50 ns function with two clock reads (~40 ns) doubles its cost and — worse — changes inlining, code layout, and cache behavior, so you’re now profiling a different program. A sampling profiler perturbs the target a few microseconds per second, statistically invisible. The rule the profiling chapter (ch09) assumes: sample to find where time goes; instrument only the specific boundaries you’ve already decided to care about (and the observability chapter (ch11) shows how trading systems instrument those boundaries cheaply). You already live by this rule — you reach for the APM flame view first and add custom spans second — this is the same discipline with better vocabulary.
4. The PMU: the APM agent baked into the silicon
your code runs ──► CPU core
┌──────────────────────────────────────┐
│ execution units │
│ │
│ PMU (a few hardware counters): │
│ ┌────────────────────────────┐ │
│ │ cycles 1,203,441 │ │
│ │ instructions 601,882 │ │
│ │ cache-misses 14,207 │ │
│ │ branch-misses 3,190 │ │
│ └────────────────────────────┘ │
└──────────────────────────────────────┘
▲
│ read by `perf stat`
-
PMU (Performance Monitoring Unit) — a small block of circuitry inside every CPU core containing a handful of programmable hardware performance counters: registers you can point at an event type (“count every cache miss”) and read later, at zero cost to the running code. — It is a Datadog agent implemented in silicon: always on, free to run, and it answers “what was my program actually doing?” one layer below any software. —
perf stat ./your-programis just: program the PMU counters, run the program, print the counters. The profiling and microbenchmark chapters (ch09, ch10) use this constantly. -
IPC (Instructions Per Cycle) — instructions retired divided by cycles elapsed: how much work the CPU completed per tick. — A modern core can retire 4–6 instructions per cycle when everything flows; think of it as throughput utilization for the silicon. — Rough field guide: IPC ≈ 0.5 means the core spent most cycles stalled, almost always waiting on memory (a cache miss is a trip to RAM: ~100 ns, ~300 cycles — a five-minute wait on our human scale, during which the core does nothing). IPC ≈ 3 means the pipeline is humming. — This is precisely the diagnosis you already make with APM: your endpoint is slow, the trace shows the handler spent 40% of wall time awaiting Postgres — the CPU wasn’t busy, it was waiting. IPC is that same busy-vs-waiting split, one layer down: not “my process waits on the DB” but “my instructions wait on RAM.” Trading systems obsess over it because the fix differs completely — low IPC means restructure your data for cache locality (the microbenchmarks chapter, ch10), high IPC with slow results means you’re doing too much work.
One habit to build now: when a hot loop is slow, your first question is no longer “what is it doing?” (profiler) but “is it computing or waiting?” (perf stat, look at IPC and cache misses). Two tools, two different questions.
5. Flamegraphs: you already read these
x-axis: fraction of samples (NOT time order — alphabetical merge!)
y-axis: stack depth (who called whom)
┌──────────────────────────────────────────────────────┐
│ main 100% │ ← +7% self
├───────────────────────────────┬──────────────────────┤
│ on_market_data 62% │ send_order 31% │
├───────────────┬───────────────┼──────────────────────┤
│ parse_msg 40% │ update_book │ encode_fix 29% │
├───────────────┤ 22% ├──────────────────────┤
│ memcpy 35% │ │ checksum 12% │
└───────────────┴───────────────┴──────────────────────┘
▲
width of "memcpy" = it was on-stack in 35% of samples
- Flamegraph — a visualization that merges thousands of sampled stacks into one picture: each box is a function, its width is the fraction of samples it appeared in (i.e., its share of CPU time), and boxes stack vertically by caller→callee. — You read these already: the React Profiler’s flame chart and the Chrome Performance tab are the same drawing. The one trap for people coming from Chrome: in Chrome’s timeline view, left-to-right is time order. In a classic flamegraph it is not — siblings are sorted alphabetically and merged, so left-to-right means nothing. Width is everything. — Diagnosis is the same skill you use on React renders: wide box you didn’t expect = the surprise cost; wide flat-topped box (nothing above it) = the leaf actually burning CPU; tall narrow towers = deep call chains that cost little. In the picture above, the actionable fact is
memcpyat 35% — a third of the CPU budget is copying bytes, which is why the microbenchmarks chapter’s zero-copy discussion (ch10) exists.
The profiling chapter (ch09) generates these from perf record; the reading skill transfers unchanged from your React profiler experience.
6. Percentiles, and why averages lie about latency
Ten payment authorizations, milliseconds:
20, 20, 21, 21, 22, 22, 23, 23, 24, 804
mean = 100 ms ← describes NONE of the ten requests
median = 22 ms ← describes the experience
max = 804 ms ← describes the incident
The mean says “typical request: 100 ms.” No request took anything like 100 ms — nine customers had a snappy 22 ms experience and one sat through 804 ms and possibly abandoned checkout. Latency distributions are skewed: there’s a floor (physics) but no ceiling (a GC pause, a lock, a page fault can stretch one request arbitrarily). The mean is dragged by the tail while describing nobody. This is why latency work speaks percentiles:
- Percentile — the value below which that fraction of samples fall: sort all samples ascending; p50 is the middle one, p99 is the value 99% of samples beat, p99.9 the value 99.9% beat. — In the dataset above p50 = 22 ms, p99 ≈ 804 ms. — You already run this instinct in payments: median auth time is a vanity metric; p99.9 is what actually trips a customer’s checkout timeout and loses the sale.
The web-vs-trading difference, and it changes everything about Part II’s measurement chapters: in a web system, one slow request is one mildly grumpy user — you manage p99 and shrug at max. In trading, one slow order is real money: the market moved while your order was in flight, and you got filled at a worse price (or an arbitrageur got there first — the industry phrase is adverse selection). The tail isn’t a quality metric, it’s a P&L line item. So trading latency reporting is p50 / p99 / p99.9 / p99.99 / max, and the max is read first, not last. When the latency-methodology chapter (ch08) spends pages on tail methodology, this is why.
7. Histograms: how you store a billion latencies
naive: keep every sample histogram: keep bucket counts
┌──────────────────────┐ ┌───────────────────────────┐
│ 1,000,000,000 × 8B │ │ bucket count │
│ = 8 GB, growing, │ │ [1.0–1.1 µs) 114,882 │
│ must sort to get │ │ [1.1–1.2 µs) 903,415 │
│ percentiles │ │ ... │
└──────────────────────┘ │ [95–100 ms) 1 │
│ fixed memory, percentile │
│ = walk buckets, O(1)-ish │
└───────────────────────────┘
-
Histogram (as a latency data structure) — instead of storing every sample, pre-define value ranges (buckets) and store one counter per bucket; recording a sample is
bucket[index]++, and any percentile is recovered by walking the buckets until you’ve passed the right fraction of the total count. — This is exactly what Prometheus histogram metrics and Datadog distribution metrics do internally — you’ve been consuming bucketed percentiles every time you read a dashboard; now you know why the p99 line looks slightly quantized. — Recording is nanoseconds and allocation-free, which matters when the recording happens on the hot path (the observability chapter, ch11). -
HdrHistogram (High Dynamic Range Histogram) — the standard implementation (Gil Tene’s), which spaces buckets logarithmically: fine-grained buckets at small values, coarser at large ones, maintaining a fixed relative precision (e.g. every recorded value within 0.1% of truth) across values from nanoseconds to minutes — six orders of magnitude — in a fixed few-hundred-KB footprint. — Same idea as logarithmic axes on your Grafana charts: equal ratios get equal resolution, because the difference between 1 µs and 2 µs matters as much as between 1 ms and 2 ms. — Every serious latency toolchain (and the methodology of ch08) speaks HdrHistogram natively; when you see
.hgrmfiles or “hiccup charts,” this is what’s underneath.
8. Coordinated omission: the interview filter
This one is worth over-learning — it’s a classic interview question in latency-sensitive shops precisely because it separates people who have thought about measurement from people who have run ab once. Learn it well enough to teach it back — here it comes.
The setup. Your load generator is closed-loop: send a request, wait for the response, then send the next. Target rate: one request per millisecond. The system under test hums along at 100 µs per response… then stalls completely for 100 ms (GC pause, lock, whatever). Ten seconds of test, 10,000 intended requests.
intended: req every 1 ms, no matter what
────────────────────────────────────────────────────────────►
t=0 t=4000ms t=4100ms
│ ││ ││ ││ │╳ ← stall begins │ ││ ││ │
│ │
what a closed-loop generator does: │
sends 1 request at t=4000, │
BLOCKS waiting for it, │
gets response at t=4100, │
records ONE sample: 100 ms │
...resumes as if nothing happened
what a real client population would have experienced:
req sent t=4000 → waited 100 ms
req sent t=4001 → waited 99 ms
req sent t=4002 → waited 98 ms
... 100 requests, waits 100,99,98,…,1 ms ...
req sent t=4099 → waited 1 ms
The generator coordinated with the system it was measuring: the moment the system got slow, the generator politely stopped sending. The stall — during which 100 requests should have been sent and would have queued — produced one bad sample instead of one hundred. The measurement omitted precisely the data from the period being measured. Hence: coordinated omission — the systematic under-counting of bad samples that happens when a blocked load generator (or any measurement loop) stops generating during the very stalls it exists to detect.
The numbers. Normal latency 0.1 ms; one 100 ms stall in a 10-second run at 1 req/ms:
| naive (coordinated omission) | corrected | |
|---|---|---|
| samples recorded | 9,901 fast + 1 slow = 9,902 | 9,900 fast + 100 slow = 10,000 |
| slow-sample share | 0.01% | 1% |
| p50 | 0.1 ms | 0.1 ms |
| p99 | 0.1 ms | ~1 ms |
| p99.9 | 0.1 ms | ~90 ms |
| max | 100 ms | 100 ms |
Read the p99.9 row twice. The naive report says “p99.9 = 0.1 ms” — three-nines excellence — for a system that went completely dark for a tenth of a second. The corrected report says p99.9 ≈ 90 ms, which is the truth: during that window, a real order (or a real checkout) sent at any point in the stall would have waited up to 100 ms. The max was never wrong — another reason trading reads max first — but every percentile between p50 and max was fabricated by the measurement methodology.
You have seen the production version of this bug: a webhook consumer falls over, your dashboard averages only the requests that completed, and the graph looks fine while a queue of unsent retries piles up out of frame. Same disease: measuring only what the sick system allowed to happen.
The fixes (detailed in the latency-methodology chapter, ch08): open-loop load generation — send on schedule from an independent timeline whether or not responses came back; or correct after the fact — when a sample exceeds the intended send interval, synthesize the missing samples (100 ms observed at 1 ms intervals ⇒ also record 99, 98, … 1 ms). HdrHistogram ships this correction built-in (recordValueWithExpectedInterval), which is not a coincidence — same author, same war.
The one-sentence version for interviews: “Coordinated omission is when a closed-loop load generator stops sending during a stall, so a 100 ms freeze that should contribute a hundred bad samples contributes one, and every percentile below max becomes fiction; you fix it with open-loop generation or expected-interval correction.”
9. Jitter: variance is the enemy, not slowness
low jitter (good even if slower): high jitter (worse even if "faster on average"):
µs µs
12 ┤ 12 ┤ ╷ ╷
10 ┤ 10 ┤ │ │
8 ┤──────────────────── 8 ┤ │ ╷ │
6 ┤ 6 ┤ ╷ │ ╷ │ │
4 ┤ 4 ┤───┴────┴───┴───┴────┴──
2 ┤ 2 ┤
└──────────────────── └─────────────────────────
every order: 8 µs usually 4 µs… sometimes 12
you can promise 8 µs you can promise nothing
- Jitter — variance in latency: the spread between your typical and your worst, as opposed to the typical itself. — It’s the difference between an API that always answers in 80 ms and one that usually answers in 40 ms but sometimes takes 400 ms — every retry policy, timeout, and SLA you’ve ever written was really about jitter, not speed. — Trading systems will happily trade a slower consistent path for a faster jittery one, because strategy decisions are priced assuming an execution latency; the orders that miss that assumption are the ones that lose money. Much of the kernel-tuning chapter (ch03) and the measurement chapters (ch10–ch12) is jitter hunting, not speed hunting.
The standard suspects, each expanded in later chapters — memorize the lineup, because “what causes latency jitter on Linux?” is another interview staple:
| suspect | what it does to you | your-world flavor |
|---|---|---|
| allocator | malloc usually takes ns; occasionally it takes a lock or asks the kernel for pages — µs to ms | GC pause in Node, but even without GC, allocation itself has a tail |
| page faults | first touch of a memory page traps into the kernel to wire it up | cold start / lazy-loading cost, at per-page granularity |
| scheduler | the kernel deschedules your thread to run something else; you’re simply not running for a while | noisy-neighbor pod stealing your CPU, at millisecond scale |
| interrupts | a NIC or timer yanks the CPU mid-function to run kernel handler code | your event loop blocked by someone else’s synchronous work |
| frequency scaling | the CPU changes clock speed (power saving, turbo) — and the CPU running your code is the meter timing it, so the meter itself speeds up and slows down | autoscaling flapping, except it’s the silicon flapping |
| thermal | chip runs hot → forcibly slows down | same, triggered by temperature |
| hyperthread neighbor | two hardware threads share one physical core’s execution units; a busy sibling halves your throughput unpredictably | two containers pinned to one vCPU |
The trading-systems countermeasures — pre-allocate everything, pre-fault memory, pin threads to isolated cores, steer interrupts elsewhere, lock the frequency — are all “remove a suspect from the lineup,” and they’re exactly what the kernel-tuning (ch03) and microbenchmark-hygiene (ch10) chapters do one suspect at a time.
What you can now read
- Clocks & time sync (ch07) — you know what the TSC, NTP/PTP/PHC, and monotonic-vs-wall distinctions are; that chapter is now deployment detail, not new concepts.
- Latency measurement methodology (ch08) — you can read percentile tables, HdrHistogram output, and coordinated-omission corrections without stopping.
- Linux profiling (ch09) —
perf statis “read the PMU,”perf recordis “sampling profiler,” flamegraphs are React profiler charts; the chapter is now tooling walkthrough. - Microbenchmarks (ch10) — IPC, cache misses, jitter suspects, and observer effect are the entire vocabulary of that chapter.
- Hot-path observability (ch11) — you know why recording is histograms-not-samples and why instrumentation must be nanosecond-cheap.
- Lab: measurement (ch12) — the lab has you produce every artifact this chapter described: a flamegraph, a
perf statreading, an HdrHistogram, and a coordinated-omission demonstration.
Exchanges, Sessions, and What’s Under a Database
Two vocabularies block Parts I–III. The first is traditional-finance market structure — you’ve built most of these concepts for crypto, but the book names them in tradfi terms and assumes you know the plumbing (feeds, FIX, colocation). The second is database internals — you drive Postgres expertly as an application developer, but the database and zero-downtime chapters (ch15, ch16) talk about it the way a DBA (database administrator) does: WAL, MVCC, replication modes, lock queues. This chapter maps both onto things you already own.
Part A — Market structure, tradfi edition
1. The venue stack: you built this already
traders / firms
│ orders in, executions out
▼
┌─────────────────────────────────────────────┐
│ EXCHANGE (the "venue") │
│ │
│ gateway ──► MATCHING ENGINE │
│ ┌───────────────────────────┐ │
│ │ ORDER BOOK for one symbol │ │
│ │ asks 100.02 │ 500 │ │
│ │ 100.01 │ 1,200 │ │
│ │ ──── spread ──── │ │
│ │ bids 100.00 │ 800 │ │
│ │ 99.99 │ 2,000 │ │
│ └───────────────────────────┘ │
│ │ │
│ ▼ │
│ market data publisher ──► everyone
└─────────────────────────────────────────────┘
Quick crypto→tradfi dictionary, because you built a matching engine and none of this is new — only the words are:
- Exchange / venue — the organization (and its machines) where orders meet; “venue” is the generic word because one instrument often trades on many of them. — Your Binance/exchange concept, except in tradfi one stock trades on a dozen venues simultaneously and firms must route between them. — Multi-venue is why “smart order routing” exists as a job title.
- Matching engine — the single-threaded-per-symbol core that matches incoming orders against the book. — You wrote one. Same object.
- Order book — the sorted resting orders per symbol, bids and asks. — Same object; tradfi says “symbol” where crypto says “pair,” and top of book / BBO (Best Bid and Offer) for the best prices.
- Price-time priority — the standard matching rule: better price wins; at equal price, earlier arrival wins — FIFO (first in, first out) per price level. — Identical to what you implemented. — The consequence is the whole reason this book exists: at equal price, queue position is won by latency. Arriving 1 µs earlier is the difference between being filled and watching.
2. Market data feeds: at-least-once delivery, but over UDP
EXCHANGE YOU
┌──────────────┐ incremental feed A ┌─────────────────────────┐
│ │ ═════════════════════► │ arbiter: │
│ publisher │ (UDP multicast, │ take whichever copy of │
│ │ seq: 101,102,103…) │ seq N arrives first, │
│ │ incremental feed B │ drop the duplicate │
│ │ ═════════════════════► │ │
│ │ (same data, second │ gap? (…103, 105…) │
│ │ network path) │ ├─ wait: B may have │
│ │ │ │ 104 in flight │
│ │ snapshot channel │ └─ else: resync from │
│ │ ─────────────────────► │ snapshot │
└──────────────┘ (periodic full book) └─────────────────────────┘
An exchange does not send you “the order book.” It sends a firehose of changes, and keeping a correct book is your problem. The machinery:
- Incremental feed — a stream of deltas (“add 500 @ 100.01”, “cancel order X”), each carrying a sequence number — a monotonically increasing per-stream counter stamped on every message. — This is a webhook event stream with an event ID, and your book is the projection you fold it into. — Sequence numbers exist because the transport (below) can drop messages, and a missed delta means your book is silently wrong — which in trading means quoting prices off a false picture.
- Snapshot channel — a slower side-channel broadcasting the full current book periodically, stamped with the sequence number it reflects. — It’s the
GET /current-statereconciliation endpoint you pair with any webhook integration: join by applying the snapshot, then replay buffered incrementals with higher sequence numbers. — This is how you bootstrap at startup and recover after falling behind. - Gap detection — noticing that you received seq 103 then 105: 104 is gone, your book can no longer be trusted until it’s recovered. — Same as detecting a missed webhook by a hole in event IDs, and the response is the same shape: backfill or resync. — Feed handlers treat a gap as an emergency: many strategies pull their quotes until the book is proven correct again.
- A/B feed arbitration — the exchange transmits the identical feed on two independent network paths (A and B); you listen to both, take whichever copy of each sequence number lands first, and use the other side to plug gaps. — It’s at-least-once delivery built from two unreliable channels plus dedup by idempotency key — the key being the sequence number. You’ve built exactly this discipline around webhook retries; here the “retry” is a redundant simulcast, because there’s no time to ask for a resend. — Bonus: taking the first arrival of each message shaves latency, since the faster path wins message by message.
Contrast with your crypto reality: a WebSocket book feed over TCP gives you ordering and retransmission for free — at the cost of TCP’s latency behaviors (the TCP/UDP chapter). Tradfi feeds choose the opposite trade: raw speed, and push reliability up to your application. That choice is why the transport and NIC chapters (ch02, ch05) exist.
- Multicast vs unicast — unicast is one sender to one receiver (every TCP connection, every HTTP call you’ve ever made); multicast is the sender transmitting once to a group address, and the network switches replicating the packet to every subscribed port in hardware. — It’s Redis pub/sub semantics, except no broker process exists: the fanout is done by the switch’s silicon, so the exchange sends each update exactly once whether 5 or 500 firms listen. — Every subscriber hears the message at nearly the same moment — a fairness property regulators care about — and it’s UDP underneath: no delivery guarantee, hence everything in the previous paragraphs. The packet-path and transport chapters (ch01, ch02) build on this.
3. FIX: the stateful session protocol
FIX SESSION (both sides persist counters across the wire)
YOU (seq out: 47) BROKER/EXCHANGE (seq out: 92)
│ │
│── Logon (my next out = 47) ────────►│
│◄────── Logon (my next out = 92) ────│
│── Heartbeat ──────────────────────► │ ...every N seconds...
│── NewOrderSingle seq 47 ─────► │
│◄───── ExecutionReport seq 92 ───────│
│── NewOrderSingle seq 48 ─────► │
│ ⚡ crash. restart. ⚡ │
│── Logon (my next out = 1) ─────────►│ ← WRONG: they expected 49
│◄──── ResendRequest / reject ────────│ session refuses to proceed
│ │
resume correctly = come back at 49,
answer their ResendRequest with either
real retransmits or a GapFill
- FIX (Financial Information eXchange) protocol — the decades-old standard wire protocol for order entry between firms, brokers, and exchanges: tag=value pairs (
35=Dmeans “new order”) over TCP. — Think “the SWIFT/ISO 8583 of trading”: ancient, verbose, and absolutely everywhere; every tradfi counterparty speaks it. — It splits into two layers, and the split is the important idea: - FIX session layer — the bookkeeping layer: logon/logout, heartbeats, and a sequence number on every message in each direction, persisted by both sides, surviving disconnects. — This is a Stripe-style event cursor made bidirectional and mandatory: each party tracks “the next number I’ll send” and “the next number I expect,” and a reconnect must resume exactly where it left off — like resuming a webhook stream from your stored cursor, except your counterparty also keeps a cursor on you. — The session layer is what makes FIX reliable over plain TCP: nothing is lost silently, because a hole in the numbers is detected immediately.
- FIX application layer — the business messages riding on top:
NewOrderSingle,ExecutionReport(fills),OrderCancelRequest. — These map one-to-one onto your matching-engine API surface. — Session mechanics are identical across counterparties; application dialects vary per venue. - Resend request / gap fill — on detecting a gap, a side sends
ResendRequest(from, to); the other side retransmits, except messages that shouldn’t be re-executed (heartbeats, and often stale orders) are replaced by aSequenceReset-GapFill— “pretend numbers 48–52 were administrative, skip ahead.” — This is your webhook backfill endpoint, plus a tombstone mechanism for events that must not be redelivered. — The dangerous part: retransmitted orders. A naive resend ofNewOrderSingleseq 48 after a crash could place a duplicate order with real money; FIX marks retransmitsPossDupFlag=Yand well-built engines treat them idempotently — the same reason you put idempotency keys on payment captures. - Why a restart is dangerous — sequence continuity is the session. Restart with the wrong counters and the counterparty either rejects your logon or fires resend traffic at you during the most fragile moment you have; meanwhile your orders may still be live at the exchange with nobody watching them. — It’s a stateful PSP integration where both sides track a message counter: you cannot “just reconnect,” you must resume at the agreed number or negotiate a reset. — Hence trading systems persist FIX sequence numbers with the same care you persist payment state, and “how do you recover a FIX session?” is a standard interview probe. The NIC chapter’s session-recovery material (ch05) and the event-sourcing chapter (ch13) both lean on this.
Your crypto reality, for contrast: WebSocket + REST (Representational State Transfer) + API keys. Disconnect → reconnect → re-authenticate → re-subscribe → re-snapshot everything, because the server keeps no cursor for you and guarantees no sequence continuity across connections. FIX’s statefulness is the price of never having to ask “wait, which of my orders are actually live?” — the question every crypto bot answers with a frantic REST burst after each reconnect.
4. Colocation: distance is time
RETAIL PATH COLOCATED PATH
your server (cloud, ~km away) your rack, INSIDE the
│ internet, ~ms exchange's datacenter
▼ │ "cross-connect": one
exchange DC ──► matching engine │ physical fiber patch
│ cable to their switch
▼ ~µs, fixed, no hops
matching engine
physics: light in fiber ≈ 200 km per millisecond (one way)
Singapore ↔ Tokyo ≈ 5,300 km straight-line, more over real cable
⇒ ~30+ ms each way, ~70 ms RTT — no software can fix geography
- Colocation (“colo”) — renting rack space in the exchange’s own datacenter and running your trading servers there. — It’s CDN-edge thinking applied to order flow: move the compute to where the event happens, because the speed of light is a hard budget. But where a CDN chases tens of milliseconds for humans, colo chases microseconds for machines. — At price-time priority (§1), the firm 5,300 km away has lost every race before it starts; you can measure this yourself — ping your Tokyo VM from Singapore and you’ll see ~70 ms RTT (round-trip time), which is mostly just fiber distance at 200 km/ms plus routing. That’s four orders of magnitude larger than the entire tick-to-trade budget of a colocated system.
- Cross-connect — the literal physical fiber cable patched from your colo rack to the exchange’s switch, ordered from the datacenter like a work ticket. — Think of it as a dedicated private link that replaces “the internet” entirely — no routers, no peering, no variance; the closest thing in your world is a VPC peering, made physical. — Latency becomes a fixed, tiny, known number, and some venues even normalize cable lengths so no rack gets a geometric advantage. Lab I (ch06) mirrors this setup at hobby scale.
5. Tick data and kdb+
- Tick data — the complete record of every market event — every trade, every quote change — at full resolution, timestamped; “tick” is tradfi for “one market data event.” — It’s your append-only events table for the market itself; a liquid symbol produces millions of rows a day, so the store lives at billions of rows. — Research, backtesting, and best-execution compliance all query it, and its natural shape drives the storage choice:
- Columnar tick store — storing each column (time, price, size…) contiguously rather than row by row, because analytical queries touch few columns across huge time ranges. — You know this trade-off from ClickHouse/QuestDB vs Postgres: scans over one column of a billion rows want columnar layout and vectorized execution. — Time-series market queries (“volume-weighted average price of every 1-minute window last quarter”) are the canonical columnar workload.
- kdb+ — the columnar, in-memory-plus-on-disk time-series database that dominates tradfi tick storage, programmed in q, a terse array language descended from APL (A Programming Language — yes, really). — Mental model: ClickHouse/QuestDB, but 25 years older, frequently faster on this exact workload, closed-source, expensive, and with a language where a production query can be 40 characters of punctuation. — It owns the niche because it was there first, banks standardized on it, and array-language operations map perfectly onto “fold over a billion ticks”; expect it named in tradfi job specs, and expect QuestDB/ClickHouse as its modern challengers.
6. Tick-to-trade: the metric the whole book optimizes
wire in wire out
│ ▲
▼ │
┌──────┐ ┌────────┐ ┌───────────┐ ┌──────────┐ ┌─────┴────┐
│ NIC │──►│ decode │──►│ update │──►│ strategy │──►│ encode + │
│ RX │ │ feed │ │ book │ │ decision │ │ NIC TX │
└──────┘ └────────┘ └───────────┘ └──────────┘ └──────────┘
└───────────────── tick-to-trade ───────────────────────┘
measured wire-to-wire (hardware timestamps; see the clocks
and latency-methodology chapters)
software systems: ~1–10 µs · FPGA systems: <1 µs
- Tick-to-trade — the elapsed time from a market data packet touching your NIC to your responding order leaving it, measured on the wire — not inside your process, where self-reported numbers flatter you. — It’s your end-to-end request latency SLO (service-level objective), except the clock starts at the network card, and it’s measured by hardware timestamping/tapping the wire rather than by APM spans. — This single number is the scoreboard for Parts I–II: Part I’s networking chapters attack the network legs, Part II’s measurement chapters make the number honest, and on the human scale from the measuring chapter, a 5 µs tick-to-trade is about 4 hours of single-cycle “seconds” — into which fits decoding, book update, decision, and encoding.
Part B — What’s under Postgres
You use Postgres the way you use Stripe: excellent command of the API, no need (until now) to know the machinery. The database and zero-downtime chapters (ch15, ch16) assume the machinery. Here it is: Postgres has been running your event-sourcing and double-entry-ledger tricks internally all along.
7. WAL: Postgres is event-sourced
COMMIT arrives
│
▼
1. append change record to WAL ──► fsync to disk ◄── THE durability moment
│ (sequential append: fast)
▼
2. tell client "committed"
│
▼
3. eventually, background writer updates the actual
table/index pages on disk (random writes: slow, unhurried)
crash between 2 and 3? → restart replays the WAL from the
last checkpoint; no committed data lost
- WAL (Write-Ahead Log) — an append-only file to which every change is written and fsynced before any table or index file is touched; commit = “it’s in the log,” and crash recovery = replay the log. — This is event sourcing, and you’ve built it twice: it’s your matching engine’s event journal, and it’s the payments append-only ledger — the log is the source of truth, tables are just a materialized projection maintained for convenient reads. — Sequential appends are the fastest thing a disk does, which is how Postgres commits fast while updating complex structures lazily. — Everything in the zero-downtime chapter (ch16) stands on this: replication is “ship the log,” PITR (point-in-time recovery) is “replay the log to timestamp T,” and CDC (change data capture) is “decode the log.” One structure, whole ecosystem.
8. MVCC: readers get a snapshot, and the corpses pile up
UPDATE accounts SET balance = 90 WHERE id = 7;
heap (table file):
┌──────────────────────────────────────────────────┐
│ row v1: id=7 balance=100 [xmin=500, xmax=612] │ ← old version stays!
│ row v2: id=7 balance=90 [xmin=612, xmax= ∅ ] │ ← new version appended
└──────────────────────────────────────────────────┘
txn 610 (started earlier) → still sees v1: its snapshot
txn 615 (started after) → sees v2
after nobody can see v1 → it's a DEAD TUPLE = bloat
VACUUM's job: reclaim it
- MVCC (Multi-Version Concurrency Control) — Postgres never updates a row in place; an
UPDATEwrites a new version and marks the old one as superseded, and every transaction reads the consistent snapshot of versions that were committed when it began — so readers never block writers and writers never block readers. — You’ve hand-rolled this pattern: an immutable ledger where “updating” a balance means appending a new entry, and a report reads “the ledger as of sequence N.” Postgres does it per row, with transaction IDs (xmin/xmaxabove) as the sequence numbers. — The costs fall out directly: dead row versions accumulate as bloat (tables physically larger than their live data), and VACUUM is the garbage collector that reclaims them. A long-running transaction pins an old snapshot, so VACUUM can’t clean anything newer — which is why one forgottenpsqlsession or stuck job can quietly balloon a busy table. The databases chapter (ch15) builds on this; it’s also why update-heavy trading schemas think hard before using Postgres for hot-path state.
9. Replication: shipping bytes vs shipping meaning
STREAMING (physical) LOGICAL
primary ──WAL bytes──► replica primary ──decode WAL──► row events
byte-identical copy "INSERT INTO orders VALUES(…)"
ALL databases, ALL tables │
same major version only ▼
replica is read-only any subscriber: newer-version PG,
~zero decode cost subset of tables, another system
- Streaming (physical) replication — the primary ships raw WAL bytes to replicas, which replay them continuously; the replica is a byte-for-byte identical copy of the entire cluster. — Analogy: restoring a binary disk snapshot, continuously — perfect fidelity, zero selectivity. — All-or-nothing and same-major-version-only (it’s literal page bytes), which is exactly why it cannot do a major-version upgrade — and that limitation is the setup for the zero-downtime chapter (ch16).
- Logical replication — the primary decodes the WAL back into row-level change events (“insert this row into
orders”) and publishes them; subscribers apply them as ordinary SQL. — This is a CDC stream — Debezium-into-Kafka energy — generated natively by Postgres from the same WAL. — Because subscribers apply meaning rather than bytes, they can be a different major version, take only some tables, or not be Postgres at all. — This is the mechanism behind zero-downtime major upgrades (the centerpiece of the zero-downtime chapter): stand up a new-version cluster, logically replicate until caught up, then switch traffic — the same expand/migrate/contract choreography you’d use to swap a payment provider without dropping transactions.
10. The lock queue: how a one-second migration takes the site down
The single most useful DBA fact for your interviews. The trap is not the lock — it’s the queue.
t=0 long analytics SELECT on orders [RUNNING, 40 min]
(holds ACCESS SHARE — the weakest lock)
t=10s ALTER TABLE orders ADD COLUMN risk_flag …
needs ACCESS EXCLUSIVE (conflicts with EVERYTHING,
even plain SELECTs) → must wait for the SELECT:
orders lock queue
┌───────────────────────────────┐
running → │ SELECT (analytics) │
waiting → │ ALTER TABLE ◄── would take 1s│
waiting → │ SELECT (app) ← blocked by │
waiting → │ INSERT (app) ← the WAITING │
waiting → │ SELECT (app) ← ALTER, not │
waiting → │ …every query… ← by the SELECT│
└───────────────────────────────┘
Locks queue FAIRLY: nobody may jump ahead of the
waiting ALTER. The whole app now waits on the
analytics query. Connections pile up. Site down.
- ACCESS EXCLUSIVE lock — the strongest table lock, required by most DDL (Data Definition Language —
ALTER TABLEand friends); it conflicts with every other use of the table, including reads. — It is a global read-and-write lock on the whole table: nothing touches it, not even a plain read, until the holder is done. — TheALTERitself is often metadata-only and takes a second — the lock, not the work, is the hazard. - The lock queue trap — Postgres grants locks in order: your DDL queues behind any long-running query, and because queueing is fair, every subsequent query queues behind your waiting DDL. A blocked one-second migration converts one slow analytics query into a full outage of the table. — It’s a head-of-line-blocking incident, the same shape as one stuck message freezing a FIFO queue — and you’ve likely felt this as “the deploy ran a migration and everything hung.” Now you know the mechanism. — The professional fix, verbatim for interviews:
SET lock_timeout = '2s';before DDL, so theALTERgives up rather than dam the queue, then retry in a loop at a quiet moment; plus the non-blocking variants —CREATE INDEX CONCURRENTLY, add columns without table rewrites,NOT VALIDconstraints validated later. The schema-evolution (ch14) and zero-downtime (ch16) chapters are applications of this one diagram.
11. Connection pooling: why pgbouncer exists
500 app connections pgbouncer Postgres
(each cheap to the app) ┌────────────┐ ┌────────────────┐
────────────────────────► │ multiplexer│───────►│ 25 backends │
────────────────────────► │ │───────►│ (each = a full │
────────────────────────► │ hands a │───────►│ OS PROCESS: │
… │ backend to │ │ MBs of memory,│
────────────────────────► │ whoever is │ │ fork cost, │
│ active NOW │ │ MVCC snapshot │
└────────────┘ │ bookkeeping) │
└────────────────┘
- Why Postgres connections are expensive — each connection is a forked operating-system process (not a thread, not a coroutine): megabytes of memory, real fork/teardown cost, and one more participant in shared bookkeeping (snapshots, locks) whose overhead grows with the crowd. — Node hands you sockets for near-free, so 500 idle connections feels normal; to Postgres, 500 processes is a genuine load before running a single query. — This is why every serious Postgres deployment fronts it with a pooler.
- pgbouncer — a small proxy that accepts thousands of cheap client connections and multiplexes them over a small pool of real Postgres connections. — You run this today: the
:5432on your multi-tenant GCP host is pgbouncer (500 client connections, 25 backends per database), with Postgres hidden on loopback:6432. — Two pooling modes, and the difference is exam material: - Session pooling — a client keeps one backend for its whole connection lifetime. — Like a dedicated phone line: safe, fully transparent, but a connected-and-idle client hogs a scarce backend. — Nothing breaks; little is saved.
- Transaction pooling — a client borrows a backend only for the duration of each transaction, then returns it; the next transaction may run on a different backend. — Like a stateless load balancer with no sticky sessions: massive multiplexing wins, but anything that assumes the same backend across transactions silently breaks. — The breakage list (know it cold): server-side prepared statements (
PREPARElives on backend A; your next transaction lands on B, which has never heard of it — the classic “prepared statement "s1" does not exist” error from ORMs), session state (SET,SET LOCALoutside the transaction, temp tables, session-scoped GUCs), advisory locks taken at session scope (the lock is held by a backend you no longer own — poisonous, since app-level advisory locking is a favorite pattern), andLISTEN/NOTIFY. — Trading and high-tenancy systems run transaction pooling for the multiplexing and design around the list; your own host is a working reference implementation to reason against when the databases chapter (ch15) discusses connection architecture.
What you can now read
- Event sourcing (ch13) — you now hold both halves: the matching-engine journal you built, and seeing that WAL is the same structure inside Postgres.
- Schema evolution (ch14) — the lock-queue diagram in §10 is the hazard that chapter’s every technique exists to avoid.
- Databases (ch15) — MVCC, VACUUM/bloat, WAL, and pooling modes are the assumed vocabulary; you have all four.
- Zero-downtime operations (ch16) — streaming-vs-logical replication (§9) is the entire mechanism; the chapter is choreography on top.
- Change management (ch17) and Lab: upgrade (ch18) — applications of §§9–10 with runbooks.
- TCP/UDP for trading (ch02) — the feed mechanics of §2 (multicast, sequence numbers, gaps, A/B arbitration) are the workload that chapter’s transport arguments are about.
- NIC internals (ch05) — §4’s cross-connects and §6’s wire-to-wire measurement are the context for why the NIC deserves its own chapter, and §3’s FIX session recovery is the state you’re protecting when hardware misbehaves.
Jargon Decoder
The lookup table for the whole book. Skim it once now; return whenever a later chapter drops a term. Grouped by theme; within a group, roughly the order you’ll meet things. “The analogy” column maps each term onto systems you already run.
1. The hardware
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| NIC | Network Interface Card | The device that turns wire signals into bytes in RAM and back | A sidecar container with its own CPU/memory that owns all network I/O | Everything in Part I; the most-tuned hardware in the rack |
| PHY | Physical layer transceiver | Analog electronics converting light/voltage into digital bits | TLS-terminating proxy, but it terminates physics | Fixed tens-of-ns cost; exotic links (microwave) compete here |
| MAC | Media Access Control layer | Digital logic framing bits into Ethernet frames, checking addresses + CRC | Body-parsing + signature-check middleware; drops garbage early | Hardware timestamping hooks here; MAC-dropped frames are invisible to software |
| PCIe | Peripheral Component Interconnect Express | The packet-switched network inside the box linking CPU/RAM to cards | The app-server↔database network, but in-chassis and ~1000× faster | ~500 ns–1 µs per device round trip; the latency floor after kernel bypass |
| DMA | Direct Memory Access | Devices read/write RAM directly; the CPU copies nothing | Presigned S3 upload — client writes straight to storage, server just gets notified | Why arrival is CPU-free; the enabling trick of kernel bypass |
| NUMA | Non-Uniform Memory Access | Each CPU socket has its own RAM; the other socket’s RAM is ~2× slower | Read replica in another region — same schema, hidden extra hop | Pin hot threads + memory to the NIC’s socket or eat ~50–100 ns per access |
| Cache line | — | The 64-byte unit in which memory actually moves | Postgres reads the whole 8 KB page, never one column | Struct layout; two hot variables per line = one fetch; two writers per line = disaster |
| MTU | Maximum Transmission Unit | Largest payload one Ethernet frame carries (default 1500 bytes) | Max request-body size — bigger payloads get chunked | Frame count per message; fragmentation is a latency tax |
| Jumbo frames | — | Raising MTU to ~9000 bytes | Batch API endpoint — fewer, bigger requests | Throughput plays (snapshots, recovery); irrelevant to small ticks |
| FPGA | Field-Programmable Gate Array | A chip whose circuitry you rewire to be your program | Compiling the hot path into custom silicon instead of running on general-purpose | The endgame: parse-and-respond on the NIC itself, sub-µs, skipping PCIe+CPU |
2. The kernel packet path
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| IRQ | Interrupt Request | Hardware forcing a core to stop and run a handler now | Webhook vs polling — the device pushes | ~1–2 µs delivery latency; also victimizes whatever thread was running |
| MSI-X | Message Signaled Interrupts, eXtended | Many independent interrupt vectors per device, each aimed at a chosen core | One webhook endpoint per event type, each with its own consumer | Enables per-queue → per-core steering; keeps IRQs off strategy cores |
| hardirq | Hard interrupt (top half) | The minimal urgent part: ack device, schedule follow-up, return | Webhook handler that enqueues a job and returns 200 in 2 ms | Runs with interrupts blocked; must be tiny |
| softirq | Soft interrupt (bottom half) | The deferred bulk work: drain ring, run the network stack | The queue worker draining what the webhook enqueued | Deferrable/migratable → classic source of p99.9 jitter |
| NAPI | New API (its real name) | Under load, kernel disables the interrupt and polls the ring in batches | Switching from per-message callbacks to batch-draining a queue under load | The kernel’s own admission that polling beats interrupts under load |
| sk_buff | Socket buffer | Kernel’s per-packet metadata object, allocated/freed per packet | Express req — one object per request, annotated by each middleware | Hundreds of ns of overhead per packet; bypass frameworks’ objects are ~a pointer |
| Descriptor ring | — | Fixed circular array in RAM where driver posts empty buffers and NIC fills them | Bounded SPSC queue: consumer pre-posts empty envelopes, producer fills | Overflow = silent drops (rx_missed); the exact interface bypass maps into your process |
| Page fault | — | CPU trap when code touches an unmapped memory page; kernel intervenes | ORM lazy loading — touching a field fires a query | µs+ landmine inside innocent code; pre-touch + mlock everything at boot |
| TLB | Translation Lookaside Buffer | Tiny cache of virtual→physical address translations | Route cache in front of a slow resolver | Miss = ~100 ns page-table walk before your real access starts |
| Hugepages | — | 2 MB / 1 GB memory pages instead of 4 KB, so the TLB covers far more | Connection pooling for address translation | Free tail insurance for big state; DPDK requires them |
3. Steering and offloads
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| RSS | Receive Side Scaling | NIC hashes each packet’s flow to pick one of several RX queues/cores | Consistent-hash sharding across consumers, in silicon | Per-flow ordering + multi-core spread; first steering knob you’ll touch |
| RPS / RFS / XPS | Receive/Flow/Transmit Packet Steering | Software versions: kernel bounces packet work to chosen cores (RFS: toward the consuming app; XPS: TX side) | App-layer load balancer when the hardware one (RSS) is missing or wrong | Fallback steering; an extra inter-core hop RSS avoids |
| Flow steering | — | Explicit NIC rules: “this port/multicast group → this exact queue” | Routing rules pinning one tenant’s traffic to a dedicated worker pool | Dedicate a queue+core to the feed that matters |
| GRO / LRO | Generic/Large Receive Offload | Kernel/NIC merges consecutive same-flow packets into one big one before processing | Batching webhook deliveries before handling | Throughput win, latency poison — off on hot paths (batching = waiting) |
| TSO | TCP Segmentation Offload | You hand the NIC one big buffer; it slices into MTU-sized frames | Chunked upload handled by the storage SDK, not your code | TX CPU saver; fine for bulk, irrelevant-to-harmful for small urgent sends |
| Multicast | — | One packet, delivered by the network to every subscribed host | Redis pub-sub, implemented by the switches themselves | How exchanges publish market data: every subscriber hears simultaneously |
| Unicast | — | Ordinary one-sender-one-receiver traffic | A normal HTTP call | Your order-entry path; contrast with multicast feeds |
4. Protocol and socket knobs
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| Nagle | Nagle’s algorithm | Kernel delays small TCP writes hoping to coalesce them into fewer packets | Auto-batching outbound webhooks to save requests | Adds up to ~40 ms (!) to small sends; the classic “why is my order slow” |
| TCP_NODELAY | — | Socket option that turns Nagle off: send every write immediately | flush: true — deliver now, efficiency be damned | Line one of every low-latency TCP setup |
| Delayed ACK | Delayed acknowledgment | Receiver waits (~40 ms max) hoping to piggyback the ACK on data | Batching read-receipts | Interacts pathologically with Nagle: request/response ping-pong stalls |
| RTO / dup-ACKs | Retransmission TimeOut / duplicate ACKs | How TCP notices loss: a pessimistic timer (~200 ms floor), or the receiver repeating “still waiting for byte N” — three repeats trigger immediate resend (“fast retransmit”) | Webhook retry with exponential backoff, unless the receiver actively reports the gap | Why one lost packet can freeze a TCP feed for 200 ms; fast retransmit needs traffic still flowing to generate the repeats |
| Receive window | TCP flow control | The receiver advertises how much buffer it has left; at zero the sender must stop sending | Stream backpressure — pause() until the consumer drains | How a slow consumer physically slows the sender; the mechanism behind every “slow subscriber” story |
| epoll | Event poll (Linux) | Register N sockets once; one blocking syscall returns whichever are ready | What libuv and tokio run on — your event loop’s engine | The C10K solution; its wake-up cost is what busy-polling deletes |
| SO_BUSY_POLL | Socket option: busy poll | Kernel spins on the driver ring for you instead of sleeping and waking | Tight queue.tryPop() loop, but the kernel runs it | Halfway house: µs savings without bypass frameworks |
| io_uring | I/O user ring | Submit and reap I/O via two shared-memory rings; syscalls optional | Job queue between you and the kernel replacing per-job RPC | Modern Linux async I/O; amortizes the syscall wall |
5. Kernel bypass
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| Kernel bypass | — | NIC DMA-writes packets into your process’s memory; kernel never sees them | Clients write directly to S3; your API server is out of the data path | The headline move: ~2–10 µs kernel path → ~1 µs |
| Zero-copy | — | Consume bytes where they landed; never duplicate them | Streaming a request body instead of buffering; sendfile | Deletes the copy in recv; pairs with bypass |
| DPDK | Data Plane Development Kit | Framework: unbind the NIC from the kernel; your app owns it, polling from userspace | Evicting Express and speaking raw TCP because the framework tax was the bottleneck | The industry-standard bypass toolkit; brings its own drivers + hugepage pools |
| PMD | Poll Mode Driver | DPDK’s userspace driver that spins on the ring; no interrupts, ever | while(true) tryPop() as a formal driver model | Why DPDK cores sit at 100% CPU by design |
| Onload | (Solarflare/AMD product) | Transparent bypass: LD_PRELOAD swaps the socket API’s guts for a userspace stack | Swapping the DB driver for a faster wire-compatible one; app code unchanged | Bypass without a rewrite — sockets API in, kernel out |
| ef_vi | (Solarflare API) | Onload’s raw layer: direct access to the NIC’s virtual interfaces, no sockets, no TCP | Dropping the ORM and the driver — hand-rolled wire protocol | Lowest-latency Solarflare path; you parse raw frames |
| XDP | eXpress Data Path | Run a small verified program (eBPF) inside the driver, at the earliest hook: drop/redirect/pass per packet | Middleware at the CDN edge instead of in the app | Filter/steer before any sk_buff exists |
| AF_XDP | Address Family XDP | Socket type where XDP redirects raw frames into your process’s ring | Kernel-sanctioned bypass — the escape hatch that’s still in the building | Bypass-lite: no vendor lock, kernel keeps coexisting |
| UMEM | User memory (AF_XDP’s buffer region) | The chunk of your process’s memory registered so the NIC can DMA into it | The shared S3 bucket both producer and consumer have keys to | Where AF_XDP packets physically land |
6. CPU isolation and tuning
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| Pinning / affinity | CPU affinity | Lock a thread to one core so its caches stay warm and it never migrates | Sticky sessions — same server every time, cache stays hot | Non-negotiable for every hot thread |
| isolcpus | Isolated CPUs (boot flag) | Remove cores from the scheduler entirely; only pinned threads run there | Dedicated instances vs shared tenancy | The strategy core shares with nothing |
| nohz_full | No HZ (tick-less) full | Stop the kernel’s periodic timer tick on chosen cores | Turning off a health-check that interrupts the worker every 4 ms | Deletes the last periodic ~µs of jitter on isolated cores |
| C-states | CPU idle states | Numbered sleep depths (C0 awake … C6 off); deeper = slower wake | Serverless cold starts, in silicon | Wake from deep sleep = tens of µs on the packet that mattered; HFT caps at C1 |
| Frequency scaling | P-states / governors | CPU clocks down when load looks light | Autoscaling that scales in right before the traffic spike | Pin the governor to performance; determinism over watts |
7. Time
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| TSC | Time Stamp Counter | Per-CPU register counting cycles since boot; read in ~10 ns | process.hrtime() if it cost nearly nothing | The only clock cheap enough for hot-path timestamps |
| rdtsc | Read TSC (the instruction) | The single instruction that reads the TSC | Calling hrtime() directly | How you instrument ns-scale code without perturbing it |
| NTP | Network Time Protocol | Classic clock sync over the network; ~ms accuracy | Cron-based reconciliation — fine daily, useless intraday | Too coarse for trading; regulation and measurement demand better |
| PTP | Precision Time Protocol | Hardware-assisted sync to ~sub-µs across machines | Distributed tracing with clocks good enough to order spans across hosts | Cross-machine latency measurement; regulatory timestamps (MiFID II) |
| PHC | PTP Hardware Clock | The clock chip on the NIC itself that PTP disciplines | The DB’s own now() vs your app server’s clock | Hardware timestamps come from this clock |
| SO_TIMESTAMPING | Socket option: timestamping | Ask for NIC-hardware timestamps on RX/TX packets | Trusting the load balancer’s access log over your app logger | Measures true wire-to-wire latency, excluding your own software’s lies |
8. Memory and CPU micro-architecture
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| False sharing | — | Two cores write different variables that share one 64-byte cache line; the line ping-pongs | Two services hammering the same DB row for unrelated columns | Silent 10–100× slowdown; fix = pad/align to 64 B |
| MESI | Modified/Exclusive/Shared/Invalid | The hardware protocol keeping all cores’ caches agreeing on each line | Cache-invalidation events between replicas, in silicon, per 64 bytes | The mechanism behind false sharing and atomic-op costs |
| CAS | Compare-And-Swap | Atomic instruction: “if it still equals X, set to Y” — the basis of lock-free code | Optimistic concurrency: UPDATE … WHERE version = 41 | Building block of every lock-free queue; ~20 ns, more under contention |
| Memory ordering | acquire / release / seq_cst | How much the CPU/compiler may reorder your reads/writes around an atomic | Read-your-writes vs eventual consistency, at nanosecond scale | Choosing correctly is the hard half of lock-free code; seq_cst is the safe-but-slower default |
| Lock-free vs wait-free | — | Lock-free: someone always progresses. Wait-free: everyone does, bounded steps | At-least-one-consumer-progresses vs per-request SLA | Queues on the hot path; wait-free = no thread can stall another |
| SPSC / MPSC | Single/Multi Producer, Single Consumer | Queue disciplines; fewer sides = far simpler and faster | One webhook source vs many, one worker draining | SPSC ring buffers are HFT’s workhorse pipe — the ring, again |
| IPC | Instructions Per Cycle | How many instructions the core actually retires per clock (typ. 0.5–4) | Requests/sec per worker — utilization vs stall | Low IPC = memory-stalled code; the first diagnosis number in perf |
| PMU | Performance Monitoring Unit | On-chip counters: cache misses, branch misses, stalls | Built-in APM (application performance monitoring) agent, in hardware, ~free | Where all real profiling data comes from |
| perf | (Linux tool) | Samples the PMU + stacks to show where cycles/misses go | The profiler tab, for native code, off the PMU | The daily driver for “why is this loop 400 ns not 80” |
| Flamegraph | — | Stack-trace samples rendered as stacked flames; width = time share | You know this one — same picture, now over CPU samples | Reading them below the runtime: kernel frames, not just your functions |
9. Measurement discipline
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| p99.9 | 99.9th percentile | The latency 1-in-1000 events exceed | Your p99 dashboards, one digit stricter | HFT lives in extreme tails: the bad tick is correlated with the valuable tick |
| Coordinated omission | — | Measuring only when the system deigns to respond, so stalls erase their own evidence | Uptime checker that skips checks while the site is down | The classic way benchmarks lie; send on schedule, count the waiting |
| HdrHistogram | High Dynamic Range Histogram | Records full latency distributions cheaply from ns to seconds | Prometheus histogram buckets minus the resolution lies at the tail | Standard tool; its docs are the coordinated-omission sermon |
| Tick-to-trade | — | Market-data packet hits your NIC → your order leaves it; the end-to-end number | Webhook-received → outbound-call-sent, measured at the wire both ends | The single metric the whole book optimizes; wire-timestamped, not app-logged |
| EWMA | Exponentially Weighted Moving Average | Running average that weights recent samples more; old data fades geometrically — one multiply per update | The rolling latency number on your dashboard, computed incrementally | How routers (and TCP itself) estimate RTT cheaply; the venue-health signal of the broker and mini-market chapters (ch26, ch28) |
10. Trading-system architecture
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| Colo | Colocation | Your servers racked in the exchange’s own data center | Deploying into the same AZ (availability zone) as the dependency — physically | Propagation delay is 5 ns/m; distance is bought, not optimized |
| Cross-connect | — | The literal dedicated fiber from your cage to the exchange’s, priced per metre | VPC peering, except it is an actual cable and metres are nanoseconds | The shortest permitted path to the matching engine |
| Feed handler | — | The component that parses the exchange’s raw feed into your book/events | The webhook-ingestion service: decode, validate, order, dedupe, fan out | The receive hot path; where bypass + parsing tricks concentrate |
| Gap fill | — | Detecting missed sequence numbers in a UDP feed and recovering them | Idempotency keys + replay for missed webhooks | UDP feeds drop; you own reliability now (level 4 of the Express-to-wire chapter) |
| Snapshot / recovery channel | — | Side channel serving current-state snapshots so late/gapped joiners can catch up | Full resync endpoint alongside the change stream — bootstrap then tail | Cold start and post-gap recovery for every feed handler |
| Sequencer | — | Single choke point stamping one global order on all events; everyone replays the same stream | Kafka single-partition total order, or your matching engine’s inbound queue | The determinism backbone of exchange-grade architectures |
| Event sourcing | — | State = fold(events); store the events, derive the state | Double-entry ledger: the journal is truth, balances are a view | Replayable, auditable, deterministic — natural fit downstream of a sequencer |
| Upcaster | — | Transformer that migrates old stored events to the current schema on read | API-version adapters for old webhook payloads | Evolving event-sourced systems without rewriting history |
| Kill switch | — | Pre-armed instant flatten-and-halt path, independent of the strategy | Circuit breaker + feature-kill flag, drilled and audited | Regulatory requirement and survival tool; must be faster than the thing it stops |
11. Orders and execution (the trader vocabulary)
Part V and the broker chapters speak this dialect constantly. None of it is hard — it’s the web-shop vocabulary of markets.
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| CLOB | Central Limit Order Book | The venue’s standing list of buy and sell orders, sorted by price, then by arrival time within a price | A sorted job queue per price level, matched first-come-first-served | “The book” — the data structure every venue chapter is about |
| L1 / L2 | Level 1 / Level 2 data | L1 = best bid and best ask only; L2 = the whole depth ladder, level by level | A summary endpoint vs the full table plus its change stream | The feed tiers venues sell; the market-data and mini-market chapters (ch25, ch28) build both |
| The touch | — | The best bid and best ask — the front of the line, where the next trade happens | The head of the queue | Queue position at the touch decides who fills first |
| Spread (tight / wide) | — | The gap between best bid and best ask. Tight = small gap (cheap to trade now); wide = big gap | The convenience fee for immediacy | “Deep” = lots of size resting behind the touch; venue quality in the mini-market lab (ch28) is exactly the tight-vs-deep trade-off |
| Tick size / lot size | — | The smallest price step / smallest quantity step a venue accepts | Prices in integer cents — no $10.001 allowed | Why hot-path prices are integers in ticks; floats never touch a price |
| Notional | — | Quantity × price — the total money at stake, ignoring direction | The cart total | Risk limits and caps are set in notional, not share counts |
| bps | Basis points | Hundredths of a percent; 100 bps = 1% | — | The unit fees and execution quality are quoted in (no relation to Gbps) |
| Parent / child order | — | Parent = the client’s whole order; children = the venue-sized slices a router cuts it into | One API request fanned out into N backend calls, results rolled back up | The SOR chapters; fills report child → parent |
| TIF (IOC / FOK) | Time In Force (Immediate-Or-Cancel / Fill-Or-Kill) | How long an order may rest. IOC: fill what you can right now, cancel the rest. FOK: fill it all right now or do nothing | Request timeout semantics — partial results accepted vs all-or-nothing | Flags every gateway must honor; venues differ in which they truly support |
| Post-only | — | An order that may only rest in the book (add liquidity); rejected if it would trade immediately on arrival | Insert-only write — abort on conflict instead of updating | Maker-fee strategies; the “rejects on cross” behavior in the mini-market lab (ch28) |
| Iceberg | — | A big resting order that shows only a small visible slice at a time; each fill reveals the next slice | Pagination — total count hidden from the client | Venue-side feature; the public feed only ever sees the tip |
| TWAP / VWAP / POV | Time-/Volume-Weighted Average Price, Percent Of Volume | Execution algos for working a big parent: drip it out evenly over time / proportional to when the market usually trades / never exceed X% of live volume | A rate-limited batch job: fixed rate / traffic-shaped / capped at a % of cluster load | What “algo selection” means in the broker chapters |
| OMS / SOR | Order Management System / Smart Order Router | OMS: the institution’s system of record for orders. SOR: the component that picks which venue gets each child | The CRUD backend of record + a load balancer with a cost model | The broker-side machine of the SOR and mini-market chapters (ch26, ch28) |
| Print / the tape | — | A print = one executed trade published on the public feed; the running public record is “the tape” | A row appearing in the public audit log | “Printing volume”, “the worst print of the day”, TCA benchmarks |
| Slippage / TCA | Transaction Cost Analysis | Score a fill against the mid-price at the moment the order arrived (“arrival mid”); the shortfall is slippage | Price drift between cart and receipt, measured and reported | How execution quality is judged; the payoff metric of the broker chapters (ch26, ch28) |
| Alpha | — | Information you can profitably trade on before the market prices it in | Knowing tomorrow’s traffic spike today | Why brokers must wall off client intent — seeing a client’s big buy is a signal (“alpha leak”) |
| Drop copy | — | A real-time duplicate stream of your own orders/fills, sent to risk and compliance systems | Mirroring prod events into the audit pipeline | Independent risk monitoring; regulator feeds (the venue and risk chapters — ch23, ch27) |
12. Data layer and deployment (the part you already half-know)
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| WAL | Write-Ahead Log | Append the intent durably before applying it; replay after a crash | You run Postgres; this is its heart — and Kafka is a WAL with an API | Trading journals/persistence reuse the pattern; append-only is also the fast path |
| MVCC | Multi-Version Concurrency Control | Writers make new versions; readers see a consistent snapshot; no read locks | Postgres again — why VACUUM exists | The same trick reappears in lock-free structures: readers never block |
| Logical replication | — | Ship decoded row-changes (not disk blocks) to subscribers | Postgres pub/sub of row deltas — CDC (change data capture) | Feeding analytics/risk systems off the trading DB without touching it |
| Expand-migrate-contract | — | Schema change in 3 deploys: add new alongside, migrate + dual-write, drop old | Your zero-downtime migration playbook | The only way to change a schema under a system that can’t stop |
| Blue-green | — | Two full environments; flip traffic atomically, flip back to roll back | You’ve done this | Deploying a trading system inside a maintenance window measured in seconds |
| Canary | — | New version takes a small traffic slice first, watched closely | You’ve done this too | For strategies: small size limits + tight risk rails before full capital |
| Shadow deploy | — | New version receives real input, its output compared but never acted on | Dark launch / dual-run diffing | The pattern for validating a rewritten hot path against the incumbent |
13. Wire formats
| Term | Expanded | Plain English | The analogy | Where it matters |
|---|---|---|---|---|
| FIX | Financial Information eXchange | The venerable text key=value protocol of institutional trading (35=D|55=AAPL|…) | JSON-over-HTTP of finance: verbose, universal, slow-ish | Order entry at most venues; parse cost is real |
| SBE | Simple Binary Encoding | Fixed-layout binary messages; fields at known offsets, zero parsing | Protobuf taken further: no varints, no decode step — cast the pointer and read | Modern feeds and gateways; decode in ~ns, and shorter messages serialize faster (level 7 of the Express-to-wire chapter) |
What you can now read
Everything. That’s this chapter’s job — it’s the index you return to, not a rung on the ladder. Concretely: Part I (machine + packet path: groups 1–6), the measurement chapters (groups 7 and 9 — read the coordinated-omission row twice), the concurrency chapters (group 8), and the architecture and operations chapters (groups 10–13, where more of your existing experience transfers than any other part of this book — group 11 is the trader dialect Part V speaks). When a later chapter uses a term that isn’t in this table and isn’t defined on the spot, that’s a bug in the book — flag it.
The Life of a Packet: Where Microseconds Go
Before you start. This chapter assumes NIC (Network Interface Card — the hardware that turns wire signals into bytes in your RAM, ch00a), PCIe (the bus connecting the NIC to the CPU/RAM, ch00a), DMA (the card writing straight into RAM without the CPU copying, ch00b), descriptor rings (the shared circular queues the NIC and driver hand buffers through, ch00b), interrupts / IRQ / softirq (how hardware gets the kernel’s attention, and the deferred half of handling it, ch00b), and syscalls / context switches (crossing the kernel↔userspace boundary and its cost, ch00b). If any are new, read those first — 20 minutes there saves an hour here. For the whole path in web-dev terms, ch00c traces an Express handler down to the wire.
You’ve built trading systems that respond in ~10 milliseconds. HFT systems respond in ~1–10 microseconds — a 1,000–10,000x difference. Almost none of that gap is closed by writing faster application code. It’s closed by understanding, and then eliminating, what the operating system does to every packet. This chapter is the anatomy lesson: every stage a packet crosses between the wire and your recv() call, and what each stage costs.
Hold one number in your head throughout: a competitive HFT tick-to-trade budget (market data in → order out) is under 5µs in software, under 1µs with FPGAs. The default Linux network stack spends 2–10µs just delivering a packet to userspace. The kernel path alone blows the entire budget. That’s the “why” behind everything in Part I.
The map
RX (wire → your code)
─────────────────────
wire
│ serialization + propagation
▼
┌───────────────┐
│ PHY / MAC │ decode symbols, check FCS
└──────┬────────┘
│ DMA write over PCIe
▼
┌───────────────┐
│ NIC RX ring │ descriptor ring in host RAM, pre-posted buffers
└──────┬────────┘
│ MSI-X interrupt (or: NAPI already polling)
▼
┌───────────────┐
│ IRQ handler │ hardirq: ack, schedule NAPI, return
└──────┬────────┘
│ softirq (NET_RX_SOFTIRQ)
▼
┌───────────────┐
│ NAPI poll │ driver pulls descriptors, builds sk_buffs
└──────┬────────┘
▼
┌───────────────┐
│ IP layer │ netfilter hooks, routing lookup, defrag
└──────┬────────┘
▼
┌───────────────┐
│ TCP / UDP │ 4-tuple demux, TCP state machine, ACK logic
└──────┬────────┘
▼
┌───────────────┐
│ socket queue │ sk_receive_queue, memory accounting
└──────┬────────┘
│ wake up sleeping task (epoll / blocking recv)
▼
┌───────────────┐
│ scheduler │ context switch onto your thread
└──────┬────────┘
│ syscall: recvmsg / recvfrom
▼
┌───────────────┐
│ copy_to_user │ kernel buffer → your buffer
└──────┬────────┘
▼
your code
Every box is work. Every arrow is a potential queue. Let’s walk it with numbers.
Stage by stage, with costs
Numbers below are order-of-magnitude for a modern x86 server (3–4 GHz, 10/25GbE NIC). Exact values vary; the ratios are what you need to internalize.
1. The wire: serialization and propagation
Before the host sees anything, physics taxes you:
- Serialization: putting bits on the wire. A 64-byte frame (+ preamble/IFG ≈ 84 bytes on wire) takes ~67ns at 10GbE, ~672ns at 1GbE, ~27ns at 25GbE. A 1500-byte frame at 10GbE: ~1.2µs. This is one reason HFT messages are tiny and why 25G/40G links matter even at low utilization — you pay serialization per hop.
- Propagation: ~5ns per meter in fiber (light travels at ~2/3 c in glass — ~200km per millisecond). Inside a colo hall this is tens of ns; across an ocean it’s tens of ms (the TCP/UDP chapter).
- Switches: a cut-through switch adds ~300–500ns port-to-port; a store-and-forward switch must buffer the whole frame first (add serialization time again). HFT-specialized L1 switches (Arista 7130, formerly Metamako) do ~5–50ns.
2. PHY/MAC and NIC processing
The NIC’s PHY decodes line symbols, the MAC validates the FCS (Frame Check Sequence — the CRC checksum trailing every frame; a bad one means the frame is silently dropped), and internal NIC logic parses headers to pick an RX queue (RSS — Receive Side Scaling, hashing flows across several queues so several cores can work in parallel; the NIC internals chapter). Budget ~300ns–1µs inside the NIC depending on the part. Low-latency NICs (Solarflare/AMD X2/X3, Mellanox/NVIDIA ConnectX) are engineered to keep this in the low hundreds of ns; commodity NICs and especially cloud virtual NICs are worse.
3. DMA to the RX ring
The driver has pre-posted descriptors — pointers to empty buffers in host RAM — into a circular descriptor ring. The NIC DMAs the frame into the next buffer over PCIe, then writes back the descriptor to mark it filled.
- PCIe Gen3/4 posted write (“posted” = fire-and-forget, the card doesn’t wait for an acknowledgement): the data lands in RAM (or L3 directly, via Intel DDIO — Data Direct I/O) in ~300–900ns.
- DDIO matters: without it, your first touch of the packet is a DRAM miss; with it, the payload is already in L3.
Up to here everything is hardware and unavoidable (bypass or not). Wire-to-RAM is roughly ~1µs. Everything after this point is what kernel bypass deletes.
4. Interrupt vs. NAPI
The NIC raises an MSI-X interrupt (Message Signaled Interrupts — the PCIe-native way for a card to poke a specific CPU core, ch00b) to say “descriptors ready.” Cost:
- Interrupt delivery + hardirq entry (hardirq = the immediate, drop-everything half of interrupt handling, ch00b): ~1–2µs by the time your handler runs — pipeline flush, mode switch, vector dispatch. On a busy or power-managed core, worse.
- Linux uses NAPI: the hardirq handler just disables further interrupts on that queue and schedules
NET_RX_SOFTIRQ. A softirq is deferred kernel work run right after the interrupt with interrupts re-enabled — the kernel’s “bottom half”, conceptually asetImmediatefor the expensive part of the job. The softirq then polls the ring, harvesting up to abudget(default 64 per NAPI instance, 300 globally per softirq round) of packets per pass. Under load, interrupts stay disabled and the kernel polls continuously — interrupt-per-packet would melt at 10G line rate (~14.8M packets/s at min frame size). - Softirq runs either on the tail of the hardirq or in
ksoftirqd(a kernel thread) if load is high — andksoftirqdis scheduled like any other thread, which adds scheduling jitter measured in tens of µs to ms in the worst case. This is a classic tail-latency source.
NAPI means the kernel already switches between interrupt-driven (idle, latency = IRQ cost) and polling (busy, latency = poll interval) modes. Kernel bypass just says: poll always, from userspace, on a dedicated core.
5. Driver + sk_buff allocation
The NAPI poll loop reads each filled descriptor and wraps the buffer in an sk_buff — the kernel’s ~200+ byte per-packet metadata object, roughly the framework Request object your web server allocates per HTTP request. Cost per packet: ~200–500ns for allocation (from the slab allocator — the kernel’s object pool, usually recycled per-CPU), initialization, and header parsing. GRO (Generic Receive Offload) may coalesce consecutive TCP segments into one super-skb here — great for throughput, adds latency (the kernel-tuning chapter).
6. IP layer and netfilter
ip_rcv(): sanity checks, then the netfilter hooks at PREROUTING and INPUT — the kernel’s packet-filtering framework, where iptables/nftables rules hang; think globally-registered Express middleware that every single request must walk, and even with empty rulesets the hook infrastructure costs. With conntrack loaded (connection tracking, the stateful table behind NAT and -m state rules), add a per-packet lookup: ~100ns–1µs and a lock hazard. Then a route lookup in the FIB (Forwarding Information Base, the kernel’s routing table — fast, tens of ns, cached) and possible defragmentation. Budget ~200ns–1µs. Rule of thumb from HFT ops: never load conntrack/iptables NAT on a trading box.
7. Transport: TCP/UDP demux and processing
- UDP: hash the 4-tuple (source IP, source port, destination IP, destination port — the four fields that identify one flow), find the socket, charge the packet to the socket’s receive buffer accounting (
SO_RCVBUF), append tosk_receive_queue. ~200–500ns. - TCP: much heavier — sequence number validation, reassembly ordering, ACK generation, RTT sampling, congestion window update, timestamp option processing. ~1–2µs per segment is typical. This is why market data is UDP (the TCP/UDP chapter).
8. Socket queue → wakeup → schedule
The packet is queued on the socket. If your thread is blocked in recv()/epoll_wait() (epoll — the readiness API that libuv/Node’s event loop sits on, ch00c), the kernel must wake it:
wake_up_interruptible→ scheduler enqueue → IPI (Inter-Processor Interrupt — one core poking another) to the target CPU if it’s idle/elsewhere → context switch into your thread. Cost: ~1–5µs, and highly variable. If the core was in a deep C-state (a CPU sleep level — deeper sleep, slower wake, ch00a), add tens of µs for wakeup (the kernel-tuning chapter). If another thread was running, add a full context switch (~1–3µs direct cost) plus indirect cost: your cache and TLB state has been trampled.- This wakeup is usually the single largest and most variable RX cost. It’s why serious latency work starts with “never sleep”: busy-poll the socket and the wakeup disappears.
blocked receiver spinning receiver
──────────────── ─────────────────
packet hits socket queue packet hits socket queue
→ wake_up_interruptible → next iteration of your
→ scheduler enqueue loop simply reads it
→ IPI to the target core
→ context switch in ≈ tens of ns, ~zero variance
→ caches/TLB now cold (cost: the core is 100% busy)
≈ 1–5µs, spiky (µs–ms tail)
9. Syscall + copy to userspace
Your recvfrom()/recvmsg():
- Syscall entry/exit: ~50–100ns bare; ~100–250ns+ with Spectre/Meltdown mitigations (KPTI, retpolines, IBRS — CPU-vulnerability workarounds that make every kernel entry more expensive). Measure on your hardware — Lab I does exactly this.
- copy_to_user (the kernel’s checked memcpy across the kernel/userspace boundary): from the sk_buff to your buffer. ~50–100ns for a 200-byte market data message (a few cachelines), ~200–400ns for a full 1500-byte frame. Small in isolation; at millions of packets/second it’s a core’s worth of memcpy.
- Then sk_buff free, socket accounting, return.
The RX bill
| Stage | Typical cost | Avoidable? |
|---|---|---|
| Serialization (64B @10G) | ~70ns | No (physics) |
| Switch hop (cut-through) | ~300–500ns | Buy better switch (~5ns L1) |
| NIC internal + DMA over PCIe | ~500ns–1µs | No (both paths pay it) |
| IRQ + softirq dispatch | ~1–3µs | Yes — poll instead |
| Driver + sk_buff | ~200–500ns | Yes — bypass |
| IP + netfilter | ~200ns–1µs | Yes — bypass |
| UDP processing | ~200–500ns | Yes — bypass |
| Wakeup + context switch | ~1–5µs (spiky) | Yes — busy-poll |
| Syscall + copy | ~150–500ns | Yes — mapped rings |
| Kernel total (wire→app) | ~2–10µs typical, ms-tail | This is the target |
A tuned kernel-bypass path (DPDK, ef_vi — the two main bypass frameworks, covered in the bypass-landscape chapter) delivers wire-to-app in ~1–2µs, dominated by the unavoidable hardware stages — the gap the rest of Part I closes.
The TX path (briefly)
Roughly the mirror image: send() syscall → copy_from_user into an sk_buff → TCP/UDP header build (TCP: congestion/flow-control gating — your send can be queued, not sent) → IP + netfilter OUTPUT → the qdisc (queueing discipline — the kernel’s per-interface outbound packet scheduler, default fq_codel/pfifo_fast; another queue, another lock). From there, the driver’s ndo_start_xmit posts a descriptor to the TX ring and writes the doorbell — one PCIe register write telling the card “new descriptors are waiting,” literally ringing a bell (~100–300ns, posted). The NIC then DMAs the frame and serializes it. App-to-wire through the kernel: ~2–5µs. Bypass TX: ~700ns–1.5µs, mostly the PCIe doorbell + DMA fetch.
TX has its own trap: the qdisc layer and TX ring are shared, so an unrelated bulk transfer (log shipping!) can queue ahead of your order. HFT practice: dedicated NIC (or at minimum a dedicated TX queue via XPS — Transmit Packet Steering, per-CPU choice of TX queue, the kernel-tuning chapter) for order traffic, and never share it with anything.
Interrupt coalescing: the throughput–latency dial
NICs batch interrupts: “fire at most every N µs, or after M packets” (ethtool -c/-C, rx-usecs/rx-frames). Defaults often sit at tens of µs or “adaptive” — tuned for throughput per CPU cycle, not latency.
rx-usecs = 50 rx-usecs = 0
pkt ─┐ pkt ── IRQ ── delivered (+~2µs)
pkt ─┤ (waiting...) pkt ── IRQ ── delivered (+~2µs)
pkt ─┴── IRQ ── batch ...
└─ up to 50µs added more IRQs, more CPU,
to first packet less latency
- Throughput box: coalesce aggressively — fewer interrupts, better cache behavior, higher packets-per-core.
- Latency box:
rx-usecs 0(or 1), adaptive moderation off. You accept a CPU-usage hit for determinism. - The mirror concept in bypass land: there are no interrupts at all; a core spins at 100% polling the ring. Coalescing is the kernel-world compromise; busy-polling is the refusal to compromise.
Why HFT cares: the three hidden killers
Beyond the per-stage bill, three systemic effects dominate tail latency — and in trading, the tail is where you get picked off:
- Context switches. Direct cost ~1–3µs, but the bigger cost is cache/TLB pollution: post-switch, your hot path runs at DRAM speed (~100ns/miss × hundreds of misses) until caches re-warm. A 1µs hot path can take 10µs+ on its first post-switch execution. Hence: pinned threads, isolated cores, nothing else schedulable there.
- Softirq interference.
NET_RX_SOFTIRQruns on whichever core took the interrupt. If that’s your trading core, your book-building thread gets preempted mid-update by someone else’s packets. Hence: IRQ affinity steering (the kernel-tuning chapter). - Cache pollution from the stack itself. Every kernel-processed packet drags sk_buffs, socket structs, netfilter tables, and kernel text through L1/L2 on your core. The stack doesn’t just take time — it evicts your data. Bypass paths keep the packet in a userspace-mapped ring your code touches directly, often still warm in L3 via DDIO.
The framing to carry into interviews: the kernel network stack is a general-purpose, multi-tenant, throughput-oriented system. It must be fair, secure, and correct for thousands of flows. You want a single-purpose, single-tenant, latency-oriented path for a handful of flows. Everything from kernel tuning to NIC internals (ch03–ch05) is either shrinking the general-purpose path (tuning) or replacing it (bypass).
Plain-English recap
If you remember nothing else from this chapter:
- The kernel network path is a deep middleware stack that every packet must walk — like an Express app where a dozen globally-registered middlewares run on every request whether you need them or not. Each is cheap; the stack of them costs 2–10µs, which is the entire HFT budget.
- The NIC and driver talk through a shared job queue (the descriptor ring): the driver pre-posts empty buffers like a worker pre-registering “I can take work here,” and the card DMAs packets into them without CPU involvement — a producer writing directly into your queue’s memory.
- Interrupts vs. polling is webhooks vs. polling loops. An interrupt (webhook) is great when idle but costs µs of dispatch per event; a poll loop notices new work in tens of ns. NAPI is a consumer that starts on webhooks and switches to draining the queue when busy. Bypass just says: poll always.
- The single biggest, spikiest cost is waking a sleeping thread (~1–5µs, sometimes tens of µs) — the cold-start problem. A blocked
recv()is a Lambda that scaled to zero; a busy-polling core is a warm instance that answers instantly but bills 100% CPU. - Syscalls and copies are like
JSON.parseper message: ~100–250ns each looks free, but at millions of packets/second it’s a whole core doing overhead instead of work. - Interrupt coalescing is webhook batching: deliver every 50µs in a batch and throughput improves — but the first event of a burst waits, and in trading the first event is the one that pays.
- Tail latency comes from state you lost, not work you did: a context switch trashes your caches the way restarting a Node process throws away its JIT-warmed state — the next request runs slow even though “nothing changed.”
Interviewer will ask
“Walk me through what happens when a packet arrives, from the wire to your application.” Tell it as three acts. Act one — hardware gets it into RAM (~1µs, unavoidable): the PHY/MAC decode the frame and check the FCS, then the NIC DMAs it over PCIe into a buffer the driver pre-posted on the descriptor ring — with DDIO it lands already in L3. Act two — the kernel processes it: the NIC raises an MSI-X interrupt; the hardirq does almost nothing except schedule the NET_RX softirq; the softirq polls the ring under a budget — that’s NAPI, interrupt-driven when idle, polling when busy — and wraps each buffer in an sk_buff. Then the middleware stack: IP layer and netfilter hooks, transport demux on the 4-tuple, onto the socket’s receive queue. Each of those stages exists so the kernel can be correct and fair for thousands of flows it knows nothing about. Act three — the kernel hands it over: wake the sleeping task, context-switch onto it, and your
recvmsgcopies the bytes to userspace. That handover is the biggest, spikiest line on the bill — ~1–5µs by itself. Total: 2–10µs wire-to-app, which alone blows a sub-5µs tick-to-trade budget — and that’s the whole case for Part I.
“Where does most of the latency actually go?” Not in any single stage — in the transitions: interrupt dispatch (~1–3µs) and especially the scheduler wakeup (~1–5µs, spiky, worse with C-states). Per-stage protocol processing is hundreds of ns each. So the biggest wins are “never sleep, never switch”: busy-polling and core isolation, before any bypass.
“Why is polling better than interrupts for latency? Isn’t polling wasteful?” Interrupt cost is ~1–3µs of dispatch plus scheduling variance; a poll loop notices a new descriptor in ~tens of ns with zero variance. Yes, you burn a core at 100%. In HFT the core is worth far less than the microseconds — a core costs a few hundred dollars; a microsecond of edge is revenue. Also note Linux itself agrees: NAPI converges to polling under load, and
SO_BUSY_POLLexists precisely to let sockets poll.
“What’s an sk_buff and why do bypass frameworks avoid it?” It’s the kernel’s per-packet
Requestobject — the ~200+ byte metadata struct a web framework allocates for every HTTP request, here tracking buffer geometry, headers, protocol state, and refcounts. The kernel needs that generality because any packet might be any protocol for any socket. But generality is billed per packet: ~200–500ns to allocate, initialize, and free, plus the cachelines it drags through L1 on your hot core. A bypass path serves one app on one known flow, so it can replace the Request object with a pre-allocated, fixed-layout buffer — DPDK’smbuf, ef_vi’s raw buffers — no allocation, no refcounts, no kernel bookkeeping. At 1M packets/s, that ~300ns of per-packet object overhead alone is ~30% of a core.
“What does interrupt coalescing do, and how would you set it on a trading box?” NIC delays interrupts to batch packets (
rx-usecs/rx-frames). Default/adaptive settings add up to tens of µs to the first packet of a burst — and the first packet is the one that moves markets. On a latency box:ethtool -C eth0 adaptive-rx off rx-usecs 0(or 1), accept the interrupt-rate cost, or better, make it moot by busy-polling.
“How much does a syscall cost, and why does it matter if it’s only ~100ns?” It’s the
JSON.parse-per-message problem: the cost is small, but it’s charged per message, and rate multiplies it. A syscall is ~50–100ns bare, ~100–250ns with the Spectre/Meltdown mitigations on. Every received message costs at least onerecvmsg— and a naive event loop paysepoll_waittoo. So run the chapter’s arithmetic: 150ns × 1,000,000 packets/s = 150ms of CPU per second — 15% of a core spent just crossing the kernel boundary, before any copy or actual work. That’s why the endgame is a ring mapped into userspace, where receiving a packet involves no syscall at all.
“Your p50 is fine but p99.9 is 100x worse. Name likely causes in the stack.” p50 is the happy path through the map; p99.9 is the same path after something made it sleep, queue, or go cold — the generator for every suspect I’d name. Slept: the core was in a deep C-state and paid tens of µs waking, or the receiver was blocked and paid the full wakeup-plus-context-switch. Queued: softirq work got pushed to
ksoftirqd, which is scheduled like any other thread, or the interrupt-coalescing timer held a lone packet waiting for a batch that never came. Went cold: a context switch trashed cache and TLB, so the first post-switch run of a 1µs hot path executes at DRAM speed — 10µs+. Tail debugging is hunting for whoever put a sleep, a queue, or a cache miss in the path; later chapters add more suspects to each verb (TCP retransmits in the TCP/UDP chapter, memory effects in Part I’s labs), and every knob in the kernel-tuning chapter is the removal of one of these.
Further reading
- “Monitoring and Tuning the Linux Networking Stack: Receiving Data” (packagecloud.io blog, Joe Damato) — the definitive line-by-line walkthrough of the Linux RX path; the TX companion post covers the send side.
- Brendan Gregg, Systems Performance (2nd ed.), Chapter 10: Network — costs, observability tooling (where you verify every number in this chapter yourself).
- LWN.net: “Driver porting: the network NAPI interface” and Jonathan Corbet’s follow-up NAPI/busy-polling articles — why the interrupt/poll hybrid exists.
- Cloudflare blog: “How to receive a million packets per second” (Marek Majkowski) — an empirical tour of exactly where the kernel stack drops packets and burns cycles.
- man 7 socket, man 7 tcp, man 7 udp — the actual tunables (SO_RCVBUF, SO_BUSY_POLL, TCP_NODELAY) you’ll use in the TCP/UDP and kernel-tuning chapters (ch02, ch03) and Lab I.
Where this goes next: TCP & UDP for Trading (ch02) — now that you know what a packet costs, which transport do you put on top, and why does every exchange send market data over UDP multicast but take orders over TCP?
TCP & UDP for Trading
Before you start. Two ideas from earlier carry this whole chapter: a socket (one network connection, handed to your code as a number — the thing
net.createServer()orfetch()gives you, ch00b), and the kernel’s socket buffers (small queues the kernel holds your bytes in between the wire and yourrecv(), ch00b). That’s it. Everything else gets built up slowly below.
The one mental model: mail vs a phone call
There are two ways to move bytes across a network. Everything in this chapter is a consequence of this one choice.
TCP = a phone call UDP = dropping postcards in a mailbox
───────────────────── ──────────────────────────────────────
• you dial, they pick up • you write a card, drop it, walk away
(a connection is established) (no connection, no handshake)
• words arrive in order • cards may arrive out of order
• nothing is lost — if they • some cards get lost, and nobody
didn't hear you, you repeat tells you
• but if the line crackles, you • but one lost card never holds up
both wait until it's clear the next card
-
TCP — the reliable one. A connection is set up first (like dialing and waiting for “hello”), then every byte you send arrives, in order, exactly once. If a piece goes missing, TCP quietly re-sends it before giving you anything after it. You already trust TCP for everything: every HTTP request, every database query, every
fetch()is TCP underneath. -
UDP — the fire-and-forget one. You hand the kernel a small message (a datagram — one self-contained packet, like one postcard) and it’s flung at the destination. No connection, no ordering promise, no “did it arrive?” Some get lost. The upside: because there’s no ordering promise, one lost message never delays the next one.
Hold that last line. It’s the whole reason trading uses both.
The trading split, in one sentence
Prices come OUT of an exchange over UDP. Orders go IN over TCP.
Why the split? Because the two directions want opposite things when something goes wrong.
-
Prices: the exchange is firehosing book updates to hundreds of firms. If one update gets lost, you do not want everything behind it to freeze while the exchange re-sends the old one — by the time it arrives it’s stale anyway. You’d rather have the newest price now and patch the one gap separately. That’s UDP’s “one lost card doesn’t block the next” behaviour — exactly what you want.
-
Orders: you are sending “buy 100” and “cancel that”. Here, losing a message, or getting it out of order (a cancel landing before the order it was cancelling), is a disaster. You’d happily wait an extra millisecond to be sure it arrives correctly. That’s TCP.
Your payments instinct already knows this: a live price ticker is like an analytics event stream — drop one, show the next, who cares. An order is like a charge request — it must be exactly-once and correctly ordered, and you’ll wait to guarantee that.
Part 1 — Prices: the UDP side
Why one-lost-card-doesn’t-block-the-next matters so much
Concretely:
TCP price feed (bad idea): packet 4 is lost
...③ ④✗ ⑤ ⑥ ⑦... → TCP holds ⑤⑥⑦ HOSTAGE until ④ is re-sent
you're now looking at a frozen, stale book
UDP price feed (good idea): packet 4 is lost
...③ ⑤ ⑥ ⑦... → ⑤⑥⑦ arrive NOW; you notice ④ is missing
and go get it separately (next section)
With TCP, one lost packet stalls everything after it until the re-send completes — milliseconds of staleness on a feed where microseconds matter. This is called head-of-line blocking (the item at the front of the line holds up everyone behind it — like one stuck request blocking a queue). UDP has none of it: each message stands alone.
“But UDP loses messages — how do you not lose data?”
Fair question: the exchange numbers every message (1, 2, 3, 4, …). You watch the numbers. If you were expecting #1004 and #1005 shows up, you know you missed one — and now you can go get it.
expected: 1004
arrives: 1005 → GAP. you missed 1004. two options to recover ↓
This is exactly the pattern you use in payments: webhooks are “at-least-once”, so you put a sequence number or cursor on each event, and if you notice a jump you call the “list events since X” endpoint to backfill. Same idea, different words.
The recovery options, cheapest first:
- The backup feed — the exchange sends the same numbered messages twice, over two different network paths (call them feed A and feed B). If A drops #1004, B almost certainly has it. You listen to both. (More on this in a second — it’s clever.)
- A “resend me #1004” request — a slower side channel where you ask for a specific missing range. Rate-limited, milliseconds.
- The snapshot channel — a separate stream that broadcasts the entire current order book every few seconds. If you’re badly behind, you wait for the next full snapshot, throw away your stale book, rebuild from the snapshot, and resume. This is the exact same thing crypto venues do: the REST “order book snapshot” endpoint plus the WebSocket “diff” stream. You’ve built this consumer already — only the transport underneath is different.
While you have a gap and haven’t recovered yet, your book is stale — you stop quoting on it. A junior forgets that step; a senior says it unprompted.
The backup-feed trick (A/B feeds)
The exchange publishes every message twice, on two independent network paths. You subscribe to both and, for each message number, use whichever copy arrives first and ignore the second.
feed A: ① ② ③ ④ ⑤
feed B: ① ② ✗ ④ ⑤ (B lost #3)
you: ① ② ③ ④ ⑤ (took #3 from A, everything else from whoever was first)
└─ result: faster (min of two paths) AND survives loss on either path
Two wins at once: you get the faster of the two paths every time, and you only lose data if both paths drop the same message (rare). It’s the same move as sending a critical webhook through two providers and deduping by idempotency key — belt and suspenders.
The crypto reality is different — and this is your edge
Everything above (UDP, one-send-many-receivers, A/B feeds) is the traditional-finance world. Crypto venues mostly don’t do it — they send prices over a separate WebSocket connection to each client (TCP, one per customer). That changes the game, and knowing why is an interview point:
- Traditional: one UDP broadcast, the network hardware copies it to everyone at the same instant. Fair and cheap no matter how many subscribers.
- Crypto: the venue maintains a separate TCP/WebSocket connection per client. 10,000 clients = 10,000 sends per update. Now a slow client is the venue’s problem — its connection backs up, and the venue has to decide whether to buffer, skip, or disconnect it. (You’ll build exactly this in the venue chapters — it’s the “slow consumer problem”, ch25.)
That’s the whole reason crypto feeds feel different from tradfi feeds: broadcast vs. per-customer connections. You lived on the crypto side; the tradfi side is the mirror.
Part 2 — Orders: the TCP side
Orders go over TCP because correct and in-order beats fast. A late order is annoying; a lost or reordered order is a reconciliation incident. Order traffic is also low-volume compared to the price firehose, so TCP’s costs are easily affordable.
What rides on top of TCP is just a message format. You only need to recognize the names:
- FIX — the old, universal one. Human-readable
tag=valuetext (like a URL query string). Easy to read, slowish to parse. - Binary formats (Nasdaq’s “OUCH”, CME’s “iLink”) — fixed-layout binary, so reading a message is basically casting bytes to a struct. They exist purely to skip the text-parsing cost.
- Your world — crypto venues use WebSocket/REST over TLS (i.e. TCP + encryption + a bit of framing). Structurally the same job as the tradfi session formats, plus encryption and JSON. When interviewing, translate out loud: “a venue’s WebSocket order channel is doing what OUCH-over-TCP does — a session with heartbeats and acks — just with TLS and JSON on top.”
The five TCP behaviours that actually bite
You do not need all of TCP. You need these five. Each gets a plain description, then the one-line fix.
1. The 40-millisecond stall (the famous one)
Two well-meaning “efficiency” features in TCP can lock together and freeze your small messages for ~40ms. Here’s the trap in plain terms:
- TCP has a feature that says “this message is tiny — let me wait a moment in case more data is coming, so I can send it all together.” (Its name is Nagle’s algorithm.)
- The receiving side has a feature that says “I just got data — let me wait a moment before acknowledging it, in case I’m about to send a reply I can piggyback the ack onto.” (Its name is delayed ACK.)
Now watch them deadlock: your side is waiting for an acknowledgement before sending the tiny order; their side is deliberately sitting on that acknowledgement. Both wait. The timer breaks the standoff after ~40ms.
you: "here's a 60-byte order" … (Nagle holds it, waiting for an ack)
them: (delayed-ack holds the ack, waiting for reply data to piggyback on)
⏳ … ~40ms … ⏳
them: timer fires → sends ack → your order finally goes. 40ms gone.
Fix: turn off the “wait to batch small messages” feature on every trading socket.
In Rust it’s literally one line — stream.set_nodelay(true). The option’s name is
TCP_NODELAY. There is never a reason to leave it on for trading. (This is the
missing-database-index of network code: a one-line default nobody notices, costing
1000x. Lab I has you measure the 40ms yourself.)
Bonus rule that follows: build each logical message in one buffer and send it with
one write(). Splitting a message into two writes (header, then body) re-invites the
same class of problem.
2. Socket buffers (the kernel’s queues)
Each socket has a small kernel queue on each side — bytes wait there between your code and the wire. Two things to know:
- Incoming prices (UDP): make the receive queue big. A burst of price updates in one busy microsecond can overflow a small queue, and overflow means silently dropped packets — which shows up later as mysterious feed gaps. Make it big, and monitor the overflow counter. This is the #1 real-world cause of “why did we gap?”
- Outgoing orders (TCP): a huge send queue can hide a problem — your
send()returns instantly while the data actually sits in the kernel aging. Some shops keep this queue small on purpose so backpressure is visible instead of hidden.
Analogy: it’s your job-queue depth. Too shallow and bursts overflow; too deep and you can’t see that you’re falling behind.
3. Congestion control — matters far, irrelevant near
TCP has logic that slows itself down when it thinks the network is congested. Whether this matters depends entirely on distance:
- Same building as the exchange (tradfi colo, sub-millisecond): the network is private and never congested, so this logic never kicks in. Ignore it.
- Across the world (crypto — Singapore to a venue in Tokyo, over the public internet): it matters a lot. A single lost packet makes TCP panic and throttle itself, and a stall can be hundreds of milliseconds. This is why serious crypto setups run several connections in parallel with failover, keep aggressive heartbeats, and place their servers in the same cloud region as the venue.
4. The idle-connection cold-start
One specific gotcha that makes a great interview answer: if an order connection sits idle for a bit and then you suddenly send a burst, that first burst can be weirdly slow. Reason: TCP “forgets” it had warmed up and resets itself to cautious-and-slow, exactly like a cold Lambda or a cold database connection.
Fix: keep the connection warm — send heartbeats so it’s never idle, and turn off the
Linux setting that re-colds idle connections (tcp_slow_start_after_idle). “Keep it
warm like a connection pool” is the whole idea.
5. Detecting a dead connection
TCP’s built-in “is the other side still there?” check defaults to two hours. Useless. In trading, a half-open connection — you think you’re connected and quoting, the venue thinks you’re gone — is how you end up with unhedged risk. So everyone relies on application-level heartbeats: FIX heartbeats, or WebSocket ping/pong, every 1–30 seconds with a hard timeout. If a few pings go unanswered, you assume dead and reconnect.
Part 3 — Crypto specifics (your interview home turf)
You lived here, so these are the points to make confidently.
Where the time actually goes on a crypto connection
Ordered biggest-to-smallest — reciting this order is the senior answer:
- Distance (milliseconds). Light in fiber travels ~200km per millisecond, and nothing beats it. Singapore↔Tokyo is ~70ms round trip no matter how good your code is.
- The venue’s own internal delay (milliseconds, worse during bursts) — their matching engine and gateways.
- Reconnect handshakes (multiple round-trips) — only when a connection drops and has to re-establish TCP + encryption + WebSocket + re-auth. This is why you pre-warm backup connections.
- JSON parsing (microseconds per message) — usually the biggest cost your own code pays, because most venues send JSON text, not binary. Serious shops use fast (SIMD) JSON parsers.
- Your own stack (microseconds) — the lock-free, zero-allocation stuff from the rest of this book.
Here’s the real interview trap: when the venue is 70ms away, shaving 5µs off your own code is pointless for taker orders (you crossing the spread to hit a resting price). But it’s decisive for maker orders (you resting a quote and racing other bots to cancel/replace when the book moves) within the venue’s own region, where everyone’s a few microseconds apart. Knowing which race you’re in is the answer.
taker vs maker, since it’s load-bearing above: a taker hits an existing resting order (crosses the spread, pays the fee, gets filled now); a maker posts a resting quote and waits to be hit (earns the spread/rebate, risks being picked off). See ch00f.
Where crypto venues actually live
“Colo” in crypto means same cloud region as the venue:
| Venue | Roughly lives in |
|---|---|
| Binance | AWS Tokyo |
| Bybit | AWS Singapore |
| Coinbase | AWS us-east-1 (Virginia) |
| Deribit | bare-metal, London |
So cross-venue arbitrage (say Binance-in-Tokyo vs Coinbase-in-Virginia) has an irreducible ~150ms+ information gap between the two — the strategy has to absorb the delay that infrastructure can’t remove. Multi-region setups run a quoting engine in each venue’s region and reconcile global risk between them asynchronously.
The 60-second recap
- Two ways to move bytes: TCP = a phone call (reliable, ordered, but a bad line makes you wait); UDP = postcards (some lost, out of order, but one lost card never blocks the next).
- Prices go UDP, orders go TCP — because for prices you want “newest now, patch gaps later,” and for orders you want “correct and in-order even if slower.”
- UDP doesn’t lose data in practice because every message is numbered; a jump in the numbers = a gap you go recover, from a backup feed / a resend request / a full snapshot. This is your payments webhook-with-a-cursor pattern.
- A/B feeds = the same stream sent twice over two paths; take whichever’s first. Faster and loss-proof.
- Crypto is different because it’s per-customer TCP/WebSocket instead of one UDP broadcast — which is why crypto has the “slow consumer” problem and tradfi doesn’t.
- The 40ms stall is the one TCP bug to know cold: two batching features deadlock; the
fix is
TCP_NODELAYon every socket, always. - Distance is the budget everything lives inside: cross-region is ~70–240ms no matter what; know whether you’re in a cross-region taker race (milliseconds, your code barely matters) or an in-region maker race (microseconds, your code is everything).
Interviewer will ask
“Why is market data UDP but order entry TCP?” Prices: you want the newest update now, and one lost packet must not freeze everything behind it — so UDP, with message numbers and a snapshot channel to patch gaps. Orders: losing or reordering a cancel is a disaster, and order volume is low, so you pay for TCP’s reliability and ordering. One line: prices tolerate loss but not staleness; orders tolerate slowness but not loss.
“You see a gap in the message numbers — walk me through recovery.” First check the backup (B) feed — usually has it. Else request the missing range on the resend channel. Else join the snapshot channel, wait for a full book snapshot at or past the gap, rebuild, and resume — buffering newer updates meanwhile and discarding the ones the snapshot already covers. The whole time, the book is marked stale and I stop quoting on it. Same shape as recovering a crypto WebSocket book from the REST snapshot.
“What’s the 40-millisecond stall?” Two batching features deadlock: my side holds a tiny message waiting for an ack; their side delays the ack waiting for reply data to piggyback on. A timer breaks it after ~40ms. Fix:
TCP_NODELAYon every trading socket, and one logical message perwrite(). I can measure it directly — Lab I does exactly that.
“What TCP tuning matters in colo vs cross-region crypto?” Colo: the LAN never congests and never loses packets, so the only enemies are TCP’s own batching timers and cold starts. Knobs:
TCP_NODELAY, keep connections warm (pre-open, heartbeat, disable idle-cold-start), sane dead-peer detection. Cross-region: distance makes loss expensive — one lost packet stalls the stream for hundreds of ms of retransmit round trip — so the game is surviving loss, not shaving µs. Knobs: redundant parallel connections with failover, aggressive heartbeats for half-open detection, pre-warmed backups — and accept the speed-of-light floor by placing servers in the venue’s region.
“Where does the latency go on a crypto venue connection, in order?” Distance (ms) → the venue’s internal delay (ms) → reconnect handshakes (round-trips, only on drops) → JSON parsing (µs) → my own stack (µs). Optimize in that order. The exception: in-region maker races are decided at the µs tier, so there my stack is the whole game.
“Why do crypto feeds feel different from traditional exchange feeds?” Traditional = one UDP broadcast the network copies to everyone simultaneously (fair, scales for free). Crypto = a separate TCP/WebSocket to each client, so the venue does N sends per update and now has a slow-consumer problem — it must buffer, conflate, or drop for clients that can’t keep up. Different fairness and different failure modes fall right out of that one design difference.
Further reading
- Stuart Cheshire, “It’s the Latency, Stupid” — the classic, plain-English essay on why more bandwidth never fixes latency. Read this first; it’s the mental model behind every distance number above.
- Nagle’s algorithm — the Wikipedia page plus John Nagle’s own (widely quoted) comments on why it and delayed-ACK should never have shipped together. Short and illuminating.
- A crypto venue’s WebSocket docs you already know (Binance or Coinbase market-data docs) — reread the “how to maintain a local order book” section and notice it’s the snapshot-plus-incremental recovery protocol from this chapter, in your own vocabulary.
- Nasdaq’s ITCH / OUCH overview — skim once to see the tradfi archetype: numbered UDP for prices, fixed-binary TCP for orders. You don’t need the field-level detail.
Where this goes next: Kernel Tuning Before Bypass (ch03) — you now know the path and the protocols; next, how far you can push a stock Linux kernel before reaching for anything exotic, and which knobs pay off first.
Kernel Tuning Before Bypass
Before you start. This chapter assumes cores and thread pinning (a CPU is several independent cores; pinning ties a thread to one, ch00a), C-states (CPU sleep levels — deeper sleep saves power but costs tens of µs to wake, ch00a), IRQ / softirq (hardware interrupts and the deferred kernel work they schedule, ch00b), NUMA (each CPU socket has its own local RAM; the other socket’s RAM is slower, ch00a), TLB and hugepages (the address-translation cache, and bigger pages that relieve it, ch00b), and busy-polling (spinning in a loop instead of blocking, in event-loop terms, ch00c). If any are new, read those first — 20 minutes there saves an hour here.
Reaching for DPDK (the take-over-the-NIC bypass framework, next chapter’s subject) before tuning the kernel is a junior move, and interviewers probe for it. A tuned kernel stack gets you from “10–100µs with millisecond tails” to “single-digit µs with tens-of-µs tails” — using standard sockets, standard tooling, standard debuggability. And the tails are the prize: the tick that arrives mid-burst, exactly when the kernel is busiest, is usually the tick carrying the price move you wanted to trade — the bad tick is the valuable tick (the packet-path chapter). For most crypto trading, where venue RTT is 1–70ms, that’s the finish line. This chapter is what a senior does first, roughly in order of return on effort.
The unifying principle: the enemy is not average cost, it’s variance. Nearly every knob below removes a source of “sometimes the kernel does something else”: sleeps, migrations, interrupts, frequency changes, memory faults. You are converting a general-purpose timesharing OS into something that leaves N cores alone.
Here is the whole chapter in one picture — the same millisecond on a trading core, before and after:
one millisecond on a trading core, before vs after tuning
─────────────────────────────────────────────────────────────────────
BEFORE ──[tick ~2µs]──[IRQ: disk]──[RCU callback ~30µs]──[C6 doze]──▶
scheduler's random kernel's deferred core sleeps;
1kHz beat device IRQ cleanup, any core wake ~100µs
▲
└─ your market-data tick lands HERE,
queued behind work it never asked for
AFTER ──────────── your pinned thread, spinning, alone ────────────▶
isolcpus: no tasks nohz_full: no tick rcu_nocbs: no callbacks
IRQs steered away C-states capped: the core never dozes
tick arrives ──▶ handled in ~1µs, every time
Every section below removes one of the BEFORE boxes.
1. Isolate the trading cores
Give your hot threads cores that the scheduler, timers, and kernel housekeeping leave alone. The first three knobs are kernel boot parameters (GRUB GRUB_CMDLINE_LINUX — settings baked into the boot line, so you can’t change them without editing the config and rebooting; the kernel’s equivalent of a redeploy):
isolcpus=2-7 nohz_full=2-7 rcu_nocbs=2-7
isolcpus: the scheduler will not place any task there unless explicitly affinitized (taskset/pthread_setaffinity_np). No random cron job, no kthread (kernel-owned background thread) load balancing onto your book-builder. Modern alternatives exist (cpusets/cset shield, systemdCPUAffinity=), butisolcpusis static, bulletproof, and still what most trading shops use.nohz_full: stops the ~1000Hz scheduler tick on those cores while a single task runs — removes a periodic ~1–5µs interruption (plus cache damage) every millisecond. Requires a tickless-capable kernel (standard now). One task per core, or the tick comes back — the tick exists to timeslice between tasks, so the moment a second runnable task appears the kernel has to resume the tick.rcu_nocbs: RCU (Read-Copy-Update) is a kernel synchronization scheme that defers its cleanup work — freeing old versions of shared data — into callbacks that run later, on whatever core is handy. Think of it as the kernel’s garbage collector: the pause lands wherever the collector happens to run, and without this flag “wherever” includes your trading core.rcu_nocbsmoves those callbacks onto housekeeping cores. RCU callbacks are a classic “why did my core stall for 30µs” answer.- Pin your threads explicitly (in Rust:
core_affinitycrate or rawsched_setaffinity), one hot thread per isolated core, and pin the memory too (below, NUMA).
2. Kill power management on those cores
C-states (sleep depth) and frequency scaling are the biggest single source of tail latency on an untuned box:
- Exiting C6 (the deepest common core sleep state) costs roughly 40–130µs on server parts. An idle-ish trading core that dozes between packets pays that on every wakeup. This alone can explain a p99 that’s 50x p50.
- Frequency:
powersave/schedutilgovernors (a governor is the kernel policy that decides each core’s clock speed) and idle cores mean your first µs of work runs at low clocks; ramp-up takes µs–ms.
# governor
cpupower frequency-set -g performance
# limit C-states (or boot params: intel_idle.max_cstate=1 processor.max_cstate=1)
cpupower idle-set -D 0 # disable all idle states deeper than C0/poll
# alternatively per-state: /sys/devices/system/cpu/cpuN/cpuidle/stateX/disable
- Or hold
/dev/cpu_dma_latencyopen with value 0 from your process. The open file works like a lease: while any process holds it, the kernel honors the latency cap; when the process exits, the file closes and the lease releases automatically — cleaner than boot params. This is whattuned(Red Hat’s tuning daemon, which applies named profiles likenetwork-latency) does internally. - Turbo: opinions differ. Turbo is the CPU running cores above their base frequency when thermal and power headroom allow — and that headroom is a budget shared by the whole chip. A neighbor core grinding through heavy AVX-512 (wide vector instructions) draws so much power that the chip down-clocks other cores — including yours — to stay inside the budget. So turbo means higher peak clocks but variable clocks. Many shops disable turbo for determinism and fix all cores at base or a known all-core frequency. The senior answer is “we pin frequency and measure, because variance costs more than mean.”
- If a core busy-polls (spins) 100% anyway, C-states never engage on it — one more argument for spinning.
- SMIs (System Management Interrupts): the one interrupter no kernel knob controls — firmware pauses every core to run BIOS code (fan control, memory scrubbing) for tens of µs, invisibly to the OS. Count them with
turbostat’s SMI column; the fix is in BIOS settings, not Linux.
3. Steer interrupts away, then steer the right flow in
Two-part game: (a) get all IRQs off trading cores; (b) get your NIC queue’s IRQ (if not busy-polling) onto the right core.
systemctl stop irqbalance && systemctl disable irqbalance # it will fight you
# find the NIC's queue IRQs
grep eth0 /proc/interrupts
# pin each queue IRQ: the value is a bitmask of allowed CPUs (bit N = CPU N)
echo 2 > /proc/irq/123/smp_affinity # 2 = 0b010 = CPU1 (queue 0 → CPU1)
echo 4 > /proc/irq/124/smp_affinity # 4 = 0b100 = CPU2 (queue 1 → CPU2)
# default for everything else: housekeeping cores only
echo 3 > /proc/irq/default_smp_affinity # 3 = 0b011 = CPUs 0+1
- RSS (hardware): NIC hashes flows across queues; each queue has its own IRQ → set queue count with
ethtool -L, pin as above. The NIC internals chapter covers steering a specific venue flow to a specific queue (ntuple/Flow Director) — the end state is “market data queue’s IRQ and consuming thread on the same core (or adjacent, sharing L3).” - RPS/RFS (software RSS): when the NIC can’t steer, the kernel does — the packet lands on whatever core took the interrupt, and the kernel then hands it to a different core to process. That hand-off is an extra inter-core hop, delivered by an IPI (inter-processor interrupt — one core poking another). RPS spreads packets across a chosen CPU set (
/sys/class/net/eth0/queues/rx-*/rps_cpus). RFS goes one step further and steers each flow toward the core where its consuming app actually runs (net.core.rps_sock_flow_entries). Useful when the NIC has too few queues (cloud!); on bare metal with good NICs, prefer hardware steering and leave RPS off — hardware puts the packet on the right core to begin with, no hop. - XPS:
/sys/class/net/eth0/queues/tx-*/xps_cpus— which TX queue each CPU uses. Each TX queue is guarded by a lock, so two cores sharing one queue serialize on it; give your order-send core its own TX queue and it never waits behind the logger.
4. Busy polling: stop sleeping
The single biggest kernel-path win (the packet-path chapter: wakeup = 1–5µs+, spiky). Three tiers:
- Kernel busy poll:
SO_BUSY_POLL(per-socket, µs value) or globallynet.core.busy_poll=50/net.core.busy_read=50. A blockingrecv/pollwill spin in the driver for up to that many µs — polling the NIC queue directly — before sleeping. Cuts the IRQ+softirq+wakeup chain out of the happy path. Typical gain: several µs off median, far more off tails. Cost: CPU on that core pegged during the window. - Userspace spin on nonblocking socket: loop on
recv(MSG_DONTWAIT)/try_recv. Still pays a syscall per attempt (~100–250ns each) but never sleeps, never gets woken — simplest deterministic option, and what Lab I demonstrates. - epoll + busy poll hybrid:
EPOLL_BUSY_POLL/epoll_pwaitwith busy_poll set — for many sockets on one thread.
The philosophical point for interviews: busy-waiting inverts the normal engineering instinct (“don’t spin, block”). In trading, blocking is the bug: sleeping costs wakeup latency, invites C-states, and cools your caches. Spin, and burn the core proudly.
5. NIC ring buffers, coalescing, offloads
ethtool -g eth0 # current/max ring sizes
ethtool -G eth0 rx 4096 tx 4096 # bigger rings: burst-loss insurance
ethtool -c eth0 # coalescing state
ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
ethtool -k eth0 # offload state
ethtool -K eth0 gro off lro off tso off gso off
- Rings (
-G): bigger = fewer overflow drops during microbursts, slightly worse cache locality and (if a queue actually builds) more queued latency. On a latency box the queue should be near-empty anyway; max the rings and treat any drop (ethtool -S | grep -i drop) as an incident. - Coalescing (
-C): the packet-path chapter. Zero it for latency; moot if busy-polling (no IRQs on the hot queue). - Offloads (
-K): GRO/LRO batch received TCP segments before delivering — deliberately adds latency to save CPU; TSO/GSO (GSO = Generic Segmentation Offload, TSO’s software cousin) batch on send, meaning your data can sit while a super-segment forms and hardware slices it. On latency-critical interfaces: off. Keep checksum offload on (rx/tx csum) — it’s free in NIC hardware and never delays delivery. Nuance for interviews: GRO-off can hurt if the box also handles bulk TCP (each segment now costs full stack traversal) — hence the real pattern: separate interfaces for trading vs. bulk traffic, tuned oppositely.
6. Memory: hugepages and NUMA
- Hugepages (ch00b): 2MB/1GB pages slash TLB pressure — a random walk over a multi-GB book with 4KB pages misses the TLB constantly, and each miss costs a page walk of ~tens of ns. Reserve at boot (
hugepagesz=1G hugepages=8orvm.nr_hugepages) and allocate viammap(MAP_HUGETLB)orhugetlbfs. DPDK requires hugepages outright (the bypass chapter). - THP (Transparent Huge Pages): the kernel promoting your 4KB pages to 2MB ones automatically. A kernel daemon,
khugepaged, rearranges your memory behind your back, and a page being moved is frozen mid-move — any thread that touches it stalls. Prefer explicit hugepages for the hot path, and set THP tomadvise— promotion happens only where you explicitly opt in withmadvise()— neveralways, on trading boxes. - Also:
mlockall(MCL_CURRENT|MCL_FUTURE)— a page fault on the hot path is a µs–ms catastrophe; lock everything, pre-fault at startup, disable swap. - NUMA (ch00a): the NIC hangs off one socket’s PCIe root complex (where that socket’s PCIe lanes originate) — so packets DMA into that socket’s RAM. Cross-socket delivery adds ~100–200ns per access, plus variance from the UPI link (the socket-to-socket interconnect).
- Keeping everything on the NIC’s node: find the node with
cat /sys/class/net/eth0/device/numa_node, then keep the IRQ core, the busy-polling thread, and all packet-touching memory there —numactl --cpunodebind=0 --membind=0 ./engine, or per-allocation withlibnuma. On dual-socket boxes, two placement defaults can silently cost double-digit percent latency: an interleave policy stripes pages across both sockets, and first-touch placement (Linux puts each page on the node of whichever core first writes it) means memory initialized by a thread on the wrong socket lives on the wrong side forever.
7. The checklist
Idempotent-ish setup block — annotate and adapt, don’t cargo-cult:
#!/usr/bin/env bash
# trading-node tuning — assumes: NIC=eth0 on NUMA node 0,
# housekeeping CPUs 0-1, trading CPUs 2-7 (isolcpus/nohz_full/rcu_nocbs set at boot)
set -euo pipefail
# power/clocks
cpupower frequency-set -g performance
cpupower idle-set -D 0 || true # cap C-states
# IRQs
systemctl stop irqbalance 2>/dev/null || true
echo 3 > /proc/irq/default_smp_affinity # 3 = 0b011: others → CPUs 0-1
for irq in $(grep eth0 /proc/interrupts | cut -d: -f1); do
echo 4 > /proc/irq/$irq/smp_affinity # 4 = 0b100: NIC queues → CPU2 (near consumer)
done
# NIC
ethtool -G eth0 rx 4096 tx 4096
ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
ethtool -K eth0 gro off lro off tso off gso off # keep csum offloads on
# sysctls
sysctl -w net.core.busy_poll=50 net.core.busy_read=50
sysctl -w net.core.rmem_max=134217728 net.core.wmem_max=134217728
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sysctl -w vm.swappiness=0
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
# verify
ethtool -S eth0 | grep -iE 'drop|err' ; nstat -az | grep -iE 'Rcvbuf|Drop' || true
Plus boot line: isolcpus=2-7 nohz_full=2-7 rcu_nocbs=2-7 intel_idle.max_cstate=1 mitigations=off hugepagesz=1G hugepages=8 — where mitigations=off (Spectre/Meltdown) buys back ~50–150ns per syscall and is defensible only on an isolated single-tenant box running fully trusted code; say the caveat out loud in interviews.
Then verify, don’t assume: perf stat (context-switches, migrations should be ~0 on hot cores), perf sched/bpftrace (tracing via small eBPF programs run safely inside the kernel) for stray wakeups, mpstat -P ALL (trading cores 100% user during busy-poll), ethtool -S and nstat for drops, and your own end-to-end histograms — every knob above should move a measured number or be reverted.
When tuned-kernel is enough — and when bypass pays
your latency venue RTT stack share
crypto, taker, x-region ~10µs ~70-200ms 0.01% ← tuning is already overkill
crypto, maker, in-region ~10µs ~0.5-2ms ~1-2% ← tuned kernel: right answer
crypto, top-tier in-AZ ~5µs ~100-500µs ~5% ← bypass starts to pay
tradfi colo maker/arb ~5µs ~10-100µs ~50% ← bypass mandatory
- Tuned kernel is enough when transit dominates: cross-region crypto (RTT ms+), or any strategy whose alpha horizon is ms+. You keep: normal sockets, tcpdump/eBPF observability, TLS libraries that just work, no dedicated-core tax, portability to cloud VMs where bypass is impossible or crippled. Most crypto trading, including profitable market making, falls in this band — a well-built tuned-kernel stack at ~5–20µs app latency is simply not the bottleneck when the venue’s own gateway jitter is hundreds of µs.
- Bypass pays when you’re in the same building/AZ as the matching engine, competitors are at 1–5µs, and the race is per-event (queue position on cancel/replace, cross-venue arb in one metro). Then the kernel’s 2–10µs and its tails are the biggest remaining term, and the bypass chapter is your menu.
- The interview frame: “I’d tune first, measure, and only bypass when the measured kernel share of my end-to-end budget justifies owning a userspace stack.” That sentence, with the table above behind it, is exactly the judgment call being tested.
Plain-English recap
If you remember nothing else from this chapter:
- The enemy is variance, not the mean — same instinct as chasing p99 API latency instead of average response time. Every knob here removes one way the OS can “sometimes do something else” on your core.
- Core isolation is a dedicated worker instance for your hot loop:
isolcpus/nohz_full/rcu_nocbsgive a thread a core where no cron job, no other pod, no kernel housekeeping ever gets scheduled — like moving batch jobs off the production database. - C-states are cold starts. A dozing core pays 40–130µs to wake, which is why p99 can be 50x p50 while p50 looks perfect. Disabling C-states (or spinning, which never lets the core sleep) is keeping the Lambda warm.
- IRQ steering is traffic routing: all interrupts off the trading cores, your hot flow’s interrupt onto the core that consumes it.
irqbalanceis an autoscaler that will fight your manual placement — kill it. - Busy-polling inverts the async instinct. In Node you’d never
while(true) tryRecv()— here, blocking is the bug: sleeping costs a wakeup, invites C-states, and cools your caches. Spin and burn the core; the core is cheaper than the microseconds. - NIC offloads and coalescing are batching: great for throughput (fewer interrupts, bigger chunks), bad for the first message of a burst. Latency interface: everything off. Bulk interface: everything on. Don’t share.
- Memory: no surprises on the hot path — hugepages so address translation stays cached,
mlockallso a page fault (a µs–ms stall, like a cache miss to disk) can’t happen mid-trade, and everything on the NIC’s NUMA node so packets don’t land in the far socket’s RAM. - Tune, then measure, then only maybe bypass: if venue RTT is milliseconds (most crypto), a tuned kernel at ~5–20µs is already not your bottleneck — spend the effort on placement and connections instead.
Interviewer will ask
“You get a fresh Linux box for a latency-sensitive service. First five things?” One principle generates the whole list: remove every way the OS sometimes does something else on my cores — scheduling, sleeping, interrupting, batching. (1) Scheduling: pin hot threads to isolated cores (
isolcpus/cpusets) on the NIC’s NUMA node, so no other task can ever land there. (2) Sleeping: performance governor plus capped C-states, so a core never pays tens of µs waking up. (3) Interrupting: IRQ affinity — everything off the trading cores, the NIC queue’s IRQ next to its consumer,irqbalancedead so it can’t fight me. (4) Batching in the sockets:TCP_NODELAYand busy-poll, so no timer ever holds my message. (5) Batching in the NIC: coalescing to zero, GRO/LRO/TSO off, rings maxed. Then measure — histograms,perf, drop counters — because every knob must move a number or be reverted.
“What do isolcpus, nohz_full, and rcu_nocbs each actually do?”
isolcpus: the scheduler won’t place unpinned tasks on those cores — because a random cron job or kthread landing there would preempt the hot loop and trash its caches.nohz_full: kills the periodic scheduler tick while one runnable task owns the core — because the tick only exists to timeslice between tasks, so with one task it’s a pointless ~1–5µs interruption a thousand times a second.rcu_nocbs: moves RCU callbacks — the kernel’s deferred garbage-collection, whose pause otherwise lands on whatever core is handy — onto housekeeping cores; RCU is the classic answer to “why did my core stall for 30µs.” Together: a core where only my pinned thread runs — and I steer the residual hardware IRQs away too.
“Why do C-states destroy tail latency, and what’s the tradeoff of disabling them?” Deep C-state exit costs tens of µs (C6 ~40–130µs) — an idle-ish core pays it per wakeup, so p99 explodes while p50 looks fine. Disabling burns power/heat and forfeits turbo headroom that C-states free up for other cores. On a spinning core it’s moot — the spin holds C0 — which is one more reason busy-poll designs are self-consistent.
“Explain SO_BUSY_POLL. How is it different from your app spinning on a nonblocking recv?” SO_BUSY_POLL makes the kernel spin in the device driver’s poll routine during a blocking receive — it polls the NIC queue itself, bypassing the IRQ→softirq→wakeup chain, falling back to sleep after the budget. App-level spin never sleeps but pays a full syscall per probe and only checks the socket queue (the packet still traversed softirq to get there). Busy poll shortcuts delivery; app spin only shortcuts the wakeup. They compose.
“When would you leave GRO on?” When the interface also carries throughput-oriented TCP (backups, log shipping, snapshots) — per-segment processing without GRO can saturate a core and cause drops, which is worse for latency than coalescing. Clean answer: separate interfaces — trading NIC with everything off, bulk NIC with offloads on. If forced to share, GRO on + accept the µs-level batching, or flow-steer trading traffic to queues you handle differently.
“Your p99.9 wire-to-app is 80µs on a tuned box, p50 is 4µs. Debug it.” Walk the BEFORE diagram from this chapter and ask: which box didn’t I remove? The C-state box first — a C6 wake is 40–130µs, which fits an 80µs tail perfectly — check
cpupower idle-infoand turbostat residency. The stray-task box: something landed on the core and brought the scheduler tick back with it —perf sched record, context-switch counters should be ~0. The RCU box: isrcu_nocbsactually on the boot line? The random-IRQ box:grepthe core’s column in/proc/interrupts— anything counting up shouldn’t be. Two suspects live outside the diagram: a page fault mid-path (mlockalldone? THP set tomadvise, notalways?) and the NIC’s coalescing timer holding a lone packet. And if every box checks clean, it’s the one interrupter no kernel knob controls: SMIs — firmware pausing every core for tens of µs, invisible to the OS;turbostat’s SMI column counts them, the fix is BIOS. Method throughout: bpftrace on scheduler and IRQ tracepoints around the bad samples, not guessing.
“Why might you NOT deploy all this on a crypto trading box in AWS?” Because the budget doesn’t warrant the ops cost and the platform blunts the tools: venue RTT is ms-scale so stack µs are noise for most flows; on VMs you don’t control C-states/SMIs, isolcpus is weaker under a hypervisor scheduler, and ENA/gVNIC limit queue/IRQ/coalescing control (the NIC internals chapter). I’d still do the cheap, robust subset — thread pinning, NODELAY, buffers, busy-poll, governor — and spend the effort on placement (same AZ as venue) and connection management, where the real ms live.
Further reading
- Erik Rigtorp, “Low Latency Tuning Guide” (rigtorp.se) — the community-standard checklist for exactly this chapter; also his articles on hugepages and spinlock/membench measurements.
- Red Hat “Low Latency Performance Tuning” guides /
tunedprofiles (network-latency, realtime) — read the profile source to see what a vendor thinks belongs in this list; even if you hand-roll, it’s a good diff-base. - Linux kernel docs:
Documentation/networking/scaling.rst— RSS, RPS, RFS, XPS from the source; andDocumentation/timers/no_hz.rstfor nohz_full’s fine print. - LWN.net: “Low-latency Ethernet device polling” (Jonathan Corbet) — origin and mechanics of SO_BUSY_POLL; see also LWN’s later coverage of NAPI busy polling and per-queue configs.
- Brendan Gregg, Systems Performance (2nd ed.), chapters 6 (CPUs) and 10 (Network) — the observability half: proving which knob moved which number.
Where this goes next: The Bypass Landscape (ch04) — when tuning isn’t enough, what does replacing the kernel path actually look like, and which framework (DPDK, Onload, ef_vi, AF_XDP, io_uring) fits which job?
The Bypass Landscape
Before you start. This chapter assumes descriptor rings (the shared circular queues the NIC and driver exchange buffers through, ch00b), DMA (the card reading/writing RAM directly, no CPU copy, ch00b), sk_buff (the kernel’s per-packet metadata object, ch00b), syscalls (the cost of crossing into the kernel, ch00b), hugepages / TLB (big memory pages that keep address translation cached, ch00b), and busy-polling (spinning instead of blocking, ch00c). If any are new, read those first — 20 minutes there saves an hour here.
The kernel-tuning chapter shrank the kernel path. This chapter replaces it.
The mailroom, and the desk
Hold one picture for the whole chapter.
Your process is an office worker. The NIC is the building’s loading dock, where packages (packets) physically arrive. Between the dock and your desk sits the kernel: the building’s mailroom. Every arriving package gets logged (sk_buff allocated), sorted (protocol stack), placed in a pigeonhole (socket buffer), and then someone walks upstairs to wake you (scheduler wakeup) so you can come down to the counter (syscall) and collect it (copy). Each step exists for a reason — the mailroom serves every tenant in the building safely and fairly. But you’re paying for that generality on every single package: ~2–10µs on a tuned kernel, with occasional multi-µs hiccups whenever the mailroom staff gets busy with someone else.
Kernel bypass fires the mailroom — for your packages. The dock delivers straight to a tray on your desk, and you glance at the tray in a tight loop instead of waiting to be called. Concretely: the NIC’s RX/TX descriptor rings are mapped into your process’s memory, the card DMA-writes arriving frames directly into buffers you own, and your pinned thread polls those rings. No interrupts, no sk_buffs, no syscalls, no copies on the hot path.
kernel path (the mailroom) bypass path (tray on your desk)
─────────────────────────── ─────────────────────────────────
NIC ──DMA──► kernel ring NIC ──DMA──► rings mapped into
IRQ → softirq → skb YOUR address space
→ stack → socket → your thread polls descriptors
wakeup → syscall → copy in a loop; TX = write descriptor
→ your buffer + doorbell. No kernel on hot path.
Two things you buy:
- Latency: wire-to-app drops from ~2–10µs to ~1–2µs.
- Variance — the more important one: nothing in the path can schedule, sleep, or be preempted, so the tail collapses. The mailroom can have a bad day; a tray on your desk cannot.
And one thing you sell: every service the mailroom provided. Nobody sorts your other mail anymore (no sockets, no TCP), nobody keeps records (no tcpdump), nobody answers the building’s front door for you (no ARP). Every framework below is a different answer to three questions: who owns the dock, who rebuilds the mailroom’s services, and how much of the building you still get to use.
First, though, one framing that shapes how you should learn this chapter: you have not run these in production, and you should say so in interviews. That’s fine — most senior candidates haven’t. What’s tested is whether you understand the architecture, the tradeoffs, and where each tool fits. The strong position: “My production latency work was lock-free, zero-alloc userspace design plus kernel tuning at ~10ms budgets dominated by venue RTT; kernel bypass is the next lever down, and here’s exactly how I’d apply it and what it would cost us.” This chapter arms that sentence. Never bluff hands-on experience — a follow-up (“what did your ef_vi event loop look like?”) kills a bluff in seconds; architecture fluency paired with knowing exactly where your experience ends reads as senior.
DPDK: fire the mailroom, run the dock yourself
DPDK (Data Plane Development Kit — Intel-originated, now Linux Foundation) is the full takeover: your application takes the loading dock away from the building and staffs it with its own people.
Watch what physically happens when a DPDK app starts:
- The kernel unbinds the NIC —
vfio-pci, the kernel facility that detaches a device from its in-kernel driver and hands userspace safe access to it. The NIC vanishes fromip link. As far as Linux is concerned, that network card no longer exists. - Your process maps the card’s rings and control registers (the memory-mapped switches that command the hardware) into its own address space, using a PMD — a poll-mode driver, DPDK’s userspace driver for that NIC family. The driver now lives inside your app.
- One or more of your threads — DPDK calls a thread pinned to its own dedicated core an lcore — start spinning:
rte_eth_rx_burst()to harvest arrived frames,rte_eth_tx_burst()to send. Those cores run at 100% CPU forever. That’s not a bug; that’s the employee standing at the dock staring at the conveyor so nothing ever waits.
A packet’s life is now: wire → NIC → DMA into a buffer your process owns → your loop notices it on the next poll → your code parses it. The kernel never learns the packet existed.
// the whole idea in one loop
while (run) {
struct rte_mbuf *bufs[BURST];
uint16_t n = rte_eth_rx_burst(port, queue, bufs, BURST);
for (uint16_t i = 0; i < n; i++) {
handle_frame(rte_pktmbuf_mtod(bufs[i], uint8_t *), bufs[i]->data_len);
rte_pktmbuf_free(bufs[i]);
}
}
The programming model is run-to-completion: each packet is handled start-to-finish inside that one loop on that one core — like an Express middleware chain inlined into a single function. No handoffs, no awaits, no scheduler between the steps.
Vocabulary you’ll meet around this loop, one at a time:
- mbuf (
rte_mbuf): DPDK’s packet buffer — the sk_buff replacement, except it’s basically just a pointer plus a length. - mempool: hugepage-backed pools of those buffers, all pre-allocated at startup, so nothing mallocs on the hot path. Hugepages are mandatory — DMA needs pinned physical memory, and GB-scale pools would thrash the TLB on 4KB pages.
- burst APIs: every poll grabs up to 32 frames at once — amortize the poll over a batch.
- rings (
rte_ring): lock-free queues (single- or multi-producer/consumer — SPSC/MPMC) that lcores use to pass work to each other. - EAL (Environment Abstraction Layer): DPDK’s runtime, which initializes all of the above. It is a framework with opinions — it takes over your process layout, your cores, and your memory at startup.
Latency: ~1–2µs wire-to-app; RX poll detection at ~100ns scale; line rate at 100G+ on a few cores.
What you just lost — recite these. You fired the mailroom, so:
- No kernel TCP/IP stack. For UDP market data that’s fine: parsing an Ethernet+IP+UDP header yourself is ~50 lines. But order entry rides TCP (for its in-order delivery guarantees, the TCP/UDP chapter) — and TCP is a mailroom service. Rebuilding it means adopting a userspace TCP stack: F-Stack (FreeBSD’s stack ported atop DPDK), mTCP, Seastar’s, or a commercial one — each an adopted codebase with its own bugs. Many shops therefore split: market data via DPDK/UDP, order entry via tuned-kernel TCP. A pragmatic hybrid worth naming in interviews.
- No tcpdump (DPDK has
dpdk-dumpcap/KNI-style reinjection, but it’s extra machinery you must build and run). - No ARP. ARP is the who-has-this-IP broadcast protocol that maps IP→MAC. The mailroom used to answer the front door; now, if your process doesn’t reply to ARP queries itself, the rest of the network slowly forgets your address and stops delivering.
- Burned cores (100% spin by design), hugepage provisioning, a NIC dedicated to one process, ops/debug retraining, and PMD/firmware/NIC-model compatibility as a permanent maintenance line-item.
- Rust story:
rust-dpdk-style bindings exist but are rough; realistically you write the datapath against C FFI or in C/C++.
Solarflare/Xilinx/AMD: Onload and ef_vi — the tradfi default
Solarflare NICs (acquired by Xilinx, then AMD; today’s X2/X3 series) plus their software are the incumbent default in traditional HFT. Join a tradfi desk and this is most likely what’s in the racks. Two layers, same card:
OpenOnload: the mailroom is secretly replaced, and you never notice. Your app keeps dropping letters through the same mail slots — socket(), send(), recv(), epoll_wait(), unchanged — but behind the wall, a faster private crew has replaced the building staff. Mechanically it’s an LD_PRELOAD shim: a loader trick that injects a library ahead of libc at process start, monkey-patching the socket API. TCP and UDP now run in a userspace stack inside your process, which talks to the NIC through mapped VIs (virtual interfaces — each VI is a private slice of the NIC’s rings). The result:
onload ./your_engine— zero code changes, no recompile — and latency drops from ~5–10µs to ~2–3µs one-way.- Anything the shim can’t accelerate silently falls back to the real kernel.
- It stays debuggable-ish (
onload_stackdumpinstead of tcpdump).
That “recompile nothing” property is why it won tradfi: the entire estate of existing socket code — including third-party FIX engines you don’t have source for — accelerates overnight.
ef_vi: skip the mail slots, take packages raw off the belt. This is the raw layer beneath Onload: an event-queue + descriptor-ring API against the NIC. No protocol stack at all; you build Ethernet frames yourself. ~1µs-class. The model: post receive buffers with ef_vi_receive_post(), poll ef_eventq_poll() for events, and on TX write the frame and call ef_vi_transmit(). It sits at DPDK’s level of abstraction with one big difference: per-VI, not whole-NIC. You claim specific flows/queues; the kernel keeps the interface, and ordinary traffic (ssh, monitoring) flows normally alongside your bypass path. You took over one conveyor belt, not the whole dock.
Also know the name TCPDirect (Solarflare’s ~µs “zockets” TCP built on ef_vi), and the vendor’s slogan: “Onload for the estate, ef_vi for the crown jewels.”
Cost: proprietary-ish ecosystem tied to one NIC vendor; licenses; Onload’s stack has its own tuning surface (spin options, stack-per-process layout). But operationally it’s the lowest-friction bypass in existence — which is exactly why it’s the industry default.
AF_XDP: the mailroom installs an express chute
Upstream Linux’s own answer (since ~4.18). Nobody gets fired. Instead, the mailroom bolts an express chute onto the loading dock: a small rule that says “packages matching THIS description skip sorting and drop straight into that office’s bin.” Everything else goes through normal sorting, untouched.
The rule is an XDP program — eBPF, small verified code the kernel runs safely inside itself — executing in the NIC driver at the earliest possible point, before an sk_buff is even allocated. When it matches your packets, it redirects them into a UMEM: a chunk of your process’s memory organized into frame slots. The bin on your desk.
You and the kernel then coordinate through four rings — two request/response pairs, one pair per direction:
RX side (two rings) TX side (two rings)
you ──fill ring─────▶ kernel you ──TX ring─────────▶ kernel
"here are empty slots" "send these frames"
you ◀─RX ring─────── kernel you ◀─completion ring── kernel
"slots now hold packets" "sent; slots reusable"
You post empty slots on the fill ring; the driver (in zero-copy mode, supported by modern Intel/Mellanox drivers) DMAs packets straight into your UMEM; you poll the RX ring. Like DPDK you get raw frames — no TCP for you. Rust support is decent (xsk-rs, plus libbpf/aya for the XDP program).
Latency/throughput: far better than the socket path; close-to-DPDK throughput in zero-copy mode; latency typically a hair above DPDK. The reason is visible in the picture: the mailroom still owns the dock. Ring refills and TX doorbells go through the kernel driver rather than your process touching hardware directly (kernel flags — “need-wakeup” and poll mode — tune whether that mediation costs a syscall or a spin). Realistic: ~2–4µs class wire-to-app, sub-2µs achievable with busy-polling zero-copy on good drivers.
Why choose it: no NIC takeover — the interface stays a normal kernel netdev (it still appears in ip link; tcpdump, ssh, and the rest of your traffic all still work); no vendor lock; no out-of-tree drivers; plays cleanly with containers and modern infra tooling. Why not: fewer absolute-lowest-latency guarantees than ef_vi/DPDK, zero-copy support varies by driver, and you’re still writing an L2/L3/UDP parser yourself. In one line: AF_XDP is what you pick when you want 80% of DPDK’s win while remaining a normal Linux citizen.
io_uring: not bypass — and worth knowing anyway
Be precise here; interviewers use io_uring as a trap. io_uring does not bypass the network stack. The mailroom is completely unchanged — every package still gets logged, sorted, and pigeonholed exactly as in the packet-path chapter. What changes is your trips to the counter. Instead of walking down for every operation (a syscall per send/recv), you and the mailroom share an inbox/outbox tray pair:
- The inbox is the SQ (submission queue) — a ring in shared memory where you write one entry per I/O request.
- The outbox is the CQ (completion queue) — where the kernel posts each result.
- With
SQPOLL, a kernel thread watches your inbox for you, so steady-state operation needs zero syscalls. - Registered buffers and files skip per-operation reference and mapping costs.
- Multishot recv keeps a standing receive armed — one submission keeps producing a completion per arriving message.
Where it fits in trading infra: exactly the places that are latency-sensitive but not latency-critical — order gateways handling thousands of TCP sessions, logging/journaling (its original file-IO home), drop-copy (the real-time duplicate stream of your own orders and fills that goes to risk and compliance systems), market data recording, web/API layers. It’s also the modern answer to “epoll event-loop overhead” in a crypto stack: a Rust gateway on io-uring/tokio-uring with registered buffers gets meaningful syscall-batching wins with zero exotic ops burden. Saying “I’d use io_uring for the gateway tier and reserve real bypass for the market-data hot path” is a well-calibrated senior sentence.
The exotic tier: names to know
FPGA NICs / tick-to-trade — a robot at the dock. The top of the market deletes the office worker entirely. An FPGA (Field-Programmable Gate Array — a chip whose logic circuits you rewire with code, so your “program” runs as hardware, not instructions) sits on the NIC itself, pre-programmed: “if a package matches this description, immediately drop this pre-written reply into the outgoing truck.” Feed parsing, book-delta extraction, and a (simple, pre-armed) trigger all evaluate in NIC silicon, and the response order goes out wire-to-wire in ~50–500ns — vs ~1–5µs for the best software. Software still exists, but as the “slow path” (now meaning: microseconds): it computes strategy state and arms the robot with conditions and order templates. The decision was made earlier; the hardware only executes it. One consequence follows immediately: the pre-trade risk checks (price bands, size caps, kill conditions) must live in the same hardware path — a robot that fires in 500ns can also mis-fire faster than any software can stop it, so a check that runs in software is a check that runs too late.
- Platforms: AMD/Xilinx Alveo, historically Exablaze (now Cisco), Enyx cores.
- Cost: HDL/HLS skillsets (hardware description languages / high-level synthesis — how FPGA logic is written), months-long dev cycles, painful iteration. A different engineering culture.
- Below that: full-custom ASICs (an FPGA design burned permanently into silicon) at a handful of top firms, and layer-1 switch tricks — a layer-1 switch rewires signals between ports without ever reading frames, and some can hold a pre-armed order and blast it onto the wire on a trigger, skipping the server entirely.
Kernel modules / custom drivers: the pre-DPDK-era approach (put strategy logic in a kernel module to skip user/kernel crossings). Obsolete — all downside (crash the box, GPL entanglement, undebuggable) now that userspace bypass exists — but you mention it to show you know why it’s obsolete.
Comparison table
| Latency (wire→app) | Protocol stack | Code changes | Kernel coexistence | Ops burden | Typical use | |
|---|---|---|---|---|---|---|
| Tuned kernel (ch03) | ~2–10µs | full kernel TCP/IP | none | n/a | low | crypto default; everything non-colo |
| io_uring | ~kernel minus syscalls | full kernel TCP/IP | moderate (new IO model) | perfect | low | gateways, logging, many-conn tiers |
| AF_XDP | ~2–4µs (under 2 with zero-copy + busy-poll) | none (DIY L2–L4) | high (raw frames) | good — NIC stays a netdev | medium | feed handlers w/o vendor lock |
| OpenOnload | ~2–3µs | userspace TCP+UDP (BSD API) | zero (LD_PRELOAD) | good (fallback path) | medium (vendor) | tradfi estate-wide default |
| ef_vi / TCPDirect | ~1µs | none / minimal TCP | high | partial (per-VI claim) | medium-high | tradfi crown-jewel paths |
| DPDK | ~1–2µs | none (or F-Stack etc.) | very high (framework) | poor — NIC consumed | high | max-control feed/TX engines |
| FPGA / ASIC | ~50–500ns wire→wire | hardware-implemented subset | different discipline | n/a | very high | top-tier tick-to-trade triggers |
Read it through the mailroom lens: the top rows keep the mailroom and shave trips to the counter; the middle rows dodge it — AF_XDP and Onload while keeping the rest of the building’s services, DPDK by seizing the dock outright; the bottom row replaces the worker.
How you’d actually choose (the interview arc)
- Establish the budget: measure end-to-end and the kernel share (Lab I). If transit/venue dominates (most crypto): tuned kernel + io_uring at the edges; done.
- Colo/in-AZ with µs races: split the flows. Market data (UDP, one-way, loss-tolerant-by-design): AF_XDP if staying vendor-neutral/cloud-adjacent, ef_vi or DPDK on owned metal. Order entry (TCP, correctness-critical): Onload if on Solarflare (zero-change TCP acceleration is unbeatable ROI), else tuned-kernel TCP until proven insufficient.
- Only then the exotic tier, bought not built, when strategy economics prove that the last µs pays for an FPGA team.
Plain-English recap
If you remember nothing else from this chapter:
- The kernel is a mailroom; bypass is delivery straight to your desk. The NIC’s queues get mapped into your process; your loop polls them. No interrupts, no kernel objects, no syscalls — and also no mailroom services. You traded the staff for raw speed.
- DPDK fires the whole mailroom and runs the dock itself: the NIC leaves the OS, one core stands at the conveyor forever, and you get ~1–2µs and total control. Like leaving the npm ecosystem: no sockets, no tcpdump, no ARP — you reimplement or go without. TCP becomes bring-your-own-stack, which is why real shops often run UDP market data on DPDK and keep orders on tuned-kernel TCP.
- Onload secretly replaces the mailroom behind the same mail slots:
LD_PRELOADswaps the socket layer under your unmodified binary —onload ./engineand existing socket code (even a vendor’s FIX engine) drops to ~2–3µs. Zero code changes is why it’s the tradfi default; ef_vi is its raw ~1µs layer for paths that justify a rewrite. - AF_XDP is the mailroom’s own express chute: a filter in the driver drops your packets straight into your process’s memory while everything else flows normally. ~80% of DPDK’s win while staying a normal Linux citizen — no vendor lock, observability intact.
- io_uring is NOT bypass — it batches your trips to the counter, not the sorting. Packets still walk the full kernel stack; io_uring replaces syscall-per-operation with shared submission/completion trays. Right for gateways and loggers; conflating it with DPDK is the junior tell.
- FPGAs are a pre-programmed robot at the dock, not application code: software decides the strategy and arms conditions; hardware fires a pre-built order in ~50–500ns when the condition hits. The decision was made earlier — the hardware only executes it.
- Choose by measured budget share, not coolness: if venue RTT dominates (most crypto), tuned kernel + io_uring at the edges wins. Bypass pays only when the kernel’s 2–10µs is the biggest remaining term.
Interviewer will ask
“Have you used DPDK or Onload in production?” Straight answer: “No. My production latency work was zero-allocation, lock-free userspace design and kernel tuning at budgets where venue RTT dominated — bypass wasn’t the binding constraint. I know the landscape well: what I’d reach for is Onload for estate-wide TCP because it’s zero-code-change, ef_vi or DPDK for a UDP feed handler, AF_XDP if I need to stay vendor-neutral — and I know the costs: burned cores, DIY protocol stacks, lost tcpdump-style observability.” Honest, specific, decision-shaped: that’s what passes.
“Why does DPDK need hugepages?” Two reasons. DMA: the NIC writes physical addresses, so packet memory must be pinned with stable physical mappings, and huge contiguous regions make buffer→physical translation trivial. TLB: mempools span GBs; 4KB pages would thrash the TLB on every packet touch, 2MB/1GB pages keep translations resident. Bonus: hugepages are also just good hot-path hygiene outside DPDK (the kernel-tuning chapter).
“You take over the NIC with DPDK. What just broke?” Everything the mailroom did — I just fired it, so I walk its services and read off the loss and the fix for each. Sorting: no kernel protocol stack, so no sockets on that interface — ssh and monitoring agents go dark, and order-entry TCP needs a userspace stack or another NIC. Records: no tcpdump — nobody logs the packages anymore; I mirror traffic or run DPDK’s own capture machinery. The front door: no ARP — nobody answers “who has this IP,” so peers’ ARP queries time out, the network forgets my address, and delivery stops unless my datapath answers ARP itself. Standard mitigation is the split design: a second (or flow-bifurcated) NIC keeps management and TCP in the kernel, ARP answered inline in the datapath. The question tests operational understanding, and the checklist is just “what did the mailroom do for me yesterday?”
“Onload vs ef_vi — when each?” “Onload for the estate, ef_vi for the crown jewels.” Onload keeps the mail slots: the app still calls
socket()/send()/recv(), butLD_PRELOADswaps a faster private crew in behind the wall — so existing socket code, including a vendor FIX engine you have no source for, drops to ~2–3µs with zero changes and full TCP semantics. ef_vi skips the slots and takes packages raw off one conveyor belt: no protocol stack, you build the frames yourself, ~1µs — worth it only for the few paths that justify a rewrite, typically market-data ingest and the final order-TX. They coexist on the same NIC — that layering is the standard tradfi deployment.
“Is AF_XDP as fast as DPDK? Why would you pick the slower one?” The gap falls out of who owns the dock. DPDK fires the mailroom and runs the loading dock itself, so your process touches the hardware directly. AF_XDP is the mailroom’s express chute: your packets skip the sorting, but the kernel driver still owns the dock — ring refills and TX doorbells go through it, and that mediation is the residual cost. So in zero-copy + busy-poll mode they’re close, with DPDK slightly ahead at the extreme. Pick AF_XDP anyway when the ops delta matters: the NIC stays a normal netdev — tcpdump works, other traffic flows, containers behave, no vendor lock. Pick DPDK when the last few hundred ns and total control of the device justify owning the dock outright. The framing: AF_XDP is ~80% of DPDK’s win while staying a normal Linux citizen.
“Where does io_uring fit — is it kernel bypass?” No — it batches your trips to the counter, not the sorting. Every packet still walks the full mailroom path from the packet-path chapter; what changes is that instead of a syscall per operation, you and the kernel share an inbox/outbox tray pair — the SQ and CQ rings — and with SQPOLL a kernel thread watches your inbox, so steady state needs zero syscalls. That makes it the right tool for tiers that are latency-sensitive but not latency-critical: order gateways holding thousands of TCP sessions, recorders, journaling. Conflating io_uring with DPDK is the exact junior tell this question screens for.
“Sketch the architecture of an FPGA tick-to-trade system.” Feed parsing, book-delta extraction, and pre-armed trigger evaluation in NIC-resident FPGA logic; on trigger, splice a pre-built order template (IDs, price/qty fields patched in hardware) onto the wire — ~50–500ns wire-to-wire. Host software runs the actual strategy asynchronously: computes desired triggers/prices, arms/disarms the FPGA, handles everything non-critical (risk recalc, cancels, recovery). Key design point: the FPGA executes decisions already made; software makes decisions. Risk checks must be in the hardware path too — a hardware bug can send orders faster than software can stop it.
Further reading
- DPDK Programmer’s Guide and Getting Started Guide (doc.dpdk.org) — EAL, mbufs, mempools, PMDs from the source; skim the
l2fwd/l3fwdsample apps to internalize the loop. - Cloudflare blog: “Kernel bypass” (Marek Majkowski) — the classic survey of why and how to escape the stack, with measurements; pairs with their “How to receive a million packets per second.”
- OpenOnload / ef_vi documentation (AMD/Xilinx Solarflare) — the Onload User Guide’s tuning chapters double as the best public description of the tradfi deployment model.
- LWN.net: “Accelerating networking with AF_XDP” (Jonathan Corbet) plus Karlsson & Töpel’s “The Path to DPDK Speeds for AF_XDP” (Linux Plumbers) — design and performance from the authors.
- Jens Axboe, “Efficient IO with io_uring” and the “Lord of the io_uring” tutorial site — canonical io_uring model docs; read alongside the
io_uringman pages.
Where this goes next: NIC Internals (ch05) — every bypass framework programs the NIC itself, so what is that card actually doing in silicon (RSS, flow steering, hardware timestamps), and how much of it survives when your “colo” is an AWS VM?
NIC Internals
Before you start. This chapter assumes NIC / PHY / MAC (the network card and its wire-facing layers, ch00a), descriptor rings and RX queues (the shared circular queues packets arrive through, ch00b), MSI-X interrupts (how each queue pokes a CPU core, ch00b), DMA (the card writing straight into RAM, ch00b), NUMA and L3 cache (which socket’s RAM/cache the packet lands near, ch00a), and clocks / why timestamps are hard (ch00e). If any are new, read those first — 20 minutes there saves an hour here.
The NIC stopped being a dumb serializer twenty years ago. A modern server NIC is a packet-processing computer — a small router, not a modem: it hashes, classifies, steers, timestamps, checksums, segments, and virtualizes — all before the host sees a byte. The tuning and bypass chapters (ch03, ch04) told you to steer flows and trust hardware timestamps; this chapter is what’s actually happening in silicon, and what survives when you move to the cloud (where, as a crypto trader, you probably live).
RSS: hashing flows across queues
A single core can’t process 10–100G of packets, so NICs implement RSS (Receive Side Scaling): N independent RX queues, each with its own descriptor ring and MSI-X interrupt (ch00b), packets distributed by flow hash.
┌─ NIC ─────────────────────────────┐
packet ──► │ parse 5-tuple │
│ │ │
│ Toeplitz hash(src_ip,dst_ip, │
│ src_port,dst_port) │
│ │ low bits → indirection table │
│ ▼ │
│ [Q0][Q1][Q2][Q3] rings + IRQs │
└───┬────┬────┬────┬────────────────┘
▼ ▼ ▼ ▼
CPU2 CPU3 CPU4 CPU5 (one IRQ+consumer per core)
Mechanics worth knowing cold:
- The hash is usually Toeplitz over the 4/5-tuple (src IP, dst IP, src port, dst port, and optionally protocol — the fields that identify one flow) with a configurable secret key. The result’s low bits index an indirection table (128–512 entries) mapping hash→queue.
- The commands:
ethtool -x eth0shows key+table;ethtool -Xrewrites it (e.g., weight queues unevenly, or exclude a queue reserved for your hot flow);ethtool -Lsets queue count;ethtool -N eth0 rx-flow-hash udp4 sdfnpicks which fields hash for UDP (sdfn= hash on src/dst IP and src/dst port; the letters are ethtool’s field codes, not initials). - Same flow → same queue, always: per-flow ordering preserved, per-queue state stays core-local (no cross-CPU locking on TCP state).
- The trading caveat: RSS is a load balancer, and you don’t want your venue flow load-balanced — you want it placed. A hash doesn’t know that queue 3’s core also handles your logging flows. Hence flow steering.
Flow steering: pinning a venue’s flow to its queue
This is the routing rule for your VIP customer: “this exact flow → this exact queue, always” — no load balancer in the way. ethtool ntuple rules let you write exact-match classification rules that override RSS. (Vendors brand the same hardware differently: Intel calls it Flow Director; on Mellanox/NVIDIA ConnectX parts it’s flow tables.)
ethtool -K eth0 ntuple on
# CME-style: this multicast market data group:port → queue 0
ethtool -N eth0 flow-type udp4 dst-ip 224.0.31.1 dst-port 14310 action 0
# order-entry TCP session from the venue gateway → queue 1
ethtool -N eth0 flow-type tcp4 src-ip 203.0.113.10 src-port 40000 action 1
ethtool -n eth0 # list rules
The end-state design on a trading box:
- Queue 0 ← venue A market data only; IRQ pinned to (or busy-polled by) the core running venue A’s feed handler, same NUMA node, same L3 (the last-level cache shared between neighboring cores, ch00a).
- Queue 1 ← order-entry session(s). Queue N (the rest) ← RSS for everything else (ssh, monitoring), IRQs on housekeeping cores.
- Result: your hot flow’s packets land in a ring that exactly one core ever touches — no queue sharing with noise traffic, no cache-line bouncing, deterministic cache locality. This is the software-side prerequisite that makes both busy-polling (ch03) and per-queue bypass (ef_vi VIs, AF_XDP on a specific queue — ch04) surgical instead of whole-NIC.
- Related names to recognize, then skip: ATR (Intel’s Application Targeted Routing) and aRFS (accelerated RFS) — hardware that learns where to steer a flow instead of being told; for trading, explicit rules beat adaptive magic — determinism again.
Hardware timestamping: the difference between measuring and guessing
Every latency claim needs a timestamp, and where the timestamp is taken decides what you measured — the difference between reading the CDN edge log and starting a timer inside your route handler:
wire ──► PHY/MAC ──► DMA ──► IRQ/poll ──► stack ──► recv() ──► app clock_gettime()
▲ HW timestamp ▲ SW timestamp (kernel rx) ▲ app timestamp
(PHC clock, ~ns) (µs later) (more µs later)
-
Software timestamps (
SO_TIMESTAMPNS, or your ownclock_gettimeafter recv) include the entire kernel path + wakeup — µs of jitter that has nothing to do with the network. Fine for coarse work; useless for claims like “our wire-to-wire is 4µs.” -
Hardware timestamps: the NIC’s PHY/MAC stamps the packet against the PHC (PTP Hardware Clock — a clock on the NIC, exposed as
/dev/ptp0). On RX, the stamp arrives beside eachrecvmsgas acmsg— a control message, metadata the kernel attaches next to the payload, like response headers riding alongside a body. On TX, the stamp of your outgoing packet comes back on the socket error queue (MSG_ERRQUEUE) — a famously awkward API; you read your own echoes.ethtool -T eth0shows what the NIC supports. The exact flags:// request hardware RX timestamps on this socket int f = SOF_TIMESTAMPING_RX_HARDWARE // stamp packets in the NIC on receive | SOF_TIMESTAMPING_RAW_HARDWARE; // deliver the raw PHC value in the cmsg setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &f, sizeof(f)); -
PTP (IEEE 1588) — Precision Time Protocol, NTP’s precise cousin (ch00e) — syncs the PHC to a grandmaster, the network’s reference clock, usually GPS-fed. The measurement itself is a four-timestamp exchange:
master ──Sync────────▶ slave t1 = master sends (stamped in master's NIC) t2 = slave receives (stamped in slave's NIC) master ◀─Delay_Req──── slave t3 = slave sends t4 = master receives (echoed back to slave)Four timestamps, two unknowns, so both fall out: clock offset = ((t2−t1)−(t4−t3))/2 and one-way path delay = ((t2−t1)+(t4−t3))/2. The stamps must be hardware stamps — a software stamp would add the kernel’s µs of jitter to a ns-scale measurement — and switch queuing would poison the path-delay estimate, which is why PTP-aware switches participate as boundary clocks (re-run the exchange per port) or transparent clocks (write their own queuing delay into the packet) so their queues don’t corrupt the math. Two daemons split the work.
ptp4lruns this exchange and disciplines the NIC’s clock from the network — to discipline a clock is to continuously nudge it toward a reference (measure the offset, adjust the rate, repeat) rather than jumping it — achieving sub-µs, often tens-of-ns accuracy on a clean LAN.phc2systhen disciplines the system clock from the NIC’s clock. Trading venues and colos distribute PTP feeds (often GPS-disciplined); regulation (MiFID II RTS-25) requires 100µs-or-better clock sync for HFT firms — timestamping is compliance, not just engineering hygiene. -
Why you care beyond honesty: one-way latency measurement (feed handler A-to-B across hosts) is impossible without synced clocks; A/B feed arbitration analytics, venue SLA disputes, and tick-to-trade attribution (“of our 4.2µs, 0.9 was NIC-to-app”) all hang off hardware stamps. The interview-grade sentence: “software timestamps measure your kernel, hardware timestamps measure your network — never present the first as the second.”
-
Top-tier shops go further: optical taps (a passive splitter spliced into the fiber — a wiretap that can’t perturb what it measures) + dedicated capture cards (or switch-based timestamping à la Arista 7130 + MetaWatch) stamping every packet at multiple points — latency measured by watching the wire, not by asking the servers.
Interrupt moderation, revisited at the hardware level
The packet-path chapter (ch01) gave the concept; the knobs live in the NIC:
rx-usecs/tx-usecs: hold-off timers per queue (“wait up to N µs before raising the IRQ”);rx-frames: fire after N packets regardless. Adaptive modes retune these per-queue against traffic rate — throughput-friendly, latency-hostile (the algorithm optimizes packets-per-interrupt exactly when you want interrupt-per-packet).- Trading-box setting:
ethtool -C eth0 adaptive-rx off rx-usecs 0 rx-frames 1on the hot queues — or, since ntuple gave you per-flow queues, the refined version: per-queue coalescing (ethtool --per-queue eth0 queue_mask 0x1 --coalesce rx-usecs 0) — zero moderation on the market-data queue, generous moderation on the noise queues so housekeeping cores take fewer interrupts. Best of both, and a detail that signals real hands-on-ness. - If the hot queue is busy-polled or bypassed, its IRQ never fires and moderation is moot — the knob only matters for whatever still uses interrupts.
SR-IOV: slicing one NIC into many
SR-IOV is NIC multi-tenancy: one physical card carved into many virtual cards, each tenant getting its own. The physical NIC (PF, physical function) exposes multiple lightweight PCIe devices (VFs, virtual functions). Each VF gets its own rings, IRQs, and MAC address; an embedded switch on the NIC bridges them. A VF can be handed directly to a VM (PCI passthrough) or a container — DMA goes straight into the guest, hypervisor not in the datapath.
- Why it exists for you: it’s how you get bypass-grade access in shared environments. A VF passed into your VM/container can run DPDK or a PMD at near-bare-metal latency — add ~hundreds of ns for the embedded switch plus IOMMU translation. (The IOMMU is the memory-management unit for devices: it translates and polices the addresses a card may DMA into.) The alternative is the software-virtualized path: virtio, a NIC emulated in software, feeding through the hypervisor’s software switch (the vswitch) — which adds tens of µs and jitter.
- Costs/limits: VF count and features are constrained (fewer queues, limited filtering vs the PF), the embedded switch is another black box, live migration breaks, and the cloud providers you trade on don’t hand you raw VFs of their choosing: they expose their own paravirtual-ish devices built on this machinery. Which brings us to:
Cloud NICs: what AWS ENA and GCP gVNIC actually give you
Crypto reality: Binance ≈ AWS Tokyo, Bybit ≈ AWS Singapore, OKX ≈ Alibaba HK — so your “colo” is a VM near the venue, and your NIC is what the hypervisor says it is. ENA and gVNIC are the paravirtual NIC devices AWS and GCP respectively expose to guests. On AWS the platform underneath is Nitro — Amazon’s custom hypervisor plus hardware-offload cards; several rows in the table depend on it. Know precisely what you keep and lose vs. bare metal:
| Capability | Bare metal (ConnectX/X2) | AWS ENA | GCP gVNIC |
|---|---|---|---|
| Multiqueue + RSS | full control | yes (queues scale w/ instance size) | yes |
| ntuple / flow steering | full | limited/none (no ethtool ntuple) | no |
| Coalescing control | full, per-queue | partial (rx-usecs on modern ENA, adaptive) | limited |
| HW timestamps (PHC) | yes, full PTP | partial: PTP hardware clock on Nitro (/dev/ptp0, sync to AWS ref) on supported instances; per-packet rx stamping limited | no PHC; GCP offers NTP-ish time; no per-packet HW stamps |
| Kernel bypass | DPDK/ef_vi/AF_XDP all | ENA has a DPDK PMD and AF_XDP (zc on recent drivers) — works, but latency floor set by Nitro | gVNIC DPDK PMD exists; similar caveats |
| Latency floor (intra-AZ RTT) | ~1–10µs switch fabric | ~40–100µs typical (cluster placement group helps) | similar order |
| Jitter sources | you control them | hypervisor, neighbors, fabric — not yours | same |
Two takeaways:
- The cloud sets a latency floor you cannot tune through. Intra-AZ RTTs of tens of µs with occasional ms-tails come from the virtualization fabric, not your stack — so heroic guest-side tuning past a point buys nothing. The levers that do matter in cloud: instance family (Nitro-based, ENA Express/SRD where offered — SRD is AWS’s Scalable Reliable Datagram, a multi-path fabric protocol that cuts tail latency), cluster placement groups (same-rack-ish adjacency), same-AZ-as-venue placement (worth ~100µs–1ms vs cross-AZ — and venues’ AZs are community-known), biggest-instance-size for dedicated NIC queues and no CPU neighbors, and connection/session management (ch02). This is why crypto HFT is more about placement and protocol than stack — and being able to articulate that hierarchy is exactly the judgment interviews probe.
- Timestamping honesty degrades in cloud. Without per-packet hardware stamps, your latency numbers include guest-kernel and hypervisor noise. Practical crypto approach: Nitro PTP plus chrony (the standard Linux NTP client) for decent absolute time (sub-100µs), measure distributions not single numbers, cross-check with venue-provided timestamps (exchange
tsfields in feed messages — mind their own accuracy), and never quote “wire-to-wire” numbers you didn’t measure at the wire.
Plain-English recap
If you remember nothing else from this chapter:
- A modern NIC is a small router, not a modem: it parses, hashes, classifies, steers, and timestamps every packet in silicon before your OS sees a byte.
- RSS is a consistent-hash load balancer with sticky sessions: flows hash across N queues so N cores share the load, and one flow always lands on the same queue (ordering preserved, state stays core-local). But it’s statistical — and your venue feed shouldn’t be load-balanced, it should be placed.
- Flow steering (ntuple rules) is a routing rule for your VIP customer: “this exact multicast group → queue 0, always” — a dedicated queue, consumed by a dedicated core, instead of round-robin with the noise. It’s the prerequisite that makes busy-polling and per-queue bypass surgical.
- Hardware vs. software timestamps is measuring at the CDN edge vs. inside your handler: the NIC stamps at the wire against its own on-card clock (the PHC); software stamps include your whole kernel path and scheduling jitter. HW measures the network, SW measures your host — never present one as the other.
- PTP is NTP’s precise cousin: it disciplines NIC clocks across machines to sub-µs (NTP is ms-grade), which is what makes one-way, cross-host latency numbers meaningful at all — and regulators require it.
- SR-IOV is NIC multi-tenancy: one card carved into many virtual cards, each DMA-ing straight into its VM — the substrate cloud NICs are built on, except the provider keeps the knobs.
- In the cloud, placement beats tuning: ENA/gVNIC epoxy over most controls and the hypervisor sets a latency floor (tens of µs, ms tails) you cannot tune through. Spend effort on same-AZ placement, placement groups, instance choice, and the protocol layer — that’s what moves the milliseconds.
Interviewer will ask
“What is RSS and what problem does it solve? What’s the trading-specific problem with it?” First beat — the problem it solves: one core can’t process 10–100G of packets. So the NIC Toeplitz-hashes each flow’s tuple into an indirection table and spreads flows across N queues, one queue+IRQ per core. Same flow → same queue, always — so per-flow ordering holds and flow state stays core-local, no cross-CPU locking. Second beat — the trading problem: RSS is a load balancer, and a venue feed shouldn’t be load-balanced, it should be placed. The hash doesn’t know that queue 3’s core also handles my logging. So I override RSS with an ntuple/Flow Director rule for the hot flow — dedicated queue, dedicated core — and leave RSS to spread the noise.
“How would you pin a venue’s market data to one core, end to end?”
ethtool -Nntuple rule matching the multicast group:port → dedicated queue; that queue’s IRQ affinitized to (or the queue busy-polled from) a core on the NIC’s NUMA node reserved via isolcpus; feed-handler thread pinned to that core; consumer of its output on a neighboring core sharing L3, connected by an SPSC ring (single-producer single-consumer lock-free queue). Then verify withethtool -Sper-queue counters and/proc/interruptsthat only that queue moves when the feed bursts.
“Hardware vs software timestamps — what’s the difference and when does it matter?” Where the stamp is taken decides what you measured — it’s reading the CDN edge log versus starting a timer inside your route handler. Hardware: the NIC stamps at the PHY/MAC against its on-card PHC clock, ns-grade, so it measures the network; it needs PTP discipline to be comparable across hosts. Software: the kernel or app stamps only after the full stack and a scheduler wakeup — µs of jitter that measures your host, not the wire. So it matters whenever the claim reaches past your host: one-way numbers, A/B feed arbitration, network-vs-stack latency attribution, MiFID II clock-sync rules. One-liner: software timestamps measure your kernel, hardware timestamps measure your network — never present the first as the second. Hands-on tell: TX hardware stamps come back on the socket error queue (
MSG_ERRQUEUE) — you read your own echoes.
“Explain PTP in two minutes.” NTP’s precise cousin, built on a four-timestamp exchange. The grandmaster (usually GPS-fed) sends a Sync message, hardware-stamped leaving its NIC (t1) and arriving at mine (t2); my side sends a Delay_Req back, stamped leaving (t3) and arriving (t4). Four timestamps, two unknowns, so both fall out: the asymmetric part is my clock offset, ((t2−t1)−(t4−t3))/2, and the symmetric part is the path delay, ((t2−t1)+(t4−t3))/2. The stamps must be hardware — a software stamp would inject the kernel’s µs of jitter into a ns-scale measurement — and switch queues would poison the path-delay estimate, which is why PTP-aware switches join in as boundary or transparent clocks and correct their own queuing delay. Operationally:
ptp4lruns the exchange and disciplines the NIC’s PHC,phc2sysslews the system clock to the PHC. Result on a clean LAN: tens-of-ns to sub-µs sync — enough to make one-way latency numbers meaningful. Without hardware assist it degrades toward NTP’s milliseconds, which is why cloud time is the weak link.
“What’s SR-IOV and why does it matter for low-latency in virtualized environments?” One physical NIC exposes many virtual functions — real PCIe devices with own rings/IRQs — passed directly into VMs/containers so DMA skips the hypervisor’s software switch. It’s the difference between virtio-through-vswitch (tens of µs, jittery) and near-bare-metal (~µs + a few hundred ns for the embedded switch/IOMMU). It’s also the substrate cloud NICs are built on — except the provider keeps the controls, which is exactly why ENA/gVNIC feel like a NIC with most knobs epoxied over.
“You move a strategy from bare-metal colo to AWS next to a crypto venue. What changes in your networking approach?” The cloud sets a latency floor I cannot tune through — tens of µs with ms tails, made by the hypervisor and fabric, not my stack — so placement beats tuning. Three buckets follow. Placement: same AZ as the venue (worth ~100µs–1ms by itself), cluster placement group, Nitro/ENA-Express instance family, biggest size for dedicated queues and quiet neighbors. Stack and protocol: keep only the cheap robust tuning — pinning, NODELAY, busy-poll, buffers, max queue counts — and move the real effort up to the protocol layer: connection warmup, redundancy, reconnect storms, parsing. Hardware flow steering is gone; RPS/XPS — the kernel’s software steering knobs from the tuning chapter (ch03) — approximate the intent. Measurement honesty: ntuple rules and per-packet hardware stamps don’t exist here, so I measure distributions on PHC/chrony-disciplined clocks, cross-check venue timestamps, and say plainly that my numbers are host-inclusive.
“How do you verify your NIC config is actually doing what you think?”
ethtool -S(per-queue packet/drop counters — burst the feed, watch exactly one queue increment),/proc/interruptsdeltas per core,ethtool -x/-nto dump RSS table and ntuple rules,ethtool -Tfor timestamp capabilities,ethtool -c/-g/-kfor coalescing/rings/offloads, and end-to-end: latency histogram before/after each change. Config that isn’t verified by a counter or a histogram is folklore.
Further reading
- Linux kernel docs:
Documentation/networking/timestamping.rst— SO_TIMESTAMPING, PHC, MSG_ERRQUEUE semantics, with example code; the primary source for the timestamp APIs. - linuxptp project:
ptp4l(8)andphc2sys(8)man pages — the operational half of PTP; skim the IEEE 1588 overview in the ptp4l docs. - Linux kernel docs:
Documentation/networking/scaling.rst— RSS/RPS/RFS/XPS definitions straight from the maintainers; pair withethtool(8)for-N/-X/-L/-C/--per-queue. - Intel Ethernet Controller datasheets (e.g., X710/E810) Flow Director sections, and AMD Solarflare ef_vi docs — what steering hardware actually implements; heavier reading, high credibility yield.
- AWS
amzn-driversGitHub (ENA driver docs/README) and the AWS “PTP hardware clock on Nitro” documentation; GCP gVNIC docs — the authoritative statements of what cloud NICs do and don’t expose.
Where this goes next: Lab I: Measuring the Stack You’re Bypassing — the chapters behind you (ch01–ch05) handed you numbers; the lab makes you earn them: reproduce the kernel RTT, the Nagle trap, syscall cost, and the price of sleeping on your own machine, in Rust, in under an hour.
Lab I: Measuring the Stack You’re Bypassing
Before you start. This lab assumes syscalls and the vDSO (crossing into the kernel, and the kernel code mapped into your process that lets some calls avoid crossing at all, ch00b), TSC / rdtsc (the CPU’s cycle counter and the instruction that reads it, ch00e), percentiles, warmup, and benchmark hygiene (why p99.9 matters and means lie, ch00e), busy-polling vs. blocking (spin vs. sleep, in event-loop terms, ch00c), and the Nagle/delayed-ACK trap from chapter 2. If any are new, read those first — 20 minutes there saves an hour here.
The chapters before this lab (ch01–ch05) gave you numbers. Never quote a number you haven’t measured — this lab makes every headline claim reproducible on a stock Linux box: kernel RTT (round-trip time), the Nagle/delayed-ACK trap, syscall cost, the price of sleeping — and then two parts that were missing from this lab’s first edition: measuring what kernel tuning actually buys (Part F, the kernel-tuning chapter’s claims put on a scale) and running real kernel bypass (Part G, AF_XDP on a virtual wire). Parts A–D take under an hour; E–G are stretch goals worth a second session. Everything compiles with stable Rust. Run on Linux x86_64 (a cloud VM is fine — expect noisier tails, which is itself a lesson; the rdtsc part needs x86_64).
Run everything --release, and pin the process to quiet cores if you can (taskset pins a process to specific CPU cores — pinning matters because a thread that migrates mid-run lands on a core with cold caches, and that shows up as tail noise):
cargo new hft-lab1 && cd hft-lab1
# ... add files below ...
nproc # know your core count first
taskset -c 2,3 cargo run --release --bin udp_rtt # 4+ cores
taskset -c 0,1 cargo run --release --bin udp_rtt # 2-core VM: these are the cores you have
Know your box before you start. taskset -c 2,3 in the listings assumes 4+ cores — on a 2-vCPU cloud VM those cores don’t exist and taskset fails; substitute -c 0,1 (or -c 1 for single-core pins) throughout. Parts F and G want sudo and a few packages (sudo apt-get install -y stress-ng linux-cpupower; Part G’s extras are listed there). If your daily box is 2 vCPUs, Part F wants a throwaway 4-vCPU spot VM for an hour — separating “the load” from “the measured thread” needs cores to separate them onto.
Scaffold
Cargo.toml:
[package]
name = "hft-lab1"
version = "0.1.0"
edition = "2021"
[dependencies]
libc = "0.2"
# Part E (optional stretch) only; requires Linux 5.6+
[target.'cfg(target_os = "linux")'.dependencies]
io-uring = "0.7"
# Part G (optional stretch) only; see Part G's setup for system packages
xsk-rs = { version = "0.8", optional = true }
[features]
xdp = ["dep:xsk-rs"]
# Part G binary only builds when you ask for it — plain builds stay dependency-light
[[bin]]
name = "xdp_rx"
required-features = ["xdp"]
[profile.release]
opt-level = 3
lto = true
codegen-units = 1
src/lib.rs — one shared histogram reporter:
Two things in this file you won’t have met if you’ve only written safe Rust. First, a libc:: call. Picture Rust’s standard library the way you picture Node’s fs module: a friendly wrapper around the C functions the operating system actually speaks. When the wrapper doesn’t expose the thing you need, you go one floor down and call the C function yourself — libc::clock_gettime here is the same C function that sits under Node’s process.hrtime(). Second, an unsafe block. Rust’s compiler normally proves your memory access is sound before it lets you compile — but it can’t read C code, so for these calls unsafe is you signing the guarantee instead: “compiler, trust me on this one.” Both appear below, glossed where they land.
#![allow(unused)]
fn main() {
// src/lib.rs
// Sorts the samples and prints one row of percentiles (min/p50/p99/p99.9/max).
// Every part of the lab funnels its numbers through this.
pub fn report(name: &str, mut ns: Vec<u64>) {
ns.sort_unstable();
let n = ns.len();
let pct = |p: f64| ns[((n as f64 * p) as usize).min(n - 1)];
println!(
"{:28} n={:<7} min={:>9} p50={:>9} p99={:>9} p99.9={:>9} max={:>9}",
name, n, fmt(ns[0]), fmt(pct(0.50)), fmt(pct(0.99)),
fmt(pct(0.999)), fmt(ns[n - 1])
);
}
// Renders a nanosecond count as "850ns" / "12.3us" / "40.1ms" so columns stay readable.
fn fmt(ns: u64) -> String {
if ns < 10_000 { format!("{}ns", ns) }
else if ns < 10_000_000 { format!("{:.1}us", ns as f64 / 1e3) }
else { format!("{:.1}ms", ns as f64 / 1e6) }
}
// Instant is a private stopwatch: each process starts it at its own zero, so
// a stamp from one process means nothing in another. CLOCK_MONOTONIC is one
// machine-wide clock — every process on the box reads the same one — which is
// why Parts F/G can stamp a packet in the sender process and subtract in the
// receiver process. (Returns nanoseconds.)
pub fn mono_ns() -> u64 {
// An empty two-field form (whole seconds + leftover nanoseconds) that we hand
// to the kernel to fill in. C functions return data by writing into memory
// you provide — like passing an object for a callback to mutate.
// (libc::timespec is that C struct.)
let mut ts = libc::timespec { tv_sec: 0, tv_nsec: 0 };
// `unsafe` = "compiler, trust me": Rust can't read C code, so it can't prove
// clock_gettime writes only into our form and nothing else. We sign for it.
unsafe { libc::clock_gettime(libc::CLOCK_MONOTONIC, &mut ts) };
ts.tv_sec as u64 * 1_000_000_000 + ts.tv_nsec as u64
}
}Percentiles, not averages: latency distributions are heavy-tailed, and the p99.9 column is where every kernel pathology from the packet-path chapter (ch01) shows up — wakeups, preemption, C-state exits, IRQs, page faults.
Part A — Baseline: UDP round-trip through the kernel
The whole journey first:
THREAD A (userspace) │ KERNEL │ THREAD B (userspace)
── syscall boundary ──┤ ├── syscall boundary ──
1 a.send() ──────────►│ copy into B's socket queue │
│ softirq: w1 wake B ─────────►│ (was parked in recv)
│ copy out on recv return ────►│ 2 b.recv() returns
│ copy into A's socket queue ◄─│─ 3 b.send()
(parked in recv) │ softirq: w2 wake A │
4 a.recv() returns ◄──│ copy out on recv return │
1 = a.send(), 3 = b.send() — syscalls: cross in, copy into the peer's queue
2 = b.recv(), 4 = a.recv() — the syscalls each thread was parked inside
w1, w2 = scheduler wakeups un-parking them → 4 syscalls + 2 wakeups per RTT
src/bin/udp_rtt.rs:
// Baseline: what does a full kernel round trip cost on loopback?
use std::net::UdpSocket;
use std::time::Instant;
const WARMUP: usize = 10_000;
const ITERS: usize = 100_000;
// Spawns an echo thread on socket b, then times WARMUP+ITERS blocking
// round trips from socket a and prints the percentile row.
fn main() {
let a = UdpSocket::bind("127.0.0.1:0").unwrap();
let b = UdpSocket::bind("127.0.0.1:0").unwrap();
// connect() on UDP dials nothing — there's no handshake to perform. It just
// saves the peer's address, like filling in a default "to:" field, so the
// plain send()/recv() calls below don't need an address every time.
a.connect(b.local_addr().unwrap()).unwrap();
b.connect(a.local_addr().unwrap()).unwrap();
// echo peer
std::thread::spawn(move || {
let mut buf = [0u8; 64];
loop {
let n = b.recv(&mut buf).unwrap();
b.send(&buf[..n]).unwrap();
}
});
let msg = [0u8; 32]; // tick-sized payload
let mut buf = [0u8; 64];
let mut samples = Vec::with_capacity(ITERS);
for i in 0..WARMUP + ITERS {
let t0 = Instant::now();
a.send(&msg).unwrap();
a.recv(&mut buf).unwrap();
if i >= WARMUP {
samples.push(t0.elapsed().as_nanos() as u64);
}
}
hft_lab1::report("udp_rtt blocking loopback", samples);
}One RTT = 4 syscalls + 2 loopback traversals + 2 scheduler wakeups (crossings 1–4, w1/w2 above). Loopback (127.0.0.1 — packets short-circuit inside the kernel and never touch a NIC) skips the NIC/DMA/IRQ hardware stages, so this measures the software stack — the part bypass deletes. Watch p50 vs p99.9 diverge; then re-run under load (stress-ng --cpu 4 elsewhere) and watch the tail explode — under load the echo thread queues up behind the stress-ng workers before the scheduler gets it back on a core, and that wait lands directly in your tail.
Part B — TCP_NODELAY A/B: catching Nagle in the act
The trap is a standoff between two timers. Time flows down:
time CLIENT userspace │ CLIENT KERNEL (Nagle) ═ wire ═ SERVER KERNEL (delayed ACK)
│ ── syscall boundary ──
│ 1 write 40B ────►│ nothing unACKed → out ──[40B]───► queued to app; server app
│ 2 write 60B ────►│ small + 40B still unACKed reads 40/100 → no reply.
│ │ → Nagle HOLDS the 60B: ACK owed for the 40B, but
│ │ "wait for the ACK" the delayed-ACK timer HOLDS
│ ~40ms │ ▲ it: "wait — I might piggy-
│ deadlock │ │ each side waits back it on a reply"
│ window │ │ for the other │
▼ │ └──────[ACK]◄───────────── timer expires (~40ms)
3 read returns ◄─│ ACK frees Nagle ──[60B]─────────► 100/100 → reply ─► client
1 = c.write_all(&[1u8; 40]) — "header": sails through, no unACKed data yet
2 = c.write_all(&[2u8; 60]) — "body": small write + unACKed data → Nagle queues it
3 = c.read_exact(&mut resp) — completes only after the ~40ms timer breaks the tie
src/bin/tcp_nodelay.rs:
// The Nagle + delayed-ACK interaction (the TCP chapter), reproduced on demand.
// Run: cargo run --release --bin tcp_nodelay -- on
// cargo run --release --bin tcp_nodelay -- off
use std::io::{Read, Write};
use std::net::{TcpListener, TcpStream};
use std::time::Instant;
const REQ: usize = 100;
const ITERS: usize = 200; // 40ms stalls make big runs slow
// Starts an echo server, then times ITERS request/response round trips where
// each request is deliberately split into two small writes — Nagle bait.
fn main() {
let nodelay = std::env::args().nth(1).map(|s| s == "on").unwrap_or(true);
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = l.local_addr().unwrap();
std::thread::spawn(move || {
let (mut s, _) = l.accept().unwrap();
s.set_nodelay(true).unwrap(); // server side kept sane
let mut buf = [0u8; REQ];
loop {
if s.read_exact(&mut buf).is_err() { return; }
s.write_all(&buf).unwrap(); // responds only after the FULL request
}
});
let mut c = TcpStream::connect(addr).unwrap();
c.set_nodelay(nodelay).unwrap();
let mut resp = [0u8; REQ];
let mut samples = Vec::with_capacity(ITERS);
for _ in 0..ITERS {
let t0 = Instant::now();
// One logical request as TWO small writes: "header" then "body".
c.write_all(&[1u8; 40]).unwrap();
c.write_all(&[2u8; 60]).unwrap();
c.read_exact(&mut resp).unwrap();
samples.push(t0.elapsed().as_nanos() as u64);
}
hft_lab1::report(&format!("tcp rtt nodelay={nodelay}"), samples);
}With nodelay=on the Nagle hold never happens and the standoff can’t form. Expect p50 to move from tens of µs to ~40ms — a 1000x regression from one missing setsockopt. Also internalize the second lesson: even with NODELAY, that’s two packets for one message — assemble one buffer, one write.
Part C — Syscall cost with rdtsc
The binary asks “what time is it?” three ways, and the only thing that differs is how far the question travels (journey 2 is the vDSO from ch00b):
USERSPACE │ KERNEL
── syscall boundary ──
1 getpid ────────────────────────────────────┼──► run handler, come back
◄─────────────── ~100–250ns ───────────────┼────┘ (a full crossing)
2 clock_gettime ──► ┌─────────────────────┐ │
◄──── ~20ns ───── │ vDSO clock page: │◄─┼──── kernel refreshes the page
│ kernel data, mapped │ │ from its side
│ INSIDE your process │ │ (the call never crosses)
└─────────────────────┘ │
3 rdtsc ── ~6–10ns ── one instruction: never leaves the core, let alone userspace
1 = libc::syscall(SYS_getpid) — a guaranteed kernel entry and exit
2 = libc::clock_gettime(CLOCK_MONOTONIC) — answered by the vDSO in userspace
3 = _rdtsc() — reads the CPU's own counter register
src/bin/syscall_cost.rs (x86_64 only):
// What does crossing into the kernel cost, cycle-counted with rdtsc?
use std::time::Instant;
const N: u64 = 2_000_000;
// The odometer read. The TSC (timestamp counter) has been counting ticks since
// boot, and on modern CPUs it keeps a constant rate even when the core changes
// clock speed ("invariant TSC") — which upgrades it from rev counter to clock.
// (_rdtsc compiles to the single `rdtsc` instruction: ~6-10ns, no kernel involved.)
fn rdtsc() -> u64 {
// `unsafe` = "compiler, trust me": this block does something Rust can't
// check for you. Here it's harmless — we're just reading a counter the
// CPU exposes.
unsafe { core::arch::x86_64::_rdtsc() }
}
// Calibrating the odometer: nobody told us how fast it ticks, so we race it
// against ~200ms of wall clock and divide — ticks counted / seconds elapsed
// = the counter's rate in cycles per second (Hz).
fn tsc_hz() -> f64 {
let t0 = Instant::now();
let c0 = rdtsc();
while t0.elapsed().as_millis() < 200 {}
(rdtsc() - c0) as f64 / t0.elapsed().as_secs_f64()
}
// Warms up, runs f() N times inside one rdtsc bracket, prints avg cycles and ns/op.
fn bench(name: &str, hz: f64, mut f: impl FnMut()) {
for _ in 0..10_000 { f(); } // warmup
let c0 = rdtsc();
for _ in 0..N { f(); }
let cycles = (rdtsc() - c0) as f64 / N as f64;
println!("{:22} {:>7.1} cycles {:>7.1} ns/op", name, cycles, cycles / hz * 1e9);
}
// Calibrates the TSC, then cycle-counts three clocks: a real syscall,
// a vDSO call, and Rust's Instant::now.
fn main() {
let hz = tsc_hz();
println!("TSC ~{:.2} GHz", hz / 1e9);
let mut ts = libc::timespec { tv_sec: 0, tv_nsec: 0 };
// 1. A full crossing. Every kernel service has a number, and libc::syscall
// enters the kernel by that number with no library wrapper in between —
// a guaranteed entry and exit. (getpid as the probe: uncached by
// glibc since 2.25, so every call really crosses.)
bench("getpid syscall", hz, || unsafe {
libc::syscall(libc::SYS_getpid);
});
// 2. No crossing at all. Same C-function shape as a syscall, but the vDSO
// answers from a page the kernel keeps updated inside YOUR process —
// the question never crosses.
bench("clock_gettime (vDSO)", hz, || unsafe {
libc::clock_gettime(libc::CLOCK_MONOTONIC, &mut ts);
});
// 3. What your Rust code actually calls. black_box is a wall the optimizer
// can't see through: without it, the compiler notices nobody reads the
// Instant, deletes the loop, and you time an empty bracket.
bench("Instant::now", hz, || {
std::hint::black_box(Instant::now());
});
}Three lessons in one binary. (1) A genuine kernel entry costs ~100–250ns — how much depends on “mitigations”, the Spectre/Meltdown security patches that add extra work to every kernel entry; compare a box booted with mitigations=off to see their price. getpid is the probe precisely because glibc stopped caching it in 2.25 — every call genuinely crosses. (2) The vDSO means timestamping is not a syscall (~15–25ns), which is why you can afford to timestamp everything. (3) rdtsc itself (~6–10ns) is the only sane way to time nanosecond-scale operations — Instant::now inside the measured region would dominate it. One caveat rides along: the CPU can reorder rdtsc relative to nearby work; the clocks chapter (ch07) covers the fences.
Part D — The price of sleeping: blocking vs busy-poll vs spin
src/bin/busy_poll.rs:
The three receive modes first — all three answer the same question, how does a thread wait for a packet; they differ in who does the waiting and where. Blocking parks the thread: it tells the kernel “wake me when a packet arrives” and leaves the core entirely — free for other work — and when data lands the kernel must reschedule the thread and refill its caches. That wakeup is the microseconds this part measures. Busy-poll keeps the single blocking recv(), but on an empty queue the kernel itself polls the NIC driver’s ring for up to a bounded time before parking the thread — still one syscall; the waiting moves in-kernel, and any packet it finds is pulled through the stack right there instead of waiting for the interrupt path to deliver it. Spin makes the socket non-blocking and loops recv() from userspace: the waiting is your loop, one cheap syscall per check, and the thread never parks. The two delete different costs: spin deletes the wakeup; busy-poll deletes the interrupt-delivery leg while keeping parking as its fallback. They also compose — SO_BUSY_POLL on a non-blocking socket makes each spin-loop recv() poll the driver as well, deleting both costs at once; that combination on a pinned core is the strongest tuned-kernel receive short of bypass (inert on loopback like busy-poll alone, so measure it on the Part G veth or two hosts). Where each mode does its waiting:
USERSPACE │ syscall boundary │ KERNEL
block 1 recv() ───────────────────┼─────────────────►│ empty → thread PARKED here
(thread off the core) │ │ packet lands (softirq)
3 recv() returns ◄──────────┼──── 2 WAKEUP ────│ reschedule, refill caches
│ ▲ the expensive, spiky crossing — the tail
busy- 1 recv() ───────────────────┼─────────────────►│ empty → kernel itself polls
poll 2 recv() returns ◄──────────┼──────────────────│ the driver ring for ≤200µs
│ one crossing; the waiting stays in-kernel
spin 1 recv() → WouldBlock ◄────►│ cheap, immediate │ each call just peeks the
2 recv() → WouldBlock ◄────►│ round trips │ queue; the thread never
n recv() → data ◄──────────►│ │ parks, so nothing to wake
block: rx.recv() on a blocking socket — 2 is the µs-scale wakeup this part measures
busypoll: set_busy_poll(&rx, 200), then the same rx.recv()
spin: rx.set_nonblocking(true); each WouldBlock loops via std::hint::spin_loop()
One new construct in this listing: SO_BUSY_POLL has no Rust wrapper, so we set it the C way, with setsockopt. A C API can’t see your types — C has no generics — so you hand it a raw pointer (“the data starts here”) and a byte count (“it runs this long”), and it takes your word for what’s there. That’s why the code below passes a pointer plus a size where Rust would normally pass a typed value.
// One-way latency into an IDLE receiver: blocked-and-woken vs spinning.
// Run: ... --bin busy_poll -- block | busypoll | spin
use std::net::UdpSocket;
use std::os::unix::io::AsRawFd;
use std::time::{Duration, Instant};
const ITERS: usize = 2_000;
const GAP: Duration = Duration::from_micros(500); // receiver idles between packets
// Tells the kernel: when a recv finds no data, keep checking the driver for up
// to `usec` microseconds before parking me. Warns (and continues) if refused.
fn set_busy_poll(s: &UdpSocket, usec: libc::c_int) {
// `unsafe` = "compiler, trust me": Rust can't check a C function's paperwork.
// We're vouching that the pointer and byte count below really describe one int.
let r = unsafe {
libc::setsockopt(
// as_raw_fd(): underneath the Rust socket object sits a plain integer —
// the number the kernel issued when the socket was opened (its
// file descriptor). C APIs speak these numbers, not wrapper types.
// SOL_SOCKET = "a socket-level option, not a TCP- or IP-level one".
s.as_raw_fd(), libc::SOL_SOCKET, libc::SO_BUSY_POLL,
// The pointer half — where the data starts:
// &int → pointer-to-int → pointer-to-anything (void*). The casts
// erase the type because C reads memory, not types...
&usec as *const libc::c_int as *const libc::c_void,
// ...and the length half — how many bytes sit there (socklen_t is
// just C's name for "a length").
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
)
};
if r != 0 {
eprintln!("SO_BUSY_POLL failed: {} (older kernels want CAP_NET_ADMIN; try sudo)",
std::io::Error::last_os_error());
}
}
// Sends a timestamped packet every 500µs from one thread and measures, in the
// chosen receive mode, how long each packet took to reach the receiver.
fn main() {
let mode = std::env::args().nth(1).unwrap_or_else(|| "block".into());
let rx = UdpSocket::bind("127.0.0.1:0").unwrap();
let tx = UdpSocket::bind("127.0.0.1:0").unwrap();
tx.connect(rx.local_addr().unwrap()).unwrap();
match mode.as_str() {
"busypoll" => set_busy_poll(&rx, 200),
// Non-blocking recv never parks the thread. If the queue is empty it
// returns instantly with WouldBlock — "nothing yet, ask again" — and
// what happens next becomes OUR decision instead of the kernel's.
"spin" => rx.set_nonblocking(true).unwrap(),
_ => {}
}
// Instant is Copy: each thread carries its own copy of the SAME stopwatch
// start, so their elapsed() readings share one zero and can be subtracted.
let epoch = Instant::now();
let sender = std::thread::spawn(move || {
for _ in 0..ITERS {
std::thread::sleep(GAP);
let t = epoch.elapsed().as_nanos() as u64;
// to_le_bytes: the u64 as its 8 raw bytes, little-endian
// (least-significant byte first) — our one-line wire format.
tx.send(&t.to_le_bytes()).unwrap();
}
});
let mut buf = [0u8; 8];
let mut samples = Vec::with_capacity(ITERS);
while samples.len() < ITERS {
if mode == "spin" {
loop {
match rx.recv(&mut buf) {
Ok(_) => break,
// WouldBlock: "queue empty right now", delivered as an error
// value — not a failure, just the cue to ask again.
Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {
// We hold the core and ask "anything yet?" millions of
// times per second — polling instead of waiting to be
// woken. spin_loop() is the one courtesy: it tells the
// CPU "this is a poll loop", easing power. It never
// sleeps or yields.
std::hint::spin_loop();
}
Err(e) => panic!("{e}"),
}
}
} else {
rx.recv(&mut buf).unwrap();
}
let sent = u64::from_le_bytes(buf);
samples.push(epoch.elapsed().as_nanos() as u64 - sent);
}
sender.join().unwrap();
hft_lab1::report(&format!("one-way idle-rx [{mode}]"), samples);
}Design notes, because they’re the transferable skill: the 500µs gap guarantees the receiver is idle when each packet lands — you’re measuring the wakeup path, which steady-throughput benchmarks hide. Both modes pay the same sender-side send() cost, so the delta between modes isolates the receive side. Honesty caveat you should repeat in interviews: on loopback there’s no NAPI driver to poll — NAPI being the kernel’s interrupt/poll hybrid for real NICs (ch00b) — so SO_BUSY_POLL is mostly inert here: expect busypoll ≈ block on localhost. The real kernel-busy-poll win only appears on a physical NIC; re-run with rx/tx split across two hosts, or across a veth pair (a virtual ethernet device pair), to see it. The spin mode is the one that shows the sleep tax on any box. Run block vs spin pinned to separate cores (taskset -c 2,3). And watch the units when you compare rows: report prints each value in its own unit — anything under 10µs comes out in ns — so a spin p50 of 4104ns is 4.1µs, beating a block p50 of 14.4us by ~3.5×, even though the digit string looks bigger.
Part E (stretch) — io_uring: fewer syscalls, same stack
The picture first. io_uring replaces the syscall-per-op pattern with two conveyor belts in memory both sides share: you place order forms on the submission queue (SQ) — “recv on this socket, into this buffer” — and the kernel drops finished-work receipts on the completion queue (CQ): “done, 8 bytes.”
classic (Parts A–D): one crossing PER OP N ops = N crossings
USERSPACE │ syscall boundary │ KERNEL
recv() ────────────────────┼─────────────────►│ do the op, come back
recv() ────────────────────┼─────────────────►│ do the op, come back ...
io_uring: the rings straddle the boundary — shared memory both sides can touch
USERSPACE │ │ KERNEL
1 push order form ─────► [ SQ ring — ON the boundary ] ◄─── kernel reads forms
3 pop receipt ◄───────── [ CQ ring — ON the boundary ] ◄─── kernel writes receipts
2 submit_and_wait ─────────┼─────────────────►│ drain SQ, run ops, fill CQ
│ N ops = 1 crossing (SQPOLL: 0 — kernel polls SQ)
1 = ring.submission().push(&e) — a plain memory write, no crossing
2 = ring.submit_and_wait(1) — the ONE syscall in the loop
3 = ring.completion().next() — a plain memory read, no crossing
src/bin/uring_recv.rs (Linux 5.6+):
// One submit_and_wait replaces the recv syscall-per-packet pattern.
use io_uring::{opcode, types, IoUring};
use std::net::UdpSocket;
use std::os::unix::io::AsRawFd;
use std::time::Instant;
// Sends 100k ticks to itself, receiving each via an io_uring submission
// instead of a recv() syscall, and reports the one-way latency distribution.
fn main() -> std::io::Result<()> {
let rx = UdpSocket::bind("127.0.0.1:0")?;
let tx = UdpSocket::bind("127.0.0.1:0")?;
tx.connect(rx.local_addr()?)?;
let mut ring = IoUring::new(8)?;
let mut buf = [0u8; 64];
let mut samples = Vec::with_capacity(100_000);
for _ in 0..100_000 {
// Fill out one order form for the submission belt: "recv on this fd,
// into this buffer". user_data is your tag — it comes back stamped on
// the receipt, so you can match receipts to forms when many are in flight.
let e = opcode::Recv::new(types::Fd(rx.as_raw_fd()),
buf.as_mut_ptr(), buf.len() as u32)
.build().user_data(1);
// `unsafe` = "compiler, trust me": the form carries a raw pointer to buf,
// and Rust can't see when the kernel finishes writing there. We promise
// not to move, reuse, or free buf until the receipt comes back.
unsafe { ring.submission().push(&e).expect("sq full") };
let t0 = Instant::now();
tx.send(b"tick")?;
ring.submit_and_wait(1)?; // ONE syscall: hand over the belt AND wait for a receipt
let cqe = ring.completion().next().expect("cqe");
assert!(cqe.result() > 0);
samples.push(t0.elapsed().as_nanos() as u64);
}
hft_lab1::report("io_uring recv one-way", samples);
Ok(())
}As-is this shows parity, not victory — one op per submit means one syscall either way. The wins arrive when you go further (good exercises): multishot recv (opcode::RecvMulti + buffer rings — arm the receive once and completions keep flowing without resubmitting), batching N submissions per syscall, SQPOLL (a kernel thread polls your SQ — zero steady-state syscalls), and registered buffers (described to the kernel once, not re-validated per op). Remember the bypass-landscape framing (ch04): the packet still walks the whole kernel stack — io_uring economizes the doorway, not the hallway.
Part F — The tuning dividend: put the kernel-tuning chapter on a scale
Everything so far measured the stock kernel. The kernel-tuning chapter (ch03) claims its knobs buy you the tail — never quote that claim unmeasured either. No new Rust here: you re-run Part A and Part D’s binaries while turning ch03’s knobs one at a time, and watch which percentile moves.
First, discover what your box even allows — on a cloud VM, several knobs simply aren’t yours, and that observation is Part F’s first result (the NIC-internals chapter’s cloud lesson, ch05: the hypervisor owns the floor):
sudo apt-get install -y stress-ng linux-cpupower
sudo cpupower frequency-info 2>/dev/null | grep -iA1 governor # governor visible? settable?
ls /dev/cpu_dma_latency 2>/dev/null || echo "no C-state control here (VM?)"
cat /sys/devices/system/cpu/cpu0/cpuidle/state*/name 2>/dev/null # which sleep states exist?
F1 — C-states and the governor (bare metal, or a VM that exposes them). Baseline: record Part D block mode’s p50/p99.9. Then cap sleep depth — holding /dev/cpu_dma_latency open with a zero written to it tells the kernel “no sleep state with more than 0µs wake latency” for as long as the file stays open (the file-as-lease mechanism from ch03):
sudo sh -c 'exec 3<>/dev/cpu_dma_latency; printf "\x00\x00\x00\x00" >&3; sleep infinity' &
CAP=$!
sudo cpupower frequency-set -g performance 2>/dev/null
taskset -c 2 cargo run --release --bin busy_poll -- block # 2-core box: -c 1
kill $CAP # releasing the file releases the C-state cap
Read the result like ch03 taught: p50 barely moves; p99.9 collapses. The median wakeup was already from a shallow state; the tail was the occasional deep-C6 exit (~40–130µs), and you just made deep sleep illegal. If block mode’s tail now approaches spin mode’s, you have measured exactly what spinning was buying you — and what a C-state cap buys instead, without burning the core.
F2 — Pinning and isolation under load (works on any box, including 2 vCPUs). The enemy here is the scheduler’s freedom to put the load where you are:
run B — fence up (4+ cores shown; on 2 cores it's core 0 vs core 1):
core 0 core 1 │ core boundary (taskset fence) │ core 2 core 3
stress-ng stress-ng │ scheduler may not place │ busy_poll (idle)
worker worker │ either side's threads │ rx thread
(--taskset 0,1) │ across this line │ (taskset -c 2)
run A — no fence: same cores, boundary erased — load and receiver mix freely
A/B it:
# A: unpinned receiver, load everywhere — the scheduler mixes them freely
stress-ng --cpu $(nproc) --timeout 70 &
cargo run --release --bin busy_poll -- block
# B: same load, but fenced — load on core 0, receiver pinned to core 1
# (4+ cores: stress on 0-1 with --taskset 0,1, lab pinned to 2)
stress-ng --cpu 1 --taskset 0 --timeout 70 &
taskset -c 1 cargo run --release --bin busy_poll -- block
Expected: run A’s p99.9 explodes (your receiver queues behind stress workers for whole scheduler timeslices — milliseconds); run B pulls the tail most of the way back to the quiet-box number, using nothing but placement. That is isolcpus in miniature: the boot flag makes this fencing permanent and kernel-enforced instead of per-command and advisory.
F3 — What loopback can’t show you. IRQ affinity, coalescing, GRO, flow steering — the NIC-side half of ch03 — are invisible here by construction: loopback has no NIC, no IRQs, no rings. Say that out loud in an interview when you present these numbers; the two-host version of this lab (real NIC, ethtool -C/-K, /proc/irq/*/smp_affinity) is where those knobs become measurable. A null result you can explain beats a positive one you can’t — same lesson as SO_BUSY_POLL in Part D.
Record everything in one table — stock / C-state-capped / pinned-under-load — per mode. The artifact you want at the end is one sentence with three numbers in it: “blocking receive went from p99.9 of X stock, to Y with sleep states capped, to Z when I fenced the load — the median never moved; tuning buys the tail.”
Part G (stretch) — Real bypass: AF_XDP on a virtual wire
io_uring economized the doorway. This part actually skips the hallway: an AF_XDP socket (ch04’s express chute) receiving raw frames into a UMEM your process owns — running against a veth pair, a virtual ethernet cable, so you can do it on any Linux box or VM without touching the interface your SSH session rides on. The A/B: the same UDP sender, received two ways — once through the kernel stack, once through the chute:
═══ wire (veth va) ═══ the same frame, two arms
ARM A — kernel socket (veth_recv) │ ARM B — AF_XDP (xdp_rx)
KERNEL │ KERNEL (driver hook only)
1 alloc sk_buff, copy frame in ✂ │ 1' XDP hook fires in the driver —
2 walk the IP/UDP stack ✂ │ before any sk_buff exists
3 socket lookup, queue to rx ✂ │ 2' frame lands in a UMEM frame —
4 wakeup: schedule the thread ✂ │ YOUR memory, already mapped into
── syscall boundary ── │ your process: nothing crosses
5 recv() returns, copy to user │ 3' descriptor slip → RX ring
USERSPACE │ (shared memory ON the boundary)
6 payload handed to you, │ USERSPACE
already parsed by the kernel │ 4' harvest the slips off the ring
│ 5' read the frame in place
│ 6' parse eth/ip/udp yourself @42
✂ = stage arm B deletes │ no sk_buff, no stack walk, no wakeup
A: 1–4 happen behind the scenes; 5 = rx.recv(), the one call in veth_recv.rs
B: 4' = rx_q.poll_and_consume, 5' = umem.data(d), 6' = the PAYLOAD_AT parse
Honesty up front: on veth there’s no real NIC, so AF_XDP runs in copy (“SKB”) mode — you get the full programming model (UMEM, fill/RX rings, frames-not-sockets, you-are-the-parser) and a real syscall/wakeup win, but not the DMA-into-your-memory zero-copy numbers a physical NIC gives. This is the flight simulator: every control is real, the physics are approximated.
Setup — the wire, and a room at the far end of it (a network namespace, so the kernel actually routes packets over the veth instead of short-circuiting via loopback):
# system packages Part G's build wants (bindgen + libxdp build chain):
sudo apt-get install -y clang llvm libelf-dev gcc make m4 pkg-config
# if the build still asks for libxdp explicitly: sudo apt-get install -y libxdp-dev
sudo ip link add va numrxqueues 1 numtxqueues 1 type veth peer name vb numrxqueues 1 numtxqueues 1
sudo ip netns add lab1
sudo ip link set vb netns lab1
sudo ip addr add 10.77.0.1/24 dev va && sudo ip link set va up
sudo ip netns exec lab1 ip addr add 10.77.0.2/24 dev vb
sudo ip netns exec lab1 ip link set vb up
sudo ip netns exec lab1 ip link set lo up
# Static ARP: once the XDP program owns va's queue it swallows EVERYTHING —
# including ARP requests — so the far side must not need to ask.
MAC_A=$(cat /sys/class/net/va/address)
sudo ip netns exec lab1 ip neigh replace 10.77.0.1 lladdr $MAC_A dev vb
src/bin/veth_send.rs — the constant across the A/B — a plain UDP sender stamping the cross-process monotonic clock:
// Run INSIDE the namespace: sudo ip netns exec lab1 ./target/release/veth_send
use std::net::UdpSocket;
use std::time::Duration;
// Sends 20k UDP packets across the veth, each carrying a mono_ns() timestamp
// the receiving process can diff against its own clock.
fn main() {
let tx = UdpSocket::bind("10.77.0.2:0").unwrap();
tx.connect("10.77.0.1:7777").unwrap();
for _ in 0..20_000 {
std::thread::sleep(Duration::from_micros(500)); // idle receiver, like Part D
tx.send(&hft_lab1::mono_ns().to_le_bytes()).unwrap();
}
}src/bin/veth_recv.rs — arm A, the kernel-stack path:
// The control: same wire, same sender, ordinary blocking socket.
use std::net::UdpSocket;
// Receives 20k timestamped packets through the normal kernel stack and
// reports one-way latency (receive time minus the sender's stamp).
fn main() {
let rx = UdpSocket::bind("10.77.0.1:7777").unwrap();
let mut buf = [0u8; 8];
let mut samples = Vec::with_capacity(20_000);
while samples.len() < 20_000 {
rx.recv(&mut buf).unwrap();
let sent = u64::from_le_bytes(buf); // reverse of to_le_bytes: 8 bytes -> u64
// saturating_sub: clamp at 0 instead of wrapping if clocks disagree slightly.
samples.push(hft_lab1::mono_ns().saturating_sub(sent));
}
hft_lab1::report("veth one-way, kernel socket", samples);
}src/bin/xdp_rx.rs — arm B, the chute. Hold the whole mechanism as one picture before reading a line of it. You allocate one big slab of your own memory — the UMEM, think a single Buffer.alloc() done once at startup — and chop it into 4096 fixed-size frames: empty envelopes. Two shared queues connect you to the driver (each is a ring: a conveyor loop in memory both sides can see). On the fill ring you hand the driver your empty envelopes. When a packet arrives, the driver writes it straight into one of them and drops a slip in your tray — the RX ring. A slip (a descriptor) never carries the packet itself, only “envelope at offset N, M bytes used.”
YOUR PROCESS DRIVER
┌───────────────────┐ fill ring ┌─────────────┐
│ UMEM: 4096 │ ──empty envelopes──► │ packet in? │
│ fixed-size │ │ write it │
│ frames │ ◄──slips: "envelope │ into an │
│ (your memory) │ N, M bytes"───── │ envelope │
└───────────────────┘ RX ring └─────────────┘
read the payload in place, then return the
envelope to the fill ring — the loop never allocates
The listing follows the hello_xdp example that ships with the xsk-rs crate (pinned at 0.8); if the API has drifted by the time you run this, the crate’s examples/ directory is the source of truth — the shape above is the lesson. Every unsafe in it means the same thing: the rings traffic in raw offsets into your UMEM, and it’s on you — not the compiler — to only hand over slips (descriptors) that really point at envelopes you own:
// Run as root: sudo ./target/release/xdp_rx va
// Build: cargo build --release --features xdp
use xsk_rs::{config::{SocketConfig, UmemConfig}, Socket, Umem};
const FRAMES: u32 = 4096;
const ITERS: usize = 20_000;
// eth(14) + ipv4(20) + udp(8): you own the protocol stack now — "parse UDP"
// is a pointer offset. This is ch04's "you rebuild the mailroom's services".
const PAYLOAD_AT: usize = 42;
// Attaches an AF_XDP socket to the interface, harvests 20k raw ethernet frames
// straight from the driver hook, parses the UDP payload itself, and reports latency.
fn main() {
let iface = std::env::args().nth(1).unwrap_or_else(|| "va".into());
// The envelope slab: one allocation, chopped into FRAMES envelopes, plus the
// stack of slips (descs) that point into it. This memory is YOURS — packets
// will land here without ever living in a kernel buffer.
// (try_into().unwrap(): converts usize -> the exact integer type the API
// wants, panicking only if it wouldn't fit — it always fits here.)
let (umem, mut descs) =
Umem::new(UmemConfig::default(), FRAMES.try_into().unwrap(), false).unwrap();
// Socket on (interface, queue 0). fq is the fill ring from the diagram —
// your empty-envelope belt. cq is its TX-side counterpart (the completion ring,
// for sends; unused here). The ch04 ring pairs, live.
let (_tx_q, mut rx_q, fq_and_cq) =
Socket::new(SocketConfig::default(), &umem, &iface.parse().unwrap(), 0).unwrap();
let (mut fq, _cq) = fq_and_cq.expect("fill/comp rings present when umem is unshared");
// Hand the driver the entire stack of empty envelopes up front.
// `unsafe` = "compiler, trust me": we vouch every slip points at a UMEM
// envelope nobody else is using — ring math the compiler can't check.
unsafe { fq.produce(&descs) };
let mut samples = Vec::with_capacity(ITERS);
while samples.len() < ITERS {
// Check the tray (5ms timeout): take a batch of slips off the RX ring;
// each one now points at an envelope the driver has filled.
// (unsafe: descs must be scratch space we own for the slips to land in.)
let n = unsafe { rx_q.poll_and_consume(&mut descs, 5).unwrap() };
for d in &descs[..n] {
// Open the envelope: a raw view into the UMEM at the slip's offset.
// (unsafe: sound only because d is a slip the RX ring just handed
// us — the compiler can't know that; we can.)
let frame = unsafe { umem.data(d) };
let bytes = frame.contents();
if bytes.len() >= PAYLOAD_AT + 8 {
let sent = u64::from_le_bytes(bytes[PAYLOAD_AT..PAYLOAD_AT + 8].try_into().unwrap());
samples.push(hft_lab1::mono_ns().saturating_sub(sent));
}
}
// Close the loop: the envelopes we just read go back on the fill ring,
// empty again. This is the entire allocation story — there isn't one.
unsafe { fq.produce(&descs[..n]) };
}
hft_lab1::report("veth one-way, AF_XDP (copy mode)", samples);
}Run the A/B (build first: cargo build --release --features xdp):
# Arm A — kernel stack:
./target/release/veth_recv &
sudo ip netns exec lab1 ./target/release/veth_send
# Arm B — kill arm A first (the XDP program will steal its packets anyway):
sudo ./target/release/xdp_rx va &
sudo ip netns exec lab1 ./target/release/veth_send
# Teardown when done:
sudo ip netns del lab1 && sudo ip link del va 2>/dev/null
What to expect and how to read it: arm A lands near Part D’s block numbers (it is Part D over a virtual wire). Arm B typically lands in spin-mode territory with a flatter tail — it took the right-hand arm of the diagram, every ✂ stage gone. What you should narrate, though, is what your hands just did: posted empty frames to a fill ring, harvested raw ethernet off an RX ring, parsed UDP at byte 42 yourself, and recycled frames — that is the ch04 model executed, and it’s the same shape DPDK and ef_vi have. Copy mode on a veth is the mechanism without the magnitude; on a real NIC in zero-copy mode with a busy-polling core, this same code shape is the sub-2µs path — and you now know precisely which stages it deleted, because you measured them one at a time in Parts A–F.
Expected results
Reference: bare-metal-ish 3–4GHz x86_64, Linux 6.x, mitigations on, quiet cores. Cloud VMs: p50 similar-to-2x, tails 5–20x worse.
| Measurement | Typical p50 | Typical p99.9 | What it proves |
|---|---|---|---|
| A: UDP RTT loopback (blocking) | 8–25µs | 30–200µs | Kernel software path alone ≫ a 5µs HFT budget (ch01) |
| B: TCP rtt, nodelay on | 10–30µs | 50–300µs | Healthy small-message TCP ≈ UDP + protocol overhead |
| B: TCP rtt, nodelay off | ~40ms | ~45ms+ | Nagle × delayed-ACK = 1000x from one missing sockopt (ch02) |
| C: getpid syscall | 100–250ns | — | Kernel entry cost; why per-packet syscalls add up (ch01) |
| C: clock_gettime vDSO | 15–25ns | — | Timestamps are ~free → instrument everything |
| C: Instant::now | 20–35ns | — | Rust’s clock = vDSO + small wrapper |
D: one-way, block | 4–15µs | 20–100µs+ | Wakeup + schedule dominates idle-receiver latency (ch01) |
D: one-way, spin | 1–4µs | 5–20µs | Never sleeping removes the biggest, spikiest term (ch03) |
D: one-way, busypoll | ≈ block on loopback | — | SO_BUSY_POLL needs a real NAPI driver — knowing why is the point |
| E: io_uring single-op | ≈ A one-way | — | io_uring ≠ bypass; value is batching/multishot/SQPOLL (ch04) |
| F1: D-block, C-states capped | ≈ stock p50 | collapses toward spin | Tuning buys the tail, not the median (ch03) |
| F2: D-block under load, unpinned | ≈ stock p50 | ms-scale | Scheduler timeslices land in your tail |
| F2: D-block under load, fenced | ≈ stock p50 | ≈ quiet-box tail | Placement alone recovers it — isolcpus in miniature |
| G: veth one-way, kernel socket | ≈ D block | ≈ D block tails | Part D over a real (virtual) wire — the control arm |
| G: veth one-way, AF_XDP copy mode | ≈ D spin | flatter | The chute skips skb/stack/socket/wakeup; mechanism real, magnitude needs a NIC (ch04) |
If your numbers differ by 2–3x, fine — the ordering and the ratios (spin ≪ block; nodelay-off catastrophic; vDSO ≪ syscall) are the results. If ordering differs, debug with the kernel-tuning toolkit (ch03): governor, C-states (ch00a), pinning, noisy neighbors.
One chain from Part D is worth assembling in full before you narrate it: a blocked receiver’s core has nothing to run, so it idles, and an idle core sinks into a C-state — a hardware sleep level (ch00a) that gets cheaper to sit in and more expensive to wake from the deeper it goes. When the packet finally lands, you pay the whole chain in reverse — wake the core, exit the sleep state, reschedule the thread — and that is the 4–15µs (with ugly tails) the block row pays and the spin row deletes.
Narrating this in an interview
The lab’s real product is sentences you’ve earned. The shape that lands:
- Claim → number → mechanism. “Blocking receive into an idle thread cost me ~8µs p50 with 50µs+ tails; spinning on the same socket was ~2µs and flat — that’s the scheduler wakeup and C-state exit, which is why hot paths busy-poll” beats any amount of recited theory, because it’s yours.
- Show calibrated honesty. “I measured this on loopback, which skips the NIC/DMA/IRQ stages — so it’s a floor for the software stack, not a wire number. On hardware I’d expect X, and I’d verify with hardware timestamps (ch05).” Knowing what your benchmark doesn’t show is the senior tell; so is mentioning that SO_BUSY_POLL did nothing on loopback and exactly why.
- Connect to decisions. “getpid cost ~150ns on my box; at one syscall per packet on a million-packet feed that’s 15% of a core before any work — that’s the case for recvmmsg (a single syscall that drains many queued packets at once) or io_uring batching at the gateway tier, and for kernel-bypass rings (ch04) on the true hot path.”
- Tie to your production story. “My prod systems ran ~10ms budgets where venue RTT dominated, so tuned-kernel was the right call — but I’ve measured where the next 10µs live: wakeups, syscalls, Nagle-class footguns, and that’s the order I’d attack them before reaching for DPDK.”
- Close with the ladder you climbed. “Stock kernel, then tuned — C-state cap and core fencing bought back the tail, median untouched — then AF_XDP, where I posted fill-ring frames and parsed UDP off raw ethernet myself. Copy mode on a veth, so I’ll claim the mechanism, not the zero-copy numbers — but I know exactly which stage each rung deleted, because I measured them separately.” That sentence is Part I of this book, compressed.
Plain-English recap
If you remember nothing else from this lab:
- The software stack alone blows the HFT budget: a loopback UDP round trip — no wire, no NIC, pure kernel — costs 8–25µs p50. That’s like discovering your framework’s per-request overhead already exceeds the SLA before your handler runs. This is the packet-path chapter’s claim (ch01), now measured on your box.
- One missing
set_nodelay(true)costs 1000x: nodelay-off turns tens-of-µs round trips into ~40ms. It’s the missing-DB-index of sockets — invisible in code review, catastrophic in production, fixed with one line. - Kernel entries cost real money; clock reads are free: a genuine syscall is ~100–250ns, but
clock_gettime/Instant::nowis ~20ns because the vDSO keeps it in userspace — so timestamp everything, but batch your syscalls. - Sleeping is the tax: a blocked receiver pays 4–15µs (with ugly tails) just to be woken — the cold-start penalty. A spinning receiver gets 1–4µs, flat. Warm worker vs. scale-to-zero, measured.
- A null result you can explain beats a positive one you can’t:
SO_BUSY_POLLdid nothing here because loopback has no driver ring to poll — knowing the mechanism behind the flat line is the difference between running benchmarks and understanding them. - io_uring at one op per submit is parity, not victory — the wins are in batching, multishot, and SQPOLL. It economizes the doorway; the packet still walks the hallway.
- Tuning buys the tail, not the median — capping C-states left p50 alone and collapsed p99.9; fencing the load off your core undid a milliseconds-scale tail with placement alone. The kernel-tuning chapter’s whole thesis, now three numbers in your notebook.
- Bypass is a programming model you have now used: fill ring, RX ring, frames in your own memory, UDP parsed at byte 42 by you. On a veth in copy mode it’s the flight simulator; on a real NIC in zero-copy it’s the sub-2µs path — same code shape.
- Methodology is the transferable skill: warmup, 100k+ samples, percentiles never means, one variable per A/B, and a measuring clock cheap enough not to perturb the measurement.
Interviewer will ask
“How did you measure that? Walk me through the methodology.” Warmup iterations discarded (cold caches, page faults, slow-start effects); 100k+ samples; report full percentiles, never means; rdtsc for ns-scale sections with TSC calibrated against the monotonic clock; pinned cores; separate the thing measured from the measuring clock (vDSO reads at ~20ns so timestamping doesn’t perturb µs-scale results). And each A/B changes exactly one variable — nodelay flag, receive mode — on otherwise identical code.
“Why percentiles and not averages?” Latency is heavy-tailed: p50 of 8µs with 1% of samples at 100µs+ averages to a lie. Trading death lives in the tail — the slow response is correlated with the busy market moment when it costs most. p99.9/max are the engineering targets; the mean is marketing.
“Your loopback RTT was 15µs. What would wire-to-wire on real hardware add or remove?” Remove: nothing — loopback already charges the full software stack twice per round trip. Add: the physical path, per direction. Wire serialization is ~70ns to clock a 64-byte frame onto a 10G link. A cut-through switch hop is ~300–500ns — the switch starts forwarding once it has read the header. NIC internal processing plus PCIe DMA is ~1µs. The IRQ path, if I’m not polling, is another ~1–3µs. So hardware RTT lands around the loopback RTT plus roughly 2–4µs of physical path per direction — the software cost is unchanged, and the total stays consistent with the packet-path chapter’s (ch01) 2–10µs-per-direction kernel figure. And I’d verify it with NIC hardware timestamps, not app clocks.
“Why did busy-poll not beat blocking in your test?” Loopback has no NAPI context to poll — packets are delivered synchronously by the sender’s kernel path, so SO_BUSY_POLL’s driver-spin never engages and both modes reduce to queue-check semantics. On a physical NIC the blocking path eats IRQ + softirq + wakeup while busy-poll spins in the driver ring. Knowing the mechanism behind a null result is the difference between running benchmarks and understanding them.
“What’s wrong with timing a single operation with Instant::now before and after?” Clock read cost (~20–30ns) and its serialization effects swamp ns-scale operations; one sample tells you nothing about a distribution; the compiler may reorder or elide un-black-boxed work; and the first execution measures cold caches, not the operation. Hence: loops, warmup, rdtsc, black_box, percentiles.
“You ran AF_XDP. What did the kernel stop doing for you — and what did you have to start doing?” Stopped: allocating an sk_buff per packet, walking IP/UDP through the stack, the socket lookup and queue, and the wakeup — my frames landed straight in a UMEM my process owns. Started: everything the mailroom used to do — I posted empty frames to the fill ring myself, parsed ethernet/IP/UDP myself at fixed offsets, recycled frames back after reading, and static-ARP’d the far side because my XDP program was swallowing ARP along with everything else on that queue. That last one is the lesson in miniature: bypass takes the whole queue, not just your packets — the ch04 costs are real, and I hit one in a lab within the hour.
Further reading
- rigtorp.se (Erik Rigtorp): the low-latency tuning guide plus his measurement posts and tools — the closest public analog to how HFT shops actually benchmark.
- man pages:
recv(2),socket(7)(SO_BUSY_POLL, SO_RCVBUF),tcp(7)(TCP_NODELAY, TCP_QUICKACK),vdso(7)— the exact semantics behind every knob this lab touches. - Brendan Gregg: Systems Performance ch. 2 (Methodology) and his “Active Benchmarking” material — how to know your benchmark measures what you claim.
io_uringcrate docs (docs.rs) and the “Lord of the io_uring” tutorial — from Part E’s single-op toy to multishot/SQPOLL designs.- Intel’s “How to Benchmark Code Execution Times on Intel IA-32 and IA-64” white paper (Gabriele Paoloni) — the canonical rdtsc-methodology reference: serialization, invariant TSC, and pitfalls.
Where this goes next: Chapter 7: Clocks — this lab trusted Instant::now and a quick TSC calibration; Part II starts by asking whether you should: what do TSC, PTP, and the kernel’s clocks actually guarantee, and whose timestamp can you believe?
Clocks: TSC, PTP, and Lying Timestamps
Before you start — this chapter leans on a handful of primer ideas:
- TSC /
rdtsc— the CPU’s built-in cycle counter and the instruction that reads it, the rawest clock you have: ch00e- Crystals and clock drift — why every clock is a vibrating crystal that runs slightly fast or slow, and what NTP/PTP do about it: ch00e
- Kernel vs userspace and syscalls — why “calling the kernel for the time” used to be expensive and how the vDSO fixed it: ch00b
- The NIC and the PHY — the network card hardware that can stamp packets the instant they hit the wire: ch00a
- Tick-to-trade — the market-data-in to order-out latency this chapter teaches you to decompose: ch00f
Read those first — 20 minutes there saves an hour here.
You have a ~10ms system — the web-stack trading setup you’ve run in production, where venue RTT dominated — and you want to make claims about microseconds. Every one of those claims rests on a clock, and most engineers have never audited theirs. This chapter is about knowing exactly what your timestamps mean, what they cost to take, and when they lie.
The clock hierarchy on one box
Your machine does not have “a clock.” It has one raw counter, then layers of kernel
arithmetic on top of it, then the NIC’s own watch off to the side. The raw counter is
the TSC (the CPU’s cycle counter, read by the rdtsc instruction with no
kernel involved — ch00e): it counts ticks since reset,
costs a few nanoseconds to read, and has no idea what time it is. The kernel takes
that counter and, per clock ID, applies a scale and an offset to turn ticks into
nanoseconds — that’s the whole clock_gettime family. And the NIC keeps a physical
clock of its own, so a packet can be stamped the instant it touches the wire rather
than whenever your software got around to noticing it.
What separates the kernel’s clocks is adjustment policy. A clock is disciplined
when a daemon keeps nudging it so it tracks some reference — like a thermostat,
forever measuring the error and correcting toward the target. The nudge comes in two
flavors: slewing (gently stretch or shrink the length of the second until the
clock catches up — time never jumps) and stepping (yank the hands straight to the
right time — a discontinuous jump that can even go backwards). CLOCK_MONOTONIC_RAW
is the undisciplined hardware rate; CLOCK_MONOTONIC is slewed but never stepped;
CLOCK_REALTIME can be both.
Every kernel clock here is built on the TSC
on a modern x86 Linux box (clocksource=tsc). clock_gettime is the kernel reading
the TSC in the vDSO and applying a scale/offset. When you call rdtsc yourself, you
are just cutting out the middleman — and losing the calibration the kernel maintains.
The whole hierarchy as a summary table, cheapest/rawest to most expensive/most-meaningful:
| Source | Cost to read | What it counts | Lies about |
|---|---|---|---|
rdtsc | ~6–10 cycles (~2–4ns) | CPU reference cycles since reset | Wall time, cross-socket offsets |
rdtscp / lfence; rdtsc | ~20–35 cycles | Same, but ordered | Same |
CLOCK_MONOTONIC_RAW | ~20–30ns (vDSO) | Hardware time, no NTP discipline | Nothing much; drifts vs true seconds |
CLOCK_MONOTONIC | ~20–30ns (vDSO) | Hardware time, NTP-slewed rate | Its “second” stretches under NTP |
CLOCK_REALTIME / gettimeofday | ~20–30ns (vDSO) | Wall clock, NTP-stepped/slewed | Can jump backwards |
| NIC hardware timestamp | free at capture, cost to retrieve | Packet at the PHY/MAC | Nothing — this is ground truth for wire time |
Remaining terms in that table, in one clause each: vDSO (the kernel page
mapped into your process that keeps clock reads out of the kernel —
ch00b);
NTP (network time protocol — sync over the ordinary network, millisecond-class
accuracy — ch00e); “ordered” / lfence (a fence
instruction that stops the CPU reordering the read relative to your work — the
serialization section below); PHY/MAC (the NIC’s physical-layer
and link-layer hardware, the last silicon a packet touches before the wire —
ch00a).
rdtsc: what you must know before using it
Invariant TSC
Old CPUs ticked the TSC at the current core frequency, so it stopped in sleep states and changed speed with turbo. Every CPU you will trade on since roughly ~2008 (Intel Nehalem) has an invariant TSC: it ticks at a fixed frequency (the “TSC frequency”, near the base clock, e.g. 2.994 GHz on a “3.0 GHz” part) regardless of P-states, C-states, or turbo (the CPU’s frequency-scaling and sleep states — ch00a).
Verify, don’t assume:
- CPUID leaf
0x80000007, EDX bit 8 = invariant TSC (cpuidis the x86 instruction that reports the CPU’s features; a “leaf” is just which page of answers you ask for). - Linux:
grep -o 'constant_tsc\|nonstop_tsc' /proc/cpuinfo | sort -u— you want both. cat /sys/devices/system/clocksource/clocksource0/current_clocksourceshould saytsc. If it sayshpetoracpi_pm, the kernel demoted the TSC because it observed it misbehaving — investigate before trusting any timing on that box.
Frequency: never use /proc/cpuinfo MHz
The “cpu MHz” field is the current core frequency, which turbos and idles all over the place. The TSC frequency is a different, fixed number. Get it from:
dmesg | grep 'tsc:'— kernel prints the refined calibration, e.g.tsc: Refined TSC clocksource calibration: 2994.374 MHz.- CPUID leaves
0x15/0x16(crystal clock ratio) on Skylake+. - Or calibrate it yourself against
CLOCK_MONOTONIC_RAW(code below) — this is what you should do anyway, because it makes your code robust and is a one-time startup cost.
Serialization: rdtsc is not ordered
rdtsc is just another instruction to the out-of-order engine (modern CPUs execute
instructions in whatever order keeps the pipeline busy, not program order —
ch00a). The CPU is free to
execute it before the work you’re trying to time has finished, or hoist work from
after it to before it. For coarse pipeline stamps (microseconds apart) this doesn’t
matter — a few nanoseconds of skid is noise. For microbenchmarks of 20-cycle
operations it destroys the measurement.
The modern discipline (Intel’s recommendation since the Paoloni whitepaper era) uses
lfence (a “load fence” instruction — a barrier that stops the CPU reordering
instructions across it; the out-of-order engine is ch00a’s
territory):
start: lfence; rdtsc ; lfence stops earlier insns' results arriving late
end: rdtscp ; waits for all prior insns to retire (fully
; finish and commit their results)...
lfence ; ...and lfence stops later insns starting early
rdtscp is only partially serializing — it waits for prior instructions but does
not fence subsequent ones, hence the trailing lfence. The old recipe — issuing a
cpuid instruction as the barrier, because it fully serializes the pipeline —
works but cpuid costs hundreds of cycles and has variable latency; use lfence.
In Rust:
#![allow(unused)]
fn main() {
#[cfg(target_arch = "x86_64")]
#[inline(always)]
pub fn rdtsc_ordered() -> u64 {
use core::arch::x86_64::{__rdtscp, _mm_lfence, _rdtsc};
unsafe {
_mm_lfence();
let t = _rdtsc();
_mm_lfence();
t
}
}
}For pipeline stamps taken microseconds apart, plain _rdtsc() without fences is fine
and is what you want in the hot path — 2–4ns, no pipeline drain.
Per-core sync caveats
On a single modern socket, cores’ TSCs are synchronized at reset and the hardware
keeps them together; the kernel checks this at boot (tsc: Synchronized across N CPUs)
and via IA32_TSC_ADJUST (a per-core register recording any offset software applied
to that core’s TSC — nonzero means someone shifted it). Practical rules:
- Same socket, pinned threads: comparing TSC values across cores is fine to within a few cycles. This is what makes cross-thread pipeline stamping work.
- Multi-socket: usually still synchronized (same reset signal), but verify; NUMA-era (ch00a — multiple CPU sockets, each with its own local memory) horror stories exist. Run the kernel’s check, or measure a ping-pong round trip and confirm one-way ≈ RTT/2 in cycles both directions — RTT/2 is legitimate here because a same-box ping-pong is symmetric by construction, unlike the cross-network case skewered later in this chapter.
- VMs: all bets off. Live migration rewrites TSC offsets; some hypervisors trap
rdtsc(intercept the instruction and emulate it in software — slowly, which is the whole reason VM timing numbers are worthless). If you’re timing inside a VM, you’re characterizing the hypervisor. - Unpinned threads: a thread migrating mid-measurement between synchronized cores is fine; between unsynchronized sockets it’s not. One more reason you pin.
clock_gettime and the vDSO
clock_gettime(CLOCK_MONOTONIC) does not make a syscall on any kernel you’ll run:
the vDSO maps a page with the TSC scale/offset into your process and the “call” is a
userspace function that does rdtsc, multiply, shift. ~20–30ns. Verify with
strace (the Linux tool that prints every syscall a process makes): you should see
no clock_gettime syscalls in steady state. (Trap:
CLOCK_MONOTONIC_RAW wasn’t in the vDSO until around Linux 4.16 — it fell back to a
real syscall — so on an ancient kernel that “cheap” call is 100ns+ and, being a
kernel entry, a moment where the scheduler may take your core away.)
Distinctions that matter:
CLOCK_REALTIME: wall time. NTP can step it backwards. Never compute a duration from it. Its only job is correlating with the outside world.CLOCK_MONOTONIC: never steps backwards, but NTP slews its rate (stretches or shrinks the second by up to 500ppm) to chase true time. Fine for timeouts; a subtle lie for precision measurement — your “1ms” might be 0.9995ms.CLOCK_MONOTONIC_RAW: the undisciplined hardware rate. This is what you calibrate the TSC against, because both are lies in the same direction. Mechanically: the TSC andCLOCK_MONOTONIC_RAWare derived from the same physical crystal on the board, so if that crystal runs 30ppm fast, both run 30ppm fast together and the ratio between them stays fixed. A calibration againstCLOCK_MONOTONICwould chase NTP’s slew instead.
So is gettimeofday in a hot path a sin? Less than folklore says — it’s a ~25ns vDSO
call now, not a 1µs syscall. But the standard in trading systems is raw rdtsc in the
hot path anyway: it’s 5–10× cheaper, it’s immune to NTP slew, and cycles are the
natural unit when you’re also reading performance counters. Take cycles hot, convert
to nanoseconds cold.
Calibrating cycles → nanoseconds in Rust
#![allow(unused)]
fn main() {
use std::time::Instant;
/// TSC ticks per nanosecond, measured at startup. Do this once, on a pinned
/// thread, and sanity-check against dmesg's "Refined TSC" value.
pub fn calibrate_tsc_ghz() -> f64 {
let mut best = f64::MAX;
for _ in 0..5 {
let t0 = Instant::now();
let c0 = rdtsc_ordered();
// Long enough to swamp the ~20ns measurement edges: 50ms.
while t0.elapsed().as_millis() < 50 {
std::hint::spin_loop();
}
let c1 = rdtsc_ordered();
let ns = t0.elapsed().as_nanos() as f64;
let ghz = (c1 - c0) as f64 / ns;
// Take the minimum-noise (most consistent) sample.
if ghz < best { best = ghz; }
}
best
}
// Usage: ns = cycles as f64 / ghz. Store 1.0/ghz and multiply in the cold path.
}Two production notes: (1) do the division off the hot path — stamp raw cycles, convert during aggregation; (2) recheck the calibration periodically in long-running processes and alarm if it moves — a shifting apparent TSC rate means the crystal feeding your reference clock has heat-shifted (crystals speed up and slow down with temperature — ch00e) or there’s a clocksource problem.
Cross-machine time: where the real lies live
Everything above was one box. The moment your latency claim spans two machines — “exchange gateway to our server in 40µs” — you need both clocks to agree, and this is where most published numbers are fiction.
NTP: ±milliseconds, and it won’t tell you
NTP over a LAN under good conditions gets you within tens to hundreds of
microseconds; over anything congested or asymmetric, single-digit milliseconds of
error is routine — and NTP happily reports itself “synchronized” the whole time. If
your one-way latency claim is 50µs and your clock error budget is ±1ms, your
measurement is 100% noise. Any cross-machine latency figure derived from
NTP-disciplined CLOCK_REALTIME deserves exactly zero trust below the millisecond.
The RTT/2 fallacy
The tempting dodge: measure round trip with one clock, divide by two. This assumes the path is symmetric. In trading infrastructure it reliably isn’t: different fiber routes in each direction, asymmetric queuing (your order enters a busy gateway, the ack returns on an idle path), different switch hop counts, NIC send vs receive path costs. Asymmetries of 2:1 are common. RTT is a real, useful number — quote it as RTT. One-way numbers require synchronized clocks, full stop.
PTP: tens of nanoseconds
IEEE 1588 (PTP) with hardware timestamping gets machines within tens to hundreds of nanoseconds of each other:
- The NIC’s PHY stamps sync packets on the wire (removing OS jitter from the sync loop entirely), maintaining a PHC (PTP Hardware Clock — an actual clock that lives on the NIC itself, separate from the system clock — ch00e).
ptp4ldisciplines the PHC to the grandmaster — the one reference clock in the network that everyone chases;phc2sysdisciplines the system clock to the PHC. Boundary/transparent clocks in the switches correct for queuing delay on the sync path itself.- Software-timestamped PTP is a halfway house: ~µs-tens-of-µs accuracy. Better than NTP, not good enough to decompose a 10µs path.
The whole sync chain on one screen:
GRANDMASTER ──► switch ──────────────► NIC's PHC ──────────► system clock
(the reference (boundary/transparent (clock chip on (what clock_gettime
everyone clock: corrects for the NIC) reads)
chases) its own queuing delay)
└── ptp4l disciplines ──┘ └── phc2sys disciplines ──┘
Colos serving exchanges run PTP infrastructure precisely because clients demand defensible one-way numbers. If you’re asked “how would you verify a vendor’s claimed one-way latency” — the answer is PTP-disciplined hardware timestamps at both ends, or you refuse to state one-way numbers and quote RTT.
NIC hardware timestamps
Independent of PTP, the NIC can stamp your traffic: enable SO_TIMESTAMPING with
SOF_TIMESTAMPING_RX_HARDWARE / TX_HARDWARE (config via ethtool -T to check
capability, hwtstamp_config to enable) and each packet arrives with the PHC time it
hit the wire, delivered in the socket’s error queue / control messages (a side
channel on the socket where the kernel attaches per-packet metadata — despite the
name, nothing has gone wrong). Kernel-bypass
stacks (Onload, ef_vi, DPDK) surface the same hardware stamps directly. This is the
only timestamp that is not polluted by interrupt latency, softirq scheduling (the
kernel’s deferred packet-processing work — ch00b),
or your process getting around to calling recv.
Timestamping discipline for a trading pipeline
The four stamps that decompose tick-to-trade, and the clock each uses:
| # | Stamp | Clock | What it captures |
|---|---|---|---|
| 1 | t_wire_in | NIC hardware (PHC) | Market data packet hits your NIC |
| 2 | t_recv | rdtsc | Your thread has the packet in hand |
| 3 | t_decision | rdtsc | Strategy decided; order constructed |
| 4 | t_wire_out | NIC hardware TX stamp | Order left your NIC |
The three deltas: (1→2) is your network stack + wakeup cost — this is the delta where kernel vs
bypass shows up, and the one most people have never measured. (2→3) is your
code — the only part your Rust hot path controls. (3→4) is the send-side stack.
End-to-end (1→4) is the number you quote; the decomposition is the number you debug.
To mix PHC stamps with rdtsc stamps you need the PHC↔TSC relationship, which
phc2sys maintains (or sample both at effectively one instant yourself — read
clock A, read clock B, read A again, and pair B with the midpoint of the two A
readings — and keep the offset).
Anyone who quotes tick-to-trade without saying which two of these points they measured between is quoting an incomparable number. “1.2µs tick-to-trade” measured 2→3 is a completely different claim from 1→4.
Plain-English recap
- The clock hierarchy is like timestamp columns in a payments system. A payment
has a
created_atfrom the client, one from your API server, one from the database, and one from the PSP — four different clocks meaning four different things.rdtsc,CLOCK_MONOTONIC,CLOCK_REALTIME, and the NIC stamp are the same idea; the sin is reading one column and thinking it means another. - rdtsc vs
CLOCK_REALTIMEisperformance.now()vsDate.now(). One is a raw monotonic tick counter that’s cheap and never jumps; the other is wall time that NTP can yank around. You’d never compute a duration fromDate.now()across a DST change —CLOCK_REALTIMEdurations are the same bug at microsecond scale. - NTP error swamping your measurement is a reconciliation problem. It’s like diffing your ledger against a PSP settlement report where each side stamped events with its own clock, off by an unknown couple of seconds: you cannot order events across the two systems, no matter how precise each timestamp looks. PTP is both sides agreeing to a shared, audited clock before anyone compares timestamps.
- The RTT/2 fallacy is “webhook delivery time = API round trip ÷ 2”. Your request went out over one path and the webhook came back over a completely different one (different queues, different retries). Halving the round trip assumes symmetry that isn’t there.
- NIC hardware timestamps are the PSP’s own
received_at. Stamped at the front door, not when your worker finally pulled the job off the queue. Every software timestamp includes “how long until my process got around to it”; the wire stamp doesn’t — that’s why it’s ground truth. - The four tick-to-trade stamps are spans in an APM trace. Wire-in → thread has it → decision made → wire-out is exactly a Datadog trace of gateway → worker → handler → response. Quoting “latency” without saying which two spans you measured between is as meaningless in trading as it is in an APM dashboard.
Interviewer will ask
Q: Why can’t you measure a 50µs latency with
SystemTime::now()? A:SystemTimeis Rust’sDate.now()— it readsCLOCK_REALTIME, the wall clock NTP is allowed to yank around. NTP can slew or even step that clock mid-measurement, so end−start isn’t a duration — it’s two wall-time readings with an adjustment of unknown size hiding between them. And that adjustment is millisecond-class, while the thing being measured is 50µs — the error bar is bigger than the measurement. So single-box durations come from the TSC orCLOCK_MONOTONIC_RAW, the clocks nothing yanks; cross-machine one-way numbers need PTP or they’re fiction.Q: What must be true before you trust raw rdtsc as a clock? A: Each condition rules out one specific way the counter lies. It must tick at a fixed rate through turbo and sleep — invariant TSC — checked via
constant_tscandnonstop_tscin /proc/cpuinfo (CPUID leaf 0x80000007 if you want the hardware’s own word). The kernel must still trust it too: ifcurrent_clocksourcesays hpet instead of tsc, the kernel caught the TSC misbehaving and demoted it. I need its real frequency, not a guess — dmesg’s refined calibration or my own againstCLOCK_MONOTONIC_RAW, never /proc/cpuinfo’s MHz field, which is the wandering core frequency. My threads must be pinned, because cross-socket TSC agreement is verify-not-assume. And on a VM none of this holds — live migration rewrites TSC offsets — so there I don’t time with rdtsc at all.Q: When do you fence rdtsc, and when is it a waste? A: rdtsc is unordered — the out-of-order engine can hoist it before the work under test finishes. For a microbenchmark of a 20-cycle operation that skid is the whole measurement, so:
lfence; rdtscat the start,rdtscp; lfenceat the end (rdtscp only waits for prior instructions, hence the trailing lfence). For pipeline stamps microseconds apart, a few ns of skid is noise and the fences’ pipeline drain costs more than the error — plain_rdtsc()in the hot path, fenced variants in benches.Q: Your two servers are NTP-synced. Can you quote “gateway to server in 40µs”? A: No. NTP’s actual error on a LAN is tens to hundreds of µs, and it reports “synchronized” regardless — a 40µs claim with a ±500µs clock is 100% noise. I either quote the round trip measured on one clock, as RTT, or I get both ends onto PTP with hardware timestamping — PHC-stamped at the PHY, ptp4l to the grandmaster — which brings inter-machine error to tens of nanoseconds and makes a one-way number defensible.
Q: Why not just measure RTT and divide by two? A: Because the path isn’t symmetric and in trading infrastructure it reliably isn’t: different fiber routes each way, asymmetric queuing (busy ingress gateway, idle return path), different hop counts. 2:1 asymmetries are common, so RTT/2 can be off by half the RTT. RTT is a real number — I quote it as RTT. One-way numbers require synchronized clocks, full stop.
Q: Hardware vs software timestamps — what does the NIC stamp buy you? A: The NIC stamp is the PSP’s own
received_at— stamped at the front door, not when my worker finally pulled the job off the queue. A software stamp is taken whenever my code ran, so it silently includes interrupt latency, softirq scheduling, and scheduler mood. The NIC’s PHY stamps the packet against the PHC the instant it hits the wire, before any of that can pollute it. So the delta between the wire stamp and my first software stamp is my network-stack-plus-wakeup cost — the number that justifies (or kills) a kernel-bypass project. Enable it with SO_TIMESTAMPING; bypass stacks surface the same stamps directly.Q: Decompose tick-to-trade for me. Which clocks, which stamps? A: It’s an APM trace with four spans, so: four stamps, three deltas, then endpoint honesty. The stamps: t1 wire-in on the NIC’s hardware clock (PHC), t2 packet-in-hand and t3 decision-made on rdtsc, t4 wire-out on the NIC’s TX stamp. The deltas each blame one owner: 1→2 is network stack plus wakeup, 2→3 is my code, 3→4 is the send-side stack — and mixing PHC with rdtsc stamps needs the PHC↔TSC offset that phc2sys maintains. Endpoint honesty: quote 1→4, debug with the deltas, and never state a tick-to-trade figure without naming its two endpoints — “1.2µs” measured 2→3 differs from 1→4 by the entire network stack.
Q: I don’t believe your 8µs number. Convince me. A: A number is the output of a measurement, so I defend the measurement — and a measurement is trusted through exactly three things: its clock, its endpoints, and its reproducibility. The clock: raw TSC with invariant-TSC verified, calibration cross-checked against dmesg’s refined value, kernel clocksource still tsc. The endpoints: I say which two of the four stamps, and they’re wire-referenced — NIC hardware stamps, not when my process got around to it. The reproducibility: the whole distribution, worst case included — next chapter’s machinery — under replayed production bursts, plus the recipe: same capture, same pinned cores, run it yourself. Show all three and 8µs stops being a claim and becomes a result; the strongest move is handing over the capture file so the skeptic reproduces it.
Further reading
- Intel SDM, Volume 3, the Time-Stamp Counter section (invariant TSC, IA32_TSC_ADJUST) — plus Gabriele Paoloni’s Intel whitepaper “How to Benchmark Code Execution Times”, the source of the lfence/rdtscp discipline.
- IEEE 1588 / PTP overviews — the linuxptp project’s documentation (linuxptp.org)
is the practical entry point;
ptp4l(8)andphc2sys(8)man pages for the actual sync chain. clock_gettime(2),vdso(7), andtime(7)man pages — the authoritative word on what each clockid means and which calls avoid the syscall.- The Linux kernel’s timestamping documentation
(
Documentation/networking/timestamping.rst) — SO_TIMESTAMPING, hardware RX/TX stamps, and PHC plumbing. - Gil Tene, “How NOT to Measure Latency” — the methodology talk the next chapter builds on; watch it before making any latency claim in an interview.
Where this goes next: you can now take honest timestamps — Chapter 8 is about turning millions of them into honest statistics: percentiles, HdrHistograms, and the coordinated-omission trap that invalidates most published latency numbers.
Latency Methodology
Before you start — this chapter leans on a handful of primer ideas:
- Percentiles and why averages lie — p50/p99/p99.9 as the vocabulary of latency, and what a heavy tail is: ch00e
- HdrHistogram — the fixed-memory histogram that records billions of samples without keeping them: ch00e
- Coordinated omission — the load-testing bug where you stop measuring exactly when the system is worst: ch00e
- TSC / rdtsc stamps — where the raw timestamps come from (and the calibration discipline of the clocks chapter, ch07): ch00e
- The jitter suspects — page faults, IRQs, scheduler preemption and friends, the kernel-side causes of tail spikes: ch00b
Read those first — 20 minutes there saves an hour here.
You know how to build a fast system. This chapter is about how to know it’s fast — which is a different skill, and the one interviewers use to separate people who have operated latency-critical systems from people who have merely written them.
Distributions, not averages
An average latency is close to useless in trading, for two reasons:
- Latency distributions are heavy-tailed. Your ~10ms system almost certainly has a p50 nowhere near 10ms; the mean is dragged around by a small number of huge outliers, and it tells you nothing about either the common case or the bad case.
- The money is in the tail. In a competitive strategy, the distribution of your latency relative to competitors determines fill rates. One slow order during a volatile window isn’t a rounding error — it’s a fill you missed or adverse selection (getting filled at a stale price) you ate. A system with p50=5µs, max=50ms can lose more money than one with p50=8µs, max=100µs.
So you report and track: p50, p90, p99, p99.9, p99.99, max (percentiles — p99 is the value 99% of samples fall under; ch00e if the notation is new) — and you treat the right side of that list as more important than the left. p50 tells you about your architecture. p99.9 and max tell you about your discipline: allocator behavior, page faults, scheduler interference, GC-like stalls in dependencies. In trading interviews, saying “our p50 was X” and stopping is a red flag; saying “p50 X, p99.9 Y, max Z, and here’s what the max was caused by” is the credential.
Why max specifically, when statisticians hate it? Because at realistic sample counts it’s not noise. At 100k events/sec, a full trading day is ~2.3 billion samples; your p99.99 covers all but 230,000 of them. The max is the answer to “what is the worst thing my system actually did today,” and in trading that question has a dollar value.
One more piece of tail arithmetic, because it changes what “1% of requests” means: percentiles are per-request, but users — and strategies — experience sessions, runs of many requests in a row. A session of 100 requests dodges the p99 only if every single one does: probability 0.99¹⁰⁰ ≈ 37%. So ~63% of sessions eat at least one p99 event — “1% of requests” quietly becomes “most sessions.” The trading version is a burst: 500 correlated ticks in a volatile window almost certainly contain your p99.5, so the “rare” tail is effectively guaranteed during exactly the multi-message moments that matter.
HdrHistogram mechanics
Recording billions of samples means you can’t keep them all; you need a histogram. Naive linear buckets force a resolution/range tradeoff. HdrHistogram (High Dynamic Range histogram — ch00e), from Gil Tene, solves it with logarithmic bucketing with linear sub-buckets:
- You declare a range (say 1ns to 60s) and a precision in significant figures (usually 3).
- Values are bucketed so that relative error is bounded: 3 sig figs means any recorded value is within 0.1% of the true value. 1.000µs and 1.001µs land in different buckets; 10.000ms and 10.001ms don’t need to.
- Mechanically: the exponent range is covered by log2 buckets, each subdivided into
2^n linear sub-buckets; bucket index is found with a couple of shifts and a
leading_zeros— recording is O(1), a handful of nanoseconds, no allocation after construction. - Memory is fixed and small (tens of KB for ns→minutes at 3 sig figs), histograms are mergeable (per-thread histograms combined off-path), and percentile queries are cheap iterations.
Concretely, recording a 1,250ns sample: leading_zeros (a hardware instruction that
finds the highest set bit of a number — which is its log2) puts 1,250 in the major
bucket covering [1024, 2048); a shift of the remaining bits picks the linear
sub-bucket inside that range; one counter at one array index is incremented. That’s
the entire record path — no search, no allocation, one memory write.
#![allow(unused)]
fn main() {
use hdrhistogram::Histogram;
// 1ns to 60s range, 3 significant figures.
let mut h = Histogram::<u64>::new_with_bounds(1, 60_000_000_000, 3).unwrap();
h.record(1_250).unwrap(); // 1.25µs, in ns
println!(
"p50={}ns p99={}ns p99.9={}ns max={}ns",
h.value_at_quantile(0.50),
h.value_at_quantile(0.99),
h.value_at_quantile(0.999),
h.max()
);
}The shape of the whole pipeline, from a stamped event to a number on a dashboard:
HOT PATH (per event, ~ns) COLD PATH (per second, off the hot core)
───────────────────────── ───────────────────────────────────────
t0 = rdtsc() ┌──────────────┐
…work… │ thread A hist│──┐
t1 = rdtsc() ├──────────────┤ │ merge
│ │ thread B hist│──┼──────────► merged
│ cycles (u64) ├──────────────┤ │ histogram
▼ │ thread C hist│──┘ │
┌─────────────────┐ └──────────────┘ │
│ per-thread │ record() is O(1): shift, clz, query percentiles
│ HdrHistogram │ increment one bucket. Each hot │
│ (no lock, no │ thread owns one histogram — the ▼
│ allocation) │ A/B/C hists above ARE these. p50 p99 p99.9 p99.99 max
└─────────────────┘ │
no mutex — a shared histogram behind a lock ▼
would itself become the jitter you're hunting dashboard / SLO
The hdrhistogram crate is a faithful Rust port. Use one histogram per thread per
stage, merge in the aggregator. Never share one behind a mutex (the observability
chapter, ch11).
Coordinated omission: the flagship failure
This idea has a name because Gil Tene spent years yelling about it: coordinated omission (ch00e has the gentle version) is when your load generator conspires with the system under test to not measure during the worst moments — precisely the samples you care about.
The worked example
You want to test at a constant 10,000 requests/sec, one request every 100µs. You write the obvious closed loop:
loop {
t0 = now();
send(); wait_for_response();
record(now() - t0);
sleep_until_next_interval();
}
The system runs happily at 50µs per response. Then it stalls for 1 second — page fault storm, whatever. What does your histogram show?
- One sample of ~1 second.
- Then the loop resumes and records 50µs samples again.
But you intended to send 10,000 requests during that second. A real open-world client population (orders arriving from the market) doesn’t politely stop arriving because you stalled. The request that would have arrived 100µs into the stall would have waited ~999.9ms. The one at 200µs, ~999.8ms. And so on: 10,000 samples ranging uniformly from ~0 to ~1s should be in the histogram. Instead there is one bad sample — the other 9,999 were simply never taken, because the load generator was blocked, coordinating with the stall.
Run the numbers for a 100-second test: 1,000,000 intended samples. Honest accounting puts ~10,000 samples (1%) spread uniformly across 0–1s — the entire top percentile is that stall block, so the honest p99.5 sits near 500ms and the p99.9 near 900ms. The coordinated-omission version shows ~50µs at both marks, plus one weird max. The reported tail is wrong by four orders of magnitude — 500ms against 50µs is 10,000×. This is not a subtle statistical quibble; it is the difference between “our system is fine” and “our system dropped the ball for a full second of market activity.”
The correction: intended send time
Account for when each request should have been sent:
- Schedule request i at
intended[i] = start + i * interval. - Record
response_time = completion - intended[i], notcompletion - actual_send. The time a request spent queued behind your own blocked load generator is real latency a real client would have seen.
HdrHistogram also offers post-hoc correction — record_correct(value, expected_interval) back-fills the missing samples by synthesizing the linearly
decreasing series — but intended-time accounting at the source is strictly better;
use record_correct only when you can’t fix the generator.
#![allow(unused)]
fn main() {
use hdrhistogram::Histogram;
use std::time::{Duration, Instant};
let interval = Duration::from_micros(100); // 10k/s intended rate
let mut h = Histogram::<u64>::new_with_bounds(1, 60_000_000_000, 3).unwrap();
let start = Instant::now();
for i in 0u64.. {
let intended = start + interval * (i as u32);
// Open-loop: wait until the *scheduled* send time — never later because
// a previous response was slow.
while Instant::now() < intended { std::hint::spin_loop(); }
do_request_and_wait(); // the system under test
// Latency measured from *intended* send time: queuing delay caused by
// our own backlog is charged to the system, as a real client would see.
h.record(intended.elapsed().as_nanos() as u64).unwrap();
if start.elapsed() > Duration::from_secs(30) { break; }
}
}(A fully open-loop harness sends from a paced thread regardless of outstanding responses and matches completions asynchronously; the code above is the minimal single-threaded version that still gets the accounting right.)
Open vs closed loop, and the throughput–latency curve
- Closed loop: N virtual clients, each sends, waits, sends again. Arrival rate adapts to system speed. Models: a fixed pool of synchronous callers. Inherently prone to coordinated omission.
- Open loop: arrivals come from an external schedule (Poisson — randomly spaced arrivals, like independent customers walking in — or fixed-rate), regardless of completions. Models: the market. Market data does not slow down because you’re busy — trading systems must be tested open-loop.
The deliverable of a load test is not a number, it’s a curve: sweep offered rate, plot p50/p99/p99.9 vs throughput. Every queueing system shows the same shape: flat latency at low utilization, then a knee where queueing delay explodes as you approach capacity (queueing theory: delay ∝ 1/(1−utilization)). Picture a checkout line at 95% utilization: there is no slack left to absorb a clump of arrivals, so the queue — and the wait — explodes. Find the knee, then state capacity as “we run at X, the p99.9 knee is at 4X.” A single “we handle 1M msgs/sec” claim without the latency curve is meaningless — anything can “handle” any rate if you let the queue grow.
The jitter-source checklist
When the tail is worse than the median by more than ~10×, walk this list. Each item is a distinct mechanism with a distinct signature and fix. Most of the vocabulary is already yours: page faults, TLB, IRQ/softirq, and scheduler preemption are ch00b’s kernel mechanisms; NUMA and hyperthreads are ch00a’s hardware topology; IPC is instructions-per-cycle (ch00e). The genuinely new terms: SMIs (System Management Interrupts — firmware-level interrupts the OS literally cannot see), an arena (a preallocated block you carve allocations out of and free all at once), khugepaged (the daemon that merges regular pages into hugepages in the background — it can freeze a page mid-move, right under your hot path), SCHED_FIFO (the run-until-yield real-time scheduler class), and an AVX license transition (the CPU briefly downclocking while its wide-vector units power up).
| Source | Signature | Fix |
|---|---|---|
| Allocator | occasional µs–ms spikes on alloc-heavy events | zero-alloc hot path; preallocate; arena |
| Page faults | first-touch spikes, spikes after idle | pre-fault + mlockall; touch all pages at startup |
| TLB misses / THP | spikes correlated with large working set; khugepaged stalls | hugepages (explicit, not THP defrag on hot path) |
| Scheduler preemption | multiples of timeslice; other runnable threads | isolcpus/cpusets, pinning, SCHED_FIFO carefully |
| IRQs / softirq | short (µs) spikes, network-correlated | IRQ affinity away from hot cores; busy-poll/bypass |
| Frequency scaling | first-op-after-idle slow; AVX transitions | performance governor; disable deep C-states; watch AVX license |
| SMIs | rare, large (10µs–ms), invisible to the OS | check smi counter / turbostat; BIOS settings; vendor fight |
| Hyperthread contention | 10–40% throughput noise, IPC drop | isolate the sibling; don’t share a physical core |
| NUMA | consistent extra ~60–100ns on remote lines | pin memory + threads to one node |
The interview version: don’t recite the list; explain that each has a measurable signature (the profiling chapter, ch09, shows how to catch scheduler noise with ftrace and the rest with perf) and that you eliminate them by measurement, not superstition.
Measuring under realistic load
Steady-state numbers lie. Market load is violently non-stationary:
- Open/close auctions: message rates 10–100× the daily median in the first and last minutes. If you sized queues and measured latency at median load, the open is where you find out.
- News/econ prints: near-instantaneous bursts — thousands of ticks in a millisecond across correlated symbols. This is also when your strategy most wants to trade, so tail latency during bursts is the only tail latency that matters.
- Quiet periods: paradoxically dangerous — caches cool, pages get reclaimed,
frequencies drop, branch predictors (the CPU’s learned guesses about which way
your
ifs go) decay and retrain. The first message after a lull is often your worst message. (Countermeasure: keep the path warm with synthetic traffic / cache-warming dummy work.)
Methodology consequences:
- Replay captured market data with original timestamps, including the worst bursts you’ve recorded — not a Poisson generator at average rate.
- Report percentiles conditioned on load regime: p99.9 during burst windows vs overall. A system can have a beautiful overall p99.9 that is entirely composed of quiet periods.
- Test the burst after the lull — the cold-start-into-burst transition is the realistic worst case, and steady-state harnesses never exercise it.
Plain-English recap
- You already do the first half of this in Datadog. Tracking p95/p99 per endpoint instead of averages is standard APM practice; trading just extends the discipline to p99.99 and max, because one slow “request” is a missed fill with a dollar sign, not one grumpy user.
- Coordinated omission is the synthetic-monitor blind spot. Your checkout service freezes for 30 seconds; your health checker (one synchronous loop) logs one slow check and resumes. The 3,000 customers who would have arrived during the freeze were never measured. That’s exactly what a closed-loop load generator does to a latency test.
- The intended-time fix is “measure from enqueue, not from dequeue”. If you time webhook processing from when the worker picked the job up, queue wait is invisible — and queue wait is precisely what the user experienced. Measuring from the scheduled send time charges your own backlog to the system, like measuring payment latency from the user’s click.
- Open loop vs closed loop is Black Friday vs a polite single customer. Real traffic doesn’t slow down because you’re struggling; a closed-loop tester does, and thereby flatters you. Markets are Black Friday all day.
- The throughput–latency knee is connection-pool saturation. A pgbouncer pool that’s fine at 60% utilization goes vertical near 100% — same 1/(1−u) queueing math. Capacity is the rate at the tail knee, not the biggest number the box survived.
- Conditioning on load regime = “p99 during the flash sale”. An overall p99 dominated by 3am quiet hours is flattering fiction; the only tail that matters is the one during the moments you actually need to perform — auctions, news bursts, your Black Friday.
Interviewer will ask
Q: Explain coordinated omission like I’m a skeptical SRE. A: If your load generator waits for each response before sending the next, then during a stall it stops sampling — exactly when latency is worst. One 1s stall at an intended 10k/s costs you 10,000 samples that should have recorded up to 1s of wait; you record one. Your tail percentiles come out wrong by four orders of magnitude — the chapter’s worked example reports ~50µs where the honest p99.5 is ~500ms. Fix: schedule sends at intended times and measure from the intended time, or use HdrHistogram’s
record_correctas a patch.Q: Why do you care about max latency? Any statistician will call it noise. A: At 100k events/sec a day is billions of samples — the max is a real event that really happened to a real order, and in trading one slow order has a direct cost: a missed fill or adverse selection. Also, maxes recur: today’s unexplained max is tomorrow’s p99.9. I track max per interval and I want a causal story for every spike.
Q: Open vs closed loop — which do you use and why? A: Open loop for anything market-shaped: the market’s arrival rate doesn’t adapt to my system’s speed, so a closed-loop test both understates latency (coordinated omission) and overstates capacity. Closed loop only models fixed pools of synchronous callers, which almost nothing in trading is.
Q: How does HdrHistogram get huge range and fine precision in fixed memory? A: By bounding relative error, not absolute — and the chapter’s 1,250ns sample is the whole trick in one record.
leading_zerosfinds the highest set bit, which is the log2, so 1,250 lands in the major bucket covering [1024, 2048); a shift of the remaining bits picks the linear sub-bucket; one counter at one index increments. Sub-buckets are sized so every value stays within the configured significant figures — 0.1% at 3 sig figs — which is why 1.000µs and 1.001µs get separate buckets while 10.000ms and 10.001ms don’t need to. That’s O(1) record — shift, clz, increment — no allocation, mergeable, tens of KB for the whole ns-to-minutes range.Q: Your p99 is fine but customers complain. What’s going on? A: Percentiles are per-request, but a customer experiences a session — the chapter’s tail arithmetic. A session of 100 requests dodges the p99 only if all 100 do, and 0.99¹⁰⁰ ≈ 37% — so ~63% of sessions eat at least one p99 event. “1% of requests” is therefore “most customers,” and the complaints are exactly what the math predicts. The trading version: a 500-tick burst almost certainly contains the p99.5, during precisely the moments the strategy trades. And before defending the p99 itself, I’d check it wasn’t measured with coordinated omission — a flattering p99 is this chapter’s other classic lie.
Q: How do you find the capacity of a service? A: Sweep offered rate open-loop, plot p99/p99.9 vs throughput, find the knee where queueing delay diverges. Capacity is the rate at the knee with an SLO on the tail, not the max rate the box survived. Then re-run with recorded burst traffic because the knee under bursty arrivals is lower than under smooth arrivals.
Q: Steady-state p99.9 is 8µs. What number do you tell the desk? A: Neither that nor anything single. I’d give the tail conditioned on regime: p99.9 during open/close and news-burst windows from replayed captures, plus the worst-case cold-into-burst number. Steady-state tails are the flattering, irrelevant case — the tail during bursts is when the strategy actually trades.
Further reading
- Gil Tene, “How NOT to Measure Latency” (talk, many recordings) — coordinated omission, percentile fallacies, the service-time vs response-time distinction.
- HdrHistogram documentation and the original Java repo’s design notes (hdrhistogram.org);
the Rust
hdrhistogramcrate docs mirror the API. - Brendan Gregg, Systems Performance (2nd ed.) — methodology chapters (USE method, workload characterization, latency analysis).
- The
wrk2README — the canonical short explanation of constant-throughput, corrected-latency load generation. - Neil Gunther / standard queueing-theory treatments of utilization vs response time (any presentation of M/M/1 and the 1/(1−ρ) blow-up) for the knee.
Where this goes next: you now know what to measure and how to avoid lying to yourself — Chapter 9 is the toolbox that answers where the time went and who caused the spike: perf counters, flamegraphs, ftrace, and a worked tail-regression diagnosis.
Linux Profiling
Before you start — this chapter leans on a handful of primer ideas:
- Sampling vs counting profilers, PMU counters, and IPC — the two ways
perfwatches a program and the numbers it emits: ch00e- Flamegraphs — what the axes mean (width = samples, x ≠ time): ch00e
- Caches, cache lines, and coherence — L1/L2/LLC and why a line “bouncing” between cores is expensive: ch00a
- Kernel vs userspace, context switches, IRQs/softirq — the scheduler machinery this chapter hunts for on isolated cores: ch00b
- Pages and the TLB — what a dTLB miss even is: ch00b
Read those first — 20 minutes there saves an hour here.
[Linux] — everything here assumes a Linux box with perf and root or
perf_event_paranoid configured. This is the chapter that turns “it got slower” into
a diagnosis.
perf fundamentals: counting vs sampling
perf fronts the kernel’s perf_event subsystem, which exposes two distinct modes
you should never conflate:
- Counting (
perf stat): program the PMU (Performance Monitoring Unit — the CPU’s built-in event-counting hardware, ch00e) counters, run the workload, read totals at the end. Near-zero overhead, exact totals, no attribution — you learn what happened, not where. - Sampling (
perf record): take a sample every N events — cycles by default. Each sample records the IP (instruction pointer — the address of the instruction the CPU was executing at that instant, “where in the code you were”) plus an optional call stack. Statistical attribution to code. Overhead is real but controllable via frequency. One catch: the sample can land a few instructions after the real culprit — that drift is called skid. Appending:p/:ppto an event asks the CPU’s hardware assist (Intel’s is called PEBS) to pin the sample exactly.
Workflow discipline: count first, sample second. perf stat tells you which
resource is the problem; perf record on the corresponding event tells you where.
perf stat literacy
perf stat -d -- ./router --replay ticks.bin
# or attach to a running pinned process for 10s:
perf stat -p $(pidof router) -e cycles,instructions,branches,branch-misses,\
cache-references,cache-misses,LLC-loads,LLC-load-misses,dTLB-load-misses -- sleep 10
What the numbers mean on a modern x86 core (4–6 wide issue — the core can start 4–6 instructions per clock tick, so IPC around 4 is the practical ceiling):
- IPC (instructions per cycle): the single most information-dense number.
- IPC ≈ 0.5 or below: the core is stalled — almost always memory: cache misses,
or atomics ping-ponging cache lines. Or (check separately) it’s not stalled at
all but spinning: a polite poll loop executes the
pauseinstruction, which deliberately does nothing for many cycles — the loop is waiting by design, so IPC craters with nothing actually wrong. - IPC ≈ 1–2: typical mixed code; nothing screaming.
- IPC ≈ 3+: compute-dense, well-fed pipeline — L1-resident data, predictable branches. Your hot loop should look like this; if your SPSC drain loop shows IPC 0.4, the queue’s cache behavior is the story.
- IPC ≈ 0.5 or below: the core is stalled — almost always memory: cache misses,
or atomics ping-ponging cache lines. Or (check separately) it’s not stalled at
all but spinning: a polite poll loop executes the
- branch-misses / branches: >2–3% in a hot loop is worth attention. The CPU is an assembly line that speculatively runs ahead: it starts executing instructions past a branch before it knows which way the branch actually goes. Guess wrong and it throws away everything already on the line and restarts — each miss is ~15–20 cycles of that pipeline flush. Trading hot paths with unpredictable data-dependent branches (order type dispatch) are classic offenders — fix with branchless forms or sorting work by type.
- cache-misses / cache-references, LLC-load-misses (LLC = last-level cache, the
big L3 shared by all cores on the socket — ch00a):
“memory bound” looks like:
low IPC + high LLC misses + high memory-stall counters (PMU events that count the
cycles the core spent waiting on memory; Intel’s catch-all is named
cycle_activity.stalls_mem_any). Every LLC miss is a trip to DRAM: ~60–100ns, i.e. ~200–400 cycles — one miss costs more than an entire well-tuned queue operation. - dTLB-load-misses (the TLB caches virtual→physical address translations; a miss means a page-table walk — ch00b): elevated with large scattered working sets → hugepages conversation.
Counter multiplexing trap: the PMU has ~4–8 programmable counters; ask for more
events and perf time-slices them and scales the results (see the [xx.x%]
annotation). For precise work, run multiple passes with few events each.
perf record / report and flamegraphs
# Sample on-CPU cycles with call graphs, 99Hz to avoid lockstep with timers:
perf record -F 99 -g -p $(pidof router) -- sleep 30
perf report # TUI; use --no-children to see self time
(Why 99Hz and not a round 100: the kernel’s own timers fire at round rates. Sample at exactly the same rate and every sample lands at the same phase of the timer cycle, showing you the same instant over and over. An odd rate drifts relative to the timers and sweeps the whole range.)
Call-graph capture has two modes and the choice matters for Rust:
- Frame pointers (
-g=--call-graph fp): each function keeps one register pointing at its caller’s stack frame, so the live call stack forms a linked list — the profiler just follows the chain from the sampled IP back tomain. Cheap, reliable if frames exist — but compilers omit the frame pointer exactly to free that register for real work, and Rust/LLVM omits it by default in release. Fix in.cargo/config.tomlor RUSTFLAGS:-C force-frame-pointers=yes. Cost is ~1% (one register); every serious low-latency shop just leaves it on in production builds precisely so perf works when it matters. - DWARF (
--call-graph dwarf): when there’s no frame-pointer chain to follow, perf snapshots a chunk of raw stack memory with every sample (8KB by default) and reconstructs the call chain offline — “unwinding” — using DWARF, the standard debug-info format, whose tables describe each function’s frame layout. Works without frame pointers but is heavy, can truncate deep stacks, and slows recording. Use when you can’t rebuild.
Flamegraphs (Brendan Gregg’s stackcollapse-perf.pl | flamegraph.pl, or
cargo flamegraph which wraps the whole pipeline):
cargo flamegraph --bin router -- --replay ticks.bin
# or from an existing perf.data:
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg
Reading (ch00e walks the axes slowly): x-axis is
alphabetical, not time; width = fraction of samples; you’re
looking for wide plateaus (where cycles go) and unexpectedly-present frames (why is
memmove 8% of my hot path?). Remember it shows on-CPU time only.
Rust-specific perf hygiene
[profile.release] debug = true(ordebug = "line-tables-only"): keeps DWARF so perf maps addresses to source lines. Zero runtime cost — it only fattens the binary.- Symbols: the compiler encodes each function’s module path and generic parameters
into one flat, linker-safe string (“mangling” —
my_crate::foobecomes something like_ZN8my_crate3foo17h…); perf demangles these reasonably well, andrustfiltcleans up the rest. - Inline noise: release Rust inlines aggressively, so samples attribute to the
caller into which code was inlined.
perf report --inline(with debug info) expands inlined frames. When a flamegraph shows a fat frame for a function that “does nothing,” it’s usually full of inlined callees. - Iterator chains compile to loops that attribute strangely; profile with
optimizations on always (a debug-build profile is fiction), and use
#[inline(never)]temporarily to force attribution boundaries when you need to isolate a suspect.
perf c2c: catching false sharing
perf c2c (cache-to-cache) samples loads and stores with PEBS (the precise-sampling
hardware from earlier) and looks for one specific event. The picture: core A writes
a cache line, so A’s cache now holds the only current copy (MESI’s “Modified” state —
false sharing and MESI are unpacked in ch00a). When core B
then loads from that line, the data has to come out of A’s cache, not from RAM —
each such load counts as one HITM (“hit in another core’s modified line”). A few
are normal; thousands mean the line is commuting between cores — bouncing — which is
exactly the false-sharing signature.
perf c2c record -p $(pidof router) -- sleep 10
perf c2c report --stats # then the full report
Worked interpretation of the report:
- Top section lists cache lines sorted by HITM count. A line with thousands of Rmt/Lcl HITM events (Rmt = the other core was on the remote socket, Lcl = the local one) is contended.
- For each hot line, the per-offset breakdown shows which byte offsets within the 64-byte line are touched, by which code (symbol+source line), from which CPUs.
- True sharing: all accesses hit the same offset — that’s a genuinely shared variable (e.g., both threads on one atomic head index). Fix = algorithmic.
- False sharing: different offsets on the same line — e.g., offset 0x00 written
by CPU 2 (producer’s head) and offset 0x08 read by CPU 14 (consumer’s cached
tail copy). Fix = pad to 64B with
#[repr(align(64))]wrappers — and on Intel pad to 128B: the adjacent-line prefetcher speculatively pulls in the neighboring 64B line alongside every fetch, so two variables 64B apart can still end up colliding.
The classic trading-system find: two per-thread counters declared adjacently in a struct, “independent” but sharing a line, silently taxing both threads ~100ns per increment pair. c2c is how you prove it rather than pad everything superstitiously.
ftrace: the scheduler-silence check
For an isolated hot core, the invariant is brutal and testable: no scheduler
activity on that core, ever. ftrace (the kernel’s built-in tracer — no install,
you drive it through /sys/kernel/tracing) tracepoints verify it:
cd /sys/kernel/tracing
echo 0 > tracing_on
echo > trace
echo sched_switch sched_wakeup > set_event
echo 1 > tracing_on; sleep 60; echo 0 > tracing_on
grep 'CPU:3\|cpu=3' trace # your isolated core — this should print NOTHING
Expected result on a properly isolated core (isolated via the isolcpus boot flag
or its runtime cousin, cpusets — plus IRQ affinity + nohz_full): silence, or a
single switch-in of your pinned thread.
Anything else — kernel background threads like ksoftirqd, kworker, or
migration, or a timer tick — is a named
intruder with a named fix. This check takes two minutes and settles arguments that
otherwise run for days. Two cheap corroborations: perf stat’s
context-switches counter must read zero over the window, and a before/after diff
of that core’s column in /proc/interrupts — the kernel’s per-core interrupt
scoreboard — must show no counts moving beyond any IRQ you deliberately routed there.
(perf sched record / perf sched timehist gives the same
data with per-event wakeup latencies when you need detail.)
Off-CPU analysis
Flamegraphs show where you burn cycles; they’re blind to where you wait. Off-CPU analysis (ch00e — profiling the time a thread spends not running) attributes blocked time (locks, page faults, I/O, involuntary preemption) to stacks. Options:
perf sched timehist— per-wakeup scheduling latencies.offcputimefrom bcc/bpftrace — toolkits for running small programs inside the kernel (bpftrace gets its proper introduction in the next section); it sums blocked time by stack, kernel+user, cheaply in BPF. Renderable as an off-CPU flamegraph.
A hot path that should never block makes this a null-check: any off-CPU stack for the hot thread other than your intended park/poll site is a bug report writing itself.
bpftrace one-liners worth memorizing
bpftrace is a one-liner language over eBPF — small verified programs the kernel runs at probe points, so you can ask production questions without patching anything (ch00b).
# Syscall latency histogram for one process (should be EMPTY for a hot thread):
bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == 1234/ { @t[tid] = nsecs; }
tracepoint:raw_syscalls:sys_exit /@t[tid]/
{ @lat = hist(nsecs - @t[tid]); delete(@t[tid]); }'
# Who is sending my hot thread signals / waking it?
bpftrace -e 'tracepoint:sched:sched_wakeup /args->pid == 1234/
{ @wakers[comm, kstack] = count(); }'
# Page faults on the hot process after warmup (want: zero):
bpftrace -e 'software:page-faults /pid == 1234/ { @[ustack] = count(); }'
The pattern: for a well-behaved hot thread, most of these tools should return nothing, and “instrumented silence” is exactly the evidence you bring to a review.
Worked session: “router p99.9 regressed 3µs — walk the diagnosis”
The setup: order router, pinned to isolated core 3, historically p99.9 = 9µs. After Tuesday’s deploy: 12µs. p50 unchanged at 4.1µs. Walk it:
1. Frame the symptom. p50 flat + tail worse = not a straight-line code slowdown; something episodic. Prior: new allocation, new fault, new interference, or new contention.
2. Cheap wide net first — counting, attached to prod replica under replay:
perf stat -e cycles,instructions,cache-misses,LLC-load-misses,page-faults,\
context-switches,dTLB-load-misses -p $(pidof router) -- sleep 30
Result: IPC 2.1 → 2.0 (noise), page-faults 0 (good — mlockall, the call that
locks every page of the process into RAM so none can fault, is holding),
context-switches: 0 before, 41 now. On an isolated core that number must be 0.
Tail regression + nonzero context switches ≈ found the mechanism; now find the actor.
3. Name the intruder with ftrace/perf sched:
perf sched record -C 3 -- sleep 30 && perf sched timehist -C 3
Result: rdkafka-metrics thread scheduling onto core 3 every ~750ms, 2–8µs each.
The deploy added a metrics client whose background thread inherited the process’s
CPU mask before main() pinned the hot thread — nothing repinned the spawned thread,
and the cpuset allowed it.
4. Corroborate against the latency data. Pull the event ring (the in-process
log of per-event timestamps the hot path records — the observability chapter,
ch11, builds it) for
the p99.9 outliers: outlier timestamps line up with sched_switch events on core 3
at ~750ms cadence. Mechanism, actor, and correlation all agree — this is the bar for
“diagnosed,” not “the flamegraph looked different.”
5. Fix and verify. Spawn ancillary threads with an explicit non-isolated affinity (or move pinning before any thread spawns); re-run the 60s ftrace silence check (clean); re-run the replay: p99.9 back to 9.1µs. Attach the before/after histograms and the sched trace to the postmortem.
Total wall time: ~an hour, and no code was read until step 3 named a thread. That’s the shape of counter-driven diagnosis: symptom → resource → actor → correlation → fix → re-verify.
Plain-English recap
perf statvsperf recordis dashboard vs profiler. Counting is your Datadog metrics view — CPU%, DB time, error rate — it names the resource that’s wrong. Sampling is the profiler flame view — it names the code. Same discipline you use today: look at the dashboard before opening the profiler.- A flamegraph is the Chrome DevTools flame chart with one crucial difference: the x-axis is not time. Stacks are merged and sorted, so width = “share of all samples,” and you read it by hunting wide plateaus, not left-to-right.
- IPC is work-per-tick, like rows-per-second per connection. Low IPC means the CPU is mostly waiting — usually on memory — the way a worker with low throughput is usually blocked on I/O, not short of CPU.
- False sharing is two services updating unrelated columns of the same DB row.
Each write invalidates the other side’s cached copy, and both pay for a conflict
that exists only because of physical layout.
perf c2cis the tool that shows you the row and the columns. - The ftrace silence check is asserting your container runs nothing else. For an isolated core the correct trace output is empty — any line is a named intruder. Instrumented silence is evidence, like a clean Sentry release.
- Off-CPU analysis is the “waiting” spans in an APM trace. A CPU profiler only sees running code; time blocked on locks, faults, or the scheduler is invisible to it — exactly like DB-wait time that never shows in a CPU profile.
- The worked session is a Datadog-first incident review. Symptom → counters → actor → correlate with the latency spikes → fix → re-verify, and no code gets read until the mechanism has a name. You already work this way; here the counters are just closer to the metal.
Interviewer will ask
Q: IPC is 0.5 on your hot thread. What are your hypotheses and next steps? A: Stalled or spinning, so first split those: a polite poll loop is mostly
pauseinstructions, and low IPC while waiting is by design — check whether the loop was actually doing work. If it’s genuinely stalled, it’s memory-bound until proven otherwise, and I count before I sample: LLC-load-misses and the memory-stall counters name the resource, thenperf record -e cycles:ppnames the loads. The final fork is three diseases, each with its own counter signature: high LLC misses with a large working set means capacity — the cache is simply too small; HITM lines inperf c2cmean coherence — cores fighting over shared lines; elevated dTLB-load-misses means TLB — and starts the hugepages conversation.Q: Frame pointers or DWARF unwinding for production profiling? A: Frame pointers, compiled in always (
-C force-frame-pointers=yes). ~1% cost for the ability to profile any incident live with cheap, reliable stacks. DWARF unwinding copies stack per sample — heavy, truncates, and I don’t want to be rebuilding binaries during an incident.Q: How do you distinguish true from false sharing in perf c2c output? A: Look at the per-offset breakdown of the hot line: same offset hammered by multiple CPUs = true sharing (fix the algorithm); different offsets on one line = false sharing (pad/realign, 64B minimum, 128B on Intel because the spatial prefetcher pulls line pairs).
Q: Prove to me a core is actually isolated. A: Enable
sched_switch/sched_wakeuptracepoints filtered to that CPU for a minute of production traffic — output must be empty. Plus:context-switchescounter zero in perf stat,/proc/interruptsdeltas zero for that core, and the timer tick confirmed off vianohz_full. Silence in the trace is the proof; anything else names the intruder.Q: Your flamegraph looks identical before and after a tail regression. Why? A: Flamegraphs are on-CPU and dominated by the common case; a p99.9 event is 1 in 1000 samples — invisible. Tails need targeted tools: off-CPU analysis for blocking, sched tracing for preemption, the in-process event ring for outlier timestamps to correlate against. Sampling profilers answer “where do cycles go,” not “what happened at 14:31:07.”
Q: What does
perf statcost the target? Andperf record? A: They’re the chapter’s two modes, and each cost follows from its mechanism. Counting programs the PMU’s hardware registers and reads totals at the end — the hardware counts whether or not you look, soperf statis effectively free and safe in production. Sampling pays an interrupt per sample, so its cost scales with rate and with what each sample carries: 99Hz with frame pointers is negligible; 10kHz with DWARF copying 8KB of stack per sample very much isn’t. So the workflow discipline doubles as the safety rule: count first, sample second — and on live systems keep record frequencies low and windows short.Q: Why do Rust release-build profiles attribute time to the “wrong” function? A: Aggressive inlining — callee cycles land in the caller frame. Keep
debug = truein the release profile so DWARF inline info exists, useperf report --inline, and when isolating a suspect,#[inline(never)]it temporarily to force a real frame boundary.
Further reading
- Brendan Gregg, Systems Performance (2nd ed.) — chapters on CPUs, perf, and methodology; and his website’s perf examples, flamegraph, and off-CPU analysis pages.
- The perf wiki (perf.wiki.kernel.org) — canonical reference for events, call-graph
modes, and
perf c2c. - Joe Mario’s Red Hat blog write-up on
perf c2c— the worked false-sharing interpretation this section compresses. - Brendan Gregg, BPF Performance Tools — offcputime, syscall tracing, and the bpftrace idioms above.
- Denis Bakhvalov, Performance Analysis and Tuning on Modern CPUs (free book) — PMU literacy, top-down analysis, skid/PEBS details.
Where this goes next: profiling finds where time goes in a running system — Chapter 10 is about measuring a single function honestly: Criterion, black_box, contention regimes, and why a microbenchmark win can still be a system-level loss.
Microbenchmarking Without Fooling Yourself
Before you start — this chapter leans on a handful of primer ideas:
- Caches, cache lines, and coherence — why the same queue costs 4ns on one thread and 40ns across two cores: ch00a
- Cores, hyperthreads (SMT), and sockets — the topology words the contended benchmarks depend on: ch00a
- False sharing — two “independent” variables on one cache line taxing each other: ch00a
- Cycles vs nanoseconds, and PMU counters — why comparisons are done in cycles: ch00e
- rdtsc and frequency scaling — the clocks chapter’s discipline (ch07), which this chapter assumes: ch00e
Read those first — 20 minutes there saves an hour here.
Microbenchmarks are the easiest measurements to produce and the easiest to be wrong about. The failure modes are systematic — the compiler deletes your work, the CPU warms into an unrepresentative state, your “uncontended” queue bench never exercises coherence traffic — and each has a specific countermeasure. This chapter is those countermeasures, plus the calibration table that lets you judge whether a number is even plausible.
Criterion: the baseline harness
Criterion (the de-facto Rust benchmarking crate) is the standard because it does statistics you’d otherwise skip: warmup, many samples, outlier classification, bootstrap confidence intervals (“bootstrap” = resampling your own measurements many times over to estimate how confident to be in the result), and regression comparison against the saved baseline.
# Cargo.toml
[dev-dependencies]
criterion = "0.5"
[[bench]]
name = "spsc"
harness = false
[profile.bench]
debug = true # so perf can symbolize the bench binary too
#![allow(unused)]
fn main() {
// benches/spsc.rs — skeleton
use criterion::{criterion_group, criterion_main, Criterion};
use std::hint::black_box;
fn bench_push_pop(c: &mut Criterion) {
let mut g = c.benchmark_group("spsc");
g.bench_function("uncontended_push_pop", |b| {
let (mut tx, mut rx) = spsc::channel::<u64>(1024);
b.iter(|| {
tx.push(black_box(42u64)).unwrap();
black_box(rx.pop().unwrap());
});
});
g.finish();
}
criterion_group!(benches, bench_push_pop);
criterion_main!(benches);
}Useful discipline: cargo bench -- --save-baseline main before a change,
--baseline main after — Criterion reports the delta with confidence intervals, which
kills “it looks maybe 2% faster” conversations.
black_box and dead-code elimination
The optimizer’s job is to delete work whose result is unused — the pass is called dead-code elimination (DCE); your benchmark’s job is to do work. Left unresolved, that conflict yields benches that measure an empty loop at 0.3ns/iter — a number that should trigger immediate suspicion (that’s one cycle; almost nothing real is one cycle).
std::hint::black_box is an identity function the optimizer must treat as opaque:
- Wrap inputs so the compiler can’t constant-fold the computation across
iterations:
compute(black_box(x)). - Wrap outputs so the result is “used”:
black_box(compute(x)). - It is a hint, not a guarantee, but on current rustc it reliably forces the value to materialize (typically to a register/stack slot) without adding a memory fence — that is, without smuggling in an extra CPU-ordering instruction whose own cost would pollute the measurement.
Three more traps, all with the same shape — the loop you wrote is not the loop that ran:
-
Loop-invariant hoisting. If every iteration computes the same thing, the compiler computes it once, outside the loop, and your timed loop is a no-op. Feed it varying inputs instead:
#![allow(unused)] fn main() { b.iter(|| lookup(&table, 42)); // hoisted: computed once, timed never b.iter_batched(|| rng.gen(), // fresh input per iteration |k| lookup(&table, k), BatchSize::SmallInput); } -
Bounds checks as the measurement. Indexing inside the bench loop can make the bounds check — not your function — the dominant cost, a cost the real call site may never pay:
#![allow(unused)] fn main() { b.iter(|| { for i in 0..n { sum += data[i]; } }); // measures bounds checks b.iter(|| { for x in &data { sum += x; } }); // measures the loop body } -
iter_batchedsetup pollution. For a tiny measured function, the per-batch machinery around it becomes a real fraction of the number — pick the batch size deliberately rather than accepting a default:#![allow(unused)] fn main() { b.iter_batched(setup, tiny_op, BatchSize::LargeInput); // batch overhead drowns tiny_op b.iter_batched(setup, tiny_op, BatchSize::SmallInput); // sized for small routines }
Warmup, frequency, and thermals
The first iterations run cold: icache misses (the icache is the instruction cache — the L1 that holds code rather than data, ch00a), branch predictor untrained, and — biggest — the CPU may be at idle frequency. Criterion’s warmup (default 3s) handles training; it does not control the platform:
- Governor:
performance, notschedutil/powersave, or your first benchmark runs at 1.2GHz and your comparison across runs tracks the governor’s mood. - Turbo: turbo frequency depends on how many cores are active and thermal
headroom — a single-threaded bench turbos higher than the same code will run in
your 8-thread production process. For comparable numbers either disable turbo
(
no_turbo=1) or at least know your bench frequency (turbostat). - Thermals: a 5-minute bench suite on a small box slowly clocks down; benchmark A (run first, cold package) beats benchmark B (run second, hot) for no code reason. Randomize/interleave order or fix frequency.
- Report cycles when comparing algorithms. Nanoseconds confound your code with the frequency circus; cycles (rdtsc, or perf’s cycle counter) isolate the code. Nanoseconds are for budgets; cycles are for comparisons.
- Laptops (and especially MacBooks) are for writing benches, not for believing them. Numbers you’ll quote come from the pinned, governed, isolated Linux target.
Benchmarking lock-free structures: contention is the benchmark
Here is the mistake that invalidates most published queue benchmarks: an SPSC/MPMC (single-producer single-consumer / multi-producer multi-consumer queue) structure has two completely different performance regimes, and a single-threaded bench only ever measures the first:
- Uncontended: producer and consumer never race; every access hits L1 (ch00a); you’re measuring instruction count and store-buffer behavior — the store buffer is the core’s small private outbox: writes queue up there and drain to the cache a moment later. Good SPSC: a few ns/op.
- Contended (cross-core): head/tail lines bounce between cores; you’re measuring the coherence protocol (the MESI machinery that keeps caches consistent — ch00a). Same code: 20–100+ns/op, and throughput depends on which cores you picked.
Both are real workloads (a queue drained in bursts runs mostly-uncontended; a saturated pipeline runs contended). So you write both harnesses and report both:
#![allow(unused)]
fn main() {
// Harness 1: uncontended — same thread, alternating push/pop (as above).
// Measures pure instruction cost. Expect single-digit ns.
// Harness 2: contended — two pinned threads, sustained streaming.
fn bench_contended(c: &mut Criterion) {
c.bench_function("spsc/contended_throughput_1M", |b| {
b.iter_custom(|iters| {
let (mut tx, mut rx) = spsc::channel::<u64>(1024);
let n = iters.max(1_000);
let consumer = std::thread::spawn(move || {
core_affinity::set_for_current(core_affinity::CoreId { id: 4 });
let t0 = std::time::Instant::now();
for _ in 0..n {
loop { if let Some(v) = rx.pop() { black_box(v); break; }
std::hint::spin_loop(); }
}
t0.elapsed()
});
core_affinity::set_for_current(core_affinity::CoreId { id: 2 });
for i in 0..n {
while tx.push(i).is_err() { std::hint::spin_loop(); }
}
consumer.join().unwrap() // elapsed / iters = ns per transfer
});
});
}
}Notes on that harness:
iter_custombecause Criterion’s default timing loop can’t span two threads; you time the whole stream and divide.- Pin both threads, explicitly, in the bench (
core_affinitycrate). Unpinned, the scheduler sometimes lands both threads on SMT siblings — two logical CPUs on one physical core (ch00a) — which share L1/L2 and look suspiciously fast, and sometimes across sockets (suspiciously slow), and your bench has multi-modal results that track nothing in your code. Choose the same topology production uses: same-socket, different physical cores, and say so in the results. - Report pairs: same-core-SMT / cross-core / cross-socket are three different (interesting) numbers. Cross-socket can be 3–5× cross-core — the cache line has to travel over the physical link between the two CPU chips, not just within one.
- Batch effects: benching
pushof 1 item repeatedly is different from streaming. A good SPSC keeps a private cached copy of the other side’s index — the producer remembers where it last saw the consumer’s tail — and only re-reads the shared index when that cached copy says the queue might be full or empty. Streaming amortizes that expensive re-read over many items; single-item ping-pong forces one per item and measures the worst case. Know which one your harness exercises.
Measuring allocation
“Zero-alloc hot path” is a claim; measure it:
- dhat (
dhatcrate): heap profiling with allocation-site stacks. Run the replay underdhat::Profiler, assert the hot phase allocates nothing. Slow, thorough — a CI job, not a hot-path tool. - Counting allocator: a
GlobalAllocwrapper that increments an atomic. Cheap enough to keep in test builds, and it turns “zero allocations after warmup” into a unit test:
#![allow(unused)]
fn main() {
use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
pub static ALLOCS: AtomicU64 = AtomicU64::new(0);
pub struct Counting;
unsafe impl GlobalAlloc for Counting {
unsafe fn alloc(&self, l: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Relaxed);
System.alloc(l)
}
unsafe fn dealloc(&self, p: *mut u8, l: Layout) { System.dealloc(p, l) }
}
#[global_allocator]
static A: Counting = Counting;
// In the test: warm up, snapshot ALLOCS, run 1M events, assert delta == 0.
}The assertion version is the valuable one: it catches the intern who adds a
format! to the hot path six months from now, mechanically.
Calibration: theoretical limits your numbers must respect
A number without context can’t be judged. One term before the table: an atomic
RMW is a read-modify-write — e.g. fetch_add; on x86, a lock-prefixed
instruction like lock xadd. Memorize the ladder (typical modern x86 server,
order-of-magnitude):
| Operation | Cost |
|---|---|
| L1 load hit | 4–5 cycles (~1.5ns) |
| L2 hit | ~12–14 cycles |
| L3 hit | ~40–50 cycles |
| DRAM load | ~60–100ns (200–350 cycles) |
| Uncontended atomic RMW (line in L1) | ~15–25 cycles |
| Cross-core cache-line transfer (HITM) | ~40–100+ cycles, more cross-socket |
| Contended atomic RMW (line bouncing) | 100+ cycles each, throughput collapses |
| Branch mispredict | ~15–20 cycles |
| Store-forwarding stall (a write re-read immediately, before it reached cache — the core trips over its own outbox) | ~10–15 cycles |
| Memory bandwidth (per socket) | order 100–400 GB/s; a single core saturates at ~10–30 GB/s |
Use it in both directions. Your contended SPSC does a transfer in 25ns? Plausible — that’s one-and-change line transfers, believable for a cached-index design. It does a transfer in 3ns cross-core? Not plausible — a single cross-core line transfer costs more than that; your harness isn’t crossing cores (SMT siblings, or the consumer is batching in a way you didn’t intend, or DCE ate the work). Implausibly good numbers are bugs in the bench far more often than breakthroughs in the code.
The trap: microbench won, system regressed
You optimized a function; Criterion says −30%; you ship; the replay harness says end-to-end p99.9 got worse. This is common, and the mechanisms are worth knowing cold:
- icache/code-size pressure: the “faster” version is 4× more code (unrolling, inlining, a LUT — lookup table). Alone in a microbench loop, it’s icache-resident and wins. In the real system it evicts other hot code; total icache misses rise; the system loses. Microbenches systematically favor code bloat because the bench binary has no competing working set.
- Inlining changes: your edit pushed a function past the inlining threshold — callers across the crate now make real calls; or the reverse, and register pressure spilled somewhere else. (x86-64 has only ~16 general-purpose registers; inline too much code into one function and the compiler runs out, “spilling” variables to stack memory — and every spilled access is a memory access.) The diff you benched is not the diff that ran.
- D-cache working set: a 64KB lookup table beats computation in isolation, and then evicts the order book from L2 in production.
- Branch predictor training: the microbench’s input distribution trains the predictor to near-perfection; production’s distribution doesn’t.
The rule: a microbenchmark is evidence about a mechanism, never a verdict about the system. The verdict comes from the macro replay harness — captured market data, full pipeline, HdrHistograms, A/B against baseline (the measurement lab, ch12, builds exactly this). The microbench tells you why the macro result moved; only the macro result tells you whether to ship. Institutionalize it: no perf PR merges on Criterion output alone.
Plain-English recap
black_boxis the jsperf lesson. A JIT (or LLVM) will happily delete a loop whose result nobody reads, and your “benchmark” measures an empty loop. Wrapping inputs and outputs inblack_boxis how you force the work to be real — and a sub-nanosecond result means you forgot.- Criterion baselines are perf snapshot tests.
--save-baselinebefore,--baselineafter, and the delta comes with confidence intervals — the same mechanical “did this PR regress it?” gate you’d want in CI, killing “looks maybe 2% faster” debates. - Warmup and thermals: never trust a benchmark from a laptop on battery. Cold CPU at idle frequency vs warm CPU at turbo is the cold-Lambda vs warm-Lambda problem; if you don’t pin the governor and know the frequency, you’re benchmarking the thermostat, not the code.
- Contended vs uncontended is hot-row contention in Postgres. The same UPDATE costs wildly different amounts uncontended vs when every transaction hammers one row. Queues are identical: single-threaded cost and cross-core cost are two different numbers, and a benchmark must state which one it measured — and which cores it used.
- The counting allocator is the N+1-query assertion. Like a test asserting an
endpoint issues exactly 3 SQL queries, asserting “zero allocations per million
events” turns a performance claim into a mechanical regression gate that catches
next quarter’s accidental
format!. - Microbench won, system lost = bundle-size thinking. Inlining and lookup tables win in isolation the way a heavyweight dependency “wins” one page — then the bigger footprint evicts everything else and the whole app slows. The end-to-end replay harness is your E2E suite: it, not the unit-level number, decides whether to ship.
- The calibration table is your plausibility linter. You already know a network round trip to Postgres can’t take 10µs, so a test claiming it is broken. Same instinct, new ladder: a cross-core transfer can’t cost 3ns, so a bench claiming it isn’t actually crossing cores.
Interviewer will ask
Q: What does
black_boxactually do, and when do you need it? A: It’s an optimizer-opaque identity — the compiler must assume the value is read and produced arbitrarily, so it can’t dead-code-eliminate the computation or constant-fold across iterations. Wrap benchmark inputs and outputs. It’s needed whenever the measured work’s result doesn’t otherwise escape. Sanity check: a sub-nanosecond result usually means I forgot it.Q: Your SPSC benches at 4ns/op. Ship it? A: First question: which regime? 4ns is believable single-threaded/uncontended — that’s an L1-resident instruction-cost measurement. Cross-core it’s below the price of one cache-line transfer, so I’d suspect the harness: threads not actually on separate physical cores, or batching hiding the coherence cost. I want both numbers, from pinned threads, with the topology stated.
Q: Why pin threads inside a benchmark? A: Because cross-core cost depends on topology: SMT siblings share L1/L2 and look fast; cross-socket pays interconnect and looks slow; the scheduler picks differently each run, so unpinned benches are multi-modal noise. I pin to the production topology and state it with the result.
Q: How do you verify a zero-allocation claim? A: Mechanically, two ways: dhat in CI for allocation sites, and a counting
GlobalAllocwrapper with a test that runs the warmed hot path for a million events and asserts the allocation counter delta is zero. Claims that aren’t asserted regress silently.Q: Microbench improved 30%, system p99.9 regressed. Explain. A: Most likely icache: the faster variant is bigger, wins alone, and evicts other hot code in the full binary. Or the change moved inlining decisions so the production call sites compile differently than the benched one. Diagnosis: perf stat icache/frontend-stall counters on the full system before/after, and the macro replay harness as arbiter. This is why microbenchmarks are evidence, not verdicts.
Q: What would make you distrust a Criterion result of “−3%, p < 0.05”? A: The delta is smaller than what the frequency circus alone can cause — a governor swing from idle clocks to turbo moves nanosecond numbers by tens of percent, and a package that warmed up between the baseline run and the candidate run moves them too. Criterion’s p-value only covers sampling noise within a run; it can’t see that the two runs happened at different clock speeds. So before believing a small delta I remove frequency from the experiment: compare cycles, not nanoseconds; fix the frequency — performance governor, turbo off; and interleave baseline and candidate runs so thermal drift hits both equally. If the −3% survives that, it’s real; if it doesn’t, I was benchmarking the thermostat, not the code.
Q: Roughly what does an uncontended atomic fetch_add cost? Contended? A: Uncontended with the line in L1: ~15–25 cycles — a
lock-prefixed RMW. (Ancient x86 froze the whole memory bus for everylockinstruction; modern parts just hold on to the one cache line, which is why the uncontended case is this cheap.) Contended: the line ping-pongs, each RMW waits ~40–100+ cycles for line ownership and total throughput collapses to line-transfer rate — which is why per-thread counters aggregated off-path beat a shared counter (the observability chapter, ch11).
Further reading
- Criterion.rs user guide — statistics model,
iter_custom/iter_batched, baselines. - Brendan Gregg, Systems Performance (2nd ed.) — benchmarking chapter (“benchmarking sins” checklist).
- Denis Bakhvalov, Performance Analysis and Tuning on Modern CPUs — measurement bias, frequency management, counter-based bench validation.
- Agner Fog’s optimization manuals (instruction tables + microarchitecture guide) — the source for instruction/atomic cost intuition.
- Paul McKenney, Is Parallel Programming Hard, And, If So, What Can You Do About It? — counting and cache-coherence cost chapters behind the contended-vs- uncontended distinction.
Where this goes next: benchmarks live in the lab — Chapter 11 answers how to keep measurement running inside the production hot path, always on, for a cost you can state and defend.
Observability Inside the Hot Path
Before you start — this chapter leans on a handful of primer ideas:
- Cache lines and false sharing — why counters get padded to 64/128 bytes and never shared between threads: ch00a
- rdtsc / cycle stamps — the ~2ns timestamps the whole design is built around: ch00e
- HdrHistogram — where all the recorded samples end up: ch00e
- Syscalls and their cost — why even
getpidor a pipe write is banned from the hot loop: ch00b- Jitter suspects — the tail-spike mechanisms this instrumentation exists to catch: ch00e
Read those first — 20 minutes there saves an hour here.
Measurement costs latency, and the paths most worth measuring are the ones with the least latency to spare. Done naively, the instrumentation becomes the jitter you’re hunting. Done well, the hot path stays observed in production — during the incident, not just in the lab — for a cost you can state and defend.
Budget it like everything else
You already accept this cost in web systems: the Datadog or New Relic agent runs in production, and you know it isn’t free. The only difference here is that the overhead gets a stated number and an owner.
Rule of thumb worth adopting and quoting: observability spends ≤1% of the path budget. A 5µs tick-to-trade path (ch00f’s wire-in → order-out scoreboard number) affords ~50ns of instrumentation — roughly a dozen rdtsc stamps and counter bumps, and nothing else. The exact number matters less than the discipline it enforces: instrumentation is a line item in the latency budget with an owner, not a free action. Every technique below exists to fit under it.
Corollary: the instrumentation must be always on. This is the Sentry principle — you don’t install error tracking after the outage. A measurement path that’s compiled out in production has two failure modes — it perturbs the system when you finally enable it, and it’s off during the incident you needed it for. The perturbation is physical, not superstition: compiling the stamps back in shifts every instruction address after them, which reshuffles cache and branch-predictor layout — timings move even in code you didn’t touch. It’s a Heisenbug factory: the act of looking changes the thing observed. Pay the 1% permanently; design so 1% is enough.
Cheap counters: per-thread, padded, aggregated off-path
This section is statsd done right: each worker bumps counters nobody else touches, and a scraper sums them once a second. The anti-pattern is every worker incrementing one shared row — the hot-row contention you’d never design into Postgres.
The wrong way: a shared AtomicU64 incremented by several threads — the cache line
ping-pongs between cores and every increment pays a cross-core transfer
(the microbenchmarks chapter’s table, ch10;
ch00a for the mechanism). The
mutex-guarded metrics struct is the same mistake with extra steps.
The right way: each thread owns its counters on cache lines nobody else writes; a cold aggregator thread reads them at 1Hz. Writes are plain-ish stores to an L1-resident line (~1ns); the reader’s once-a-second reads cost the hot thread at most one line transfer per second. The accounting behind that claim, in two sentences: when the aggregator reads your line, your core keeps a copy but gives up exclusive ownership, so your next write has to fetch the line back before it can proceed. That fetch-back is the entire cost — and it happens once per aggregator read, i.e. once per second.
hot thread 1 hot thread 2 hot thread 3
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ one padded counter
│ counters #1 │ │ counters #2 │ │ counters #3 │ block per thread —
│ (own lines, │ │ (own lines, │ │ (own lines, │ sole writer, cheap
│ sole writer)│ │ sole writer)│ │ sole writer)│ stores
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└───────────────────┼───────────────────┘
│ 1Hz reads (cold thread)
┌───────▼───────┐ ┌─────────────────┐
│ aggregator │──────►│ /metrics scrape │
│ (sums blocks) │ │ (Prometheus) │
└───────────────┘ └─────────────────┘
One wrinkle in the padding: Intel’s adjacent-line prefetcher pulls cache lines in
pairs, so “your own line” really means “your own 128-byte pair” — hence the
align(128) below rather than 64.
#![allow(unused)]
fn main() {
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
/// One cache line (2 on Intel to defeat the adjacent-line prefetcher pair).
#[repr(align(128))]
pub struct StageCounters {
pub events: AtomicU64,
pub bytes: AtomicU64,
pub queue_full: AtomicU64,
pub max_cycles_seen: AtomicU64, // reset by aggregator
}
impl StageCounters {
#[inline(always)]
pub fn bump(&self, bytes: u64) {
// Sole writer -> no RMW contention; Relaxed is correct: these are
// statistics, not synchronization.
self.events.fetch_add(1, Relaxed);
self.bytes.fetch_add(bytes, Relaxed);
}
}
}Because each counter block has exactly one writer, fetch_add(Relaxed) never
contends; on x86 you could even use plain load+store, but the uncontended RMW is
~20 cycles and saves you an argument with Miri, Rust’s undefined-behaviour
checker. The aggregator sums per-thread
blocks and that is what your Prometheus endpoint serves — the scrape never touches
hot-thread state directly, and there is no lock anywhere a hot thread can see.
Ring-buffer event logging
Counters tell you rates; incidents need events: “what were the last 10,000 things this thread did, with timestamps.” The tool is a fixed-size ring of fixed-size binary records, written by the hot thread, drained (or deliberately not drained — see tail capture) by a cold thread.
This is the pattern the event-sourcing chapter (ch13) will formalize. The two-sentence sketch: instead of storing current state, you store an append-only log of small immutable facts (“what happened”), and any state you want is rebuilt by replaying the log. Here the “domain” is the pipeline itself: the hot path appends facts, and consumers rebuild any view they want — histograms, dashboards — off-path. Same log discipline, same replayability, same single-writer append — if you can defend this ring in an interview, that chapter’s event store will feel familiar, and connecting the two unprompted is worth doing.
#![allow(unused)]
fn main() {
/// 32-byte POD record ("plain old data": fixed-size bytes, no pointers, no heap).
/// No strings, no Debug formatting — ever.
#[derive(Clone, Copy)]
#[repr(C)]
pub struct Event {
pub tsc: u64, // raw cycles; convert off-path
pub kind: u16, // enum discriminant
pub stage: u16, // pipeline stage id
pub a: u32, // e.g. symbol id
pub b: u64, // e.g. order id / seq
pub c: u64, // payload (price, size, latency, ...)
}
pub struct EventRing {
buf: Box<[Event]>, // power-of-two length
mask: usize,
head: std::sync::atomic::AtomicU64, // writer-owned; reader loads
}
impl EventRing {
#[inline(always)]
pub fn push(&self, ev: Event) {
use std::sync::atomic::Ordering::*;
let h = self.head.load(Relaxed);
// SAFETY: single writer; slot ownership by index math.
unsafe {
let slot = self.buf.as_ptr().add((h as usize) & self.mask) as *mut Event;
slot.write(ev);
}
self.head.store(h + 1, Release); // publish: Release orders the slot
// write first, so any reader that sees the new head value is
// guaranteed to also see the slot's bytes
}
}
}Cost of push: an rdtsc (if stamping), a few stores to a line that’s usually
L1-resident, one release store. ~5–15ns. The drain thread walks head snapshots,
converts cycles→ns, feeds HdrHistograms, writes to disk — all at its leisure, on a
non-isolated core.
Overwrite policy is a choice: a drained ring must handle wrap (drop + count
ring_dropped — never block the writer); an undrained “flight recorder” ring is
supposed to overwrite (below).
Sampling and late materialization
Two multipliers that stretch the 1% budget: sampling and late materialization.
Sampling (1-in-N). Anything expensive to capture — full order-book snapshots,
deep stamp sets, payload copies — gets recorded for only a fraction of events:
if seq & 1023 == 0 { capture_expensive() }, where the bit-mask just means
“every 1024th event.” The guard branch is ~free because the CPU’s branch
predictor guesses the skip in advance and is right 99.9% of the time.
Won’t sampling miss the tail event? No — because you always record the cheap latency stamp for every event, and only sample the expensive context around it.
Better yet, bias the sampling: capture the expensive context conditionally on the observed latency exceeding a threshold. Now the expensive data exists for exactly the events you’ll end up investigating.
Late materialization — the same instinct as storing user_id, not the user’s
full name, in every log row. The hot path logs ids, not strings: symbol id, not
"AAPL"; enum discriminant, not a message; raw cycles, not formatted time. The
drain thread joins ids against tables and formats. format! is a heap allocation
plus a traversal — 100ns–1µs — and every one of them in a hot path is someone’s
future p99.9 spike. Format off-path, always.
Same idea at stage boundaries: the hot thread’s whole timing duty is
ring.push(Event{ tsc: rdtsc(), stage: STAGE_DECISION, .. }). Histogram math,
percentiles, cycle conversion: all cold.
What NOT to do in a hot path
Each of these has appeared in a real trading system’s hot path, and each is a tail event generator:
- syslog / any logging framework call: formatting + locks + possibly a syscall (and a blocking write if the disk hiccups). ms-scale worst case.
format!/to_string: allocation + formatting. Even “just for the error path” — error paths during a burst are exactly when you can’t afford it.- Mutex-guarded metrics (
Mutex<HashMap<String, f64>>is the classic): the failure mode is priority inversion — the hot thread hits the lock at the exact moment the scrape thread holds it, and the scrape thread, being low-priority, has been descheduled by the kernel mid-critical-section. Your fastest thread is now waiting on the slowest thread’s nap: a 1Hz scrape holding the lock for 100µs turns into hot-thread stalls. - Prometheus client structs with internal locks / registry lookups per event: metric lookup by string name per increment is a hash + lock. Resolve handles at startup; better, keep the whole scrape surface on the aggregator only.
- Unbounded anything: a Vec of events that grows until the reallocation lands on your worst burst.
- Innocent syscalls:
getpid,write(2)to a pipe “just for a heartbeat” — syscalls are 100ns+ and a scheduling opportunity: crossing into the kernel is exactly where the scheduler is allowed to take your core away, so a “harmless” heartbeat write can return milliseconds later. The bpftrace syscall check (the profiling chapter, ch09) should stay empty.
Watchdogs: heartbeat and deadline monitoring
Latency observability tells you how fast you were; a watchdog tells you that you’ve stopped — and in trading, a stalled system with live orders is the emergency.
- Heartbeat: each hot thread stores
last_seen = rdtsc()into its padded counter block every loop iteration (a store it already pays). A cold monitor thread checks each block at ~1ms:now - last_seen > threshold→ alert / cancel orders / trip the kill switch. Cost to the hot path: one store. Value: a stall detector with millisecond reaction, immune to the stalled thread’s own inability to report. - Deadline monitor: for event-driven stages, the watchdog checks progress against input: producer seq is advancing while consumer seq isn’t → the stage is wedged even though its thread might be spinning “alive.” This is exactly Kafka consumer-lag alerting: the consumer process is up, but its offset has stopped moving while the topic’s head keeps advancing. Compare sequence numbers, not just heartbeats.
- Escalation is domain logic: a wedged market-data thread means your book is stale — the correct automated response (pull quotes) belongs to the watchdog, not to a human reading a dashboard 45 seconds later.
Always-on tail capture: the flight recorder
The p99.99 event will not happen while you’re watching. The trick that catches it is Sentry’s breadcrumbs / session replay, applied to nanoseconds: always be recording the recent past over itself, and when something goes wrong, freeze the recording. The pattern:
- Keep a pre-crisis ring: the last N events (say 64k), always being overwritten, never drained — an aircraft flight recorder.
- On a trigger — watchdog trip, latency threshold breach, crash handler, SIGTERM — freeze and dump the ring: you now hold the complete, timestamped, per-stage event sequence for the milliseconds leading into the incident.
- Triggered from the latency path itself: consumer observes end-to-end cycles > threshold → snapshot the ring alongside. You get causality (“the 40µs outlier was preceded by 900 queue_full events in stage 2”), not just a number.
- Cost: the ring writes you were already making. The freeze/dump is off-path and rare.
This closes the loop with the profiling chapter (ch09): perf and ftrace reconstruct the system’s view of an incident; the flight recorder holds the application’s view; the rdtsc stamps let you join the two timelines.
Plain-English recap
- The 1% budget is the APM-agent overhead you already accept — made explicit. You run the Datadog agent in production knowing it costs something; here the cost is a stated line item (~50ns on a 5µs path) with an owner, and every technique exists to fit under it.
- Always-on is the Sentry principle. You don’t install error tracking after the outage. Instrumentation that gets compiled in “when needed” is off during the incident and perturbs the system when enabled — so it’s on permanently and its cost is part of every number you quote.
- Per-thread counters + cold aggregator is statsd done right. Each worker
bumps its own local counters; a scraper sums them once a second. The anti-pattern
— every worker incrementing one shared row — is exactly the hot-row contention
you’d never design into Postgres, yet
Mutex<HashMap>metrics do it in memory. - The event ring is event sourcing (ch13) applied to telemetry. Small immutable facts appended by one writer, projections (histograms, dashboards) built downstream at leisure — a double-entry ledger for the pipeline itself.
- Late materialization is storing
user_id, not the user’s name, in every log row. The hot path logs ids and raw cycle counts; the drain thread does the joins and formatting — the same reason you don’t denormalize and stringify at write time in a high-volume table. - Watchdogs are liveness probes plus consumer-lag alerts. The heartbeat store is a k8s liveness check at 1000× the resolution; the deadline monitor (producer seq advancing, consumer seq stuck) is exactly Kafka consumer-lag alerting — a thread can be “alive” and still wedged.
- The flight recorder is Sentry breadcrumbs / session replay. The last 64k events are always being recorded over themselves; the error (latency breach, watchdog trip) freezes and dumps them, so you get the milliseconds leading into the incident, not just the incident.
Interviewer will ask
Q: How much latency does your instrumentation add, and how do you know? A: It’s budgeted: ≤1% of the path — for a 5µs path, ~50ns, which buys a handful of rdtsc stamps, per-thread counter bumps, and one ring push. And it’s measured like any other change: A/B replay with instrumentation compiled in vs stubbed, comparing full histograms — the stubbed build exists only in the lab harness, to price the stamps; production never runs that way. It’s always on, so production numbers include its cost.
Q: Why not a shared atomic counter for a metric several threads bump? A: Contended atomic RMW makes the cache line ping-pong — each increment pays a cross-core transfer, ~100+ cycles, and it scales negatively with threads. Per-thread counters on padded (128B on Intel) lines are ~1ns sole-writer increments; a 1Hz aggregator sums them. Same totals, none of the coherence tax.
Q: Your logging is a binary ring. What happens when it fills? A: Policy is explicit per ring. Drained telemetry rings drop-and-count on overrun — the writer never blocks, and
ring_dropped > 0is itself an alert that the drain is undersized. Flight-recorder rings are the opposite: designed to overwrite forever and only read after a trigger freezes them.Q: How do you log “what happened” without strings in the hot path? A: Late materialization: fixed-size POD records — tsc, enum discriminants, symbol/order ids, raw values. The cold drain joins ids to names and formats. Formatting is allocation plus traversal, 100ns to 1µs; ids are stores. Same reason the wire protocols we parse are binary.
Q: How would you detect that a hot thread has stalled, within a millisecond, without adding a syscall to its loop? A: Heartbeat store: the thread writes rdtsc to its own padded slot each iteration — a store it can afford — and a cold watchdog polls all slots at 1kHz comparing against now. Plus deadline monitoring on queue sequence numbers to catch a thread that’s spinning but not progressing. The watchdog owns the escalation — pull quotes first, page humans second.
Q: Prometheus in a trading system — where does it fit? A: At the edge only. Hot threads write per-thread counters and rings; an aggregator thread materializes those into Prometheus metrics and serves the scrape. The scrape path and any locks it needs exist only on the aggregator. The hot path neither knows nor cares that Prometheus exists.
Q: You see a 40µs outlier in the histogram. What’s your next artifact? A: The flight-recorder dump keyed to it: the consumer that observed the breach froze the last-64k-event ring, so I have the timestamped per-stage sequence leading into the spike — queue depths, stage deltas, event kinds. I correlate its tsc range with sched/ftrace data to decide app-cause vs system-cause. Histograms locate that something happened; the ring says what.
Further reading
- Gil Tene, “How NOT to Measure Latency” — the service-time/response-time framing that motivates always-on production measurement.
- Brendan Gregg, Systems Performance (2nd ed.) — observability-tool overhead and the “observer effect” discussions.
- The LMAX Disruptor technical paper (Thompson et al.) — the single-writer principle and mechanical sympathy behind the ring design.
- HdrHistogram docs — interval histograms and the recorder/double-buffer pattern for lock-free histogram handoff to a reader thread.
- Martin Fowler’s write-ups on Event Sourcing and the LMAX architecture — the architectural pattern this chapter’s telemetry design mirrors.
Where this goes next: Chapter 12 welds the measurement chapters (ch07–ch11) into one runnable lab — a TSC-stamped three-stage pipeline you deliberately break four ways and watch each break appear in the histograms.
Lab II: Instrumenting a Pipeline End-to-End
Before you start — this lab exercises the whole of Part II at once; the primer pages worth having fresh:
- TSC / rdtsc and cycle→ns calibration — every stamp in the lab is raw cycles: ch00e
- HdrHistogram and percentiles — where the samples land and how to read the output table: ch00e
- Coordinated omission / open-loop load — why the producer paces by intended send time: ch00e
- Cores, pinning, hyperthreads, false sharing — what experiments (b) and (c) actually break: ch00a
- perf counters (page faults, context switches, IPC) — the fingerprints step (d) checks: ch00e
Read those first — 20 minutes there saves an hour here.
Everything from the measurement chapters (ch07–ch11), welded into one runnable artifact: a three-stage pipeline (producer → SPSC → transformer → SPSC → consumer; SPSC = the single-producer single-consumer queues from the microbenchmarks chapter, ch10), TSC-stamped at every hop, aggregated off-path into per-stage HdrHistograms. Then you break it four ways on purpose and watch each break appear in the numbers. This lab is the difference between having read Part II and being able to say things in an interview.
Climb the ladder first. The capstone below arrives fully assembled — calibration, open-loop pacing, off-path aggregation all pre-built — and reading someone else’s finished instrument teaches much less than building each instrument yourself. So three warm-ups come first, one per craft, ~15 minutes each, each a standalone binary in the same crate. Do them in order; every design decision in the capstone will then be something your own hands have already made.
| Warm-up | Chapter | What you build | The moment it lands |
|---|---|---|---|
| W1 | ch07 | Calibrate the cycle counter; price a fence | Your counter drifts by this many ppm; a fenced read costs this much |
| W2 | ch08 | Two harnesses over one stalling system | Same stall: one harness reports 52µs p99.9, the other 103ms |
| W3 | ch10 | Bench a queue; check it against the ladder | Your “work” benchmarks at 0.000ns/op — because it was deleted |
Warm-up 1 — Own your clock (ch07)
The capstone calls calibrate() and moves on. Do it yourself once, and answer the three questions ch07 says every timestamp rests on: is the ratio stable, does it drift, and what does a correct read cost?
Fair warning: this file reaches below the language, so it contains three things a Node developer has likely never typed — a compile-time architecture switch, an unsafe block, and one line of inline assembly. Each is glossed in a comment at first use; the comments are part of the lesson.
Two pictures to hold before you read it.
The fence. A modern CPU does not run your instructions in the order you wrote them: it executes whatever is ready first, and only guarantees the results come out as if they ran in program order. Normally that reordering is free speed. But a timestamp read is an ordinary instruction — the CPU may execute it before the work you meant to time has finished, and the as-if guarantee does nothing to protect a measurement. A fence (lfence on x86, isb on ARM) forbids that: the counter read may not start until every earlier instruction has completed. Without it, your stopwatch can click before the race ends.
Calibration. The cycle counter is a car’s odometer that counts in ticks, not kilometres: it tells you how many ticks passed, never how long a tick is. Calibration is driving a known distance — let the OS clock (which does speak nanoseconds) run for 200ms, count the ticks that elapsed, divide. From then on you own the exchange rate: ticks per nanosecond.
Those pictures say why; here is where each read actually travels:
USERSPACE │ KERNEL
│
(1) Instant::now() ─► read the vDSO page ◄─┼── kernel updates this page
~20–40ns — the page is kernel data │ on its own tick; the
mapped into your process, so no │ crossing happened earlier,
boundary is crossed at call time │ not on your call
───────────────────────────────────────────┤
(2) cycles() ─► rdtsc: copy the TSC │ (never involved)
register — ~6–10ns, one instruction, │
and the value never leaves the core │
src/bin/w1_clock.rs:
// Calibrate the cycle counter yourself, then price a fence.
use std::time::Instant;
// The CPU keeps its own tick counter running in hardware — a count that has
// been ticking since boot: no epoch, no unit, just a number, no OS involved.
// This function copies it out. On x86 it
// ticks billions of times a second; ARM's counter can be far coarser (~24MHz
// on some parts — see the note below). Either way: the cheapest timing read you have.
#[inline(always)] // "paste the body in place of every call" — a function
// call costs as much as the thing we're trying to time
fn cycles() -> u64 {
// Compile-time switch: only the branch for the CPU you're building for
// exists in the binary at all.
#[cfg(target_arch = "x86_64")]
// `unsafe` = "compiler, you can't verify this; I vouch for it". Copying
// out a counter the CPU maintains anyway is harmless — it just sits
// outside Rust's safety model. _rdtsc grabs the Time Stamp Counter
// (TSC), x86's name for that hardware tick counter.
unsafe { core::arch::x86_64::_rdtsc() }
#[cfg(target_arch = "aarch64")]
// Same tick counter, ARM flavour — no library helper exists, so we write
// the single CPU instruction ourselves; it copies the counter into `v`.
// (asm! = embed one raw CPU instruction; `mrs` reads a CPU-internal slot;
// `cntvct_el0` is that counter's name; `out(reg) v` = "put the answer in v".)
{ let v: u64; unsafe { core::arch::asm!("mrs {}, cntvct_el0", out(reg) v) }; v }
}
// Fenced read: every earlier instruction completes before the counter is
// sampled. Without this, the out-of-order CPU can click your stopwatch
// before the work it's supposedly timing has finished (ch07).
#[inline(always)]
fn cycles_fenced() -> u64 {
#[cfg(target_arch = "x86_64")]
// _mm_lfence is the fence: the CPU may not start the counter read until
// every earlier instruction has actually finished.
unsafe { core::arch::x86_64::_mm_lfence(); core::arch::x86_64::_rdtsc() }
#[cfg(target_arch = "aarch64")]
// `isb` (instruction synchronization barrier) — ARM's equivalent fence:
// everything earlier finishes, then the counter is read.
{ let v: u64; unsafe { core::arch::asm!("isb; mrs {}, cntvct_el0", out(reg) v) }; v }
}
// The odometer calibration from the picture above: drive `ms` milliseconds
// by the OS clock, count the ticks that passed, return ticks (cycles) per ns.
fn calibrate(ms: u64) -> f64 {
let t0 = Instant::now();
let c0 = cycles();
// Busy-wait: keep re-checking the clock in a tight loop rather than
// handing the core back to the OS — NOT a sleep, NOT a yield. spin_loop()
// adds the CPU's `pause` hint, an "I'm just waiting" courtesy to the
// core and its hyperthread sibling.
while t0.elapsed().as_millis() < ms as u128 { std::hint::spin_loop(); }
(cycles() - c0) as f64 / t0.elapsed().as_nanos() as f64
}
// Answer ch07's three questions: is the ratio stable, does it drift,
// and what does a correct (fenced) read cost?
fn main() {
// 1. Three calibrations: is the ratio stable run to run?
for _ in 0..3 { println!("calibration: {:.4} cycles/ns", calibrate(200)); }
// 2. Drift: does a 3-second window in cycles agree with the OS clock?
let ghz = calibrate(200);
let t0 = Instant::now();
let c0 = cycles();
while t0.elapsed().as_secs() < 3 { std::hint::spin_loop(); }
let os_ns = t0.elapsed().as_nanos() as f64;
let tsc_ns = (cycles() - c0) as f64 / ghz;
println!("3s window: os={:.3}ms tsc={:.3}ms drift={:+.1}ppm",
os_ns / 1e6, tsc_ns / 1e6, (tsc_ns - os_ns) / os_ns * 1e6);
// 3. What a read costs, unfenced vs fenced.
let mut raw = Vec::with_capacity(100_000);
let mut fen = Vec::with_capacity(100_000);
// wrapping_sub is odometer-rollover math: if the free-running counter
// ever rolls past its maximum between two reads, wrapping around zero
// still gives the true distance travelled (plain `-` would panic in
// debug builds instead).
for _ in 0..100_000 { let a = cycles(); let b = cycles(); raw.push(b.wrapping_sub(a)); }
for _ in 0..100_000 { let a = cycles_fenced(); let b = cycles_fenced(); fen.push(b.wrapping_sub(a)); }
raw.sort_unstable(); fen.sort_unstable();
println!("read cost unfenced p50={:.1}ns fenced p50={:.1}ns",
raw[50_000] as f64 / ghz, fen[50_000] as f64 / ghz);
}Read your output against ch07: calibrations agreeing to ~3 decimals means the counter is invariant (it doesn’t change rate with frequency); drift in the low hundreds of ppm is normal crystal error — which is exactly why the capstone recalibrates at every startup rather than hardcoding a GHz. The fence delta is the number that decides your instrumentation budget: it’s what each honest hot-path stamp costs, and it’s why the capstone stamps a handful of points rather than everywhere.
Warm-up 2 — Make the benchmark lie to you (ch08)
The capstone’s producer is open-loop, and one comment tells you why. That’s a claim. This warm-up is the demonstration — and it’s the single most valuable fifteen minutes in Part II, because it makes coordinated omission something you watched happen rather than a term you recite.
One toy system with one injected 100ms freeze, measured two ways. Time flows down; the freeze hits at request #10000 in both lanes:
time CLOSED — wait, then send OPEN — send at timetable ticks
│
│ (1) send #9999 ─► reply: 20µs (1) tick 9999: send ─► 20µs
│ (2) send #10000 ─► ▓▓▓▓▓▓▓▓ (2) tick 10000: send ─► ▓▓▓▓▓▓▓
│ 100ms STALL — harness (3) ticks 10001…11000 come due
│ WAITS; the ~1,000 sends during the stall: each stamped
│ due in this gap never exist, from its INTENDED tick, all
│ their waits unrecorded queueing up behind #10000
│ (3) reply ─► ONE 100ms sample (4) stall ends, backlog drains:
│ (4) sends #10001+ bunch up #10000 records 100ms, #10001
▼ here, ~20µs each again ≈99.9ms … ~1,000 true waits
closed: (1,2) t0=Instant::now(); service(seq) (3,4) h.record(t0.elapsed())
open: (1,2) sleep to intended=start+period·seq (3,4) h.record(intended.elapsed())
src/bin/w2_co.rs:
// Run: w2_co closed then w2_co open
use hdrhistogram::Histogram;
use std::time::{Duration, Instant};
const N: u64 = 20_000;
const RATE_HZ: u64 = 10_000; // 100µs between intended sends
const STALL_AT: u64 = 10_000; // one freeze, mid-run
const STALL: Duration = Duration::from_millis(100);
// The system under test: ~20µs of "work" per request, except one injected
// 100ms freeze at request STALL_AT.
fn service(seq: u64) {
if seq == STALL_AT { std::thread::sleep(STALL); } // the freeze
else { std::thread::sleep(Duration::from_micros(20)); } // normal work
}
// Drive N requests through service() closed- or open-loop; print percentiles.
fn main() {
let mode = std::env::args().nth(1).unwrap_or_else(|| "closed".into());
let mut h = Histogram::<u64>::new_with_bounds(1, 60_000_000_000, 3).unwrap();
let period = Duration::from_nanos(1_000_000_000 / RATE_HZ);
let start = Instant::now();
for seq in 0..N {
if mode == "closed" {
// CLOSED LOOP: wait for the reply, then send the next request.
// During the freeze we simply stop sending, so
// the requests that *should* have gone out never exist to measure.
let t0 = Instant::now();
service(seq);
h.record(t0.elapsed().as_nanos() as u64).unwrap();
} else {
// OPEN LOOP: a bus timetable — send times fixed in advance whether
// or not the system keeps up. Latency runs from the INTENDED send
// time, so the freeze lands in every sample that was due during it.
let intended = start + period * seq as u32;
let now = Instant::now();
if intended > now { std::thread::sleep(intended - now); }
service(seq);
h.record(intended.elapsed().as_nanos() as u64).unwrap();
}
}
let us = |v: u64| v as f64 / 1000.0;
println!("{mode:>6}: n={} p50={:.1}us p99={:.1}us p99.9={:.1}us max={:.1}us",
h.len(), us(h.value_at_quantile(0.50)), us(h.value_at_quantile(0.99)),
us(h.value_at_quantile(0.999)), us(h.max()));
}Real output from this exact code:
closed: n=20000 p50=29.3us p99=48.3us p99.9=52.0us max=105054.2us
open: n=20000 p50=47.6us p99=91029.5us p99.9=103678.0us max=105054.2us
Sit with that. Same system. Same 100ms freeze. Same sample count. The closed-loop harness reports a p99.9 of 52µs — a system that looks healthy — the left lane of the diagram, where the stall’s victims never existed to be measured. Only max betrays it, which is precisely why ch08 says a max wildly detached from your percentiles is a coordinated-omission fingerprint, not an outlier to discard. The open lane’s thousand victims each carry their share of the freeze: p99 of 91ms, p99.9 of 104ms — the truth.
Now you know what the capstone’s let intended = start + seq * interval; line is defending against, and you have the sentence: “I’ve built the same measurement both ways over an injected stall — closed loop hid a 100ms freeze behind a 52µs p99.9.”
Warm-up 3 — Bench one component, then distrust it (ch10)
The capstone uses rtrb and mentions in passing that you could swap in your own queue “to bench it.” Benching it is a skill, and it has a trapdoor.
The trapdoor has a name: dead-code elimination. If a computation’s result is never used, the optimizer doesn’t make the work faster — it deletes it, loop and all, and your benchmark times an empty shell. black_box is a wall the optimizer can’t see through: it must assume the value going in gets used and the value coming out could be anything, so the work on your side of the wall has to actually happen.
And hold this picture of what the queue benchmark below does — and doesn’t — exercise:
core N — the whole benchmark lives in one lane
┌────────────────────────────────────────────────────────┐
│ (1) p.push(i) ─► ring slot lands in this core's L1 │
│ (2) c.pop() ◄── same slot, same L1, still warm │
└────────────────────────────────────────────────────────┘
── core boundary ── never crossed: that's why push+pop benches
at single-digit ns below — and why the number says nothing about
the cross-core case, where every handoff must move a cache line
src/bin/w3_bench.rs:
// black_box: the wall from the paragraph above — the optimizer must actually
// compute what goes in and may assume nothing about what comes out, so the
// measured work can't be deleted or hoisted out of the loop.
use std::hint::black_box;
use std::time::Instant;
// Cycle-counter read, exactly as in warm-up 1 (rdtsc / `mrs cntvct_el0`).
#[inline(always)]
fn cycles() -> u64 {
#[cfg(target_arch = "x86_64")]
unsafe { core::arch::x86_64::_rdtsc() }
#[cfg(target_arch = "aarch64")]
{ let v: u64; unsafe { core::arch::asm!("mrs {}, cntvct_el0", out(reg) v) }; v }
}
// Cycles per nanosecond, calibrated as in warm-up 1.
fn calibrate() -> f64 {
let t0 = Instant::now();
let c0 = cycles();
while t0.elapsed().as_millis() < 200 { std::hint::spin_loop(); }
(cycles() - c0) as f64 / t0.elapsed().as_nanos() as f64
}
const ITERS: u64 = 5_000_000;
// Three benchmarks: one the optimizer deletes, one it can't, one real queue.
fn main() {
let ghz = calibrate();
// (1) THE TRAP: nothing consumes the result, so the optimizer deletes the work.
let t = cycles();
for i in 0..ITERS { let _ = i * 7 + 3; }
let dce = (cycles() - t) as f64 / ITERS as f64 / ghz;
// (2) Same arithmetic, but black_box makes the optimizer believe it's used.
let t = cycles();
for i in 0..ITERS { black_box(black_box(i) * 7 + 3); }
let real = (cycles() - t) as f64 / ITERS as f64 / ghz;
println!("multiply-add without black_box: {dce:.3}ns/op with: {real:.3}ns/op");
// (3) The component under test: same-thread push+pop, everything hot in L1.
let (mut p, mut c) = rtrb::RingBuffer::<u64>::new(1024);
let t = cycles();
for i in 0..ITERS {
p.push(black_box(i)).ok();
black_box(c.pop().ok());
}
let spsc = (cycles() - t) as f64 / ITERS as f64 / ghz;
println!("rtrb push+pop (same thread, hot in L1): {spsc:.2}ns/op");
}Real output:
multiply-add without black_box: 0.000ns/op with: 0.331ns/op
rtrb push+pop (same thread, hot in L1): 2.68ns/op
0.000ns/op. Dead-code elimination, caught red-handed in your own harness — ch10’s first trap, on your box, in ten seconds. Any benchmark result at or near zero is not a fast implementation; it is an absent one.
Then apply ch10’s plausibility ladder to the 2.68ns: the diagram above shows why single-digit ns is physically reasonable — no boundary crossed. But now predict before you measure: split producer and consumer across cores and the floor jumps to tens of ns. If a cross-core version still reports 2.68ns, the harness is lying (compiler hoisting, or the consumer never actually seeing the producer’s writes). This is the habit ch10 exists to build: a benchmark number is a hypothesis you test against physics, not a result you report. In production use criterion for this (outlier classification, confidence intervals); the hand-rolled version here is to make the trapdoor visible.
The design
core 1 │ core 2 │ core 3 │ unpinned core
producer │ transformer │ consumer │ aggregator
│ │ │
(1) t0 = intended │ │ │
(2) p1.push ═q1═══╪═► (3) c1.pop │ │
Msg{t0} │ (4) t1 = rdtsc │ │
│ (5) work ~100ns │ │
│ (6) t2 = rdtsc │ │
│ (7) p2.push ═q2═══╪═► (8) c2.pop │
│ Msg{t0,t1,t2} │ (9) t3 = rdtsc │
│ │ (10) pa.push ═ring═╪═► (11) ca.pop
│ │ Sample{deltas}│ cycles→ns →
── core boundary ─┴─ core boundary ─────┴── core boundary ───┴ HdrHistograms
d_q1 = t1−t0 spans cores 1→2 (queue + wake) d_work = t2−t1 inside core 2
d_q2 = t3−t2 spans cores 2→3 d_e2e = t3−t0 spans them all
- The SPSC rings are drawn on the core boundaries because that’s where they live: the only shared state between adjacent lanes. Hop (10)→(11) is the “event ring drained by a cold thread” pattern from ch11.
- All hot-path stamps are raw cycles; the aggregator’s cycles→ns conversion uses a startup calibration (ch07, ch08).
- The producer is open-loop — warm-up 2’s right-hand lane, wired in: sends at intended times paced by cycles (coordinated omission, ch08).
- Runs on x86_64 (rdtsc) and aarch64 (
cntvct_el0, ARM’s counterpart to the TSC), so you can develop on the Mac and take real numbers on the Linux box. Believe only the Linux numbers: macOS won’t let you pin threads to cores, and the ARM counter is far coarser than the TSC (quantified in the notes below).
Cargo.toml
[package]
name = "pipeline-lab"
version = "0.1.0"
edition = "2021"
[dependencies]
rtrb = "0.3" # proven SPSC ring; warm-up 3 benches it
hdrhistogram = "7"
core_affinity = "0.8"
[profile.release]
debug = true # keep symbols for perf (the profiling chapter)
The three warm-up binaries (src/bin/w1_clock.rs, w2_co.rs, w3_bench.rs) and the capstone (src/main.rs) all live in this one crate and share these dependencies. Run a warm-up with cargo run --release --bin w1_clock.
Four pictures before the code. The capstone reads much faster with these held in your head; every comment below hangs off one of them.
The 64-byte cache line. A core never fetches memory one byte at a time — it fetches a fixed 64-byte block called a cache line, and only one core may hold a line’s writable copy at a moment. Put two counters on the same line and two cores that each only touch their own counter still pull the entire line back and forth on every bump — the line, not the byte, is the unit of ownership. That’s false sharing (the ping-pong is drawn step by step at experiment (c) below), and the capstone’s SLOTS array is wired to switch it on and off: slots 0 and 1 sit on one line; slots 0 and 16 sit 128 bytes apart, safely on separate lines.
AtomicU64, Relaxed. An AtomicU64 is a counter that many threads may increment at the same instant without a lock — the hardware guarantees no increment is ever lost. Relaxed is the cheapest promise level you can ask for: “count correctly, promise nothing about ordering relative to anything else” — which is all a statistics counter needs.
The rtrb ring. Each queue is a conveyor belt with exactly one loader and one unloader — single producer, single consumer, nobody else allowed to touch it. push sets an item on the belt, pop lifts one off, and if the belt is full, push hands the item back to you inside the error — nothing is ever dropped silently.
pin(). By default the OS scheduler may move a thread to a different core mid-run. Pinning fixes it to one core for the whole measurement, so “which core was it on” stops being a variable in your experiment.
src/main.rs (complete)
use hdrhistogram::Histogram;
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
use std::time::Instant;
// ---------- cycle counter (the clocks chapter) ----------
// The warm-up 1 read — the CPU's hardware tick counter, a raw count since
// boot: rdtsc on x86; on ARM, inline asm reading `cntvct_el0`.
#[inline(always)]
fn now_cycles() -> u64 {
#[cfg(target_arch = "x86_64")]
unsafe { core::arch::x86_64::_rdtsc() }
#[cfg(target_arch = "aarch64")]
{
let v: u64;
unsafe { core::arch::asm!("mrs {}, cntvct_el0", out(reg) v) };
v
}
}
/// Warm-up 1's odometer calibration: drive 200ms by the OS clock, count the
/// ticks that passed, return cycles per nanosecond.
fn calibrate() -> f64 {
let t0 = Instant::now();
let c0 = now_cycles();
while t0.elapsed().as_millis() < 200 { std::hint::spin_loop(); }
(now_cycles() - c0) as f64 / t0.elapsed().as_nanos() as f64
}
// ---------- pipeline messages ----------
#[derive(Clone, Copy, Default)]
struct Msg { seq: u64, payload: u64, t0: u64, t1: u64, t2: u64 }
#[derive(Clone, Copy)]
struct Sample { d_q1: u64, d_work: u64, d_q2: u64, d_e2e: u64 } // cycles
// ---------- experiment toggles (CLI) ----------
#[derive(Default, Clone, Copy)]
struct Cfg { alloc: bool, no_pin: bool, share_line: bool }
const N: u64 = 2_000_000; // events per run
const RATE: u64 = 200_000; // offered rate, events/sec (open loop)
const Q: usize = 4096; // queue capacity
/// Pin the calling thread to one core: the scheduler may never move it
/// to another core mid-measurement, taking placement out of the experiment
/// (ch00a).
fn pin(core: usize, cfg: &Cfg) {
if cfg.no_pin { return; } // experiment (b)
if let Some(ids) = core_affinity::get_core_ids() {
if let Some(id) = ids.get(core) { core_affinity::set_for_current(*id); }
}
}
// Counter slots: transformer bumps [ti], consumer bumps [ci].
// This array IS the cache-line picture from above. Each slot is an 8-byte
// AtomicU64 (lock-free counter), so index 1 sits 8B away — the SAME 64-byte
// line — while index 16 sits 16*8 = 128B away, safely on its own line
// (128B not 64B because Intel prefetches lines in adjacent pairs).
// share_line picks indices (0,1) — one shared line, experiment (c);
// otherwise (0,16) — separate lines, no ping-pong.
// `[const { ... }; 32]` is the idiom for building an array of atomics at
// compile time (they aren't copyable, so plain `[value; 32]` won't do).
static SLOTS: [AtomicU64; 32] = [const { AtomicU64::new(0) }; 32];
// Wire up the three SPSC rings, spawn the four threads, join, report.
fn main() {
let cfg = Cfg {
alloc: std::env::args().any(|a| a == "--alloc"),
no_pin: std::env::args().any(|a| a == "--no-pin"),
share_line: std::env::args().any(|a| a == "--share-line"),
};
let (ti, ci) = if cfg.share_line { (0usize, 1usize) } else { (0usize, 16usize) };
let ghz = calibrate();
eprintln!("calibrated {:.3} cycles/ns; cfg: {:?} {:?} {:?}",
ghz, cfg.alloc, cfg.no_pin, cfg.share_line);
let (mut p1, mut c1) = rtrb::RingBuffer::<Msg>::new(Q);
let (mut p2, mut c2) = rtrb::RingBuffer::<Msg>::new(Q);
let (mut pa, mut ca) = rtrb::RingBuffer::<Sample>::new(1 << 16);
// ---- producer: open-loop, paced by intended send time (latency chapter) ----
let interval = (ghz * 1e9 / RATE as f64) as u64; // cycles between sends
let producer = std::thread::spawn(move || {
pin(1, &cfg);
let start = now_cycles();
for seq in 0..N {
let intended = start + seq * interval;
// Busy-wait to the scheduled tick (spin_loop = pause hint, warm-up 1).
while now_cycles() < intended { std::hint::spin_loop(); }
// ..Default::default() = "every field I didn't name gets its zero
// value" — like spreading in an all-zeroes object literal.
let mut m = Msg { seq, payload: seq.wrapping_mul(0x9E37_79B9), t0: intended, ..Default::default() };
// Belt full: push hands the message back inside the error —
// take it back, wait a beat, set it on the belt again.
while let Err(rtrb::PushError::Full(v)) = p1.push(m) { m = v; std::hint::spin_loop(); }
}
});
// ---- transformer: dequeue, "strategy" work, enqueue ----
let transformer = std::thread::spawn(move || {
pin(2, &cfg);
for _ in 0..N {
let mut m = loop {
if let Ok(m) = c1.pop() { break m; } std::hint::spin_loop();
};
m.t1 = now_cycles();
let mut acc = m.payload; // fixed work: ~100ns
for _ in 0..40 { acc = acc.wrapping_mul(6364136223846793005).wrapping_add(1); }
if cfg.alloc { // experiment (a)
let v: Vec<u64> = vec![acc; 32]; // heap alloc in hot path
acc ^= v[31];
}
m.payload = acc;
// Relaxed = "count correctly, promise nothing about order" —
// the cheapest atomic mode, and all a stats counter needs.
SLOTS[ti].fetch_add(1, Relaxed); // stage counter
m.t2 = now_cycles();
while let Err(rtrb::PushError::Full(v)) = p2.push(m) { m = v; std::hint::spin_loop(); }
}
});
// ---- consumer: final stamp, ship deltas to cold aggregator ----
let consumer = std::thread::spawn(move || {
pin(3, &cfg);
let mut dropped = 0u64;
for _ in 0..N {
let m = loop {
if let Ok(m) = c2.pop() { break m; } std::hint::spin_loop();
};
let t3 = now_cycles();
SLOTS[ci].fetch_add(1, Relaxed);
// saturating_sub floors at zero instead of wrapping: if two cores'
// counter reads ever land a hair out of order, a tiny negative
// would otherwise wrap into a near-2^64 "latency" and wreck the
// histogram. A clamped zero beats a corrupt maximum.
let s = Sample {
d_q1: m.t1.saturating_sub(m.t0),
d_work: m.t2.saturating_sub(m.t1),
d_q2: t3.saturating_sub(m.t2),
d_e2e: t3.saturating_sub(m.t0),
};
if pa.push(s).is_err() { dropped += 1; } // never block the hot path
}
dropped
});
// ---- aggregator: cold thread, cycles->ns, HdrHistograms (observability chapter) ----
let aggregator = std::thread::spawn(move || {
let mk = || Histogram::<u64>::new_with_bounds(1, 10_000_000_000, 3).unwrap();
let (mut q1, mut wk, mut q2, mut e2e) = (mk(), mk(), mk(), mk());
let mut n = 0u64;
while n < N {
match ca.pop() {
Ok(s) => {
n += 1;
let ns = |c: u64| ((c as f64 / ghz) as u64).max(1);
q1.record(ns(s.d_q1)).ok(); wk.record(ns(s.d_work)).ok();
q2.record(ns(s.d_q2)).ok(); e2e.record(ns(s.d_e2e)).ok();
}
// Nothing waiting: hand the core back to the OS — the
// opposite of the hot threads' spin. A cold thread can
// afford that; a hot one never does it.
Err(_) => std::thread::yield_now(),
}
}
for (name, h) in [("q1+wake", &q1), ("work", &wk), ("q2+wake", &q2), ("e2e", &e2e)] {
println!("{:8} p50={:>7}ns p99={:>7}ns p99.9={:>8}ns max={:>9}ns",
name, h.value_at_quantile(0.5), h.value_at_quantile(0.99),
h.value_at_quantile(0.999), h.max());
}
});
producer.join().unwrap();
transformer.join().unwrap();
let dropped = consumer.join().unwrap();
aggregator.join().unwrap();
let transformed = SLOTS[ti].load(Relaxed);
let consumed = SLOTS[ci].load(Relaxed);
println!("processed: transform={transformed} consume={consumed} (expect {N} each)");
println!("aggregation ring dropped {dropped} samples (expect 0; >0 means the cold thread fell behind)");
}Build and run the baseline, pinned, on the Linux box:
cargo run --release # baseline
cargo run --release -- --alloc # experiment (a)
cargo run --release -- --no-pin # experiment (b)
cargo run --release -- --share-line # experiment (c)
Notes: cores 1/2/3 are assumed free (ideally isolated per Part I); the aggregator is
deliberately unpinned. On aarch64 cntvct_el0 ticks at ~24MHz–1GHz, so per-stage
resolution is coarser — at 24MHz one tick is ~42ns, bigger than some of the deltas
you’re trying to measure — the structure still works, the fine numbers don’t. If
dropped is ever nonzero, your aggregator ring is undersized; that counter is
itself a lesson from the observability chapter (ch11).
The experiments
Run baseline ≥3 times first; know your run-to-run variance before attributing anything. Then one variable at a time:
(a) --alloc — heap allocation in stage 2. One small Vec per event, and
the work path grows a side-trip:
USERSPACE — core 2, stage-2 work path │ KERNEL
│
(1) m.t1 = now_cycles() │
(2) 40× wrapping_mul (~100ns, fixed work) │
(3) --alloc: vec![acc; 32] — ask free list │
├─ usually: slot ready, pointer math, │
│ ~20–50ns, kernel never knows ─────┐ │
└─ sometimes: list empty ── syscall ─┼──┼─► (4) map + zero fresh
boundary crossed ─────────────────┘ │ pages, page faults —
(5) m.t2 = now_cycles() ◄────────────────────┼───── microseconds, not ns
the "sometimes" branch IS the tail: p50 barely moves, p99.9 spikes
Watch work p50 rise modestly (the usual branch) — but watch p99.9 and
max: the boundary crossings (free-list refills, madvise calls returning
memory to the kernel, occasional page faults) fire episodically. That is why
zero-alloc is a tail discipline, not a throughput one.
(b) --no-pin — let the scheduler place threads. Same pipeline, but the
lanes stop being fixed — the boundary is now the scheduler’s choice:
core 2 │ core 4 (boundary = wherever
transformer, caches warm │ the scheduler decides)
│
(1) c1.pop / work / p2.push … │
(2) scheduler evicts it ──────┼─► (3) resumes HERE: L1/L2 cold —
mid-run (another thread, │ every hot line refetched from
an IRQ, load balancing) │ L3 or the old core (~µs of misses)
│ (4) or worse: parked on the run queue
│ behind someone else — a spinning,
│ descheduled consumer takes ms to
│ notice new data waiting in its ring
Watch every stage’s p99+ inflate and become bimodal across runs (results
cluster into two distinct groups rather than one): sometimes two stages land
on SMT siblings of one physical core, sometimes migration (2)→(3) lands
mid-burst; q1/q2 wake latencies get noisy — (4) is what “wake” costs when
it goes wrong. Nothing in the code changed — only placement. Re-run five times and note the
variance explosion; that irreproducibility is the finding. Then close the loop
with the profiling chapter’s ftrace silence check (ch09): enable sched_switch/sched_wakeup
for a pinned run and again for a --no-pin run. The pinned run’s hot cores
trace silent; the unpinned run’s trace names every migration and preemption the
histograms felt.
(c) --share-line — stage counters on one cache line. Transformer and
consumer now bump adjacent AtomicU64s — the cache line from before the code,
in motion:
core 2 (transformer) │ core 3 (consumer)
L1: [ctrA|ctrB…] ← owns the line │ L1: (copy of the same 64B line)
│
(1) SLOTS[0].fetch_add ───────────┼─► (2) core 3's copy INVALIDATED —
needs the line writable │ ctrB rides on the same line
─── the 64-byte line transfers ───┼─► so core 3 must refetch it
│ (3) SLOTS[1].fetch_add ──────────
(4) core 2's copy INVALIDATED ◄───┼─── needs it back: invalidate,
…and the line comes back │ transfer, ~40–100ns each way
neither core ever reads the other's counter — but the line, not the
byte, is the unit of ownership, so it ping-pongs on ~every increment
Watch throughput (wall time for the run) degrade and work/q2 medians rise:
hops (2) and (4) now tax every bump. Confirm the mechanism with perf c2c record — the slots array shows
up with HITM at two offsets on one line (HITM = the line was fetched dirty out of
the other core’s cache — the smoking gun of ping-pong; the false-sharing
signature from ch09).
(d) [Linux] correlate with perf stat. For baseline vs each experiment:
perf stat -e cycles,instructions,cache-misses,LLC-load-misses,\
context-switches,page-faults -- cargo run --release -- --share-line
Predictions to verify: (a) adds page-faults and instructions; (b) adds context-switches/migrations; (c) tanks IPC on the hot cores and raises cache-misses with flat instruction count — the same instructions are executing, but each one now stalls waiting for the line transfer. Three different fingerprints for three different diseases, visible in counters before you look at code.
(e) [Linux] Profile it — and watch the flamegraph fail you. Counters said
what; the profiling chapter’s tools (ch09) say
where. The debug = true in your release profile has been waiting for this:
perf record -F 999 -g --call-graph dwarf -- ./target/release/pipeline-lab
perf report --stdio | head -40 # or build a flamegraph from perf script
perf record -F 999 -g -- ./target/release/pipeline-lab --share-line # compare
Predict first, then look. Prediction: both profiles are dominated by the spin
loops — now_cycles, spin_loop, the queue poll — and the baseline and
--share-line flamegraphs look nearly identical, even though (c) measurably
hurt. That is not a broken tool; it is ch09’s own interview question happening to
you. A sampling profiler answers “which instruction was the CPU on,” and under
false sharing the CPU is on the same instruction, stalled — the time moved into
memory stalls that on-CPU sampling attributes to the very same frames. The
counters from (d) saw it (IPC down, cache-misses up, instructions flat) and
perf c2c localized it to a cache line. Flamegraph for “which code,” counters
for “why it’s slow,” c2c for “which line of memory.” Having run all three on one
artifact is the difference between owning a profiler and owning a method.
(f) Close the loop: catch your own outlier (ch11). Everything so far aggregated. Aggregates locate a problem in the distribution and then go quiet about the individual event — but the flight recorder from ch11 exists precisely to answer “what happened during that one.” Extend the lab (~30 lines, the last exercise):
- Give the aggregator a ring of the last 4,096
Samples (fixed array, wrapping index — it already receives them). - On any sample whose
d_e2eexceeds a threshold — set it at your measured baseline p99.9 — freeze: stop overwriting, and dump the ring to stderr with per-stage deltas. - Run
--alloc, which you know produces episodic spikes, and read the dump.
The mechanism you’re building, as dataflow:
core 3 (consumer) │ (unpinned) aggregator
│
(1) pa.push(Sample) ─┼─► (2) ca.pop ─► (3) ring[n & 4095] = s
hot path never │ (last 4,096 samples, wrapping)
blocks or knows │ (4) s.d_e2e > threshold (baseline p99.9)?
the ring exists │ │ no → keep overwriting, ring rolls on
│ ▼ yes
│ (5) FREEZE — stop overwriting; (6) dump ring
│ to stderr: the outlier AND its ~4,095
│ neighbours, per-stage deltas for each
What to look for is the payoff: the frozen window shows the neighbours of the
bad event. Was one event slow in work alone (an allocator slow path), or were
twenty consecutive events slow in q2 (the consumer stalled and the queue
backed up)? A histogram can never distinguish those two; the ring does it at a
glance. That is ch11’s whole argument — histograms locate, the ring says what
happened — and you now have it as a thing you built, on a bug you injected,
caught by an instrument you wrote.
Expected results (indicative — a modern x86 server, isolated cores; your numbers will differ, your ratios shouldn’t differ much)
| Run | e2e p50 | e2e p99.9 | e2e max | Signature |
|---|---|---|---|---|
| baseline | ~0.4–1µs | ~2–5µs | ~10–30µs | flat, reproducible |
| (a) –alloc | +50–100ns | 2–10× worse | ~ms possible | episodic spikes, page-faults > 0 |
| (b) –no-pin | +0–50% | 5–50× worse, varies per run | ~ms | run-to-run variance, ctx-switches > 0 |
| (c) –share-line | +100–300ns | 2–5× worse | modest | throughput down, IPC down, HITM in c2c |
The meta-lesson sits in the columns: (a) and (b) are tail diseases, (c) is a throughput/median disease. If you only tracked p50, you’d ship (a) and (b); if you only tracked throughput, you’d ship (a) and (b) and catch only (c). This is the latency-methodology chapter’s argument (ch08), now demonstrated on your own hardware.
Interview narration of the findings
The lab’s product is that you can now narrate cause → measurement → fix in one breath. The template, using experiment (a):
“I decomposed the pipeline with TSC stamps at each hop — cycles carried in the message, converted and histogrammed off-path so instrumentation stayed under a percent of the budget. Baseline e2e was ~600ns p50, 3µs p99.9. Introducing one small allocation in the middle stage left p50 almost untouched but multiplied p99.9 and produced ms-scale maxima; perf stat showed the page faults, and the per-stage histograms put the growth entirely in the transformer’s work delta. That’s the general shape I look for: medians measure the design, tails measure the discipline.”
The same 15-second story for experiment (b):
“Same binary, one flag — I let the scheduler place the threads instead of pinning them. p50 barely moved; p99.9 blew out 5–50×, and differently on every run: results clustered into groups, because some runs landed two stages on hyperthread siblings and others migrated a thread mid-burst. perf stat showed the fingerprint — context switches went from zero to dozens. Nothing in the code changed, so the run-to-run variance itself was the finding, and the ftrace silence check closed it: the pinned run’s hot cores trace empty, the unpinned run’s trace is full of migrations. Placement is a variable; pinning removes it.”
And for experiment (c):
“I moved two ‘independent’ stage counters onto one cache line — transformer and consumer each bumping their own adjacent AtomicU64. Wall time for the run got worse and the work and q2 medians rose a few hundred nanoseconds, because every bump now dragged the line between cores 2 and 3. perf stat showed the fingerprint — IPC down, cache-misses up, instruction count flat: same instructions, each now stalling on a line transfer. perf c2c gave the smoking gun: HITM hits at two offsets on one line of the slots array. Padding the counters 128 bytes apart restored the baseline. That’s false sharing — a median-and-throughput disease, the opposite signature from allocation’s tail disease.”
And the warm-up that earns you the most credit, because most candidates only recite the term:
“I built the same measurement two ways over one injected 100ms stall. Closed-loop — wait for the system, then send — reported a p99.9 of 52µs and a max of 105ms: healthy percentiles, because while the system was frozen the harness wasn’t sending, so the thousand requests that should have arrived were never measured. Open-loop, timing from intended send time, reported p99 of 91ms. Same system, same stall, same sample count. That’s why the max detaching from the percentiles is a coordinated-omission fingerprint, not an outlier to throw away — and why every load generator I trust paces on a schedule.”
Plain-English recap
- The ladder matters as much as the capstone. You calibrated a clock and priced a fence; you made a benchmark lie and then caught it; you watched dead-code elimination report 0.000ns/op. Only then did the assembled pipeline arrive — and every piece of it was a decision you had already made by hand.
- You just built a miniature APM. Stamps carried inside the message are trace context propagation; the aggregator is the metrics backend; the printed percentile table is the dashboard. The difference from Datadog is only the scale: nanosecond spans, ~zero overhead, no vendor.
- Experiment (a) is the GC-pause lesson. One innocent allocation per request is the Node service whose p50 is fine but whose p99 is eaten by GC. Zero-alloc is a tail discipline, which is why code review can’t verify it but a counting allocator can.
- Experiment (b) is noisy neighbors. Unpinned threads are pods without CPU pinning, and the signature isn’t “slower” — it’s irreproducible, run-to-run variance you can’t explain. Placement, not code, was the variable.
- Experiment (c) is two services updating one row. The stage counters are logically independent but physically adjacent — hot-row contention at nanosecond scale, provable with perf c2c instead of pg_stat_activity.
- Step (d) is what dashboards are for: distinct diseases, distinct signatures. Allocation shows page faults; bad placement shows context switches; false sharing shows IPC down with instructions flat. Counters diagnose before anyone reads code — the same triage you already do from metrics.
- The open-loop producer is the honest load generator. It sends on schedule whether or not downstream is keeping up, so a stall shows up as latency instead of silently lowering the offered rate.
The 5 sentences you now get to say in interviews, truthfully
- “I’ve decomposed a pipeline’s latency budget stage-by-stage with TSC timestamps and per-stage HdrHistograms, aggregated off the hot path, and I know what my instrumentation itself costs.”
- “I’ve measured, on my own hardware, what a single hot-path heap allocation does to p99.9 versus p50 — and that’s why I treat zero-alloc as a tail-latency discipline and verify it with a counting allocator, not by code review.”
- “I’ve demonstrated false sharing between two ‘independent’ per-thread counters, watched it in the per-stage histograms, and confirmed the HITM signature with perf c2c before padding the layout.”
- “I generate load open-loop with intended-send-time accounting, because I’ve seen how a closed-loop harness coordinates with stalls and understates the tail by orders of magnitude.”
- “I can walk a tail regression from symptom to mechanism with counters first — perf stat, then sched tracing or c2c depending on the fingerprint — and I consider it diagnosed only when the outlier timestamps correlate with the mechanism.”
Interviewer will ask
Q: Why carry timestamps inside the message instead of logging at each stage? A: It makes correlation free — all four stamps for one event arrive together, no joining logs by sequence number across threads — and the hot-path cost is just the rdtsc plus stores into a message already in cache. The trade-off is message size; four u64s is cheap. For wide fan-out topologies you’d switch to per-stage rings keyed by seq and join off-path — a road deliberately not taken here: this lab’s pipeline is a straight line, so in-message stamps stay the cheaper design.
Q: Your q1 delta includes both queue residency and consumer wakeup. How would you split them? A: Add a stamp at enqueue-complete vs dequeue-start — t0 is intended send here, so I’d stamp t0’ after the push returns: t0’−t0 is producer-side delay and pacing, t1−t0’ is residency plus wake. To isolate pure wake latency, run at a rate low enough that the consumer always drains before the next message arrives — then the queue is empty at every enqueue, residency ≈ 0, and the whole t1−t0’ delta is wake.
Q: Is it load-bearing that the aggregator ring is SPSC? A: Yes — it has exactly one producer (the consumer stage) and one consumer (the aggregator); if multiple stages shipped samples I’d give each its own SPSC to the aggregator rather than share one MPSC (multi-producer single-consumer — many writers pushing into one ring), keeping one writer per ring — the rule all of the observability chapter (ch11) ran on — and letting the aggregator merge histograms, which HdrHistogram supports natively.
Q: The producer spins to pace. Isn’t that a core wasted? A: In this lab, yes, deliberately — open-loop pacing needs a reliable clock and spin-waiting is the jitter-free way. In production the “producer” is the NIC; the pacing question becomes replay fidelity: I replay captured data on original timestamps, including bursts, for exactly the reasons steady-state numbers lie.
Q: What would this lab miss about a real trading system? A: Plenty, and knowing it matters: no network (so no IRQ/softirq path, no NIC hardware stamps), trivially small working set (no icache/dcache pressure from a real book), uniform message sizes, no bursty arrival process unless I replay one, and no exchange at the other end. It teaches the measurement machinery; the production replay harness (the change-management chapter, ch17, built on ch13’s event log) is where verdicts come from.
Q: Why is max latency in your baseline tens of µs even when healthy? A: Because I ran 2M samples — at that count you will observe rare platform events: a stray IRQ on an imperfectly isolated core, a TLB shootdown (one core unmapping memory forces every other core to flush its translation cache — a cross-core interrupt), a C-state exit. The follow-up is exactly Part I’s tuning list, verified by the ftrace silence check; a healthy isolated setup pulls the max down toward single-digit µs, and I’d want a flight-recorder dump for whatever remains.
Further reading
- Gil Tene, “How NOT to Measure Latency” — re-watch it after running this lab; it lands differently once you’ve seen coordinated omission in your own harness.
- HdrHistogram documentation — interval histograms and merging, for turning this lab’s one-shot histograms into continuously-published percentiles.
- Brendan Gregg, Systems Performance (2nd ed.) — the CPU and scheduling chapters to explain experiment (b)’s fingerprints.
- The LMAX Disruptor paper — the architectural ancestor of the stamp-and-drain pattern used here.
- The
rtrbcrate documentation — a clean, real SPSC implementation worth reading end-to-end and comparing against your own queue in this same harness.
Where this goes next: Part II (measurement) is done — Chapter 13 opens Part III with the question the rest of the book hangs on: how does one design decision — the event log — make deployment, recovery, testing, and compliance all fall out for free?
Event Sourcing as a Deployment Primitive
Before you start — this chapter leans on a handful of primer ideas:
- WAL, snapshots, and replay in databases — Postgres does internally exactly what this chapter does architecturally; seeing it there first makes everything here familiar: ch00f
- Exchange feed architecture and sequence numbers — the ITCH-lineage “sequenced stream” pattern this chapter generalizes: ch00f
- Memory pages and copy-on-write — needed only for the fork-based snapshot pattern: ch00b
- Tick-to-trade and the trading-system vocabulary (fills, books, positions, venues): ch00f
Read those first — 20 minutes there saves an hour here.
You already built an event-sourced matching engine with snapshot/replay and a hot standby. Good — you know the shape. This chapter is about the parts that only show up in production and in interviews:
- The determinism contract as an auditable property, not a hope.
- Snapshotting without stalling the writer.
- Replay speed as an engineered number, not whatever you get.
- Replaying old logs through new code — the versioning trap.
- The sequencer (the component that assigns a global order — sequence numbers — to every input event; full section below) as the architectural center of the system.
By the end, “event sourcing” should stop meaning “a pattern I used” and start meaning “the mechanism that makes deployment, recovery, testing, and compliance all fall out of one design decision.”
Why the event log is a deployment primitive, not a persistence trick
Most engineers meet event sourcing as a persistence pattern: instead of storing state, store the deltas. In a trading engine it’s more than that. The log is:
- The database of record. The book, positions, open orders — all derived state. If it’s not in the log, it didn’t happen.
- The replication protocol. Your hot standby is just a second consumer of the same log. You already know this; hold onto it, because it generalizes.
- The deployment mechanism. A new binary that can replay the log to identical state can take over from the old binary. That’s the zero-downtime chapter (ch16).
- The test oracle. Replay yesterday through the candidate build, diff decisions. That’s the change-management chapter (ch17).
- The compliance artifact. Regulators asking “why did you send this order at 14:32:07.123456” get an exact answer, not a log-grep guess.
One design decision buys all five — if you hold the determinism contract. Lose determinism and you lose all five at once, usually silently.
The determinism contract
The contract: same log in, same state out, on any conforming binary, any machine, any time. Formally, your engine is a pure function fold(initial_state, events) -> state. If you know Array.reduce or a Redux reducer, you already know fold — same shape: (state, event) => newState, applied to every event in order. And the contract is the pure-reducer rule. Redux bans Date.now() and Math.random() inside reducers for the same reason: anything impure goes into the action (the event), so replaying the events always rebuilds the same state. Everything that violates purity must be pushed out of the fold and into the log itself.
The forbidden inputs
-
Wall clock.
SystemTime::now()inside the state machine is the classic sin. Time must arrive as an event field, stamped by the sequencer when the event was ordered. If your engine needs “current time” (for order expiry, session close), it consumes timer events that are themselves in the log. Replay then sees the exact same timestamps. -
Randomness. No
rand::thread_rng()in the fold. If you genuinely need randomness (randomized order queue priority on some venues, jittered internal IDs), the seed or the drawn value goes into the log as part of the input event. -
Iteration order of unordered collections.
HashMapiteration order in Rust is randomized per-process (SipHash with a random key). If you ever iterate aHashMapand the iteration order affects an output — say, cancelling all orders for a client and the sequence of cancel events matters — you’ve broken determinism. UseBTreeMap,IndexMap(insertion-ordered), or sort before iterating. This is the one that passes every unit test and fails in production three weeks later. -
Floats — with nuance. IEEE 754 arithmetic is actually deterministic for the same operations in the same order on the same settings. The dangers are: (a) compiler reassociation under
-ffast-math-style flags (Rust doesn’t do this by default — a point worth stating in an interview), (b) different builds rounding intermediate results differently — e.g. one build computesa*b+cas a fused multiply-add with one rounding while another rounds after the multiply and again after the add, so the last bit of the result differs (the historical version of this is x87 hardware keeping 80-bit intermediates), (c) accumulation order changing when you refactor. The professional answer: use fixed-point integers for prices and quantities (price in ticks, quantity in lots, i64 everywhere). Floats are for analytics, never for the state machine. -
External I/O and channel timing. Any
if socket.ready()branch, any “batch until the queue is empty” logic where batch boundaries affect state, any cross-thread race. Concretely: if a fee discount applies per batch and live processing happened to drain 10 events as two batches of 5, a replay that drains them as one batch of 10 computes a different fee — state diverges even though the events are identical. The fold must be single-threaded over a totally ordered input. -
Config read at runtime. If a config value affects decisions, either it’s constant for the life of the log segment, or config changes are themselves events in the log (
ConfigUpdated { key, value }). Otherwise replaying with today’s config against last month’s log produces different state.
Auditing for determinism
You will be asked “how do you know it’s deterministic?” Weak answer: “we’re careful.” Strong answers, layered:
- Structural audit: make the forbidden inputs unimportable — the same idea as an ESLint rule banning
Date.nowinside reducers, enforced at the dependency level. The state machine crate has#![no_std]-adjacent discipline — nostd::time, norand, no I/O in its dependency graph — enforced with clippy lints, acargo denyrule on the crate’s dependencies, and code review convention: theengine-corecrate takes events in, emits events out, nothing else. - Dual-run in CI: replay the same log twice in two separate processes (fresh state each time), compare a state hash. Cheap, and it catches HashMap-iteration bugs precisely because hash randomization differs per process.
- Cross-run in production: your primary and standby are already dual-running live. Continuously compare rolling state hashes (e.g., every 10k events, both sides publish
hash(state)keyed by sequence number). Divergence pages you before failover would have hurt. This is the tie-in to your hot-standby: you built the replication; the hash comparison is the cheap upgrade that turns it into a determinism monitor. - Nightly replay: replay today’s full log on a different machine class, compare final hash against the primary’s end-of-day hash.
A practical state hash: fold a structural hash over the book (per level: price, total qty, order count) plus positions plus sequence number. Don’t hash incidental fields (internal pointer-ish IDs, capacity of vectors).
#![allow(unused)]
fn main() {
fn state_hash(book: &Book, seq: u64) -> u64 {
use std::hash::{Hash, Hasher};
let mut h = twox_hash::XxHash64::with_seed(0); // fixed seed!
seq.hash(&mut h);
for (price, level) in book.bids.iter() { // BTreeMap: ordered
price.hash(&mut h);
level.total_qty.hash(&mut h);
for o in &level.orders { o.id.hash(&mut h); o.qty.hash(&mut h); }
}
// ... asks, positions ...
h.finish()
}
}Fixed seed matters: DefaultHasher with a random key gives you a hash that can’t be compared across processes.
Snapshot mechanics: don’t stall the writer
Naive snapshotting: pause the engine, serialize state, resume. At 1M events/sec with a book that serializes in 200ms, that’s a 200ms hole in your latency distribution — unacceptable. Three production patterns:
1. Secondary replayer snapshots (the usual answer)
The hot path never snapshots. A separate process (or your standby!) consumes the same log, maintains the same state, and snapshots its copy at leisure. Its pauses cost nothing. Snapshot = (state_blob, last_applied_seq). Recovery = load blob, replay log from seq+1.
This is the pattern to lead with in interviews because you’ve effectively already run it: your hot standby is a secondary replayer; giving it a “write a snapshot every N minutes” job is a small delta. It also means snapshots are implicitly determinism-checked — if the standby’s snapshot replays to a different hash than the primary’s live state, you’ve caught divergence.
2. Copy-on-write fork
fork() the engine process; the child inherits a copy-on-write (COW) view of memory frozen at the fork instant — parent and child share pages until one of them writes (ch00b) — and serializes it while the parent keeps trading. Redis does exactly this for RDB saves. Costs: fork itself is not free (page-table copy — hundreds of µs to ms for big heaps, and it stalls the parent for that duration), COW page faults add jitter to the parent as it writes, and memory can transiently double under heavy write load. Viable, used in practice, but the page-fault jitter is why latency-sensitive shops prefer pattern 1.
3. Immutable / persistent data structures
Book built on persistent structures (e.g., an immutable tree per side); snapshot = grab the current root pointer, serialize from it on another thread while the writer keeps producing new versions. The root pointer works like a git commit: grabbing it pins an immutable view of the whole tree at that instant, while new “commits” build on top without disturbing it. Bounded jitter, but you pay per-operation allocation/indirection cost on the hot path forever to make occasional snapshots cheap. Usually the wrong trade for a matching engine; worth naming to show you know the space.
Snapshot atomicity details that interviewers probe: write to snapshot.tmp, fsync, rename to snapshot.{seq} (rename is atomic on POSIX), fsync the directory — the rename lives in the directory’s own data, so skipping that fsync means a crash can forget the file was ever renamed. Keep the last K snapshots — a corrupt latest snapshot must not be fatal; fall back to the previous one and replay a longer tail. Checksum the blob (xxhash/crc32c) and validate on load. Record the exact seq inside the blob, not just the filename.
Log compaction and retention economics
The log grows without bound; state doesn’t. Retention policy is snapshots + tail:
- Operational tier: you need
latest_snapshot + tailto recover. Keep several snapshot generations and the log back to the oldest one you’d trust. On fast NVMe. - Regulatory/analytical tier: full raw log, compressed, shipped to object storage. Order/quote data compresses extremely well (delta-encoded integers, repetitive symbols) — 10–20x is normal. Multi-year retention is a compliance requirement in tradfi (order record-keeping obligations), and it’s cheap: even 100GB/day raw is single-digit GB/day compressed, dollars per month in S3-class storage.
- Compaction in the Kafka sense (keep last value per key) is mostly wrong for a trading log — you need the history. Compaction applies to derived/reference topics (latest config, latest position snapshot), not the order-flow log.
Do the arithmetic out loud in interviews: 500k msgs/sec × 64 bytes ≈ 32 MB/s ≈ 115 GB per 8h session raw; ~10 GB/day compressed; three years ≈ 11 TB — one cheap object-storage bucket. Retention is never the technical bottleneck; the bottleneck is replay time, next section.
Replay speed engineering
Recovery time = snapshot load + tail replay. Tail replay speed also gates your nightly regression replays (ch17). Target: replay an 8-hour day in minutes. That means replaying at 50–200x real time. How:
- No-I/O replay mode. In live mode, applying an event emits outbound messages (acks, fills, market data) to sockets. In replay mode, outputs are either discarded or written to a buffer for diffing — never sent. Gate this with a mode enum, not scattered
ifs: the fold returnsVec<OutboundEvent>(or writes into a caller-supplied sink), and the caller decides live-publish vs. discard vs. record. The state machine itself doesn’t know which mode it’s in — that’s what keeps replay honest. - Sequential, batched reads. The log is append-only, so replay is a pure sequential scan. Read in large chunks (4–64 MB), decode in-place with zero-copy framing. NVMe gives you multiple GB/s; decode is usually the bottleneck, not the disk.
- Cheap decode. Fixed-layout binary events (ch14) decode at memory bandwidth. If your replay is slow, the usual culprits are per-event allocation, serde-style dynamic deserialization, or logging. Replay profile should show ~all time in the fold itself.
- Bound the tail. With a snapshot every N minutes, tail replay is bounded regardless of day length. Snapshot cadence is a knob: snapshot interval × replay speed = worst-case tail time. If you replay at 5M events/sec and snapshot every 50M events, tail replay ≤ 10s.
- Parallelism — careful. The fold itself is inherently sequential (that’s the contract). You can parallelize decode/checksum ahead of the fold (pipeline: reader thread → decode threads → single apply thread), and you can replay independent symbols/shards in parallel only if they’re truly independent partitions with independent logs. Cross-symbol state (margin, or self-match prevention across books — blocking your own buy order from trading against your own sell) breaks that; know which side of the line your engine is on.
Numbers to have ready: a clean Rust fold applying simple book events does 5–20M events/sec/core. An 8h day of 500k/sec = ~14.4B events… which is why nobody replays whole days from genesis: 14.4B ÷ 10M/sec ≈ 24 minutes — fine for nightly regression, too slow for recovery. Hence snapshots: recovery replays minutes of tail, not hours.
Replaying old logs through new code: state-machine versioning
Event sourcing meets deployment here, and it’s where interviewers separate people who’ve read the blog posts from people who’ve operated the thing.
The question: binary v2 replays a log written (and originally applied) by binary v1. When is the result valid?
Valid when: v2’s fold is semantically identical on all event types that appear in the log. Adding new event types, adding fields with defaults (via upcasters — ch14), refactoring internals, performance work — all fine. The state hash after replaying the v1 log through v2 must equal v1’s hash. This is precisely your deploy gate: no behavior change intended → hashes must match, and you verify that in CI by replaying recorded prod logs.
Dangerous when the change is a semantic fix. Suppose v1 had a bug: it matched against a stale level in some edge case. v2 fixes it. Now replaying the v1 log through v2 produces a different book than production actually had — but production’s history really happened; real fills were sent to real counterparties. You cannot retroactively “fix” the past. Options, in order of preference:
- Log outputs, not just inputs. If fills/executions are themselves events in the log (the sequencer logs the engine’s decisions, not just requests), replay applies recorded fills verbatim and you sidestep the problem: old segments replay old decisions, new events get new logic. Many real engines log decisions for exactly this reason — it decouples “reconstruct state” from “re-run logic.”
- Effective-version epochs. Log a
LogicVersionChanged{v2}event at deploy time. The fold dispatches on the active logic version: events before the marker replay with v1 semantics, after with v2. You’re keeping the old code path alive — bitemporal in spirit (what we knew/did then vs. what we’d do now). Costs code retention; prune once segments age past retention. - Snapshot fence. Deploy v2 with a fresh snapshot taken at cutover; declare logs before the fence non-replayable through v2 (only through archived v1 binaries — keep them!). Simple, common, and what most shops actually do. The compliance archive still has the raw log plus the v1 binary artifact if a regulator asks.
Say the word “bitemporal” and then explain it plainly: two time axes — when it happened vs. what logic/knowledge applied — and your log must let you reconstruct along the first axis without contamination from the second.
Sequencer patterns: who owns the order of events
Determinism requires a total order of inputs. Something must impose it. That thing is the sequencer, and it is the real single point of design in every serious engine. The whole architecture in one picture:
orders ────────────┐
market data ───────┤ ┌──────────────────┐ sequenced log
timer events ──────┴───► │ SEQUENCER │──► ①②③④⑤...
│ stamps each input│ │
│ with 1, 2, 3, … │ ├──► matching engine
└──────────────────┘ ├──► risk
├──► hot standby
└──► drop-copy
Everything left of the sequencer is unordered chaos arriving on many wires; everything right of it consumes one identical, totally ordered stream. Once order is imposed exactly once, every consumer — engine, risk, standby, drop-copy — is just a deterministic fold over the same log.
- Single-sequencer architecture (the classic tradfi pattern). One process receives all inputs (orders, market data callbacks, timers), assigns monotonically increasing sequence numbers, writes the log, and multicasts/streams the sequenced log to every consumer — matching engine, risk, drop-copy (the duplicate feed of your own executions kept for reconciliation — ch00f), standby. Everything downstream is a deterministic function of the sequenced stream. This is the design behind exchange architectures in the NASDAQ/ITCH lineage and most prop-shop internal buses. (ITCH is NASDAQ’s sequenced multicast market-data feed — ch00f.) The payoff: replication, recovery, and fan-out are all “consume the log.” Its cost: the sequencer is a SPOF (single point of failure) and the latency floor (everything crosses it).
- Sequencer failover is then the hard sub-problem: a standby sequencer must take over without gapping or double-assigning sequence numbers. Options: shared reliable log the standby resumes from; or consensus.
- Raft / Aeron Cluster. (Raft: the standard consensus algorithm for getting several nodes to agree on one log; Aeron: a low-latency messaging and clustering library from the LMAX/Real Logic lineage — the same people as the Disruptor.) Aeron Cluster is the production-grade off-the-shelf version of “replicated deterministic state machine”: Raft consensus orders the input log across 3–5 nodes, each node runs your deterministic service (clustered service model), snapshots and log replay are built into the framework, and leadership transfer is the failover story. Used in real tradfi matching engines and post-trade systems. The trade: consensus adds a quorum round-trip to every input — a majority of the 3–5 nodes must acknowledge each event’s position in the log before it counts (~tens of µs on a good LAN with kernel bypass) — versus a naive single sequencer, in exchange for principled failover. Know both designs and the trade; interviewers love “single sequencer vs. Raft — when and why.”
- Your world: your primary/standby pair with log shipping is the single-sequencer pattern with a manually-managed standby. The interview upgrade is being able to say what you’d need for automatic failover: fencing (old primary must be unable to write after takeover — every append carries the leader’s epoch number, and the log and its consumers reject appends stamped with an old epoch, so a deposed primary’s writes simply bounce), gap-free handover (standby confirms it has the full log to seq N before claiming N+1), and split-brain prevention (leases/quorum, never “ping timed out so I’m leader”).
Recovery drills: RTO from snapshot + tail
An untested recovery path is a rumor. Components of RTO (recovery time objective):
- Detect failure (health checks, watchdog): target seconds.
- Load latest snapshot: size / disk bandwidth — a 5 GB snapshot on NVMe ≈ 2–3s plus deserialize.
- Replay tail: bounded by snapshot cadence (see arithmetic above — engineer this to seconds).
- Re-establish venue sessions and reconcile open orders (often the longest pole — the zero-downtime chapter (ch16) covers session takeover).
- Resume, initially in a safe mode (cancel-only or reduced limits) until reconciliation confirms state matches the venues’ view.
Drill it: monthly, kill the primary for real in production-like conditions (staging with recorded feed at minimum; the brave do game-days in prod with tiny limits). Measure each stage. The number you quote in interviews should sound measured, not aspirational: “our snapshot+tail recovery was ~X seconds; the venue re-logon dominated at Y; here’s what we did about Y.” You have hot-standby failover experience — mine it for one concrete story with numbers before interview day.
Plain-English recap
- The log is a double-entry ledger. You never edit a posted entry; you append. Balances (books, positions) are derived by summing the entries, and any “correction” is a new entry with a reason — the accounting discipline you already trust with money, applied to all state.
- The determinism contract is the pure-reducer rule. Impurity belongs in the
action, not the reducer — push clocks, randomness, and config into the logged
event and replay can never disagree with itself. The
HashMap-iteration-order trap is just the sneakiest impurity. - State-hash comparison is reconciliation with a statement date. Comparing primary and standby hashes at the same sequence number is exactly comparing your ledger balance to the PSP settlement report at a common cutoff — without the common cutoff, recon chases its own tail.
- Snapshot + tail replay is backup + WAL. Load the checkpoint, replay everything after it. Snapshot cadence × replay speed = worst-case recovery time, a knob you engineer, not a hope.
- The semantic-fix trap is “you can’t retroactively re-price settled payments.” A bug fix changes what the engine would have done, but real fills went to real counterparties — history happened. Logging decisions (not just inputs) is the ledger answer: old entries replay verbatim, new logic applies only going forward.
- The sequencer is your single Kafka partition / single Postgres primary. Total order has to come from somewhere; one process assigning sequence numbers is the cheapest way, and then replication is “everyone consumes the same partition.” The failover problems (fencing, split-brain) are the same ones Patroni — Postgres’s automated-failover agent — solves in the databases chapter (ch15).
Interviewer will ask
Q1: “How do you guarantee your replay is deterministic?”
Same picture as the Redux reducer: the fold must be pure, and determinism breaks wherever the same log could produce two different states. So you hunt the leak sources one by one — anything the fold reads that isn’t in the log. Wall clock: time enters as an event, so replay sees identical timestamps. Randomness: the seed or drawn value is itself logged. Float rounding: fixed-point integers, so no build can round differently. Map iteration order: ordered collections wherever iteration touches output. Then don’t promise it — audit it: the core crate can’t even import time or rand, CI replays the same log in two separate processes and diffs state hashes, and production compares primary/standby hashes continuously. The caveat: HashMap iteration passes every single-process test, because the random hash key only differs across processes — exactly why the CI dual-replay uses two processes.
Q2: “How do you snapshot a live engine without pausing it?” Name the problem first: pausing to serialize a big book is a 200ms hole in the latency distribution, so the hot path must never snapshot. The standby already consumes the same log and holds the same state, so it snapshots its copy at leisure — a state blob tagged with the last applied sequence number. Mention COW-fork (Redis RDB) as the alternative and why it loses: the fork itself stalls the parent, and copy-on-write page faults add jitter afterward. Then the durability details, each with its why: tmp file plus atomic rename, because a crash mid-write must never leave a half-snapshot under the real name; fsync the directory, because the rename lives in the directory’s own data; checksum the blob, because corruption must fail loudly at load, not silently at failover; keep K generations, because a corrupt latest must not be fatal — fall back one and replay a longer tail.
Q3: “Your standby’s state hash diverged from primary. What now?” First: the divergent side stops being a failover candidate immediately — a standby with wrong state is worse than none. Then bisect: replay the log from the last matching snapshot on both binaries offline, find the first event where hashes diverge, inspect. Usual suspects: unordered iteration, a float sneaking in, or config skew between the two hosts. It’s almost never “cosmic rays”; it’s almost always a determinism-contract violation that CI’s dual-replay didn’t cover.
Q4: “You fixed a matching bug. What happens to replay of old logs?” Show you see the trap: replaying old logs through fixed code produces state that never existed — real fills went to real counterparties under the old logic, and history can’t be re-run. Three ways out, each with its mechanism. Log decisions, not just inputs: replay then applies recorded fills verbatim, so old segments never re-run any logic at all — the best answer. Logic-version epochs: a marker event in the log, so the fold applies v1 semantics before it and v2 after — bitemporal, at the cost of keeping old code alive. Snapshot fence: fresh snapshot at cutover, older logs replayable only through the archived v1 binary — the pragmatic answer for most shops. Then say which you’d pick, and why, for the system at hand.
Q5: “How fast can you recover, and how do you know?” Run the clock on a concrete config instead of reciting a formula. Say snapshots every 30 seconds and a peak log rate of 100k events/sec. Second 0: the primary dies; the watchdog misses a few 1kHz heartbeats and declares it dead inside a second. Seconds 1–3: load the last snapshot — a few GB, mostly sequential read and deserialize. Seconds 3–5: replay the tail — worst case 30 seconds of log, ~3M events, and because replay is the pure fold with no network waits it runs at millions of events/sec, so ~1–2 seconds. State is now current: books, positions, sequence numbers. But nobody can trade yet — venue sessions still have to come back: re-logon, sequence negotiation, resend processing, 10–20 seconds per venue in parallel, and reconciliation against venue state is the gate before orders flow. So: state in ~5 seconds, trading in tens of seconds — and the tuning knob is snapshot cadence, because halving the interval halves the worst-case tail. How do I know? We drilled it by killing the process, and the drill is what exposed that session re-logon dominated, not replay — recovery numbers you haven’t measured by killing the process are fiction.
Q6: “Why a single sequencer? Isn’t that a SPOF?” Yes, deliberately: a total order must be imposed somewhere, and one process assigning sequence numbers is the lowest-latency way to do it; everything else becomes a deterministic consumer, which makes replication and recovery trivial. The SPOF is then handled by standby-with-fencing or by paying a quorum round-trip for Raft (Aeron Cluster) when you need automatic failover. The wrong answer is distributing ordering ad hoc — then nothing agrees on history.
Q7: “Kafka is an event log. Why not build the engine on Kafka?” Anchor in the sequencer picture: the engine needs one totally ordered stream, and something must impose that order. Kafka gives total order only within a single partition, so the engine would use exactly one partition — the parallelism Kafka exists to provide buys you nothing. And every event would cross the network to a broker and back before the fold sees it — a hop the microsecond budget can’t pay, versus an in-process append to a memory-mapped log. So: right shape, wrong tier. The caveat that shows judgment: Kafka still belongs in the architecture — downstream, shipping the sequenced log to analytics, risk, and archive, where its fan-out is exactly right.
Q8: “What’s in your snapshot, exactly — and what’s deliberately not?” Everything needed to resume the fold: books, orders, positions, session-level counters, active timers, active config, and the sequence number it corresponds to. Not in it: anything derivable that’s cheaper to rebuild than to store, and anything non-deterministic (socket state, wall-clock). Also: versioned snapshot format with its own compatibility story — a snapshot is just a big event, and it needs the same never-mutate versioning the log itself needs (next chapter).
Further reading
- Martin Kleppmann, Designing Data-Intensive Applications — ch. 5 (Replication), ch. 7 (Transactions), ch. 9 (Consistency and Consensus — the total order broadcast section is exactly the sequencer problem), ch. 11 (Stream Processing — event sourcing, log compaction, “the log is the database”).
- Martin Fowler, “Event Sourcing” and “Memory Image” articles on martinfowler.com — the Memory Image piece is the LMAX-flavored “keep it all in RAM, log the inputs” argument.
- Aeron Cluster documentation (aeron.io / real-logic GitHub) — clustered service model, snapshotting, log replay, leadership transfer. Read it even if you never use it; it’s the best public writeup of Raft-ordered deterministic services.
- Martin Thompson’s talks on the LMAX Disruptor and Aeron (various conference recordings) — mechanical sympathy meets the sequencer architecture.
- Jim Gray, “The Transaction Concept: Virtues and Limitations” — the ancient source of “log is truth, state is cache.”
- Redis documentation on persistence (RDB) — a candid engineering discussion of fork/COW snapshotting costs.
Where this goes next: the log outlives every binary that writes it — Chapter 14 answers how schemas and protocols evolve without ever breaking a reader: versioning, upcasters, and the N/N+1 compatibility rule.
Schema & Protocol Evolution
Before you start — this chapter leans on a handful of primer ideas:
- The event log and the replay contract — why old bytes must stay decodable forever: chapter 13
- FIX and venue protocols — the session/message world your schemas talk to: ch00f
- Feed handlers and normalization — the per-venue adapter layer this chapter uses as its isolation boundary: ch00f
- Struct layout and byte offsets — why “fixed-offset” formats are fast and why inserting a field mid-struct corrupts everything: ch00a
Read those first — 20 minutes there saves an hour here.
Your event log lives for years. Your binaries live for weeks. Your venues change their protocols whenever they feel like it. The gap between those lifetimes is schema evolution, and it’s the single most-probed “can this person run a system, not just build one” topic in infrastructure interviews. The discipline is small and rigid: never mutate a published schema; only add; version everything; translate at the boundary.
The prime directive: never mutate v1
Once an event with schema v1 has been written to a log that outlives the current binary, v1 is frozen forever. Not deprecated-then-changed. Frozen. A byte layout, field meaning, and unit convention that some reader will need to understand in three years.
What “mutate” covers, because people rationalize all of these:
- Changing a field’s type (
u32qty →u64qty) in place. - Changing a field’s meaning or units (price in cents → price in ticks) without changing anything structural — the worst kind, because nothing crashes, numbers are just silently wrong.
- Renaming with reuse — deleting
fooand adding a differentfoo. - Reusing a numeric field id/tag of a deleted field. Protobuf identifies fields on the wire by number, not name (mechanics in the wire-formats section below), so reusing a dead number makes years-old bytes silently decode as the new field. This is protobuf’s cardinal sin;
reservedexists for it. - Changing enum variant discriminants (the discriminant is the integer the variant is stored as — 0=buy, 1=sell) or reordering variants in a format where the discriminant is the wire value.
What’s allowed, format permitting: adding new optional fields, adding new event types, adding enum variants (if readers tolerate unknowns), widening semantics in a way old readers can safely ignore.
Everything else is a new version: OrderPlacedV2 alongside OrderPlacedV1, both decodable forever, with an upcaster bridging them.
Upcasters: translate at read time, once, at the boundary
An upcaster is a pure function v_old -> v_new applied when reading old events, so that everything past the deserialization boundary sees only the newest version. The state machine (the event-sourcing fold — ch13) handles exactly one version: current. All version sprawl is quarantined in the codec layer.
Rules:
- Upcasters are pure and total. For every valid v1 event, a defined v2 result. New fields get explicit, documented defaults — and the default must reproduce old behavior (“
source: Unknownbehaves exactly as v1 did”), otherwise you’ve broken the replay contract (ch13). - Chain them. v1→v2→v3, not v1→v3 direct. N versions cost N-1 upcasters, not N². Each is written once, when the new version ships, while the semantics are fresh.
- Upcast on read, never rewrite the log. The log is immutable and often a compliance artifact; rewriting it destroys the audit trail and risks corruption. (A deliberate offline “re-encode the archive to vCurrent” migration is a separate, rare operation — and you keep the original.)
- Test with golden files. Committed binary fixtures of real v1/v2 bytes; CI decodes them with today’s code and asserts equality with expected structs. This catches “someone touched the old decoder” — the mutation you swore wouldn’t happen.
Rust: enum-versioned events + upcasting
#![allow(unused)]
fn main() {
// codec layer — the ONLY place old versions exist
#[derive(Clone, Debug, PartialEq)]
pub struct OrderPlacedV1 { pub id: u64, pub side: Side, pub price: i64, pub qty: u64 }
#[derive(Clone, Debug, PartialEq)]
pub struct OrderPlacedV2 {
pub id: u64, pub side: Side, pub price: i64, pub qty: u64,
pub source: Source, // NEW in v2
}
pub enum WireEvent { // what the log actually contains
OrderPlacedV1(OrderPlacedV1),
OrderPlacedV2(OrderPlacedV2),
// every version ever written, forever
}
impl From<OrderPlacedV1> for OrderPlacedV2 {
fn from(v1: OrderPlacedV1) -> Self {
OrderPlacedV2 {
id: v1.id, side: v1.side, price: v1.price, qty: v1.qty,
source: Source::Unknown, // default MUST reproduce v1 behavior
}
}
}
/// Boundary function: the engine only ever sees `Event` (== current versions).
pub fn upcast(w: WireEvent) -> Event {
match w {
WireEvent::OrderPlacedV1(v1) => Event::OrderPlaced(v1.into()),
WireEvent::OrderPlacedV2(v2) => Event::OrderPlaced(v2),
}
}
}On disk, each record carries a header the decoder dispatches on:
#![allow(unused)]
fn main() {
#[repr(C)]
pub struct RecordHeader {
pub len: u32, // payload length
pub event_type: u16, // OrderPlaced = 1, OrderCancelled = 2, ...
pub version: u16, // schema version of THIS record
pub seq: u64,
pub ts_ns: u64, // sequencer-assigned time (event-sourcing chapter!)
pub crc32c: u32,
}
}(event_type, version) pairs are append-only registry entries. Keep the registry in one file with a comment per entry: date shipped, what changed, default rules. That file is your schema history — interviewers respond well to “we kept a single append-only registry, code-reviewed like an API.”
Wire formats and their evolution rules
You’ll be asked to compare these. Every format picks a point on the triangle of decode cost, evolution flexibility, and self-description. Know each format’s specific evolution mechanics — the exact add/remove/reorder rules — not a general impression of which is faster.
SBE (Simple Binary Encoding)
The binary standard from the FIX Trading Community (the standards body behind the FIX protocol — ch00f); the tradfi latency-tier default. Fixed-offset fields — decode is pointer-cast plus field reads, zero allocation, effectively memory-bandwidth speed.
Evolution model: extension, not flexibility. Messages declare a schema version; new fields are appended after existing fixed fields — never inserted, because every field sits at a fixed byte offset and inserting shifts all the later ones:
v1 layout: [id @0][price @8][qty @16]
v2 by APPEND: [id @0][price @8][qty @16][source @24] old offsets still valid
v2 by INSERT: [id @0][source @8][price @16][qty @24] every later offset shifted
^^^^^^^^^^ old readers read source as price, price as qty — garbage
Old readers, told the message is a newer version, simply don’t read past what they know (“extension” semantics); new readers of old messages must apply defaults for missing trailing fields. Variable-length data goes at the end.
What you can’t do: remove or retype fields in place, insert mid-message. The pragmatic pattern is pre-allocated reserved fields — pad the layout at design time, claim padding later without changing size or offsets:
#![allow(unused)]
fn main() {
#[repr(C, packed)]
pub struct OrderPlacedWire {
pub id: u64,
pub price: i64, // ticks (units documented AT the field)
pub qty: u64, // lots
pub side: u8, // 0=buy 1=sell
pub source: u8, // v2: was _reserved[0]; 0 = Unknown = v1 behavior
pub _reserved: [u8; 6],// claim bytes here in future versions; zeroed on write
}
// Size and every offset are frozen. Zero means "not set / old default" by
// convention, so a v1 writer's zeroed padding IS a valid v2 message.
}That “zero = legacy default” convention is what makes reserved-field evolution safe: v1 writers produce valid v2 messages for free.
Protobuf
The general-purpose default. Tag-length-value: every field carries its numeric id; decoders skip ids they don’t know (unknown-field pass-through). Proto3 also retains unknown fields on re-serialize, which matters for proxies: a middlebox that decodes a message and forwards it re-serialized no longer silently strips the fields it didn’t understand.
Evolution rules: never reuse or renumber a field id (reserved 5; after deletion); adding fields is free; several type changes are wire-compatible (int32↔int64 with truncation caveats), most aren’t; required was removed from the language because it made evolution brittle — everything is optional, and application-level defaults do the work.
The cost is decode speed. Integers are varints — a variable-length encoding where small numbers use fewer bytes — so parsing is field-by-field with allocations, not a pointer cast. Fine for control plane, config, and cold path; too slow and too allocation-happy for a market-data hot path.
FlatBuffers / Cap’n Proto
Zero-copy access like SBE, with a vtable/pointer layer that buys protobuf-like evolution — a vtable here is a small per-message lookup table mapping field → byte offset, consulted on every read. Fields located via these offset tables can be added and unknown ones skipped without fixed-position rigidity. The tax: indirection on every access (vtable lookup vs. SBE’s compile-time offset), bigger messages, and alignment discipline. A defensible middle choice for internal buses; in practice tradfi picked SBE (with FIX heritage) and most crypto shops picked JSON-because-the-venue-did plus an internal binary format.
JSON
Your venues’ reality: crypto exchange WebSocket feeds are JSON. Evolution is trivially flexible (add keys; readers ignore unknowns) and totally undisciplined (nothing stops a venue renaming a key, changing a number to a string — Binance famously sends quantities as strings — or changing units; you find out in production). Parse cost is brutal: hundreds of ns to µs per message, allocation-heavy. The professional stance: JSON is a boundary format you normalize out of immediately (see feed handlers below); simd-json-style parsers and arena allocation if the boundary itself is hot.
Comparison one-liner for interviews: “SBE when both ends are mine and latency is the product; protobuf when evolution across many teams matters more than nanoseconds; FlatBuffers when I want both and accept indirection; JSON when the counterparty chose it for me — and then I normalize it away at the edge.”
Rolling upgrades: the N/N+1 compatibility guarantee
You have many services on the internal bus (your internal message stream between components) — feed handlers, engine, risk, gateways, drop-copy — and you deploy them one at a time (ch16). During any deploy window, versions N and N+1 coexist on the same streams. The rule that makes this safe:
Every message schema change must be compatible in both directions across one version step. N+1 readers accept N’s messages (upcast/defaults); N readers accept N+1’s messages (unknown-field skip / extension semantics). You never need N and N+2 live simultaneously because deploys are serialized — but note that the log makes the guarantee stronger than it looks: the current reader must handle every version ever written to retained logs, not just N. Live traffic needs N/N+1; replay needs the current reader to decode arbitrarily old versions — N back to the start of the archive — via the upcaster chain.
The rollout choreography for a breaking-ish change (new required-by-logic field):
- Ship readers first. Deploy code that understands v2 everywhere, still writing v1.
- Flip writers. Once all readers speak v2, writers start emitting v2 (config/startup flag, not code deploy, ideally).
- Deprecation window. v1 write-capability is removed after readers have been v2-aware for a defined period (and logs containing v1 remain decodable forever regardless).
Reader-before-writer ordering is what keeps this safe; almost every “mysterious deserialization error during deploy” post-mortem is that order violated. For deprecation windows, be concrete: internal services, one or two release cycles; anything persisted, effectively never (decoder lives as long as the archive).
Venue protocol upgrades: the normalize layer as isolation boundary
This is your daily reality with 20+ venue integrations, so own it in interviews. A venue announces: “on March 15 we migrate to WS API v5; new auth flow; qty field renamed; new message for liquidations; old API sunset in 90 days.” You do not get a vote.
The architecture that makes this survivable: per-venue feed handlers that normalize to your internal schema at the edge. The venue’s protocol exists only inside its handler. Everything downstream — book builders, strategies, the engine — consumes your internal events (this chapter’s and ch13’s discipline applies to those, and you control their evolution). A venue migration is then a change to exactly one process, deployable venue-by-venue (rolling-by-venue — ch16), testable in isolation.
Operational playbook for a venue migration:
- Capture first. Record raw v5 traffic (they usually run new API in parallel before sunset). Raw capture — bytes with timestamps, before parsing — is your regression fixture and your dispute evidence.
- Build the v5 handler as a new module/binary, not edits to v4 in place — you’ll run both during transition.
- Shadow it: v5 handler consumes live, normalized output diffed against the v4 handler’s output in real time. Diffs reveal the semantic changes the changelog didn’t mention (different snapshot depth, different trade-side convention, timestamps now in µs not ms — units again).
- Cut over per venue with instant fallback to v4 while it still exists.
- Keep the v4 decoder as long as you retain v4-era raw captures.
The normalized schema itself needs the same evolution discipline: when a venue exposes something new you actually want (a liquidation flag), that’s an additive internal-schema change rolled out readers-first — venue chaos on the outside, boring N/N+1 on the inside. That sentence is the interview answer.
Config schema evolution: same discipline, smaller egos
Config is code-adjacent input to a deterministic system, so it gets the same treatment, and interviewers increasingly probe it because config changes cause more incidents than code changes.
- Version the config schema. A
schema_versionfield in the file; the loader upcasts old configs exactly like old events (defaults must preserve behavior). - Config as code: files in git, reviewed, CI-validated (parse + semantic lint: “limit must be > 0”, “every enabled venue has credentials”), deployed as artifacts with a rollback path — never hand-edited on hosts.
- Renames are additive: accept both keys for a window, warn on old, remove later. A binary that crashes on an old config file during rollback has broken N-1 compatibility exactly as badly as a message schema would (the rollback discipline of ch16 depends on this).
- If config affects the deterministic fold, config changes are events in the log (ch13). Startup-loaded config that never changes mid-session is exempt; anything hot-reloaded is not.
Plain-English recap
- “Never mutate v1” is Stripe API versioning. A published API version is frozen forever; changes ship as new versions, and old clients keep working indefinitely. Your event log makes every past writer an “old client” you can never break — same contract, enforced by your own archives.
- Upcasters are Stripe’s version-transform layers. Old-shaped requests pass through a chain of pure transforms (v1→v2→v3) so core code only ever sees the latest shape. Version sprawl is quarantined at the boundary — exactly like keeping all the legacy-request shims in the API gateway, never in the domain logic.
- Golden files are snapshot tests for bytes. Committed fixtures of real old payloads, decoded in CI and compared to expected structs — the same reflex as Jest snapshots, protecting decoders someone will otherwise “clean up.”
- Readers-first is the expand–contract deploy you already do with Postgres. Add the nullable column and deploy code that tolerates it before anything writes it; drop the old column only after nothing reads it. Flipping writers before readers is the same outage in both worlds.
- N/N+1 on the bus is rolling deploys with queued webhooks. During a deploy, old and new pods coexist and messages written by either must be readable by both. And the retained log stretches the rule: today’s reader must decode every version ever archived, like a webhook consumer that might receive a replay of events from 2019.
- A venue migration is a PSP API migration. When Stripe or Adyen announce a breaking change, only your per-PSP adapter changes; the rest of the system sees your internal, stable schema. Shadow-diffing the new handler against the old is running both integrations in parallel and reconciling their outputs before cutover — the anti-corruption layer (the adapter layer that keeps an external system’s weirdness out of your core) earning its keep.
Interviewer will ask
Q1: “How do you evolve an event schema when the log is retained for years?”
Open from the Stripe picture: a published API version is frozen forever, and the log makes every past writer an old client you can never break. The log outlives every binary, so bytes written today must still decode in three years. Therefore v1 freezes the moment it’s written, and every change is a new type — OrderPlacedV2 beside OrderPlacedV1, both decodable forever. Upcasters quarantine the sprawl at the read boundary: pure v1→v2→v3 translations with behavior-preserving defaults, so the engine only ever sees the current version. Golden files keep the freeze honest: committed real v1 bytes decoded in CI, so “someone cleaned up the old decoder” fails a test instead of corrupting replay. Land: never mutate, only add, translate at the boundary — and the log is never rewritten. The registry detail: (event_type, version) pairs live in one append-only, code-reviewed file.
Q2: “Protobuf vs. SBE — when and why?”
One fork decides everything: how does a reader find a field — compile-time offset, or per-field tag on the wire? SBE picks offsets: decode is a pointer cast plus field reads, memory-bandwidth fast. But frozen offsets mean evolution is append-only — new fields at the end, or claim pre-reserved padding — because inserting mid-message shifts every later offset into garbage. Protobuf picks tags: every field carries its id, so readers skip unknown fields and cross-team evolution is easy — but you pay varint decode and field-by-field parsing, too slow and allocation-happy for the hot path. Land: SBE when both ends are mine and latency is the product; protobuf for control plane and cold path. Volunteer each choice’s signature failure: reusing a dead protobuf field id, so years-old bytes silently decode as the new field (reserved exists for this); and inserting a field mid-SBE-message.
Q3: “You’re deploying a change that adds a field consumers need. Walk me through the rollout.” Start from the asymmetry that dictates the order: a new reader given old bytes is safe — the missing field takes a default; an old reader given new bytes is not — it can’t use a field it never learned. So readers ship first: deploy v2-aware readers everywhere while everything still writes v1. Only when reader coverage is total do writers flip, via config rather than another deploy. Then a deprecation window before v1 write capability is removed — while the v1 read path lives as long as any log contains v1. Name the guarantee — N/N+1 both directions on live traffic, N back to forever on replay — and the classic failure: flipping writers before readers, the root of almost every “mysterious deserialization error during deploy” post-mortem.
Q4: “A venue announces a breaking API change with 90 days notice. What do you do?” Same shape as a PSP API migration: when Stripe announces a breaking change, only your Stripe adapter changes — here, the venue’s protocol lives only in that venue’s feed handler; build the new handler alongside the old, capture raw traffic, shadow-diff normalized output against the current handler to catch undocumented semantic changes (units, side conventions, snapshot behavior), cut over per-venue with fallback. Downstream sees zero change unless we choose an additive internal-schema update. This is the question where your 20-venue experience should carry the answer — have one real migration story ready.
Q5: “What breaks if you just add a field to a #[repr(C)] struct you write to the log?”
Every prior record now decodes at wrong offsets — silent corruption, not a crash, because the bytes still “parse.” Fixed-layout formats require either explicit versioning in the record header with per-version decoders, or pre-allocated reserved space with zero-as-default so old records remain valid new-version records. This is the trap question checking you’ve actually done binary logging.
Q6: “How do old readers handle fields they don’t understand — compare formats.” This question reduces to one requirement: skipping a field you don’t understand means knowing where it ends. Protobuf builds that in — tag-length-value framing tells the reader how many bytes to skip — and proto3 even retains skipped fields on re-serialize, so proxies stop stripping them. SBE has no per-field framing, so old readers can’t skip; instead the extension model — read only your known prefix, safe because new fields only ever append — which needs the version in the header. FlatBuffers: the vtable lookup misses, and the reader gets a default. JSON: keys are self-describing, so unknown keys are simply ignored. Land the design lesson: unknown-field tolerance is what makes reader/writer skew survivable, so a hand-rolled format must design it in — a version field plus length-prefixed records, letting readers skip whole records they can’t parse.
Q7: “Does config get the same treatment as code?” Yes, and say why with force: config changes cause more trading incidents than code changes because they skip the pipeline. Config as code in git, schema-versioned, CI-validated, deployed and rolled back like binaries, N-1 compatible (old binary must load new-ish config during rollback), and hot-reloaded config that affects the deterministic path enters as log events.
Q8: “How would you version snapshots?” Trap check — people version events and forget snapshots. A snapshot is one giant event, so in principle it needs the same versioned header and upcasters. The pragmatic shortcut is two steps: the upgraded binary reads the previous snapshot once — one version step, so N-1 compatibility suffices — then immediately writes a fresh snapshot in the new format. After that, the only cross-version read that can ever happen is a rollback inside the bake window. And the escape hatch if that read fails: event-level replay from an older snapshot — the log is the safety net under the shortcut.
Further reading
- Martin Kleppmann, DDIA, ch. 4 (“Encoding and Evolution”) — the canonical treatment: Avro/protobuf/Thrift evolution rules, forward/backward compatibility, rolling upgrades. If you read one thing for this chapter, this is it.
- Simple Binary Encoding specification and the SBE GitHub wiki (FIX Trading Community / real-logic) — especially the “Message Versioning / Schema Extension” sections.
- Protocol Buffers language guide — “Updating a Message Type” section; the
reservedkeyword rationale; proto3 unknown-field semantics. - Greg Young, Versioning in an Event Sourced System (free ebook) — upcasters, weak schema, “never rewrite the log,” from the CQRS/ES lineage.
- FlatBuffers documentation — “Writing a schema / Evolution” section, for the vtable trade-off.
- Martin Fowler, “Evolutionary Database Design” (with Pramod Sadalage) — the mindset bridge into the migration material of the databases chapter (ch15).
Where this goes next: with the log as system of record and schemas that evolve safely, Chapter 15 asks where actual databases fit — the hot/warm/cold tiering, tick stores, and the Postgres operational depth (replication, failover, online migration) that platform interviews test hardest.
Databases in Trading Systems
Before you start — this chapter leans on a handful of primer ideas:
- WAL and MVCC — how Postgres actually writes and why updated rows leave bloat behind: ch00f
- Streaming vs logical replication — the two ways a replica can follow a primary: ch00f
- Lock queues and the ALTER TABLE trap — how a “fast” migration freezes production without doing any work: ch00f
- pgbouncer pooling modes — session vs transaction vs statement, and what transaction pooling breaks: ch00f
- Tick data and kdb+ — what a tick store is and the query shape it exists for: ch00f
Read those first — 20 minutes there saves an hour here.
This is your declared weak spot, so this chapter goes deepest. The good news: trading systems use databases in a strongly opinionated, tiered way, and once you can articulate the tiering, every interview question about “how do you store X” has a slot to fall into. The second half is Postgres operational depth — replication, failover, pooling, and above all online schema migration — because that’s what “we need someone who can also touch the platform” interviews actually test. You run pgbouncer and a multi-tenant Postgres 17 on your own GCP host, and Redis as an ephemeral tier at work; use both as story hooks — it’s rare and it lands.
The tiering: hot, warm, cold
Rule zero: the hot path never touches a database. No exceptions, and interviewers will probe until you say it. A matching or strategy decision path budgeted in microseconds cannot tolerate a round trip to anything with a query planner, a lock manager, or a network hop to a storage tier. Sub-rule: it doesn’t touch a remote cache either — Redis at ~100µs+ RTT is just as disqualified as Postgres.
So where does state live?
Hot tier: memory + the event log
The engine’s working state — books, orders, positions, risk counters — is in-process memory, laid out for the access pattern (Part II material). Durability comes from the event log (ch13), not from a DB: append to a memory-mapped or O_DIRECT (write straight to disk, skipping the kernel’s page cache) sequential log, replicate to the standby, and that is the database of record. The phrase to use: “the log is the system of record; every database downstream is a derived, eventually-consistent view.” This inverts the enterprise mental model where the DB is truth and logs are exhaust — say the inversion explicitly, it’s the key idea of the whole Part.
Downstream consumers tail the log and project it into whatever store suits their query pattern. Which brings us to:
Warm tier: tick stores / time-series
The quant and ops query load: “give me every trade and top-of-book for BTC-perp across venues between 09:30 and 10:00,” “compute realized spread per venue per hour for the last quarter.” Billions of rows, append-mostly, time-ordered, scanned in ranges, aggregated by column. That shape is why columnar time-series stores own this tier:
- kdb+ (ch00f for the gentle intro) — the tradfi incumbent, and you should be able to say why it won rather than just name-drop it. (1) Columnar on-disk layout: a date-partitioned table is a directory per date, a file per column; “average spread over 3 months for one symbol” reads only the columns touched, at sequential-scan speed. (2) The same language, q, runs against in-memory real-time tables and on-disk historical ones — the canonical deployment is a ticker plant: a real-time database (RDB) holding today in RAM, appended from the feed, written down at end-of-day to the historical database (HDB), with q queries spanning both. (3) It’s a full programming environment, so the analytics run inside the store instead of hauling billions of rows out. (4) Decades of trust and installed base in banks. Costs: eye-watering per-core licensing, a famously terse language, key-person risk. Being conversant — columns, splayed/partitioned tables (splayed = one file per column on disk), RDB/HDB, why
xasc(sort ascending) andaj(as-of join) matter — signals tradfi literacy even if you’ve never run it. As-of joins deserve one sentence in any answer here: “join each trade to the most recent quote at or before its timestamp” is the canonical tick-store query, and native as-of support is half the reason these engines exist. - ClickHouse — the open-source columnar workhorse; crypto-native shops overwhelmingly land here. MergeTree tables ordered by
(symbol, ts), aggressive compression (delta + zstd on timestamps and prices compresses hard), materialized views for rollups,ASOF JOINbuilt in. Operationally heavier than it looks — in plain terms: constant background compaction (merges), whole-part rewrites for updates/deletes (mutations), and its own coordination service (Keeper) to run replication — but the query performance per dollar is the draw. - QuestDB / TimescaleDB — QuestDB: purpose-built TSDB, SQL with ASOF joins, strong ingest, simpler ops story than ClickHouse at smaller scale. Timescale: Postgres extension — you keep the Postgres operational model (next section applies verbatim) and get hypertables (one virtual table auto-partitioned into time chunks) plus compression; the right choice when your tick volumes are modest and you value one database technology to operate.
- Arctic / ArcticDB — Man Group’s open-source approach: versioned dataframe storage over object storage or LMDB (an embedded key-value store), Python-native. Less “database,” more “columnar dataframe store for research”; the research-cluster complement rather than the production tick store.
- The crypto-world equivalent: raw capture files (compressed JSONL or your normalized binary log) in object storage as the immutable record, loaded/projected into ClickHouse-or-similar for querying. Many crypto desks run exactly that and it’s a perfectly respectable answer — the log-plus-projection pattern again.
Cold tier: Postgres for reference data, accounts, compliance
Everything low-rate, high-value, relational, and audit-sensitive: instrument reference data (symbols, tick sizes, multipliers, venue mappings), accounts and permissions, credentials metadata, fee schedules, end-of-day positions and P&L snapshots, reconciliation results, compliance/audit records, config history. Tens to thousands of writes per second at most, but correctness and queryability matter, transactions matter, and this data feeds humans and regulators. Postgres is the default and nobody gets fired for it. This tier is also where your operational experience lives: you run a shared Postgres 17 behind pgbouncer for your own multi-tenant platform — say so.
Redis sits beside the tiers, not in them: ephemeral coordination state — sessions, queues, cursors, distributed-ish locks you don’t bet money on. The discipline is that nothing in Redis is the record of anything; lose it and you re-derive. You run this exact split at work; one sentence about “Redis is allowed to lose data by policy” shows tier thinking.
Postgres operational depth
Now the part you’re weakest on and interviews for platform-adjacent trading roles genuinely test. Four topics: WAL and replication, failover, pooling, and online migration.
WAL mechanics in one page
Every change in Postgres is written twice: to the write-ahead log (WAL) (ch00f walks it slowly) first, then to the actual table/index pages (“heap”) in shared buffers, which reach disk lazily at checkpoints. Commit = WAL flushed to disk (fsync), nothing more; crash recovery = replay WAL from the last checkpoint. If that sounds familiar, it should: Postgres is internally an event-sourced system — WAL is the event log, the heap is the snapshot, recovery is snapshot+tail replay. Making that connection out loud in an interview (“Postgres does internally what my engine does architecturally”) is a genuinely strong move because it’s true and it shows transfer.
Three consequences worth knowing, one at a time.
The durability knob. synchronous_commit decides whether COMMIT waits for the WAL fsync (durable, slower) or returns as soon as the record is in the WAL buffer (fast, but a crash can lose the last few hundred milliseconds of “committed” transactions) — a per-transaction trade of durability for latency. Concrete scenario: an internal metrics table can run synchronous_commit = off and take the risk for the throughput; the fills table a regulator will ask about cannot.
Checkpoint storms. A checkpoint is the moment Postgres flushes all its dirty table/index pages to disk so old WAL can be recycled. Mistune it and you get I/O storms: let max_wal_size grow too large and each checkpoint arrives with an enormous backlog to flush at once, spiking every query’s latency when it hits. checkpoint_completion_target is the smoothing knob — it spreads the flushing across the checkpoint interval instead of one burst, turning a periodic I/O cliff into a steady hum.
Torn pages. A Postgres page is 8 kB but a disk only guarantees atomic writes of a smaller unit, so a crash mid-write can leave a page half-old, half-new — “torn,” and unrepairable from ordinary WAL records alone. full_page_writes is the defense: the first change to each page after a checkpoint writes the entire page into the WAL, giving recovery a known-good copy to restore before replaying changes — hence the write-amplification spike right after every checkpoint.
Streaming vs. logical replication
Two different machines that ship the same WAL:
- Streaming (physical) replication ships WAL bytes; the replica is a block-for-block copy, replaying continuously. Same major version, same architecture, whole cluster or nothing. Replicas serve read-only queries, with visibility lag and one built-in conflict: WAL replay sometimes needs to remove a row version (the primary’s vacuum already cleaned it up) that a replica query is still reading, and the replica must then either cancel that query or pause replay and fall further behind.
hot_standby_feedbackis the escape hatch — the replica tells the primary “don’t vacuum row versions my queries can still see,” so queries stop dying, but the dead rows now pile up on the primary (bloat) for as long as replica queries run. That’s the trade: query cancellation on the replica vs. bloat on the primary. Sync vs. async:synchronous_standby_namesmakes commits wait for replica flush — durability across host loss, at latency cost. This is your HA mechanism. - Logical replication decodes WAL back into row-change events and publishes them per-table:
CREATE PUBLICATION/CREATE SUBSCRIPTION. The plumbing is a replication slot — the primary’s durable promise to retain WAL until this subscriber confirms it received it (think: a webhook queue that never truncates until the consumer acks) — plus an output plugin (e.g.pgoutput) that does the decoding. Replica is a live, writable database applying changes — so it can be a different major version (this is how near-zero-downtime major upgrades are done), a subset of tables, or a differently-indexed copy. Limitations to name: DDL is not replicated (schema changes must be applied on both sides — coordinate with the migration discipline below), sequences don’t replicate, and an abandoned replication slot pins WAL on the primary until the disk fills — the classic 3am logical-replication incident. - CDC: the same logical decoding mechanism feeds Debezium-style change-data-capture into Kafka/ClickHouse — “DB as event producer,” the mirror image of your engine’s log-projection pattern.
Rule of thumb to say: streaming for HA/failover, logical for upgrades, migrations, selective copies, and CDC.
Failover: the Patroni pattern
Manual failover of Postgres is a pager-driven ritual; Patroni (an open-source agent that automates Postgres leader election and failover) is the standard automation. Shape: each Postgres node runs a Patroni agent; agents coordinate through a consensus store (etcd/Consul/ZooKeeper) holding a leader lease; the leader holds/renews the lease, replicas watch. Leader dies or loses the lease → healthiest sufficiently-caught-up replica (by WAL position) wins an election, promotes, others re-point. Client routing via HAProxy/vip-manager or Patroni’s REST health endpoints (/primary, /replica).
┌───────────────────────────────┐
│ etcd: lease "leader = pg1" │◄── pg1 renews every few seconds
└───────────────────────────────┘
pg1 (primary) ──WAL──► pg2 (replica)
└────────WAL────► pg3 (replica)
pg1 dies → lease expires → least-lagged replica (pg2) promotes
→ proxy re-points clients at pg2
Underneath the name, three concepts an interviewer actually probes. Fencing: the old primary must be prevented from accepting writes when it comes back — lease expiry plus demote-on-start; without fencing you get split-brain and divergent timelines. Data-loss window: async replication means promoting a lagged replica loses the tail — maximum_lag_on_failover bounds it; only sync replication makes it zero. Timeline forks: the demoted primary’s un-replicated WAL must be discarded — pg_rewind reconciles it back into the cluster as a replica:
shared history ──1──2──3──┬── old primary keeps writing: A4──A5 (never replicated)
└── new primary writes: B4──B5 (the surviving timeline)
pg_rewind = cut A4–A5 off the old primary, rejoin it as a replica on the B timeline
Notice these are exactly your hot-standby failover problems — fencing, gap-free handover, split-brain — wearing DB clothes. Say that; it converts your engine experience into DB credibility.
Connection pooling: pgbouncer
You literally run one, so own this. Why pooling exists: each Postgres connection is a backend process with real memory cost, and connection storms (a fleet of app instances × their pools) collapse a server that’s happy at 50 active backends. pgbouncer multiplexes thousands of client connections onto tens of server connections.
The modes, because this is the standard probe:
- Session pooling — server conn held for the client’s whole session; safe, least sharing.
- Transaction pooling — server conn borrowed per transaction; the production default and the big multiplexing win.
- Statement pooling — per-statement; forbids multi-statement transactions; rare.
Transaction pooling buys its multiplexing by breaking session state: no session-level PREPARE/prepared-statement caching by name (pgbouncer 1.21+ added protocol-level support), no SET that must persist, no session advisory locks, no LISTEN. Your concrete hook: your platform runs host-native pgbouncer on :5432 routing by database name to a loopback Postgres, ~25 server conns per tenant DB against a 500-client ceiling — that’s transaction-pooling economics in one sentence, from your own infra, plus a real caveat you documented yourself (your ReadySet cache tier bypasses per-tenant auth — an isolation trade you made consciously). Interviewers remember candidates who volunteer the caveat they own.
Online schema migration — the flagship skill
The scenario every platform-flavored interview reaches: “the orders table has 500M rows and the system trades 24/7. Add a column / change a type / add an index. Go.” The framework: know your locks, expand-migrate-contract, backfill in batches.
Lock analysis first. DDL takes locks; the killer is ACCESS EXCLUSIVE (the strongest table lock — conflicts with everything including plain SELECTs; lock levels are decoded in ch00f). Worse trap: lock queuing — your ALTER waits behind one long-running query, and every subsequent query (even reads) queues behind your waiting ALTER. A “fast” migration can freeze production for minutes without doing any work:
time ──►
[long-running SELECT]════════════════════╗ holds ACCESS SHARE on orders
ALTER TABLE orders ... ──waits──►║ wants ACCESS EXCLUSIVE — queued
SELECT ... ──waits──►║ queued BEHIND the waiting ALTER
SELECT ... ──waits──►║ queued
UPDATE ... ──waits──►║ queued
╚═ table effectively frozen: nothing new
runs until the long query ends AND the
ALTER finishes (or gives up)
Defense: SET lock_timeout = '2s' — the ALTER aborts after 2s of waiting,
the queue drains, you retry later. The freeze is bounded by the timeout.
Defenses: SET lock_timeout = '2s' in every migration session and retry, run migrations when long transactions aren’t running, and know the lock cost of each operation:
| Operation | Lock / cost | Verdict |
|---|---|---|
ADD COLUMN (nullable, no default) | ACCESS EXCLUSIVE, metadata-only, instant | Safe (with lock_timeout) |
ADD COLUMN ... DEFAULT <constant> | Instant since PG 11 (default stored in catalog, not rewritten) | Safe on modern PG; table rewrite pre-11 |
ADD COLUMN ... DEFAULT <volatile fn> | Full table rewrite under ACCESS EXCLUSIVE | Never online |
ALTER COLUMN TYPE (e.g. int→bigint) | Usually full rewrite + index rebuilds, ACCESS EXCLUSIVE | Never online — use expand-contract |
SET NOT NULL | Full validation scan under ACCESS EXCLUSIVE (PG <12); instant if a validated CHECK constraint already proves it (PG 12+) | Use the CHECK trick |
ADD CONSTRAINT ... NOT VALID then VALIDATE CONSTRAINT | NOT VALID is instant; VALIDATE takes only SHARE UPDATE EXCLUSIVE (reads/writes continue) | The safe pattern for constraints/FKs |
CREATE INDEX | Blocks writes (SHARE) for the whole build | Use CONCURRENTLY |
CREATE INDEX CONCURRENTLY | No write block; two table scans + wait for old snapshots | Safe, with failure modes below |
DROP COLUMN | ACCESS EXCLUSIVE, metadata-only (space reclaimed lazily) | Safe-ish; it’s the semantic contract you’re breaking |
Expand–migrate–contract, step by step. The worked example interviewers love: orders.qty is int4 and you need int8. In-place ALTER TYPE rewrites 500M rows under an exclusive lock — hours of downtime. Instead:
-- EXPAND: add the new column (instant, metadata-only)
ALTER TABLE orders ADD COLUMN qty_v2 bigint;
-- Dual-write: deploy app code writing BOTH columns (or a trigger while old code drains):
CREATE OR REPLACE FUNCTION orders_qty_sync() RETURNS trigger AS $$
BEGIN NEW.qty_v2 := NEW.qty; RETURN NEW; END $$ LANGUAGE plpgsql;
CREATE TRIGGER t_qty_sync BEFORE INSERT OR UPDATE ON orders
FOR EACH ROW EXECUTE FUNCTION orders_qty_sync();
-- MIGRATE: backfill in keyed batches — small transactions, no long locks,
-- throttled, resumable from a checkpoint of last_id:
UPDATE orders SET qty_v2 = qty
WHERE id > $last_id AND id <= $last_id + 10000 AND qty_v2 IS DISTINCT FROM qty;
-- loop, sleeping between batches; watch replication lag and bloat as you go
-- Verify: count mismatches, spot-check; then enforce NOT NULL the online way:
ALTER TABLE orders ADD CONSTRAINT qty_v2_nn CHECK (qty_v2 IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT qty_v2_nn; -- scan, but only SHARE UPDATE EXCLUSIVE
ALTER TABLE orders ALTER COLUMN qty_v2 SET NOT NULL; -- instant on PG12+: proven by the CHECK
-- CONTRACT (a separate deploy, days later, after reads are switched):
-- deploy code reading qty_v2 only → drop trigger → drop old column
DROP TRIGGER t_qty_sync ON orders;
ALTER TABLE orders DROP COLUMN qty; -- instant; optionally rename v2→qty
Narrate the deploy interleaving — that’s what the question is really testing: every schema state must work with both the previous and next app version (N/N+1 again — the schema-evolution guarantee of ch14, applied to DDL). Expand ships before any code depends on it; contract ships only after no running code touches the old column; each step is independently rollback-safe.
Batched backfill details that show production scars: batch by primary-key range, not LIMIT/OFFSET (offset re-scans); keep batches small enough that each transaction is milliseconds (row locks held briefly, replicas keep up); make it resumable and idempotent (IS DISTINCT FROM guard).
Then expect bloat, and plan the cleanup. Every updated row is a new row version under MVCC (multi-version concurrency control — Postgres never overwrites a row in place, it writes a new version and leaves the old for vacuum; ch00f), so a 500M-row backfill leaves up to 500M dead versions behind. That’s the job for pg_repack: an extension that rebuilds a bloated table/index online by copying it to a new table while a trigger captures deltas, then swaps under a brief exclusive lock — the online alternative to VACUUM FULL (which takes ACCESS EXCLUSIVE for its whole run).
Finally, name-drop the linting layer: tools like squawk / strong_migrations exist to catch unsafe DDL in review, and lock_timeout-plus-retry belongs in the migration runner, not in tribal memory.
Zero-downtime index builds. CREATE INDEX CONCURRENTLY: builds without blocking writes — it scans the table once to build the index, scans a second time to catch rows written during the first pass, then waits out every transaction older than itself, so no in-use snapshot predates the index. Failure modes to recite: (1) it’s slower and can wait indefinitely behind long-running transactions (including idle-in-transaction — pgbouncer clients misbehaving, or a forgotten psql); (2) if it fails or is cancelled it leaves an INVALID index — still maintained on writes (pure cost), unusable for reads — you must DROP INDEX CONCURRENTLY and retry; (3) can’t run inside a transaction block, so your migration tool needs a non-transactional mode; (4) unique-index builds can fail late on duplicates. Same story for REINDEX CONCURRENTLY (PG 12+). “INVALID index left behind” is the detail that proves you’ve done this.
Reconciliation: engine state vs. DB drift
Your engine’s truth is the log-derived in-memory state; the DBs downstream are projections; venues hold their version of your orders and balances. These will drift — a projection consumer crashes mid-batch and double-applies, a venue fill never reaches you, a manual DB fix bypasses the log. Reconciliation is the immune system:
- Engine vs. warehouse/DB: periodically (end-of-day at minimum, hourly better) compare authoritative aggregates — position per instrument, open-order count/qty per venue, cash — between engine snapshot at sequence N and the projection’s view as of N. Sequence numbers make “as of the same point” well-defined; without them recon chases its own tail. Row-level diff on mismatch; alert with materiality thresholds (a 1-lot drift pages differently than a 10k-lot drift).
- Engine vs. venue: the one that costs money. Venue drop-copy / execution reports / REST “open orders” and balance endpoints diffed against engine state; on restart this is a mandatory gate before trading resumes (ch16). Crypto reality you know firsthand: REST snapshots are rate-limited and eventually consistent with their own WS stream, so recon logic needs tolerance windows, not equality asserts.
- Design stance to state: recon detects, humans-or-runbooks decide, and every correction is applied as a new logged event (
ManualAdjustment { reason, ticket }), never by mutating state or DB rows in place — otherwise you’ve just created drift the next recon can’t explain and broken replay besides.
Plain-English recap
- Rule zero is “the card-authorization decision never waits on the warehouse.” A microsecond decision path can’t afford a network hop to anything with a query planner — not Postgres, not even Redis. Decisions run on in-process state; the databases are downstream.
- Log-as-truth is CDC turned into the whole architecture. You know the pattern from change-data-capture: events stream out, projections consume them. Here the event log is the system of record and every DB — tick store, Postgres, dashboards — is a derived read model. Postgres itself works this way internally: WAL is the event log, tables are the projection, recovery is replay.
- Tick stores are your analytics warehouse, specialized for time. Columnar, append-only, range-scanned — ClickHouse-shaped. The one exotic bit is the as-of join: “match each trade to the quote in force at that moment,” which is the same query as “match each payment to the FX rate in force when it settled.”
- The lock-queue trap is the senior half of migrations you already run. You’ve
done expand–contract column changes; the trap is that even an instant
ALTERcan freeze production for minutes by queuing behind one long query while everything else queues behind it.lock_timeout+ retry bounds the freeze — that detail is what interviewers listen for. - pgbouncer you literally run. Transaction pooling’s broken session state
(prepared statements,
SET, advisory locks,LISTEN) is the standard probe; your own host — 500 clients multiplexed onto ~25 backend conns per DB — is the worked example, and the ReadySet auth-bypass caveat you documented is the volunteered trade-off that lands. - Reconciliation is payments recon, verbatim. Engine vs DB is ledger vs read-model at a common sequence number (the statement date); engine vs venue is ledger vs PSP settlement report, tolerance windows included; and every correction is a new journal entry, never an UPDATE on history.
Interviewer will ask
Q1: “Where’s the database in your hot path?” Nowhere — the card-authorization rule: the decision never waits on the warehouse. The arithmetic is the argument: the budget is microseconds, and anything with a query planner, a lock manager, or a network hop costs orders of magnitude more — even Redis at ~100µs RTT blows the whole budget, never mind Postgres. So working state lives in process memory, and durability comes from the append-only event log, replicated to the standby — the log is the record. Every database is a downstream projection of that log: columnar tick store for time-series analytics, Postgres for reference/accounts/compliance, Redis for ephemeral-by-policy coordination. Land the inversion: log-as-truth, DB-as-view — the opposite of the enterprise model where the DB is truth and logs are exhaust. The discipline: nothing in Redis is the record of anything; lose it and you re-derive.
Q2: “Why does kdb+ dominate tradfi tick data?” Because the tick workload has a specific shape, and kdb+ matches it three ways. Query shape: billions of time-ordered rows, range-scanned and aggregated by column — and the canonical query is the as-of join, each trade matched to the quote in force at its timestamp, which kdb+ supports natively. Storage shape: date-partitioned, one file per column, so “average spread over three months” reads only the columns it touches, at sequential-scan speed. Language shape: q runs identically over today-in-RAM (RDB) and history-on-disk (HDB), so analytics execute inside the store instead of hauling billions of rows out. Add two decades of installed trust in banks, and it won. The costs: license cost, terse language, key-person risk — which is why crypto shops run ClickHouse or QuestDB, same shapes, open source.
Q3: “Add a NOT NULL column with a default to a 500M-row table that’s live. Go.”
State the governing rule first: every step exists to avoid holding ACCESS EXCLUSIVE — the lock that blocks even reads — for longer than a metadata flip. And name the trap that has nothing to do with work: your ALTER queues behind one long query, everything else queues behind your ALTER, so lock_timeout + retry bounds the freeze. Then the plan. On PG 11+, ADD COLUMN DEFAULT constant is instant — the default lives in the catalog, no rows rewritten — so the literal question is already solved. If the default must be computed, each arrow has its reason: nullable add (metadata-only, instant) → dual-write (new rows arrive complete) → batched keyed backfill (each transaction holds row locks for milliseconds) → CHECK NOT NULL NOT VALID (instant, no scan yet) → VALIDATE (the scan, but under a lock that lets reads and writes continue) → SET NOT NULL (instant, because the validated CHECK already proved it). Mentioning the lock-queue trap is what marks the answer senior.
Q4: “int4 → int8 on a live orders table.”
One invariant generates the whole procedure: every intermediate schema works with app versions N and N+1, and every step rolls back independently. In-place ALTER TYPE violates it immediately — a 500M-row rewrite under ACCESS EXCLUSIVE, hours of downtime. So each step is the invariant applied: add qty_v2 (old code ignores it); dual-write via trigger or app code (both columns stay true whichever version runs); backfill in keyed, idempotent batches (either version reads correctly mid-backfill, and it resumes after interruption); verify, then constraint-then-NOT-NULL the online way; switch reads; contract — drop the old column — in a later deploy, once nothing running touches it. Land: expand–migrate–contract is the N/N+1 guarantee of the schema-evolution chapter (ch14) applied to DDL.
Q5: “CREATE INDEX CONCURRENTLY — what can go wrong?”
Derive the failures from the mechanism: it avoids blocking writes by scanning twice, then waiting out every transaction older than itself, so no in-use snapshot predates the index. That wait is failure one: it hangs indefinitely behind a long-running or idle-in-transaction session — a forgotten psql, a misbehaving pooled client — because the wait has no timeout. Failure two comes from the index being registered before it’s valid: cancellation or crash leaves an INVALID index, maintained on every write (pure cost) but unusable for reads — DROP INDEX CONCURRENTLY and retry. Two more from the same machinery: it can’t run inside a transaction block, so your migration tool needs a non-transactional mode; and unique builds can fail late, on a duplicate the second scan finds. “INVALID index left behind” is the giveaway that you’ve run this in anger. Bonus: same CONCURRENTLY machinery for REINDEX, and pg_repack for bloat.
Q6: “Streaming vs. logical replication — when each?” One contrast carries everything: streaming ships raw WAL block bytes, which only mean something to a bit-identical cluster; logical decodes WAL into row events, which can land anywhere. Everything follows. Streaming: an exact block-for-block replica, same major version, whole cluster or nothing — which is precisely the HA/failover job, sync or async per durability need. Logical: a live, writable subscriber applying row changes — so a different major version (near-zero-downtime upgrades), a subset of tables, or CDC into Kafka/ClickHouse. Rule of thumb: streaming for HA, logical for upgrades, migrations, selective copies, and CDC. Volunteer the two operational teeth: DDL doesn’t replicate logically, and an abandoned slot pins WAL until the primary’s disk fills.
Q7: “How does Postgres failover actually work in production?”
Patroni: agents + leader lease in etcd, election of the least-lagged replica, promotion, client re-routing via proxy/health endpoints. Then the three concepts under the tooling, each with its why: fencing — the old primary must be unable to take writes when it returns, or split-brain gives you two divergent histories; the data-loss window — async means a promoted replica may lack the tail, so maximum_lag_on_failover bounds the loss and only sync replication zeroes it; and pg_rewind — the demoted primary’s un-replicated WAL is a fork that must be cut off before it rejoins as a replica. Then the bridge: identical problem shape to your engine’s hot-standby failover — different layer, same fencing/gap/split-brain checklist.
Q8: “How do you know your engine and your DB agree?” You don’t — you check: scheduled reconciliation of aggregates as-of a common sequence number — the statement date; without a common cutoff, recon chases its own tail — row-drill on mismatch, materiality-tiered alerts; venue-side recon against drop-copy/REST as a hard gate on restart; and all corrections applied as logged adjustment events so the audit trail and replay stay intact. “Recon detects, events correct, nothing mutates in place.”
Further reading
- Kleppmann, DDIA — ch. 3 (Storage and Retrieval: B-trees vs LSM, columnar storage — the tick-store theory), ch. 5 (Replication: leader-based, sync/async, failover pitfalls — maps 1:1 onto the Patroni section), ch. 11 (derived data / “turning the database inside out”).
- Postgres official docs — chapters on WAL configuration and reliability, high-availability & replication (streaming), Logical Replication (restrictions section especially),
ALTER TABLEnotes (lock levels per subform), andCREATE INDEX(“Building Indexes Concurrently”). - Patroni documentation — architecture and failover/switchover semantics; skim
pg_rewinddocs alongside. - pgbouncer docs — the features/pooling-modes page; the transaction-pooling caveat list is interview gold.
- kdb+ / KX whitepapers (code.kx.com) — “Building Real-time Tick Subscribers,” the ticker-plant architecture papers, and the columnar/splayed-table storage docs; skim one so your kdb+ paragraph is grounded.
- ClickHouse docs — MergeTree internals and
ASOF JOIN; the Braintree/GitLab-style public write-ups on zero-downtime Postgres migrations (GitLab’s migration style guide is public and excellent) for battle-tested expand-contract discipline.
Where this goes next: databases can be migrated online — but your engine holds state no load balancer can flip. Chapter 16 is how you deploy a stateful trading engine with zero downtime: hot-standby cutover, session takeover, shadow deploys, and rollback discipline.
Zero-Downtime Deploys of Stateful Engines
Before you start — this chapter leans on a handful of primer ideas:
- FIX sessions and sequence numbers — the venue-side session state that makes takeover hard: ch00f
- Feed snapshot + delta resync — what a reconnecting consumer must do to rebuild a book: ch00f
- The determinism contract and state hashes — how a new binary proves it agrees with the old one: chapter 13
- N/N+1 schema compatibility — why the new version must not write what the old can’t read: chapter 14
Read those first — 20 minutes there saves an hour here.
Blue-green deployment is a solved problem for stateless web services: run two fleets, flip the load balancer, done. Fowler wrote it up two decades ago. Web blue-green works because state lives in the database and sessions live in a cookie — a load-balancer flip moves nothing that matters. The reason this is an interview topic for trading infrastructure is that your engine is the opposite of stateless, in three compounding ways:
- Derived state — books and positions that took hours of message-by-message accumulation to build up in memory. A load-balancer flip transfers none of it.
- Open orders — orders resting at venues, which keep existing whether or not your process does. This is state held at a third party.
- Live sessions — FIX connections with sequence numbers (ch00f), authenticated WebSockets: stateful protocols whose other half the counterparty owns — and the counterparty didn’t agree to your deploy.
Any deploy story that doesn’t address all three is a web-app answer wearing a trading costume — interviewers are listening for exactly that tell.
The escape hatch from all three is the event log. Because state is a deterministic fold over the log (the fold of ch13: state = events.reduce(apply, empty) — feed the same events in, get the same state out), “transfer state to the new version” reduces to “let the new version consume the log.” Every pattern below is a variation on that move.
Pattern A: hot-standby cutover (the flagship)
This is the pattern you’ve actually run, so it’s your anchor story. The choreography:
-
Start the new binary (green) as a standby. It loads the latest snapshot, replays the tail, then consumes the live sequenced stream, staying at the head. Those three steps on one picture of the log:
log: [ events the snapshot already summarizes ][ tail ]──► head (live, still growing) green: 1. load snapshot ═══════════════════════► 2. replay tail ► 3. follow the headIt is a full engine with outputs disabled — same fold, sink discarded (the no-I/O replay mode of ch13 doing production work).
-
Health-verify while both run. Green publishes rolling state hashes; they must match blue’s at the same sequence numbers. This is the payoff of the determinism contract: you get a proof the new version agrees with the old on live traffic before it owns anything. If the release intentionally changes behavior, you instead pre-verify via replay-diff in CI (ch17) and accept expected-diff categories; unexplained divergence aborts the deploy.
Preview: replay-regression testing (ch17). Replay recorded production days through the candidate build and diff its decisions — orders, cancels, prices — against what production actually did. Sort every diff into two piles: intended by this change (predicted in advance, in roughly the predicted amounts) and anything else. Anything else fails the gate. That’s all this chapter means by “replay-diff” and “expected-diff categories”; the change-management chapter builds the full machinery.
-
Cutover. Quiesce briefly (go quiet) at a sequence boundary: blue stops accepting new inputs (or the sequencer marks a
LeadershipTransferevent in the log — cleaner, because the handover point is itself logged and replayable), green confirms it has applied through that sequence, then green’s outputs go live and inputs route to it. With both processes hot, the gap is the routing flip — milliseconds. -
Fence blue. The old primary must be unable to act after the flip — never rely on “we told it to stop.” The mechanism is the epoch (also called a term or generation number): every leadership transfer bumps it by one, and every log append and outbound order carries the writer’s epoch stamp; anything downstream rejects stale epochs. Concretely: blue, still running because a shutdown script hung, emits an order stamped epoch 42; the gateway’s current epoch is 43; the order is rejected at the door. It’s a rotated API key — the old process can still make requests, but its credential no longer opens anything. Deploys are just voluntary failovers with the same fencing, and saying that sentence in an interview reframes your whole standby experience as a deployment skill.
-
Blue drains and lingers. Keep it running (outputs fenced) as the instant rollback target for the bake period (the watch-and-wait window after cutover, before you fully trust the new version).
The whole choreography on one timeline:
time ──►
BLUE (v42) live: applying log, outputs ON ─────────────────╳ fenced: epoch 42
│ rejected downstream;
│ stays hot as rollback
log (seq) ─1000──1001──1002──[LeadershipTransfer{epoch:43}]─┬─1003──1004──►
│
GREEN (v43) snapshot ► tail replay ► live tail, outputs SINKED
└── rolling state-hash vs BLUE: must match ──┘ └► outputs ON,
at same seq numbers (gate) owns sessions
The session takeover problem
The hard residue is step 3’s unstated assumption: green can talk to the venues. Sessions don’t transfer through the log.
FIX (tradfi): a FIX session is (SenderCompID, TargetCompID, inbound/outbound sequence numbers) over TCP. Three takeover options, in ascending sophistication.
Option (a): re-logon. Blue logs out, green logs on. You must persist and hand over the sequence numbers — they’re session state, so put them in the log/snapshot like everything else. If the seqnums don’t line up on logon, the two sides negotiate a resend/gap-fill. In miniature:
- Green logs on: “my next outbound is 5001, and I expect your 8200.”
- Venue: “I only got through your 4990 — resend 4991–5000.”
- Green resends them (or gap-fills the ones that are now stale), and the session is level again.
The classic incident is the naive seqnum reset: green comes up claiming sequence 1 while the venue’s counter says 12000. One side now believes 11999 messages went missing and asks for all of them — the venue replays, or rejects, what looks like a full day of traffic. Even done right, re-logon costs seconds of session downtime; resting orders at the venue survive (they live in the venue’s book), but you’re blind and can’t cancel during the gap.
Option (b): session handover via a FIX gateway tier. The venue-facing TCP session lives in a thin, rarely-deployed gateway process; engines behind it come and go without the venue ever seeing a logout. This separation — long-lived dumb edge, frequently-deployed smart core — is the architectural answer interviewers want, and it’s the same isolation move as the feed handlers of ch14.
Option (c): TCP handoff / connection-migration tricks. These exist but are exotic; name them only to dismiss them.
Crypto (your world): WebSocket sessions with auth tokens; there is no seqnum continuity contract, which is simpler and worse. Deploy = green must re-authenticate and resubscribe N venues × M streams, which raises six concerns at once:
- Rate limits — the auth burst hits per-connection and per-endpoint limits (you have felt this).
- Resubscription storm — N × M subscribe messages, all at once.
- Book resync — a snapshot-plus-buffered-deltas window per venue before its book is trustworthy again (your daily bread).
- Pre-warming — open green’s connections before cutover where venues allow duplicate sessions; many do.
- Session-kill quirks — some venues kill the older session on new auth. Know it per venue; that per-venue quirks table is itself an asset worth mentioning.
- Reconciliation as a gate — open orders rest at the venue under venue-assigned IDs, so green needs the ID map (in the log) and must reconcile — compare its own open-order picture against the venue’s, line by line (ch15) — before it may trade. A gate, not a nicety.
The structural mitigation is the same gateway-tier answer as FIX: keep the connection owners out of the deploy blast radius.
Pattern B: drain-and-replace (for gateways and order-path services)
For services where in-flight work is short-lived — order gateways, risk checkers, REST/API frontends — you don’t need state transfer at all:
- Mark the instance draining: stop routing new orders/requests to it.
- Let in-flight work complete or time-bound it: wait for outstanding acks, cancel-on-timeout stragglers.
- When quiesced, kill and replace; new instance registers for traffic.
The design prerequisites are the interview substance:
- An upstream router that can exclude an instance — or venue-session multiplexing, so a draining gateway’s sessions move elsewhere.
- Idempotent order handling — client-order-ID dedupe, so a retry through the new instance can’t double-submit.
- A hard drain deadline — so one stuck order can’t wedge the deploy.
In crypto, “drain” also means “stop the strategy quoting through this gateway and let its resting orders be cancelled or adopted by another gateway.” Adoption is less mystical than it sounds: the orders never move — they rest at the venue the whole time. Gateway 2 loads the venue-order-ID map from the log and takes over cancel/replace duty for them.
Pattern C: shadow deployment (decision diffing)
Run the candidate against reality with zero risk: green consumes the live production feed and order flow, runs its full logic, and its outbound orders go to a sink that records instead of sends. Then diff green’s decisions against blue’s actual decisions, streamed, with a triage UI or even just a diff log.
This is strictly stronger than replay-based testing for one reason: it exercises today’s regime (the market’s current personality: its volatility, volumes, and quirks) — including inputs your recorded days don’t contain. It’s the cheapest high-fidelity test in the industry if and only if you have determinism and log-consumption as primitives, which you do; shadow mode is literally your standby with a recording sink. Limits to volunteer, because they’re the senior half of the answer: shadow can’t see market impact (its phantom fills come from a fill simulator against the live book), can’t test venue interaction (rejects, rate-limit behavior, partial-fill sequencing), and decision-diffs need noise discipline (an intended pricing change diffs everywhere; you need expected-vs-unexpected diff classification before the signal is usable — ch17). The fill simulator gets dishonest about queue position: the shadow’s order never actually stood in the price level’s line, so “would it have filled” is a guess about where in the queue it would have sat. Those guesses skew optimistic — the simulator awards fills that a real order, waiting behind everyone who arrived first, would not have gotten.
Pattern D: rolling by shard / venue (bounding blast radius)
How it works. If the system is sharded — per-venue feed handlers and gateways, per-symbol-group engines — deploy one shard at a time. A canary is the web-deploy idea unchanged: ship to one small, low-risk slice and watch it before the rest; here the slice is a small, liquid, forgiving venue — not your biggest P&L venue. So: canary venue first, bake, proceed in waves, halt-and-rollback on any regression.
Prerequisites. Shards genuinely independent (a shared risk service or cross-venue arb strategy couples them — know your coupling before claiming independence); per-shard health metrics with automatic gate checks between waves; and N/N+1 message compatibility (ch14), since mixed versions now coexist for hours, not minutes, on the bus.
When it applies — and when it doesn’t. For your 20-venue reality this is the default deploy mode for handler/gateway changes: venue-by-venue is both blast-radius control and a natural fit to per-venue protocol quirks. Engine-core changes, by contrast, are usually all-or-nothing per engine instance — which is why Pattern A exists.
Maintenance windows: 24/7 crypto vs. tradfi
Tradfi hands you a nightly maintenance window and a weekend; a huge fraction of “zero-downtime” pressure evaporates: you deploy at 5:30pm after the close, with the whole evening to verify and roll back. Session-close rituals (EOD snapshots, seqnum resets on many FIX venues at start-of-day) even give you natural state boundaries. Crypto gives you nothing: markets never close, weekends are often the highest-volatility periods, and there is no moment when open orders and positions are flat by nature. Consequences you should state as lived experience: every deploy is a market-hours deploy, so the Patterns above aren’t aspirational — they’re the only way to ship; you create synthetic windows by choosing low-activity hours (and your desk knows its venue-local quiet hours) and by flattening or reducing exposure pre-deploy as a policy decision with a real P&L cost (missed volume) that engineering must justify; and venue-side maintenance (exchanges restart their own matching engines, announced or not) doubles as your chaos testing. Tradfi interviewers enjoy hearing that last inversion: crypto infra people get failover drills for free because the venues perform them on you.
Rollback discipline
Forward-fix vs. roll back — the decision tree. Default is roll back: the old binary is a known-good artifact, the new one is a hypothesis you just falsified. Forward-fix only when: (a) rollback is unsafe because the new version has already written state the old can’t read — which you architect to avoid, next paragraph; (b) the defect predates the deploy (rolling back changes nothing); or (c) the fix is truly trivial and the bake/verify pipeline can validate it faster than a rollback — rarer than 3am-you believes. The discipline that makes the tree usable: decide the criteria before the deploy, in the runbook, because judgment during an incident is the worst judgment you own; and the moment rollback is on the table, kill switches (ch17) flatten risk first — you can think clearly about binaries once you’re not bleeding.
State compatibility with the N-1 binary is what makes rollback real. Rollback = the old binary must consume what the new one wrote: log events, snapshots, config. Rules: the new version must not emit new event/snapshot versions until it has baked past the rollback horizon — the point in time after which you’d no longer roll back, so N-1 compatibility can finally be relaxed. (This is the readers-first choreography of ch14: ship the ability to read the new format everywhere first, and only later let anything write it — here, v2-write capability flips on after bake, via config.) If the new version already wrote v2 events, rollback targets a patched N-1 that can at least skip-or-parse v2 (length-prefixed records and unknown-type tolerance make this survivable), or you accept replaying from the pre-deploy snapshot and reconciling — an incident, not a rollback. Test it: CI replays new-binary-written logs through the previous release. A rollback path that was never exercised is the untested recovery path of ch13 with worse timing.
The worked runbook: engine deploy at 3pm, market open
The interview set-piece. Deploying matching-engine build v42 → v43 (perf work + one intended behavior change in cancel handling), markets live. Three gates recur below; the legend:
- Gate 1 — replay-regression clean: only the intended diffs, in roughly the predicted amounts.
- Gate 2 — N-1 compatibility: the old binary can read what the new one writes.
- Gate 3 — live agreement: rolling state hashes match while both versions run on live traffic.
Narrate it in phases:
T-1 day — pre-verification. The day before, you prove the change is exactly what you think it is — nothing more. Replay-regression: last 5 prod days through v43; decision-diff classified — only expected cancel-path diffs, counts within predicted bounds (gate 1). Determinism dual-replay green. N-1 check: v42 replays a v43-written staging log (gate 2). Runbook reviewed; rollback criteria written down: any unexplained decision divergence, p99.9 order-path latency +20%, recon mismatch, or venue session instability → roll back, no debate.
T-30 min. Half an hour out, you freeze the world and warn the humans. Freeze other changes. Page-out to the desk: deploy window, what changes, abort authority (desk can veto). Verify snapshots current, standby healthy, kill switches tested today. Reduce exposure per policy: widen quotes / cut size on the canary scope.
T-15 min. Now the new version starts running for real — with its outputs still off. Start v43 as shadow/standby: snapshot load + tail replay + live consumption (outputs sinked). Watch it reach head; rolling state-hash comparison against v42 running — matching on all paths except the expected cancel diffs, each one auto-classified (gate 3). Pre-warm v43’s venue connections where duplicate sessions are allowed.
T-0 — cutover. The flip itself is one logged event and a routing change. Sequencer writes LeadershipTransfer{epoch: 43}; v42 stops emitting at that boundary (fenced by epoch on every downstream); v43 confirms applied-through-boundary, enables outputs, takes the sessions (gateway-tier handover, or scripted re-logon per venue in dependency order). Total order-path gap target: <1s; measured and recorded.
T+0 to T+15 — verification gates. The first fifteen minutes are verification, not celebration. Open-order reconciliation vs. every venue (hard gate — trading stays reduced until clean). Latency histograms vs. baseline. Fill/reject/cancel rates per venue vs. same-hour baseline. First N decision spot-checks on the changed path. v42 stays hot, fenced, at head — instant rollback.
T+15 to T+2h — bake. Then you hand back the full keys, slowly. Restore full size stepwise. v43 still writing v1-compatible events (write-flip comes tomorrow, after the rollback horizon). Desk sign-off closes the deploy; v42 stays resident until end of day.
Rollback branch (rehearsed). If any tripwire fires, the path back is already rehearsed: kill switch to cancel-only on affected scope → LeadershipTransfer{epoch:44} back to v42 (it’s hot, at head) → sessions back → recon gate → resume → post-mortem with the shadow-period logs, which — because everything is in the log — reproduce the divergence exactly.
Two sentences of meta land well after the walkthrough: every gate is mechanical (a number and a threshold decided in advance), and the whole procedure is only possible because state transfer, verification, and rollback all reduce to log operations — the deployment story is the event-sourcing story.
Plain-English recap
- Why blue-green fails here: the engine is a stateful WebSocket server with money attached. Web blue-green assumes state lives in the DB and sessions in a cookie; the engine holds hours of accumulated in-memory state, open orders resting at third parties, and live authenticated connections whose other half the counterparty owns. A load-balancer flip transfers none of that.
- Hot-standby cutover is promoting a Postgres replica — with a checksum gate. Green syncs from the ledger (the log), catches up to zero lag, and must prove byte-identical state (matching hashes at the same sequence number) before it’s allowed to lead. Determinism turns “we think it’s ready” into a mechanical gate.
- Session takeover is the PSP-connection problem. Your OAuth sessions and webhook registrations with a PSP don’t move because you redeployed. The gateway tier is the same cure you already use: keep a thin, stable edge (API gateway/proxy) that owns the external connections, and redeploy the smart backends behind it freely.
- Drain-and-replace is a k8s rolling deploy plus idempotency keys. Stop routing new work, let in-flight work finish with a hard deadline, and rely on client-order-ID dedupe — your idempotency-key reflex — so a retry through the new instance can’t double-submit an order.
- Shadow deployment is traffic mirroring / a dark launch. Run the candidate on live traffic with its outputs recorded instead of sent, then diff decisions — GitHub’s “Scientist” pattern with a P&L. Its blind spot is the same one mirroring has: the world never responded to the shadow’s actions, so market impact and venue reactions are unmeasured.
- Rollback discipline is “new app version, old DB schema” until bake ends. The new binary must not write formats the old binary can’t read until the rollback horizon passes — the same reason you don’t run destructive migrations in the same deploy as the code that needs them. And criteria are written down before the deploy, like auto-rollback thresholds in a pipeline, because 3am judgment is the worst judgment you own.
Interviewer will ask
Q1: “Why can’t you just blue-green a matching engine like a web service?” Three kinds of state a load-balancer flip ignores: derived in-memory state (hours of book/position accumulation — solved by log replay + live catch-up), open orders resting at venues (survive your process; need ID mapping and reconciliation before green may act), and stateful venue sessions (FIX seqnums / WS auth — the counterparty holds half the state). Then: every workable pattern is “new version consumes the log,” which is why event sourcing is a deployment primitive.
Q2: “How do you know the new version is safe before it takes over?” Layered: replay-regression over recorded prod days with classified decision-diffs (pre-deploy); live shadow consumption with rolling state-hash comparison against the incumbent (during deploy); recon and metric gates with pre-committed thresholds (post-cutover). Emphasize that determinism turns “hope” into “proof of agreement on live traffic,” and that intended changes need a diff-classification story — an intended change makes hashes diverge by design, so without classified diffs you can’t tell planned divergence from a bug.
Q3: “Walk me through FIX session continuity across an engine restart.” Seqnums are session state, so they live in the log and snapshot like all other state. Re-logon then negotiates from the persisted numbers: each side says where its counters stand, and they gap-fill the difference. Name the naive-reset incident — coming up claiming sequence 1 against a venue counter at 12000, so one side demands what looks like a full day of traffic back. Even done right, resting orders survive at the venue during the gap, but you’re blind and can’t cancel.
Better architecture: a gateway tier owns the venue-facing TCP session, and engines deploy behind it. The venue never sees a logout — long-lived dumb edge, frequently-deployed smart core.
Then the crypto contrast, in its own breath: WebSockets have no seqnum contract at all — less to manage, more that breaks silently. A deploy means re-auth bursts into rate limits, resubscription storms, and a book-resync window per venue before its data is trustworthy. Different failure surface, same cure: the thin, long-lived edge.
Q4: “Design a shadow deployment. What does it not tell you?” Candidate consumes the live feed and order flow, runs full logic into a recording sink; stream-diff its decisions against production’s, classified expected vs. unexpected. Then the blind spots, unprompted — and derive the big one from the queue picture: the shadow’s order never actually stood in the price level’s line. So “would it have filled” is a guess about queue position, and those guesses skew optimistic — the simulator awards fills a real order, waiting behind everyone who arrived first, would not have gotten. That’s why shadow can’t measure market impact. It can’t test venue interaction either — rejects, rate limits, partial-fill sequencing — because nothing was ever sent. And an intended change diffs everywhere, so without classification the noise drowns real regressions. Land: shadow complements replay — today’s regime versus curated hard days — it doesn’t replace it.
Q5: “Deploy went out, metrics look bad. Roll back or fix forward?” Default roll back — old binary is known-good, new is a falsified hypothesis; but first, kill switch to flatten/limit risk so the decision isn’t made while bleeding. Forward-fix only if rollback is state-unsafe (new-format events already written), the bug predates the deploy, or the fix genuinely validates faster than rollback. The deciding factor: criteria were written in the runbook before the deploy, and rollback works because we hold N-1 compatibility (write-flips after bake) and test old-binary-reads-new-log in CI.
Q6: “When can the old binary NOT read what the new one wrote, and what then?” When the new version emitted new event/snapshot schema versions before the rollback horizon — a self-inflicted wound the readers-first/write-later choreography exists to prevent. If it happens anyway: patched N-1 that skips unknown record types (possible because records are length-prefixed and type-tagged), or restore from pre-deploy snapshot and reconcile against venues — which you classify as an incident with an RTO, not a rollback.
Q7: “How do you deploy across 20 venues without betting the firm?” Rolling by venue: canary on a small forgiving venue, mechanical health gates between waves (fills/rejects/latency/recon per venue), halt-and-rollback on regression, N/N+1 bus compatibility because mixed versions coexist for hours. Note the coupling caveat — shared risk or cross-venue strategies mean shards aren’t as independent as the deploy plan assumes — and that per-venue rollout doubles as per-venue-quirk testing.
Q8: “There’s no maintenance window in crypto. How does that change your engineering?” It removes the escape hatch that lets tradfi defer this whole chapter to 5:30pm: every deploy is market-hours, so hot-cutover, shadow, and rolling-by-venue are the baseline, synthetic windows are created by policy (quiet hours, pre-deploy exposure reduction with acknowledged P&L cost), and venue-side restarts function as involuntary failover drills. Then flip it: this is why crypto infra experience transfers up to tradfi — you’ve been doing the hard version daily.
Further reading
- Martin Fowler, “BlueGreenDeployment” and “CanaryRelease” on martinfowler.com — the baseline vocabulary, so you can say precisely where stateful engines break its assumptions.
- Kleppmann, DDIA ch. 5 (Replication — leader failover pitfalls: split brain, fencing, lost updates map directly onto cutover) and ch. 11 (log-centric integration underpinning “deploy = new log consumer”).
- Aeron Cluster documentation — leadership transfer, snapshot + replay on join, and multi-node determinism: the productized version of Pattern A.
- FIX protocol session-layer specification (fixtrading.org) — logon, sequence numbers, resend/gap-fill; skim once so the seqnum story is precise, not folklore.
- Kief Morris, Infrastructure as Code — the drain/replace and immutable-artifact discipline generalized; useful vocabulary for the runbook framing.
- Public exchange post-mortems of failed upgrades and matching-engine outages (several venues publish them) — read two or three; interviewers love candidates who cite real failure modes rather than hypotheticals.
Where this goes next: the mechanisms exist — Chapter 17 is the process wrapper: how a change earns its way from a branch to production money through replay regression, canaries with hard caps, and kill switches that actually work.
Change Management
Before you start — this chapter leans on a handful of primer ideas:
- Determinism and replay — the contract that makes “re-run production history through the candidate” possible at all: chapter 13
- Shadow deploys and mechanical gates — the deploy patterns this process wraps: chapter 16
- Config as code and N-1 compatibility — why config gets the full pipeline: chapter 14
- Market/venue vocabulary (fills, quotes, resting orders, adverse selection): ch00f
Read those first — 20 minutes there saves an hour here.
The previous four chapters (ch13–ch16) gave you the mechanisms: determinism, versioned schemas, tiered storage, cutover patterns. This chapter is the process wrapper — how changes get from a branch to production money without betting the firm, and what happens when one goes wrong anyway. Interviewers probe this because most trading blowups are change-management failures wearing a technology mask, and the genre’s founding cautionary tale is worth knowing cold:
The Knight Capital story (August 2012). Knight deployed new order-routing code to 7 of its 8 servers — the 8th was missed, and it still carried dead test code called “Power Peg,” wired to a feature flag whose name the new release had reused. When the flag flipped on, the dead code woke up on that one server and started buying — and kept buying for 45 minutes, because nobody could tell which system was doing it. Roughly $440M gone; the firm was effectively dead within a week. Not a clever bug — a partial deploy plus a repurposed flag.
Replay-based regression: the backbone
Your event log makes possible a test most industries can’t have: run actual production history through the candidate build and compare what it would have done against what production did.
The nightly loop. Every night, CI replays a library of recorded prod days through the current release candidate: yesterday (regime freshness), plus a curated set of hard days — the flash-crash day, the venue-outage day, the day the feed gapped, the highest-volume day, the day that triggered last quarter’s incident. Curating that library is real work and worth mentioning: hard days are your most valuable test assets, and every incident post-mortem (below) contributes its day to the library.
Decision-diff gating. The output isn’t pass/fail on state hash — that only works for refactors. For real changes you diff decisions (orders sent, cancels, prices, sizes) and classify: expected diffs (the change’s intent — the new cancel logic should diff on exactly the cancel path; predict the category and rough magnitude before running) versus unexpected diffs (anything else — automatic gate failure, no human override culture).
The gate itself is mechanical: zero unexpected diffs; expected diffs within predicted bounds; plus invariant checks that must hold diff-or-not — position limits never breached in replay, no self-crosses (your own buy order filling your own sell order), risk checks fire where they should.
Replay’s limit: it assumes the market would have behaved identically despite your different orders — fine for regression (“did I change what I didn’t mean to change”), invalid for strategy evaluation. That’s backtesting with impact models (models of how the market would have reacted to your orders), a different discipline — don’t conflate them in an interview.
Determinism tests in CI are the substrate (the determinism audit of ch13): dual-process replay with hash comparison on every PR touching the engine; golden-file decode tests (a golden file is a checked-in expected-output fixture — snapshot tests, in Jest terms) on every PR touching codecs; and the N-1 compatibility replay (old binary reads new log) on every release build. Cheap, fast, and each one guards a contract the deploy patterns depend on.
Canary with real money
After replay and shadow (ch16) comes the only test that includes market impact: trading real money, made survivable by hard-capped limits. Canary scope: the new build runs one strategy, or one venue, or a slice of flow, with independent enforced caps — order size, open-order count, gross/net notional (the total dollar value at stake), max loss. A sane opening posture is ~1% of normal notional. The caps live in the risk layer, not the strategy config — the canary must not be able to mis-config its own cage; enforcement belongs to a component the change didn’t touch. Graduation is stepwise (1% → 5% → 25% → 100%) with mechanical gates between steps — same-hour baselines for fill rates, rejects, latency, P&L-vs-expectation — and any gate failure returns to zero, not to the previous step. Time-box each step: a canary that lingers at 5% for three weeks is a decision nobody made.
Feature flags vs. binary deploys in hot paths
The web-industry default — runtime feature flags everywhere, deploy dark, flip flags — needs modification in a latency engine, and interviewers use this to test whether you cargo-cult practices across domains.
The case against runtime flags in the hot path: every flag costs a branch, a load from flag storage (cache traffic if it’s shared/atomic state), and combinatorial state (2^N flag combinations, of which you tested maybe three; Knight again — the repurposed flag). The branch cost is worse than it looks: the CPU’s branch predictor guesses each branch’s direction in advance and pays a pipeline flush when it guesses wrong, so a flag that almost never flips is exactly the branch it mispredicts the one time it matters. Worst of all, a runtime-flippable flag means the system’s behavior can change without a deploy, without CI, without replay-regression — you’ve built a bypass around your entire verification pipeline.
The pattern that survives: config-at-startup. Behavior toggles are read once at process start into immutable config; changing one = restart = a deploy, going through the same gates (config is code — ch14). If a branch is truly hot-path-critical, resolve it at startup via monomorphization — generics/const-generics compiling the chosen variant to straight-line code; think build-time tree-shaking: the untaken variant doesn’t exist in the shipped binary, so there is no branch left to mispredict. The lighter alternative: dispatch chosen once at init, not per-message. What remains legitimately runtime-mutable is a small, enumerated set of operational controls: kill switches, limit values, throttles — things that must move faster than a deploy in an emergency. That’s not a feature-flag system; that’s the risk-control plane, next section, and the two must not blur: features go through deploys, stopping goes through switches.
Kill switches: taxonomy and drills
The inverted priority of trading infrastructure: you must be able to stop faster and more reliably than you can do anything else. Taxonomy, in expanding blast radius:
- Per-strategy: stop quoting/taking for one strategy; optionally cancel its resting orders. First resort; desk-level authority; used weekly in normal life.
- Per-venue: halt all activity on one venue (venue misbehaving, feed suspect, session flapping) — cancel opens there, block new sends. Your 20-venue world uses this constantly.
- Per-symbol / per-account: finer scopes as the risk model demands.
- Global (“the big red button”): stop all order flow firm-wide, cancel everything cancellable. Anyone on the desk can pull it; nobody needs permission; un-pulling requires seniority and a checklist. The asymmetry is the design: cheap to trigger, expensive to reset, because false-positive stops cost basis points (hundredths of a percent) while false-negative non-stops cost the firm.
- Flat-position button: one level beyond stop — stop and actively liquidate to flat. This one is dangerous — market orders into a dislocated market realize the loss at the worst price. Concretely: the price just gapped 5% down and the bid side of the book is empty (a liquidity vacuum); “flatten now” sells into nothing and locks in the worst print of the day. So it’s tiered: passive-flatten with a timeout, then aggressive. But it must exist and be rehearsed, because “we couldn’t get flat” is how bad hours become mortal days.
Engineering requirements:
- Minimal-dependency path — a kill switch that traverses the whole stack dies with the stack; it should be enforceable at the gateway/risk edge even if the engine is wedged.
- Persisted state, survives restart — a rebooting engine must come up stopped if the switch was pulled.
- Every pull logged — who, when, why.
- Drills — the part that separates real shops: pull each tier on a schedule against production (in a quiet window, with the desk warned), measure time-to-stopped and time-to-flat, and treat a failed drill as a P1. An untested kill switch is a decorative button.
One more reason to build all this properly: MiFID II (the EU’s markets regulation) literally requires the kill capability and evidence that it works (below).
Config as code + staged rollout
Restating the config discipline of ch14 as process, because config changes outnumber code changes and cause a disproportionate share of incidents:
- Config lives in git.
- Schema-validated and semantically linted in CI (limits positive, venues have credentials, referenced strategies exist).
- Deployed as versioned artifacts through the same pipeline stages as binaries — replay-regression where behavior-affecting (a new risk limit changes decisions; replay it).
- Canary scope first, then staged rollout, with instant rollback to the previous artifact.
- Hand-edits on hosts are findable (drift detection) and treated as incidents even when harmless.
The one-sentence interview version: “we deploy config with exactly the ceremony of code, because the system can’t tell the difference — and neither can the P&L.”
Incident discipline
When it goes wrong anyway:
First minutes — stop the bleeding, in order: appropriate-scope kill switch (smallest that plausibly contains it; global if unsure — the asymmetry rule), assess exposure (positions, open orders, venue state — this is why recon tooling must work during chaos, not just nightly), flatten if risk demands (the tiered flat button), then stabilize and only then debug. Roles matter: one incident commander, one person on comms, hands-on-keyboard separated from decision-making. Say the discipline plainly: no debugging while bleeding — the binary/rollback decision tree (ch16) comes after the position is safe.
Post-incident replay forensics — your structural advantage. Because every input is in the log, the incident is exactly reproducible: replay the day into the incident window, single-step the decisions, test the “what if the fix had been live” counterfactual by replaying through the patched build, and confirm the fix kills the failure without collateral diffs. No “couldn’t reproduce,” no log-archaeology guesswork — the event log converts post-mortems from forensics into re-execution. Then the loop closes: the incident’s day joins the nightly replay library, the post-mortem is blameless-but-specific (mechanism, not villain), and every action item is a gate, test, or drill — not a “be more careful.”
Audit and compliance trail (regulated-venue flavor)
Name-drop depth only — enough to signal you know this world exists and maps onto machinery you already have. EU MiFID II’s RTS 6 (algorithmic-trading systems requirements — SEC 15c3-5, the “market access rule,” is the US cousin) requires, roughly:
- Pre-trade controls on every order — price collars (reject any order priced absurdly far from the current market), max order size, max notional, repeated-order throttles — hard-coded in the flow, evidenced.
- Kill functionality — the switch taxonomy above, mandated, with proof it works.
- Annual self-assessment and stress testing of algo systems.
- Testing before deployment, including non-live environments — your replay/shadow/canary pipeline is the evidence pack.
- Real-time monitoring with alerting.
- Record-keeping — orders, quotes, decisions, time-synchronized to UTC within regulated tolerances (RTS 25 clock-sync flavor), retained for years.
The interview move: your event-sourced architecture makes most of this nearly free — the sequenced log with sequencer timestamps is the record-keeping and the reproduction evidence; the kill drills and canary gates are the self-assessment artifacts. Firms without the log retrofit compliance as a bolt-on; you get it as a projection. That contrast, stated calmly, is a senior answer.
Plain-English recap
- Replay regression is record-replay testing with production traffic. Imagine replaying yesterday’s actual webhook stream through a candidate build and diffing every side effect against what production really did. The hard-day library is your incident-fixture collection — every outage contributes its day.
- Decision-diff classification is snapshot-test discipline. Expected diffs are the snapshots you meant to update (predicted category and rough count, in advance); any other diff fails CI, no human-override culture. The gate is mechanical, by design.
- Canary with hard caps is processing 1% of live payments with a spend limit the new code can’t touch. The caps live in the risk layer — a component the change didn’t modify — because the canary must not be able to misconfigure its own cage. Graduation is stepwise, and any failure goes back to zero.
- The feature-flag argument is about your verification pipeline, not flags. A runtime-flippable flag changes production behavior with no deploy, no CI, no replay-regression — a bypass around every gate you built (that’s the Knight Capital shape). Config-at-startup turns every behavior change back into a deploy. The legitimate runtime-mutable set is the circuit-breaker plane: kill switches, limits, throttles.
- Kill switches are your PSP “pause payouts” button. Cheap to trip, expensive to reset, scoped by blast radius (strategy → venue → global), enforceable at the edge even when the core is wedged, and drilled — an untested kill switch is a decorative button.
- Incident discipline is a SEV process with money on fire. Smallest sufficient kill switch first, exposure check, flatten if needed, IC/comms/hands roles — no debugging while bleeding. Then the structural advantage: the log makes every incident exactly reproducible, so the postmortem is re-execution, not archaeology, and the incident’s day joins the regression library.
- The compliance section is audit-trail requirements, payments-style. Like PCI/SOC2 evidence, RTS 6 wants records, controls, and proof of testing — and an event-sourced system emits all of it as a byproduct of the log.
Interviewer will ask
Q1: “How do you test a change to the matching/strategy path before it sees money?” State the principle that generates the ladder: each rung adds a reality the previous one can’t see. CI tests — determinism dual-replay, golden files, N-1 replay — see only the code’s own contracts. Replay-regression adds your own history: recorded prod days plus the hard-day library, gated on classified decision-diffs (zero unexpected, expected within predicted bounds). Shadow adds today’s regime — live inputs your recorded days don’t contain. Canary adds your own market impact — the one thing no offline test can show — under risk-layer-enforced ~1%-notional caps with stepwise mechanical graduation. You climb because each rung answers a question the rung below structurally cannot. Then the caveat that marks seniority: replay is regression, not strategy evaluation — impact isn’t modeled, so P&L claims come from the canary, not the replay.
Q2: “Feature flags in a low-latency system — yes or no?” Runtime flags in the hot path: no — branch and cache cost, combinatorial untested states, and a behavior-change channel that bypasses replay-regression entirely; cite Knight’s repurposed flag as the canonical disaster. Config-at-startup instead, with startup-time monomorphization for hot branches (the tree-shaking move from above: the untaken branch doesn’t exist in the binary) — and every behavior change becomes a deploy through the gates. Carve-out: the runtime-mutable set is the enumerated risk-control plane — kill switches, limits, throttles — which is deliberately not a feature system.
Q3: “Design the kill-switch system.” Three parts: taxonomy, engineering, drills. Taxonomy is scoped by blast radius — strategy, venue, symbol, global — plus the tiered flat-position button (passive first, then aggressive, because market-ordering into a dislocated book locks in the worst print), with the authority asymmetry: cheap to pull, expensive to reset. Engineering: enforcement at the minimal-dependency edge so it works when the engine is wedged; persisted state so a restart comes up stopped; every pull logged. Drills: pull each tier on a schedule against production, measure time-to-stopped and time-to-flat, and treat a failed drill as a P1. Close with: regulators mandate this anyway (RTS 6 kill functionality), so build it once, properly.
Q4: “A bad change made it to prod and is losing money. Walk me through your first ten minutes.” Minute 0: kill switch, sized to what I actually know — if I can name the strategy, its switch; if all I know is “we’re bleeding,” global, because the expensive mistake is scoping the kill by optimism. New flow stops, resting orders cancel, and the loss rate is now bounded by open positions instead of by a runaway algo. Minutes 1–3: exposure assessment with the recon tooling — what positions do we actually hold versus what the bad change believed — which is where drop-copy recon earns its keep, because the sick system’s own view is a suspect witness. Minutes 3–5: flatten if risk demands, per the tiered policy — passive first, aggressive only if the market is moving against the position, because market-ordering into a dislocated book locks in the worst print. In parallel the roles split: one incident commander, one on comms to desk and compliance, one pair of hands on the system — the same person doing all three is how a bad ten minutes becomes a bad hour. Minutes 5–10: only now the rollback-vs-forward-fix tree, read from the runbook written before the deploy — almost always rollback, and the criteria were pre-committed precisely so nobody reasons under adrenaline. What never happens inside these ten minutes: debugging. No debugging while bleeding. Afterward: exact replay reproduction of the failure, counterfactual validation of the fix, and the incident day goes into the regression library so this exact mistake can never ship twice.
Q5: “What does ‘config as code’ mean concretely in your world?” Rationale first: config changes outnumber code changes, get fewer reviewing eyes than binaries do, and the engine can’t tell the difference — so config earns the full ceremony. Concretely: git plus schema validation and semantic lint in CI; versioned artifacts through the same replay/canary/staged pipeline as binaries when behavior-affecting; rollback to the previous artifact; host-drift detection, with hand-edits treated as incidents; and hot-reloadable config that touches the deterministic path enters as log events (ch13).
Q6: “How would you catch a change that’s subtly wrong — not crashing, just worse?” Layered nets for the quiet failures: decision-diff replay catches “different where it shouldn’t be”; shadow catches regime-dependent divergence replay can’t; canary gates on same-hour baselines (fill rate, reject rate, adverse selection, latency percentiles) catch “statistically worse”; and invariant monitors (self-cross, limit proximity, quote-to-trade ratios) catch “categorically wrong.” What none of them catch: a change that’s wrong only via market impact at full size gets past all of it until graduation steps expose it — which is why graduation is stepwise with returns-to-zero.
Q7: “What do regulators actually require of algo trading systems?” RTS 6 flavor (15c3-5 in the US): evidenced pre-trade controls (collars, size, notional, throttles), mandated and tested kill functionality, deployment testing and annual self-assessment, real-time monitoring, and UTC-synchronized order/decision record-keeping with multi-year retention. Then the architecture point: an event-sourced engine produces the records and reproduction evidence as a byproduct — compliance as projection of the log, not a bolt-on — and the drills/canary artifacts double as the self-assessment pack.
Further reading
- The SEC’s Knight Capital order (2013, admin proceeding re: 15c3-5) — the primary-source post-mortem of the genre-defining change-management failure; ten minutes to read, permanently quotable.
- Kleppmann, DDIA ch. 11–12 — derived data and “the log as the system of record,” the substrate for replay-regression and audit-as-projection.
- Martin Fowler, “CanaryRelease” and the “FeatureToggle” article (Pete Hodgson, martinfowler.com) — read the toggle taxonomy so you can argue against runtime toggles in hot paths from an informed position.
- ESMA MiFID II RTS 6 text (and a broker-published summary — several banks publish readable digests) — skim for the control vocabulary: pre-trade limits, kill functionality, self-assessment.
- Google SRE book (sre.google), chapters on release engineering and postmortem culture — the blameless post-mortem and staged-rollout discipline, translated here to money-loss incident response.
- Aeron Cluster docs + Martin Thompson’s talks — for the “deterministic replay as testing primitive” framing from the people who productized it.
Where this goes next: Chapter 18 compresses chapters 13–16 into a 250-line runnable lab — build an event-sourced order book, ship a v2 schema, and live-upgrade it mid-stream with hash-verified handover.
Lab III: Live-Upgrading an Event-Sourced Book
Before you start — this lab compresses Part III; have these fresh:
- The determinism contract and state hashes — every PASS in this lab is that contract, executed: chapter 13
- Upcasters and wire versioning — the one-line
if ver >= 2the whole lab turns on: chapter 14- Hot-standby cutover and fencing — what phase 4 is a miniature of: chapter 16
- What a limit order book is (bids, asks, price levels, resting orders): ch00f
Read those first — 20 minutes there saves an hour here.
Chapters 13–16 in one runnable file. You will build a tiny event-sourced limit-order-book, log it to disk in a v1 wire format, snapshot it, then perform the exercise that is the interview: ship a v2 schema (adds an order source tag), write the upcaster (the read-time version translator of ch14), and prove two things with state hashes: (a) the v2 binary replays the v1 log to bit-identical state, and (b) a v2 “process” (a second engine instance in the same binary; Extension 1 makes it real) can tail a live v1 writer, take over mid-stream, and continue the log in v2 — after which a cold replay of the mixed log reproduces the leader’s state. Zero dependencies, so nothing hides the mechanics. This code compiles and the output below is real (rustc 1.92).
The five phases on one timeline:
phase 1 A (v1 binary) writes 1000 v1 records ─────────────────► log
phase 2 B (v2 binary) cold-replays that log ► hash == A's? PASS
phase 3 B snapshots, reloads the snapshot ► hash == B's? PASS
phase 4 A writes on; B tails to head ► hash match ► A fenced,
B leads — and now writes v2 records
phase 5 fresh cold replay of the MIXED v1+v2 log ► hash == B's? PASS
Setup
cargo new lab-upgrade && cd lab-upgrade
Cargo.toml:
[package]
name = "lab-upgrade"
version = "0.1.0"
edition = "2021"
[dependencies]
# none — the lab is dependency-free so nothing hides the mechanics
The code
src/main.rs, complete. How to read it: skim main() first — it’s at the bottom, and it is just the five phases from the timeline above, a few lines each. Then read the helpers it calls, in file order: the event model and wire codec (encode/decode — decode is the entire upcaster), the Book fold, the log/snapshot I/O, and the seeded workload generator. Two deliberate details — the torn-tail break and the unknown-id no-op — are explained after the run.
// Lab III: live-upgrading an event-sourced limit order book.
// Zero dependencies. One run walks all five phases; every PASS is an assert.
use std::collections::BTreeMap;
use std::fs;
use std::io::{Read, Seek, SeekFrom, Write};
use std::path::Path;
// ---------------- current (v2) in-memory event model ----------------
// The engine ONLY knows this. Old wire versions exist solely in the codec.
#[derive(Clone, Copy, Debug, PartialEq)] // derive = the compiler writes the boilerplate (copy, print, ==) for you
pub enum Side { Buy, Sell }
#[derive(Clone, Debug, PartialEq)]
pub enum Event {
Place { id: u64, side: Side, price: i64, qty: u64, source: u8 }, // source: NEW in v2
Cancel { id: u64 },
Execute { id: u64, qty: u64 },
}
pub const SRC_UNKNOWN: u8 = 0; // upcast default: MUST reproduce v1 behavior
// ---------------- wire codec: [len u32][type u16][ver u16][payload] ----------------
// Picture one record as a strip of bytes; the log is these strips laid nose to tail:
//
// byte: 0 1 2 3 4 5 6 7 8 ...............
// [ len: u32 ][ type ][ ver ][ payload: len bytes ]
// "payload is "which "which fields packed at fixed
// N bytes" event" dialect" positions — no names
//
// Unlike JSON, no field names travel with the data — the positions ARE the
// names. That is why a shipped layout can never be rearranged: v2 may only
// append bytes at the end.
const T_PLACE: u16 = 1;
const T_CANCEL: u16 = 2;
const T_EXECUTE: u16 = 3;
// A u16 is 2 bytes; to_le_bytes lays them out least-significant byte first:
// 0x0102 is stored as [0x02, 0x01]. That fixed order IS the wire format: any
// machine reading the bytes back in that order rebuilds the same number.
fn put_u16(b: &mut Vec<u8>, v: u16) { b.extend_from_slice(&v.to_le_bytes()); }
// Same idea, wider: a u64 becomes its 8 bytes, smallest-first, appended to the buffer.
fn put_u64(b: &mut Vec<u8>, v: u64) { b.extend_from_slice(&v.to_le_bytes()); }
// And the signed flavor: i64 as 8 bytes. Negatives ride along fine — two's
// complement is just the agreed-on bit pattern for "below zero", and both ends agree.
fn put_i64(b: &mut Vec<u8>, v: i64) { b.extend_from_slice(&v.to_le_bytes()); }
// The mirror image: grab the 2 bytes sitting at offset `o`, rebuild the u16.
// (`try_into().unwrap()` turns the slice into the fixed-size array from_le_bytes
// wants; the unwrap can't fail — the slice length always matches.)
fn get_u16(b: &[u8], o: usize) -> u16 { u16::from_le_bytes(b[o..o + 2].try_into().unwrap()) }
// Same read-back for u64: the 8 bytes at `o` become the original number again.
fn get_u64(b: &[u8], o: usize) -> u64 { u64::from_le_bytes(b[o..o + 8].try_into().unwrap()) }
// And for i64.
fn get_i64(b: &[u8], o: usize) -> i64 { i64::from_le_bytes(b[o..o + 8].try_into().unwrap()) }
/// Encode = lay one strip down: each field lands at a known offset,
/// in write order, no names. `schema_ver` is what THIS binary writes: the v1
/// process writes ver=1 (no source byte); the v2 process writes ver=2.
pub fn encode(ev: &Event, schema_ver: u16) -> Vec<u8> {
let mut p = Vec::with_capacity(32); // payload buffer; with_capacity pre-reserves space — a perf hint, not format
let ty = match ev {
Event::Place { id, side, price, qty, source } => {
put_u64(&mut p, *id);
p.push(if *side == Side::Buy { 0 } else { 1 }); // side as one byte: 0=buy, 1=sell
put_i64(&mut p, *price);
put_u64(&mut p, *qty);
if schema_ver >= 2 { p.push(*source); } // the entire v2 schema change: one byte, appended last
T_PLACE
}
Event::Cancel { id } => { put_u64(&mut p, *id); T_CANCEL }
Event::Execute { id, qty } => { put_u64(&mut p, *id); put_u64(&mut p, *qty); T_EXECUTE }
};
// Assemble the strip from the diagram above: 8-byte header, then the payload.
let mut rec = Vec::with_capacity(p.len() + 8);
rec.extend_from_slice(&(p.len() as u32).to_le_bytes()); // bytes 0-3: payload length
put_u16(&mut rec, ty); // bytes 4-5: event type
put_u16(&mut rec, schema_ver); // bytes 6-7: wire version
rec.extend_from_slice(&p);
rec
}
/// Decode = read the strip back at the known positions — and UPCAST at the
/// boundary: whatever wire version comes in, a current-model Event comes out.
/// This function is the entire v1->v2 upcaster.
pub fn decode(ty: u16, ver: u16, p: &[u8]) -> Event {
// These offsets mirror encode's write order byte for byte. With no names on
// the wire, the positions are the schema: shift one offset and every record
// ever written silently decodes to garbage.
match ty {
T_PLACE => Event::Place {
id: get_u64(p, 0), // payload bytes 0-7
side: if p[8] == 0 { Side::Buy } else { Side::Sell }, // byte 8
price: get_i64(p, 9), // bytes 9-16
qty: get_u64(p, 17), // bytes 17-24
source: if ver >= 2 { p[25] } else { SRC_UNKNOWN }, // <-- the upcast
},
T_CANCEL => Event::Cancel { id: get_u64(p, 0) },
T_EXECUTE => Event::Execute { id: get_u64(p, 0), qty: get_u64(p, 8) },
_ => panic!("unknown event type {ty}"),
}
}
// ---------------- the book: a pure fold over events ----------------
// "Fold" = replaying a bank statement: start at zero, apply every transaction
// in order, arrive at the balance. Same statement, same order -> same balance,
// on any machine. The log is the statement; the Book is the running balance.
#[derive(Default, Clone)]
pub struct Book {
pub seq: u64,
// id -> (side, price, remaining qty, source). BTreeMap iterates in sorted
// key order, so the book reads out identically on any machine. (HashMap's
// iteration order is arbitrary and differs per process, so hashes would
// differ; the ch13 contract.)
pub orders: BTreeMap<u64, (Side, i64, u64, u8)>,
pub bids: BTreeMap<i64, u64>, // price -> total qty
pub asks: BTreeMap<i64, u64>,
}
impl Book {
// Pick the price-level map for a side: bids for buys, asks for sells.
fn level(&mut self, side: Side) -> &mut BTreeMap<i64, u64> {
match side { Side::Buy => &mut self.bids, Side::Sell => &mut self.asks }
}
// Shrink or remove order `id`: Some(n) = execute up to n, None = cancel it all.
fn reduce(&mut self, id: u64, by_qty: Option<u64>) {
// `if let Some(...)` reads "only if the id existed": remove() hands back the
// order's fields (unpacked right in the pattern), and an absent id skips the
// whole block — no null check needed, the shape of the code is the check.
if let Some((side, price, qty, src)) = self.orders.remove(&id) {
let take = by_qty.unwrap_or(qty).min(qty);
let lv = self.level(side);
let left = lv[&price] - take;
if left == 0 { lv.remove(&price); } else { *lv.get_mut(&price).unwrap() = left; }
if take < qty { self.orders.insert(id, (side, price, qty - take, src)); } // partial fill: re-insert the remainder
} // unknown id: deterministic no-op (idempotent replay)
}
/// Fold one event into the book: bump the sequence number, then mutate state.
pub fn apply(&mut self, ev: &Event) {
self.seq += 1;
match *ev {
Event::Place { id, side, price, qty, source } => {
self.orders.insert(id, (side, price, qty, source));
*self.level(side).entry(price).or_insert(0) += qty; // entry/or_insert: fetch the level, creating it at 0 if new
}
Event::Cancel { id } => self.reduce(id, None),
Event::Execute { id, qty } => self.reduce(id, Some(qty)),
}
}
/// Canonical state hash: a fingerprint of the entire book. Two processes
/// showing the same fingerprint hold the same state — that comparison is
/// every PASS in this lab.
pub fn hash(&self) -> u64 {
// FNV-1a: xor a byte in, multiply, repeat. Same recipe over the same bytes
// gives the same fingerprint on any machine — and that is all it promises.
// It detects divergence; it is NOT security (nobody here is forging books).
let (mut h, prime) = (0xcbf29ce484222325u64, 0x100000001b3u64);
// `mix` is a closure that captures `h`: each call feeds one value's 8 bytes
// into the running fingerprint. wrapping_mul lets the multiply overflow and
// wrap around on purpose — the wrap is part of the recipe, not a bug.
let mut mix = |v: u64| { for b in v.to_le_bytes() { h = (h ^ b as u64).wrapping_mul(prime); } };
mix(self.seq);
for (id, (side, price, qty, src)) in &self.orders {
mix(*id); mix(*side as u64); mix(*price as u64); mix(*qty); mix(*src as u64);
}
for m in [&self.bids, &self.asks] {
for (price, qty) in m { mix(*price as u64); mix(*qty); }
}
h
}
}
// ---------------- log + snapshot I/O ----------------
/// Encode one event and append its bytes at the end of the log file.
pub fn append(log: &mut fs::File, ev: &Event, ver: u16) {
log.write_all(&encode(ev, ver)).unwrap();
}
/// Read every complete record from `pos` onward, apply each to the book, return
/// the new position. Picture tailing a log file someone may STILL be writing:
/// take whole records only, and if the last one is half-written, stop and
/// remember where you got to. This one loop is cold replay, standby catch-up,
/// AND live tailing — the only difference is whether the writer has finished.
pub fn replay_from(path: &Path, book: &mut Book, mut pos: u64) -> u64 {
let mut f = fs::File::open(path).unwrap();
f.seek(SeekFrom::Start(pos)).unwrap(); // jump the file cursor to `pos` bytes from the start
let mut buf = Vec::new();
f.read_to_end(&mut buf).unwrap(); // slurp cursor-to-end into memory in one go
let mut o = 0usize; // read offset within buf
while buf.len() - o >= 8 { // is at least one full 8-byte header left?
let len = u32::from_le_bytes(buf[o..o + 4].try_into().unwrap()) as usize; // header bytes 0-3: payload length
// Torn tail: the header promises more bytes than the file has yet — a
// half-written record. Not an error: park here; the next call resumes at `pos`.
if buf.len() - o < 8 + len { break; }
let (ty, ver) = (get_u16(&buf, o + 4), get_u16(&buf, o + 6)); // header bytes 4-5, 6-7
book.apply(&decode(ty, ver, &buf[o + 8..o + 8 + len]));
o += 8 + len;
pos += (8 + len) as u64;
}
pos
}
/// Serialize the book to one file — a JSON.stringify of the whole state, except
/// positional bytes instead of named text: seq, log position, then every open
/// order in id order (BTreeMap order, so the bytes come out identical every time).
pub fn write_snapshot(path: &Path, book: &Book, log_pos: u64) {
let mut b = Vec::new();
put_u64(&mut b, book.seq);
put_u64(&mut b, log_pos);
put_u64(&mut b, book.orders.len() as u64);
for (id, (side, price, qty, src)) in &book.orders {
put_u64(&mut b, *id);
b.push(if *side == Side::Buy { 0 } else { 1 });
put_i64(&mut b, *price);
put_u64(&mut b, *qty);
b.push(*src);
}
fs::write(path, &b).unwrap(); // prod: write tmp, fsync, rename over — a reader sees old or new, never half (Q3)
}
/// Read a snapshot back: rebuild the book, return it plus the log position to resume from.
pub fn load_snapshot(path: &Path) -> (Book, u64) {
let b = fs::read(path).unwrap();
let mut book = Book { seq: get_u64(&b, 0), ..Default::default() }; // like { ...emptyBook, seq }: every field not named starts empty
let (log_pos, n) = (get_u64(&b, 8), get_u64(&b, 16));
let mut o = 24usize;
for _ in 0..n {
// A snapshot entry is laid out byte-for-byte like a v2 Place payload
// (26 bytes), so the existing codec reads it for free — one layout, two files.
let ev = decode(T_PLACE, 2, &b[o..o + 26]);
if let Event::Place { id, side, price, qty, source } = ev {
book.orders.insert(id, (side, price, qty, source));
*book.level(side).entry(price).or_insert(0) += qty;
}
o += 26;
}
(book, log_pos)
}
// ------- deterministic workload -------
// Lcg is a seeded pseudo-random generator: one u64 of state, scrambled on
// each call. Same seed, same sequence — seed it with 42 and it emits the same
// "random" workload on every machine, every run, which is
// why your output below will match this book's byte for byte. (LCG = linear
// congruential generator; no rand crate needed.)
struct Lcg(u64);
impl Lcg {
// Next pseudo-random u64. wrapping_mul/_add let the math overflow and wrap
// mod 2^64 on purpose — the wrap IS the scramble; `>> 33` keeps the
// better-mixed high bits.
fn next(&mut self) -> u64 { self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407); self.0 >> 33 }
}
// Roll one random event: ~60% places, ~20% cancels, ~20% executes (cancels and
// executes may name ids that no longer exist — fine, reduce() shrugs those off).
fn gen_event(rng: &mut Lcg, next_id: &mut u64, source: u8) -> Event {
match rng.next() % 10 { // picks 0-9; `0..=5` is a range pattern — matches 0 through 5
0..=5 => { let id = *next_id; *next_id += 1; Event::Place {
id, side: if rng.next() % 2 == 0 { Side::Buy } else { Side::Sell },
price: 10_000 + (rng.next() % 200) as i64 - 100, qty: 1 + rng.next() % 50, source } }
6..=7 => Event::Cancel { id: rng.next() % (*next_id).max(1) },
_ => Event::Execute { id: rng.next() % (*next_id).max(1), qty: 1 + rng.next() % 10 },
}
}
// ---------------- the five phases ----------------
// Runs the five phases in order; every PASS is an assert_eq! that aborts on mismatch.
fn main() {
let dir = std::env::temp_dir().join("lab-upgrade");
fs::create_dir_all(&dir).unwrap();
let log_path = dir.join("book.evlog");
let snap_path = dir.join("book.snap");
let _ = fs::remove_file(&log_path); // start clean; `let _` ignores "file didn't exist"
// Open in append mode, creating if missing — fs.createWriteStream(path, {flags: "a"}):
// every write lands at the current end of the file, after whatever is already there.
let mut log = fs::OpenOptions::new().create(true).append(true).open(&log_path).unwrap();
let mut rng = Lcg(42);
let mut next_id = 1u64;
// Phase 1: process A (v1 binary) writes 1000 v1 records, folding as it goes.
let mut a = Book::default();
for _ in 0..1000 {
let ev = gen_event(&mut rng, &mut next_id, SRC_UNKNOWN); // v1 has no source concept
append(&mut log, &ev, 1);
a.apply(&ev);
}
// flush() on a raw File is actually a no-op: there is no userspace buffer to
// drain (that's a BufWriter thing) — every write() above already went to the
// OS, so readers already see the bytes. Surviving power loss is a separate
// promise: that needs sync_all() (fsync), deliberately skipped here — ch13
// covers when the log must be synced.
log.flush().unwrap();
println!("phase 1: v1 writer seq={:>4} hash={:016x}", a.seq, a.hash());
// Phase 2: v2 binary cold-replays the v1 log through the upcaster.
let mut b = Book::default();
let mut b_pos = replay_from(&log_path, &mut b, 0);
println!("phase 2: v2 replay seq={:>4} hash={:016x}", b.seq, b.hash());
assert_eq!(a.hash(), b.hash());
println!("phase 2: DETERMINISM CHECK PASS (v2 binary == v1 state on v1 log)");
// Phase 3: snapshot B, reload, verify roundtrip.
write_snapshot(&snap_path, &b, b_pos);
let (c, c_pos) = load_snapshot(&snap_path);
assert_eq!(b.hash(), c.hash());
assert_eq!(b_pos, c_pos);
println!("phase 3: SNAPSHOT ROUNDTRIP PASS (snap at seq={} pos={})", c.seq, c_pos);
// Phase 4: hot cutover. A keeps writing v1; B tails the log live; at the
// handover boundary the hashes must match, then B takes over writing v2.
for i in 0..500 {
let ev = gen_event(&mut rng, &mut next_id, SRC_UNKNOWN);
append(&mut log, &ev, 1);
a.apply(&ev);
if i % 100 == 99 { log.flush().unwrap(); b_pos = replay_from(&log_path, &mut b, b_pos); }
}
log.flush().unwrap();
let _ = replay_from(&log_path, &mut b, b_pos); // B reaches head
assert_eq!(a.hash(), b.hash());
println!("phase 4: HANDOVER PASS at seq={} hash={:016x} (A fenced, B leads)", b.seq, b.hash());
for _ in 0..250 { // B is primary now, writing v2 records with a real source tag
let ev = gen_event(&mut rng, &mut next_id, 2);
append(&mut log, &ev, 2);
b.apply(&ev);
}
log.flush().unwrap();
println!("phase 4: v2 leader seq={:>4} hash={:016x}", b.seq, b.hash());
// Phase 5: fresh cold replay of the MIXED v1+v2 log must equal B exactly.
let mut d = Book::default();
replay_from(&log_path, &mut d, 0);
assert_eq!(b.hash(), d.hash());
println!("phase 5: MIXED-LOG REPLAY PASS seq={} hash={:016x}", d.seq, d.hash());
println!("all phases PASS (log: {})", log_path.display());
}Run it
cargo run --release
Expected output (byte-for-byte reproducible — LCG seed 42, fixed-seed FNV hash):
phase 1: v1 writer seq=1000 hash=3067adf006947f53
phase 2: v2 replay seq=1000 hash=3067adf006947f53
phase 2: DETERMINISM CHECK PASS (v2 binary == v1 state on v1 log)
phase 3: SNAPSHOT ROUNDTRIP PASS (snap at seq=1000 pos=27389)
phase 4: HANDOVER PASS at seq=1500 hash=5d27565fba06ac86 (A fenced, B leads)
phase 4: v2 leader seq=1750 hash=daf16d78feb8e7ad
phase 5: MIXED-LOG REPLAY PASS seq=1750 hash=daf16d78feb8e7ad
all phases PASS (log: /tmp/lab-upgrade/book.evlog)
Your hashes will match these because every source of nondeterminism was designed out — seeded LCG instead of thread_rng, BTreeMap instead of HashMap, fixed-seed FNV instead of DefaultHasher, integers instead of floats, time absent entirely. That exact reproducibility is the lesson the exercise exists to teach.
What each phase proves, and where it maps to production
Phase 1 — one event, two destinations. Log first; state is derived, never authoritative.
ENGINE A (v1 writer) │ DISK
│ ◄─ disk (fs write) boundary
gen_event ──► Event │
│ │ │
│ └─(1) encode(ev,1) ─► [len|ty|ver=1|payload] strip
│ (3) a.apply(&ev) │ │ (2) append
▼ │ ▼
A's in-memory Book │ book.evlog [v1][v1][v1]…
│ (4) a.hash() │
▼ │
3067adf006947f53 │
Phase 2 — the determinism check.
DISK │ ENGINE B (fresh v2 instance)
◄─ disk (fs read) │ ◄─ upcast boundary: v1 bytes cross
boundary │ into the v2 model inside decode
book.evlog │
[v1][v1][v1]… ──(1) replay_from(path, &mut b, 0)
each strip ──────►(2) decode(ty, ver=1, payload)
│ │ ver < 2 → source = SRC_UNKNOWN
│ ▼
│ Event { …, source: 0 }
│ │ (3) b.apply(&ev)
│ ▼
│ B's Book ──(4) b.hash()
(5) assert_eq!(a.hash(), b.hash()) → DETERMINISM CHECK PASS
The PASS holds because the upcaster (decode, one if ver >= 2 line) defaults the new field to a value that reproduces v1 behavior, and the fold never branches on anything outside the event stream. Production equivalent: the pre-deploy replay gate of ch13/ch17. Note where the versions live: the engine model has no V1 type — v1 exists for one line of the codec, at the marked boundary.
Phase 3 — snapshot = state + cursor.
ENGINE B │ DISK
│ ◄─ disk (fs write) boundary
B's Book ──(1) write_snapshot ─► book.snap
+ b_pos (cursor) │ [seq][log_pos][orders…]
│ │ cursor: pairs the state
│ │ with a byte offset
│ ▼
│ book.evlog [v1]…[v1]▌◄ pos 27389 (head)
fresh Book c, c_pos ◄──(2) load_snapshot ── book.snap
│
(3) assert c.hash()==b.hash() && c_pos==b_pos → ROUNDTRIP PASS
recovery = load_snapshot, then replay_from(c_pos) for the rest
The comment about tmp+fsync+rename in write_snapshot is the production delta (ch13).
Phase 4 — hot cutover.
ENGINE A (v1, leader) │ DISK: book.evlog │ ENGINE B (v2, follower)
─── process boundary (in this lab: two instances, one binary) ───
(1) append(ev,1) ───► [v1] │ time ↓
a.apply(&ev) [v1] │
│ [v1] ◄──(2) replay_from(b_pos): tail;
│ [v1] cursor advances strip by strip;
▼ [v1] torn tail at head → break, park
… 500 v1 records … ─► [v1] │
[v1] ◄──(3) replay_from: B reaches head
│ │
(4) hash gate: assert_eq!(a.hash(), b.hash()) → HANDOVER PASS
(5) A FENCED ✕ │ │
lane ends — A │ │ B leads now:
never writes [v2] ◄──(6) append(ev,2), b.apply(&ev)
again [v2] │
[v2] … │
the WRITE ROLE crossed the process boundary: same file, new writer,
new wire version — B could read v2 (phase 2) before anyone wrote it
The handover assert is the go/no-go gate from the runbook of ch16: leadership flips only when the follower’s hash matches the leader’s at the boundary sequence. Then B writes v2 into the same log — the readers-first choreography of ch14 compressed into one file.
Phase 5 — mixed-log replay.
DISK: book.evlog (two dialects) │ ENGINE D (fresh Book)
│
[v1][v1]… 1500 v1 strips …[v1]║[v2]… 250 v2 strips …[v2]
version boundary ─┘ (ver flips 1→2 mid-file)
│ │
(1) replay_from(path, &mut d, 0) — one pass, no mode switch
└──── every strip ──────────►(2) decode:
│ ver 1 → upcast source=0
│ ver 2 → read source byte
│ │ (3) d.apply(&ev)
│ ▼
│ D's Book ──(4) d.hash()
(5) assert_eq!(b.hash(), d.hash()) → MIXED-LOG REPLAY PASS
This is the property that makes rollback horizons and long retention livable: any conforming binary, at any later date, can rebuild any historical state from the heterogeneous log.
Two deliberate details worth noticing: replay_from’s torn-tail break (a half-written record at the file end is waited on, not an error — that’s what makes the same loop serve live tailing), and reduce’s unknown-id no-op (cancels/executes for absent orders are deterministic no-ops, which keeps the generated workload — and real-world duplicate-message replay — idempotent).
Extensions if you have another hour
- Make it actually two processes: split into a lib plus two bins (
writer,follower) sharing the file; the follower pollsreplay_fromin a loop and prints its hash; kill the writer and promote for real. The code already supports it. - Break determinism on purpose: swap
ordersto aHashMapand iterate it inhash(). Watch phase 2 pass in-process but fail across two separate processes (per-process SipHash keys) — the exact reason ch13 insists dual-replay tests run in separate processes. - Add a v3 that widens
qtysemantics or adds a second field via the reserved-byte trick from ch14, chaining upcasters v1→v2→v3. - Add a
crc32to the record header and corrupt a byte mid-file withdd; makereplay_fromstop at the corruption and report the last good(seq, pos).
Plain-English recap
- The lab is a mini ledger plus a read model, with a versioned serialization
format. Events are journal entries; the
Bookis the derived balance; the wire codec is your API’s payload schema. - Phase 2 is “new service, old archive, identical balances” — the replay-regression gate from ch17, in miniature.
- Phase 3’s
(snapshot, log_pos)pair is Kafka consumer-offset checkpointing — materialized state plus its cursor, the same reason every stream consumer persists its offset. - Phase 4 is promoting a replica after lag reaches zero and checksums match. Only after promotion does the new format flow — the readers-first choreography of ch14 in one file (Q6 shows where the lab deliberately cheats).
- Phase 5 is a webhook archive with old and new payload shapes both parseable forever — the property that makes long retention, audit replay, and rollback horizons livable.
- The two “deliberate details” are your at-least-once instincts. The
torn-tail
breaktreats a half-written record as “wait for more,” not an error; the unknown-id no-op makes duplicate or stale cancels/executes idempotent — the idempotency-key reflex, applied to a book.
Interview narration
How you tell this in an interview, sixty seconds, first person: “I keep the engine as a pure fold over a length-and-version-prefixed record log. When I ship a schema change I never touch the old decoder — the new field is appended on the wire, and a one-line upcaster defaults it so old records reproduce old behavior exactly. My deploy gate is mechanical: the candidate binary replays the production log and must hash-match the running engine’s state at the same sequence number — same fixed-seed structural hash both sides. For the cutover itself, the new version runs as a follower tailing the live log; when it’s at head and hashes match at a boundary sequence, leadership flips, the old process is fenced, and the new one continues writing the new version into the same log. And because replay tolerates mixed-version logs, rollback and audit replay keep working across the upgrade — I’ve got a 250-line toy that demonstrates the whole cycle end-to-end, and the same shape scaled to my production engine with its hot standby.” Every clause of that paragraph is a line of code you just ran.
Interviewer will ask
Q1: “Why default the new field to SRC_UNKNOWN = 0 instead of something meaningful?”
Because the upcast default must make v2 semantics degenerate to v1 semantics — replaying old logs must reproduce the state that actually existed, and any “smart” default (inferring source from order id ranges, say) is a silent history rewrite. Zero-as-legacy also composes with fixed-layout formats where old writers emit zeroed padding (the reserved-byte trick of ch14).
Q2: “Your handover compares hashes once at the boundary. Is that enough?” For the toy, yes; in production you compare rolling hashes continuously during the whole shadow period (every N sequence numbers), because a single end-point match can mask transient divergence that happened to cancel out, and because you want divergence to page you hours before a cutover, not fail the gate at T-0. You’d also gate on the follower being at head within a lag bound, not just matched at some past boundary.
Q3: “What’s missing from this snapshot for a real engine?” Atomic write, unpacked: write to a tmp file, fsync it, then rename over the real name — rename is atomic on POSIX, so a reader sees old or new, never half. Then fsync the directory too, because the rename lives in the directory’s own data — skip it and a crash can forget the file was ever renamed (ch13). Plus a checksum, a format version field, and K retained generations so a corrupt latest falls back to the previous one. And content beyond open orders: session-level state like venue sequence numbers, active timers, in-force config — anything the fold reads. Also note this snapshot cheats pleasantly: it rebuilds price levels from orders on load, trading load time for snapshot simplicity — a legitimate trade you should be able to defend either way.
Q4: “What happens if the process dies mid-append?”
The log ends in a torn record; replay_from refuses to apply it (length check) and reports the last durable position — recovery resumes from complete records. In production you’d add a per-record checksum so a torn or bit-rotted record is distinguishable from a clean tail, and the writer would fsync on a policy (per event, per batch, or rely on the replicated standby as the durability story — a latency/durability trade you should name explicitly).
Q5: “Why is the hash function hand-rolled FNV instead of DefaultHasher?”
DefaultHasher is SipHash with a per-process random key — hashes are incomparable across processes, and comparing across processes is exactly what this hash is for (primary vs standby, old binary vs new). Any fixed-seed hash works; the requirements are canonical field order (hence BTreeMap traversal), fixed serialization, and covering all state the fold can produce while excluding incidentals like capacity.
Q6: “The v2 leader starts writing v2 records immediately after taking over. Is that safe?” In the lab, yes, because there’s no rollback consumer; in production it violates the rollback horizon — the fenced v1 binary can’t parse v2 records, so if you rolled back you’d strand the tail. The discipline of ch16: new leader keeps writing v1-compatible records through the bake period and flips write-format later by config. Being able to point at the exact line of the lab that’s “wrong on purpose” is a strong interview move.
Further reading
- Chapters 13, 14, and 16 of this book — every line of the lab is one of their patterns in miniature; re-read them with the code open.
- Greg Young, Versioning in an Event Sourced System — the upcaster pattern this lab implements, with the enterprise-scale edge cases.
- Aeron Cluster documentation — compare its snapshot + log-replay + leadership-transfer lifecycle to phases 3–4; the shape is identical at production scale.
- Kleppmann, DDIA ch. 4 and ch. 11 — encoding evolution and log-centric state, the two theories this lab welds together.
Where this goes next: Part III is complete — Chapter 19 opens Part IV, which turns the whole book into interview reps: question banks with model answers, starting with networking.
You Are the Venue: Exchange Architecture
Before you start. This chapter assumes event sourcing / snapshot + replay (state as a log of events, rebuildable from any snapshot, ch13), deterministic single-writer (one thread owns the state; same inputs → same outputs, ch13), TCP vs UDP trade-offs (ch02), clocks and timestamping (why “who was first” is a measurement problem, ch07), latency percentiles and door-to-ack measurement (ch08), and the CLOB / price-time priority basics from ch00f. If any are new, read those first.
You’ve spent Parts 0–III on one side of the wire: your process races to hear the venue and reply faster than everyone else. This chapter flips the table. You are now the venue. Every trick you learned as a taker — timestamping, feed gaps, order acks, rate limits — has a mirror image on this side, and the mirror image is usually harder, because the venue’s problems are everyone’s-flow problems, not your-flow problems.
Here’s the good news: you’ve already built the hardest single component. Your Crypto.com matching engine — deterministic single-writer, price-time priority, event-sourced with snapshot/replay, hot-standby — is the center box of this diagram. What you haven’t built is everything around it, and the surround is what makes a matching engine into an exchange of record.
The canonical architecture
THE MODERN EXCHANGE, END TO END
clients edge the heart deterministic consumers
──────── ───────────── ───────────────── ──────────────────────────────
FIX ───┐ ┌───────────┐
OUCH ──┼──► │ order │
WS ────┘ │ gateway │──┐
└───────────┘ │ ┌─────────────┐ ┌────────────────────┐
┌───────────┐ ├──────►│ SEQUENCER │───┬───►│ matching engine(s) │──► acks/fills
FIX ───────► │ gateway │──┤ │ (assigns │ │ │ (per-symbol shard)│ (back out via
└───────────┘ │ │ global seq │ │ └────────────────────┘ gateways)
┌───────────┐ │ │ number to │ │ ┌────────────────────┐
WS ────────► │ gateway │──┘ │ EVERYTHING)│ ├───►│ market-data │──► L1/L2/L3 feeds
└───────────┘ └──────┬──────┘ │ │ publishers │
│ │ └────────────────────┘
▼ │ ┌────────────────────┐
┌────────────┐ ├───►│ drop copy / │──► clearing,
│ EVENT LOG │ │ │ clearing feed │ brokers' risk
│ (sequenced,│ │ └────────────────────┘
│ durable, │ │ ┌────────────────────┐
│ replayable│ └───►│ surveillance / │──► regulator,
└────────────┘ │ regulatory capture │ audit trail
└────────────────────┘
Read it left to right and notice the shape: many chaotic inputs → one total order → many deterministic outputs. Everything left of the sequencer is concurrent, racy, and unfair by nature (packets arrive when they arrive). Everything right of it is a pure function of the sequenced stream. The entire architecture exists to make that boundary as early, as fast, and as defensible as possible.
The sequencer: partition-0 for the whole market
Start with the analogy from your world: a Kafka topic with exactly one partition. Recall what that means — Kafka is an append-only log, and each message’s offset is its position number in that log; one partition means one single, total order that every consumer sees identically. The sequencer is that idea promoted to be the market’s backbone: one component that stamps a global, gap-free sequence number on every inbound message — orders, cancels, admin events, even clock ticks. (Why time itself is a message: the determinism checklist below.) It is the single place where “what happened, in what order” is decided for the whole market; the partition offset is market truth. And every other property of an exchange — fairness, replayability, recovery, audit — is downstream of that one number.
You built the consumer side of this at Crypto.com: your matching engine was a deterministic single-writer that applied events in order. The sequencer answers the question you never had to ask: who decides the order? In your system, the order was whatever your single writer happened to dequeue. Fine for one application. Not fine for a market, because at a venue the ordering is the product — firms paid for colocation to fight over it. The sequencer settles that fight; everything after it is bookkeeping.
Why this design won over, say, a cluster of matching engines with distributed coordination:
- Determinism. Same sequenced input stream → same books, same fills, same feed, bit for bit. You know this property from your engine; the sequencer extends it from one component to the entire venue. The market-data publisher, the clearing feed, and surveillance never talk to the matching engine — they consume the same log and derive consistent state independently. No cross-service RPC on the hot path, no “did the feed and the fills disagree” incidents.
- Fairness you can defend. “Order A matched before order B because A got sequence 4,412,907 and B got 4,412,908” is an answer you can give a regulator or an angry HFT firm. “A won a mutex race inside engine shard 3” is not.
- Trivial recovery. A downstream component crashes → restart, load a snapshot, replay the tail from its last applied sequence number — exactly your snapshot/replay design (ch13). No distributed reconciliation.
- One number tells you everything under load. Lag = sequencer head minus consumer position. You’ve run Kafka consumers; same dashboard.
The cost: the sequencer is a single point of serialization — every message in the market funnels through one code path. That sounds insane until you do the arithmetic. A tight sequencer does almost nothing per message: validate framing, stamp number + timestamp, append to the log, hand off. Call it 100–200 ns per message on one core → ~5M messages/sec on a single thread — and NASDAQ’s entire equity market peaks in the low tens of millions of messages/sec across all instruments. One well-written thread genuinely covers most markets, which is why this “obviously unscalable” design runs the world’s exchanges. LMAX, a UK exchange, made the argument famous; Aeron, a trading-messaging library, and modern exchange stacks industrialized it.
Here is what the sequenced log physically is, since “append to the log” is doing a lot of work in that sentence. (mmap’d = the file mapped straight into the process’s memory, so appending is just a memory write — no write() syscall per record.)
sequencer memory: [ mmap'd segment files, append-only, fixed-size records ]
│ written sequentially (the disk-friendly pattern
│ from ch00f — same physics as Postgres WAL /
│ Kafka segments)
├──► shipped to standby/quorum (replication, below)
└──► consumed by engines/publishers via shared-memory
ring buffers or reliable multicast — consumers
POLL forward through it; the sequencer never
waits for any consumer
Two properties matter. Sequential append is the one disk pattern that keeps up with the message rate (your WAL intuition transfers directly), and the sequencer never blocks on consumers — a lagging engine or publisher falls behind in the log and catches up; backpressure toward the sequencer would let the slowest component in the building set the market’s pace. If a consumer falls off the retained window (the log keeps only recent history in fast storage; older segments age out to archives), that’s an incident for that consumer (snapshot + replay to recover), never a brake on the market. Hold this asymmetry; it returns with teeth in ch25’s slow-consumer problem.
Where the matching engine sits: sharding by symbol
One matching engine process per instrument shard — a group of symbols one engine owns exclusively:
sequenced stream ──► demux by symbol ──► [engine shard 1: BTC-USD, BTC-PERP]
[engine shard 2: ETH-*, SOL-*]
[engine shard 3: long tail, 3000 symbols]
The partitioning is legitimate because no cross-symbol ordering guarantee is needed: an order on AAPL and an order on MSFT never interact inside a book, so they can match in parallel without violating price-time priority. (Cross-symbol products — futures spreads, implied liquidity across a curve — are the exception; venues offering them co-locate those legs on one shard or accept real complexity. Know the caveat; it’s a favorite interview follow-up.)
The operational problem is the hot symbol. Load isn’t uniform: on a big day one instrument can be 40% of all market messages. It’s the noisy tenant on your multi-tenant Postgres host — the one whose table gets all the writes, where “add more tenants per box” solves nothing. And here the usual escape hatch is welded shut: you cannot sub-shard a single order book, because price-time priority demands a single writer per book. That leaves exactly three levers:
- make the hot engine itself faster;
- give it a dedicated core or host;
- evacuate every other symbol off its shard.
Venue capacity planning is substantially “which symbol goes hot after the next listing or news print, and is its shard ready.”
Determinism and fairness as product features
Two clients send an order in the “same microsecond.” Who wins? At a venue the answer must be: whoever’s message was sequenced first — and arrival at the sequencer is the definition of first. Not gateway receive time, not client send time: sequence number. Everything else is evidence about the ordering, not the ordering itself.
That sounds circular until you see what it buys:
- Gateways timestamp on ingress (hardware timestamps where the venue is serious, ch07) so the venue can demonstrate that sequencing tracked arrival — the timestamp is audit evidence, not the tiebreaker.
- Serious venues publish their fairness model: how gateways feed the sequencer, whether gateway→sequencer paths are latency-equalized, what happens on ties. CME, Eurex, NASDAQ all document this publicly — because sophisticated clients (your former self, running an SOR) will reverse-engineer it empirically anyway, and a fairness model clients discover before you disclose it is a scandal in waiting.
- The sequenced log doubles as the regulatory audit trail. When the regulator asks “reconstruct 14:30:00–14:30:10 on the day of the flash event,” the venue replays the log and produces exact book state at any sequence number. It’s your engine’s event-sourced replay, except the output is legal evidence. That’s why “deterministic consumers of a durable log” is near-mandatory rather than merely elegant: CAT (the Consolidated Audit Trail, the US regulator’s every-order database) in US equities, MiFID II record-keeping in Europe.
Fairness, in short, is a line item clients pay for and regulators examine. Your matching engine had determinism for correctness; a venue has it for defensibility.
Sequencer failover: the genuinely hard problem
Here’s where venue-side is harder than what you built. Your hot-standby followed the primary’s event stream and could take over. Now ask the venue-grade question: when the primary dies, can you prove no acknowledged order was lost?
The trap: primary sequences message N, sends the ack, crashes before the standby saw N. Standby takes over at N−1. The client holds an ack for an order the new primary has never heard of. For an exchange of record this isn’t a bug, it’s an existential event — the ack is a legal commitment.
The two production-grade answers:
Option A: primary/standby, synchronous replication
┌─────────┐ seq N ┌─────────┐
│ primary │─────────►│ standby │ rule: the client ack for N goes out
│ │◄─────────│ (ack) │ ONLY after the standby confirms N
└────┬────┘ └─────────┘
└──► client ack (after standby ack)
Option B: Raft-style consensus cluster (e.g. Aeron Cluster)
┌────┐ ┌────┐ ┌────┐ a message is "sequenced" when a majority
│ n1 │ │ n2 │ │ n3 │ holds it in their log; leader failover is
└────┘ └────┘ └────┘ automatic; ack only after commit
Both make the same trade: an acknowledged message exists on ≥2 machines before the ack leaves the building. That synchronous hop sits inside your door-to-ack path — a first-class citizen of the latency budget, and the reason sequencer nodes share a low-latency fabric (a dedicated private network between the nodes). “We async-replicate and accept a tiny loss window” — a perfectly reasonable call in most systems you’ve shipped — is off the table when the ack is a contract. If your Crypto.com hot-standby was async (most application-layer ones are), that’s the gap between what you built and venue grade — name it crisply.
Aeron Cluster is the open-source embodiment of Option B: Raft — the standard recipe for leader election plus majority-ack replication — driving a replicated log, with deterministic state machines on top. Several production crypto and FX venues run on it or on the same design; know it by name.
The other consumers: drop copy, clearing, surveillance
The right-hand column of the big diagram has two boxes you never touched as a client but will be asked about at any broker-platform or exchange interview:
Drop copy. Picture a webhook fanout where every payment.settled event also goes to a second, audit-owned endpoint that the merchant’s finance team controls. Drop copy is that for trading: a real-time copy of an account’s execution reports (and often order events), delivered to a different session than the one doing the trading. Why it exists: brokers and clearing firms are on the hook for their clients’ risk. ch24’s credit checks are pre-trade; drop copy is how the risk desk watches post-trade in real time. It’s also how a firm’s own independent risk system cross-checks what its trading system believes. Implementation is nearly free in this architecture: a drop-copy session is one more filtered projection of the sequenced log — filter by account, serialize as execution reports, deliver on a FIX session. No new source of truth, so it cannot disagree with the fills.
Clearing feed — the post-trade stream to the clearing house / settlement layer: matched trades with counterparties, quantities, prices — the thing that turns “the engine printed a fill” into “money and assets actually move.” At a crypto venue with internal custody this loop is short; in tradfi it’s an external institution (DTCC, CME Clearing) with its own formats and its own timeliness rules.
Surveillance / regulatory capture — a consumer that stores everything and runs pattern detection over it: spoofing (fake orders cancelled before they trade), layering (spoofing stacked at several price levels), wash trades (trading with yourself to fake volume), marking the close (pushing the official closing price with last-minute orders). Two design notes worth having: it must consume the full sequenced log, not a summarized feed, because manipulation lives in the order events that never trade (a spoofer’s signature is orders placed and cancelled — invisible in a trades-only view); and it’s the one consumer where falling behind is tolerable — surveillance can lag minutes without harm, so it runs on cheap batch-friendly infrastructure, while the market-data publisher lags microseconds at most. Same log, wildly different consumer SLAs — a nice concrete instance of the architecture’s flexibility.
Determinism gotchas: what actually breaks replay
You know these from building your engine, but the venue interview version wants them as a checklist, because every consumer of the sequenced log must obey them, not just the engine:
- Wall-clock reads in logic. Any
now()inside a decision path breaks replay. Time must arrive as sequenced events (the sequencer stamps a timestamp into each message; timers become injected tick events). Your engine’s timers were driven by event time, not machine time — same rule, venue-wide. - Hash-map iteration order. Iterating an unordered map to, say, expire orders produces machine-dependent order. Sorted structures or insertion-ordered containers only, anywhere order can leak into output.
- Floating point. Cross-platform/compiler FP differences are tiny but nonzero; venues use integer ticks and fixed-point (ch00f) so replay is bit-exact. You did this in your engine; here it’s non-negotiable because the regulatory replay must match production.
- Threads inside a consumer. Parallelism inside one deterministic consumer reintroduces racing. Parallelism lives between consumers (shards, projections), never inside one.
- Randomness and uninitialized memory. Any RNG must be seeded from the log; any uninitialized read is a latent divergence bomb that detonates weeks later on the standby.
The test that keeps you honest: continuously replay production’s log on a shadow instance and diff state hashes at checkpoints. Divergence pages someone. This is your snapshot/replay regression testing promoted to a permanent production invariant.
Auditing an execution: explain <order_id>
Here is what all that determinism discipline buys. A client (or a regulator) asks: “why did my order fill at that price, against that counterparty, at that moment?” At a web company this question triggers log-spelunking and a shrug. At a venue it triggers a query, because the venue is a deterministic fold over the sequenced log — the audit is a replay.
The procedure, mechanical from end to end:
- Find the order’s ingress chain. Gateway receipt (hardware timestamp, session, account), the risk-gate verdict, and the sequence number the sequencer stamped — say seq N. Every hop stamped its passage; the gaps between stamps are evidence too (they show who delayed what).
- Rebuild the world as of seq N−1. Load the nearest snapshot at or before N−1, replay the tail up to N−1. You now hold the exact book the order walked into: every resting order, in exact queue position, each tagged with the sequence number it arrived at.
- Re-run the match. Feed event N to the same engine version. Determinism guarantees the same fills fall out — and now the “why” reads straight off the state: “best ask was 10001 with order Y resting first (arrived seq M, never modified — a modify would have sent it to the back of the queue); X was a marketable buy for 500; price-time priority filled Y’s 300, then Z’s 200 at the next level.” Every clause points at a sequence number. No opinions — arithmetic.
- Version-stamp the rules. The replay proves the decision only if it runs the same rules: engine version and matching config are themselves events in the log (the change-management chapter’s config-as-events, ch17), so the answer includes “matched under rules vX, config as of seq K.”
Build it as a tool, not a runbook: explain <order_id> locates the seq range, loads the snapshot, replays, and emits the human narrative plus the machine dump. The same replay engine powers surveillance queries and incident forensics — you don’t build audit infrastructure; you build determinism plus snapshots once, and audit falls out as a query. And this is why a diverging replay is not a quality bug but a compliance incident: a replay that diverges is an audit that proves nothing. The shadow-replica hash check above is your continuous proof that the audit machinery still works.
Two supporting pieces close the loop. Drop copy (previous section) answers “what is happening right now” for watchers outside the system. The log itself goes to write-once retained storage — regulators demand years of retention — with a hash chain over the records so tampering is evident. Drop copy is the live witness; the log is the court record.
Snapshots: what, when, how — without stopping the market
The snapshot mechanics are the event-sourcing chapter’s (ch13), applied venue-scale. What goes in follows one generative rule — everything the fold reads, nothing it derives:
- per-symbol books with resting orders in queue order (queue position is the fairness product; a snapshot that loses it is worthless),
- account and risk-counter state (the pre-trade gate’s memory),
- both sequence spaces (engine seq and feed seq — the two counters the mini-market lab makes concrete),
- config epoch, engine version, and open auction state if snapshotted mid-auction.
When: every N events or T seconds per shard, always at a sequence boundary — the label snapshot.{seq} means “state exactly as of seq N, nothing mid-event,” and that label is what makes step 2 of the audit legal.
How, without a pause the latency distribution would wear: three standard mechanisms, chosen per shard. Fork the process at a boundary and let the child serialize while the parent keeps matching (the OS’s copy-on-write shares pages until the parent writes one). Or keep the book as a persistent structure and hand the snapshotter the old root pointer — a git commit, while the writer moves on. Or, for cold symbols, micro-quiesce: hold intake for the microseconds a double-buffer swap takes. Durability is ch13’s ritual verbatim: write to a temp file, fsync, rename to snapshot.{seq}, fsync the directory, checksum, keep K generations.
The closure worth saying in an interview: snapshot + log tail is one mechanism serving three masters — crash recovery, failover (ch16), and the audit entry point above. That triple duty is why venues treat snapshot cadence as a product decision, not an ops afterthought: it bounds recovery time and bounds how long explain takes to answer.
Throughput shape: you now receive everyone’s flow
As an SOR operator you sent orders — your flow, your rate. The venue receives the sum of all participants, and the sum has a brutal shape:
- Steady state is a lie. Opens, closes, economic prints (scheduled data releases — CPI, payrolls — hitting the market), and liquidation cascades (one forced sale pushing the price into triggering the next — an avalanche of margin calls) produce 10–100× bursts over median load, concentrated into milliseconds. A venue provisioned for 2× median falls over exactly when being up matters most — and when it’s on the news.
- Concrete anchors: NASDAQ ITCH peaks in the tens of millions of messages/sec market-wide on volatile opens; a top crypto venue sees hundreds of thousands to millions of order-messages/sec at cascade peaks. Human translation: at 5M msgs/sec a message arrives every 200 ns — roughly one per L3 cache miss. Burst capacity, not average capacity, is the spec.
- Order-to-trade ratios of 20:1 to 100:1 mean the flow is overwhelmingly cancel/replace churn from market makers. The load profile is metadata churn, not fills — which is why messaging policies exist (ch24).
Latency numbers to hold
| Path | Door-to-ack (gateway in → ack out) |
|---|---|
| Your old client-side world (WS over internet to a cloud venue) | ~10 ms round trip |
| Decent cloud-hosted crypto venue, software path | ~50–500 µs |
| Serious colo venue, tuned software (kernel bypass, ch04) | ~10–50 µs |
| CME/NASDAQ class, hardware-assisted edge | sub-10 µs; wire-to-wire budgets in single-digit µs |
Human scale: the gap between your old 10 ms client-side world and CME’s sub-10 µs is three orders of magnitude — all of it the physics and architecture from Parts 0–I, applied on the receiving side.
The trading day has a shape: sessions and auctions
One more venue-side concept your continuous-trading crypto background skips: the market itself has states, and state transitions are the venue’s highest-stress moments.
pre-open ──► OPENING AUCTION ──► continuous trading ──► CLOSING AUCTION ──► closed
(orders (one batch (the CLOB you (one batch
accumulate, cross at a know) cross; sets
no matching) single price) official close)
An auction (call auction / uncrossing) is batch matching instead of streaming — the end-of-day job that nets a whole day of card transactions in one pass. Instead of matching continuously, the venue collects orders for a window, then computes the single price that maximizes matched volume and executes everyone crossable at that one price.
A worked uncrossing, to make “maximizes matched volume” concrete:
willing buyers (limit ≥ P): P=$100 → 600 sh P=$101 → 500 sh P=$102 → 300 sh
willing sellers (limit ≤ P): P=$100 → 200 sh P=$101 → 400 sh P=$102 → 700 sh
crossable = min(buy, sell): 200 400 ◄ max 300
→ $101 wins; 400 shares trade, everyone at that single price
Why the venue engineer cares — three stakes:
- The open is the burst. The accumulated overnight order flow hits the book at once — part of why bursts are 10–100×.
- It’s a second deterministic code path. The uncrossing algorithm is separate from continuous matching, but it must live inside the same sequenced-log discipline as everything else.
- The close is real money. The closing auction’s print (the executed trade published on the public feed) is the official close that trillions in index funds benchmark against — a correctness bug there reprices ETFs.
Crypto mostly trades 24/7 continuous, but even there, listings-day opens and post-halt reopens are auction-shaped problems (a mass of accumulated orders needing a fair single crossing), and venues that reopen a halted book straight into continuous matching produce the wild first-print artifacts you saw as a client.
Halts themselves — circuit breakers, per-symbol limit-up/limit-down pauses — are sequenced admin events like everything else: the halt, the quote-only window, the reopen auction all flow through the same log, so the audit trail of why the market stopped is as replayable as the trades.
Crypto-venue specifics (the outside view, confirmed from inside)
Things you observed as a 20-venue client that now make architectural sense:
- WebSocket gateways for both orders and data: TCP per client, JSON or bespoke binary, no multicast possible over the public internet. The fanout consequences are the whole story of the feed-publishing chapter (ch25).
- Rate limits per API key (you lived under these): the gateway protecting the sequencer’s inbound funnel, not arbitrary meanness — ch24.
- Matching engines behind cloud load balancers: some venues front the order path with a cloud LB. Serious venues don’t — an LB adds jitter (two identical clients get different paths, so fairness becomes indefensible), hides client identity from the edge, and inserts a hop the venue can’t timestamp or reason about. When you saw a venue’s ack latency go bimodal for a week, this class of middlebox was often why. As the venue, the rule is: nothing between the client and your timestamping gateway that you don’t control.
Anatomy of an ack: one order, door to door
Tie the whole chapter together by tracing a single marketable order through a good software-path venue (no FPGA), with the clock running:
t=0 order's last byte hits the gateway NIC (HW timestamp — the
"door" in door-to-ack, the clocks chapter)
t+2µs gateway: session lookup, decode, risk chain, token bucket,
stamp, forward (the gateway budget)
t+4µs sequencer: assigns seq 4,412,907; message is now "real"
t+6µs sync replication: standby/quorum confirms 4,412,907
◄── the ack is now LEGAL to send; nothing was allowed
to promise anything before this line
t+7µs engine shard (deterministic consumer): matches against the
book → fill events, themselves sequenced outputs
t+9µs publisher emits the book delta + trade on the feed;
gateway serializes the execution report back to the client
t+11µs ack/fill's first byte leaves the venue NIC — door-to-ack ≈ 11µs
Three things to notice, because they’re the chapter in miniature. First, where the point of no return sits: not at the engine, but at replication confirm — the order “happened” when it was durably sequenced, and matching is downstream bookkeeping (this is why a venue can honestly ack receipt before the match completes, and why acked-but-crashed is recoverable). Second, everything after the sequencer could run at different speeds without breaking correctness — if the publisher lags 50µs behind the engine, the feed is late but never wrong; ordering, not scheduling, is the invariant. Third, the budget’s big rocks are the replication RTT and the two NIC traversals — which is why sequencer fabric latency and kernel bypass (ch04) dominate venue tuning, and why the remaining software must live in the L1/L2 cache regime you learned in ch00a. Multiply this 11µs picture by “a message every 200ns at peak” and you have the venue’s entire performance problem on one page.
Plain-English recap
- The sequenced log is physically boring on purpose: append-only segments (WAL/Kafka physics), consumers poll forward, and the sequencer never waits for anyone — the slowest component in the building must never set the market’s pace.
- The point of no return is replication-confirm, not the match: an order “happened” when it was durably sequenced; matching, publishing, and clearing are all downstream bookkeeping that can lag without ever being wrong.
- An exchange is a funnel: many chaotic inputs → one component that decides the order of everything (the sequencer) → many independent consumers deriving state from that one ordered log. Kafka with a single partition, where the partition is the market.
- You already built the most famous consumer — the matching engine. The venue-shaped work is the funnel, the durable log, and proving the ordering was fair.
- The trading day has states — open auction, continuous, close auction, halts — all flowing through the same log as sequenced admin events; the close’s single print is what index funds benchmark against, so the uncrossing code path carries real-money correctness weight.
- Sharding is by symbol because AAPL and MSFT never interact; the hot-symbol problem is the noisy tenant on your multi-tenant Postgres box, except you can’t split the tenant’s table — a book demands a single writer.
- Fairness is a documented product feature, not an emergent property: sequence number is the tiebreaker, timestamps are the audit evidence, the log is what you hand the regulator.
- Failover’s hard rule: no acknowledged order may be lost — an ack leaves only after the message exists on two machines. Synchronous replication or Raft; the replication hop lives inside the ack-latency budget.
- Drop copy, clearing, and surveillance are just more projections of the same log — a drop-copy session is a filtered webhook fanout of an account’s events, and it can’t disagree with the fills because it has no independent source of truth.
- Determinism is a venue-wide discipline with a checklist: no wall-clock in logic, no unordered-map iteration into output, integer ticks not floats, no threads inside a consumer, no unseeded randomness — enforced by continuously replaying prod’s log on a shadow and diffing state hashes.
- Provision for the open and the liquidation cascade, not the average: 10–100× bursts are when a venue earns or torches its reputation.
Interviewer will ask
“You built a matching engine at Crypto.com — what’s the difference between that and running a venue?” “My engine was the center box of this chapter’s diagram — a deterministic single-writer consuming an ordered event stream. But I produced that stream myself, so the ordering only had to be internally consistent. A venue picks up three obligations my engine never had, and each one maps to a component. It must decide the order among competing external clients — that’s the sequencer, the one place ‘what happened, in what order’ is settled for the whole market. It must defend that decision to regulators and angry HFT firms — that’s latency-equalized gateways with hardware timestamps at ingress, plus the sequenced log doubling as the audit trail. And it must never lose an acknowledged order — that’s synchronous replication, the ack leaving only after the message exists on a second machine. So the engine is the famous component, but the venue-shaped work is the surround: the funnel, the durable log, the proof of fairness. I built the engine — and spent years on the client side probing exactly that surround, because my SOR empirically reverse-engineered venues’ fairness properties.”
“Why does a single sequencer scale? Isn’t a global serialization point a bottleneck?” “Start from the chapter’s picture: the sequencer is Kafka with one partition, and the partition’s per-message work is tiny — validate, stamp, append, hand off — call it 100–200 ns, so one core clears ~5M msgs/sec, which covers most entire markets. Now price the alternative. A total order is required for fairness within a book, so a distributed design still has to make one ordering decision per message — but consensus makes each decision with a network round trip between nodes, microseconds, where the single thread makes it with one memory write, nanoseconds. The ‘bottleneck’ is orders of magnitude faster than anything you’d replace it with. So you keep the ordering on one thread and scale the genuinely expensive work — matching, publishing — as deterministic consumers behind it, sharded by symbol since cross-symbol ordering isn’t needed. It’s the shape I ran in production: the single-partition log was never the bottleneck; the consumers were.”
“Two orders arrive in the same microsecond on different gateways. Who wins?” “Whichever is sequenced first — the sequence number is definitionally the answer. The real engineering question is whether gateway topology makes that fair: gateways must be interchangeable, paths to the sequencer latency-equalized, ingress hardware-timestamped so you can audit that sequencing tracked arrival. And you publish the model — clients like my former SOR will empirically reverse-engineer it anyway, so it had better be disclosed and defensible.”
“How do you fail over the sequencer without losing orders?” “The invariant is: no ack leaves until the message is durable on a second machine. Two shapes — lockstep primary/standby where the primary waits for standby confirmation before acking, or Raft-style like Aeron Cluster where ‘sequenced’ means majority-committed. My hot-standby at Crypto.com followed the primary’s stream but the primary acked before replication confirmed — acceptable for an internal system, not for an exchange of record where the ack is a contract. Venue-grade means eating the replication RTT inside the door-to-ack path.”
“How would you capacity-plan a new venue?” “For the burst, not the mean. Opens and liquidation cascades run 10–100× median load, concentrated into milliseconds, and 20–100:1 order-to-trade ratios mean most of it is maker cancel/replace churn — so a venue provisioned for 2× median falls over exactly when being up matters most. Inside that burst, the binding constraint is the hottest single shard: on a big day one symbol can be 40% of all messages, and you can’t sub-shard a book because price-time priority demands a single writer. Which means the plan is about that shard, not aggregate capacity — budget it for the worst credible burst, keep it on dedicated hardware, evacuate every other symbol off it. Then prove the headroom rather than assert it: replay captured real bursts through the sequenced log, which event sourcing gives you for free — the same replay discipline I used for my engine’s regression testing.”
“What happens if the matching engine and the public feed disagree?” “Run the incident. A market maker’s recon desk calls: their private execution report shows a fill at 14:31:07 that the public tape never printed. First call to make: which side is true? The engine’s fill — money moved, clearing saw it on drop copy — so the feed is what’s lying, and every client trading off it has been quoting against a false book since the divergence began. Immediate moves, in order: mark the feed suspect, force a snapshot republish so consumers resync to true state, then diff the publisher’s book against a reference replay of the sequenced log to find the first divergent sequence number. And that diff exposes something structural: in this architecture a divergence should be impossible — engine and feed are both deterministic consumers of the same sequenced log, so they can’t drift apart; a divergence at seq N means one of them computed a different state from identical input. That’s a determinism bug, not a synchronization bug — wall-clock leaking into logic, hash-map iteration order, uninitialized memory — and you find it by replaying the log through both consumers offline and bisecting to the first event where their state hashes split. Which is exactly why venues ban nondeterminism in anything downstream of the sequencer. From the client side I have caught venues whose private fills contradicted their public feed — that’s the signature of a venue not built this way, and it was a real input to my SOR’s venue-quality scoring.”
Further reading
- Martin Fowler, “The LMAX Architecture” (martinfowler.com, 2011) — the canonical write-up of single-threaded deterministic matching plus event sourcing; the intellectual ancestor of this chapter.
- Aeron Cluster documentation and Martin Thompson’s talk “Cluster Consensus: when Aeron met Raft” (QCon) — production Raft-replicated deterministic state machines for trading systems.
- NASDAQ TotalView-ITCH 5.0 and OUCH protocol specifications (nasdaqtrader.com) — read a real venue’s order-entry and feed contracts end to end; short documents, worth every page.
- Brian Nigito, “How to Build an Exchange” (Jane Street tech talk, on YouTube) — the best single hour on sequencer-centric exchange design, by a practitioner.
- CME Globex public documentation on matching algorithms and market-data channels (cmegroup.com) — how a tier-1 venue describes its own fairness model.
Where this goes next: ch24 zooms into the left edge of the diagram — the gateways where sessions, pre-trade risk, and fairness at the door actually happen.
Gateways, Sessions & Fairness at the Edge
Before you start. This chapter assumes the sequencer-centric exchange layout (gateways → sequencer → deterministic consumers, ch23), TCP session behavior (connections, retransmits, what a dropped connection means, ch02), the FIX / OUCH / ITCH jargon (ch00d), timestamping and clock discipline (ch07), and latency budgeting by component (ch08). If any are new, read those first.
For years you were on the outside of this door: twenty venues, twenty API-key schemes, twenty flavors of rate limit, twenty ways to get disconnected at the worst moment. This chapter is the view from inside the door. The order gateway is the venue’s edge tier — the only component that talks to untrusted parties — and its job description is a contradiction: be maximally paranoid (auth, risk, rate limits, abuse defense) while adding minimal and equal latency for everyone. Everything interesting about gateway design falls out of that tension.
The gateway’s job, end to end
ONE ORDER'S PATH THROUGH THE GATEWAY (budget: single-digit µs total)
client ──TCP──►┌──────────────────────────────────────────────────────────────┐
│ ① ingress timestamp (HW tstamp at NIC where serious) │
│ ② session layer (auth'd? seq numbers ok? HB ok?) │
│ ③ protocol translation (FIX 4.4 / OUCH / WS-JSON │
│ → internal fixed binary msg) │
│ ④ pre-trade risk, in-line: │
│ • fat-finger limits (qty/notional caps) ~100ns │
│ • price bands (limit px vs reference) ~100ns │
│ • self-match check tag (attach STP group id; ~50ns │
│ STP = self-match prevention, §below) │
│ • credit/position (per-account exposure) ~200ns │
│ ⑤ rate limiter (token bucket per session) ~50ns │
│ ⑥ stamp session id + gateway id + ingress time, │
│ forward to SEQUENCER │
└──────────────────────────────────────────────────────────────┘
│ ▲
▼ │
[ sequencer ] acks/fills routed back
to the owning session
Your mental model from the SaaS world is right, and one adjustment away: this is an API gateway + auth middleware + rate limiter in front of a monolith — except the “monolith” is one sequenced pipe for the whole market, and the p99 budget for the entire middleware stack is microseconds, not milliseconds. Every check in box ④ has an explicit per-check latency budget, the way you’d budget a Postgres query — but the budgets are in nanoseconds, which forces the implementation style: no allocation, no locks, no syscalls, all limits pre-loaded into gateway-local memory and updated out-of-band.
Walk the boxes:
② Session auth. In FIX land: a Logon message opens a session identified by (SenderCompID, TargetCompID), authenticated by credentials and increasingly by TLS or network identity (in colo, “you’re on port 7 of my switch” is part of auth). In crypto land: API key + HMAC signature per request or per connection — the scheme you implemented twenty times as a client. Either way the gateway resolves connection → account → risk profile once at logon and caches it, because re-resolving per order costs a lookup you can’t afford.
③ Protocol translation. Clients speak FIX (verbose, tag=value, self-describing), OUCH (NASDAQ’s lean fixed-width binary order protocol — the whole spec is a few pages), or WS-JSON (crypto). The internal world downstream speaks one fixed-layout binary format, because the sequencer and engines must never pay parsing costs (ch00c). The gateway performs the last parse. Translation cost is why serious venues offer a native binary protocol next to FIX and why their FIX gateway is documented as slower — a fact your SOR exploited when it chose which protocol to send on.
④ Pre-trade risk, in-line, before the sequencer. This ordering is a design decision that earns its place three times over. First, a rejected order must never consume a sequence number or touch a book — the sequenced log is the market’s official history (ch23), and garbage doesn’t belong in the record. Second, the sequencer is the one single-threaded resource the whole market funnels through, so every cycle it spends stamping a message that was never eligible is capacity stolen from legitimate flow — precisely during the bursts when that capacity is scarce. Third, a reject decided at the edge turns around inside the gateway in microseconds, with no round trip through the core and back — the client gets a fast, explicit answer from the one tier that’s cheap to scale. The checks:
- Fat-finger limits — max order quantity/notional per message: the “did a human or a bug add three zeros” check, the per-transaction card limit of the PSP edge. It matters because one fat order can walk an entire book before anyone reacts.
- Price bands — reject limit prices further than X% from a reference price — the venue-side companion to circuit breakers.
- Credit / position limits — per-account max exposure, maintained gateway-locally with async reconciliation — the check that turns “client went insolvent” from a venue loss into a rejected order. (In tradfi this is regulatory: SEC 15c3-5, the “market access rule,” legally requires pre-trade risk checks on every order before it reaches the market. Sponsored access is a member lending its exchange access to a client; doing that without checks — “naked access” — was banned in 2010.)
- Self-match tagging — attach the account’s STP group so the engine can act on it (below).
⑤ Rate limiting. A token bucket per session — plain English: you accrue N message-credits per second up to a burst cap; each message spends one — you spent years on the receiving end of exactly this, engineering your order flow to stay under venues’ buckets. From this side, its purpose is crisp: the sequencer is a shared funnel, and one runaway client must not be able to consume the market’s headroom. Same reason your multi-tenant Postgres has per-tenant connection caps: protect the shared resource, make blast radius per-tenant.
Sequence-number bookkeeping per session. FIX sessions number every message in both directions (MsgSeqNum), and both sides track expected-next. This is not the sequencer’s global number — it’s per-session, and it exists for gap detection and recovery on that session (see the state machine below). Your idempotency-key discipline from payments is the same instinct: both sides must agree on exactly which messages were exchanged, and replays must be detectable.
The protocol zoo at the front door
The gateway speaks whatever its clients speak, and the choice each client makes is itself informative:
| FIX | OUCH-style native binary | WS-JSON (crypto) | |
|---|---|---|---|
| Encoding | tag=value ASCII (35=D|55=AAPL|...) | fixed-width binary, ~10 msg types | JSON over WebSocket |
| Typical msg size | 200–400 bytes | 30–50 bytes | 300–800 bytes |
| Gateway parse cost | ~0.5–2 µs | ~50–200 ns | ~1–5 µs (JSON + TLS) |
| Session recovery | full resend machinery (below) | thin — relies on daily reset + drop copy | ad-hoc: REST resync |
| Who uses it | institutions, brokers, anything cross-venue | latency-sensitive prop flow | everyone in crypto |
The pattern to articulate in an interview: FIX is the venue’s compatibility API, native binary is its performance API, and serious venues ship both — same resource, two contracts, exactly like a REST API next to a gRPC endpoint in your world.
FIX won institutional trading not on merit of encoding (the encoding is famously awful) but on network effects: every OMS (order management system — an institution’s system of record for orders), every broker, every compliance system already speaks it. It’s the SWIFT of trading connectivity. That network effect is also why Talos — a vendor whose product is exactly this layer, an institutional FIX face over twenty bespoke crypto WS APIs — exists as a company, and why Talos is the shape of your interview target: the wrapper itself is a product.
Your SOR chose the native protocol wherever a venue offered one; now you know the gateway-side reason it was faster: you were skipping the expensive parse row of this table.
Identity is a hierarchy, not a flat key. The session that connects is the bottom of a tree the gateway must understand, because risk limits attach at every level:
FIRM (member) ── firm-wide credit/exposure cap
├─ ACCOUNT A (prop desk) ── per-account position & notional limits
│ ├─ session 1 (colo, OUCH) ── per-session rate, COD config
│ └─ session 2 (backup, FIX)
└─ ACCOUNT B (client of the member, sponsored access)
└─ session 3 ── 15c3-5 checks applied by the
SPONSOR's limits, at YOUR gateway
Multi-tenant SaaS shape again — org → project → API key, quotas at each tier, exactly your Postgres tenant hierarchy — with one venue-specific twist: limit checks roll up (session 1’s order must pass session, account, and firm caps), and the firm-level check is the one that can’t be sharded away, since two sessions on two different gateways draw on one shared credit number.
Here’s how that shared number gets checked without a cross-gateway lock. Say the firm’s cap is $10M and its sessions land on three gateways. Each gateway holds a $2M lease carved out of the cap and admits orders against its lease locally — a nanosecond memory check, no network hop. The remaining $4M stays with the credit service, which reconciles actual usage continuously and re-sizes leases out-of-band, shrinking them as the firm approaches its cap. The trade you’re making: a small, bounded over-admission risk (leases can momentarily add up to more than what’s truly left) instead of a synchronous cross-gateway round trip on every order. The trade has a limit, though: as the firm approaches its cap the leases shrink, and eventually the possible over-admission is no longer small next to what’s genuinely left — so near the cap, large orders fall back to a synchronous check against the shared number, paying the round trip only when the error would matter. Same pattern as distributed rate limiting across your API fleet: local buckets, eventual global truth, bounded error.
When the bucket runs dry: reject, don’t queue
A detail you experienced from the client side and now get to decide as the operator: what happens to the message that exceeds the rate limit?
token bucket empty:
Option A: QUEUE the message until tokens accrue ◄── tempting, wrong
Option B: REJECT immediately with an explicit error ◄── correct
Queueing feels polite — it’s what your HTTP-world rate limiters often did — but it’s wrong here for a reason specific to trading: a delayed order is a different order. An instruction priced against the book at T, silently executed against the book at T+50ms, is not what the client sent — you’ve converted their limit order into a worse one without telling them.
Rejection with an explicit, machine-readable throttle error returns the decision to the only party who has current context: the client, whose strategy can re-price, re-route (your SOR did exactly this — venue throttled → next venue in the table), or drop the intent.
The corollary you lived: venues that silently queued under load were the ones whose ack latencies went bimodal and whose fills came back mysteriously stale. As the operator, the rule is: the gateway may be a filter, never a buffer. The same logic is why pre-trade risk rejects synchronously instead of parking orders for review — in this domain, fast explicit failure is a feature, and every queue you add between client and sequencer is latency variance you’ll have to explain.
One refinement venues actually ship: separate buckets (or reserved headroom) for cancels. Throttling a client’s new orders protects the market; throttling their cancels traps them in positions during exactly the volatile moments when buckets empty — which is dangerous for them and, if the venue extends credit, for you. “Cancels always get through” (or get a much deeper bucket) is a common and defensible asymmetry, and a good interview flourish because it shows you’re reasoning about the risk semantics of message types, not treating throughput as undifferentiated load.
A worked latency budget
Numbers make the design concrete. A respectable software gateway (no FPGA), door-to-sequencer:
| Stage | Budget | Notes |
|---|---|---|
| NIC → userspace (kernel bypass, ch04) | ~1–2 µs | or ~4–8 µs through the kernel stack (ch01) |
| Ingress timestamp + session lookup | ~50–100 ns | pre-resolved at logon; one predictable read |
| Protocol decode (OUCH/binary) | ~50–200 ns | FIX tag=value parse: ~0.5–2 µs — the price of verbosity |
| Risk-check chain (④, all checks) | ~0.5–1 µs | each check ~50–200 ns, in-cache, branch-predictable |
| Token bucket + stamp + enqueue to sequencer link | ~100–200 ns | |
| Total, gateway ingress → sequencer inbound | ~2–4 µs | FIX clients pay ~1–2 µs more than binary clients |
Human scale: the entire paranoid middleware stack — auth, parse, five risk checks, rate limit — fits in the time one accidental syscall would cost (ch00b). That’s the discipline: the budget doesn’t survive a single allocation, lock contention, or cache miss on a cold limits table, which is why limits are pre-loaded, structures are fixed-layout, and the hot loop runs pinned on an isolated core with the ch03 treatment. And it’s why hardware-assisted venues (FPGA risk checks at the NIC) can quote sub-microsecond gateways: box ④ is embarrassingly parallel fixed-function logic — each check is independent of the others, and what runs branch-predictable in software unrolls to branch-free logic in hardware — the most FPGA-shaped workload in the whole venue.
Fairness at the edge
Every gateway adds latency. So the fairness rules are:
client A ──► gateway 1 ──┐
client B ──► gateway 1 ──┤ RULE 1: within a session/gateway,
client C ──► gateway 2 ──┼──► sequencer arrival order preserved
client D ──► gateway 3 ──┘ RULE 2: ACROSS sessions, order is
decided ONLY at the sequencer
gateways must be interchangeable:
same hardware, same code, same path length to the sequencer
- Per-session ordering is guaranteed by the gateway (it’s a TCP stream; keeping order is free).
- Cross-session ordering is decided only at the sequencer (ch23). The gateway must not create ordering — it must merely not distort the race. Hence: gateways are interchangeable (identical hardware, identical code path, no “premium gateway” unless it’s a disclosed product), and gateway→sequencer links are latency-equalized.
- The famous tradfi expression of this: equalized cable lengths in colo. Exchanges like NYSE literally provision the same optical fiber length from every colo cage to the matching-engine access switch — a client one rack closer gets the same propagation delay as one across the hall. It sounds like theater until you remember Part 0’s numbers: light in fiber covers ~1m in 5ns, cages differ by tens of meters, and 100ns is a winnable race margin. Know this as lore; interviewers use it to test whether you grasp that fairness is enforced physically, not just in software.
The gateway’s fairness obligations, stated as invariants: never reorder within a session; never add client-dependent latency (no per-client code paths on the hot path); timestamp at ingress so distortions are measurable. That last one is your ch08 discipline turned into a compliance artifact — venues monitor per-gateway latency distributions and investigate skew, because skew is a fairness incident, not a perf bug.
Self-match prevention and cancel-on-disconnect
Two features you used as a client, now seen from the operator’s side:
Self-match prevention (STP) stops one firm’s buy from trading against the same firm’s sell. Why clients want it: a firm running many independent strategies (or your SOR racing your own maker flow) can accidentally cross with itself, and in regulated markets self-trades look like wash trading (fake volume, potentially market manipulation) — STP shields the client from that compliance exposure.
The venue has its own stake: printed volume that’s really one firm trading with itself is garbage data, and a tape polluted with self-crosses undermines the venue’s core product. Preventing them protects the venue’s credibility, not just the client’s.
Mechanics: orders carry an STP group ID, tagged at the gateway (④). When the engine detects that an aggressing order would match a resting order in the same group, policy applies — cancel-newest (aggressor dies), cancel-oldest (resting order dies), or cancel-both. Note the split of labor: the gateway tags (it has the account context), the engine enforces (only it sees both sides of the prospective match). A gateway can’t do STP alone — it never sees the book.
Cancel-on-disconnect (COD): when your session drops, the venue pulls all your resting orders. You configured this on every venue that offered it, because a maker with dead connectivity and live quotes is a sitting duck. From the venue side it’s equally self-interested: orders whose owner can’t manage them are stale liquidity that will print bad trades, generate disputes, and — for the venue extending credit — grow unmanageable exposure.
Design detail that matters: COD must trigger on effective death, not just TCP FIN — which is what heartbeats are for (below), and why heartbeat intervals (commonly 1–30s, negotiated at logon) are effectively the COD detection latency. The gateway turns “session declared dead” into a burst of cancel messages injected into the sequencer like any other flow — which itself is a burst-capacity line item: one big maker disconnecting can be 50k cancels in one shot.
Abuse: quote stuffing and messaging economics
The gateway is also the venue’s abuse-defense tier. The canonical attack is quote stuffing: flooding the market with orders and cancels you never intend to trade, to congest competitors’ feed processing or the venue itself. Think of an application-layer DoS that is fully authenticated and well-formed — a logged-in API client hammering a legitimate endpoint. That’s exactly what makes it hard: classic DoS defenses (drop unauthenticated junk at the edge) don’t apply, because every message is from a paying member.
Venues respond on two axes:
- Technical: the per-session token buckets (⑤); per-account aggregate limits across sessions; port-level throughput caps in colo.
- Economic: order-to-trade ratio policies and messaging fees. Your messages are free until your ratio of orders to actual trades crosses a threshold (venues set thresholds anywhere from tens to ~500:1 depending on the program); after that, each excess message costs money. It’s API pricing tiers: the free tier is sized for legitimate use, and abuse gets priced out rather than blocked. When your abusers are authenticated paying customers, pricing is a better throttle than blocking, because it scales pressure smoothly and doesn’t require you to adjudicate intent.
Scale shape: a wide, shallow fleet
thousands of sessions × modest per-session rate
5,000 sessions ──► [ gw1 ] [ gw2 ] ... [ gwN ] ──► one sequencer
│ (sticky: a session lives
│ entirely on one gateway)
└── p99 per-gateway budget: single-digit µs, flat under load
The gateway tier’s load profile is the opposite of the sequencer’s: wide and shallow. Thousands of concurrent sessions, each individually modest (a big market maker might run 1–10k msgs/s per session; most sessions do far less), summing to the market’s total. So it scales horizontally like any stateless-ish edge tier.
The one big caveat: sessions are sticky. A FIX session’s state (auth, sequence numbers, in-flight orders, heartbeat timers) lives on one gateway; you can’t round-robin messages of one session across the fleet. Losing a gateway = bouncing its sessions = every affected client runs recovery (below) and COD fires for those who configured it. This is your multi-tenant SaaS scaling story — horizontal fleet, per-tenant stickiness, blast radius = one box’s tenants — with the twist that “failover” is client-visible by design, because the protocol makes session death explicit rather than hiding it behind a retry.
Session state machines: FIX mechanics for a WebSocket native
You know this dance from the other side — twenty venues’ worth of reconnect/re-auth/resubscribe logic. FIX formalizes it:
┌────────┐ Logon(seq, HB interval) ┌─────────────┐
TCP up ─►│ CONNECT│ ────────────────────────────►│ ESTABLISHED │◄─┐
└────────┘ (auth + seq negotiate) └──────┬──────┘ │
│ │ Heartbeat
no msg for HB interval ───────────────┤ │ every N sec
▼ │ (both ways)
┌──────────────┐ │
│ TestRequest │──┘ reply in time
│ sent (ping) │
└──────┬───────┘
no reply │
▼
┌────────┐ Logout / force close ┌──────────────┐
│ CLOSED │◄───────────────────────────│ DECLARED DEAD│──► trigger COD
└────────┘ └──────────────┘
Mapping to your WebSocket world: Logon = connect + auth handshake; Heartbeat = ping/pong frames; TestRequest = “I haven’t heard from you, prove you’re alive” (an explicit, in-protocol ping with a required correlated reply — WS ping/pong made mandatory and audited); Logout = graceful close. The part with no clean WS analogue is recovery:
Resend Request — when a session re-establishes mid-day, both sides compare sequence numbers. If the client logs on saying “my next inbound should be 8,001” but the gateway already sent through 8,240, the client issues a Resend Request for 8,001–8,240 and the gateway replays those application messages from its per-session outbound store. What this means for you as the operator:
- Every gateway keeps a durable-enough per-session outbound message store for the day — a real storage/retention design item, not a nicety.
- Replayed execution reports are flagged (PossDup) so the client’s idempotency layer — and this is literally your payments idempotency-key pattern, down to the semantics — can distinguish “new fill” from “redelivery of a fill you may have seen.”
- The dangerous asymmetry: a reconnecting client mostly wants your messages resent (did I get filled while dark?); for orders the venue missed from the client, resending is usually wrong — you don’t want 90-second-old order instructions entering today’s market, so convention is to gap-fill them with sequence resets rather than replay them. Stale intents die; facts get redelivered. Get this backwards as an operator and you inject a client’s dead orders into the book — the kind of incident that ends up in a venue’s disciplinary notices.
Compare crypto: most venues punt — on reconnect you re-auth, re-subscribe, and poll REST for open orders and recent fills to resync. Same problem, coarser tool. Having implemented the coarse version twenty times, you can explain precisely why the FIX version exists: it turns “resync after disconnect” from a bespoke racy dance into a protocol-guaranteed replay with explicit dup marking.
What the gateway operator watches
Running a gateway fleet, your observability (ch11) centers on a short list — worth having ready because “what would you monitor?” is a standard interview close:
- Per-gateway ingress→sequencer latency distributions, compared across gateways. Absolute p99 matters; skew between gateways matters more — skew is a fairness incident (some clients’ door is slower than others’), and sophisticated clients will find it before you do. Alert on divergence, not just degradation.
- Reject rates by reason code, per account. A spike in fat-finger rejects on one account = their bug (call them before they call you — the venue-side version of a PSP flagging a merchant’s retry storm). A spike across many accounts = your bug — a bad reference price feeding the price bands is the classic: suddenly every legitimate order is “outside the band,” and you’ve effectively halted the market from the edge. Reference-price staleness needs its own alarm.
- Token-bucket saturation per session — who is riding their limit, which is both a capacity-planning signal and a sales lead (“your flow has outgrown your tier”).
- Heartbeat/TestRequest statistics and COD firings — a cluster of sessions going dark together is a network incident on your side, not twenty coincidences; auto-correlating “who died together” by gateway/switch/path is the fastest triage signal you have.
- Sequence-gap and resend-request counts per session — a client constantly gap-filling has a flaky path or broken engine; either way it becomes your support ticket and, in the disputes above, your evidence.
Plain-English recap
- FIX is the compatibility API, native binary is the performance API — REST next to gRPC — and the identity model under both is a multi-tenant hierarchy (firm → account → session) with limits that roll up, enforced via local leases against shared caps rather than cross-gateway locks.
- The gateway is an API gateway + auth middleware + rate limiter in front of a monolith — except the monolith is one sequenced pipe, and the whole ingress stack — auth, decode, five risk checks, rate limit — fits a 2–4 µs budget, roughly the cost of one stray syscall. Hence: no locks, no allocation, no syscalls, no cold cache lines, limits cached in local memory.
- Pre-trade risk runs before the sequencer so a rejected order never consumes a sequence number — and in US markets, pre-trade checks are literally the law (15c3-5), not a best practice.
- Rate limits you suffered as a client are, from this side, per-tenant caps protecting a shared funnel — same logic as connection caps on your multi-tenant Postgres.
- Fairness at the edge = gateways are interchangeable and add equal latency; ordering across clients is created only at the sequencer. Tradfi enforces this down to equalized fiber lengths in the colo hall.
- Self-match prevention: gateway tags, engine enforces — because only the engine sees both sides of a prospective match. Cancel-on-disconnect: heartbeat interval is your detection latency, and one dead maker means a 50k-cancel burst.
- Abusers here are authenticated paying customers, so the best throttle is pricing (messaging fees on high order-to-trade ratios) layered on top of token buckets.
- When the rate limit trips: reject explicitly, never queue — a delayed order is a different order, and silent buffering converts clients’ instructions into worse ones. Cancels get a deeper bucket, because trapping a client in a position helps no one.
- FIX session recovery is your payments idempotency discipline as a wire protocol: redeliver facts (fills, flagged PossDup), never replay stale intents (old orders).
- The operator’s dashboards watch fairness, not just health: cross-gateway latency skew, reject-reason spikes (one account = their bug; all accounts = your reference price), and sessions dying together (your network, not coincidence).
Interviewer will ask
“You consumed 20 venues’ gateways. Now design one. What’s on the hot path and what’s not?” “Hot path, in order: ingress timestamp, session lookup (pre-resolved at logon, one cache-friendly read), protocol decode into our internal binary, pre-trade risk as a chain of pure in-memory checks — fat finger, price band, STP tag, credit — each budgeted at ~50–200ns, token bucket, forward to sequencer. Off the hot path: auth resolution (logon-time), limit updates (pushed out-of-band from a risk service, gateway just reads local memory), the per-session outbound store writes (async, but must be durable enough to serve resend requests), and all observability via the ch11 patterns. The design rule I’d carry from the client side: anything that can vary per-client must not vary in latency per-client, or I’ve built an unfair gateway.”
“Why must risk checks run before the sequencer rather than in the matching engine?” “Picture the funnel from ch23: everything the market does squeezes through one sequenced pipe, and the gateway is the door in front of it. Three reasons the risk gate sits at the door. The sequenced log is the market’s official, replayable history, so a rejected order must never consume a sequence number — garbage doesn’t get a place in the record. The sequencer is the scarcest single-threaded resource in the building, so cycles it spends stamping ineligible messages are capacity stolen from legitimate flow — exactly during the bursts when everyone’s fighting for it. And a reject decided at the edge turns around in microseconds inside the gateway, with no round trip through the core — a fast explicit answer from the tier that’s cheap to scale. The one check that can’t live at the edge is anything needing book state — the actual self-match decision at match time — which is why STP is split: gateway tags the group ID, engine enforces. I saw both halves as a client: instant rejects from the door, STP cancels only at the moment a match would have printed.”
“A big market maker’s session drops at 14:30 on CPI day. Walk me through what your venue does.” “Heartbeat/TestRequest declares the session dead — worst case one heartbeat interval plus the test-request timeout, so if they negotiated 5s heartbeats, up to ~10s of dark time. If they’ve armed cancel-on-disconnect, the gateway injects cancels for all their resting orders into the sequencer — potentially tens of thousands of messages, which my burst budget must include, on a day that’s already 10–100×. Their per-session outbound store retains every execution report from the dark window. When they reconnect: logon, sequence-number comparison reveals the gap, they issue a Resend Request, I replay their fills flagged PossDup so their idempotency layer dedupes. Having been the client in exactly this scenario, the thing I’d obsess over as the operator is that replayed fills are complete and correctly flagged — a missing fill during a disconnect is how clients end up unknowingly short on a volatile day, and it becomes the venue’s dispute.”
“How do you stop quote stuffing without hurting legitimate market makers?” “The hard part is the chapter’s framing: the attacker is an authenticated paying member sending well-formed messages, so I can’t drop them at the edge like anonymous DoS junk. So the defense layers. Token buckets per session and per account are the technical ceiling, sized so legitimate maker behavior never touches them. Then economics does the discrimination that intent-guessing can’t, and the discriminator falls out of the order-to-trade ratio. A legitimate maker cancels constantly because they’re re-quoting — many hold quoting obligations, contractual commitments to keep two-sided quotes posted, so every price move forces a cancel/replace — but that churn resolves into actual trades, so their ratio stays inside the venue’s program thresholds. A stuffer’s messages exist only to congest; almost none ever trade, so their ratio goes pathological. Hence the policy: order-to-trade thresholds with messaging fees above them — pricing applies pressure smoothly, scales with the abuse, and never requires me to adjudicate intent in real time. I lived under these regimes as a client and they shaped our SOR’s cancel discipline — the venue’s fee schedule is an API that trains client behavior.”
“Why do exchanges equalize cable lengths in colo? Isn’t nanosecond fairness theater?” “It’s not theater once you do Part-0 arithmetic: light in fiber does ~1m per 5ns, colo cages differ by tens of meters, so unequalized runs hand some firms a 100–200ns structural edge — and races at the top of book are won by less. More than that, fairness is the product: firms pay colo fees precisely for a credible level playing field, so the venue’s job is to push all systematic advantage out of the infrastructure and let firms compete on their own stack. Software mirror of the same principle: interchangeable gateways, no per-client hot-path branches, ingress timestamps so any skew is measurable and fixable.”
“Why offer FIX at all if your native binary protocol is 10× cheaper to parse?” “Because FIX is the compatibility surface, not the performance surface — every institutional OMS, broker bridge, and compliance stack already speaks it, so a venue without FIX has no institutional on-ramp. You run both, like REST next to gRPC: FIX gateway documented as the slower path, native binary for flow that cares. The subtle operator obligation is honesty about the gap — clients choose protocols based on your published gateway latencies, and my SOR did exactly that arithmetic per venue. The Talos-shaped observation: the FIX-to-many-crypto-APIs translation layer is valuable enough that it’s an entire product category, which tells you how sticky the FIX network effect is.”
“Two sessions of one firm sit on two different gateways sharing one firm-level credit limit. How do you check it without a cross-gateway lock?” “Local reservations against the shared number: each gateway holds a lease — say 20% of remaining firm credit — admits orders against its lease locally at nanosecond cost, and reconciles with the credit service out-of-band, shrinking leases as utilization climbs. It’s distributed rate limiting across an API fleet: local buckets, eventual global truth, bounded over-admission error. The design conversation is about the tolerance band — how much over-limit exposure is acceptable for how many microseconds. And the scheme degrades gracefully at its edge: as the leases shrink near the cap, the possible over-admission stops being small next to what’s genuinely left, so large orders fall back to a synchronous check against the shared cap — you pay the round trip only when the error would matter. What you never do is put a cross-gateway round trip on every order’s hot path to make the error zero.”
“Your gateway fleet — what happens when one gateway host dies?” “Sessions are sticky, so that host’s sessions hard-drop; blast radius is its tenant list, like losing one shard of a multi-tenant fleet. Clients see explicit session death — which is correct in this domain; unlike a web LB I must not transparently fail over mid-session, because session state includes sequence numbers and in-flight order context that another box doesn’t have, and pretending continuity risks silent gaps. So: COD fires for opted-in sessions, clients reconnect to surviving gateways via their connection lists, sequence negotiation + resend recovers the facts. My job is making that path boring: per-session outbound stores replicated off-box so resend works after host loss, and capacity headroom so N−1 gateways absorb the re-logon stampede.”
Further reading
- FIX Session Protocol specification (FIXT.1.1) and the FIX Trading Community’s session-layer docs (fixtrading.org) — logon, heartbeats, resend, PossDup: the exact machinery in this chapter.
- NASDAQ OUCH 4.2 / 5.0 specification (nasdaqtrader.com) — a real venue’s lean binary order-entry protocol; contrast its ~10 message types with FIX’s sprawl.
- SEC Rule 15c3-5 (“Market Access Rule”) adopting release — why pre-trade risk checks are legally mandatory in US markets and what “naked access” was.
- CME Globex documentation on messaging policy, order-to-trade programs, and Cancel-on-Disconnect (cmegroup.com) — a tier-1 venue’s published edge-tier rules.
- Donald MacKenzie, “Trading at the Speed of Light” (Princeton, 2021) — the colo fairness arms race, including cable-length equalization, from a sociologist embedded with the firms.
Where this goes next: ch25 walks out the other door — the same sequenced stream, published to thousands of consumers who all want it first.
Publishing Market Data: Building the Feed
Before you start. This chapter assumes the sequencer + deterministic-consumer layout (the feed is derived from the same log as the fills, ch23), UDP multicast vs TCP fanout (one-packet-many-receivers vs one-connection-per-client, ch02), feed-consumer mechanics — gap detection, snapshots, book building — from the client side (ch00f, ch00d), event sourcing / snapshot + replay (ch13), and queueing under bursts (ch08). If any are new, read those first.
You spent years on the receiving end of market-data feeds — 20+ venues’ worth of gap detection, snapshot recovery, and book building in your ingestion pipeline. Every quirk you handled defensively (missed sequence numbers, snapshots inconsistent with increments, venues that silently dropped you under load) was a producer-side design decision, made well or badly. This chapter is where you make those decisions. The matching engine decides what happened, but the publisher has to tell ten thousand people at once — and at crypto scale, the fanout tier dwarfs the matching engine.
The publisher is just another consumer
┌──────────────────────────► [ matching engine ] ──► fills
[ SEQUENCER ] ───┤ same sequenced stream
(one ordered └──────────────────────────► [ FEED PUBLISHER ]
event log) │
│ maintains its own book replica
│ (deterministic → bit-identical
│ to the engine's book, by determinism)
▼
┌──────────────┬──────────────┐
│ L1 channel │ L2 channel │ L3 channel
│ (top of book)│ (depth) │ (every order)
└──────────────┴──────────────┘
│
fanout tier (this
chapter's real topic)
Start with where the feed comes from: the feed is not a report from the matching engine — it’s an independent derivation from the same sequenced log. The publisher consumes the sequenced stream, maintains its own book replica, and emits deltas. Because everything downstream of the sequencer is deterministic (ch23), the publisher’s book is bit-identical to the engine’s without ever talking to it. No RPC from the engine’s hot path, no “publish” call that can slow matching, no possibility of feed-vs-fills divergence that isn’t a determinism bug. Your event-sourced engine had exactly this property — you could hang any number of read-model projections off the event log without touching the writer. A market-data feed is a read-model projection with ten thousand subscribers and a latency SLA.
Product tiers: L1, L2, L3
| Tier | What it is | Your-world analogy | Who buys it |
|---|---|---|---|
| L1 / top of book | Best bid, best ask, last trade — the one-line summary | The webhook that says “payment settled” without the line items | Retail platforms, charting, anyone who needs a price, not the book |
| L2 / depth | Aggregated quantity per price level, top N levels or full depth — the book as a histogram | An API response with the pagination depth you choose | Most algorithmic traders; your old SOR (queue-depth-aware routing needs it) |
| L3 / order-by-order (ITCH-style full feed) | Every individual order’s add, cancel, replace, execute, with order IDs — the raw event stream itself, minimally disguised | Not the API response but the change-data-capture stream off the database | HFT firms who rebuild the book themselves and mine it for microstructure signal — signals hiding in the order flow itself, not the price. Queue-position estimation needs this tier; an L2 consumer literally cannot compute what an L3 consumer can |
Why the venue tiers them, beyond bandwidth: market data is a product line, often rivaling trading fees as a revenue source. Tiering is price discrimination by information content — same instinct as your API product tiers (webhook granularity levels, basic vs firehose), except here the premium tier’s customers can measure its value in basis points (hundredths of a percent, the unit execution quality is scored in). Note the derivation direction: L3 → L2 → L1 are each computable from the previous, so the publisher builds once from the sequenced stream and projects three ways. One book replica, three serializers.
The incremental + snapshot architecture
This is the contract you coded against for years. Now specify it from the producer side:
CHANNEL A: incremental (every change, seq-numbered)
──► seq 1001: bid 64999.5 qty 3.2 (level update)
──► seq 1002: ask 65000.0 qty 0 (level delete)
──► seq 1003: trade 65000.5 qty 0.4
... continuous, low-latency, the "real" feed ...
CHANNEL B: snapshot (periodic full book state)
──► snapshot { as_of_seq: 1000, bids: [...], asks: [...] } every N sec
──► snapshot { as_of_seq: 1450, ... }
consumer recovery = your old client-side dance:
buffer increments ► fetch snapshot(as_of_seq=S) ► drop increments ≤ S
► apply buffered > S ► live
The producer-side book-building contract — the invariants your consumers’ correctness rests on. You know what breaking each one does to a client, because you handled the breakage; now you’re the one who must never break them:
- Gap-free, monotonic sequence numbers within a channel. The consumer’s only loss-detection mechanism is “I saw N, next must be N+1.” Implication for you: per-channel sequence assignment is sacred, and any internal publisher failover must resume without gaps or dups — which the sequenced log makes possible (replay from last published seq) and ad-hoc designs get wrong.
- Every snapshot carries an exact consistency point (
as_of_seq), and applying increments from that point onward yields the true book. The classic producer bug — snapshot generated from a book mid-mutation, or stamped with a fuzzy sequence — gives consumers permanently corrupt books that look plausible. You debugged venues with exactly this bug; it costs your consumers days of “our book drifts from reality” forensics. - Deterministic emission: same log → same feed messages. This makes the feed itself replayable for your own testing (ch13) and lets you regression-test the publisher the way you regression-tested your engine.
- Documented conflation semantics (below) — if the feed may skip states, consumers must know which states can vanish and what is never skipped (trades, in any sane design).
The snapshot channel is cheap insurance: a snapshot every 1–60s costs little and bounds every consumer’s recovery time. As a client you cursed venues with 60s snapshot intervals during volatile gaps; as the producer, snapshot frequency is a knob trading your bandwidth against their worst-case recovery — put it in the spec and make it burst-aware.
Tradfi fanout: multicast, and why it’s a different universe
TRADFI (UDP multicast) one send, N deliveries
┌─► subscriber 1
publisher ──1 packet──►│switch│──► subscriber 2 the SWITCH replicates
(one send() total) └─► subscriber N in hardware, ~300ns
A/B feeds: two independent multicast groups, disjoint network paths,
same payload — consumers arbitrate (first-arrival wins per seq),
loss on one path masked by the other [you knew these as a consumer;
see the TCP/UDP chapter]
Multicast fanout: the publisher sends each packet once, to a group address, and the switches replicate it to every subscribed port in hardware. It’s a CDN edge duplicating your origin’s single stream to every viewer — except the “CDN” is L2 switching silicon and adds nanoseconds.
The consequence: publisher cost is O(1) in subscriber count.
ITCH-class feeds peak at millions of messages/sec (NASDAQ TotalView peaks in the tens of millions market-wide). The price: it’s UDP, so no delivery guarantee — which is why the A/B dual-feed pattern and the retransmission/snapshot infrastructure exist, and why the seq-number contract above is the load-bearing wall.
The recovery back-office: retransmission and replay services
Multicast’s fire-and-forget speed pushes reliability to dedicated side services — infrastructure you interacted with as a consumer without necessarily naming it:
┌───────────────┐ lost pkts 1001-1005? ┌──────────────────────┐
│ consumer │─────request (TCP/UDP)───►│ RETRANSMISSION server │ small gaps:
│ (gap detected)│◄────those packets────────│ (recent-history cache)│ re-request
└───────────────┘ └──────────────────────┘
│ too far behind / too big a gap?
└──────────────────────────────────────────► SNAPSHOT/REPLAY service
(start over from a
consistency point)
NASDAQ’s MoldUDP64 layer is the canonical example: sequenced UDP packets on the multicast group, plus a re-request server that serves recent history to consumers who name a sequence range. Design decisions you now own as the producer:
- The retransmission window is deliberately small (seconds of history, bounded memory). It exists for microbursts and single-packet drops, not for consumers that went to lunch. Past the window, the answer is the snapshot channel — this two-tier split (tiny fast re-request, big slow resync) bounds the cost of your recovery infrastructure no matter how broken the consumer.
- Re-request capacity is itself rate-limited per consumer, or one badly-written feed handler in a loop becomes a DoS on the recovery path during exactly the burst that caused everyone’s gaps. (You saw venues throttle recovery endpoints; now you know why.)
- A/B arbitration reduces re-requests to nearly zero in practice — two independent paths rarely drop the same packet — which is why the dual-feed pattern is cheaper than it looks: you pay 2× bandwidth to almost never touch the recovery path.
The crypto equivalent is coarser: the REST depth-snapshot endpoint plus WS resubscribe is the recovery service, and its rate limits during volatile periods — which you cursed — are the same “protect the recovery path from stampedes” logic, minus the fast small-gap tier that would have made your life easier.
Crypto reality: per-client TCP fanout, and the arithmetic of pain
No multicast over the public internet. Every consumer is a WebSocket — a private TCP connection with its own send queue, its own congestion state, its own slowness:
┌─► [queue][TLS][TCP] ─► client 1 (fast, fine)
publisher ──►(copy)─┼─► [queue][TLS][TCP] ─► client 2 (fast, fine)
every message, ┼─► [queue][TLS][TCP] ─► client 3 (SLOW ◄── problem)
every client └─► ... × 10,000
The arithmetic: 10,000 connected clients × 1,000 msgs/sec of book updates = 10,000,000 sends/sec — each a userspace copy, TLS encryption, and TCP transmission. Meanwhile the matching engine that generated those 1,000 msgs/sec is loafing on one core. The fanout tier dwarfs the matching engine — at a crypto venue, market-data distribution is commonly the largest compute fleet in the building, an inversion that surprises people who assume matching is the expensive part. Human scale: the engine’s day is a single busy Postgres writer; the fanout tier is a CDN origin under permanent load. This is why crypto venues shard fanout fleets by symbol and subscription tier, and why their engineering blogs are full of “how we rewrote our WS distribution layer” posts.
The slow consumer problem
This is the chapter’s central engineering problem. On TCP, a slow reader backpressures its connection: their receive window fills, your socket buffer fills, your per-client queue grows. The one thing you must never do is let one slow client backpressure the publisher — the feed is shared fate, and the market does not slow down because someone’s book-builder is GC-pausing. Your options, worst to best:
- Unbounded per-client queues: memory grows until the fanout host OOMs. One bad client kills service for everyone on that box. Never.
- Disconnect policy: bounded queue; on overflow, cut the client. Simple, predictable, and standard as the backstop — but as the only tool it’s harsh: on a volatile spike, everyone’s queues spike, and you’d mass-disconnect exactly when clients most need data (and their reconnect-plus-snapshot stampede hits you at the worst time).
- Conflation — with disconnect as the backstop.
Conflation: when a client falls behind, don’t queue every tick — keep only the latest state per price level and send that when the connection drains. It’s the webhook consumer that can’t keep up: you don’t slow the producer or buffer a million events, you skip to current state and let them re-sync.
You’ve built this before, pointed the other way. Your pipeline did fan-in conflation — 20 venues feeding your strategies, keep-latest per book level when your consumers lagged. This is the same data structure as fan-out: one keep-latest map per slow client:
per-client conflation map (bounded by book size, NOT by message rate):
updates while {bid 64999.5 → qty 3.2} later updates to the same
client is slow: {bid 64999.5 → qty 1.1} ──► level OVERWRITE in place:
{ask 65000.0 → qty 0 } map holds ONE entry per level
client drains ──► send current map contents (+ seq jump marker)
What makes conflation cheap: the conflation map’s size is bounded by book width, not message rate — a client can be behind by a million messages and owe you only a few thousand level-states. The costs, which you must document as feed semantics: the client loses tick-by-tick history (intermediate book states vanish — fine, because the current state supersedes them), and the client must be told (a seq discontinuity or explicit conflation flag) so they know their view skipped states.
Trades are facts and are never conflated. A missed book state is harmless — the latest state replaces it. A missed trade is a lost fact: trades must be queued faithfully or recovered via the snapshot/recovery channels, never collapsed away.
Venues run this as product tiers: full-rate feed for those who keep up, conflated feeds (e.g., 100ms-interval book states) as an explicitly throttled cheaper product — your API-tier instinct again, and tradfi vendors sell exactly this split.
Timestamps in the feed: the producer side of “when”
Your feed handlers compared venue timestamps to local receive time for years (ch07, ch08). Now you’re the one stamping, and each message wants several times, because consumers use them for different jobs:
one feed message, three producer times:
┌───────────────────────────────────────────────────────────────┐
│ event_time : when the sequencer sequenced the cause │ ◄ market truth
│ (a.k.a. transact/match time — same for every consumer, │ (use for
│ replay-stable, tied to the seq number) │ research/backtests)
│ send_time : when THIS publisher put it on the wire │ ◄ measures the
│ │ venue's own lag
│ [consumer adds] recv_time : their NIC timestamp │ ◄ measures the path
└───────────────────────────────────────────────────────────────┘
send_time − event_time = publisher lag (the venue's problem — publish it honestly)
recv_time − send_time = network path (the consumer's problem)
The design rules:
- event_time must come from the sequenced log, never from the publisher’s wall clock. It’s part of the deterministic output (same log → same event_times on replay), it’s what backtests and surveillance key on, and it’s identical across L1/L2/L3 so a consumer can join tiers.
- send_time is diagnostic, not truth. It differs across A/B feeds and across publisher restarts; its whole value is letting sophisticated consumers decompose “the data was late” into “the venue was slow” versus “my path was slow.” You did exactly that decomposition from the outside, usually with worse tools; publishing an honest send_time is the producer-side courtesy that makes it tractable.
- Never flatter send_time. Venues get caught stamping it early to improve their published latency, because colo consumers with hardware timestamps (ch07) can measure the lie.
- One clock for everything. Gateways, sequencer, and publishers PTP-synced to the same grandmaster (the one reference clock every machine in the building disciplines to), or your own timestamps can’t be compared across components — the venue-internal version of the multi-venue clock problem your pipeline fought.
Fairness, again: everyone “at the same time”
Tradfi multicast makes simultaneity credible: one packet, hardware replication with nanosecond skew, measured cable lengths from switch to every colo cage (ch24) — the venue can defend “all subscribers were sent the data at the same instant” as physical fact.
Per-client TCP fanout cannot be perfectly fair, and you should be able to say why precisely: sends are serialized (a loop over sockets — client #1 in iteration order beats client #10,000 by whole microseconds every single time), each connection’s TLS/TCP state differs, and kernel scheduling adds jitter.
Mitigations, not cures:
- Randomize send order per tick — no client is systematically first; a structural advantage becomes zero-mean noise, which is what fairness means in practice.
- Shard clients evenly across fanout hosts — no host’s send loop gets disproportionately long.
- Keep per-tick fanout loops tight — so the first-to-last spread stays small.
As a client you suspected some venues’ WS feeds had favorites; as the producer, randomized send order is how you make that accusation false — and provably so, because you can show the shuffle in code.
The reconnect stampede
The failure mode that couples everything in this chapter together: something blips — a fanout host dies, a network path flaps, or you mass-disconnect slow consumers during a volatility spike — and now thousands of clients simultaneously run the recovery dance: reconnect, re-auth, request snapshot, replay increments.
t=0 fanout host dies (2,000 clients)
t+1s 2,000 reconnects hit surviving hosts ◄─ TLS handshakes: CPU spike
t+2s 2,000 snapshot requests ◄─ snapshot service: 100×
... during the same volatile burst normal load, worst moment
that caused the disconnects ...
This is a thundering herd with a cruel correlation: recovery load peaks exactly when live load peaks, because volatility causes both the disconnects and the message-rate spike. Standard mitigations, all of which you’d recognize from web-scale work but must re-derive under microsecond-adjacent constraints:
- Jittered reconnect backoff, enforced server-side — clients won’t do it voluntarily; you didn’t, when reconnect speed was money.
- Pre-generated snapshots served from memory, never computed per-request — the snapshot at
as_of_seq=Sis identical for every requester, so build once per interval and serve many: your CDN-cache instinct exactly. - Connection-accept rate limiting — surviving hosts degrade gracefully instead of collapsing under the TLS-handshake spike.
- Capacity math for N−1 hosts during a burst — steady-state sizing is the wrong question, because the stampede arrives mid-burst by construction.
A venue that mass-disconnects on a spike and then can’t absorb the re-entry has converted a slow-consumer policy into a full outage — this coupling is why the disconnect threshold and the recovery capacity have to be designed as one system, not two settings owned by two teams.
Testing the publisher
The determinism dividend again: because the feed is a pure function of the sequenced log, the publisher is about as testable as a stateful component gets — provided you build the harness:
- Golden-feed regression: replay a captured production log through the candidate publisher; diff emitted bytes against the previous version’s output. Any unexplained diff is a bug or an intentional (and therefore documented) format change. This is exactly your engine’s replay-based regression testing pointed at a different output.
- Invariant checking in CI and prod: run a reference consumer that does what your clients do — build books from snapshot + increments — and continuously assert it matches a directly-derived book replica. This catches the consistency-point bugs (invariant 2) that plague real venues, before clients do the catching.
- Adversarial consumer simulation: a load harness of deliberately slow, gappy, reconnect-happy fake clients hammering the fanout tier — because the slow-consumer and stampede machinery above is exactly the code that never gets exercised until the worst day of the year, and “tested only in production during incidents” is how feed reputations die.
The other end of the wire: building the book as a consumer
Everything above is you publishing. Flip to the seat you’ve actually sat in — broker, trader, anyone consuming this feed — and two questions every market-data interview asks: how do you build the book in the first place, and how do you get the book as of some time N?
Building the live book
The algorithm depends on which product tier (above) you bought.
Level-based feed (L2/MBP — most crypto websockets). Events are absolute level updates: “bid 10001 now 400.” Your book is two sorted maps px → qty per side; apply each update, delete the level when qty is 0. A trivial fold — if you start from a correct base. Establishing that base is the interview question: the bootstrap ordering.
- Subscribe to the diff stream first. Apply nothing — you have no book yet. Buffer the deltas.
- Then fetch the snapshot. It carries a sequence number, say S.
- Discard buffered deltas with seq ≤ S — they’re already inside the snapshot. Verify the first surviving delta is S+1; a hole here means your book would be silently wrong until the next reconnect, so re-bootstrap instead. Apply the buffered tail, then go live.
- Any sequence gap later → the book is stale: mark it, stop trading on it, re-bootstrap. (The router rule from the broker chapter: a book you can’t trust is a book you don’t act on.)
Subscribe-then-snapshot is the load-bearing choice. Reverse it and every update that arrived between the snapshot fetch and the stream connect is lost forever — you carry a corrupt level for hours with no error anywhere. Some venues also publish book checksums with each update; apply, compare, and a mismatch is your corruption detector firing — re-bootstrap.
Order-by-order feed (L3/MBO — ITCH-style). Events are Add(order_id, side, px, qty), Execute(order_id, qty), Cancel(order_id), Replace(…). Your state is a hashmap order_id → (side, px, qty) plus per-price aggregates maintained as orders arrive and leave; the L2 ladder is now derived. Start from the venue’s start-of-day reset or its snapshot channel, apply in venue-sequence order. The reward for the extra work is queue position: you know exactly which orders rest ahead of yours at the touch — which is why L3 costs more (product tiers, above).
A production feed handler runs four checks continuously: sequence continuity; checksum where offered; the book never crossed against itself; and a staleness heartbeat — no update and no heartbeat for X ms means the feed is lying by silence.
The book as of time N
The move: the book at time N is not stored anywhere — it is derived. Nearest snapshot at-or-before N, plus a replay of deltas up to N. The same snapshot-plus-tail move as the venue’s audit tool (ch23) — except as a consumer you must manufacture the raw material yourself:
- Capture the feed as it arrives. Gold standard: a passive tap with hardware timestamps (ch05). Practical minimum: journal every normalized event with its venue sequence number and your receive timestamp. Add periodic snapshots of your built book (hourly, or every M events) so replay cost stays bounded.
- Storage shape: snapshot files plus compressed delta segments, keyed
(symbol, day), time-indexed. Recognize it — it’s a WAL plus checkpoints, the same design making its fourth appearance in this book. - Reconstruction: binary-search snapshots for the last one ≤ N, load it, replay the segment’s deltas while
ts ≤ N, stop. Cost is bounded by snapshot cadence — which is why cadence is a decision, not an afterthought. - Which clock is “time N”? Venue event-time and your receive-time differ by transit plus your own lag (ch07’s whole lesson). For TCA and best-execution the correct clock is your receive time — the question is “what could we have known when we routed,” not “what had objectively happened.” For market research, venue time. An answer that doesn’t state its clock isn’t an answer.
- Why your own capture beats vendor data: a best-ex dispute asks “what did we see” — a vendor archive is someone else’s clock and someone else’s gaps. Vendor data is fine for research; evidence needs your wire.
Who uses it: TCA’s arrival-mid (the book at parent arrival), markout computation, backtests, and the 2am dispute where someone claims your fill was off-market — you load the snapshot, replay to 14:32:07 on your clock, and read the answer off the screen.
Numbers to hold
| Quantity | Value | Human scale |
|---|---|---|
| ITCH-class full-feed peak | millions–tens of millions msgs/sec market-wide | a message per ~100ns at peak — hardware-timestamp territory (ch07) |
| Multicast publisher cost | O(1) in subscribers | 1,000th subscriber is free |
| Crypto WS fanout | 10k clients × 1k msg/s = 10M sends/sec | fanout fleet ≫ matching engine |
| Per-send cost (copy+TLS+TCP) | ~1–5 µs of CPU | 10M sends/sec ≈ tens of cores just moving bytes |
| Snapshot interval | 1–60 s typical | bounds every consumer’s worst-case recovery |
| Conflation map bound | book width (≈10²–10⁴ levels) | behind by 1M msgs, owe only the current book |
| Retransmission window | seconds of history, in memory | microburst repair only — beyond it, snapshot resync |
| Multicast replication skew | nanoseconds (switch hardware) | vs whole microseconds first-to-last in a TCP send loop |
Plain-English recap
- Feed messages carry two producer times with different jobs: event_time from the sequenced log (replay-stable market truth for backtests and surveillance) and send_time from the wire (diagnostic — it splits “data was late” into the venue’s lag vs the consumer’s path).
- The feed isn’t the engine reporting out — it’s a second, independent projection of the same event log, like a read model hanging off your event store. Determinism is what lets the fills and the feed never disagree.
- One book replica, three serializers: L3 → L2 → L1 are successive projections, so the publisher builds state once from the log and prices the projections as products.
- L1/L2/L3 are the same data at three information densities — summary webhook, paginated API, raw CDC stream — and they’re a revenue line, tiered like any API product.
- Incremental + snapshot is a contract, and you already know every clause from the consumer side: gap-free monotonic seq per channel, snapshots with exact
as_of_seqconsistency points, documented conflation semantics. Producer bugs here cost every consumer days of book-drift forensics. - Tradfi multicast = the switch is your CDN: one send, hardware replication, O(1) in subscribers, nanosecond skew. Crypto reality = one TCP/WS connection per client, so 10k clients × 1k msg/s = 10M sends/sec and the fanout tier dwarfs the matching engine.
- Slow consumers: never backpressure the producer. Bounded per-client queues, conflate book state to keep-latest-per-level (bounded by book width, not message rate), never conflate trades, disconnect as the backstop. It’s your fan-in conflation flipped to fan-out.
- Recovery is two-tier: a small in-memory retransmission window for microbursts (MoldUDP64-style, itself rate-limited), and the snapshot channel for everyone else. A/B dual feeds exist so the recovery path almost never gets touched.
- The reconnect stampede is the coupling failure: recovery load peaks exactly when live load peaks. Server-enforced jittered backoff, pre-built snapshots served from memory, and N−1 capacity math during a burst — or a slow-consumer policy becomes a full outage.
- Per-TCP fanout can’t be perfectly fair — someone is always first in the send loop — so you randomize send order per tick to turn a systematic edge into noise, and you keep the receipts.
- Determinism makes the publisher highly testable: golden-feed byte-diffs against replayed prod logs, a reference consumer continuously asserting snapshot+increments equals truth, and adversarial slow/gappy client simulations for the code that otherwise only runs on the worst day of the year.
Interviewer will ask
“You consumed 20 venues’ feeds. Design the feed you wished they’d built.”
“Two seq-numbered channels per product tier — incremental and snapshot — with three invariants I’ll never break because I paid for every venue that broke them: gap-free monotonic seq per channel, every snapshot stamped with an exact as_of_seq so the buffer-snapshot-replay recovery dance is deterministic, and trades never conflated even when book updates are. Publisher is a deterministic consumer of the sequenced log — no coupling to the engine, feed replayable for regression tests. Documented conflation semantics and an explicit flag when a client’s view has skipped states — the worst venues were the ones where I had to discover their conflation behavior empirically during a volatile open.”
“Why is the feed derived from the log rather than emitted by the matching engine?” “Decoupling and provable consistency. If the engine publishes directly, the fanout tier’s problems — slow consumers, TLS costs, reconnect stampedes — are one backpressure bug away from the matching path, and any engine/feed mismatch becomes an unanswerable reconciliation question. As a log consumer, the publisher can’t slow matching, scales independently — which matters since fanout is the bigger fleet — and determinism guarantees its book is bit-identical to the engine’s. It’s exactly the read-model projection pattern from my event-sourced engine: writers never wait for projections.”
“One client on your 10k-client WS fanout reads at half rate. What happens, and what do you do?” “Their TCP receive window fills, my socket send buffer fills, their per-client queue grows — and the design requirement is that this is completely invisible to the other 9,999 and to the publisher. Bounded per-client queue; on threshold, switch that client to conflation: collapse queued book updates to latest-state-per-level, so their debt is bounded by book width instead of message rate; queue trades faithfully since those are facts, not states. Mark the seq discontinuity so their book-builder knows to treat it as a partial resync. Hard overflow past that: disconnect and let them re-enter through snapshot recovery. I built the mirror image of this — fan-in conflation across 20 venues when my downstream lagged — so I’d also insist the conflation semantics be in the public spec, because as a consumer I had to reverse-engineer them.”
“Why can’t a TCP fanout be fair, and does it matter?” “Sends are serialized — some client is first in the loop, and iterating in a fixed order hands client #1 a systematic multi-microsecond edge over client #10,000, every tick, which sophisticated clients will detect and either exploit or complain about. Randomizing send order per tick converts the systematic edge into zero-mean noise — that’s the honest definition of fairness available on TCP. Contrast tradfi: one multicast packet, switch replicates in hardware, cable lengths equalized — simultaneity is a physical claim there. On WS it can only ever be a statistical claim, and the venue should be able to demonstrate the shuffle.”
“Your incremental feed and your snapshot service disagree during an incident. How?”
“If both are deterministic consumers of the same sequenced log, disagreement is a determinism bug or a consistency-point bug — my first suspect is the snapshot’s as_of_seq: a snapshot cut from a live book without a coherent sequence point, or off-by-one on which increments it includes. That bug ships consumers a book that’s subtly wrong forever after recovery — I’ve debugged it from the client side against a real venue and it presents as slow book drift, which is why I’d build the snapshot service as a replay-from-log at an exact sequence number, never a read of live mutable state, and continuously verify snapshot-plus-increments against a reference book replica in CI and in prod.”
“What timestamps go in a feed message and where do they come from?” “Two from the producer: event_time — when the sequencer sequenced the cause — which must be derived from the log, never the publisher’s wall clock, because it’s part of the deterministic output, identical across tiers and across replays, and it’s what backtests and surveillance key on. And send_time — when this publisher hit the wire — which is purely diagnostic: send minus event is the venue’s own lag, honestly published; consumer receive minus send is their path. I spent years doing that decomposition from the outside, sometimes against venues whose timestamps were flattering rather than true. The producer-side lesson from that: colo consumers with hardware timestamps can measure the lie, so flattery always gets caught. Which means the only defensible posture is one clock — gateways, sequencer, and publishers PTP-synced to the same grandmaster, so my own timestamps are comparable across components. And once the numbers are honest, publish the lag distribution yourself — better clients read it from your docs than discover it in their receive logs.”
“ITCH peaks at millions of messages a second. What does that force on the publisher?” “At tens of millions market-wide, a message arrives every ~100ns at peak. Hold that against costs you know: a single syscall or a single allocation costs more than that entire per-message budget, so anything the publisher does per message has to be a plain memory operation. That forces the pipeline’s shape — messages pre-serialized into fixed binary layouts so emission is a copy, sends batched so one syscall amortizes across dozens of messages, and the multicast path on kernel bypass (ch04) so there’s no per-packet kernel toll at all. But the deeper implication is for the retransmission and recovery infrastructure: at that rate a 100ms consumer glitch is a million-message gap, so gap recovery must come from snapshots and dedicated replay services, never from ‘please resend the increments’ — which is exactly why the incremental+snapshot split exists rather than a reliable-delivery protocol. Reliability is pushed to the edges; the feed itself stays fire-and-forget fast.”
“You connect to a venue’s L2 feed. Walk me through getting a correct book — and what goes wrong if you snapshot first.” “Subscribe to the diff stream first and buffer — I have no book yet, so there’s nothing to apply to. Then fetch the snapshot; it carries sequence S. Discard buffered deltas at or below S — the snapshot already contains them — check the first survivor is exactly S+1, apply the tail, go live. If I snapshot first instead, every update that lands between my snapshot fetch and my stream connect is simply gone: no gap, no error, just a level that’s wrong until the next reconnect — I’ve debugged exactly that as hours of slow book drift. And the standing rule afterward: any sequence gap makes the book stale — stop trading on it, re-bootstrap — because a book that might be wrong is worse than no book. Where the venue publishes checksums, I apply them per update; a mismatch is my corruption detector firing early.”
“Show me the book as it was at 14:32:07 last Tuesday. How?” “The book at a time isn’t stored anywhere — it’s derived: nearest captured snapshot at or before that moment, then replay my journaled deltas up to it. That presumes I built the capture: every normalized event journaled with venue sequence and my receive timestamp, plus periodic snapshots of my built book so the replay is bounded — a WAL plus checkpoints, same design as everywhere else in this stack. The senior half of the answer is the clock: 14:32:07 on whose clock? For best-execution and TCA it’s my receive time — the question is what I could have known when I routed, not what had objectively happened at the venue. And it has to be my own capture, not a vendor’s: a dispute asks what we saw, and a vendor archive is someone else’s clock with someone else’s gaps.”
“How do you know the rebuilt book is right — that you didn’t lose anything?” “Three layers. Completeness is sequence continuity: every delta carries the venue’s sequence number, and I gap-check twice — at capture, where a gap permanently marks that window degraded, and again at replay, where a gap means the archive itself is damaged. Unbroken sequence from the snapshot to the target is the proof nothing is missing — that’s what venue sequence numbers exist for. Correctness is hashes: where the venue publishes book checksums I recompute them during replay — the counterparty certifying my rebuild — and my own snapshots store a state hash the rebuild must reproduce bit-for-bit, which integer-tick prices make possible (the event-sourcing chapter’s determinism contract paying off again). And the systemic layer: adjacent snapshots verify the deltas between them — replay snap-14:00 plus its segment and it must hash-equal the independently-cut snap-15:00; run that over the whole archive nightly and the archive audits itself, with any failure localized to one segment of one symbol. Anything that fails any layer is served as degraded, never silently — same stale-book discipline as the live path.”
Further reading
- NASDAQ TotalView-ITCH 5.0 specification and MoldUDP64 (nasdaqtrader.com) — the canonical L3 feed and the multicast framing/retransmission layer under it; read both, they’re short.
- CME MDP 3.0 market-data documentation (cmegroup.com) — incremental + snapshot channels, A/B feed arbitration, and conflation at a tier-1 futures venue.
- Aeron documentation (github.com/aeron-io/aeron) — open-source high-throughput messaging with multicast and flow control; study its handling of slow receivers.
- Coinbase Exchange WebSocket feed documentation — a crypto venue’s public feed contract (full vs level2 channels, sequence numbers, snapshot recovery); compare its guarantees clause-by-clause against ITCH’s.
- Brian Nigito, “How to Build an Exchange” (Jane Street tech talk, YouTube) — includes the publisher-and-retransmitter side of the sequenced-log architecture.
Where this goes next: ch26 climbs one level up the stack — the broker that wraps many venues, where your SOR experience stops being background and becomes the whole job.
You Are the Broker: SOR Across Venues
Before you start. This chapter leans on:
- Exchange/venue anatomy — gateways, sequencer, matching engine — because the broker is a client of N of them: ch23
- Order gateways and sessions — the venue-side view of the connections your adapters maintain: ch24
- Market-data publishing — snapshot+incremental feeds, because your normalized-data layer consumes them: ch25
- Event sourcing and the determinism contract — the OMS/SOR here is an event-sourced system with compliance obligations: ch13
- Trading vocabulary (parent/child orders, TIF, tick size, TWAP/VWAP, and the rest of the dialect) — the decoder’s “Orders and execution” table: ch00d
Read those first — this chapter assumes all of them.
You already built a smart order router: cost-model routing across 20+ venues, ~10ms decision on CLOB legs (a leg is the per-venue piece of one trade), venue scoring, one desk consuming it. This chapter takes that exact system and asks the question a Talos-style firm will ask you in the first ten minutes: what changes when the SOR is a product serving N external clients instead of an internal tool serving one desk? The answer is not “add auth and a billing table.” Almost every hard property of the system — risk, fairness, state, compliance — multiplies in a specific, nameable way, and interviewers at execution-platform firms are testing whether you can name the multiplications.
The one-sentence framing to open with: a broker/execution platform is a multi-tenant SaaS whose tenants’ requests compete for the same scarce external resources (liquidity, venue rate limits) and whose every decision must later be defensible to the tenant and to a regulator. You have built the single-tenant version. This chapter is the delta.
The whole machine
Client A ─┐
Client B ─┤ FIX / WS / REST ┌───────────────────────────────┐
Client C ─┼─► [client gateways] ──► │ OMS │
... │ authn, sessions, │ parent orders, client accts, │
Client N ─┘ per-client rate │ per-client risk & buying │
limits, CoD │ power, allocation policy │
└──────────────┬────────────────┘
│ approved parents
┌──────────────▼────────────────┐
│ SOR / algo engine │
│ slicing, cost model, │
│ venue scoring, child orders │
└──────────────┬────────────────┘
│ child orders
┌────────────────────────────────▼───────────────────────┐
│ venue adapters × 20+ │
│ symbology map, order-semantics normalization, │
│ per-venue sessions, heartbeats, shared rate budget │
└───┬──────────┬──────────┬─────────────┬────────────────┘
▼ ▼ ▼ ▼
[venue 1] [venue 2] [venue 3] ... [venue 20+]
│ │ │ │
└──────────┴────┬─────┴─────────────┘
│ fills, acks, rejects
┌───────────────────▼───────────────────┐
│ normalized market data (your pipeline) │──► SOR cost model
└───────────────────┬───────────────────┘
│
[fills → allocation → client reporting → ledger → TCA (§below)]
The top half is new (multi-client OMS). The middle is your existing SOR. The bottom is your existing adapter + market-data layer, now with multi-tenant complications. The fills pipeline at the bottom grows from “update my position” to “allocate, report, invoice, and prove.”
OMS vs EMS vs SOR: untangling the triad
The industry uses these three acronyms with maximal sloppiness. Untangle them once and you sound native.
client intent ──► OMS ──► EMS ──► SOR ──► venues
what how where
- OMS — Order Management System — owns the what: the parent order as a durable business object — client, account, instrument, side, quantity, limit, instructions — plus client accounts, positions, buying power, and post-fill allocations. Analogy from your world: Stripe’s PaymentIntent — the durable object representing “customer wants to pay $50,” which survives retries, partial captures, and whatever routing happens underneath it. Why it matters: the OMS is the system of record; when everything else disagrees, the OMS’s ledger is what you reconcile against.
- EMS — Execution Management System — owns the how: working the parent over time — algo selection, slicing schedule, urgency. The algo names, in plain English: TWAP drips the order out evenly over N hours; VWAP drips it out proportional to when the market usually trades; POV never lets you be more than X% of current volume; liquidity-seeking hunts for size wherever it appears. Analogy: the retry/orchestration logic in a PSP that decides when and in what sizes to attempt captures — not what the customer owes, not which acquirer.
- SOR — Smart Order Router — owns the where: for one child slice, right now, which venue(s), at what price, given fees, latency, fill probability, and current books. Analogy: the routing step that picks an acquirer for a single authorization based on cost, auth-rate history, and health.
In practice the EMS/SOR boundary blurs (an aggressive multi-venue sweep is slicing and routing in one motion), and vendors sell combined “OEMS” products. The clean claim for interviews: you built an SOR with some EMS behavior (slicing, cost-model timing) and a thin single-tenant OMS (your desk’s position and risk state); productizing means building the OMS out into a multi-tenant system of record.
What multiplies at N clients
1. Risk moves in front of routing
One desk: the risk limits were your own, checked wherever convenient, and a breach hurt only you. N clients: per-client pre-trade risk — buying power, position limits, notional caps (notional = quantity × price, the dollar size of the order), price collars (defined in ch27), duplicate-order detection — must run before the SOR ever sees the parent, per client, with per-client configuration, and with the results logged. In the US this is literally law (SEC Rule 15c3-5, ch27): a broker providing market access must apply pre-trade financial and regulatory checks; “the client says they’re good for it” is not a control. Latency budget: these checks sit on the client’s critical path, so they get the same treatment as a venue’s risk gate — microseconds, in-memory limit counters, no database on the hot path, async persistence.
Buying power — the client’s remaining capacity for new exposure: cash/collateral, minus positions, minus the reserved exposure of working orders. The subtlety that trips people: open child orders reserve buying power the moment they’re sent and release it on cancel or fill — a hold/capture lifecycle. Analogy: card auth holds — an authorization reserves funds; capture settles; void/expiry releases — and the classic payments bugs (double-release, a hold leaked forever after a lost webhook) have exact counterparts in order-state reconciliation.
2. Client segregation is an engineering requirement, not a policy document
One client’s order flow is alpha — information you can trade profitably on; knowing A is buying big is a trading signal. To work an order is to execute it slice by slice; a thin book is one with little resting size, where a big order moves the price. If client B can infer that client A is working a large buy in a thin book — from timing, from shared-queue backpressure, from a support dashboard, from a log line — you have leaked information that B can trade against. Information barriers — controls preventing one client’s trading information from reaching another — sound like a compliance concept; in an execution platform they are concrete engineering:
- No shared mutable state whose observable behavior reveals another client’s flow. A shared unbounded queue where A’s burst delays B’s acks is a side channel. Per-client queues, fair-scheduled into shared downstream stages.
- Per-client authorization on every read path — reporting APIs, dashboards, support tooling. A support engineer’s “all open orders” view is itself an information-barrier surface, and access to it is audited.
- The platform’s own trading desk (if any) is the hardest wall. If the firm also trades principal, “the SOR operator sees everyone’s flow” is the FTX/Alameda lesson. Say this in interviews: the architecture must make leakage structurally hard — separate services, separate credentials, audited access — not just contractually forbidden.
Analogy: multi-tenant SaaS tenancy isolation, except a “data leak” here isn’t PII embarrassment — it’s directly monetizable against the victim, so the threat model is adversarial and partly internal.
3. Fair allocation is a product decision you must document
Two clients want the same liquidity: A and B both send buys in the same instrument, and venue X has one resting offer big enough for only one of them. Who gets it? At one desk the question doesn’t exist. At N clients it needs a written allocation policy — the documented rule for how competing orders share access to liquidity and how fills on aggregated child orders split back to parents. Common answers: strict time priority of parent arrival at the platform; pro-rata splits when the platform batches several parents into one child (proportional to size: A wants 10, B wants 4, the fill is 7 → A gets 5, B gets 2); never-aggregate (each parent gets its own children; venue queue position decides). Each is defensible; having no documented answer is not, because the disadvantaged client’s lawyer will ask. Analogy: a PSP splitting a partial settlement across merchants needs a deterministic, documented rule — ad-hoc splits are how you fail an audit. Engineering consequence: allocation must be deterministic and replayable — same fills in, same allocation out — your event-sourcing contract (ch13) applied to a new deterministic aggregation.
4. Pricing and fees enter the cost model — with a conflict of interest attached
Per-client fee schedules: some clients pay cost-plus (venue fees passed through plus an itemized platform fee), others all-in (one bundled rate; the platform keeps or eats the venue-fee difference). Why an engineer cares: under all-in pricing the platform has an incentive to route to cheap venues even when a pricier venue is better for the client — precisely the conflict best-execution rules (defined below) police. Your cost model now needs two outputs per candidate route — cost-to-client and cost-to-platform — and the routing decision must optimize the client’s number. Log both; the gap between them is exactly what a regulator asks about.
The venue adapter layer: the crown jewels
This is your daily craft, so own this section. Talos’s public positioning amounts to “we normalized dozens of venues so you don’t have to” — the adapter layer is the moat, because every venue integration is months of quirk-discovery a competitor must repeat.
What normalization actually means, from someone who has done 20+ of these:
-
Instrument master — the subsystem mapping every venue’s symbology into one canonical instrument space:
BTC-USDTvsBTCUSDTvsXBTUSDvs a numeric instrument ID; spot vs perp vs dated future as different canonical instruments even when a venue reuses a ticker; tick size (smallest price step), lot size (smallest quantity step), min notional (smallest dollar size accepted), contract multiplier (how many units one contract represents) per venue per instrument. This is a real service with its own storage, update pipeline (venues list and delist constantly), and versioning — a stale instrument master sends orders at the wrong tick and gets rejects, or worse, gets accepted at an unintended price scale. Analogy: currency-exponent reference data in payments — boring, and the source of the worst incidents when wrong (the “amount in cents vs units” bug class). -
Order-semantics normalization — the same order flag means different things at different venues:
Term What the client means How venues differ postOnlyrest in the book only, never trade on arrival venue A rejects the order if it would cross; venue B silently re-prices it to sit passive TIF (IOC / FOK) how long the order may live — IOC: fill what you can now, cancel the rest; FOK: fill all of it now or nothing flavors vary; some venues have no true FOK, so the adapter emulates it or refuses Iceberg show only a small visible slice of a big resting order support varies — native, absent, or emulated by the adapter STP flags self-trade prevention: never match against my own resting orders per-venue enums with different cancel-newest / cancel-oldest / cancel-both semantics The adapter exposes a canonical order-type set and, per venue, maps, emulates, or explicitly refuses. Silent approximation is the sin: the client asked for FOK semantics and got something else.
-
Per-venue health: heartbeats, sequence-gap counters, ack-latency histograms, reject rates by reason — feeding the venue score the SOR consumes. You built this; say so, with numbers.
-
The new multi-tenant problem — shared rate budgets. Venue X allows the platform 100 orders/sec total. That budget is now shared by N clients, and one client’s algo burst can starve everyone else’s cancels — and a starved cancel is a risk event, not an inconvenience. Design: per-client sub-budgets inside each venue budget (weighted fair queuing — each client gets a guaranteed share of the 100/sec, and unused share is redistributed; cancels strictly prioritized over new orders — cancels must never queue behind entries), burst allowances, and throttling surfaced to the client as an explicit signal rather than silent queuing. Analogy: an API gateway doing per-tenant rate limiting inside a global upstream quota — except the upstream quota is a hard external constraint and “just queue it” changes execution prices.
Where the HFT skillset pays on the broker side — and where it doesn’t
You came to this book for the HFT toolbox: kernel tuning, lock-free queues, µs measurement. The broker seat uses all of it — but the payoff is asymmetric, and knowing where it pays is itself the senior skill. Start the way the kernel-tuning chapter (ch03) taught you: with the budget table, not with the coolest tool.
| Segment of a broker’s order path | Typical cost | Can engineering shrink it? |
|---|---|---|
| WAN hop to an internet crypto venue | ~1–70ms | No — buy placement (region, colo), don’t code |
| Venue’s own processing | ~ms | No — it’s their machine |
| Your adapter/gateway path (sign, session, submit) | ~1–50ms | Yes — usually the biggest controllable term |
| Your SOR decision loop | µs–ms | Yes — the classic hot-path skillset |
| Your feed ingestion + normalization | µs–ms | Yes — same |
The rule is the one from the kernel-tuning chapter (ch03): optimize the biggest controllable term first. For an internet-venue broker that’s almost never the kernel — it’s warm sessions, pre-computed auth, and not blocking the decision loop. The µs disciplines still transfer wholesale; they just aim at different segments.
Why speed converts to money for a broker — three concrete mechanisms:
- A stale book is a lying cost model. The SOR splits parents using the normalized books. If one venue’s feed is seconds stale (silent WS death, missed reconnect), you route into prices that no longer exist — rejects, slippage, re-plans. Fast, health-checked market data is routing correctness. Hence: staleness stamps on every update, per-venue feed-lag tracking, and a hard rule — a venue you can’t currently see is a venue you don’t route to.
- Slow orders eat adverse selection. Between the routing decision and the child’s arrival at the venue, the market moves — and it moves against you more often than chance, because the counterparties who rest orders are watching too. Every ms shaved off the adapter path is slippage not paid. This is tick-to-trade discipline with the finish line moved: decision-to-venue-ack, measured per venue with the ack-RTT EWMA feeding straight back into the cost model as a latency handicap.
- Slow terminal-state detection freezes capital and blocks re-plans. The double-fill race (below) forces the discipline that unfilled size can only move once a child is terminal — filled, canceled, or rejected, with nothing in flight that could still execute. The faster you process acks and fills, the faster parents complete and the less reserved buying power sits idle.
And the build order the whole book has been teaching: measure first, change behavior second. Feed staleness stamps, ack-RTT EWMAs, and TCA logging change nothing about routing — they build the scoreboard. Only when the scoreboard exists do you let it drive behavior (health demotion, latency penalties, re-route policy), because otherwise you cannot tell whether any of the fast-path work paid. That’s the lesson of the latency-methodology chapter (ch08) wearing a broker suit.
Anticipating the crypto-reality section below: with internet venues, a ~10ms decision loop is genuinely adequate, and the edge is venue knowledge — quirks, health, credit — not nanoseconds. The HFT skillset’s biggest broker-side dividend isn’t raw speed; it’s the discipline — measure everything, never block the hot path, make every state transition explicit — applied to a system whose scoreboard is TCA instead of tick-to-trade.
The synthetic book: one market view, published to N clients
Your adapters maintain a normalized book per venue. The SOR reads them. The next product step — and a favorite interview design question — is publishing a consolidated (synthetic) book: one merged view of all N venues, streamed to clients. You become, from the client’s perspective, the venue — which means the market-data-publishing chapter (ch25) now applies to you, plus some broker-specific honesty rules.
Hold the pipeline as one straight line first:
venue A ws ─► adapter A ─► ΔA ─┐
venue B ws ─► adapter B ─► ΔB ─┼─► one queue ─► MERGER ─► Delta{mseq,…} ─► PUBLISHER ─► clients
venue N ws ─► adapter N ─► ΔN ─┘ (per symbol) (one thread) │
│ └─► journal (evidence)
└─► merged book, read by the SOR
Left to right: each adapter turns its venue’s feed into normalized book changes (ΔA = “on venue A, bid level 10001 now has 400”). All changes for a symbol funnel into one queue. One merger thread drains it, updates the merged book, and — the step most descriptions skip — emits deltas as a by-product of applying changes. The publisher fans those deltas out. That’s the whole machine.
Where a delta actually comes from
You never receive a delta for the merged book — you manufacture it, in the merger, as the diff your own update caused. Your mini-market lab already does exactly this on the venue side: MatchOut carries the book deltas that a match produced (ch28, Step 2). The broker version is the same move one level up.
Trace one update end to end. State before: venue A shows 250 at bid 10001, venue B shows 150 at bid 10001. The merged level is therefore 10001 → {A: 250, B: 150}, total 400, and that total is what clients currently see.
- Venue A’s websocket delivers: “bid 10001 now 400” (A’s own book-delta format, whatever it is).
- Adapter A normalizes it — canonical symbol, integer ticks — updates its local book A replica, and pushes one message to the merger’s queue:
BookChange { venue: A, side: Bid, px: 10001, qty: 400 }. - The merger applies it to the merged level:
{A: 250→400, B: 150}. Total was 400, is now 550. The total changed, so the merger emits:Delta { mseq: 8813, side: Bid, px: 10001, total: 550, by_venue: {A: 400, B: 150} }— and increments mseq. - If the total had not changed (say A revised 250→250 metadata, or the change only touched a depth tier you don’t publish), no delta is emitted. The merged book moved; the published view didn’t; clients hear nothing.
Two design choices hiding in step 3, both worth saying in an interview: deltas carry the absolute new total (“level now has 550”), not the increment (“+150”) — absolute levels are idempotent, so a client that somehow applies one twice is still correct, and conflation (below) becomes trivial. And mseq is assigned inside the single merger thread, which is what makes the stream gap-detectable: a client holding mseq 8813 that receives 8815 knows it missed one.
The structs, in the lab’s style:
#![allow(unused)]
fn main() {
struct BookChange { venue: VenueId, side: Side, px: Px, qty: Qty } // adapter → merger
struct MergedLevel { total: Qty, by_venue: SmallMap<VenueId, Qty> } // merger state, per px
struct Delta { // merger → publisher → clients; also the journal record
mseq: u64,
side: Side, px: Px,
total: Qty, // absolute: "this level now has"
by_venue: SmallMap<VenueId, Qty>, // attribution tier only
}
struct Snapshot { mseq: u64, bids: Vec<(Px, MergedLevel)>, asks: Vec<(Px, MergedLevel)> }
}And the merger loop is a fold, nothing more:
#![allow(unused)]
fn main() {
loop {
let ch = rx.recv(); // one queue in, one thread
if stale(ch.venue) { book.evict(ch.venue); emit_evict_deltas(); continue; }
let lvl = book.level_mut(ch.side, ch.px);
let old_total = lvl.total;
lvl.by_venue.insert(ch.venue, ch.qty); // qty 0 removes the venue's contribution
lvl.total = lvl.by_venue.values().sum();
if lvl.total != old_total { // published view changed?
let d = Delta { mseq: next_mseq(), side: ch.side, px: ch.px,
total: lvl.total, by_venue: lvl.by_venue.clone() };
journal.append(&d); // evidence first
pub_tx.send(d); // then fan out
}
}
}How publishing actually works
The publisher owns one bounded outbound queue per client (never shared — one slow client must not delay another, and observable backpressure is an information-barrier leak, per the segregation section above). The loop:
- Fast client: every
Deltais pushed to its queue; the socket writer drains it in mseq order. The client appliesbook[px] = total, deleting the level when total is 0. That’s the entire client-side algorithm — a consequence of absolute-total deltas. - Slow client (queue full): switch that client to conflation mode. Stop queueing every delta; instead keep a per-client dirty set of price levels touched since it last kept up. When its socket drains, walk the dirty set and send one delta per level with the current total — the flickers in between are gone, the end state is identical. This is why absolute totals matter: conflating relative increments would require summing them; conflating absolute levels is “just send the latest.”
- Trades are never conflated — trades are facts (ch25’s rule). If a client can’t keep up with the trade stream either, disconnect it; it re-enters through the snapshot door like any late joiner.
- Snapshot service: every K deltas (or T ms) the publisher serializes the merged book as
Snapshot { mseq, … }. A connecting client gets: latest snapshot, then every delta withmseq > snapshot.mseq, then the live stream. Gap detected mid-stream → client re-requests a snapshot. Exactly the late-joiner contract you demand from venues, now offered by you. - Entitlements are filters on the way into each client’s queue: depth tier (L1 only? ten levels? full ladder), update-rate tier (real-time vs 100ms-conflated — the conflation machinery doubles as the product knob), and whether
by_venueattribution is included or stripped. ch25’s L1/L2 product ladder, yours to sell.
“But ch25 rebuilds the feed from the sequencer — we don’t have one”
Half right, and the half matters. You have no sequencer of the market — reality already happened N times, at N venues, each with its own sequence space, and the arrival order of their updates at your doorstep is a race with no true answer (the lab’s Step 7 nondeterminism). But look at the merger loop above: one thread, draining one queue, stamping mseq on each emitted delta. That is a sequencer — of your published view, not of the market. The venue’s seq answers “what order did the market happen in”; your mseq answers “what order did we show the world to our clients in” — and the second question is precisely the one best-execution evidence needs.
The distinction rewrites recovery too. The venue replays its log because the log is reality. Your merged book is derived state — always reconstructible from upstream — so recovery is re-derivation, not replay: adapters re-snapshot from their venues (the gap-fill machinery they already have), the merger rebuilds and bumps an epoch, the publisher emits a fresh snapshot, every client bootstraps from it. The journal the merger writes (journal.append above, before fanout) exists for a different job: evidence. “What did clients see at 14:32:07” = binary-search the journal by time, replay deltas since the prior snapshot — same query shape as the venue’s explain tool (ch23), answering display instead of matching. The venue event-sources because its log is the truth; you journal because you must prove what you displayed.
Merge honesty rules
Each merged level keeps per-venue attribution — 10001 → {A: 400, B: 150} — because the SOR routes on it and sophisticated clients pay for it. Beyond that, four decisions make the book “synthetic” rather than merely summed:
- Crossed books are real. Venue A’s bid at 10002 above venue B’s ask at 10001 happens legitimately — latency skew, or fees that make the “arbitrage” unprofitable. Publish it as-is and flag it; “de-crossing” the view means publishing prices nobody can trade.
- Raw prices, not fee-adjusted. B’s 10001 plus a 20 bps taker fee can cost more than A’s 10003 at zero. You could publish an effective-price book — but fees are per-client-tier, so that book is different for every client. Standard answer: publish the raw consolidated book to everyone; the fee arithmetic lives where it already lives, in the SOR’s cost model.
- Staleness eviction. A venue whose feed has gone stale leaves the merge entirely — otherwise you are publishing phantom liquidity, levels that stopped existing seconds ago. Same rule the router follows: a venue you can’t currently see is a venue whose liquidity you don’t show.
- Self-exclusion. Your own resting child orders sit on those venues. The naive merge shows your clients your own orders as market depth — flag or subtract broker-own liquidity, or client B can be routed into crossing with client A’s order through the venue, which is the self-match problem wearing market-data clothes.
And one thing to teach clients (it will come up in their TCA reviews): ghost liquidity. The same market maker quotes on all N venues; the merged book shows several times the real depth; sweep every level at once and the maker pulls the other venues the moment the first fill prints. Consolidated depth is an upper bound, not a promise — which is precisely why post-fill markout lives in the venue scorecard.
The one-source-of-truth rule. The published book must be derived from the same merged state the SOR reads. If clients see book X while the router routed on book Y, every best-execution conversation becomes unwinnable. One merged state, one sequence — and that state snapshot is exactly what the decision log (below) records, so “what did the market look like at 14:32:07” is one lookup, serving client support, TCA, and the regulator alike.
The two-sided order state machine
One desk: one state machine per order — you versus the venue. Platform: every parent order has a client-facing state (what you’ve told the client: NEW → ACKED → PARTIALLY_FILLED → FILLED / CANCELED) and a set of venue-facing states (one per live child per venue), evolving asynchronously. The parent state is a fold: an aggregation over child states plus OMS decisions.
client view: parent: BUY 100 BTC [ACKED, filled 37.5]
▲
aggregation │ allocation
│
platform view: child 1 ──► venue A FILLED 20
child 2 ──► venue B PARTIAL 12.5, working 12.5
child 3 ──► venue C CANCEL_PENDING (re-route in flight)
child 4 ──► venue D NEW_PENDING (sent, no ack yet)
Reconciliation between the two sides is continuous, not end-of-day: every venue execution report updates a child; every child update recomputes the parent; and a periodic sweep compares platform-believed child state against venue-reported open orders (venues expose order-status queries for exactly this). Ambiguous states — order sent, no ack, session dropped — get the payments treatment you know cold: the order is state-unknown, never assumed dead, and must be resolved by query or cancel-with-confirmation before its reserved quantity is released.
The double-fill race
The canonical platform-SOR failure, and a guaranteed interview question. Sequence: the child on venue A isn’t filling → the SOR re-routes → cancel to A, new child to B → A’s fill arrives after the cancel was sent (the cancel lost the race to a match already through A’s sequencer). Both A and B fill: the client bought more than they asked for.
- Prevention: cancel-ack discipline. Never send the replacement child until venue A confirms the cancel and reports final filled quantity (a proper cancel-ack carries cumulative fill). Cost: one venue round-trip added to every re-route — 10–100ms on internet crypto venues. This is the correct default, and it is exactly the idempotency discipline from payments: don’t retry the charge until the first attempt’s outcome is known, because “probably failed” is how double-charges happen.
- Mitigation when speed forces optimism: if an aggressive algo routes to B before A’s cancel-ack, cap B’s child at parent-remaining assuming A fully fills, and run an over-fill handler: excess lands in a platform error account, gets traded out, with explicit policy (and disclosure) on whether a client ever wears an over-fill. The error account is a real subsystem with its own P&L and audit — the reconciliation suspense account of your payments world.
Say the payments version out loud in interviews — “this is the double-charge problem, except the retry moves the market” — it lands.
Best execution: your event-sourcing instinct, now legally mandated
Best execution — the broker’s obligation to take all sufficient (MiFID II, the EU regime) or reasonable (FINRA, the US regime) steps to obtain the best possible result for the client, weighing price, cost, speed, likelihood of execution and settlement, and size; both regimes are toured in ch27. The architectural forcing function: you must be able to prove, after the fact, that each routing decision served the client. So every SOR decision is logged with the market snapshot it saw — per-venue books, venue scores, cost-model inputs and outputs, at decision time — because “we routed to venue B” is only defensible as “and here is what every venue’s book looked like at that moment, and here is the arithmetic.” You built decision logging for debugging and venue scoring; here it is the compliance artifact. An event-sourced SOR gets this nearly free: the decision log is the log.
TCA — Transaction Cost Analysis — measuring execution quality after the fact — is both your internal feedback loop and a client-facing product:
- Slippage vs arrival: average fill price vs mid-market (the midpoint between best bid and best ask) at parent arrival — the headline number (“your 100 BTC buy cost 4.2 bps vs arrival”).
- vs VWAP / participation benchmarks for scheduled algos.
- Venue scorecards: per-venue fill rates, effective spread, reject rates, post-fill markout (does the price move against you right after fills there?). Markout is a toxicity signal: toxic flow is counterparties who only trade with you when you’re about to lose. You already compute venue scores for routing; TCA is the same data productized into client reports.
Analogy: the auth-rate and cost dashboards a PSP shows merchants to justify its acquirer routing — same data, same “we route in your interest, here’s proof” purpose.
Failure modes the platform must survive
- Venue down mid-parent. Children on the dead venue go state-unknown. Playbook: mark the venue unroutable (score → 0); do not release the unknown children’s reserved quantity; keep working the parent elsewhere only up to remaining-minus-unknown; reconcile on venue recovery (order-status queries or the venue’s recovery feed reveal what happened in the dark). The conservative quantity arithmetic is the whole game — optimism here is the double-fill race at venue scale.
- Client disconnect with live children. Per-client policy, configured up front: cancel-on-disconnect (safe default for takers), or keep-working — an algo running a 6-hour TWAP shouldn’t die because the client’s monitoring session dropped; the OMS owns the parent, not the client’s TCP connection. The session-vs-order-ownership distinction is the design point; ch24 covers the venue-side mirror.
- The platform’s own kill switches, layered: per-client (their breach → halt their flow, cancel their children), per-venue (venue misbehaving → stop routing, mass-cancel there), global (platform incident → everything stops, mass-cancel everywhere). Each layer gets its own drilled big red button, and the mass-cancel path must be the fastest path in the system (ch27).
Crypto spice: the venue is also a counterparty
In tradfi, the broker’s venue risk is mostly operational — central clearing means settlement risk sits with a clearinghouse, which legally becomes the buyer to every seller (and seller to every buyer), so a member’s default is the clearinghouse’s problem, not yours. In crypto, the venue holds your assets: pre-funding means the platform (or its clients) keeps balances on each exchange, and an exchange failure is a credit loss, not an outage. FTX made it concrete: routing 100% of flow to the venue with the best prices was catastrophic when that venue was also insolvent.
Engineering consequences:
- The venue score includes credit and counterparty terms, not just latency and fees: withdrawal-latency monitoring (withdrawals quietly slowing is the canonical early-warning signal), proof-of-reserves posture, jurisdiction, and hard concentration caps (“never more than X% of platform assets on one venue”). Your venue scorer grows slow-timescale risk inputs alongside the fast microstructure ones.
- Treasury/rebalancing as a first-class subsystem: moving balances between venues so routable inventory sits where the flow is, netting against withdrawal fees and on-chain confirmation times (minutes, sometimes hours). This is the multi-currency treasury problem from payments — prefund the local rails where the volume is, sweep to safety otherwise.
- Stablecoin/fiat legs:
BTC-USDon one venue andBTC-USDTon another are different instruments with an FX-like basis (a persistent price gap, like a currency pair that never quite sits at 1.00); the SOR either keeps them as distinct books or explicitly models the USDT/USD leg. Pretending stablecoin = USD is a routing bug with a case study: in the March 2023 USDC depeg, routers that hardcoded $1.00 “arbitraged” themselves into depegged inventory. - 24/7, no close: no end-of-day window for reconciliation, upgrades, or resets. Every maintenance operation is a live operation (ch16); reconciliation is continuous; “we’ll fix it after the close” is not in the vocabulary.
Plain-English recap
- A broker/execution platform is your one-desk SOR wrapped in a multi-tenant OMS: OMS = the durable what (PaymentIntent), EMS = the how (retry/orchestration), SOR = the where (acquirer selection).
- Going from 1 desk to N clients multiplies four things: risk moves before routing and becomes per-client and (in the US) legally mandatory; client flow must be segregated like adversarial tenant data; competing clients need a documented, deterministic allocation policy; and fees enter the cost model carrying a conflict of interest that best-ex rules police.
- The venue adapter layer — instrument master, order-semantics normalization, health scoring — is the product moat; its new multi-tenant problem is fairly sharing each venue’s rate limit across clients, with cancels always winning.
- The HFT skillset transfers to the broker seat with the finish line moved: speed pays through routing correctness (never route on a stale book), less adverse selection (decision-to-ack, EWMA-scored per venue), and faster re-plans (terminal-state discipline) — measured first, behavior-changing second, with TCA as the scoreboard instead of tick-to-trade.
- Every parent order is two state machines — client-facing and venue-facing — continuously reconciled; the double-fill race on re-route is the double-charge problem, and cancel-ack discipline is its idempotency key.
- Best execution turns decision logging into a legal obligation: every routing decision stored with the market snapshot it saw; TCA (slippage vs arrival, venue scorecards) is the client-facing proof.
- In crypto the venue is also a counterparty: withdrawal monitoring, concentration caps, and treasury rebalancing belong in the router’s venue score, next to latency and fees.
Interviewer will ask
Q1: “You built an SOR for one desk. What actually changes when it serves external clients?” I’d name the multiplications rather than list features. Risk checks move in front of the router and become per-client and mandatory — buying power with hold/release semantics on working orders, which is card-auth-hold logic I’ve built before. Client segregation becomes an engineering property: no shared state whose timing or backpressure leaks one client’s flow to another. Allocation between competing clients needs a documented, deterministic, replayable policy — at one desk that question doesn’t exist. And every routing decision becomes evidence: logged with the market snapshot it saw, because best execution means proving the route served the client. My existing SOR — cost model, venue scoring, 20+ adapters — is the engine; the productization is the OMS shell and the fairness-and-evidence layer around it.
Q2: “Untangle OMS, EMS, and SOR.” OMS owns the what: the parent order as a durable business object, plus accounts, buying power, allocations — the system of record, like a PaymentIntent in Stripe. EMS owns the how: working the parent over time — algo choice, slicing schedule, urgency. SOR owns the where: for one child right now, which venue, given fees, books, and fill probability — the acquirer-selection step. In practice EMS and SOR blur into one engine, and what I built was an SOR with EMS behaviors and a thin single-tenant OMS. The multi-client OMS is the part I’d build fresh, and I’d build it event-sourced, because allocation determinism and best-ex evidence both demand exact replay.
Q3: “Venue X gives you 100 orders/sec total. Client A’s algo wants all of it. Go.” A doesn’t get all of it, and here’s the second-by-second version. Say A’s contracted share is 40 of the 100. A’s algo bursts 100 orders in one second: the first 40 pass. If B and C are quiet this second, their unused share redistributes, so A might actually get 85–90 — but never the last slice, because a floor stays reserved so that B’s next order doesn’t have to wait behind A’s burst. Whatever A sends beyond its share comes straight back rejected with an explicit “throttled by platform” reason — not silently queued. Why not queue it? A queued order executes later at a different price; silently changing a client’s execution price is a best-execution violation, but an explicit reject lets A’s algo make its own choice: re-pace, or route the flow to another venue. Now the moment that matters: mid-burst, B sends a cancel. The cancel does not join any line — cancels preempt new orders, always, because a cancel that queues behind A’s entries means B is locked into market risk they’re trying to exit. That’s a risk event, not a fairness event. So the picture is API-gateway per-tenant rate limiting with three trading-specific amendments: the upstream quota is hard (the venue enforces it), cancels preempt, and throttling is explicit. I ran per-venue rate budgets for one desk; the per-client scheduler that decides whose order passes this second is the genuinely new layer.
Q4: “Walk me through the double-fill race and your answer to it.” Child on venue A, not filling; I re-route: cancel to A, new child to B. A’s fill was already through its sequencer when my cancel arrived, so both venues fill and the client is over-bought. Prevention is cancel-ack discipline: don’t send B’s child until A confirms the cancel with final cumulative quantity, paying one venue round-trip per re-route. Same idempotency rule as payment retries — never retry until the first attempt’s outcome is known, because “probably failed” is how double-charges happen. And that round trip is exactly where the speed work pays: the faster I turn A’s cancel-ack into a terminal state, the sooner the parent’s reserved quantity is free to move — the cost buys certainty, and the ack-path speed buys the cost back. Where an aggressive algo can’t wait, I cap the new child assuming worst-case fill on A and run an error account for over-fills, with explicit policy on whether a client ever wears one. Prevention is the default; optimism is opt-in and accounted for.
Q5: “A client claims a fill wasn’t best execution. Prove them wrong — or right.” The proof exists because I decided to log for it on day one: the SOR is event-sourced and every routing decision carries its full input snapshot — per-venue books, fees, health scores, cost-model output — keyed by decision time. So “why venue B at 14:32:07” gets a replayable answer: here’s what every venue showed, here’s the arithmetic, B won on all-in client cost including expected slippage. TCA closes the loop: that parent’s slippage vs arrival, and B’s scorecard showing the fill wasn’t an outlier. It cuts both ways: the replay can show the router was wrong — and that’s what the scoreboard exists for. The staleness stamp on every book update and the per-venue ack-RTT EWMAs sit inside the decision snapshot too, so the same log that defends a good route exposes a stale feed or a bad score when that’s the true story — and then it’s how I fix it. I built this logging at Crypto.com for debugging; the upgrade is treating it as a compliance artifact with retention and tamper-evidence.
Q6: “How is routing crypto different from routing equities?” Three structural differences. First, the venue is a counterparty: pre-funded balances mean venue failure is a credit loss, so my venue score carries slow risk signals — withdrawal-latency trends, concentration caps — next to the fast microstructure ones; FTX is why “best price” can’t be the only axis. Second, the instrument space is messier: BTC-USD and BTC-USDT are different instruments with a real basis, so the router keeps them as separate books or explicitly models the stablecoin leg — hardcoding a stablecoin at $1.00 is a bug with a 2023 case study, the USDC depeg. Third, 24/7 with no close: reconciliation, treasury rebalancing, and deploys are continuous live operations. The compensation is that crypto venues are internet-distant, so my ~10ms routing budget was genuinely adequate — the edge was venue knowledge, not nanoseconds.
Q7: “Two clients’ buys compete for one resting offer. Who gets it, and how do you defend that?” Whatever the answer, it must be written, deterministic, and replayable — the indefensible position is having no rule. My default: strict time priority of parent arrival at the platform — simple, incentive-compatible, and it mirrors what the venue itself does. If the platform aggregates parents into shared children, fills allocate back pro-rata by a documented formula computed inside the event-sourced fold, so the same fills always produce the same allocation. Allocation decisions get the same evidence logging as routing decisions, because the disadvantaged client in a fast market is exactly who audits you. And I’d want product and compliance in that design review — the policy is a business commitment, not an engineering preference.
Q8: “A venue goes dark with your child orders live on it. Next 60 seconds?” Immediately: venue marked unroutable, its children flip to state-unknown, and — the key move — their reserved quantities are not released. The parent keeps working on other venues only up to remaining-minus-unknown, so the worst case (every dark order filled) cannot over-fill the client. Affected clients get the honest state: quantity X unconfirmed on venue Y. Then reconciliation on reconnect: order-status queries or the venue’s recovery feed resolve the dark window, and reserved quantities settle into fills or releases. The discipline, from having lived venue outages across 20+ integrations: unknown quantity is treated as filled for risk purposes and as nothing for revenue purposes — conservatism points in exactly one direction.
Q9: “You come from HFT infrastructure. Where does that skillset actually pay at a broker — and where doesn’t it?” I’d start where the kernel-tuning chapter (ch03) taught me to start: with the budget table, not the tool. Cost each segment of a child order’s path: the WAN hop to an internet venue is 1–70ms, and you buy that down with placement — region, colo — you don’t code it away; the venue’s own processing is theirs; my adapter path — sign, session, submit — is 1–50ms and entirely mine. So the rule is optimize the biggest controllable term, and for an internet-venue broker that’s almost never the kernel — it’s warm sessions, pre-computed auth, and a decision loop that never blocks. That’s where the milliseconds live. Speed then converts to money through three specific mechanisms: a stale book is a lying cost model, so fast health-checked market data is routing correctness; every ms on the adapter path is adverse selection, because the market moves against you between decision and arrival; and slow terminal-state detection freezes reserved buying power and blocks re-plans. But the build order is measure first — staleness stamps, ack-RTT EWMAs, TCA logging change no routing behavior at all; they build the scoreboard — and only once the scoreboard exists does it drive health demotion and latency penalties, because otherwise I can’t tell whether any of the fast-path work paid. With internet venues my ~10ms loop was genuinely adequate, and the edge was venue knowledge — quirks, health, credit. What the HFT seat really hands the broker seat is the discipline — measure everything, never block the hot path, make every state transition explicit — pointed at a scoreboard that reads TCA instead of tick-to-trade.
Further reading
- SEC Rule 15c3-5 (Market Access Rule) — the adopting release (SEC Release 34-63241) is readable and is the canonical statement of “pre-trade risk at the broker is mandatory.”
- MiFID II best-execution materials — ESMA’s best-execution Q&As and the (now-retired) RTS 27/28 reporting regime, for what “prove your routing” means operationally in the EU.
- FIX Trading Community specifications — ExecutionReport, order-state, and allocation message semantics; reading the FIX order-state model is the fastest way to internalize the two-sided state machine.
- Talos engineering blog and product documentation — the closest public description of the multi-client execution-platform architecture this chapter describes.
- Larry Harris, Trading and Exchanges — the chapters on brokers and order routing; old, but the conceptual frame for best execution and agency conflicts is unchanged.
Where this goes next: every arrow in this chapter’s diagram crosses a risk check — Chapter 27 specifies the control plane both venue and broker must carry: pre-trade budgets, kill switches, drop copy, surveillance, and the regulators forcing all of it.
Risk, Limits & the Regulator — Both Sides
Before you start. This chapter leans on:
- Venue anatomy — gateway → sequencer → matching engine, because pre-trade risk has an exact seat in that pipeline: ch23
- Gateways and sessions — cancel-on-disconnect and mass-cancel live at the edge: ch24
- The broker/OMS side — per-client risk before routing: ch26
- Event sourcing — surveillance and regulator queries run on the sequenced log: ch13
- Change management — kill switches and replay-tested config, extended here to risk limits: ch17
Read those first — 20 minutes there saves an hour here.
Every trading system you’ll interview about — venue or broker — carries a second system inside it: the control plane that decides what is allowed to trade, stops it when something goes wrong, and can prove afterwards what happened. This chapter is that control plane, from both sides of the wire. It’s also where your payments background pays off most directly, because the shapes — pre-auth checks, idempotency keys, the big red pause button, triple reconciliation, suspicious-activity monitoring — are the shapes you already operate; the vocabulary and the latency budgets are what’s new. Interviewers use this material to separate “built a fast thing” candidates from “could be trusted with production” candidates.
The control plane, both sides
BROKER SIDE VENUE SIDE
─────────── ──────────
client order participant order
│ │
[OMS pre-trade risk] [gateway pre-trade risk]
buying power, limits, price collar, size, rate,
duplicate check ── SEC 15c3-5 ── self-match, session limits
│ │
[SOR] ──► child orders ──────────────────► [sequencer] ─► [matching engine]
│ │
│◄──────── acks / fills ───────────────────┤
│ │
[position keeper] [drop copy] ──► clearing/risk
│ │
[recon: internal book vs venue [surveillance jobs on the
statements vs custodian] sequenced log: wash, spoof]
│ │
[kill switches: per-client / [kill switches: per-session /
per-venue / global] per-symbol / market-wide]
Same skeleton on both sides: an in-line gate before anything reaches the market, a real-time copy of what happened flowing to watchers, reconciliation loops proving the books agree, and a fast path to stop everything. The regulators (bottom of the chapter) are there to force each box to exist.
Pre-trade risk: an in-line latency budget
Pre-trade risk — the checks that run on the order’s critical path, before it can reach a matching engine — is the one part of the control plane that costs latency on every single order, so it’s engineered like a hot path, not like a compliance afterthought. The venue runs it in the gateway before the sequencer (a bad order must never consume a sequence number it didn’t deserve — rejects don’t enter history); the broker runs it in the OMS before the SOR (a breach must never reach a venue). Why rejects must stay out, restated from ch13: the sequenced log is the replayable official history every downstream system folds over, so a reject inside it would make garbage part of the permanent record.
The standard battery, with realistic per-check budgets on a venue-grade gateway (nanoseconds):
| Check | What it stops | Typical cost |
|---|---|---|
| Price collar / band — reject orders further than X% (or N ticks) from reference price | fat fingers, decimal-place bugs | ~10–20 ns (two compares vs cached band) |
| Max order size / notional cap | “sell 1,000,000” instead of “1,000” | ~5–10 ns |
| Position + open-order limit — current position ± all working orders must stay inside limit | slow-bleed breaches across many small orders | ~20–50 ns (atomic read + add) |
| Credit / buying-power check | trading beyond collateral | ~20–50 ns against an in-memory counter; the update pipeline behind it is the hard part |
| Rate limit (orders/sec per session) | runaway algos, DoS on the book | ~10 ns (token bucket) |
| Self-match prevention | wash-looking prints from one firm’s own flow | ~50–100 ns (check resting-order ownership at the touch — the best bid/ask, where the next trade happens) |
| Duplicate / replay detection via client order ID | double-submits after timeouts/reconnects | ~50–100 ns (hash-set probe on recent IDs) |
Total budget: a few hundred nanoseconds venue-side — small against a ~5–50 µs gateway-to-ack path, which is exactly the argument you make when someone proposes skipping checks “for latency.” A crypto platform on cloud hardware runs the same battery at low-microsecond cost, and that’s still fine: the WAN hop to the venue, measured in milliseconds, dwarfs it.
Two of these are old friends renamed. Client order ID dedup is the idempotency key: the client stamps each order with a unique ID; a resubmit after an ambiguous timeout hits the dedup set and returns the original outcome instead of a second order — the same mechanism that stops double-charges in payments, for the same reason (retries against an uncertain outcome are mandatory, so the server must make them safe). The credit check is the auth-hold pipeline: reserve on order entry, release on cancel, convert on fill, and the failure modes are leaked holds and double-releases, found by reconciliation.
Self-match prevention (SMP) deserves its own beat because interviewers probe the policy options. When an incoming order from firm F would match F’s own resting order, the venue can: cancel-newest (reject/cancel the incoming — the aggressor loses), cancel-oldest (cancel the resting order, let the incoming trade on — the aggressor keeps its intent), or cancel-both. Venues offer these as flags because different participants want different semantics (a market maker re-quoting wants cancel-newest; an algo sweep wants cancel-oldest). Two words first: a print is an executed trade appearing on the public feed, and the feed’s running record is “the tape.” Why SMP exists at all: self-matches print volume that looks like wash trading (surveillance) and can be used to paint the tape, so the venue prevents them mechanically rather than adjudicating intent afterwards.
Implementation craft: the limit counters these checks read are updated from multiple flows (entries reserve, cancels release, fills convert) while being read on every order. Venue-side that means the padded-atomics discipline from your hot-path work (ch11) — per-session counters cache-line-aligned to kill false sharing (padded so no two counters share a 64-byte line), relaxed loads on the check path (the cheapest atomic read — safe here because the counter isn’t a synchronization primitive), no locks anywhere near the gateway. The reference data (bands, limits) is versioned config swapped in atomically by pointer — never a mid-order partial update.
Stopping: kill switches, cancel-on-disconnect, mass-cancel
Kill switch — a pre-built, tested control that halts order flow at some scope — is a taxonomy, not a single button. Both sides layer them:
| Scope | Venue side | Broker side | Who can pull |
|---|---|---|---|
| Session/client | disable one session, cancel its orders | halt one client, cancel their children | risk desk, on-call, the client themselves |
| Symbol/venue | halt one instrument | stop routing to one venue + mass-cancel there | ops, automated circuit breakers |
| Firm/global | disable a participant firm entirely | stop everything, cancel everything everywhere | senior risk officers, named individuals |
Design rules that interviewers listen for: the kill path is pre-authorized and drilled (MiFID II RTS 6 literally mandates kill functionality and periodic testing — a kill switch you’ve never pulled is a rumor, same as an untested backup); pulling it must not require the system being killed to cooperate (out-of-band path to the venue: a dedicated mass-cancel endpoint, or the venue’s own participant portal); and it’s cancel then investigate, never the reverse. Analogy: the “pause payouts” button every payments platform builds after its first fraud incident — pre-wired, permissioned, logged, and the postmortem question is always “why did it take N minutes to press.”
Cancel-on-disconnect (CoD) — the venue automatically cancels a session’s resting orders when its connection drops — is the default safety rail both sides negotiate. The subtlety: CoD triggering on a network blip while your strategy is fine mass-cancels your queue position (expensive); CoD not being armed while your strategy is dead leaves stale quotes in the market getting picked off (more expensive). So CoD is per-session configuration with heartbeat-timeout tuning, and the broker-side mirror (ch26) is the client-disconnect policy: takers get CoD, long-running algos keep working because the OMS owns the parent, not the TCP connection.
Mass-cancel must be the fastest path in the system. One message cancels everything for a session/symbol/firm. When it’s used, the market is moving against someone and every millisecond of cancel latency is money; a mass-cancel that walks the book order-by-order through the normal pipeline is a design failure. Venues implement it as a first-class sequenced operation; your own engine should too (index orders by owner so cancel-all is O(orders-owned), pre-reserved capacity in every queue so the cancel can’t be backpressured by the very flood it’s trying to stop).
Post-trade: drop copy, triple recon, and the liquidation engine
Drop copy
Drop copy — a real-time duplicate feed of your executions (and often order events), delivered on a separate session to risk, clearing, and compliance systems — is the venue telling you what you did, as you do it. Analogy: a CDC stream off the ledger — the same events your trading session already saw, but delivered independently, so a bug in your trading-session handling can’t blind your risk view. Why it matters architecturally: your risk system’s position should be built from drop copy (venue-authoritative), not from your trading gateway’s view of its own acks — independence of the watcher from the watched. It’s also the input regulators expect your firm-wide kill decision to be based on: 15c3-5 asks brokers to monitor aggregate exposure in real time, and drop copy is the venue-side feed for it.
Reconciliation: the triple loop
Straight from your payments world — internal ledger vs PSP report vs bank statement — with the nouns swapped:
internal book ◄──recon──► venue statements ◄──recon──► custodian / chain
(event-sourced (drop copy live; (where assets actually
positions) EOD statements) sit: custody, wallets)
Three-way, because any two can agree and still be wrong together (your book and the venue agree you hold X on-venue; the custodian view says the venue can’t cover it — that’s the FTX shape). Live recon runs continuously against drop copy (position drift alarms in seconds); statement recon runs on venue cutoffs; custody recon runs on withdrawal/deposit confirmations and on-chain balances. Breaks go to a suspense workflow with aging alarms — an unexplained break that survives an hour is an incident, not a ticket. In a 24/7 crypto shop there is no end-of-day batch window; recon is a streaming job with rolling cutoffs, which is genuinely harder than tradfi’s nightly batch and worth saying in interviews.
The liquidation engine (operator side)
You know perps as a consumer of the mechanics; the venue-design interview asks you to specify them. The chain:
-
Funding rate — periodic payments between longs and shorts keeping the perp tethered to spot. Operator concern: computed from observable inputs on a published schedule, because participants arbitrage any discretion.
-
Mark price vs last price — margin and liquidations are computed on a mark price (an index of external spot venues, smoothed), not the venue’s own last trade. Why: if liquidations keyed off last price, a thin book lets an attacker print one small trade at an absurd price and cascade-liquidate everyone — so mark-price design is manipulation resistance, and it forces index composition rules: multiple constituent venues, outlier rejection (drop the deviant constituent), staleness handling (a constituent that stops updating leaves the index), and published weights. This is consensus-from-unreliable-oracles, an engineering problem you can whiteboard.
-
Liquidation waterfall — the ordered stages a losing position falls through as its margin runs out. One number to hold first: the bankruptcy price is the price at which the trader’s margin hits exactly zero. Tiny worked example: a 10x long opened at $100 has margin worth a 10% move, so its bankruptcy price is $90, and the venue starts intervening early, say a margin call around $92. The stages:
- Margin call — the position is flagged; the trader can add margin or reduce.
- Partial liquidation — the engine reduces the position (reduce, don’t nuke).
- Full liquidation via the book — the remainder is closed as limit orders, rate-limited, so the engine doesn’t crash its own market.
- Insurance fund — absorbs the gap if the position closed worse than its bankruptcy price.
- ADL (auto-deleveraging: forcibly closing profitable opposing positions, by a published leaderboard) — the last resort when the fund is exhausted.
Each stage is a documented, tested policy — the waterfall is the venue’s version of “who eats the loss,” which in payments you know as the chargeback/liability waterfall.
-
The liquidation engine is itself a trading system with the same needs: deterministic, sequenced inputs (mark-price ticks are events in the log), rate limits, and its own kill switch — a runaway liquidation engine is one of the worst self-inflicted incidents a venue can have.
Surveillance on the sequenced log
Market surveillance — detecting manipulative patterns in order flow — is, architecturally, a set of stream jobs consuming the sequenced log, and the single-sequencer design (ch13) pays its compliance dividend here: because every order event has one global sequence number, questions like “what did the book look like when this order was placed” have exact answers, not log-grep approximations.
The two patterns every interviewer names:
- Wash trading — trading with yourself (directly or via colluding accounts) to print fake volume. Detection: join trades where buyer and seller resolve to the same beneficial owner (accounts, funding sources, withdrawal addresses in crypto — an entity-resolution graph problem), plus statistical tells: self-crossing rates, volume with zero net position change, round-trip times. Crypto’s historical incentive: exchange volume rankings — which is why “real volume” studies embarrassed so many venues, and why running SMP + surveillance is a credibility signal for a serious one.
- Spoofing / layering — posting orders you intend to cancel to fake pressure (spoofing: one large order; layering: a stack of them) and trading the other side. Detection features on the log: order-to-trade ratio per account, cancel-latency distributions (spoofers cancel fast when approached), imbalance placed opposite to subsequent aggression, repeated patterns across sessions. These run as windowed stream jobs; flagged cases go to human review with a replayable book reconstruction as the evidence bundle — the surveillance analyst’s UI is a book-replay tool, which is why venues that can’t replay their book can’t really do surveillance.
The payments analogy is exact: transaction-monitoring rules (velocity, structuring, mule networks) running on the payment ledger, alerts to a case-review queue, SAR filings. Same architecture, different features.
The regulators: name-drop depth
You are not expected to be a lawyer. You are expected — at Talos-style firms especially — to know these regimes exist and what each one forces architecturally. That’s the depth interviewers check: one sentence of what, one sentence of forcing function.
- SEC Rule 15c3-5 (US, “Market Access Rule,” 2010): brokers providing market access must have pre-trade financial and regulatory risk controls under the broker’s own control — you cannot rent your pipe to a client unchecked (“naked access” ban). Forcing function: the OMS pre-trade gate in ch26 is mandatory, must be broker-controlled (not client-configurable-off), and aggregate credit exposure must be monitored in real time.
- MiFID II RTS 6 (EU, algorithmic-trading controls): firms running algos must have kill functionality, pre-trade limits, real-time monitoring, annual self-assessment of their algo systems, and testing of algos against disorderly-market scenarios. Forcing function: the kill-switch taxonomy above stops being optional engineering hygiene and becomes an audited requirement with named owners; “we test our kill switches” needs evidence.
- MiFID II more broadly: best execution (ch26), clock synchronization (RTS 25 — timestamps traceable to UTC, which is why ch07’s PTP chain is a regulatory artifact in the EU), and order-record keeping (years of retention — the log-retention economics of ch13).
- Crypto’s messier map: no single rulebook; per-jurisdiction licensing. MAS (Singapore — your home turf): the Payment Services Act’s DPT (digital payment token) licensing covers exchange and transfer services, with the Financial Services and Markets Act extending custody coverage — MAS licensing is the credibility bar for SG-based platforms, and you can say you’ve watched that regime professionally from Singapore. VARA (Dubai’s virtual-asset regulator) and MiCA (the EU’s Markets in Crypto-Assets regulation, phasing in from 2024) are the other names to have: one-liners suffice. The architectural consequence of the messiness: per-jurisdiction feature flags (which clients may touch which products), geofencing as a real subsystem, and travel-rule data exchange on transfers (the travel rule: regulations requiring sender and recipient identity to accompany crypto transfers between platforms).
Interview frame, verbatim if you like: “I’m not a compliance officer, but I know 15c3-5 means the pre-trade gate is legally mine as the broker, RTS 6 means my kill switches get audited annually, and MiFID’s clock-sync rules mean my PTP chain is evidence. I design assuming the log will be read by a regulator.”
Limits are config, config is code
Who changes a risk limit, how, and how fast — this is ch17 applied to the control plane, and it’s an interview topic because it’s where discipline usually fails.
- Limits-as-code: risk configuration lives in version control, deploys through a pipeline with schema validation, dry-run against current positions (“this change would put 3 clients in breach — proceed?”), staged rollout, and automatic audit trail of who/what/when/why. Not a database row someone UPDATEs.
- The 2am call: a client (or your own desk) hits a limit mid-move and wants it raised now. The answer is never “ops edits prod config”; it’s a pre-built emergency path: dual-approval (risk officer + on-call), bounded pre-approved uplift sizes, auto-expiry (the emergency raise reverts in N hours unless ratified), and the same audit trail as the slow path. Process beats heroics because the 2am raise that stuck around is how firms discover, months later, that their real limits are nothing like their documented ones. Payments version: the merchant screaming to lift their processing cap during a flash sale — same pressure, same answer.
- Intraday ownership: named roles own limit changes (RTS 6 wants this anyway); engineering owns the mechanism, risk owns the numbers, and the system enforces that separation — engineers shouldn’t be able to change a client’s credit limit, risk officers shouldn’t need a deploy.
Plain-English recap
- Every trading system carries a control plane: an in-line pre-trade gate, layered kill switches, a real-time copy of what happened (drop copy), reconciliation loops, surveillance on the log, and a config discipline for the limits themselves.
- Pre-trade risk is a hot path: price collars, size/notional caps, position and credit checks, rate limits, self-match prevention, and client-order-ID dedup — a few hundred nanoseconds venue-side, built on padded atomic counters. Dedup is the idempotency key; the credit check is the auth-hold pipeline.
- Kill switches are a taxonomy (client/session, symbol/venue, global), pre-authorized, drilled, and out-of-band; cancel-on-disconnect is per-session policy; mass-cancel must be the fastest path in the system — the big red “pause payouts” button, pre-wired.
- Post-trade: drop copy is CDC off the venue’s ledger and should feed a risk view independent of your trading session; reconciliation is the triple loop (internal book / venue / custodian) you know from payments, running continuously because crypto has no close.
- A perps venue’s margin stack — funding, mark price with manipulation-resistant index rules, liquidation waterfall, insurance fund, ADL — is itself a deterministic trading system with its own kill switch.
- Surveillance (wash trading, spoofing) is stream jobs on the sequenced log; the deterministic log is what makes both surveillance and regulator queries tractable.
- Regulatory anchors to name, not lawyer: SEC 15c3-5 (pre-trade risk mandatory at the broker), MiFID II RTS 6 (kill switches + annual self-assessment), RTS 25 (clock sync as evidence), MAS PSA/DPT in Singapore, VARA/MiCA one-liners. Each one forces a box in the diagram.
Interviewer will ask
Q1: “What pre-trade checks would you run, and what’s the latency budget?” Placement first, because it matters as much as the list. At the venue, the checks run in the gateway before the sequencer — the sequenced log is the replayable official history, so a bad order must never consume a sequence number. At the broker, they run in the OMS before the SOR — a breach must never reach a venue.
Then the battery, each with its one-line why. Price collar against a reference band — catches fat fingers and decimal-place bugs before they walk the book. Max size and notional cap — the “sell 1,000,000 instead of 1,000” stopper. Position-plus-open-orders limit — catches the slow bleed across many small orders. Credit/buying power — no trading beyond collateral. Per-session rate limit — contains runaway algos. Self-match prevention — stops wash-looking prints from a firm’s own flow. Duplicate detection on client order ID — makes retries after an ambiguous timeout safe. Two of these I’ve effectively built in payments: the dedup check is an idempotency key, and the credit check is an auth-hold pipeline with reserve/release/convert semantics.
The budget closes the argument. Each check is a compare or an atomic read against in-memory, cache-line-padded counters, reference data swapped atomically by pointer — a few hundred nanoseconds total, venue-side, on a gateway path that’s already 5–50µs. So skipping checks “for latency” buys back a few percent of the path at best while removing the only thing standing between a bug and the book.
Q2: “Design self-match prevention. What are the policy options and who wants which?” Mechanically: at match time, if the aggressing order and the resting order at the touch resolve to the same firm or SMP group, don’t print — apply the configured policy. Options: cancel-newest (aggressor dies, rester keeps queue position — market makers re-quoting want this), cancel-oldest (rester dies, aggressor trades on — sweepers want their intent to survive), or cancel-both. It’s a per-session or per-group flag because different participants legitimately want different semantics. The reason venues do this mechanically rather than adjudicating afterwards: self-matches print volume that’s indistinguishable from wash trading on the tape, so preventing them is both a surveillance and a credibility measure. Cost is a ~tens-of-ns ownership check at the touch — cheap because order structs already carry owner IDs.
Q3: “Tell me about kill switches. Who can pull what?” Start with the ladder of scopes, because a kill switch is a taxonomy, not a button. Per-session/client: disable one session, cancel its orders — pullable by risk, on-call, and the client themselves. Per-symbol or per-venue: halt an instrument venue-side; stop routing and mass-cancel broker-side — ops or automated breakers. Global: everything stops — named senior individuals only.
Then three design rules, each with its because. The kill path is pre-authorized and drilled, because an unpulled kill switch is a rumor — RTS 6 makes the testing an audited requirement in the EU. It must not depend on the sick system cooperating, because the thing being killed is by definition misbehaving — so there’s an out-of-band route. And the doctrine is cancel-then-investigate, never the reverse, because while you investigate, the market is moving against someone.
Mass-cancel itself is engineered as the fastest path in the system: one sequenced operation, orders indexed by owner, capacity pre-reserved so the cancel can’t be backpressured by the very flood it’s stopping. It’s the payments “pause payouts” button, pre-wired — and the postmortem question is always why it took N minutes to press.
Q4: “What is drop copy and why does it exist if you already get execution reports?” Drop copy is a real-time duplicate feed of your fills and order events on a separate session, delivered to risk and clearing independently of your trading session — CDC off the venue’s ledger. It exists because the watcher must be independent of the watched: if my risk position is built from my trading gateway’s own view of its acks, a bug there corrupts both trading and risk simultaneously; drop copy gives risk a venue-authoritative stream that my trading code never touches. It’s also the practical input for firm-wide real-time exposure monitoring, which 15c3-5 expects broker-side. In my triple-recon design, drop copy is the live leg — internal book vs drop copy in seconds, vs statements at cutoffs, vs custodian on settlement — three-way because any two can agree and still be wrong together, which is the FTX shape.
Q5: “Why mark price instead of last price for liquidations, and what does that force?” Because keying liquidations off your own last trade makes a thin book a weapon: one small print at an absurd price cascades liquidations, and the attacker profits from the carnage. So margin runs on a mark price — an index over multiple external spot venues, smoothed — and that forces index-composition engineering: several constituents, published weights, outlier rejection so one deviant venue is dropped, staleness rules so a stalled feed leaves the index, and a fallback when too few constituents survive. It’s consensus from unreliable oracles, and it’s specified publicly because participants arbitrage any discretion. Downstream sits the waterfall — margin call, partial liquidation, full liquidation via rate-limited book orders, insurance fund, ADL last — each stage a documented policy, and the liquidation engine itself gets sequenced deterministic inputs and its own kill switch, because a runaway liquidator is the worst self-inflicted incident a perps venue can have.
Q6: “How would you detect wash trading and spoofing on your venue?” As stream jobs on the sequenced log — which is the architectural point: one global sequence means “what did the book look like when this order arrived” has an exact, replayable answer, and surveillance without book replay isn’t really surveillance. Wash trading: entity-resolve accounts into beneficial owners (shared funding sources, withdrawal addresses — a graph problem in crypto), then flag self-crossing rates, volume with zero net position change, and tight round-trips. Spoofing/layering: per-account order-to-trade ratios, cancel-latency distributions — spoofers cancel fast when approached — and size imbalance posted opposite subsequent aggression. Flagged cases go to human review with a book-replay evidence bundle. This is the same architecture as payments transaction monitoring — velocity rules on a ledger feeding a case queue — with different features, and self-match prevention upstream removes the innocent-explanation cases before they reach the queue.
Q7: “What do 15c3-5 and RTS 6 actually force you to build?” 15c3-5 — the US market-access rule — says a broker giving clients access must run pre-trade financial and regulatory checks under the broker’s own control: so the OMS gate before my SOR is legally mandatory, cannot be switched off per client request, and aggregate credit exposure needs real-time monitoring, which is what drop copy feeds. RTS 6 — MiFID II’s algo-trading standard — mandates kill functionality, pre-trade limits, real-time monitoring, and an annual self-assessment: so my kill-switch taxonomy needs named owners, test evidence, and documentation that survives an audit. I’d add MiFID’s RTS 25: timestamps traceable to UTC, which turns the PTP chain into a regulatory artifact. I’m not a lawyer and say so in the room — but I design assuming the event log will be read by a regulator, which is cheap if you’re event-sourced from day one and impossible to retrofit if you’re not. In Singapore, my home market, the equivalent gate is MAS’s PSA/DPT licensing regime for crypto platforms.
Q8: “A client calls at 2am demanding a limit raise mid-move. What happens?” Run the clock. 02:03 — the call routes to the risk on-call, not to engineering, because “ops edits prod config” is not a path that exists; the mechanism won’t accept a raw edit. 02:05 — on-call opens the emergency-uplift tool and sees the client’s current limit, live utilization, and a short menu of pre-approved uplift sizes — say 1.5× or 2×, sized in a design review months ago, not invented on the phone. Meanwhile the client’s orders above the old limit are still bouncing with explicit limit-breach rejects — the system stays correct while the humans decide. 02:07 — the second approver, another risk officer, confirms from their phone: two distinct identities required, so one tired human can’t wave it through alone. 02:08 — the uplift goes live, the client’s next order passes, and they’re told both the new number and its expiry. 06:00 — the uplift auto-expires and the limit snaps back, unless someone ratified it through the normal daytime pipeline with full review. Every beat — who called, who approved, what size, when it lapsed — lands on the same audit trail as the slow path. The ceremony exists because of the counterfactual: the 2am raise that quietly sticks is how firms discover, months later, that their real limits bear no resemblance to their documented ones. I’ve watched the payments version — a merchant demanding their processing cap lifted mid-flash-sale — and the answer is identical: the emergency path exists, it’s fast, and it’s paved with audit. One split keeps it honest: engineering owns the mechanism, risk owns the numbers, and the system enforces that neither can do the other’s job.
Further reading
- SEC Rule 15c3-5 adopting release (Release No. 34-63241, “Risk Management Controls for Brokers or Dealers with Market Access”) — readable, and the canonical why-and-what of broker-side pre-trade risk.
- Commission Delegated Regulation (EU) 2017/589 (MiFID II RTS 6) — the actual text of the algo-controls standard; skim Articles on kill functionality, pre-trade controls, and annual self-assessment.
- FIA, “Best Practices for Exchange Risk Controls” — practitioner-level catalogue of venue-side pre-trade checks and kill mechanisms.
- BitMEX and Deribit public documentation on mark price, index composition, liquidation, insurance fund, and ADL — the most complete public specs of a perps margin stack, written by operators.
- MAS Payment Services Act (DPT service provider) guidance — for the Singapore licensing frame; the MAS website’s DPT pages are the primary source.
- CFTC and SEC spoofing enforcement actions (e.g., the 2020 JPMorgan spoofing settlement) — read one to see what surveillance evidence actually looks like: reconstructed books and cancel-timing patterns.
Where this goes next: Chapter 28 makes all of Part V concrete — you build a mini market, venue and broker end-to-end, with the risk gate and kill switch from this chapter wired in; then Chapter 29 drills the whole part as interview questions.
Lab IV: A Mini Market — Venue + Broker End-to-End
Before you start. This lab assembles almost everything from Part V: ch23 (venue architecture: gateway → sequencer → engine), ch25 (feed publishing, snapshots, gap recovery), and ch26 (broker adapters, SOR, the double-fill race) — plus ch13 (event sourcing and replayability) and the mental model from ch00f.
You have built a matching engine before. You have built a market-data pipeline before. What you have probably never done is run both sides of the market at once and watch them argue with each other: the venue dropping deltas on a slow subscriber, the broker detecting the gap and routing around the blind spot, the SOR refusing to re-route until a cancel-ack comes back. The interesting interview questions come out of that argument, and after this lab you will have watched it happen in your own terminal.
One word before anything else: a delta is one incremental book change — “level 10004 now has 250” — the small message a venue publishes instead of resending the whole book every time anything moves. Drop one and your copy of the book silently diverges from the venue’s; most of this lab’s drama comes from exactly that.
Everything runs in one process on your Mac: threads + crossbeam-channel, no tokio, no sockets. Every channel boundary is labeled with what it would be in production (TCP session, multicast group), so the logic transfers 1:1.
VENUE A (thread) VENUE B (thread)
gateway→sequencer→engine gateway→sequencer→engine
│ │ │ │
acks/fills feed (L2 incr+snap) acks/fills feed
└───┬────┴───────┬────────────────┬────────┘
▼ ▼ ▼
[broker: venue adapters × 2 → normalized books]
│
[SOR: cost model picks venue(s), splits]
│
[parent-order manager: fills, re-routes, client report]
▲
client script: sends parent orders
(In the diagram: L2 = price-level depth — the whole ladder of prices and sizes, not just the best bid/ask; incr+snap = incremental deltas plus a periodic snapshot, so a late or gapped subscriber can always rebuild.)
Venue A is configured fast but wide (2 ms, deep book, 0.6-tick fee), Venue B slow but tight (20 ms, thin book, 0.2-tick fee) — so routing is a genuine trade-off, not a foregone conclusion. Unpacking the trader shorthand: wide = worse prices at the top of the book, deep = lots of size resting behind them; tight = better prices at the top, thin = less size behind them — so which venue is “cheapest” depends on how much you’re buying.
Step 0 — Workspace
mini-market/
├── Cargo.toml
└── src/
├── main.rs # scenarios + determinism check
├── types.rs # shared message types
├── book.rs # price-time matching engine
├── venue.rs # gateway → sequencer → engine → feed
├── adapter.rs # broker-side feed handler (normalized book)
└── sor.rs # cost model, splitter, parent-order manager
# Cargo.toml
[package]
name = "mini-market"
version = "0.1.0"
edition = "2021"
[dependencies]
crossbeam-channel = "0.5"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
Step 1 — Shared types
Fixed structs everywhere on the hot path; serde only for the human-readable client report at the end. One convention to notice before you scroll: Px is an integer count of ticks — floats never touch a price (ch00d’s tick-size rule) — so $100.00 is 10_000 everywhere in this lab.
#![allow(unused)]
fn main() {
// src/types.rs
use serde::Serialize;
pub type Px = i64; // integer ticks (1 tick = $0.01); 10_000 = $100.00
pub type Qty = u64;
pub type Seq = u64;
// #[derive(...)] is like a decorator that writes boilerplate at compile time:
// Clone/Copy/Debug for copying and printing, Serialize (serde) for JSON later.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
pub enum Side { Buy, Sell }
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize)]
pub enum VenueId { A, B }
#[derive(Clone, Debug)]
pub struct NewOrder { pub client_id: String, pub side: Side, pub px: Px, pub qty: Qty }
/// Order entry. In production: FIX/OUCH over a per-member TCP session.
#[derive(Clone, Debug)]
pub enum GatewayIn { New(NewOrder), Cancel { client_id: String }, CancelAll }
#[derive(Clone, Debug, PartialEq, Serialize)]
pub enum ExecType { Ack, Reject(String), Fill { px: Px, qty: Qty }, CancelAck { qty: Qty } }
/// Private execution report back to the order sender.
#[derive(Clone, Debug, Serialize)]
pub struct ExecReport { pub venue: VenueId, pub client_id: String, pub exec: ExecType, pub seq: Seq }
#[derive(Clone, Debug, Serialize)]
pub struct Level { pub px: Px, pub qty: Qty }
/// Public market data. In production: seq-numbered UDP multicast per channel.
#[derive(Clone, Debug)]
pub enum FeedMsg {
Delta { seq: Seq, side: Side, px: Px, qty: Qty }, // qty = new total at level; 0 = level removed
Snapshot { seq: Seq, bids: Vec<Level>, asks: Vec<Level> },
}
#[derive(Serialize)]
pub struct FillLine { pub venue: VenueId, pub px: Px, pub qty: Qty }
#[derive(Serialize)]
pub struct ClientReport { pub parent: String, pub filled: Qty, pub avg_px: f64, pub fills: Vec<FillLine> }
}Step 2 — The matching engine
Minimal price-time book — you have built a bigger one; this one exists to emit the right events, not to be fast. The two data structures below carry the two priorities directly: BTreeMap keeps the price levels permanently sorted (price priority), and each level’s VecDeque is a first-come-first-served queue of resting orders (time priority).
“The right events” is MatchOut, the triple every submit returns: fills for the taker (the incoming order), fills for the makers it hit (the resting orders on the other side), and the book deltas the feed will publish. That triple is exactly what the venue thread in Step 3 fans out — private reports to each party, public deltas to everyone.
#![allow(unused)]
fn main() {
// src/book.rs
use crate::types::*;
use std::collections::{BTreeMap, VecDeque};
// One price level = a first-come-first-served queue: first to arrive is filled
// first — time priority. (VecDeque = a double-ended queue;
// each entry is one resting order, (client_order_id, qty).)
type LevelQ = VecDeque<(String, Qty)>;
// The two sides. A BTreeMap stores its keys permanently sorted, so "best
// price" is just the first (or last) key — no searching, ever.
#[derive(Default)]
pub struct Book { bids: BTreeMap<Px, LevelQ>, asks: BTreeMap<Px, LevelQ> }
pub struct MatchOut {
pub taker_fills: Vec<(Px, Qty)>,
pub maker_fills: Vec<(String, Px, Qty)>,
pub deltas: Vec<(Side, Px, Qty)>, // (book side, price, new aggregate qty)
}
// Total quantity resting at one price: sum every order in that level's queue.
// (Each entry is a tuple (client_order_id, qty); `.0`/`.1` pick tuple fields by
// position, so `e.1` is the qty. You'll see this indexing throughout the file.)
fn lvl_qty(q: &LevelQ) -> Qty { q.iter().map(|e| e.1).sum() }
impl Book {
/// One side's full ladder, best price first — the snapshot payload.
pub fn depth(&self, side: Side) -> Vec<Level> {
match side {
// maps iterate ascending, so bids are reversed: highest (best) bid first
Side::Buy => self.bids.iter().rev().map(|(p, q)| Level { px: *p, qty: lvl_qty(q) }).collect(),
Side::Sell => self.asks.iter().map(|(p, q)| Level { px: *p, qty: lvl_qty(q) }).collect(),
}
}
/// Match an incoming order against the opposite side; any remainder rests.
/// Returns the MatchOut triple: taker fills, maker fills, feed deltas.
pub fn submit(&mut self, o: &NewOrder) -> MatchOut {
let (mut tf, mut mf, mut deltas, mut qty) = (Vec::new(), Vec::new(), Vec::new(), o.qty);
loop {
let opp = match o.side { Side::Buy => &mut self.asks, Side::Sell => &mut self.bids };
// best opposing price: lowest ask (first key) when buying, highest bid (last key) when selling
let best = match o.side { Side::Buy => opp.keys().next().copied(), Side::Sell => opp.keys().next_back().copied() };
let Some(px) = best else { break }; // nobody left on the other side of the market — stop. (let-else: unpack the Some or bail)
let crosses = match o.side { Side::Buy => px <= o.px, Side::Sell => px >= o.px };
if !crosses || qty == 0 { break; }
let q = opp.get_mut(&px).unwrap();
while qty > 0 {
let Some(front) = q.front_mut() else { break };
let take = qty.min(front.1);
mf.push((front.0.clone(), px, take));
tf.push((px, take));
front.1 -= take; qty -= take;
if front.1 == 0 { q.pop_front(); }
}
let left = lvl_qty(q);
if left == 0 { opp.remove(&px); }
deltas.push((if o.side == Side::Buy { Side::Sell } else { Side::Buy }, px, left));
}
if qty > 0 { // remainder rests at its limit, at the back of the level queue
let map = match o.side { Side::Buy => &mut self.bids, Side::Sell => &mut self.asks };
// entry(): fetch that price's level, creating it if it doesn't
// exist yet; push_back = join the back of the queue (time priority kept)
map.entry(o.px).or_default().push_back((o.client_id.clone(), qty));
deltas.push((o.side, o.px, lvl_qty(&map[&o.px])));
}
MatchOut { taker_fills: tf, maker_fills: mf, deltas }
}
/// Remove one client's resting order, wherever it sits on the book.
pub fn cancel(&mut self, cid: &str) -> Option<(Side, Px, Qty, Qty)> { // (side, px, canceled, level left)
for (side, map) in [(Side::Buy, &mut self.bids), (Side::Sell, &mut self.asks)] {
let hit = map.iter().find(|(_, q)| q.iter().any(|e| e.0 == cid)).map(|(p, _)| *p);
if let Some(px) = hit {
let q = map.get_mut(&px).unwrap();
let before = lvl_qty(q);
q.retain(|e| e.0 != cid); // pull this one client out of the line; everyone else keeps their place (retain = filter in place)
let left = lvl_qty(q);
if left == 0 { map.remove(&px); }
return Some((side, px, before - left, left));
}
}
None
}
/// Kill-switch mass cancel: clear the whole book, sparing "SEED" background liquidity.
pub fn cancel_all(&mut self) -> Vec<(String, Qty)> { // returns canceled non-seed orders
let mut out = Vec::new();
for map in [&mut self.bids, &mut self.asks] {
for q in map.values() { for e in q { if e.0 != "SEED" { out.push((e.0.clone(), e.1)); } } }
map.clear();
}
out
}
}
}Step 3 — The venue: gateway → sequencer → engine → two outputs
One thread per venue. The gateway checks run before the sequencer, so a rejected order never consumes a sequence number — the sequence is the replayable truth of the market, and a reject never happened as far as the market is concerned. The sequencer is simply the fact that one consumer drains the channel — consuming in arrival order is the total order (ch23). Two output paths: private exec reports to the sender, and a public seq-numbered feed with a periodic snapshot — the late-joiner contract from ch25: a new subscriber can only bootstrap from a snapshot, so one is published every snap_every deltas.
Two more things to watch for in the code. First, the seed_liq orders in the config: they stand in for other market participants’ resting quotes — background liquidity that isn’t yours — which is why Step 2’s cancel_all (the kill-switch path) deliberately spares anything tagged "SEED". Second, the two sequence-number spaces, seq and fseq — two independent counters, one numbering the private exec-report stream, one the public feed. The diagram below marks where each advances; they never sync and never need to — real venues keep them separate too.
This file is also where the lab’s Rust concurrency toolkit first appears. Node gives you one event loop — nothing runs at the same time as your code. This lab runs several such loops at once: each venue is its own OS thread, its for msg in rx.iter() loop genuinely executing in parallel with yours. Threads share no variables by default; the only doors between them are channels — typed one-way mail chutes, postMessage between workers, except receiving sleeps until a message drops in. Every ══ channel ══ line in the diagrams from here on is one such chute (a crossbeam-channel), labeled with the production wire it stands in for.
VENUE THREAD (one lane — one consumer draining order_rx IS the sequencer)
══ order_rx channel (thread boundary — prod: TCP order-entry session, FIX/OUCH) ══
(1) rx.iter() — order arrives
(2) GATEWAY: dup client_id? size ≤ max? collar ±10%? tokens left?
│ fail ─▶ exec_tx.send(Reject) — exits HERE, pre-sequencer: NO seq consumed
▼ pass
(3) SEQUENCER: seq += 1 ← private counter `seq`: numbers YOUR events
(4) ENGINE: book.submit(&o) → { taker fills, maker fills, deltas }
├─▶ (5) PRIVATE: exec_tx.send(Ack/Fill/CancelAck) — one `seq` each
│ ══ channel (prod: TCP session) ══ reliable: send() waits, never drops
└─▶ (6) PUBLIC: feed try_send(Delta/Snapshot) — one `fseq` each
══ channel, bounded (prod: UDP multicast) ══ best-effort: full slot =
DROPPED, never blocks. fseq contiguous: see 5 then 7 ⇒ 6 went missing
Each idiom gets a comment at first use, so read the comments as part of the text.
#![allow(unused)]
fn main() {
// src/venue.rs
use crate::{book::Book, types::*};
use crossbeam_channel::{unbounded, Receiver, Sender};
use std::collections::HashSet;
use std::thread::{self, JoinHandle};
use std::time::Duration;
pub struct VenueConfig {
pub id: VenueId, pub ref_px: Px, pub fee_ticks: f64, pub latency: Duration,
pub max_qty: Qty, pub gw_tokens: u32, pub snap_every: u64,
pub seed_liq: Vec<(Side, Px, Qty)>,
}
/// What callers hold: the order-entry sender, plus a JoinHandle — a handle to
/// the thread's eventual return value (its event log). Where JS would `await`
/// a Promise, a JoinHandle is redeemed with a blocking `.join()`.
pub struct VenueHandle { pub tx: Sender<GatewayIn>, pub join: JoinHandle<Vec<String>> }
/// Start the venue thread; return its order-entry sender and join handle.
/// In production `tx` is a TCP order-entry session and `feeds` are multicast groups.
pub fn spawn(cfg: VenueConfig, exec_tx: Sender<ExecReport>, feeds: Vec<Sender<FeedMsg>>) -> VenueHandle {
let (tx, rx) = unbounded(); // a mail chute with no depth limit: send always succeeds instantly, the pile just grows
// A JS closure *shares* what it captures; `move` instead hands the closure sole
// ownership of cfg, rx, exec_tx, feeds — it must, because the new thread outlives
// this function, and two threads may never share unguarded data.
let join = thread::spawn(move || run(cfg, rx, exec_tx, feeds));
VenueHandle { tx, join }
}
/// Fan one message out to every feed subscriber — multicast in miniature.
fn publish(feeds: &[Sender<FeedMsg>], msg: FeedMsg) {
// Each subscriber's feed queue is a mail slot of fixed depth. try_send delivers
// only if there's room — a stuffed slot means the letter is DROPPED and the venue
// moves on: a slow reader must never set the venue's pace (real UDP multicast
// behaves exactly like this when a subscriber lags).
for f in feeds { let _ = f.try_send(msg.clone()); } // let _ = "it can fail; dropping is the plan"
}
// Package the current book as a Snapshot feed message.
fn snap(book: &Book, seq: Seq) -> FeedMsg {
FeedMsg::Snapshot { seq, bids: book.depth(Side::Buy), asks: book.depth(Side::Sell) }
}
// Publish one delta, plus a fresh snapshot every `every` deltas (the late-joiner contract).
fn delta(book: &Book, feeds: &[Sender<FeedMsg>], fseq: &mut Seq, since: &mut u64, every: u64,
side: Side, px: Px, qty: Qty) {
*fseq += 1; publish(feeds, FeedMsg::Delta { seq: *fseq, side, px, qty });
*since += 1;
if *since >= every { *fseq += 1; publish(feeds, snap(book, *fseq)); *since = 0; }
}
// The venue thread body: gateway checks -> sequencer -> engine, fanning out
// private exec reports and public feed messages. Returns the event log.
fn run(cfg: VenueConfig, rx: Receiver<GatewayIn>, exec_tx: Sender<ExecReport>, feeds: Vec<Sender<FeedMsg>>) -> Vec<String> {
let (mut book, mut seq, mut fseq) = (Book::default(), 0u64, 0u64);
let (mut last_px, mut tokens) = (cfg.ref_px, cfg.gw_tokens);
let (mut seen, mut log, mut since) = (HashSet::new(), Vec::new(), 0u64);
for (s, p, q) in &cfg.seed_liq {
book.submit(&NewOrder { client_id: "SEED".into(), side: *s, px: *p, qty: *q });
}
fseq += 1; publish(&feeds, snap(&book, fseq)); // late-joiner contract: snapshot first
for msg in rx.iter() { // the venue's event loop: like `for await` on a stream — sleeps until mail arrives, ends when every sender is gone
thread::sleep(cfg.latency); // artificial venue latency (A fast, B slow)
match msg {
GatewayIn::Cancel { client_id } => {
seq += 1;
let (qty, d) = match book.cancel(&client_id) { Some((s, p, c, l)) => (c, Some((s, p, l))), None => (0, None) };
log.push(format!("{seq} CXL {client_id} qty={qty}"));
// send, not try_send: the private line is reliable — an exec report
// must arrive, so we'd wait rather than drop. Contrast publish() above.
// (.ok() shrugs only if the receiver already shut down.)
exec_tx.send(ExecReport { venue: cfg.id, client_id, exec: ExecType::CancelAck { qty }, seq }).ok();
if let Some((s, p, l)) = d { delta(&book, &feeds, &mut fseq, &mut since, cfg.snap_every, s, p, l); }
}
GatewayIn::CancelAll => { // venue-side kill switch: mass cancel
for (cid, qty) in book.cancel_all() {
seq += 1; log.push(format!("{seq} KILLCXL {cid} qty={qty}"));
exec_tx.send(ExecReport { venue: cfg.id, client_id: cid, exec: ExecType::CancelAck { qty }, seq }).ok();
}
fseq += 1; publish(&feeds, snap(&book, fseq)); since = 0;
}
GatewayIn::New(o) => {
// ---- gateway: per-session risk checks, BEFORE the sequencer ----
let rej = if seen.contains(&o.client_id) { Some("duplicate client_order_id") }
else if o.qty == 0 || o.qty > cfg.max_qty { Some("max order size") }
else if (o.px - last_px).abs() * 10 > last_px { Some("price collar +-10%") }
else if tokens == 0 { Some("gateway rate limit") } else { None };
if let Some(r) = rej {
exec_tx.send(ExecReport { venue: cfg.id, client_id: o.client_id, exec: ExecType::Reject(r.into()), seq: 0 }).ok();
continue; // rejected orders never reach the sequencer: no seq consumed
}
seen.insert(o.client_id.clone());
tokens -= 1;
// ---- sequencer: THE total-order point for this venue ----
seq += 1;
log.push(format!("{seq} NEW {} {:?} {}@{}", o.client_id, o.side, o.qty, o.px));
exec_tx.send(ExecReport { venue: cfg.id, client_id: o.client_id.clone(), exec: ExecType::Ack, seq }).ok();
// ---- matching engine ----
let out = book.submit(&o);
for (px, qty) in &out.taker_fills {
last_px = *px; seq += 1;
log.push(format!("{seq} FILL {} {qty}@{px}", o.client_id));
exec_tx.send(ExecReport { venue: cfg.id, client_id: o.client_id.clone(), exec: ExecType::Fill { px: *px, qty: *qty }, seq }).ok();
}
for (cid, px, qty) in &out.maker_fills {
if cid == "SEED" { continue; }
seq += 1;
exec_tx.send(ExecReport { venue: cfg.id, client_id: cid.clone(), exec: ExecType::Fill { px: *px, qty: *qty }, seq }).ok();
}
for (s, p, l) in out.deltas { delta(&book, &feeds, &mut fseq, &mut since, cfg.snap_every, s, p, l); }
}
}
}
log // event log, returned at shutdown for the determinism check
}
}Step 4 — Broker adapters: normalized book + gap recovery
One adapter thread per venue, each producing a normalized book: both venues’ feeds reduced to the same plain price→qty maps, so the SOR can compare venues without caring whose wire format the update arrived in. lag simulates a slow consumer; the bounded channel it reads is the “socket buffer”. The gap-then-snapshot path in the diagram is exactly ch25’s recovery protocol.
One new idiom here, and it deserves a picture. Each normalized book is a whiteboard: the adapter thread writes price updates on it, the SOR thread reads it to plan routes. Arc means both hold a handle to the same whiteboard — not photocopies; when the adapter writes, the SOR’s next glance sees it. Mutex is the single marker tied to the board: you must hold the marker to touch the board at all — even to read it reliably, since glancing mid-erase shows half-updated numbers — and lock() grabs the marker, waiting if the other thread has it. TypeScript never needs this because one event loop owns all the data; with real threads that guarantee has to be built, and Arc<Mutex<…>> is the smallest way to build it.
══ feed channel, bounded (thread boundary — prod: multicast socket buffer) ══
ADAPTER THREAD │ SOR THREAD
(1) feed_rx: msg arrives (after `lag`) │
(2) book.lock() — take the marker ──────────┼── shared state (lock): NormBook,
(3) Delta{seq}: contiguous? seq == last+1? │ Arc<Mutex<…>> — one board, both
│ no ─▶ gaps += 1, recovering = true │ threads hold handles to it
│ book UNTRUSTED — deltas now │
▼ yes SKIPPED until a snapshot │
(4) apply delta: bids/asks insert/remove │ (6) plan() takes the same lock:
(5) Snapshot: replace book wholesale, │ mid() / ladder() / healthy()
recovering = false ◀── the recovery │ — reads the very board the
path re-enters trusted state HERE │ adapter writes, never a copy
#![allow(unused)]
fn main() {
// src/adapter.rs
use crate::types::*;
use crossbeam_channel::Receiver;
use std::collections::BTreeMap;
use std::sync::{Arc, Mutex};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
/// Broker-side normalized view of one venue, built from the public feed.
pub struct NormBook {
pub bids: BTreeMap<Px, Qty>, pub asks: BTreeMap<Px, Qty>,
pub last_seq: Seq, pub gaps: u32, pub recovering: bool, pub last_update: Instant,
}
impl NormBook {
// A fresh book starts `recovering`: untrusted until the first snapshot lands.
pub fn new() -> Self {
NormBook { bids: BTreeMap::new(), asks: BTreeMap::new(), last_seq: 0, gaps: 0, recovering: true, last_update: Instant::now() }
}
// Midpoint of best bid and best ask. The `?` works like optional chaining `?.`:
// if either side of the book is empty there's no best price, so the answer is None.
pub fn mid(&self) -> Option<f64> {
Some((*self.bids.keys().next_back()? + *self.asks.keys().next()?) as f64 / 2.0)
}
// One side's resting levels as (px, qty) — the ladder the SOR's cost walk consumes.
pub fn ladder(&self, resting: Side) -> Vec<(Px, Qty)> { // best price first
match resting {
Side::Sell => self.asks.iter().map(|(p, q)| (*p, *q)).collect(),
Side::Buy => self.bids.iter().rev().map(|(p, q)| (*p, *q)).collect(),
}
}
// Trustworthy feed = no unresolved gap and updated recently.
pub fn healthy(&self, max_age: Duration) -> bool { !self.recovering && self.last_update.elapsed() <= max_age }
}
/// Adapter thread: drain one venue's feed onto `book`, the whiteboard shared with
/// the SOR (Arc = both hold the SAME board, Mutex = the one marker you must hold).
/// In production this thread reads a multicast socket. `lag` = slow-consumer simulation.
pub fn spawn(venue: VenueId, rx: Receiver<FeedMsg>, book: Arc<Mutex<NormBook>>, lag: Option<Duration>) -> JoinHandle<()> {
thread::spawn(move || {
for msg in rx.iter() {
if let Some(d) = lag { thread::sleep(d); }
// Grab the whiteboard marker; waits if the SOR is mid-read. (unwrap: if a
// holder crashed mid-write the lock is "poisoned" — crash too, don't trust
// a half-updated board.)
let mut b = book.lock().unwrap();
match msg {
FeedMsg::Snapshot { seq, bids, asks } => {
b.bids = bids.into_iter().map(|l| (l.px, l.qty)).collect();
b.asks = asks.into_iter().map(|l| (l.px, l.qty)).collect();
if b.recovering && b.last_seq > 0 { println!(" [adapter {venue:?}] recovered via snapshot seq={seq}"); }
b.last_seq = seq; b.recovering = false; b.last_update = Instant::now();
}
FeedMsg::Delta { seq, side, px, qty } => {
// We number-check every incoming letter. Letter 7 after letter 5
// means 6 is lost in the mail — every conclusion drawn from this
// board is suspect until a fresh snapshot replaces it wholesale.
if seq != b.last_seq + 1 && !b.recovering {
b.gaps += 1; b.recovering = true; // stop applying deltas: the book is untrusted
println!(" [adapter {venue:?}] GAP: expected seq {} got {seq} — waiting for snapshot", b.last_seq + 1);
}
b.last_seq = seq;
if b.recovering { continue; }
let m = match side { Side::Buy => &mut b.bids, Side::Sell => &mut b.asks };
if qty == 0 { m.remove(&px); } else { m.insert(px, qty); }
b.last_update = Instant::now();
}
}
}
})
}
}Step 5 — SOR + parent-order manager
This is the biggest block in the lab, and it does exactly five things. Read them here first — the code is just these five, in order:
The frame for 1–4 is shopping across two stores: one has lower sticker prices but a service fee and a slow checkout, the other is pricier but quick. You compare what it costs to walk out with the goods, not stickers — and you’ll happily buy part of the list at each store.
- Cost model — each visible level is priced as
price + per-venue fee + latency penalty: sticker plus that store’s fee plus its checkout queue. “Cheapest” means all-in cost. - Latency penalty from a measured ack-RTT EWMA — an EWMA (exponentially weighted moving average) is a running estimate that each new reading nudges 30% toward what just happened (
0.7 × old + 0.3 × new); older readings fade geometrically — never gone, just fainter. Every ack round-trip is one reading, so the penalty tracks what the venue is doing now, not its spec sheet. - Cross-venue merge — merge both venues’ eligible levels into ONE list, sorted by that all-in cost.
- Marginal-depth walk — walk the merged list cheapest-first, taking quantity until the parent order is covered. Splitting isn’t a special case: the walk naturally takes from both venues the moment one venue’s next level is dearer than the other’s best remaining.
- Health demotion and the penalty box — venues with a gapped or stale feed are excluded before pricing; venues that rejected this parent are penalty-boxed for the next pass.
Worked through with Step 6’s config, buying 500: B’s 150@10001 is cheapest, B’s 200@10003 next; then A’s 400@10004 beats B’s 300@10005, so the walk finishes there — 500 shares split 350 to B, 150 to A. That is the exact split you’ll see in scenario (a).
Two vocabulary bridges for the code. This is ch26’s parent/child split: the parent is the client’s whole order; the children are the venue-sized slices the SOR cuts it into. And a child is terminal when it is filled, canceled, or rejected — no state left at the venue that could still execute. cancel_children is the double-fill guard from ch26: re-route only after every child is terminal, because a merely sent cancel can still lose the race to a fill already in flight.
Read the code in this order: plan (the five mechanisms above), apply_exec (how acks, fills, rejects, and cancel-acks update state — including the RTT EWMA), execute_parent (the 3-pass send → collect → re-plan loop), then cancel_children (the guard). This is one parent order’s whole life, every thread boundary marked:
time ↓ BROKER / SOR thread │ VENUE A thread │ VENUE B thread
(1) execute_parent: arrival_mid() stamped — TCA benchmark, BEFORE anything sends
(2) plan(): lock both books, merge ladders by all-in cost, walk → child slices
(3) v.tx.send(New P1-C1) ═════════════════▶ gw→seq→engine │
(4) v.tx.send(New P1-C2) ═════════════════╪═══════════════════▶ gw→seq→engine
══ order channels (prod: one TCP order-entry session per venue) ══
(5) collect(): exec_rx.recv_timeout loop │ │
◀═══════════ Ack ═════════════════════╡ │
◀═══════════ Fill{px,qty} ════════════╡ │
◀═══════════ Ack, Fill ═══════════════╪═══════════════════╡
apply_exec: Ack→EWMA, Fill→leaves−=qty │
══ shared exec channel (prod: exec reports on each session) ══
(6) remaining > 0 → cancel before re-route: │
(7) cancel_children: tx.send(Cancel) ═════▶ book.cancel │
◀═══════════ CancelAck{unfilled} ═════╡ │
only NOW may the remainder move — the double-fill guard │
(8) re-plan: back to (2), ≤ 3 passes │ │
(9) remaining == 0 → report_tca(): avg fill px vs arrival mid │
#![allow(unused)]
fn main() {
// src/sor.rs
use crate::{adapter::NormBook, types::*};
use crossbeam_channel::{Receiver, Sender};
use std::collections::{BTreeMap, HashMap, HashSet};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
const LAT_PENALTY_PER_MS: f64 = 0.02; // ticks of adverse-selection cost per ms of ack RTT
pub struct VenueLink {
pub id: VenueId, pub tx: Sender<GatewayIn>,
pub book: Arc<Mutex<NormBook>>, pub fee_ticks: f64, pub rtt_ms: f64, // EWMA of measured ack RTT
}
pub struct Sor {
pub venues: Vec<VenueLink>,
pub exec_rx: Receiver<ExecReport>,
next_child: u64,
open_children: HashMap<String, (VenueId, Qty)>, // child -> (venue, leaves)
inflight: HashMap<String, Instant>, // child -> send time (for RTT)
rejected: HashSet<VenueId>, // penalty box, per parent
cur_fills: Vec<(VenueId, Px, Qty)>, // fills for the current parent
}
impl Sor {
// Fresh SOR over these venue links, reading all exec reports from one shared inbox.
pub fn new(venues: Vec<VenueLink>, exec_rx: Receiver<ExecReport>) -> Self {
Sor { venues, exec_rx, next_child: 0, open_children: HashMap::new(),
inflight: HashMap::new(), rejected: HashSet::new(), cur_fills: Vec::new() }
}
// Average mid across healthy venues — the TCA benchmark taken at order arrival.
fn arrival_mid(&self) -> f64 {
let mids: Vec<f64> = self.venues.iter().filter_map(|v| {
let b = v.book.lock().unwrap();
if b.healthy(Duration::from_secs(2)) { b.mid() } else { None }
}).collect();
if mids.is_empty() { 0.0 } else { mids.iter().sum::<f64>() / mids.len() as f64 }
}
// Step 5's mechanisms 1-4 as code: merge every eligible level from every
// healthy venue into one list, priced all-in (px + fee + latency penalty), and
// walk it cheapest-first. Returns child slices as (venue index, limit px, qty).
fn plan(&self, side: Side, qty: Qty, limit: Px) -> Vec<(usize, Px, Qty)> {
let sgn = if side == Side::Buy { 1.0 } else { -1.0 }; // flips sell prices so "smaller = better" holds for both sides
let mut levels: Vec<(f64, usize, Px, Qty)> = Vec::new();
for (i, v) in self.venues.iter().enumerate() {
if self.rejected.contains(&v.id) { continue; }
let b = v.book.lock().unwrap();
if !b.healthy(Duration::from_secs(2)) {
println!(" [sor] venue {:?} unhealthy (gap/stale feed) — demoted", v.id);
continue;
}
let ladder = match side { Side::Buy => b.ladder(Side::Sell), Side::Sell => b.ladder(Side::Buy) };
for (px, q) in ladder {
let within = if side == Side::Buy { px <= limit } else { px >= limit };
if within { levels.push((sgn * px as f64 + v.fee_ticks + v.rtt_ms * LAT_PENALTY_PER_MS, i, px, q)); }
}
}
levels.sort_by(|x, y| x.0.partial_cmp(&y.0).unwrap()); // floats aren't fully ordered in Rust (NaN), hence partial_cmp
let (mut need, mut alloc) = (qty, HashMap::new()); // venue idx -> (worst px, qty)
for (_, i, px, q) in levels {
if need == 0 { break; }
let take = need.min(q);
let e = alloc.entry(i).or_insert((px, 0));
e.0 = px; e.1 += take; need -= take; // walk is per-venue price-ordered, so last px = child limit
}
let mut out: Vec<(usize, Px, Qty)> = alloc.into_iter().map(|(i, (px, q))| (i, px, q)).collect();
out.sort_by_key(|c| c.0);
out
}
// Fold one exec report into SOR state: Ack feeds the RTT estimate, Fill burns
// down leaves, Reject penalty-boxes the venue, CancelAck retires the child.
fn apply_exec(&mut self, r: &ExecReport) {
match &r.exec {
ExecType::Ack => if let Some(t0) = self.inflight.remove(&r.client_id) {
let ms = t0.elapsed().as_secs_f64() * 1e3;
// The EWMA update — nudge the running estimate 30% toward what just
// happened: keep 70% of the old value, blend in 30% of this reading.
// Old readings fade geometrically; the penalty tracks the venue *now*.
if let Some(v) = self.venues.iter_mut().find(|v| v.id == r.venue) { v.rtt_ms = 0.7 * v.rtt_ms + 0.3 * ms; }
},
ExecType::Fill { px, qty } => if let Some(e) = self.open_children.get_mut(&r.client_id) {
e.1 = e.1.saturating_sub(*qty); // subtract, flooring at zero — an unsigned count can't go negative, and wrapping would turn an over-fill into a huge bogus "leaves"
let done = e.1 == 0;
if done { self.open_children.remove(&r.client_id); }
self.cur_fills.push((r.venue, *px, *qty));
println!(" [fill] {:?} {} {qty}@{px}", r.venue, r.client_id);
},
ExecType::Reject(reason) => {
println!(" [sor] venue {:?} REJECT {}: {reason}", r.venue, r.client_id);
self.open_children.remove(&r.client_id);
self.inflight.remove(&r.client_id);
self.rejected.insert(r.venue);
}
ExecType::CancelAck { qty } => {
if *qty > 0 { println!(" [sor] cancel-ack {}: {qty} confirmed unfilled", r.client_id); }
self.open_children.remove(&r.client_id);
}
}
}
// Wait for exec reports until every child in this batch is terminal — or things go quiet.
fn collect(&mut self, batch: &[String]) {
while batch.iter().any(|c| self.open_children.contains_key(c)) {
// recv_timeout: await the next report, but give up after 250 ms of silence — never hang on a quiet channel
match self.exec_rx.recv_timeout(Duration::from_millis(250)) {
Ok(r) => self.apply_exec(&r),
Err(_) => return, // quiet: remaining children are resting at a venue
}
}
}
/// Double-fill guard: cancel, then WAIT for CancelAck (or a racing Fill)
/// for every child before the caller may re-route the remainder anywhere else.
pub fn cancel_children(&mut self, batch: &[String]) {
let pending: Vec<String> = batch.iter().filter(|c| self.open_children.contains_key(*c)).cloned().collect();
for cid in &pending {
let venue = self.open_children[cid].0;
let v = self.venues.iter().find(|v| v.id == venue).unwrap();
v.tx.send(GatewayIn::Cancel { client_id: cid.clone() }).unwrap(); // unwrap: a dead venue thread is a bug worth crashing on
}
let deadline = Instant::now() + Duration::from_secs(1);
while pending.iter().any(|c| self.open_children.contains_key(c)) && Instant::now() < deadline {
if let Ok(r) = self.exec_rx.recv_timeout(Duration::from_millis(200)) { self.apply_exec(&r); }
}
}
/// Run one parent order: up to 3 passes of plan -> send children -> collect ->
/// cancel-and-wait -> re-plan, then the TCA report.
pub fn execute_parent(&mut self, pid: &str, side: Side, qty: Qty, limit: Px) {
let mid = self.arrival_mid();
self.cur_fills.clear();
self.rejected.clear();
println!("[parent {pid}] {side:?} {qty} limit {limit} | arrival mid {mid:.1}");
let mut remaining = qty;
for _pass in 0..3 {
if remaining == 0 { break; }
let plan = self.plan(side, remaining, limit);
if plan.is_empty() { println!(" [sor] no eligible liquidity within limit — leaving {remaining} unfilled"); break; }
let mut batch = Vec::new();
for (i, px, q) in plan {
self.next_child += 1;
let cid = format!("{pid}-C{}", self.next_child);
let v = &self.venues[i];
println!(" [sor] child {cid} -> venue {:?}: {q}@{px} (fee {:.1}t, rtt {:.1}ms)", v.id, v.fee_ticks, v.rtt_ms);
self.open_children.insert(cid.clone(), (v.id, q));
self.inflight.insert(cid.clone(), Instant::now());
v.tx.send(GatewayIn::New(NewOrder { client_id: cid.clone(), side, px, qty: q })).unwrap();
batch.push(cid);
}
self.collect(&batch);
let filled: Qty = self.cur_fills.iter().map(|f| f.2).sum();
remaining = qty - filled.min(qty);
if remaining > 0 {
println!(" [sor] {remaining} unfilled — cancel any resting children, wait for acks, re-plan");
self.cancel_children(&batch);
}
}
self.report_tca(pid, side, mid);
}
// Print the TCA summary: total filled, average price, slippage vs arrival mid, per-venue split.
fn report_tca(&self, pid: &str, side: Side, mid: f64) {
let tot: Qty = self.cur_fills.iter().map(|f| f.2).sum();
if tot == 0 { println!("[parent {pid}] filled 0"); return; }
let notional: f64 = self.cur_fills.iter().map(|f| f.1 as f64 * f.2 as f64).sum();
let sgn = if side == Side::Buy { 1.0 } else { -1.0 };
println!("[parent {pid}] filled {tot} avg {:.2} | slippage vs arrival mid: {:+.2} ticks",
notional / tot as f64, sgn * (notional / tot as f64 - mid));
let mut by: BTreeMap<VenueId, (Qty, f64)> = BTreeMap::new();
for (v, px, q) in &self.cur_fills { let e = by.entry(*v).or_insert((0, 0.0)); e.0 += q; e.1 += *px as f64 * *q as f64; }
for (v, (q, n)) in by { println!(" TCA {v:?}: {q} @ avg {:.2} ({:+.2} ticks vs mid)", n / q as f64, sgn * (n / q as f64 - mid)); }
}
// Send one child straight to a venue — scenario helper for parking resting orders.
pub fn place_resting(&mut self, cid: &str, venue: VenueId, side: Side, px: Px, qty: Qty) {
let v = self.venues.iter().find(|v| v.id == venue).unwrap();
self.open_children.insert(cid.to_string(), (venue, qty));
self.inflight.insert(cid.to_string(), Instant::now());
v.tx.send(GatewayIn::New(NewOrder { client_id: cid.into(), side, px, qty })).unwrap();
}
// Mass-cancel at every venue, then drain acks until open children reconcile to zero.
pub fn kill_switch(&mut self) {
println!("[kill] global cancel — {} open child orders", self.open_children.len());
for v in &self.venues { v.tx.send(GatewayIn::CancelAll).unwrap(); }
let deadline = Instant::now() + Duration::from_secs(1);
while !self.open_children.is_empty() && Instant::now() < deadline {
if let Ok(r) = self.exec_rx.recv_timeout(Duration::from_millis(200)) { self.apply_exec(&r); }
}
println!("[kill] reconcile: open child orders = {} (must be 0)", self.open_children.len());
assert!(self.open_children.is_empty(), "orphan child orders after kill switch");
}
// The parent's fills, packaged as the JSON report a client would receive.
pub fn client_report(&self, parent: &str) -> String {
let fills: Vec<FillLine> = self.cur_fills.iter().map(|(v, px, q)| FillLine { venue: *v, px: *px, qty: *q }).collect();
let filled: Qty = fills.iter().map(|f| f.qty).sum();
let avg = if filled == 0 { 0.0 } else { fills.iter().map(|f| (f.px * f.qty as i64) as f64).sum::<f64>() / filled as f64 };
// serde_json::to_string: Serialize-derived struct -> JSON text, one call
serde_json::to_string(&ClientReport { parent: parent.into(), filled, avg_px: avg, fills }).unwrap()
}
}
}Step 6 — Wiring + four scenarios
main.rs is wiring plus four scripted scenarios. The map before you scroll:
- (a) baseline — proves the cross-venue split from Step 5’s worked example, plus the cancel-ack guard and an idempotency probe (replaying an already-used child ID).
- (b) feed lag — starves B’s feed queue until the venue drops deltas; the adapter gap-detects, the SOR routes around B, and a periodic snapshot repairs the book.
- (c) rate limit — exhausts A’s gateway token budget; the fourth parent is rejected pre-sequencer and spills to B on the re-plan.
- (d) kill switch — pulls the global cancel mid-parent and proves reconciliation reaches zero open orders.
The plumbing: build wires, per venue, one bounded feed channel (a bounded queue = a finite socket buffer), one adapter thread, and one VenueLink for the SOR. b_cap and b_lag are scenario (b)’s knobs — the size of B’s feed queue and how slowly B’s adapter drains it. Each venue’s rtt_ms starts seeded at 2.0 * lat — deliberately pessimistic (twice the configured one-way latency) so the EWMA has something sane to correct once real acks are measured. shutdown works by dropping the SOR: that drops the order-entry senders, so the venues drain and exit, their feed channels close, and the adapters exit — a clean cascade with no shutdown flag.
// src/main.rs
mod adapter; mod book; mod sor; mod types; mod venue;
use crossbeam_channel::{bounded, unbounded};
use sor::{Sor, VenueLink};
use std::sync::{Arc, Mutex};
use std::thread::{sleep, JoinHandle};
use std::time::Duration;
use types::*;
use venue::VenueConfig;
// Shared venue config; the knobs that differ per venue come in as arguments.
fn cfg(id: VenueId, lat_ms: u64, fee_ticks: f64, tokens: u32, seed_liq: Vec<(Side, Px, Qty)>) -> VenueConfig {
VenueConfig { id, ref_px: 10_000, fee_ticks, latency: Duration::from_millis(lat_ms),
max_qty: 1_000, gw_tokens: tokens, snap_every: 10, seed_liq }
}
fn cfg_a() -> VenueConfig { // fast + deep, but wide spread and higher fee
cfg(VenueId::A, 2, 0.6, 100, vec![(Side::Buy, 9_996, 400), (Side::Buy, 9_994, 500),
(Side::Sell, 10_004, 400), (Side::Sell, 10_006, 500)])
}
fn cfg_b() -> VenueConfig { // slow + thin, but tight spread and lower fee
cfg(VenueId::B, 20, 0.2, 100, vec![(Side::Buy, 9_999, 150), (Side::Buy, 9_997, 200),
(Side::Sell, 10_001, 150), (Side::Sell, 10_003, 200), (Side::Sell, 10_005, 300)])
}
struct Market { sor: Sor, venues: Vec<JoinHandle<Vec<String>>>, adapters: Vec<JoinHandle<()>> }
// Wire the whole market: per venue, one feed channel + adapter thread + VenueLink;
// one shared exec-report channel feeding the SOR.
fn build(cfgs: [VenueConfig; 2], b_cap: usize, b_lag: Option<Duration>) -> Market {
let (exec_tx, exec_rx) = unbounded();
let (mut links, mut vj, mut aj) = (Vec::new(), Vec::new(), Vec::new());
let [ca, cb] = cfgs;
for (c, cap, lag) in [(ca, 64usize, None), (cb, b_cap, b_lag)] {
let (ftx, frx) = bounded(cap); // the mail slot from Step 3, `cap` letters deep — this IS the "socket buffer" the venue drops on
let (id, fee, lat) = (c.id, c.fee_ticks, c.latency.as_millis() as f64);
let h = venue::spawn(c, exec_tx.clone(), vec![ftx]);
let book = Arc::new(Mutex::new(adapter::NormBook::new()));
aj.push(adapter::spawn(id, frx, book.clone(), lag));
links.push(VenueLink { id, tx: h.tx, book, fee_ticks: fee, rtt_ms: 2.0 * lat });
vj.push(h.join);
}
sleep(Duration::from_millis(60)); // let adapters ingest the initial snapshots
Market { sor: Sor::new(links, exec_rx), venues: vj, adapters: aj }
}
// Drop-driven teardown; the joins prove every thread actually exited.
fn shutdown(m: Market) {
drop(m.sor); // drops order-entry senders -> venues drain + exit -> feeds close -> adapters exit
for j in m.venues { j.join().unwrap(); } // join: block until the thread exits and hand back its return value; unwrap re-raises any panic it died with
for j in m.adapters { j.join().unwrap(); }
}
// (a) Baseline: the Step 5 split, an idempotency probe, and the cancel-ack re-route guard.
fn scenario_a() {
println!("\n=== (a) baseline: split across A+B, TCA vs arrival mid ===");
let mut m = build([cfg_a(), cfg_b()], 64, None);
m.sor.execute_parent("P1", Side::Buy, 500, 10_006);
println!("{}", m.sor.client_report("P1"));
// idempotency probe: replay child P1-C1's id at venue A -> gateway rejects the duplicate
m.sor.venues[0].tx.send(GatewayIn::New(NewOrder { client_id: "P1-C1".into(), side: Side::Buy, px: 10_004, qty: 10 })).unwrap();
if let Ok(r) = m.sor.exec_rx.recv_timeout(Duration::from_millis(300)) { println!("replay P1-C1 -> {:?}", r.exec); }
// partial fill + re-route: a resting child must be CANCELED AND ACKED before re-routing
m.sor.place_resting("P2-R1", VenueId::A, Side::Buy, 9_998, 200); // below the ask: rests
sleep(Duration::from_millis(50));
println!("[parent P2] re-route: cancel resting child, wait for cancel-ack, only then send remainder");
m.sor.cancel_children(&["P2-R1".into()]);
m.sor.execute_parent("P2", Side::Buy, 200, 10_006); // safe: venue confirmed 200 unfilled
shutdown(m);
}
// (b) Slow subscriber: overflow B's feed queue, watch gap -> demote -> snapshot recovery.
fn scenario_b() {
println!("\n=== (b) venue B feed lags: gap -> demote -> route around -> snapshot recovery ===");
let mut m = build([cfg_a(), cfg_b()], 2, Some(Duration::from_millis(60))); // tiny buffer + slow adapter
for i in 0..30 { // churn on B overflows the lagging subscriber's queue -> venue drops deltas
m.sor.venues[1].tx.send(GatewayIn::New(NewOrder { client_id: format!("N{i}"), side: Side::Sell, px: 10_006 + (i % 3), qty: 10 })).unwrap();
}
sleep(Duration::from_millis(500)); // by now B's adapter has hit the gap and is in recovery
m.sor.execute_parent("P1", Side::Buy, 300, 10_006);
for i in 0..12 { // slow churn: once the adapter drains, a periodic snapshot repairs the book
m.sor.venues[1].tx.send(GatewayIn::New(NewOrder { client_id: format!("R{i}"), side: Side::Sell, px: 10_009, qty: 5 })).unwrap();
sleep(Duration::from_millis(80));
}
sleep(Duration::from_millis(800));
shutdown(m);
}
// (c) Token-bucket burst: A's gateway runs dry; the re-plan spills to B.
fn scenario_c() {
println!("\n=== (c) burst: venue A gateway rate-limits, SOR spills to B ===");
let a = cfg(VenueId::A, 2, 0.1, 3, vec![(Side::Buy, 9_998, 600), (Side::Sell, 10_001, 600)]);
let b = cfg(VenueId::B, 20, 0.2, 100, vec![(Side::Buy, 9_997, 600), (Side::Sell, 10_004, 600)]);
let mut m = build([a, b], 64, None);
for p in 1..=4 { m.sor.execute_parent(&format!("P{p}"), Side::Buy, 100, 10_008); }
shutdown(m);
}
// (d) Kill switch with live children at both venues; reconciliation must hit zero.
fn scenario_d() {
println!("\n=== (d) kill switch mid-parent: global cancel, zero orphans ===");
let mut m = build([cfg_a(), cfg_b()], 64, None);
m.sor.place_resting("K-1", VenueId::A, Side::Buy, 9_997, 200);
m.sor.place_resting("K-2", VenueId::B, Side::Buy, 9_998, 150);
sleep(Duration::from_millis(80)); // children are live at both venues
m.sor.kill_switch();
shutdown(m);
}
// A seeded pseudo-random generator: same seed, same sequence, forever, on any
// machine. That predictability is the whole point — the determinism check must be
// able to feed the venue an IDENTICAL order script twice and diff the logs.
struct Rng(u64); // xorshift64: 8 bytes of state, no rand crate
impl Rng {
// seed.max(1): xorshift state must never be 0 — 0 maps to 0 forever.
fn new(seed: u64) -> Self { Rng(seed.max(1)) }
// Each call: three shift-and-XOR rounds smear the state's bits around. It looks
// random but is pure arithmetic — nothing reads a clock or an OS entropy pool.
fn next(&mut self) -> u64 { let mut x = self.0; x ^= x << 13; x ^= x >> 7; x ^= x << 17; self.0 = x; x }
}
// Drive one venue with 40 seeded pseudo-random orders; return its event log.
fn scripted_venue(seed: u64) -> Vec<String> {
let (etx, _erx) = unbounded();
let (ftx, _frx) = bounded(4_096);
let h = venue::spawn(cfg_a(), etx, vec![ftx]);
let mut rng = Rng::new(seed);
for i in 0..40 {
let side = if rng.next() % 2 == 0 { Side::Buy } else { Side::Sell };
let (px, qty) = (9_995 + (rng.next() % 11) as Px, 10 + rng.next() % 90);
h.tx.send(GatewayIn::New(NewOrder { client_id: format!("D{i}"), side, px, qty })).unwrap();
}
drop(h.tx);
h.join.join().unwrap()
}
// Same seed, two runs, byte-identical logs — the event-sourcing claim, checked.
fn determinism() {
let (r1, r2) = (scripted_venue(42), scripted_venue(42));
assert_eq!(r1, r2);
println!("\n[determinism] venue A replayed on seed 42: {} log lines, byte-identical", r1.len());
}
// Everything in sequence; each scenario builds and tears down its own market.
fn main() { scenario_a(); scenario_b(); scenario_c(); scenario_d(); determinism(); println!("\nall scenarios complete"); }cargo run --release (~5 s total). Timings and RTT figures below will wobble on your machine; prices, quantities, and event ordering will not — the scripted sleeps hold these scenarios’ broker outcomes stable, though in general (Step 7) the broker side wobbles too.
Scenario (a) — the merged cost walk plays out exactly as Step 5’s worked example: B’s best prices win first even though B is slow, and once B’s cheap levels run out, A’s bigger size takes over. (B’s RTT shows as 40 ms because we seeded the estimate at twice its 20 ms configured latency — no ack has been measured yet.) The summary at the end is the TCA report from ch26: average fill price against the mid-price at the moment the order arrived (the “arrival mid”), the difference being the slippage:
=== (a) baseline: split across A+B, TCA vs arrival mid ===
[parent P1] Buy 500 limit 10006 | arrival mid 10000.0
[sor] child P1-C1 -> venue A: 150@10004 (fee 0.6t, rtt 4.0ms)
[sor] child P1-C2 -> venue B: 350@10003 (fee 0.2t, rtt 40.0ms)
[fill] A P1-C1 150@10004
[fill] B P1-C2 150@10001
[fill] B P1-C2 200@10003
[parent P1] filled 500 avg 10002.70 | slippage vs arrival mid: +2.70 ticks
TCA A: 150 @ avg 10004.00 (+4.00 ticks vs mid)
TCA B: 350 @ avg 10002.14 (+2.14 ticks vs mid)
{"parent":"P1","filled":500,"avg_px":10002.7,"fills":[{"venue":"A","px":10004,"qty":150},{"venue":"B","px":10001,"qty":150},{"venue":"B","px":10003,"qty":200}]}
replay P1-C1 -> Reject("duplicate client_order_id")
[parent P2] re-route: cancel resting child, wait for cancel-ack, only then send remainder
[sor] cancel-ack P2-R1: 200 confirmed unfilled
[parent P2] Buy 200 limit 10006 | arrival mid 10001.0
[sor] child P2-C3 -> venue A: 200@10004 (fee 0.6t, rtt 18.7ms)
[fill] A P2-C3 200@10004
[parent P2] filled 200 avg 10004.00 | slippage vs arrival mid: +3.00 ticks
TCA A: 200 @ avg 10004.00 (+3.00 ticks vs mid)
(Notice P2’s RTT estimate jumped to ~19 ms: the resting child’s ack sat unread in the broker’s inbox during our 50 ms sleep, so when we finally read it, the measured “round trip” included our own nap — and the EWMA honestly absorbed that. Measured latency includes your own processing delays — a real SOR lesson for free.)
Scenario (b) — the overflow is pure arithmetic: B’s adapter takes 60 ms per message (b_lag), the burst of 30 orders produces deltas far faster than that, and the queue between them holds only 2 (b_cap). The queue fills in the first burst, so the venue’s try_send starts failing — those are the dropped deltas. The venue never blocks; dropping is the multicast contract:
time ↓ BROKER (adapter B + SOR) │ VENUE A │ VENUE B
(1) 30 sell orders ═══════════════════════╪═════════▶ deltas flood the feed
(2) B's feed queue (cap 2) fills → venue try_send DROPS deltas — no blocking
(3) adapter B: seq 8 after 6 → GAP — │ │
recovering = true, book untrusted │ │
(4) P1: plan() → B !healthy() → demoted │ │
(5) child ════════════════════════════════▶ fills 300@10004 — A only
(6) slow churn ═══════════════════════════╪═════════▶ adapter drains at its pace
(7) periodic Snapshot ◀═══════════════════╪═════════╡ book rebuilt wholesale
(8) recovering = false → B healthy again, back in the next merge
=== (b) venue B feed lags: gap -> demote -> route around -> snapshot recovery ===
[adapter B] GAP: expected seq 6 got 8 — waiting for snapshot
[parent P1] Buy 300 limit 10006 | arrival mid 10000.0
[sor] venue B unhealthy (gap/stale feed) — demoted
[sor] child P1-C1 -> venue A: 300@10004 (fee 0.6t, rtt 4.0ms)
[fill] A P1-C1 300@10004
[parent P1] filled 300 avg 10004.00 | slippage vs arrival mid: +4.00 ticks
TCA A: 300 @ avg 10004.00 (+4.00 ticks vs mid)
[adapter B] recovered via snapshot seq=45
Scenario (c) — A is cheaper here, so the first three parents drain its gateway token budget — a token bucket: each venue’s gateway allows a fixed number of new orders per window, and this scenario’s A was configured with only 3 tokens (production gateways refill the bucket on a timer; the lab deliberately never refills within a run). The fourth parent is rejected pre-sequencer and spills to B on the re-plan — the next pass of execute_parent’s loop from Step 5: cancel anything resting, wait for the acks, plan again with A in the penalty box:
time ↓ BROKER (SOR) │ VENUE A (3 tokens) │ VENUE B
(1) P1..P3 children ══════════════════════▶ tokens 3→2→1→0, │
(2) P4 child ═════════════════════════════▶ bucket empty │
(3) ◀══ Reject("gateway rate limit") ═════╡ pre-sequencer: │
apply_exec: A → penalty box (rejected)│ no seq consumed │
(4) cancel_children (nothing resting), then re-plan │
(5) plan(): A skipped → child ════════════╪════════════════════▶ gw→seq→engine
(6) ◀═══════════ Ack + Fill 100@10004 ════╪════════════════════╡
=== (c) burst: venue A gateway rate-limits, SOR spills to B ===
[parent P1] Buy 100 limit 10008 | arrival mid 10000.0
[sor] child P1-C1 -> venue A: 100@10001 (fee 0.1t, rtt 4.0ms)
[fill] A P1-C1 100@10001
[parent P1] filled 100 avg 10001.00 | slippage vs arrival mid: +1.00 ticks
...P2, P3 identical, filled at A...
[parent P4] Buy 100 limit 10008 | arrival mid 10000.0
[sor] child P4-C4 -> venue A: 100@10001 (fee 0.1t, rtt 3.1ms)
[sor] venue A REJECT P4-C4: gateway rate limit
[sor] 100 unfilled — cancel any resting children, wait for acks, re-plan
[sor] child P4-C5 -> venue B: 100@10004 (fee 0.2t, rtt 40.0ms)
[fill] B P4-C5 100@10004
[parent P4] filled 100 avg 10004.00 | slippage vs arrival mid: +4.00 ticks
Scenario (d) — kill switch: mass-cancel at both venues, then reconcile the broker’s open-order table against venue acks. Zero means zero:
time ↓ BROKER (SOR) │ VENUE A │ VENUE B
(1) K-1 rests at A, K-2 at B — open_children = 2
(2) kill_switch: tx.send(CancelAll) ══════▶ cancel_all
tx.send(CancelAll) ══════╪═════════▶ cancel_all
(3) ◀══ CancelAck K-1 (200 unfilled) ═════╡ │
(4) ◀══ CancelAck K-2 (150 unfilled) ═════╪═════════╡ each ack retires a child
(5) reconcile: open_children == 0 — venue-CONFIRMED zero, then the assert
=== (d) kill switch mid-parent: global cancel, zero orphans ===
[kill] global cancel — 2 open child orders
[sor] cancel-ack K-1: 200 confirmed unfilled
[sor] cancel-ack K-2: 150 confirmed unfilled
[kill] reconcile: open child orders = 0 (must be 0)
Step 7 — Determinism: one side has it, the other never will
[determinism] venue A replayed on seed 42: 66 log lines, byte-identical
VENUE lane — deterministic BROKER lane — nondeterministic (timing)
run 1: script(seed 42) ─▶ venue ─▶ log₁ threads race: ack interleaving, RTTs,
run 2: script(seed 42) ─▶ venue ─▶ log₂ book-at-plan-time all wobble run to
assert_eq!(log₁, log₂) — byte-compare ✓ run → no replay; keep a timestamped
(same sequenced input ⇒ same event log) DECISION LOG instead
The venue is deterministic because its entire state is a function of the sequenced input: one consumer, no wall-clock in any decision, artificial latency that delays but never reorders — ch13’s event-sourcing claim made concrete, and why real venues can replay a day from the sequenced log for disputes and testing.
The broker side is not deterministic and cannot be made so cheaply: the threads race, so the same parent order can legitimately split differently on two runs.
Real systems don’t fight this; they evidence it: every routing decision is logged with a timestamp and the inputs it saw (the normalized books, health flags, measured RTTs). That decision log — not a replayable world — is what backs a best-execution defense (ch26).
Plain-English recap
- A venue is three stages: gateway (per-session checks: collar, size, duplicate ID, rate limit — all before sequencing), sequencer (one consumer = the total order), engine (pure function of the sequence). Rejects consume no sequence number.
- The venue has two outputs with different contracts: private exec reports (reliable, to you) and a public feed (best-effort, seq-numbered, snapshot every N so anyone can join or recover).
- A feed publisher never blocks on a slow subscriber — it drops. Recovery is the subscriber’s job: detect the seq gap, distrust the book, rebuild from the next snapshot.
- A broker adapter’s product is a normalized book plus a health flag. The SOR consumes both: cost model over healthy venues, demotion for the rest.
- Routing cost is not just price: fees and latency (measured, not assumed) shift the split.
- Re-routing an unfilled child without waiting for the cancel-ack is how you buy the same shares twice. Cancel, wait for the terminal event, then re-route.
- A kill switch is only done when reconciliation says zero open orders — venue-confirmed, not assumed.
Interviewer will ask
“Where does an order become real?” At the sequencer. My lab’s gateway rejects (collar, dup ID, rate limit) happen before sequencing and consume no seq number; from the sequencer onward everything is a deterministic function of the sequence — I verified byte-identical logs on replay.
“Why can’t the feed publisher just block briefly on a slow consumer?” Then one slow subscriber sets the venue’s pace for everyone — an outsider controlling the matching path. I used try_send and dropped; in scenario (b) the venue stayed at full speed while one subscriber went blind and recovered later.
“How does your feed handler recover from a loss?” Contiguous seq check on every delta; on a gap I mark the book untrusted and apply nothing until the next periodic snapshot (published every N deltas), then resume from its seq. Meanwhile the SOR sees recovering and routes around that venue.
“Why cancel-then-wait before re-routing?” Because cancel is a race against a fill in flight. If I re-route while the child might still execute, both can fill — a doubled position. The venue’s CancelAck (or the racing Fill) is the only authority on how much was left; I re-route exactly that confirmed quantity.
“How does your SOR choose venues?” Effective cost per level: price + venue fee + a latency penalty from an EWMA of measured ack RTTs, merged across venues, cheapest marginal depth first — so it splits exactly when one venue’s next level is worse than the other venue’s best remaining. Health-gated: gapped or stale feeds are excluded before pricing.
“What did your kill switch actually prove?” Not that I sent cancels — that reconciliation reached zero: every child the broker believed open was matched by a venue CancelAck. Open state you can’t reconcile is risk you can’t bound.
“Which half is deterministic and why?” Venue: yes — single sequencer, state a pure function of sequenced input; identical logs across runs. Broker: no — independent venue threads race, so decisions vary. The fix isn’t determinism, it’s timestamped decision logging: record what the router saw when it chose, for best-ex evidence.
“Why check duplicate client order IDs at the gateway?” Idempotency for retransmits: if my session resends after a timeout, the dup check turns a would-be double execution into a reject before the sequencer. I demonstrated it by replaying a filled child’s ID and getting Reject("duplicate client_order_id").
Further reading
- Larry Harris, Trading and Exchanges — venue microstructure, order precedence, why the rules look like this.
- Nasdaq TotalView-ITCH 5.0 and OUCH protocol specifications — real seq-numbered feed and order-entry message sets; compare with your
FeedMsg/GatewayIn. - CME MDP 3.0 market data documentation — incremental + snapshot recovery channels in production form.
- FIX Trading Community, FIX protocol specification — ExecutionReport lifecycle (your
ExecTypein the wild). - SEC Regulation NMS (esp. Rule 611) and FINRA Rule 5310 — the regulatory frame behind “best execution evidence”.
The sentences you can now say in interviews truthfully
- “I’ve built a two-venue market end-to-end in Rust — gateway, sequencer, matching engine, seq-numbered feed with snapshot recovery, plus the broker side: adapters, SOR, and a parent-order manager.”
- “I’ve watched a venue drop feed messages on a slow subscriber and implemented the gap-detect-then-resnapshot recovery on the consuming side.”
- “My SOR splits parents using effective cost — price plus fees plus a measured latency penalty — and demotes venues on feed-health signals.”
- “I’ve implemented the cancel-then-wait-for-ack guard and can explain exactly which race it closes.”
- “I’ve verified a sequenced venue is byte-for-byte replayable, and I can explain why the multi-venue broker side isn’t — and what decision logging does about it.”
- “I’ve built a kill switch that proves completion by reconciling open-order state to zero against venue acks.”
Question Bank: Networking
25 questions, ordered easy to brutal. Model answers are written in your voice: first-person where your 20+ crypto venue integrations give you standing, and explicitly honest (“I haven’t run X in prod; here’s how I’d approach it”) for colo/bypass tech you haven’t touched. Interviewers reward that candor heavily — the fastest way to fail these rounds is to bluff kernel-bypass war stories you don’t have. Practice answering out loud; 3–8 sentences each, no more.
1. What actually happens, end to end, when a market-data packet arrives at your NIC? NIC DMAs the frame into a ring-buffer descriptor in RAM, raises an interrupt (or is polled), the driver hands it to the kernel network stack — IP reassembly, TCP/UDP demux, socket buffer copy — then the epoll/recv wakes my thread, possibly on another core, and I copy into userspace and parse. Each stage costs: interrupt + context switch (µs-scale), at least one kernel-to-user copy, cache misses from waking on a cold core. That whole tour is why the kernel path floors out around single-digit microseconds and why bypass exists: map the NIC rings into userspace and poll, skipping interrupt, syscall, and copy.
2. What does TCP_NODELAY do and when do you set it? It disables Nagle’s algorithm, which delays small writes hoping to coalesce them into fewer segments. Nagle is throughput logic; an order message is a small write whose latency is all that matters, so every order-path and market-data socket I own gets NODELAY set at connect. In my crypto stack that’s the first thing checked when someone reports “orders are slow at low volume” — Nagle plus a quiet connection produces exactly that signature.
3. Explain the Nagle + delayed-ACK interaction. Nagle holds a second small segment until the first is ACKed; delayed ACK holds the ACK up to ~40–200ms hoping to piggyback on data. Put a Nagle sender opposite a delayed-ACK receiver with a write-write-read pattern (the story from ch02: send a header, send a body, wait for the reply) and each waits on the other: the sender won’t send segment two un-ACKed, the receiver won’t ACK segment one until its delayed-ACK timer expires. Result: latency spikes of tens of ms that appear only for certain message patterns. Fix on your side with NODELAY; the failure mode is worth knowing because you often can’t fix the venue’s side.
4. Why UDP multicast for market data in tradfi, and what does it cost you? One send reaches all subscribers with no per-receiver fan-out cost, no receiver can slow the sender (no backpressure), and switches replicate in hardware — it’s the only economical way to feed hundreds of subscribers the same firehose at the same time. The cost is that UDP guarantees nothing: loss, reordering, no retransmit, so reliability becomes the application’s problem — sequence numbers on every packet and a recovery story. Crypto flipped this: venues serve fan-out over per-client TCP/WebSocket because clients are on the open internet, which is why my world’s failure modes are slow-consumer disconnects rather than multicast gaps.
5. How do you detect and recover a gap on a multicast feed? Every packet carries a sequence number; a jump from n to n+k means k−1 lost. Recovery layers, in order of preference: arbitrate the A/B feed (venues transmit each packet on two lanes — most “gaps” are filled by the twin), then a retransmission request service for small gaps, then a snapshot/recovery channel to rebuild book state when you’re too far behind, buffering live increments to splice after the snapshot. I haven’t run exchange multicast in prod, but the splice logic is identical to what I do daily on crypto: REST/WS book snapshot plus buffered deltas, apply deltas with seq > snapshot seq. The design question underneath is always “how stale before you stop trading” — gap handling must set a poisoned flag the strategy respects.
6. Busy-polling vs interrupts — the actual trade. Interrupts let the core sleep and cost you wakeup latency and jitter (µs-scale, scheduler-dependent) per event; busy-polling burns a core at 100% spinning on the ring/socket and gets you consistent sub-µs reaction. Trading hot paths busy-poll on pinned, isolated cores because the jitter matters more than the electricity. Two caveats: you need core isolation (isolcpus/nohz_full) or the spinner gets preempted anyway; and for the 18 quiet venues out of 20, parking them on a shared epoll thread is correct — burn cores only on the venues that move P&L.
7. What is kernel bypass, concretely — and name the flavors. Userspace owns the NIC: the device’s TX/RX rings are mapped into the process, which polls and builds/parses frames itself — no interrupts, no syscalls, no kernel copies. Flavors: DPDK (take over the whole NIC, bring your own TCP stack), Solarflare/Xilinx ef_vi and Onload (Onload is the gentle one — LD_PRELOAD sockets API acceleration, no code change), AF_XDP (kernel-sanctioned middle path), and full FPGA NICs beyond that. I haven’t operated these in prod — my venues are internet-distant, where bypass is rounding error — but I’d start any colo build with Onload precisely because it keeps the sockets API while I benchmark whether the extra step to ef_vi/DPDK pays.
8. When is DPDK the wrong choice? When the network isn’t your bottleneck: my crypto reality is 1–50ms internet RTTs to venues, so saving 3µs of kernel stack is noise — engineering effort belongs in strategy and venue-quirk handling. Also wrong when you need the kernel’s TCP stack maturity (DPDK means bringing your own TCP — a giant liability against internet endpoints), when ops maturity is low (you lose tcpdump/netstat tooling and standard debugging), and when Onload gets you 80% of the win for 5% of the cost. It’s a decision tree ordered by RTT scale and team capacity, not a technology preference.
9. Design flow steering for a host handling 20 venues. Goal: hot venues get dedicated resources, the long tail shares. NIC-level: RSS to spread flows across queues, then explicit flow-steering rules (ntuple/Flow Director) pinning each hot venue’s 5-tuple to a dedicated queue whose IRQ (or busy-poll thread) is pinned to a dedicated core co-located with that venue’s parser — packet arrives on the core that will process it, no cross-core handoff. Long-tail venues share a couple of queues and an epoll thread. My version of this in the crypto world is the same shape one layer up: one socket-owner thread per hot venue pinned to a core, quiet venues multiplexed, because with TCP/WS the “flow” is the connection. Include the failure mode: steering rules silently vanish on NIC reset/driver reload, so audit them continuously.
10. Your feed handler sees the book crossed (bid ≥ ask). What do you do? Treat it as a data-integrity alarm, not an arb: mark the book poisoned, stop strategies consuming it, and resync (snapshot + delta splice). Causes I’ve actually hit across venues: missed/dropped delta, venue sending updates out of order across channels, snapshot/delta seq misalignment, and occasionally the venue itself briefly crossed during an auction or outage. The design point is that every consumer must respect a validity flag — a book that silently serves stale or crossed state to a strategy is how you buy the top of a spike. Uncrossing heuristics (“just trust the newer side”) are how people lose money quietly.
11. TCP retransmission timeout just fired on your order session. What did it cost and what do you do? TCP notices loss two ways: a retransmission timer that starts pessimistic at ~200ms, or seeing the same ACK repeated — which needs traffic to keep flowing. (This RTO is TCP’s retransmission timeout — no relation to the recovery-time objective of ch13.) On a busy session fast-retransmit (3 dup ACKs) saves you, but a quiet session generates no dup ACKs, so a lost packet waits out the full timer — a single lost order message eats a fifth of a second, and my order sessions are exactly that: quiet. Mitigations: keep the session warm (heartbeats also serve as loss detectors), tune what’s tunable, monitor per-socket retransmit counters (ss -i) and alert, and at the app layer treat an un-acked order past a deadline as state-unknown — never assume delivered or not delivered; reconcile. This “order in flight, session degraded” ambiguity is the worst state in trading networking and deserves an explicit state machine.
12. How would you timestamp packets honestly, and where do people lie to themselves? Hardware timestamps at the NIC PHY (SO_TIMESTAMPING with a PTP-disciplined clock) are ground truth for wire arrival; software timestamps at recv include kernel and scheduling noise and drift unless NTP/PTP is tight. People lie by: timestamping in the app and calling it wire time (hides the whole stack), comparing timestamps across unsynchronized hosts (clock skew swamps µs claims), and quoting venue-provided timestamps as if the venue’s clock were their own. My honest version: I run software timestamps disciplined by NTP because at internet RTTs that’s proportionate, and I’d move to PTP + NIC hardware stamps the day I’m in a colo claiming single-digit-µs numbers — a µs claim is only as good as the clock chain behind it, and in tradfi that chain is a regulatory artifact (RTS 25 flavor).
13. Cloud vs colo: what latency do you actually get, and what’s different in kind, not just degree? Colo gives you stable single-digit-µs to the matching engine and, critically, low variance. Cloud gives crypto reality: 1–50ms RTTs to venues, and — the difference in kind — variance you don’t control: multi-tenant hosts, virtualized NICs, transient reroutes, neighbor noise; your p50 might be fine while p99.9 is 10x. What I do about it, from prod experience: pick regions per venue (AWS Tokyo for the Tokyo-hosted venues), measure continuously per venue (RTT histograms, not averages), run redundant paths/instances and race them, and accept that my edge cannot be pure speed — it’s being fast enough consistently plus being smarter on venue microstructure. Also worth saying: some crypto venues now sell colo-ish proximity (same-AZ placement, private links), so the two worlds are converging.
14. Design a feed handler for a venue with a known-lossy feed. Assume loss is normal, not exceptional. Structure: seq-checked ingest → gap detector → recovery ladder (A/B arbitration if the venue offers dual feeds, retransmit channel for small gaps, snapshot resync for large ones) → book builder that tracks a validity state (LIVE / GAPPED / RESYNCING) exposed to every consumer → buffered-delta splice on recovery. Key design decisions to state: bounded buffer for live deltas during resync (and what to do on overflow: re-snapshot), staleness deadline after which the book is poisoned regardless, per-gap metrics because gap rate is a venue-health signal, and idempotent delta application so overlapped splice regions are harmless. This is my daily bread in crypto — every WS venue is “lossy” via disconnects — so I’d narrate it with a real venue’s snapshot+delta protocol.
15. What’s a speed-of-light budget and why do interviewers ask it? Light in fiber does ~200km/ms (c/1.5), so geography sets hard floors: NY–Chicago ~4ms one-way in fiber, ~3.9ms line-of-sight — which is why microwave networks exist and shave ~25%; Tokyo–Singapore ~27ms+; nothing you do in software beats geography. They ask it to see if you sanity-check systems against physics: if someone claims 2ms Tokyo–London, you should reject it on arithmetic alone. I use the same arithmetic in crypto for venue-region placement and for cross-venue arb feasibility — if venue A is in Tokyo and B in Virginia, the ~75ms one-way defines the whole strategy class.
16. Why can a single TCP connection never keep up with a multicast feed during a burst, and what’s the crypto equivalent? TCP has backpressure: the receiver advertises how much buffer it has left (the receive window), and at zero the sender must stop — so during a market-data burst the venue’s TCP fan-out either buffers (adding latency that compounds) or disconnects slow clients. Multicast has no such coupling; slow receivers just lose packets and recover themselves, protecting the fast ones. The crypto equivalent I’ve lived: venues detect slow WS consumers by send-buffer depth and kill the connection mid-burst — precisely when you least want a resync — so the defense is to drain the socket unconditionally fast (dedicated reader thread that never blocks, drop-to-conflation internally if the strategy can’t keep up — conflation is defined in the next question) and make disconnect-resync cheap because it will happen at the worst time.
17. Explain conflation, and when it’s correct vs a bug. Conflation replaces queued undelivered updates with the latest state — you deliver “the current book,” not every intermediate tick. Correct when the consumer only cares about current state and is slower than the feed: a human UI, a slow risk recalculation, a monitoring dashboard. A bug when the consumer needs the sequence: trade prints for volume signals, or a strategy whose logic depends on order-of-events. Design pattern: conflate per-key (per price level / per symbol) in the queue between feed thread and slow consumers, never on the hot strategy path, and make it explicit in the interface — a consumer must know whether its stream is conflated. Several crypto venues conflate server-side at low tiers; knowing which streams are conflated per venue is exactly the kind of quirk table my 20-venue layer maintains.
18. What does the OS scheduler do to your tail latency, and how do you stop it? Any runnable thread can be preempted, migrated cross-core (cold caches), or delayed by timer ticks, RCU callbacks (deferred kernel cleanup work that runs on your core), and kernel housekeeping — each event is rare but shows up precisely at p99.9+. Containment: pin threads (taskset/pthread_affinity), isolate cores (isolcpus, nohz_full, rcu_nocbs) so the kernel schedules nothing else there, keep IRQs steered away from hot cores, disable deep C-states/turbo transitions for consistency, and busy-poll instead of sleeping so the scheduler has no decision to make. I’ve applied the cheap half of this list (pinning, IRQ steering) on cloud boxes to visible p99.9 effect; full isolation I’ve only lab-tested — cloud hypervisors cap how far it goes, which is itself a point worth making.
19. Two feeds for the same venue disagree. Which do you trust, and how do you even know? First, “disagree” needs machinery: run both through identical normalizers and diff top-of-book with a tolerance window — transient disagreement within propagation delay is normal, persistent disagreement is an incident. Trust hierarchy: prefer the feed whose seq/heartbeat integrity is currently clean; if both are clean but differ persistently, one is stale — check heartbeat ages and last-update timestamps; if you can’t resolve, poison the book and resync both. This shows up constantly in crypto (WS vs REST snapshots are eventually consistent with each other), so my rule from prod: never mix fields from two sources into one book state without a sequencing story — pick a primary, use the secondary for validation and failover.
20. Walk me through diagnosing ‘orders are slow to one venue since Tuesday.’ Bisect the path with data. App-side first: per-venue order-path histograms (I keep them) — did submit→ack change at p50 or only tail? p50 shift smells like route/venue; tail-only smells like my host or bursts. Then network: RTT to venue endpoint over time (ping/TCP handshake times), traceroute vs a saved baseline for route changes, ss -i for retransmits and cwnd (the congestion window — how much TCP currently dares send) on the session, socket-buffer depths for local queueing. Then the venue: status page, other participants, a canary order from a second region — my prod experience is that a plurality of these are venue-side or ISP reroutes, which is exactly why the per-venue baseline histograms exist: without Tuesday’s baseline you can’t even ask the question crisply.
21. Why is the last mile inside your own host often worse than the WAN hop, in crypto? Because 30ms of internet RTT hides sins: a GC pause, a lock, a cross-core handoff, a syslog write on the hot path — each 100µs–10ms — are invisible at p50 but stack the tail, and unlike the WAN they’re yours to fix. I’ve found more latency in JSON parsing and allocator behavior than in routing. So the priority order inverts vs tradfi: profile the process before blaming the network — the WAN sets the floor, the host sets the tail.
22. How would you build the network side of a new colo deployment, never having run one? (Honesty test.) I’d say exactly that, then show the plan: engage the exchange’s connectivity docs and a peer/vendor early (cross-connect specs, feed bandwidth, A/B multicast details); provision Solarflare-class NICs and start with Onload for sockets-API acceleration, benchmarking ef_vi as step two; PTP time infrastructure from day one because every latency claim depends on it; capture appliance or port mirror for ground-truth wire timestamps; burst-size the multicast feeds from spec, not averages (opening auction dictates buffers); and drills for gap recovery before go-live. Frame it as: the concepts — seq/gap/recovery, pinning, steering, honest timestamps — I already operate daily in crypto; what’s new is the specific tooling, and I de-risk that with vendor references and measurement rather than bravado.
23. Multicast joins, IGMP, and what goes wrong at 3am. Picture pub-sub implemented by the network hardware: a receiver’s IGMP join is a subscription message, and the switch is the broker — it snoops the joins to learn which ports want which groups, and it forgets a subscription unless a querier periodically refreshes membership. Failure modes that page people: IGMP snooping misconfigured → either flooding every port (bandwidth collapse) or pruning your port (silent feed death); a membership timeout after a querier hiccup → feed just stops with no error on your socket — from userspace it’s indistinguishable from a quiet market, which is why heartbeat-expected alarms on every feed are non-negotiable; and after switch maintenance, joins that never re-establish. I haven’t run this in prod — it’s the multicast cousin of what I do monitor: a WS connection that’s open but silent — same alarm design (“expected message rate violated”), different layer.
24. Design market-data distribution inside your own shop: 1 feed handler, 30 internal consumers. Don’t let 30 consumers each parse the venue feed — normalize once, distribute internally. Options ladder: shared-memory ring buffer (single-writer, seqlock’d or Disruptor-style — a seqlock means the writer bumps a version counter around each write and readers retry if it changed mid-read) for same-host hot consumers — nanoseconds, no syscalls, slow consumers just fall behind by sequence and know it; Aeron/multicast for cross-host fan-out with the same loss-is-the-consumer’s-problem discipline; TCP/IPC only for the slow tier, with conflation. Key properties to state: single writer per stream, sequence numbers end-to-end (consumers detect their own gaps), no backpressure from consumer to feed handler ever — a slow risk dashboard must not be able to slow the strategy path. This is the internal mirror of Q4/Q16, and interviewers ask it to see if you apply venue-side lessons to your own architecture.
25. Brutal one: your p99.9 tick-to-trade doubled but every network metric you have is flat. Argue that it’s still the network. Flat sampled metrics don’t exonerate the network: microbursts overflow NIC/switch queues in microseconds while 1-second counters average them away — check ring-buffer drop counters and switch telemetry, not rates; retransmits on a different flow can head-of-line-block a shared queue — both flows land in the same NIC RX ring; IRQ steering may have quietly reset after a driver update, landing packets on the wrong core (the “network” is fine, the delivery locality isn’t); and if timestamps come from the app, a clock or scheduling change masquerades as network time. The meta-answer interviewers want: name where each metric is measured, find the unmeasured gaps between measurement points, and instrument the gap — hardware timestamps at the NIC vs app timestamps bracket the stack and settle the argument. Then concede the prior: having chased this in cloud prod, the culprit was usually my host, and I’d say so while still checking the drop counters first — they’re cheaper to read than a profile.
Further reading
- Kleppmann, DDIA ch. 8 (“The Trouble with Distributed Systems”) — unreliable networks and clocks, the theory under Q11/Q12/Q19.
- The Linux kernel networking docs on busy polling and
SO_TIMESTAMPING, plus thess/ethtoolman pages — the practical instrumentation layer for Q6, Q12, Q20, Q25. - Solarflare/AMD Onload and ef_vi documentation; DPDK programmer’s guide — read the intros so your “haven’t run it, here’s the plan” answers cite the actual tools (Q7, Q8, Q22).
- Exchange connectivity specs — CME MDP 3.0 and Nasdaq ITCH/MoldUDP64 docs are public; MoldUDP64’s seq/gap/retransmit design is the canonical Q5/Q14 reference.
- Aeron documentation (aeron.io) — the internal-distribution design of Q24, productized.
Question Bank: Performance
25 questions, easy to brutal. These rounds test one meta-skill: whether your numbers are real — measured correctly, decomposed into stages that add up, and defended against the classic traps (coordinated omission above all — it gets two questions in this bank because it lurks in every measurement loop). Answers are 3–8 sentences, first person, written to be spoken.
1. What is tick-to-trade latency, precisely? Wire-to-wire: first bit of the triggering market-data packet arriving at my NIC to first bit of the resulting order leaving it — because that’s the only definition the market grades me on. Anything measured app-in to app-out silently excludes the network stack and serialization, which can dominate. Internally I decompose it into stages, but the headline number must be wire-to-wire with hardware timestamps or a capture device, or it’s a partial truth.
2. Decompose a tick-to-trade path and put rough numbers on each stage. Colo-class decomposition: NIC ingest + delivery to userspace (kernel path ~1–5µs, bypass ~sub-µs), feed decode (tens of ns for binary, fixed layout), book update (tens of ns, cache-resident), strategy decision (ns to µs depending on model), risk checks (tens of ns if in-process tables), order encode + TX (sub-µs bypass). Software total: single-digit µs kernel-stack, ~1µs tuned bypass, hundreds of ns for the elite, and FPGAs take the decode-to-order core to ~100ns-class. My crypto version has the same shape with a different floor: JSON decode alone can be 1–5µs, and the venue RTT of 1–50ms dominates everything — which is why I profile the host but place the strategy geographically.
3. What is rdtsc and why do we use it for timing? The x86 time-stamp counter: a per-core counter read in ~20–30 cycles, versus ~20–30ns+ for clock_gettime even via vDSO — so it’s the only clock cheap enough to sprinkle through a hot path. Modern CPUs give you invariant/constant TSC (ticks at a fixed rate regardless of frequency scaling) and synchronized-across-cores on sane systems, but you must verify both (CPU flags, and beware VMs/migration). Convert ticks to ns with a calibrated ratio, and remember what it isn’t: not wall clock, not comparable across hosts — for cross-host claims you’re back to PTP.
4. rdtsc vs rdtscp vs fences — when does it matter? rdtsc can execute out of order relative to the code you’re timing — the CPU may hoist it above or sink it below your measured region, corrupting nanosecond-scale measurements. rdtscp waits for prior instructions to retire (and returns a core ID); pairing with lfence/serializing instructions gives stricter ordering at higher cost. For coarse timing (µs+) it’s noise; for timing a 40ns book update it’s the difference between a measurement and a fiction. My rule: rdtscp or lfence+rdtsc at region boundaries when the region is <1µs, and always sanity-check by timing an empty region to know the measurement’s own floor.
5. The classic filter: what is coordinated omission? The measurement bug where your load generator waits for each response before sending the next request, so when the system stalls, you stop measuring during the stall — the periods of worst behavior generate the fewest samples. A 1-second hiccup under intended 1k req/s should contribute ~1000 terrible samples; a coordinating harness records one. Result: tail percentiles that look beautiful and are fabricated. Fix: schedule sends on the intended timeline and measure from intended send time, so a stall’s queueing delay lands in every affected sample — this is Gil Tene’s core critique and HdrHistogram’s correction mode exists precisely for it.
6. Where does coordinated omission hide outside benchmarks? Everywhere a measurement is taken only when the system is healthy enough to take it: an event loop that timestamps work when it dequeues rather than when work arrived (its own stall vanishes), metrics sampled by the same thread that stalls, replay harnesses that feed the next event only after the previous is processed (a real feed would have queued), and health checks that can’t run during the incident they exist to catch. My defense in prod: timestamp at ingress (ideally NIC), compute latency as completion-minus-arrival, and alert on absence of measurements as loudly as on bad ones.
7. Mean, p50, p99, p99.9, max — which do you engineer for and why? For trading: the tail, because losses correlate with exactly the moments that produce tails — volatile markets generate both the most opportunity and the most load, so the p99.9 during a burst is the latency of your most important trades, not your rarest. The mean is nearly useless (dominated by quiet-market samples); p50 tells you about the common path; p99.9 and max tell you what happens when it matters. I also look at conditional percentiles — latency during the top decile of message rate — which is the number that actually matters.
8. What is IPC and how do you interpret it on a hot path? Instructions per cycle from perf counters — a proxy for how much the core stalls. Modern cores can retire 4–6 µops/cycle (µops are the micro-steps instructions decode into; that 4–6 is the hardware ceiling, while typical real code achieves 0.5–4 IPC — the number from the ch00d table); hot-path code at IPC 0.5 is stall-bound (usually cache misses, branch mispredicts, or dependency chains), while IPC 3+ means you’re compute-dense. But interpret with care: high IPC isn’t the goal — low time is; you can inflate IPC with useless instructions, and a spin-wait loop shows gorgeous IPC while doing nothing. I use IPC as a triage signal: low IPC → go look at cache-miss and branch-miss counters; high IPC but slow → you’re simply executing too many instructions, go look at the algorithm.
9. How do you diagnose false sharing, and what does the fix look like?
Symptom: a multithreaded path slows down when threads increase, with perf showing heavy cache-line contention on lines no one logically shares — two threads writing distinct variables that happen to sit in the same 64-byte line, ping-ponging it between cores in Modified state. Diagnosis: perf c2c is purpose-built (shows contended lines and the offsets within them); or bisect by padding suspects and re-measuring. Fix: pad/align to 64 bytes (#[repr(align(64))] wrappers in Rust, e.g. around per-thread counters or the head/tail indices of an SPSC queue — the classic case), or restructure so each thread writes its own line and aggregation reads occasionally. The interview flourish: consecutive AtomicU64 counters in an array, one per thread, is the textbook self-inflicted version.
10. Your p50 improved but p99.9 got worse after an optimization. Hypotheses? The optimization moved work from “always” to “sometimes”: (1) added a cache — hits are faster (p50), misses now include fill cost or invalidation stalls (tail); (2) batching/deferral — common case cheaper, deferred work now lands in bursts (tail); (3) memory layout changed and a rare path now misses where it didn’t (or allocation moved: fast path pool-hits, slow path now takes the allocator lock); (4) code got bigger through inlining — pressure on the i-cache, the cache that holds the code itself, hurts the rarely-executed paths most, because big code evicts them first; (5) lock-free retry loops — cheap uncontended, unbounded under contention; (6) measurement itself changed (fixed a coordinated-omission bug and the tail was always there). My actual procedure: capture the tail samples’ context (what stage, what core, what concurrent load) rather than guessing — the histogram tells you that, only tracing tells you why.
11. How do you benchmark a lock-free queue properly? Wrong way: tight loop of enqueue/dequeue on one thread — measures nothing but cache-hot happy path. Right way: real thread topology (producer and consumer on the pinned cores production will use, because cross-core costs are what you’re there to measure), realistic arrival process (Poisson or recorded arrivals at target rate — not back-to-back, which is coordinated omission again), measure per-item latency from intended-enqueue-time to dequeue-completion into HdrHistogram, and sweep occupancy: an SPSC ring behaves completely differently near-empty (producer and consumer share the same cache lines) vs half-full vs near-full (backpressure path). Report percentiles at each rate, include the saturation knee (the arrival rate where latency turns sharply upward), and run long enough to catch periodic interference. Also benchmark the failure path — what full-queue does to the producer — because that’s the path that fires during the burst you built the queue for.
12. What observability can you afford in the hot path, and what’s the budget? Rule: the hot path may record, never emit — no syscalls, no locks, no allocation, no formatting. Affordable: rdtsc stamps into a preallocated per-thread ring buffer (tens of ns), counters in thread-local cache-line-padded slots, latency histogram increments (HdrHistogram-style array bump, ~ns). A separate core drains rings, aggregates, formats, ships. Budget it explicitly: if the path is 1µs, I’ll spend ≤5% — say 50ns — on instrumentation, which buys ~2–4 timestamps and a few counter bumps; choose stage boundaries accordingly. And never make it toggleable in a way that changes layout/branching between “observed” and “unobserved” builds, or your measurements describe a different program — always-on cheap beats sometimes-on detailed.
13. ‘How do you KNOW your 10ms number is real?’ Chain of custody for the measurement: where exactly are the two timestamps taken (NIC hardware vs app), are the clocks the same clock (same host? PTP/NTP quality if not — cross-host µs claims need PTP), does the harness coordinate-omit (send on intended schedule?), is the distribution reported or just a mean, what’s the sample count at the percentile quoted (a p99.9 needs ≥ tens of thousands of samples to mean anything), and was it measured under production-shaped load including bursts. My prod version: wire-adjacent timestamps where possible, ingress-stamped otherwise, HdrHistogram, conditional-on-load percentiles, and an external check — venue ack timestamps or a capture box — to catch my own instrument lying. If someone’s number lacks that chain, I don’t argue with it; I ask where the timestamps live and the number usually corrects itself.
14. Walk me through a p99.9 regression investigation, start to finish. First, verify it’s real: same measurement path, same load shape, sample counts adequate, and bracket it in time — “started with Tuesday’s deploy” is half the diagnosis (correlate with deploys/config/data-shape changes; my event-sourced setup lets me replay the same day through both builds, which isolates code from market regime in one step — that replay-diff is genuinely my favorite tool). If replay reproduces it: profile the two builds on identical input, diff flamegraphs, done. If replay is clean, it’s environmental: check the tail samples’ metadata for clustering (one core? one venue? periodic — smells like a timer/housekeeping; correlated with GC/allocator/page faults?), check host changes (kernel, IRQ affinity, CPU governor, neighbor VMs in cloud). Fix, then prove it with the same histogram under the same load, and add the regression as a permanent gate in CI replay.
15. What do cache misses actually cost, and how does that shape hot-path design? Rough modern numbers: L1 ~4–5 cycles (~1ns), L2 ~12–14, L3 ~40–60, DRAM ~60–100ns+ — so one DRAM miss costs as much as ~100 well-fed instructions, and a pointer-chasing structure that misses 5 times per event has spent half a microsecond doing nothing. Design consequences: arrays over linked structures, hot fields packed into the first cache line of a struct, index-based arenas instead of pointer soup, per-core data to avoid coherence traffic, and prefetch-friendly (predictable-stride) access. The book-update path is the canonical example: a flat sorted vector or array-indexed price ladder beats a node-based tree not because of big-O but because of misses — and perf’s cache-miss counters, not intuition, arbitrate.
16. Branch mispredicts: when do they matter and what do you do?
~15–20 cycles per miss, which matters when it’s per-message on a multi-million-message path. The usual sinners: data-dependent branches with near-50/50 outcomes (buy/sell, message-type dispatch over unpredictable sequences), virtual/indirect calls with mixed targets, and error-checking chains. Remedies, in order: make branches predictable (sort/partition work so the same path repeats), replace with branchless selects/arithmetic where cheap (cmov — but measure; branchless can be slower when the branch predicts well), turn indirect dispatch into direct code per stream (per-venue monomorphized parsers — which I do in spirit by giving each venue its own decode path), and keep the rare path out of line (#[cold], unlikely hints) so it doesn’t pollute the i-cache. perf’s branch-miss counters per stage tell you whether this is your problem before you contort the code.
17. What does the allocator do to your latency, and what’s the discipline?
malloc/free are locks, syscalls (sometimes), page faults (sometimes), and cache misses (always eventually) — any of which is a tail event; plus allocation-heavy steady state fragments and drifts over hours, which is deadly for an always-on engine. Discipline: no allocation on the hot path after warmup — preallocated pools/arenas for orders and messages, fixed-capacity rings for queues, Vec::with_capacity and reuse, and object recycling with generation counters (a reuse counter that catches stale references to a recycled slot). Enforce it, don’t intend it: a counting global allocator in debug/CI that panics on hot-path alloc (Rust makes this a 20-line GlobalAlloc wrapper), plus page-fault counters in prod. And the crypto-specific sin I’ve actually fixed: JSON parsing allocating per message — an arena, or simd-json-style tape parsing (parse into one flat pre-allocated array instead of allocating objects), turns the dominant cost into a bounded one.
18. NUMA: when does it bite a trading box?
Two-socket boxes have local and remote memory: remote misses cost ~1.5–2x local, and worse, the NIC DMAs into one node — if your consumer thread runs on the other socket, every packet starts with remote-memory reads. Bites: after a naive restart when the scheduler places threads differently than last time (mysterious “it’s slower since the reboot”), when the IRQ/queue core and the processing core straddle sockets, and when a memory pool was faulted in from the wrong node (first-touch policy — allocate and touch from the thread that will use it). Discipline: single-socket for the hot path if at all possible; otherwise pin NIC, IRQs, memory, and threads to one node (numactl, libnuma) and verify with numastat/perf. Cloud instances mostly hide this from me, though — it’s a colo-class concern I know the theory and tooling for, not one I’ve fought in prod.
19. Huge pages: why and what’s the catch?
4KB pages mean a large hot heap thrashes the TLB — every TLB miss is a page-walk (multiple memory accesses); 2MB/1GB pages cut TLB entries needed by ~512x, removing a whole class of tail events for big books and history buffers. Use explicit hugepages (hugetlbfs/mmap flags) for the known-large arenas. The catch: transparent huge pages (THP) can be worse than nothing — khugepaged compacts and splits pages in the background, causing exactly the multi-ms stalls you were avoiding, so the standard prescription is THP off (or madvise-only) and explicit allocation where you want it. It’s cheap to check (/proc/meminfo, TLB-miss counters) and one of the highest signal-to-effort host tunings.
20. How do you keep a rarely-taken path fast — the ‘cold branch that matters’ problem?
The risk-limit breach, the kill-switch check, the error path: taken once a month, must be fast that once, and meanwhile must not slow the common path. Techniques: keep the check itself trivially cheap and always-exercised (a compare against a cached limit — cold data is fixable: keep the limit on the hot cache line even when the branch is never taken), move the handling out of line (#[cold]/outlined function) so i-cache stays clean, and — the underrated one — exercise the path artificially: synthetic events in production quiet hours or in replay that take the branch, so its code and data aren’t ice-cold (and its correctness isn’t unverified) when the real trigger fires. This mirrors kill-switch drills: cold paths rot, and rot is both a latency and a correctness bug.
21. Interpreting a flamegraph of a hot loop: what are the traps?
Sampling profilers lie about hot loops in specific ways: cheap-but-frequent leaf functions get inflated or vanish depending on inlining (profile with debug info and check inlined frames); skid — samples attribute to instructions near, not at, the cost (use precise events, e.g. PEBS/:pp); off-CPU time is invisible entirely (a lock wait or page fault won’t show — pair with off-CPU analysis or scheduler tracing); and a flat 2% across many frames can be one cause (cache misses everywhere) that call-stack aggregation hides — event-based profiles (cache-miss-triggered sampling) regroup it. My rule: flamegraph for where, counters for why, and for tail hunting neither is enough — you need per-event tracing of the slow instances specifically, because the p99.9 samples are 0.1% of a sampled profile by construction.
22. Measure the latency of the measurement: what’s your instrumentation’s own cost and jitter? Before trusting any stage decomposition, time an empty region with the same instrumentation: back-to-back rdtsc pairs give you the floor (~20–40 cycles) and, more importantly, its distribution — occasional 1000-cycle outliers in an empty region mean SMIs, hypervisor exits, or preemption polluting all your measurements, and no amount of averaging removes a contaminant you haven’t characterized. Same discipline as instrument calibration in a lab. I also keep instrumentation always-on (Q12) precisely so its cost is a constant included in every number rather than a Heisenberg term that appears only while debugging.
23. Brutal: your engine is ‘fast’ in benchmarks but the fill quality says you’re slow to the market. Reconcile. The benchmark measures my code; fill quality measures my code plus everything I didn’t benchmark: queueing before my ingress timestamp (NIC rings, socket buffers — measure wire-to-ingress explicitly), the path after TX (my order gateway, TLS, venue-side queueing), clock error making me think I react faster than I do, and — the subtle one — selection: I only get fills when I’m late (adverse selection — when I’m fast the order I raced for is gone or I’m queued behind no one and the fill is bad news). So: instrument wire-to-wire with capture-grade timestamps to close the internal gaps, compare my order’s venue arrival time against the triggering tick’s venue publish time (venue timestamps bracket the full path including geography), and analyze fill quality conditioned on my measured latency — if fills are bad even when measurably fast, the problem is strategy, not speed. This question is really “do you know benchmarks aren’t the market.”
24. Brutal: design a latency regression gate for CI that doesn’t cry wolf. Naive gates (“p99 must be < X”) flake because CI hosts are noisy and tails need huge samples. Design: run on a dedicated, pinned, isolated benchmark host (never shared CI runners) with fixed frequency governor; feed recorded production input via the replay harness at production arrival times (determinism makes input identical across runs); compare candidate vs baseline on the same host, same run session, interleaved A/B/A/B to cancel drift; gate on distribution comparison (difference at p50/p99 beyond noise bands); require N consecutive failures before red. Two details make the gate honest. The noise bands come from A/A runs — you must first measure the harness’s own run-to-run variance before judging a candidate against it. And a Mann-Whitney-style check beats a point threshold: a rank-based test asking “are these two latency distributions actually different,” robust to outliers. And keep two gates: a tight statistical one that warns, a loose absolute one (“p99 doubled”) that blocks — warn-noise is tolerable, block-noise kills the culture of trusting the gate.
25. Brutal: ‘convince me your whole performance culture isn’t just benchmark theater.’ Three receipts. One: production numbers, not benchmark numbers, are the system of record — always-on histograms per stage per venue, conditional on load, with an external cross-check (venue timestamps / capture) so the instrument can’t grade itself. Two: the loop closes — every latency incident becomes a replayable regression test (the event log makes the exact day reproducible), and CI gates on replayed production input, so improvements are proven on the traffic that mattered, not on synthetic loops. Three: honesty artifacts — we track the known lies (coordinated omission audits of every harness, instrumentation floor characterized, sample counts printed next to every percentile) and we publish internally the numbers that got worse, because a perf culture that only produces improvements is theater by definition. Then the personal version: my p99.9 claims come with “measured at these two points, this clock chain, this sample count” attached, and I’d rather report an ugly true number than a pretty partial one — which, in this seat, is the entire job.
Further reading
- Gil Tene, “How NOT to Measure Latency” (talk, widely available) and the HdrHistogram documentation — coordinated omission from the source; watch the talk twice.
- Brendan Gregg, Systems Performance (2nd ed.) — ch. 6 (CPUs), ch. 13 (perf), off-CPU analysis; plus his flamegraph and
perf c2cwriteups online. - Ulrich Drepper, “What Every Programmer Should Know About Memory” — the cache/NUMA/TLB numbers behind Q15/18/19, still the canonical text.
- Intel Software Developer’s Manual / Agner Fog’s optimization manuals — rdtsc semantics, branch and cache costs with real numbers (Q3/4/16).
perfdocumentation and the kernel’sperf c2carticle — false-sharing diagnosis as a tool workflow, not folklore (Q9).- Kleppmann, DDIA ch. 1 — percentiles and tail latency framing (the SLO-flavored basics, useful vocabulary for Q7/24).
Question Bank: State & Deployment
25 questions, easy to brutal, covering chapters 13–17. This is the bank where your matching-engine and hot-standby experience does the most work — nearly every answer can be grounded in something you’ve built or run (engine, standby, pgbouncer/Postgres personal platform, 20-venue integrations). Speak in first person; cite the concrete thing; name the gaps rather than paper over them.
1. What does it mean for your engine to be deterministic, and why do you care? Same event log in, same state out — the engine is a pure fold over a totally ordered input stream, with time, randomness, and config changes arriving as events rather than ambient inputs. I care because everything expensive falls out of it: the hot standby is just a second consumer of the log, recovery is snapshot-plus-tail replay, testing is replaying production days through candidate builds, and incident forensics is exact re-execution. Lose determinism and all four break at once, usually silently.
2. Name the classic determinism violations and how you catch them. Wall-clock reads in the fold, RNG, HashMap iteration order (Rust randomizes it per process — the violation that passes every test and diverges in prod weeks later), floats with build- or hardware-dependent behavior (use integer ticks/lots), branching on I/O readiness or batch boundaries, and runtime config reads. Catching: dependency discipline on the core crate (no std::time or rand in its graph), dual-process replay in CI comparing fixed-seed state hashes — separate processes specifically to expose iteration-order bugs — and in production, continuous rolling hash comparison between primary and standby, which turns the replication pair I already run into a live determinism monitor.
3. How does snapshotting work without stalling the writer? The hot path never snapshots. A secondary replayer — in my case the standby, which already maintains identical state from the log — writes snapshots on its own schedule, tagged with the last applied sequence number; its pauses cost nothing. Alternatives I’d name: COW fork (Redis RDB style — works, but the fork stall and page-fault jitter land on the parent) and persistent data structures (pay per-operation forever to make snapshots cheap — usually the wrong trade for an engine). Durability details unprompted: write tmp, fsync, atomic rename, fsync the directory, checksum the blob, keep K generations so a corrupt latest snapshot isn’t fatal.
4. You replay an old log through a new binary. When is that valid? Valid when the new code is semantically identical on every event type in the log — refactors, perf work, additive features — and I prove it: CI replays recorded prod logs and requires state-hash equality. Invalid when the change intentionally alters decisions: replay then produces a history that never happened, while real fills went out under the old logic. Escapes, in preference order: log decisions (fills) not just inputs, so replay applies recorded outcomes verbatim; logic-version epoch events in the log so old segments replay with old semantics (the bitemporal move of ch13); or a snapshot fence — new code only replays from the cutover snapshot, with archived old binaries owning older segments.
5. How do you version events in a log retained for years?
Never mutate a published version — v2 is a new type beside v1; every record carries (type, version) in its header; upcasters (pure v1→v2→v3 chains applied at read time) translate so the engine only ever sees the current model, with defaults that reproduce old behavior exactly; golden-file tests freeze the old decoders; the log is never rewritten. I’ve implemented exactly this shape — in my lab version the entire v2 upcast is one if ver >= 2 line in the codec, and that locality is deliberate: version sprawl lives in one file, not in the engine.
6. What’s the N/N+1 compatibility guarantee and why does rolling deploy need it? During any deploy, versions N and N+1 coexist on the same streams, so every schema change must be readable in both directions across one version step: new readers accept old messages via defaults, old readers accept new messages via unknown-field skip or extension semantics. Choreography: ship readers first, flip writers later — by config, after bake — and the classic outage is doing it in the other order. Note the asymmetry with the log: live traffic needs N/N+1, but replay needs N back to the oldest retained segment, which is what the upcaster chain is for.
7. Compare protobuf and SBE evolution rules in two breaths.
Protobuf: tag-length-value, unknown fields skipped (and retained on re-serialize in proto3), evolution by adding fields, cardinal sin is reusing a field id — reserved exists for that; cost is varint and dynamic decode, fine for control plane. SBE: fixed offsets, decode is a pointer cast, evolution only by appending fields or claiming pre-reserved padding with zero-as-legacy-default, schema version in the header so old readers read their known prefix; cost is rigidity, and inserting a field mid-message silently corrupts every old record. Choose by tier: SBE where latency is the product, protobuf where cross-team evolution matters more than nanoseconds.
8. A venue announces a breaking protocol change. Walk me through your process. I’ve lived this repeatedly across 20+ integrations. The venue’s protocol exists only inside its feed handler and gateway — the normalize layer is the isolation boundary — so: capture raw traffic on the new API while the old still runs; build the new handler as a parallel module, never in-place edits; shadow it, diffing normalized output against the current handler live, which catches the undocumented changes (units, side conventions, snapshot depth); cut over that one venue with instant fallback; retain the old decoder as long as old captures exist. Downstream systems see zero change unless I choose an additive internal-schema update, which then follows the readers-first rollout.
9. Why doesn’t the hot path touch a database, and where does data actually live? Anything with a query planner, lock manager, or network hop is orders of magnitude off a microsecond budget — and that includes Redis, not just Postgres. State lives in process memory; durability is the append-only event log with a replicated standby; every database is a derived projection consuming the log: a columnar tick store for time-series analytics, Postgres for reference data, accounts, and compliance, Redis for ephemeral-by-policy coordination. The log is the system of record; databases are views. I run this split personally — Postgres 17 behind pgbouncer plus a Redis tier on my own platform, with Redis explicitly allowed to lose data.
10. Walk me through expand-migrate-contract for changing a column type on a live 500M-row table.
In-place ALTER TYPE rewrites the table under ACCESS EXCLUSIVE — hours of downtime — so instead: expand — ADD COLUMN qty_v2 bigint (metadata-only, instant, with lock_timeout set); dual-write via app code or a sync trigger; migrate — backfill in primary-key-range batches, small idempotent transactions (IS DISTINCT FROM guard), throttled and resumable, watching replication lag and bloat; verify counts; enforce with ADD CONSTRAINT ... CHECK (qty_v2 IS NOT NULL) NOT VALID then VALIDATE CONSTRAINT (only SHARE UPDATE EXCLUSIVE) then SET NOT NULL (instant on PG12+ because the CHECK proves it); switch reads; contract — days later, a separate deploy drops the trigger and old column. The invariant that makes it safe: every intermediate schema works with app versions N and N+1, and each step rolls back independently.
11. Which ALTER TABLE operations are traps, and what’s the meta-trap?
Traps: ALTER COLUMN TYPE (full rewrite), ADD COLUMN DEFAULT volatile-fn (rewrite — constant defaults are instant since PG11), SET NOT NULL pre-PG12 without the CHECK trick (full scan under exclusive lock), plain CREATE INDEX (blocks writes), VACUUM FULL (exclusive for its whole run — use pg_repack instead). The meta-trap is lock queuing: even an instant ALTER needs ACCESS EXCLUSIVE, waits behind one long-running query, and every subsequent statement — including plain SELECTs — queues behind it, so production freezes while your migration does literally nothing. Defense: SET lock_timeout = '2s' plus retry built into the migration runner, and a lint layer (squawk / strong_migrations style) in code review.
12. CREATE INDEX CONCURRENTLY failed halfway. What state are you in?
An INVALID index: it’s left behind, maintained on every write (pure overhead) but unusable for reads — you must DROP INDEX CONCURRENTLY and retry. Other failure modes to volunteer: it waits out every older transaction, so one idle-in-transaction session (a wedged pooler client, a forgotten psql) can stall it indefinitely; it can’t run inside a transaction block, so your migration tool needs a non-transactional mode; and unique builds can fail late on duplicates. Knowing “INVALID index” cold is the tell that you’ve actually run this, not read about it.
13. Streaming vs logical replication — pick for three scenarios: HA, major version upgrade, feeding ClickHouse. HA: streaming — physical WAL bytes, exact replica, sync or async per your durability need, with Patroni automating failover. Major upgrade: logical — decoded row events are version-independent, so you replicate into the new-version cluster and cut over near-zero-downtime. Feeding ClickHouse: logical decoding as CDC, Debezium-style — the database becoming an event producer, which is the mirror image of my engine’s log-projection pattern. Teeth to mention: logical replication doesn’t carry DDL, sequences don’t replicate, and an abandoned replication slot pins WAL until the primary’s disk fills — the classic 3am incident.
14. What does Patroni actually solve, and what are the concepts underneath?
It automates leader election and failover: an agent beside each Postgres, a leader lease in etcd or Consul (a key with a TTL that only the leader keeps renewing — lose the renewal, lose the crown), promotion of the least-lagged replica on lease loss, client rerouting via proxy or health endpoints. Underneath are exactly my engine’s hot-standby problems wearing DB clothes: fencing (the demoted primary must not accept writes, or you get split-brain and forked timelines), bounded data loss (the async lag window, maximum_lag_on_failover, or synchronous replication to zero it), and divergence repair (pg_rewind to rejoin the old primary as a replica). I make that mapping explicitly, because it’s true. I run single-node Postgres with my own failover being “restore from backup” — Patroni I know as architecture, not as scar tissue, and I’d say so.
15. Why is pgbouncer necessary and what breaks under transaction pooling?
Postgres backends are processes with real per-connection memory; connection storms from fleets of app instances collapse a server that’s perfectly happy at 50 active backends, so pgbouncer multiplexes thousands of client connections onto tens of server connections. Transaction pooling — the production default — breaks session state: named prepared statements (protocol-level support only in recent pgbouncer), persistent SETs, session advisory locks, LISTEN/NOTIFY. I run one in production on my own platform: host-native pgbouncer on the public port routing by database name to a loopback Postgres 17, roughly 25 backend connections per tenant DB under a 500-client ceiling — and I can also describe the isolation trade-off I consciously accepted in that stack — a cache tier that bypasses per-tenant auth (ch15).
16. How do you deploy a new matching engine version during market hours? Hot-standby cutover, run as a runbook with mechanical gates, in six phases:
- Pre-verify — replay-regression with classified decision-diffs, plus an N-1 read-back check.
- Follower start — the new binary comes up as a follower: snapshot load, tail replay, live consumption with outputs sinked.
- Compare — rolling state-hash comparison against the incumbent while both run.
- Cut — at a logged sequence boundary (a
LeadershipTransferevent), fencing the old primary by epoch number on every log append and outbound order. - Reconcile — open orders checked against every venue before restoring full size.
- Bake — the old binary stays hot and fenced as the instant rollback target through the bake period, and the new binary keeps writing old-format events until past the rollback horizon.
Every gate is a pre-committed number, not a judgment call at T-0.
17. How do venue sessions survive an engine restart? Mostly they don’t — that’s the hard part. FIX: sequence numbers are session state, so they belong in the log/snapshot like all state; re-logon negotiates gap-fill from the persisted seqnums (a naive reset triggers replay-or-reject storms — the peer expects seq N, sees seq 1, and either demands a resend of everything or rejects the session outright); resting orders survive at the venue, but you’re blind and can’t cancel during the gap. Crypto, my daily world: no seqnum contract — instead you get re-auth bursts into rate limits, resubscription storms, and per-venue book resyncs, mitigated by pre-warming connections where venues tolerate duplicate sessions, which is a per-venue quirk I literally keep a table of. The architectural answer for both worlds: a thin, rarely-deployed gateway tier owns the venue sessions so the frequently-deployed engine restarts behind an unbroken connection — the same isolation move as the feed-handler normalize layer.
18. Design a shadow deployment and tell me its blind spots. The candidate consumes the live feed and order flow, runs full logic, and its outputs go to a recording sink; a comparator stream-diffs its decisions against production’s, with expected-vs-unexpected classification — an intended change diffs everywhere, and without classification the real regressions drown. Shadow tests today’s regime, which recorded replays can’t. Blind spots I volunteer before being asked: market impact (its fills are simulated, and queue-position modeling is where fill simulators lie), venue interaction (rejects, rate limits, partial-fill sequencing), and anything triggered by its own orders’ effects on the market. That’s why the ladder is replay → shadow → capped canary: each rung covers the previous rung’s blindness, and only the canary — real money under hard risk-layer caps — sees impact.
19. Design the kill-switch system for a multi-venue trading firm. Taxonomy by blast radius: per-strategy, per-venue (the tier I’d use most across 20 venues), per-symbol and per-account, global cancel-everything, plus a tiered flat-position action — passive-flatten with a deadline, then aggressive, because naive market-order flattening into a dislocated book realizes the worst possible price. Engineering: enforced at the minimal-dependency edge (gateway/risk layer) so it works even when the engine is wedged; state persisted so a restarting engine comes up stopped if the switch was pulled; pull-cheap/reset-expensive authority — anyone on the desk pulls, seniority plus a checklist un-pulls; every pull logged with who/when/why; and scheduled production drills with measured time-to-stopped, where a failed drill is a P1. Close with: regulators mandate kill functionality anyway (RTS 6 flavor), so build it once, properly, and let the drill records double as compliance evidence.
20. Runtime feature flags in the hot path — argue both sides, then pick. For: instant enable/disable without a deploy, gradual rollout, operational flexibility — real virtues in web systems. Against, and decisive on a deterministic path: every flag is a branch and possibly a shared-state load; 2^N flag combinations of which you tested three; and a runtime flip changes behavior without passing replay-regression — a bypass around the entire verification pipeline, which is how Knight Capital died (a repurposed flag plus a partial deploy). My position: config-at-startup — behavior toggles read once into immutable config, so every change is a deploy through the gates, with hot-path variants monomorphized at init; the only runtime-mutable controls are the enumerated risk plane (kill switches, limits, throttles), which is deliberately not a feature system. And if a toggle must affect the fold mid-session, its changes enter as logged events so replay stays truthful.
21. Your standby diverged from primary — how do you find out, and what do you do? Find out by construction, not by luck: both sides publish rolling state hashes keyed by sequence number every N events; a comparator alerts on first mismatch, so detection latency is bounded and it pages me long before a failover would need that standby. Immediate action: the divergent standby is disqualified as a failover target — a standby with wrong state is worse than none — while the primary keeps trading and I spin a replacement standby from snapshot-plus-replay. Diagnosis: offline, replay the log from the last matching snapshot on both binaries and bisect to the first diverging sequence number, then inspect how that event was applied — the culprit is almost always a determinism violation (unordered iteration, config skew between hosts, a float path) or, rarer, torn log shipping, which per-record checksums distinguish. Then the fix becomes a permanent CI determinism test, so the class of bug dies, not just the instance.
22. How do you reconcile engine state against the venues and against your own DB?
Two loops. Venue recon — the one that costs money: diff open orders, fills, and balances against drop-copy or execution feeds plus REST snapshots, continuously at low rate and mandatorily on any restart before trading resumes; in crypto the REST view is rate-limited and eventually consistent with the venue’s own WS stream, so the logic needs tolerance windows, not equality asserts — I’ve been burned by treating a venue’s REST snapshot as instantaneous truth. DB recon: compare aggregates — positions, open quantity, cash — between an engine snapshot at sequence N and the projection’s view as of N; sequence numbers make “the same instant” well-defined, and without them recon chases its own tail. Doctrine: recon detects, runbooks decide, and every correction is applied as a new logged event (ManualAdjustment { reason, ticket }) — never by mutating state or rows in place, or you’ve corrupted replay and created drift the next recon can’t explain.
23. Rollback vs forward-fix at 3am — give me the decision tree and its prerequisites. Kill switch first — flatten or cap the risk so the decision isn’t made while bleeding — then default to rollback: the old binary is a known-good artifact, the new one is a hypothesis you just falsified. Forward-fix only if rollback is state-unsafe (new-format events already written past what N-1 can read — which the write-flip-after-bake discipline exists to prevent), the bug predates the deploy so rollback changes nothing, or the fix genuinely validates faster than the rollback — rarer than it feels at 3am. Prerequisites that make the tree real: the criteria were written in the runbook before the deploy; N-1 compatibility with everything the new binary wrote (events, snapshots, config) is tested in CI by replaying new-written logs through the previous release; and the old binary is still resident, fenced, and at the head of the log. An unrehearsed rollback isn’t a rollback — it’s a second incident.
24. Brutal: a bad fill report from a venue corrupted your position state six hours ago and you’ve been trading on it since. Unwind the situation. Stop the bleeding at the right scope first: kill switch for the affected strategies or venue, establish true exposure by reconciling against every venue’s authoritative view — their fills, not my state — and flatten to safe bounds if the divergence is material. Then the event-sourced unwind: the corrupting input is in the log with a sequence number, so I can replay to the moment of corruption, quantify exactly how state diverged, and correct forward by applying adjustment events (or a corrected interpretation of the venue’s amended report) — while the actual log stays immutable as the audit record of what the system genuinely believed and did, which both compliance and the post-mortem need. The six hours of decisions were made on false state, but those orders are real and stand; corrections are forward-looking events, never retroactive mutation. Close the loop: the bad report becomes a permanent replay-regression fixture, and the recon cadence that let it live for six hours gets shortened — the six hours was the real failure, not the bad message.
25. Brutal: “You’ve never worked in tradfi. Why should we trust you with our deployment and state architecture?” Because the hard version of this problem is the one I already operate: crypto has no maintenance windows — every deploy I’ve ever done was during market hours; venues restart their matching engines under me routinely, which amounts to involuntary failover drills; and 20+ live integrations mean protocol evolution, session chaos, and reconciliation are my daily reality, not a quarterly event. The architecture I run is the one your engineers respect: a deterministic event-sourced core, sequenced log as the record, hot standby by log consumption, replay-based verification — and I can defend each choice down to why HashMap iteration order breaks replay and which upcast defaults preserve history. What I’d be new to is specific and bounded: FIX session mechanics at your particular venues, your regulatory evidence chain, colo-grade tooling — named gaps with learning plans, not conceptual gaps. And the meta-point I’d actually say: someone who can tell you precisely what they haven’t run is safer around your production state than someone who can’t.
Further reading
- Chapters 13–17 of this book — this bank is their compression; when an answer feels thin under probing, the chapter has the depth.
- Kleppmann, DDIA ch. 4, 5, 9, 11 — encoding evolution, replication and failover, total order, log-centric state: the theory under Q1–8, 13–16, and 21.
- Postgres documentation — ALTER TABLE lock notes, “Building Indexes Concurrently,” and the logical-replication restrictions page — plus the Patroni docs: the exact references behind Q10–15.
- Greg Young, Versioning in an Event Sourced System — Q4 and Q5 at book length.
- The SEC’s Knight Capital order (2013) — the cautionary spine of Q20 and Q23; read the primary source once, it’s short.
- Aeron Cluster documentation — the productized form of Q16’s cutover choreography, with snapshots and leadership transfer built in.
Question Bank: Venue & Broker Design
25 questions, ordered easy to brutal, covering Part V (ch23–ch27). Model answers in your voice: first-person and concrete where your CLOB, market-data pipeline, and SOR at Crypto.com give you real standing; explicitly framed (“the productization layer I’d add is…”) where multi-client broker or venue-operator experience is the new territory. Being candid about which is which is what wins these rounds — you built a matching engine, a 20-venue data pipeline, and a router for one desk, and that’s more than most candidates walk in with. Practice aloud; 3–8 sentences each.
1. What does the sequencer do, and why does everything need a total order? The sequencer receives every input — orders, cancels, timer ticks, reference-data changes — assigns each a monotonically increasing sequence number, and publishes the sequenced stream; everything downstream (matching engine, risk, drop copy, standby) is a deterministic function of that stream. Total order is what makes determinism possible: if two consumers could see events in different orders, they’d compute different books, and replication, recovery, replay-testing, and audit all break simultaneously. It also is the fairness ruling — “who was first” has exactly one answer, the sequence number. I built this shape at Crypto.com: single-writer, event-sourced matching engine with a hot standby consuming the same log; the standby works precisely because the log is totally ordered.
2. Untangle OMS, EMS, and SOR. OMS owns the what: the parent order as a durable business object plus client accounts, buying power, and allocations — the system of record, like Stripe’s PaymentIntent. EMS owns the how: working the parent over time — algo choice, slicing, urgency — the retry/orchestration layer. SOR owns the where: for one child right now, which venue, given fees, books, latency, and fill probability — acquirer selection. In practice EMS and SOR blur into one engine; what I built was an SOR with EMS behaviors and a thin single-tenant OMS, and the multi-client OMS is the layer I’d build fresh — event-sourced, because allocations and best-execution evidence both demand deterministic replay.
3. What is drop copy and why not just use your execution reports? Drop copy is a real-time duplicate of your fills and order events on a separate session, feeding risk and clearing independently of the trading session — a CDC stream off the venue’s ledger. Independence of the watcher from the watched is what earns the separate session: if risk builds positions from the trading gateway’s own view of its acks, one bug corrupts trading and risk together; drop copy gives risk a venue-authoritative stream the trading code never touches. It’s also the practical feed for the real-time aggregate exposure monitoring 15c3-5 expects. In my recon design it’s the live leg of a triple loop: internal book vs drop copy continuously, vs venue statements at cutoffs, vs custodian on settlement.
4. What is cancel-on-disconnect and when is it wrong? CoD auto-cancels a session’s resting orders when its connection drops — the default safety rail, because a dead strategy leaving stale quotes in the market gets picked off. It’s wrong in two directions: triggering on a network blip while the strategy is healthy mass-cancels your queue position, which is expensive for a market maker; and broker-side, killing a client’s 6-hour TWAP because their monitoring session dropped confuses session with order ownership — the OMS owns the parent, not the client’s TCP connection. So it’s per-session configuration with tuned heartbeat timeouts, and the platform mirror is an explicit per-client disconnect policy: CoD for takers, keep-working for parked algos.
5. List the pre-trade risk checks and their latency budgets. Price collar against a reference band, max order size and notional, position-plus-open-orders limit, credit/buying power, per-session rate limit, self-match prevention, duplicate detection on client order ID. Venue-side each is a compare or an atomic read on in-memory, cache-line-padded counters — call it 10–100ns per check, a few hundred nanoseconds total, small against a 5–50µs gateway path, which is the arithmetic that shuts down “skip checks for latency.” Placement: venue runs them in the gateway before the sequencer so rejects never consume a sequence number; broker runs them in the OMS before the SOR so a breach never reaches a venue. Two of these I’ve built in payments under different names: the client-order-ID check is an idempotency key, and the credit check is an auth hold with reserve/release/convert semantics.
6. Self-match prevention: what are the options and who wants which? At the touch (the best bid and ask — where the next trade prints), if the aggressor and the rester resolve to the same firm or SMP group, don’t print — apply policy: cancel-newest (aggressor dies; market makers re-quoting want this because the rester keeps queue position), cancel-oldest (rester dies, aggressor trades on; sweepers — aggressive orders eating through multiple price levels — want their intent to survive), or cancel-both. It’s a per-session flag because both preferences are legitimate. Venues do it mechanically rather than adjudicating intent afterwards because self-matches print volume indistinguishable from wash trading. Cost is a tens-of-ns ownership check, since order structs already carry owner IDs.
7. Why is cross-session ordering decided only at the sequencer, and what does that mean for gateways? Because the gateways are parallel: two orders entering different gateways have no meaningful “first” until something imposes one, and any attempt to decide order at the gateway tier — timestamps from different NICs, queue positions on different boxes — manufactures a fake ordering from unsynchronized clocks. So the contract is: gateways validate, normalize, and forward as fast as they can; arrival at the sequencer is the ordering event; fairness engineering means making the gateway-to-sequencer path uniform (same hops, same budget per session) so no session buys an edge from topology. The venue’s job isn’t zero latency, it’s equal latency — fairness as variance control, not speed.
8. Why can multicast be fair when TCP fanout cannot? Multicast hands one packet to the switch and the switch replicates in hardware — every subscriber’s copy leaves at effectively the same instant, and no receiver’s slowness backpressures the sender or delays anyone else. TCP fanout is N serialized sends: someone is first and someone is last on every update, the ordering is an implementation accident that becomes a de-facto tiering, and a slow receiver’s closed window forces the sender to buffer or disconnect. That’s why tradfi feeds are multicast with loss handled by the receiver, and why crypto — stuck with TCP/WS across the internet — substitutes randomized send order, tiered products, and slow-consumer kills instead of true simultaneity. I’ve lived the receiving end of the crypto version across 20+ venues; building the sending side means choosing which unfairness you can defend.
9. How do you handle a slow consumer in a WebSocket fanout tier? Never let it backpressure the publisher — that’s the one inviolable rule, because a slow risk dashboard must not slow the feed. Detect via send-buffer depth or lag from head-of-stream; respond in escalation: conflate per-key so the consumer gets current state instead of every tick (correct for UIs and dashboards, a bug for anything sequence-dependent), then drop to snapshot-on-reconnect, then disconnect with a resume token. Make the contract explicit — the consumer knows whether its stream is conflated and can detect its own gaps via sequence numbers. I’ve been on the receiving end of venues that kill slow WS consumers mid-burst, precisely when resync is most expensive, so I’d also invest in making reconnect-and-splice cheap: snapshot plus buffered deltas applied by sequence.
10. Design a market-data feed: snapshot + incremental, and the consumer contract. Publish an incremental stream where every update carries the sequence number of the book state it produces, plus a periodic (or on-demand) snapshot stamped with the sequence it reflects. Consumer contract: subscribe to increments first, buffer; fetch snapshot; discard buffered increments with seq ≤ snapshot seq; apply the rest; a gap in increment seq means you’re broken — re-snapshot, and expose a validity state (LIVE/GAPPED/RESYNCING) to every downstream consumer. Publisher-side obligations: snapshots must be consistent cuts at an exact sequence (generated from a replayer of the sequenced log, not from a racing read of live state), and increment retention must cover the slowest tolerated consumer. I’ve implemented the consumer half against 20+ venues and can enumerate which venues get this wrong and how; building the publisher half is the same contract from the other side.
11. How do you shard a venue by symbol, and what about the hot symbol? Partition instruments across matching-engine shards — each shard its own sequencer+engine, totally ordered within itself, no ordering guarantee across shards; that’s valid because price-time priority is per-book. The catch: sharding gives you nothing for the hot symbol, because one book is inherently a single total order — BTC-perp at an open is one shard’s problem no matter how many shards exist. Hot-symbol answers are vertical and structural: make the single-writer path faster (the engine is small; the win is usually upstream in the gateway/risk tier, which does parallelize), move non-matching work off the shard, and accept that cross-shard products (margin across books, self-match across books) now need either an aggregation tier or asynchronous enforcement. If someone proposes splitting one book across shards, the interview answer is: you’ve just reinvented the fairness problem with extra steps.
12. How do you fail over a sequencer without losing acked orders? Define the invariant first: an ack means the event is durably sequenced, so the ack must only be sent after the event is replicated to the standby (or a quorum) — ack-after-replication, not ack-after-local-write. Then failover is: fence the old primary so it can’t append (epoch numbers on every log entry; consumers reject stale epochs), standby verifies it holds the log through sequence N, takes over at N+1, and re-establishes sessions; clients reconcile via exchange-provided order-status queries. The unacked in-flight window is the client’s problem by contract — which is why client order IDs and idempotent resubmit exist. Options ladder: manual standby with fencing (what I effectively ran — primary/standby with log shipping), or Raft via something like Aeron Cluster, where every input must reach a majority of nodes before it’s acked — so each order pays that quorum round-trip — in exchange for principled automatic failover. The trap answer to avoid: “ping timed out so standby promotes itself” — that’s split-brain, and fencing is exactly what prevents it.
13. One venue gives you 100 orders/sec. N clients share your platform. Design the budget. Per-client sub-budgets inside each venue budget — weighted fair queuing by tier or contract, unused capacity redistributed — because one client’s algo burst must not starve another’s flow. Two hard rules: cancels never queue behind new orders, since a starved cancel is a risk event, not a UX issue; and throttling is surfaced to the client as an explicit signal rather than silent queuing, because silently delaying an order changes its execution price and that’s a best-execution problem. It’s API-gateway per-tenant rate limiting where the upstream quota is hard and external. I ran the single-tenant version — per-venue budgets in my adapters — and the fair-share scheduler across clients is the productization layer.
14. Walk me through the double-fill race on re-route. Child on venue A isn’t filling; SOR re-routes: cancel to A, new child to B; A’s fill was already through its sequencer when the cancel arrived, so both fill and the client is over-bought. Prevention is cancel-ack discipline: don’t send B’s child until A confirms the cancel with final cumulative quantity — one venue RTT per re-route, and worth it. That’s the payments idempotency rule wearing trading clothes: never retry until the first attempt’s outcome is known, because “probably failed” is how double-charges happen. If an algo genuinely can’t wait, cap the new child assuming worst-case fill on A and run an error account for over-fills with explicit client-disclosure policy — optimism becomes opt-in and accounted for, never the default.
15. What does client segregation mean as engineering, not policy? One client’s flow is alpha, so the design must make leakage structurally hard, not contractually forbidden. Concretely: no shared mutable state whose observable behavior reveals another client’s activity — a shared queue where A’s burst delays B’s acks is a side channel, so per-client queues fair-scheduled into shared stages; per-client authorization on every read path including support dashboards, with access audited; and if the firm trades principal, the firm’s own desk is the hardest wall — separate services and credentials, because “the router operator sees everyone’s flow” is the FTX/Alameda lesson. It’s multi-tenant SaaS isolation with an adversarial twist: the leak isn’t embarrassment, it’s directly monetizable against the victim.
16. What evidence does best execution require, and what’s TCA? Best execution is the obligation to get the client the best available result across price, cost, speed, and likelihood — and the architectural forcing function is provability: every routing decision logged with the market snapshot it saw — per-venue books, fees, health scores, cost-model arithmetic — so “why venue B at 14:32:07” has a replayable answer. An event-sourced SOR gets this nearly free; I logged decisions at Crypto.com for debugging and venue scoring, and the upgrade is retention and tamper-evidence. TCA — transaction cost analysis — is the after-the-fact measurement: slippage vs arrival price, vs VWAP for scheduled algos, and per-venue scorecards (fill rates, effective spread, post-fill markout as a toxicity signal). It’s simultaneously the router’s feedback loop and the client-facing proof — the auth-rate dashboard a PSP shows merchants to justify its routing.
17. How does “the venue is a counterparty” change a crypto router? Pre-funding means balances sit on each exchange, so venue failure is a credit loss, not an outage — FTX is the case study for why best-price-only routing is broken. So the venue score carries slow-timescale risk inputs next to fast microstructure ones: withdrawal-latency trends (withdrawals quietly slowing is the canonical early warning), concentration caps on the share of assets per venue, jurisdiction and licensing posture. It also spawns a treasury subsystem — rebalancing inventory across venues against withdrawal fees and on-chain confirmation times, the multi-currency prefunding problem from payments. And the stablecoin legs are real instruments: BTC-USDT vs BTC-USD carry a basis, and routers that hardcoded a stablecoin at $1.00 learned about it in March 2023, when USDC broke to $0.87.
18. Sketch a liquidation engine. Why mark price, not last price? Margin and liquidations key off a mark price — an index over multiple external spot venues, smoothed — because keying off your own last trade lets an attacker print one small trade in a thin book and cascade-liquidate everyone; mark-price design is manipulation resistance, and it forces index rules: several constituents, published weights, outlier rejection, staleness eviction. The waterfall: margin call → partial liquidation (reduce, don’t nuke) → full liquidation via rate-limited orders into the book → insurance fund absorbs closes worse than bankruptcy price → ADL against profitable opposing positions as the published last resort. The engine itself is a trading system with the same disciplines: mark-price ticks as sequenced events, deterministic decisions, and its own kill switch, because a runaway liquidator is the worst self-inflicted incident a perps venue can have. I know this stack from the consumer side across venues; specifying it operator-side is the flip this book’s Part V is about.
19. How would you detect wash trading on your venue? As stream jobs on the sequenced log — the deterministic log is what makes surveillance tractable, because “what did the book look like when this order arrived” has an exact, replayable answer. Wash detection: entity-resolve accounts into beneficial owners (shared funding sources, withdrawal addresses — a graph problem in crypto), then flag self-crossing rates, volume with near-zero net position change, and tight round-trips. Spoofing/layering, its sibling: order-to-trade ratios, cancel-latency distributions — spoofers cancel fast when approached — and size posted opposite subsequent aggression. Flags go to human review with a book-replay evidence bundle, which is why a venue that can’t replay its book can’t really do surveillance. Architecturally it’s payments transaction monitoring — velocity rules on a ledger feeding a case queue — with different features, and SMP upstream removes the innocent cases first.
20. Give me 15c3-5 and RTS 6 in one line each, plus what each forces you to build. 15c3-5, the US market-access rule: a broker giving clients market access must run pre-trade financial and regulatory checks under the broker’s own control — no naked access — which forces the OMS risk gate before the SOR, non-disableable per client, plus real-time aggregate exposure monitoring, which drop copy feeds. RTS 6, MiFID II’s algo-controls standard: firms running algos must have kill functionality, pre-trade limits, real-time monitoring, and an annual self-assessment — which turns the kill-switch taxonomy into an audited artifact with named owners and test evidence. I’d volunteer the adjacent one: MiFID’s clock-sync rules (RTS 25) make the PTP chain regulatory evidence. My frame in the room: I’m not a lawyer, but I design assuming the event log will be read by a regulator — cheap if you’re event-sourced from day one, impossible to retrofit.
21. Why must mass-cancel be the fastest path in the system? Because it’s used exactly when the market is moving against someone and every millisecond of cancel latency is money — a mass-cancel that walks orders one-by-one through the normal pipeline arrives after the damage. So it’s a first-class sequenced operation: one message cancels by owner/symbol/scope, orders are indexed by owner so cancel-all is O(orders owned), and queue capacity is pre-reserved so the cancel can’t be backpressured by the very flood it’s stopping. Layered above it sits the kill-switch taxonomy — per-session, per-symbol/venue, global — pre-authorized, drilled (RTS 6 audits this), with an out-of-band path so pulling it doesn’t require the sick system to cooperate. It’s the payments “pause payouts” button: pre-wired, permissioned, logged, and the postmortem always asks why it took N minutes to press.
22. Two clients’ orders compete for the same liquidity on your platform. Who wins? Whoever the written policy says — the indefensible answer is not having one, because the disadvantaged client’s lawyer will ask. My default: strict time priority of parent arrival at the platform, simple and incentive-compatible (no client gains by gaming when or how they submit), mirroring what the venue itself does; where the platform aggregates parents into shared children, fills allocate back pro-rata by a documented formula. The engineering requirement is that allocation is deterministic and replayable — same fills in, same split out — computed in the event-sourced fold, and logged with the same evidence discipline as routing decisions. And it’s a product decision, not just an engineering one: I’d want compliance in that design review, because the allocation policy is a commitment the firm makes to every client simultaneously.
23. Design a crypto exchange from scratch. First five boxes on the whiteboard. One: gateways — sessions, authn, normalization, pre-trade risk, per-session rate limits, cancel-on-disconnect; parallel and stateless-ish. Two: the sequencer — single writer assigning the total order, ack-after-replication, the fairness and determinism anchor. Three: the matching engine — a deterministic fold over the sequenced log, price-time priority, in-memory book, sharded by symbol with the hot-symbol caveat. Four: the market-data publisher — incremental feed with sequence numbers plus consistent snapshots, fanout tier that slow consumers cannot backpressure. Five: the post-trade spine — drop copy, positions/ledger, recon, and (for perps) the margin/liquidation engine with mark-price index. Then I’d say out loud: boxes two and three I have actually built as one desk’s engine — deterministic single-writer, event-sourced, hot standby — and the venue version is the same skeleton with fairness, surveillance, and counterparty obligations bolted on where my one-desk version could ignore them.
24. Your venue’s p99.9 ack latency doubled at market open. Walk me through it. First, localize with the timestamps the pipeline already carries: gateway-in, sequencer-in, engine-out — the doubling lives in one segment, and p99.9-only (p50 flat) means queueing, not a uniformly slower path. Open-specific suspects in order: burst arrival overflowing a gateway queue (check depth high-watermarks and per-session arrival histograms — often one participant’s algo went aggressive overnight), a risk-check hitting a cold or contended path (a limit table that grew, a false-sharing regression on the counters), GC/allocator or page-cache effects from the overnight batch still settling, and the market-data fanout stealing cycles from a shared core if isolation regressed after a deploy. I’d also diff against yesterday’s open — same percentile, same minute — because “doubled” only means something against a baseline, which is why per-segment histograms exist (ch08). And I’d volunteer the uncomfortable part: at crypto’s latency scales the culprit was usually my own host, and I’d say so while checking queue depths first because they’re cheaper to read than a profile.
25. A client says your router gave them a bad fill. Prove it didn’t — or find out it did. Pull the decision record: the SOR is event-sourced, so that child’s routing decision exists with the full input snapshot — every venue’s book, fees, health scores, and the cost-model arithmetic at decision time — and I replay it, which either reproduces the choice or exposes a divergence. If the decision was right on its inputs, TCA frames the outcome: slippage vs arrival for that parent, the venue’s scorecard that day, and the counterfactual cost of the alternatives from the same snapshot — sometimes the answer is “the market moved during your order; here’s the tape.” If the inputs were wrong — stale feed, mis-scored venue — the same log shows exactly that, and now it’s an incident with a fix and possibly a client remedy, which is a better outcome than winning the argument. This is why the evidence discipline exists before the dispute does: you cannot retrofit the snapshot. It’s the chargeback-dispute flow from payments — the merchant who kept the AVS response and the signed receipt wins; the one who didn’t, pays.
Further reading
- Aeron and Aeron Cluster documentation (aeron.io) — the best public writeup of sequenced, replicated deterministic services; Q1, Q11, Q12 productized.
- Nasdaq TotalView-ITCH and OUCH protocol specifications (public on nasdaqtrader.com) — read a real venue’s market-data and order-entry contracts; MoldUDP64’s sequencing/recovery design underlies Q8–Q10.
- SEC Rule 15c3-5 adopting release and MiFID II RTS 6 (Regulation (EU) 2017/589) — the two regulatory texts worth skimming in the original; Q5, Q20, Q21.
- BitMEX/Deribit public docs on mark price, liquidation, insurance fund, and ADL — operator-written specs of the perps margin stack behind Q18.
Whiteboard Drills & Spoken Scripts
Reading is not interviewing. These six drills convert the book into spoken, timed performance — do each one against a wall clock, out loud, standing at a whiteboard or shared doc, and score yourself against the rubric after, not during. Record audio for drills 4 and 5; you will hate it and it will fix you. Each rubric is out of 20: 16+ means interview-ready for that topic, 12–15 means re-read the chapter and re-run in two days, below 12 means the gap is knowledge, not delivery — go back to the chapter and the lab first.
Drill 1 — Design the tick-to-trade path with a per-stage latency budget (35 min)
Prompt (read aloud, then start the clock): “Design the full path from a market-data packet arriving at the NIC to an order leaving it, for a market-making system. Give me a latency budget per stage and defend the total.”
Expected shape: ingest (kernel vs bypass choice, justified for the deployment context) → decode (binary fixed-layout vs JSON, per venue) → book update → signal/strategy → risk checks (in-process tables) → encode + TX. Numbers per stage with the two honest totals: colo-class (~1µs software) and your crypto-class reality (host µs + venue RTT ms, and why the RTT dominates placement decisions). Must include: where the two wire timestamps live, threading model (single pinned hot thread vs pipeline, and the handoff cost either way), what’s precomputed off-path, and the burst story — what degrades at 10x message rate.
Rubric (20): stages complete and ordered, nothing hand-waved (4); numbers plausible at each stage and self-consistent with the total (4); measurement honesty — timestamps, percentiles-not-averages, coordinated-omission awareness if a harness comes up (4); burst/degradation story unprompted (3); context-appropriate technology choices with the “when NOT” cases — e.g. DPDK wrong for internet venues (3); crisp close: restates budget, names the dominant term, says what you’d optimize first (2).
Drill 2 — Market-data fan-in for 20 venues with gap recovery (40 min)
Prompt: “Twenty venues, mixed protocols — binary multicast, FIX, WebSocket JSON. Design the market-data subsystem that feeds one strategy tier: normalization, gap handling, and how consumers learn they can’t trust the book.”
Expected shape: per-venue feed handler as isolation boundary (venue protocol never escapes it) → normalized internal schema, versioned per ch14 → sequence numbers end-to-end → per-venue book builder with explicit validity state machine (LIVE / GAPPED / RESYNCING, exposed to every consumer) → recovery ladder: A/B arbitration, retransmit channel, snapshot+buffered-delta splice (with the bounded-buffer overflow answer) → internal distribution (single writer per stream, no consumer backpressure onto the feed path, conflation only for the slow tier) → resource tiering: hot venues get pinned cores, the long tail shares an epoll thread. This is your home turf — the rubric penalizes not using your 20-venue war stories.
Rubric (20): normalize-layer isolation stated as a design principle, not an accident (4); gap detection AND full recovery ladder including the splice mechanics (4); validity/poisoned-book contract with consumers — the “crossed book” answer (3); no-backpressure internal distribution design (3); heterogeneity handled — per-venue quirks table, hot/cold resourcing, staleness deadlines (3); at least two concrete venue anecdotes deployed naturally as evidence (3).
Drill 3 — Your matching engine + a zero-downtime deploy story (45 min)
Prompt: “Walk me through the matching engine you built. Now ship a new version of it at 3pm with markets open — take me from CI to full size, including what goes wrong.”
Expected shape:
- Part one (~15 min): your real engine — event-sourced core, determinism contract with the forbidden-inputs list, snapshot mechanics, hot standby as log consumer, one real failover story with numbers.
- Part two (~25 min): the ch16 runbook, phase by phase — replay-regression gate with classified diffs; N-1 read-back check; follower start (snapshot + tail + live, outputs sinked); rolling hash comparison;
LeadershipTransfercutover with epoch fencing; venue session takeover (FIX seqnums vs crypto re-auth burst — both); open-order reconciliation as a hard gate; bake with the write-format flip deferred; and the rehearsed rollback branch. - Part three (~5 min): interviewer injects “hashes diverge at T-5 minutes” — you abort, disqualify, bisect offline, and say why cutting over anyway is never the answer.
Rubric (20): engine description is concrete and yours — real numbers, real failover story (4); determinism contract stated precisely and connected to why the deploy works (3); full runbook with mechanical pre-committed gates (4); session takeover handled for both FIX and WS worlds (3); rollback discipline — N-1 compatibility, write-flip-after-bake, kill-switch-before-decision (3); handles the injected divergence calmly with the correct call (3).
Drill 4 — Diagnose-the-regression roleplay (30 min, needs a partner or self-script)
Prompt (interviewer script — have a friend read it, or record yourself playing both sides): “Since Tuesday, p99.9 tick-to-trade on venue X doubled. p50 is flat. Nothing obvious changed. You have full access. Go — and I’ll answer questions as the system.”
Hidden cause (self-run version — pick one at random after writing three cards): (a) a deploy Tuesday added logging on an error path that fires under burst; (b) IRQ affinity reset by a driver update, feed lands on the wrong core; (c) venue X changed snapshot depth, tripling decode work in bursts.
Expected behavior — the method is the answer: verify the measurement first (where are the timestamps, sample counts at p99.9, conditional-on-load view); bracket in time and correlate with change records (deploys, config, host, venue notices); split code from environment with the replay trick — same day’s log through Tuesday’s and Monday’s builds, deterministic input isolates the binary; then bisect the path with per-stage histograms; state a hypothesis before each measurement and say what result would falsify it. Interviewers grade the loop — hypothesis, cheapest discriminating measurement, update — not the lucky guess.
Rubric (20): interrogates the measurement before the system (4); explicit hypothesis list, ranked by prior and cost-to-test (4); uses replay/determinism as an isolation tool unprompted (4); each step names the expected evidence and what would falsify it (3); reaches the planted cause — or a correctly-reasoned dead-end with a next step — inside 25 minutes (3); closes with prevention: the regression becomes a CI replay gate (2).
Drill 5 — Explain kernel bypass to a PM (30 min: 10 prep, 5 delivery, repeat twice)
Prompt: “Our PM asks: engineering wants three months to move feed handlers to kernel bypass. What is it, why does it matter, and should we do it? Five minutes, no jargon that survives without being unpacked, end with a recommendation.”
Target script shape (write yours, then say it in under 5): an analogy that carries the mechanism (mail sorted through the office mailroom vs a courier straight to your desk — the mailroom is fair and general-purpose, and slow because of it); the two numbers that matter (what we pay per message today, what bypass gets us, and — the number that dwarfs both — our venue RTT, which is 1000x either number if we’re internet-connected); what it costs (specialized NICs, losing standard tooling, one engineer’s quarter, new failure modes); the recommendation as a conditional (“for our colo tradfi legs, yes, staged, Onload first because it keeps the standard interface; for the crypto internet legs, no — the physics of distance makes it pointless, here’s the arithmetic”). What the drill trains: numbers translated into money and risk, jargon unpacked at first use, and a recommendation the PM can act on without trusting you blindly.
Rubric (20): analogy accurate enough that a technical listener wouldn’t wince (4); quantified trade-off including the “when it’s pointless” case (4); zero unexplained jargon — every term unpacked in one clause or cut (4); lands a concrete staged recommendation with a decision criterion (4); under five minutes, structured (signposted beginning/middle/end), spoken not read (4). Score the recording, not your memory of it.
Drill 6 — Full mock loop: the 14-day self-run schedule (planning drill, 45 min once; then execute)
The final drill is running yourself through a complete interview loop using the question banks (ch19–ch21) under realistic conditions: questions drawn cold, answered aloud, timed, scored against the model answers — never read-then-nod. Rules of engagement: 25 questions per qbank session is too many for one sitting — draw 8–10 randomly per session so ordering doesn’t become memorized rhythm; grade each answer 0/1/2 (missed it / got there with flab / crisp and complete) and re-queue everything below 2; any answer that runs past 90 seconds without a point being made gets a “so what?” interrupt from you-as-interviewer. Labs are re-typed from memory on day 12, not re-read — retrieval is the point.
The 14-day plan (60–90 min/day):
| Day | Focus | Work (read / lab / drill) |
|---|---|---|
| 1 | Event sourcing depth | Read ch13 fully (30 min cap), then write the determinism-contract and RTO-decomposition answers from memory. This RTO is the recovery time objective, not ch19’s TCP retransmission timeout |
| 2 | Schema evolution | Read ch14; hand-write the upcaster pattern and the SBE reserved-field struct without looking; draft your one real venue-migration story |
| 3 | Lab day I | Build and run the ch18 lab from the chapter; do extension 2 (break determinism with HashMap, observe cross-process failure) |
| 4 | Databases I | Read ch15 through the replication/Patroni sections; say the WAL-is-event-sourcing bridge and streaming-vs-logical answers aloud |
| 5 | Databases II | ch15 migration half: write the full expand-migrate-contract SQL from memory, check against the chapter; recite the ALTER lock table |
| 6 | Deploys | Read ch16; then Drill 3 part two only — speak the 3pm runbook end to end, timed, 20 min |
| 7 | Change management + rest | Read ch17 (lighter day); write your kill-switch taxonomy and canary-graduation answers as six bullet lines each |
| 8 | Qbank: state | ch21 session one — 10 questions drawn cold, spoken, scored; re-queue the weak ones |
| 9 | Drill 1 + networking qbank | Drill 1 full (35 min + scoring); then 5 questions from ch19 |
| 10 | Drill 2 + networking qbank | Drill 2 full (40 min + scoring); then 5 more from ch19, prioritizing bypass/colo honesty questions |
| 11 | Qbank: performance | ch20 session — 10 questions cold, spoken; coordinated omission and “how do you KNOW” must score 2 or the day repeats |
| 12 | Lab day II | Re-type the ch18 lab from memory (target: compiling and passing in ≤60 min); do extension 1 (real two-process cutover) |
| 13 | Drill 3 + Drill 4 | The 45-min flagship drill, recorded; then Drill 4 with a randomly drawn hidden-cause card |
| 14 | Dress rehearsal | Drill 5 twice (record, score, redo); then a mixed cold-draw: 4 questions from each qbank, plus re-queued day-8/11 failures; write your top-5 weak spots for the following week |
Interleaving is deliberate: reading and its matching drill are 2–6 days apart because retrieval after forgetting is what builds interview recall, and every qbank session mixes banks by day 14. If an interview lands mid-plan, days 6, 8, and 13 are the highest-yield subset — do those three.
Rubric for the loop itself (20): all 75 qbank questions attempted cold at least once (4); every sub-2 answer re-queued and cleared (4); drills 1–5 each run with scored rubric, 16+ achieved or repeated (5); lab re-typed from memory successfully on day 12 (3); recordings actually reviewed for drills 4–5 (2); weak-spot list written on day 14 and scheduled (2).
Interviewer will ask
This chapter’s version of the box is about the meta-round — how interviewers probe your preparation and self-assessment itself.
Q1: “How did you prepare for this interview?” Tell the truth with structure: identified my gaps against the role (DB operations, deployment of always-on stateful systems — my engine background is strong, my platform-ops vocabulary was thinner), built a study plan interleaving reading, a runnable lab, and timed spoken drills, and pressure-tested with question banks answered cold and scored. Naming a specific artifact — “I built a 250-line event-sourced book that live-upgrades v1→v2 with hash-verified cutover” — turns “I prepared” into evidence.
Q2: “What’s your biggest technical weakness for this role?” Pick a real one you’ve bounded and worked: “Production colo networking — multicast operations, kernel bypass — I know the theory cold and I’ve operated the crypto analog of every failure mode, but I haven’t carried the pager for a multicast outage. Here’s how I’d close it in the first month.” Never claim a strength dressed as a weakness; interviewers pattern-match that instantly.
Q3: “Tell me about a time you were wrong about a system.” Have one prepared from your real history — a determinism bug, a recon gap, a venue assumption that failed — told as: what I believed, what the evidence was, the moment I updated, what mechanism (not intention) now prevents it. The mechanism ending is what separates a growth story from an anecdote.
Q4: “Whiteboard question you’ve clearly seen before — do you say so?” Yes, one sentence — “I’ve worked through this class of design; want me to go fast and you push on the corners?” — then deliver at full quality. Interviewers usually know the banks; pretending to derive freshly what you’ve rehearsed reads as performance, while disclosure plus depth reads as preparation, which is the trait they’re hiring.
Q5: “You’ve talked a lot about your event log. What if we don’t use event sourcing here?” Show the transfer, not the attachment: the log is one implementation of properties I’d want anywhere — reproducibility of incidents, verifiable state transfer for deploys, an audit trail. If the shop uses checkpoint/restore or DB-backed state, I’d ask how they get those properties and adapt — and I can argue trade-offs of their approach against mine credibly precisely because I’ve operated one end of the spectrum.
Q6: “Any questions for us?” (It’s a drill too.) Prepare three that do work for you: one that shows operational depth (“what does your deploy-to-full-size timeline look like for engine changes, and what gates it?”), one about failure culture (“walk me through your last significant incident’s post-mortem — what changed after?”), and one calibration question (“what does the strongest engineer at this level here do that others don’t?”). Their answers also tell you whether the shop practices what this book preaches — which you now know how to evaluate.
Further reading
- The question banks (ch19–ch21) and labs of this book — the drills’ raw material; the drills are worthless without them.
- Gil Tene, “How NOT to Measure Latency” — re-watch before Drill 1 and Drill 4; the measurement-first instinct is the single highest-yield drill habit.
- Kleppmann, DDIA — skim chapter summaries (each chapter ends with one) the night before any loop; they’re the best-written 2-page refreshers in print.
- Aeron Cluster docs + Martin Fowler’s event-sourcing and blue-green articles — the citable anchors for Drill 3; naming real systems and real authors under pressure signals depth cheaply.
- Your own incident notes and venue-quirks table — the most further of further reading: every drill above improves more from one real story of yours than from any external source.