You Are the Venue: Exchange Architecture

Before you start. This chapter assumes event sourcing / snapshot + replay (state as a log of events, rebuildable from any snapshot, ch13), deterministic single-writer (one thread owns the state; same inputs → same outputs, ch13), TCP vs UDP trade-offs (ch02), clocks and timestamping (why “who was first” is a measurement problem, ch07), latency percentiles and door-to-ack measurement (ch08), and the CLOB / price-time priority basics from ch00f. If any are new, read those first.

You’ve spent Parts 0–III on one side of the wire: your process races to hear the venue and reply faster than everyone else. This chapter flips the table. You are now the venue. Every trick you learned as a taker — timestamping, feed gaps, order acks, rate limits — has a mirror image on this side, and the mirror image is usually harder, because the venue’s problems are everyone’s-flow problems, not your-flow problems.

Here’s the good news: you’ve already built the hardest single component. Your Crypto.com matching engine — deterministic single-writer, price-time priority, event-sourced with snapshot/replay, hot-standby — is the center box of this diagram. What you haven’t built is everything around it, and the surround is what makes a matching engine into an exchange of record.

The canonical architecture

                         THE MODERN EXCHANGE, END TO END

 clients            edge                  the heart              deterministic consumers
────────       ─────────────         ─────────────────       ──────────────────────────────
 FIX ───┐      ┌───────────┐
 OUCH ──┼──►   │  order    │
 WS ────┘      │  gateway  │──┐
               └───────────┘  │       ┌─────────────┐        ┌────────────────────┐
               ┌───────────┐  ├──────►│  SEQUENCER  │───┬───►│ matching engine(s) │──► acks/fills
 FIX ───────►  │  gateway  │──┤       │ (assigns    │   │    │  (per-symbol shard)│    (back out via
               └───────────┘  │       │  global seq │   │    └────────────────────┘     gateways)
               ┌───────────┐  │       │  number to  │   │    ┌────────────────────┐
 WS ────────►  │  gateway  │──┘       │  EVERYTHING)│   ├───►│ market-data        │──► L1/L2/L3 feeds
               └───────────┘          └──────┬──────┘   │    │ publishers         │
                                             │          │    └────────────────────┘
                                             ▼          │    ┌────────────────────┐
                                      ┌────────────┐    ├───►│ drop copy /        │──► clearing,
                                      │ EVENT LOG  │    │    │ clearing feed      │    brokers' risk
                                      │ (sequenced,│    │    └────────────────────┘
                                      │  durable,  │    │    ┌────────────────────┐
                                      │  replayable│    └───►│ surveillance /     │──► regulator,
                                      └────────────┘         │ regulatory capture │    audit trail
                                                             └────────────────────┘

Read it left to right and notice the shape: many chaotic inputs → one total order → many deterministic outputs. Everything left of the sequencer is concurrent, racy, and unfair by nature (packets arrive when they arrive). Everything right of it is a pure function of the sequenced stream. The entire architecture exists to make that boundary as early, as fast, and as defensible as possible.

The sequencer: partition-0 for the whole market

Start with the analogy from your world: a Kafka topic with exactly one partition. Recall what that means — Kafka is an append-only log, and each message’s offset is its position number in that log; one partition means one single, total order that every consumer sees identically. The sequencer is that idea promoted to be the market’s backbone: one component that stamps a global, gap-free sequence number on every inbound message — orders, cancels, admin events, even clock ticks. (Why time itself is a message: the determinism checklist below.) It is the single place where “what happened, in what order” is decided for the whole market; the partition offset is market truth. And every other property of an exchange — fairness, replayability, recovery, audit — is downstream of that one number.

You built the consumer side of this at Crypto.com: your matching engine was a deterministic single-writer that applied events in order. The sequencer answers the question you never had to ask: who decides the order? In your system, the order was whatever your single writer happened to dequeue. Fine for one application. Not fine for a market, because at a venue the ordering is the product — firms paid for colocation to fight over it. The sequencer settles that fight; everything after it is bookkeeping.

Why this design won over, say, a cluster of matching engines with distributed coordination:

Determinism. Same sequenced input stream → same books, same fills, same feed, bit for bit. You know this property from your engine; the sequencer extends it from one component to the entire venue. The market-data publisher, the clearing feed, and surveillance never talk to the matching engine — they consume the same log and derive consistent state independently. No cross-service RPC on the hot path, no “did the feed and the fills disagree” incidents.
Fairness you can defend. “Order A matched before order B because A got sequence 4,412,907 and B got 4,412,908” is an answer you can give a regulator or an angry HFT firm. “A won a mutex race inside engine shard 3” is not.
Trivial recovery. A downstream component crashes → restart, load a snapshot, replay the tail from its last applied sequence number — exactly your snapshot/replay design (ch13). No distributed reconciliation.
One number tells you everything under load. Lag = sequencer head minus consumer position. You’ve run Kafka consumers; same dashboard.

The cost: the sequencer is a single point of serialization — every message in the market funnels through one code path. That sounds insane until you do the arithmetic. A tight sequencer does almost nothing per message: validate framing, stamp number + timestamp, append to the log, hand off. Call it 100–200 ns per message on one core → ~5M messages/sec on a single thread — and NASDAQ’s entire equity market peaks in the low tens of millions of messages/sec across all instruments. One well-written thread genuinely covers most markets, which is why this “obviously unscalable” design runs the world’s exchanges. LMAX, a UK exchange, made the argument famous; Aeron, a trading-messaging library, and modern exchange stacks industrialized it.

Here is what the sequenced log physically is, since “append to the log” is doing a lot of work in that sentence. (mmap’d = the file mapped straight into the process’s memory, so appending is just a memory write — no write() syscall per record.)

   sequencer memory:  [ mmap'd segment files, append-only, fixed-size records ]
                            │ written sequentially (the disk-friendly pattern
                            │ from ch00f — same physics as Postgres WAL /
                            │ Kafka segments)
                            ├──► shipped to standby/quorum (replication, below)
                            └──► consumed by engines/publishers via shared-memory
                                 ring buffers or reliable multicast — consumers
                                 POLL forward through it; the sequencer never
                                 waits for any consumer

Two properties matter. Sequential append is the one disk pattern that keeps up with the message rate (your WAL intuition transfers directly), and the sequencer never blocks on consumers — a lagging engine or publisher falls behind in the log and catches up; backpressure toward the sequencer would let the slowest component in the building set the market’s pace. If a consumer falls off the retained window (the log keeps only recent history in fast storage; older segments age out to archives), that’s an incident for that consumer (snapshot + replay to recover), never a brake on the market. Hold this asymmetry; it returns with teeth in ch25’s slow-consumer problem.

Where the matching engine sits: sharding by symbol

One matching engine process per instrument shard — a group of symbols one engine owns exclusively:

 sequenced stream ──► demux by symbol ──► [engine shard 1: BTC-USD, BTC-PERP]
                                          [engine shard 2: ETH-*, SOL-*]
                                          [engine shard 3: long tail, 3000 symbols]

The partitioning is legitimate because no cross-symbol ordering guarantee is needed: an order on AAPL and an order on MSFT never interact inside a book, so they can match in parallel without violating price-time priority. (Cross-symbol products — futures spreads, implied liquidity across a curve — are the exception; venues offering them co-locate those legs on one shard or accept real complexity. Know the caveat; it’s a favorite interview follow-up.)

The operational problem is the hot symbol. Load isn’t uniform: on a big day one instrument can be 40% of all market messages. It’s the noisy tenant on your multi-tenant Postgres host — the one whose table gets all the writes, where “add more tenants per box” solves nothing. And here the usual escape hatch is welded shut: you cannot sub-shard a single order book, because price-time priority demands a single writer per book. That leaves exactly three levers:

make the hot engine itself faster;
give it a dedicated core or host;
evacuate every other symbol off its shard.

Venue capacity planning is substantially “which symbol goes hot after the next listing or news print, and is its shard ready.”

Determinism and fairness as product features

Two clients send an order in the “same microsecond.” Who wins? At a venue the answer must be: whoever’s message was sequenced first — and arrival at the sequencer is the definition of first. Not gateway receive time, not client send time: sequence number. Everything else is evidence about the ordering, not the ordering itself.

That sounds circular until you see what it buys:

Gateways timestamp on ingress (hardware timestamps where the venue is serious, ch07) so the venue can demonstrate that sequencing tracked arrival — the timestamp is audit evidence, not the tiebreaker.
Serious venues publish their fairness model: how gateways feed the sequencer, whether gateway→sequencer paths are latency-equalized, what happens on ties. CME, Eurex, NASDAQ all document this publicly — because sophisticated clients (your former self, running an SOR) will reverse-engineer it empirically anyway, and a fairness model clients discover before you disclose it is a scandal in waiting.
The sequenced log doubles as the regulatory audit trail. When the regulator asks “reconstruct 14:30:00–14:30:10 on the day of the flash event,” the venue replays the log and produces exact book state at any sequence number. It’s your engine’s event-sourced replay, except the output is legal evidence. That’s why “deterministic consumers of a durable log” is near-mandatory rather than merely elegant: CAT (the Consolidated Audit Trail, the US regulator’s every-order database) in US equities, MiFID II record-keeping in Europe.

Fairness, in short, is a line item clients pay for and regulators examine. Your matching engine had determinism for correctness; a venue has it for defensibility.

Sequencer failover: the genuinely hard problem

Here’s where venue-side is harder than what you built. Your hot-standby followed the primary’s event stream and could take over. Now ask the venue-grade question: when the primary dies, can you prove no acknowledged order was lost?

The trap: primary sequences message N, sends the ack, crashes before the standby saw N. Standby takes over at N−1. The client holds an ack for an order the new primary has never heard of. For an exchange of record this isn’t a bug, it’s an existential event — the ack is a legal commitment.

The two production-grade answers:

  Option A: primary/standby, synchronous replication
  ┌─────────┐  seq N   ┌─────────┐
  │ primary │─────────►│ standby │      rule: the client ack for N goes out
  │         │◄─────────│  (ack)  │      ONLY after the standby confirms N
  └────┬────┘          └─────────┘
       └──► client ack (after standby ack)

  Option B: Raft-style consensus cluster (e.g. Aeron Cluster)
  ┌────┐ ┌────┐ ┌────┐     a message is "sequenced" when a majority
  │ n1 │ │ n2 │ │ n3 │     holds it in their log; leader failover is
  └────┘ └────┘ └────┘     automatic; ack only after commit

Both make the same trade: an acknowledged message exists on ≥2 machines before the ack leaves the building. That synchronous hop sits inside your door-to-ack path — a first-class citizen of the latency budget, and the reason sequencer nodes share a low-latency fabric (a dedicated private network between the nodes). “We async-replicate and accept a tiny loss window” — a perfectly reasonable call in most systems you’ve shipped — is off the table when the ack is a contract. If your Crypto.com hot-standby was async (most application-layer ones are), that’s the gap between what you built and venue grade — name it crisply.

Aeron Cluster is the open-source embodiment of Option B: Raft — the standard recipe for leader election plus majority-ack replication — driving a replicated log, with deterministic state machines on top. Several production crypto and FX venues run on it or on the same design; know it by name.

The other consumers: drop copy, clearing, surveillance

The right-hand column of the big diagram has two boxes you never touched as a client but will be asked about at any broker-platform or exchange interview:

Drop copy. Picture a webhook fanout where every payment.settled event also goes to a second, audit-owned endpoint that the merchant’s finance team controls. Drop copy is that for trading: a real-time copy of an account’s execution reports (and often order events), delivered to a different session than the one doing the trading. Why it exists: brokers and clearing firms are on the hook for their clients’ risk. ch24’s credit checks are pre-trade; drop copy is how the risk desk watches post-trade in real time. It’s also how a firm’s own independent risk system cross-checks what its trading system believes. Implementation is nearly free in this architecture: a drop-copy session is one more filtered projection of the sequenced log — filter by account, serialize as execution reports, deliver on a FIX session. No new source of truth, so it cannot disagree with the fills.

Clearing feed — the post-trade stream to the clearing house / settlement layer: matched trades with counterparties, quantities, prices — the thing that turns “the engine printed a fill” into “money and assets actually move.” At a crypto venue with internal custody this loop is short; in tradfi it’s an external institution (DTCC, CME Clearing) with its own formats and its own timeliness rules.

Surveillance / regulatory capture — a consumer that stores everything and runs pattern detection over it: spoofing (fake orders cancelled before they trade), layering (spoofing stacked at several price levels), wash trades (trading with yourself to fake volume), marking the close (pushing the official closing price with last-minute orders). Two design notes worth having: it must consume the full sequenced log, not a summarized feed, because manipulation lives in the order events that never trade (a spoofer’s signature is orders placed and cancelled — invisible in a trades-only view); and it’s the one consumer where falling behind is tolerable — surveillance can lag minutes without harm, so it runs on cheap batch-friendly infrastructure, while the market-data publisher lags microseconds at most. Same log, wildly different consumer SLAs — a nice concrete instance of the architecture’s flexibility.

Determinism gotchas: what actually breaks replay

You know these from building your engine, but the venue interview version wants them as a checklist, because every consumer of the sequenced log must obey them, not just the engine:

Wall-clock reads in logic. Any now() inside a decision path breaks replay. Time must arrive as sequenced events (the sequencer stamps a timestamp into each message; timers become injected tick events). Your engine’s timers were driven by event time, not machine time — same rule, venue-wide.
Hash-map iteration order. Iterating an unordered map to, say, expire orders produces machine-dependent order. Sorted structures or insertion-ordered containers only, anywhere order can leak into output.
Floating point. Cross-platform/compiler FP differences are tiny but nonzero; venues use integer ticks and fixed-point (ch00f) so replay is bit-exact. You did this in your engine; here it’s non-negotiable because the regulatory replay must match production.
Threads inside a consumer. Parallelism inside one deterministic consumer reintroduces racing. Parallelism lives between consumers (shards, projections), never inside one.
Randomness and uninitialized memory. Any RNG must be seeded from the log; any uninitialized read is a latent divergence bomb that detonates weeks later on the standby.

The test that keeps you honest: continuously replay production’s log on a shadow instance and diff state hashes at checkpoints. Divergence pages someone. This is your snapshot/replay regression testing promoted to a permanent production invariant.

Auditing an execution: `explain <order_id>`

Here is what all that determinism discipline buys. A client (or a regulator) asks: “why did my order fill at that price, against that counterparty, at that moment?” At a web company this question triggers log-spelunking and a shrug. At a venue it triggers a query, because the venue is a deterministic fold over the sequenced log — the audit is a replay.

The procedure, mechanical from end to end:

Find the order’s ingress chain. Gateway receipt (hardware timestamp, session, account), the risk-gate verdict, and the sequence number the sequencer stamped — say seq N. Every hop stamped its passage; the gaps between stamps are evidence too (they show who delayed what).
Rebuild the world as of seq N−1. Load the nearest snapshot at or before N−1, replay the tail up to N−1. You now hold the exact book the order walked into: every resting order, in exact queue position, each tagged with the sequence number it arrived at.
Re-run the match. Feed event N to the same engine version. Determinism guarantees the same fills fall out — and now the “why” reads straight off the state: “best ask was 10001 with order Y resting first (arrived seq M, never modified — a modify would have sent it to the back of the queue); X was a marketable buy for 500; price-time priority filled Y’s 300, then Z’s 200 at the next level.” Every clause points at a sequence number. No opinions — arithmetic.
Version-stamp the rules. The replay proves the decision only if it runs the same rules: engine version and matching config are themselves events in the log (the change-management chapter’s config-as-events, ch17), so the answer includes “matched under rules vX, config as of seq K.”

Build it as a tool, not a runbook: explain <order_id> locates the seq range, loads the snapshot, replays, and emits the human narrative plus the machine dump. The same replay engine powers surveillance queries and incident forensics — you don’t build audit infrastructure; you build determinism plus snapshots once, and audit falls out as a query. And this is why a diverging replay is not a quality bug but a compliance incident: a replay that diverges is an audit that proves nothing. The shadow-replica hash check above is your continuous proof that the audit machinery still works.

Two supporting pieces close the loop. Drop copy (previous section) answers “what is happening right now” for watchers outside the system. The log itself goes to write-once retained storage — regulators demand years of retention — with a hash chain over the records so tampering is evident. Drop copy is the live witness; the log is the court record.

Snapshots: what, when, how — without stopping the market

The snapshot mechanics are the event-sourcing chapter’s (ch13), applied venue-scale. What goes in follows one generative rule — everything the fold reads, nothing it derives:

per-symbol books with resting orders in queue order (queue position is the fairness product; a snapshot that loses it is worthless),
account and risk-counter state (the pre-trade gate’s memory),
both sequence spaces (engine seq and feed seq — the two counters the mini-market lab makes concrete),
config epoch, engine version, and open auction state if snapshotted mid-auction.

When: every N events or T seconds per shard, always at a sequence boundary — the label snapshot.{seq} means “state exactly as of seq N, nothing mid-event,” and that label is what makes step 2 of the audit legal.

How, without a pause the latency distribution would wear: three standard mechanisms, chosen per shard. Fork the process at a boundary and let the child serialize while the parent keeps matching (the OS’s copy-on-write shares pages until the parent writes one). Or keep the book as a persistent structure and hand the snapshotter the old root pointer — a git commit, while the writer moves on. Or, for cold symbols, micro-quiesce: hold intake for the microseconds a double-buffer swap takes. Durability is ch13’s ritual verbatim: write to a temp file, fsync, rename to snapshot.{seq}, fsync the directory, checksum, keep K generations.

The closure worth saying in an interview: snapshot + log tail is one mechanism serving three masters — crash recovery, failover (ch16), and the audit entry point above. That triple duty is why venues treat snapshot cadence as a product decision, not an ops afterthought: it bounds recovery time and bounds how long explain takes to answer.

Throughput shape: you now receive everyone’s flow

As an SOR operator you sent orders — your flow, your rate. The venue receives the sum of all participants, and the sum has a brutal shape:

Steady state is a lie. Opens, closes, economic prints (scheduled data releases — CPI, payrolls — hitting the market), and liquidation cascades (one forced sale pushing the price into triggering the next — an avalanche of margin calls) produce 10–100× bursts over median load, concentrated into milliseconds. A venue provisioned for 2× median falls over exactly when being up matters most — and when it’s on the news.
Concrete anchors: NASDAQ ITCH peaks in the tens of millions of messages/sec market-wide on volatile opens; a top crypto venue sees hundreds of thousands to millions of order-messages/sec at cascade peaks. Human translation: at 5M msgs/sec a message arrives every 200 ns — roughly one per L3 cache miss. Burst capacity, not average capacity, is the spec.
Order-to-trade ratios of 20:1 to 100:1 mean the flow is overwhelmingly cancel/replace churn from market makers. The load profile is metadata churn, not fills — which is why messaging policies exist (ch24).

Latency numbers to hold

Path	Door-to-ack (gateway in → ack out)
Your old client-side world (WS over internet to a cloud venue)	~10 ms round trip
Decent cloud-hosted crypto venue, software path	~50–500 µs
Serious colo venue, tuned software (kernel bypass, ch04)	~10–50 µs
CME/NASDAQ class, hardware-assisted edge	sub-10 µs; wire-to-wire budgets in single-digit µs

Human scale: the gap between your old 10 ms client-side world and CME’s sub-10 µs is three orders of magnitude — all of it the physics and architecture from Parts 0–I, applied on the receiving side.

The trading day has a shape: sessions and auctions

One more venue-side concept your continuous-trading crypto background skips: the market itself has states, and state transitions are the venue’s highest-stress moments.

   pre-open ──► OPENING AUCTION ──► continuous trading ──► CLOSING AUCTION ──► closed
   (orders          (one batch          (the CLOB you           (one batch
    accumulate,      cross at a          know)                   cross; sets
    no matching)     single price)                               official close)

An auction (call auction / uncrossing) is batch matching instead of streaming — the end-of-day job that nets a whole day of card transactions in one pass. Instead of matching continuously, the venue collects orders for a window, then computes the single price that maximizes matched volume and executes everyone crossable at that one price.

A worked uncrossing, to make “maximizes matched volume” concrete:

   willing buyers  (limit ≥ P):   P=$100 → 600 sh    P=$101 → 500 sh    P=$102 → 300 sh
   willing sellers (limit ≤ P):   P=$100 → 200 sh    P=$101 → 400 sh    P=$102 → 700 sh
   crossable = min(buy, sell):          200               400 ◄ max          300
   → $101 wins; 400 shares trade, everyone at that single price

Why the venue engineer cares — three stakes:

The open is the burst. The accumulated overnight order flow hits the book at once — part of why bursts are 10–100×.
It’s a second deterministic code path. The uncrossing algorithm is separate from continuous matching, but it must live inside the same sequenced-log discipline as everything else.
The close is real money. The closing auction’s print (the executed trade published on the public feed) is the official close that trillions in index funds benchmark against — a correctness bug there reprices ETFs.

Crypto mostly trades 24/7 continuous, but even there, listings-day opens and post-halt reopens are auction-shaped problems (a mass of accumulated orders needing a fair single crossing), and venues that reopen a halted book straight into continuous matching produce the wild first-print artifacts you saw as a client.

Halts themselves — circuit breakers, per-symbol limit-up/limit-down pauses — are sequenced admin events like everything else: the halt, the quote-only window, the reopen auction all flow through the same log, so the audit trail of why the market stopped is as replayable as the trades.

Crypto-venue specifics (the outside view, confirmed from inside)

Things you observed as a 20-venue client that now make architectural sense:

WebSocket gateways for both orders and data: TCP per client, JSON or bespoke binary, no multicast possible over the public internet. The fanout consequences are the whole story of the feed-publishing chapter (ch25).
Rate limits per API key (you lived under these): the gateway protecting the sequencer’s inbound funnel, not arbitrary meanness — ch24.
Matching engines behind cloud load balancers: some venues front the order path with a cloud LB. Serious venues don’t — an LB adds jitter (two identical clients get different paths, so fairness becomes indefensible), hides client identity from the edge, and inserts a hop the venue can’t timestamp or reason about. When you saw a venue’s ack latency go bimodal for a week, this class of middlebox was often why. As the venue, the rule is: nothing between the client and your timestamping gateway that you don’t control.

Anatomy of an ack: one order, door to door

Tie the whole chapter together by tracing a single marketable order through a good software-path venue (no FPGA), with the clock running:

   t=0        order's last byte hits the gateway NIC (HW timestamp — the
              "door" in door-to-ack, the clocks chapter)
   t+2µs      gateway: session lookup, decode, risk chain, token bucket,
              stamp, forward                            (the gateway budget)
   t+4µs      sequencer: assigns seq 4,412,907; message is now "real"
   t+6µs      sync replication: standby/quorum confirms 4,412,907
              ◄── the ack is now LEGAL to send; nothing was allowed
                  to promise anything before this line
   t+7µs      engine shard (deterministic consumer): matches against the
              book → fill events, themselves sequenced outputs
   t+9µs      publisher emits the book delta + trade on the feed;
              gateway serializes the execution report back to the client
   t+11µs     ack/fill's first byte leaves the venue NIC — door-to-ack ≈ 11µs

Three things to notice, because they’re the chapter in miniature. First, where the point of no return sits: not at the engine, but at replication confirm — the order “happened” when it was durably sequenced, and matching is downstream bookkeeping (this is why a venue can honestly ack receipt before the match completes, and why acked-but-crashed is recoverable). Second, everything after the sequencer could run at different speeds without breaking correctness — if the publisher lags 50µs behind the engine, the feed is late but never wrong; ordering, not scheduling, is the invariant. Third, the budget’s big rocks are the replication RTT and the two NIC traversals — which is why sequencer fabric latency and kernel bypass (ch04) dominate venue tuning, and why the remaining software must live in the L1/L2 cache regime you learned in ch00a. Multiply this 11µs picture by “a message every 200ns at peak” and you have the venue’s entire performance problem on one page.

Plain-English recap

The sequenced log is physically boring on purpose: append-only segments (WAL/Kafka physics), consumers poll forward, and the sequencer never waits for anyone — the slowest component in the building must never set the market’s pace.
The point of no return is replication-confirm, not the match: an order “happened” when it was durably sequenced; matching, publishing, and clearing are all downstream bookkeeping that can lag without ever being wrong.
An exchange is a funnel: many chaotic inputs → one component that decides the order of everything (the sequencer) → many independent consumers deriving state from that one ordered log. Kafka with a single partition, where the partition is the market.
You already built the most famous consumer — the matching engine. The venue-shaped work is the funnel, the durable log, and proving the ordering was fair.
The trading day has states — open auction, continuous, close auction, halts — all flowing through the same log as sequenced admin events; the close’s single print is what index funds benchmark against, so the uncrossing code path carries real-money correctness weight.
Sharding is by symbol because AAPL and MSFT never interact; the hot-symbol problem is the noisy tenant on your multi-tenant Postgres box, except you can’t split the tenant’s table — a book demands a single writer.
Fairness is a documented product feature, not an emergent property: sequence number is the tiebreaker, timestamps are the audit evidence, the log is what you hand the regulator.
Failover’s hard rule: no acknowledged order may be lost — an ack leaves only after the message exists on two machines. Synchronous replication or Raft; the replication hop lives inside the ack-latency budget.
Drop copy, clearing, and surveillance are just more projections of the same log — a drop-copy session is a filtered webhook fanout of an account’s events, and it can’t disagree with the fills because it has no independent source of truth.
Determinism is a venue-wide discipline with a checklist: no wall-clock in logic, no unordered-map iteration into output, integer ticks not floats, no threads inside a consumer, no unseeded randomness — enforced by continuously replaying prod’s log on a shadow and diffing state hashes.
Provision for the open and the liquidation cascade, not the average: 10–100× bursts are when a venue earns or torches its reputation.

Interviewer will ask

“You built a matching engine at Crypto.com — what’s the difference between that and running a venue?” “My engine was the center box of this chapter’s diagram — a deterministic single-writer consuming an ordered event stream. But I produced that stream myself, so the ordering only had to be internally consistent. A venue picks up three obligations my engine never had, and each one maps to a component. It must decide the order among competing external clients — that’s the sequencer, the one place ‘what happened, in what order’ is settled for the whole market. It must defend that decision to regulators and angry HFT firms — that’s latency-equalized gateways with hardware timestamps at ingress, plus the sequenced log doubling as the audit trail. And it must never lose an acknowledged order — that’s synchronous replication, the ack leaving only after the message exists on a second machine. So the engine is the famous component, but the venue-shaped work is the surround: the funnel, the durable log, the proof of fairness. I built the engine — and spent years on the client side probing exactly that surround, because my SOR empirically reverse-engineered venues’ fairness properties.”

“Why does a single sequencer scale? Isn’t a global serialization point a bottleneck?” “Start from the chapter’s picture: the sequencer is Kafka with one partition, and the partition’s per-message work is tiny — validate, stamp, append, hand off — call it 100–200 ns, so one core clears ~5M msgs/sec, which covers most entire markets. Now price the alternative. A total order is required for fairness within a book, so a distributed design still has to make one ordering decision per message — but consensus makes each decision with a network round trip between nodes, microseconds, where the single thread makes it with one memory write, nanoseconds. The ‘bottleneck’ is orders of magnitude faster than anything you’d replace it with. So you keep the ordering on one thread and scale the genuinely expensive work — matching, publishing — as deterministic consumers behind it, sharded by symbol since cross-symbol ordering isn’t needed. It’s the shape I ran in production: the single-partition log was never the bottleneck; the consumers were.”

“Two orders arrive in the same microsecond on different gateways. Who wins?” “Whichever is sequenced first — the sequence number is definitionally the answer. The real engineering question is whether gateway topology makes that fair: gateways must be interchangeable, paths to the sequencer latency-equalized, ingress hardware-timestamped so you can audit that sequencing tracked arrival. And you publish the model — clients like my former SOR will empirically reverse-engineer it anyway, so it had better be disclosed and defensible.”

“How do you fail over the sequencer without losing orders?” “The invariant is: no ack leaves until the message is durable on a second machine. Two shapes — lockstep primary/standby where the primary waits for standby confirmation before acking, or Raft-style like Aeron Cluster where ‘sequenced’ means majority-committed. My hot-standby at Crypto.com followed the primary’s stream but the primary acked before replication confirmed — acceptable for an internal system, not for an exchange of record where the ack is a contract. Venue-grade means eating the replication RTT inside the door-to-ack path.”

“How would you capacity-plan a new venue?” “For the burst, not the mean. Opens and liquidation cascades run 10–100× median load, concentrated into milliseconds, and 20–100:1 order-to-trade ratios mean most of it is maker cancel/replace churn — so a venue provisioned for 2× median falls over exactly when being up matters most. Inside that burst, the binding constraint is the hottest single shard: on a big day one symbol can be 40% of all messages, and you can’t sub-shard a book because price-time priority demands a single writer. Which means the plan is about that shard, not aggregate capacity — budget it for the worst credible burst, keep it on dedicated hardware, evacuate every other symbol off it. Then prove the headroom rather than assert it: replay captured real bursts through the sequenced log, which event sourcing gives you for free — the same replay discipline I used for my engine’s regression testing.”

“What happens if the matching engine and the public feed disagree?” “Run the incident. A market maker’s recon desk calls: their private execution report shows a fill at 14:31:07 that the public tape never printed. First call to make: which side is true? The engine’s fill — money moved, clearing saw it on drop copy — so the feed is what’s lying, and every client trading off it has been quoting against a false book since the divergence began. Immediate moves, in order: mark the feed suspect, force a snapshot republish so consumers resync to true state, then diff the publisher’s book against a reference replay of the sequenced log to find the first divergent sequence number. And that diff exposes something structural: in this architecture a divergence should be impossible — engine and feed are both deterministic consumers of the same sequenced log, so they can’t drift apart; a divergence at seq N means one of them computed a different state from identical input. That’s a determinism bug, not a synchronization bug — wall-clock leaking into logic, hash-map iteration order, uninitialized memory — and you find it by replaying the log through both consumers offline and bisecting to the first event where their state hashes split. Which is exactly why venues ban nondeterminism in anything downstream of the sequencer. From the client side I have caught venues whose private fills contradicted their public feed — that’s the signature of a venue not built this way, and it was a real input to my SOR’s venue-quality scoring.”

The HFT Gap Closer