Exchanges, Sessions, and What’s Under a Database
Two vocabularies block Parts I–III. The first is traditional-finance market structure — you’ve built most of these concepts for crypto, but the book names them in tradfi terms and assumes you know the plumbing (feeds, FIX, colocation). The second is database internals — you drive Postgres expertly as an application developer, but the database and zero-downtime chapters (ch15, ch16) talk about it the way a DBA (database administrator) does: WAL, MVCC, replication modes, lock queues. This chapter maps both onto things you already own.
Part A — Market structure, tradfi edition
1. The venue stack: you built this already
traders / firms
│ orders in, executions out
▼
┌─────────────────────────────────────────────┐
│ EXCHANGE (the "venue") │
│ │
│ gateway ──► MATCHING ENGINE │
│ ┌───────────────────────────┐ │
│ │ ORDER BOOK for one symbol │ │
│ │ asks 100.02 │ 500 │ │
│ │ 100.01 │ 1,200 │ │
│ │ ──── spread ──── │ │
│ │ bids 100.00 │ 800 │ │
│ │ 99.99 │ 2,000 │ │
│ └───────────────────────────┘ │
│ │ │
│ ▼ │
│ market data publisher ──► everyone
└─────────────────────────────────────────────┘
Quick crypto→tradfi dictionary, because you built a matching engine and none of this is new — only the words are:
- Exchange / venue — the organization (and its machines) where orders meet; “venue” is the generic word because one instrument often trades on many of them. — Your Binance/exchange concept, except in tradfi one stock trades on a dozen venues simultaneously and firms must route between them. — Multi-venue is why “smart order routing” exists as a job title.
- Matching engine — the single-threaded-per-symbol core that matches incoming orders against the book. — You wrote one. Same object.
- Order book — the sorted resting orders per symbol, bids and asks. — Same object; tradfi says “symbol” where crypto says “pair,” and top of book / BBO (Best Bid and Offer) for the best prices.
- Price-time priority — the standard matching rule: better price wins; at equal price, earlier arrival wins — FIFO (first in, first out) per price level. — Identical to what you implemented. — The consequence is the whole reason this book exists: at equal price, queue position is won by latency. Arriving 1 µs earlier is the difference between being filled and watching.
2. Market data feeds: at-least-once delivery, but over UDP
EXCHANGE YOU
┌──────────────┐ incremental feed A ┌─────────────────────────┐
│ │ ═════════════════════► │ arbiter: │
│ publisher │ (UDP multicast, │ take whichever copy of │
│ │ seq: 101,102,103…) │ seq N arrives first, │
│ │ incremental feed B │ drop the duplicate │
│ │ ═════════════════════► │ │
│ │ (same data, second │ gap? (…103, 105…) │
│ │ network path) │ ├─ wait: B may have │
│ │ │ │ 104 in flight │
│ │ snapshot channel │ └─ else: resync from │
│ │ ─────────────────────► │ snapshot │
└──────────────┘ (periodic full book) └─────────────────────────┘
An exchange does not send you “the order book.” It sends a firehose of changes, and keeping a correct book is your problem. The machinery:
- Incremental feed — a stream of deltas (“add 500 @ 100.01”, “cancel order X”), each carrying a sequence number — a monotonically increasing per-stream counter stamped on every message. — This is a webhook event stream with an event ID, and your book is the projection you fold it into. — Sequence numbers exist because the transport (below) can drop messages, and a missed delta means your book is silently wrong — which in trading means quoting prices off a false picture.
- Snapshot channel — a slower side-channel broadcasting the full current book periodically, stamped with the sequence number it reflects. — It’s the
GET /current-statereconciliation endpoint you pair with any webhook integration: join by applying the snapshot, then replay buffered incrementals with higher sequence numbers. — This is how you bootstrap at startup and recover after falling behind. - Gap detection — noticing that you received seq 103 then 105: 104 is gone, your book can no longer be trusted until it’s recovered. — Same as detecting a missed webhook by a hole in event IDs, and the response is the same shape: backfill or resync. — Feed handlers treat a gap as an emergency: many strategies pull their quotes until the book is proven correct again.
- A/B feed arbitration — the exchange transmits the identical feed on two independent network paths (A and B); you listen to both, take whichever copy of each sequence number lands first, and use the other side to plug gaps. — It’s at-least-once delivery built from two unreliable channels plus dedup by idempotency key — the key being the sequence number. You’ve built exactly this discipline around webhook retries; here the “retry” is a redundant simulcast, because there’s no time to ask for a resend. — Bonus: taking the first arrival of each message shaves latency, since the faster path wins message by message.
Contrast with your crypto reality: a WebSocket book feed over TCP gives you ordering and retransmission for free — at the cost of TCP’s latency behaviors (the TCP/UDP chapter). Tradfi feeds choose the opposite trade: raw speed, and push reliability up to your application. That choice is why the transport and NIC chapters (ch02, ch05) exist.
- Multicast vs unicast — unicast is one sender to one receiver (every TCP connection, every HTTP call you’ve ever made); multicast is the sender transmitting once to a group address, and the network switches replicating the packet to every subscribed port in hardware. — It’s Redis pub/sub semantics, except no broker process exists: the fanout is done by the switch’s silicon, so the exchange sends each update exactly once whether 5 or 500 firms listen. — Every subscriber hears the message at nearly the same moment — a fairness property regulators care about — and it’s UDP underneath: no delivery guarantee, hence everything in the previous paragraphs. The packet-path and transport chapters (ch01, ch02) build on this.
3. FIX: the stateful session protocol
FIX SESSION (both sides persist counters across the wire)
YOU (seq out: 47) BROKER/EXCHANGE (seq out: 92)
│ │
│── Logon (my next out = 47) ────────►│
│◄────── Logon (my next out = 92) ────│
│── Heartbeat ──────────────────────► │ ...every N seconds...
│── NewOrderSingle seq 47 ─────► │
│◄───── ExecutionReport seq 92 ───────│
│── NewOrderSingle seq 48 ─────► │
│ ⚡ crash. restart. ⚡ │
│── Logon (my next out = 1) ─────────►│ ← WRONG: they expected 49
│◄──── ResendRequest / reject ────────│ session refuses to proceed
│ │
resume correctly = come back at 49,
answer their ResendRequest with either
real retransmits or a GapFill
- FIX (Financial Information eXchange) protocol — the decades-old standard wire protocol for order entry between firms, brokers, and exchanges: tag=value pairs (
35=Dmeans “new order”) over TCP. — Think “the SWIFT/ISO 8583 of trading”: ancient, verbose, and absolutely everywhere; every tradfi counterparty speaks it. — It splits into two layers, and the split is the important idea: - FIX session layer — the bookkeeping layer: logon/logout, heartbeats, and a sequence number on every message in each direction, persisted by both sides, surviving disconnects. — This is a Stripe-style event cursor made bidirectional and mandatory: each party tracks “the next number I’ll send” and “the next number I expect,” and a reconnect must resume exactly where it left off — like resuming a webhook stream from your stored cursor, except your counterparty also keeps a cursor on you. — The session layer is what makes FIX reliable over plain TCP: nothing is lost silently, because a hole in the numbers is detected immediately.
- FIX application layer — the business messages riding on top:
NewOrderSingle,ExecutionReport(fills),OrderCancelRequest. — These map one-to-one onto your matching-engine API surface. — Session mechanics are identical across counterparties; application dialects vary per venue. - Resend request / gap fill — on detecting a gap, a side sends
ResendRequest(from, to); the other side retransmits, except messages that shouldn’t be re-executed (heartbeats, and often stale orders) are replaced by aSequenceReset-GapFill— “pretend numbers 48–52 were administrative, skip ahead.” — This is your webhook backfill endpoint, plus a tombstone mechanism for events that must not be redelivered. — The dangerous part: retransmitted orders. A naive resend ofNewOrderSingleseq 48 after a crash could place a duplicate order with real money; FIX marks retransmitsPossDupFlag=Yand well-built engines treat them idempotently — the same reason you put idempotency keys on payment captures. - Why a restart is dangerous — sequence continuity is the session. Restart with the wrong counters and the counterparty either rejects your logon or fires resend traffic at you during the most fragile moment you have; meanwhile your orders may still be live at the exchange with nobody watching them. — It’s a stateful PSP integration where both sides track a message counter: you cannot “just reconnect,” you must resume at the agreed number or negotiate a reset. — Hence trading systems persist FIX sequence numbers with the same care you persist payment state, and “how do you recover a FIX session?” is a standard interview probe. The NIC chapter’s session-recovery material (ch05) and the event-sourcing chapter (ch13) both lean on this.
Your crypto reality, for contrast: WebSocket + REST (Representational State Transfer) + API keys. Disconnect → reconnect → re-authenticate → re-subscribe → re-snapshot everything, because the server keeps no cursor for you and guarantees no sequence continuity across connections. FIX’s statefulness is the price of never having to ask “wait, which of my orders are actually live?” — the question every crypto bot answers with a frantic REST burst after each reconnect.
4. Colocation: distance is time
RETAIL PATH COLOCATED PATH
your server (cloud, ~km away) your rack, INSIDE the
│ internet, ~ms exchange's datacenter
▼ │ "cross-connect": one
exchange DC ──► matching engine │ physical fiber patch
│ cable to their switch
▼ ~µs, fixed, no hops
matching engine
physics: light in fiber ≈ 200 km per millisecond (one way)
Singapore ↔ Tokyo ≈ 5,300 km straight-line, more over real cable
⇒ ~30+ ms each way, ~70 ms RTT — no software can fix geography
- Colocation (“colo”) — renting rack space in the exchange’s own datacenter and running your trading servers there. — It’s CDN-edge thinking applied to order flow: move the compute to where the event happens, because the speed of light is a hard budget. But where a CDN chases tens of milliseconds for humans, colo chases microseconds for machines. — At price-time priority (§1), the firm 5,300 km away has lost every race before it starts; you can measure this yourself — ping your Tokyo VM from Singapore and you’ll see ~70 ms RTT (round-trip time), which is mostly just fiber distance at 200 km/ms plus routing. That’s four orders of magnitude larger than the entire tick-to-trade budget of a colocated system.
- Cross-connect — the literal physical fiber cable patched from your colo rack to the exchange’s switch, ordered from the datacenter like a work ticket. — Think of it as a dedicated private link that replaces “the internet” entirely — no routers, no peering, no variance; the closest thing in your world is a VPC peering, made physical. — Latency becomes a fixed, tiny, known number, and some venues even normalize cable lengths so no rack gets a geometric advantage. Lab I (ch06) mirrors this setup at hobby scale.
5. Tick data and kdb+
- Tick data — the complete record of every market event — every trade, every quote change — at full resolution, timestamped; “tick” is tradfi for “one market data event.” — It’s your append-only events table for the market itself; a liquid symbol produces millions of rows a day, so the store lives at billions of rows. — Research, backtesting, and best-execution compliance all query it, and its natural shape drives the storage choice:
- Columnar tick store — storing each column (time, price, size…) contiguously rather than row by row, because analytical queries touch few columns across huge time ranges. — You know this trade-off from ClickHouse/QuestDB vs Postgres: scans over one column of a billion rows want columnar layout and vectorized execution. — Time-series market queries (“volume-weighted average price of every 1-minute window last quarter”) are the canonical columnar workload.
- kdb+ — the columnar, in-memory-plus-on-disk time-series database that dominates tradfi tick storage, programmed in q, a terse array language descended from APL (A Programming Language — yes, really). — Mental model: ClickHouse/QuestDB, but 25 years older, frequently faster on this exact workload, closed-source, expensive, and with a language where a production query can be 40 characters of punctuation. — It owns the niche because it was there first, banks standardized on it, and array-language operations map perfectly onto “fold over a billion ticks”; expect it named in tradfi job specs, and expect QuestDB/ClickHouse as its modern challengers.
6. Tick-to-trade: the metric the whole book optimizes
wire in wire out
│ ▲
▼ │
┌──────┐ ┌────────┐ ┌───────────┐ ┌──────────┐ ┌─────┴────┐
│ NIC │──►│ decode │──►│ update │──►│ strategy │──►│ encode + │
│ RX │ │ feed │ │ book │ │ decision │ │ NIC TX │
└──────┘ └────────┘ └───────────┘ └──────────┘ └──────────┘
└───────────────── tick-to-trade ───────────────────────┘
measured wire-to-wire (hardware timestamps; see the clocks
and latency-methodology chapters)
software systems: ~1–10 µs · FPGA systems: <1 µs
- Tick-to-trade — the elapsed time from a market data packet touching your NIC to your responding order leaving it, measured on the wire — not inside your process, where self-reported numbers flatter you. — It’s your end-to-end request latency SLO (service-level objective), except the clock starts at the network card, and it’s measured by hardware timestamping/tapping the wire rather than by APM spans. — This single number is the scoreboard for Parts I–II: Part I’s networking chapters attack the network legs, Part II’s measurement chapters make the number honest, and on the human scale from the measuring chapter, a 5 µs tick-to-trade is about 4 hours of single-cycle “seconds” — into which fits decoding, book update, decision, and encoding.
Part B — What’s under Postgres
You use Postgres the way you use Stripe: excellent command of the API, no need (until now) to know the machinery. The database and zero-downtime chapters (ch15, ch16) assume the machinery. Here it is: Postgres has been running your event-sourcing and double-entry-ledger tricks internally all along.
7. WAL: Postgres is event-sourced
COMMIT arrives
│
▼
1. append change record to WAL ──► fsync to disk ◄── THE durability moment
│ (sequential append: fast)
▼
2. tell client "committed"
│
▼
3. eventually, background writer updates the actual
table/index pages on disk (random writes: slow, unhurried)
crash between 2 and 3? → restart replays the WAL from the
last checkpoint; no committed data lost
- WAL (Write-Ahead Log) — an append-only file to which every change is written and fsynced before any table or index file is touched; commit = “it’s in the log,” and crash recovery = replay the log. — This is event sourcing, and you’ve built it twice: it’s your matching engine’s event journal, and it’s the payments append-only ledger — the log is the source of truth, tables are just a materialized projection maintained for convenient reads. — Sequential appends are the fastest thing a disk does, which is how Postgres commits fast while updating complex structures lazily. — Everything in the zero-downtime chapter (ch16) stands on this: replication is “ship the log,” PITR (point-in-time recovery) is “replay the log to timestamp T,” and CDC (change data capture) is “decode the log.” One structure, whole ecosystem.
8. MVCC: readers get a snapshot, and the corpses pile up
UPDATE accounts SET balance = 90 WHERE id = 7;
heap (table file):
┌──────────────────────────────────────────────────┐
│ row v1: id=7 balance=100 [xmin=500, xmax=612] │ ← old version stays!
│ row v2: id=7 balance=90 [xmin=612, xmax= ∅ ] │ ← new version appended
└──────────────────────────────────────────────────┘
txn 610 (started earlier) → still sees v1: its snapshot
txn 615 (started after) → sees v2
after nobody can see v1 → it's a DEAD TUPLE = bloat
VACUUM's job: reclaim it
- MVCC (Multi-Version Concurrency Control) — Postgres never updates a row in place; an
UPDATEwrites a new version and marks the old one as superseded, and every transaction reads the consistent snapshot of versions that were committed when it began — so readers never block writers and writers never block readers. — You’ve hand-rolled this pattern: an immutable ledger where “updating” a balance means appending a new entry, and a report reads “the ledger as of sequence N.” Postgres does it per row, with transaction IDs (xmin/xmaxabove) as the sequence numbers. — The costs fall out directly: dead row versions accumulate as bloat (tables physically larger than their live data), and VACUUM is the garbage collector that reclaims them. A long-running transaction pins an old snapshot, so VACUUM can’t clean anything newer — which is why one forgottenpsqlsession or stuck job can quietly balloon a busy table. The databases chapter (ch15) builds on this; it’s also why update-heavy trading schemas think hard before using Postgres for hot-path state.
9. Replication: shipping bytes vs shipping meaning
STREAMING (physical) LOGICAL
primary ──WAL bytes──► replica primary ──decode WAL──► row events
byte-identical copy "INSERT INTO orders VALUES(…)"
ALL databases, ALL tables │
same major version only ▼
replica is read-only any subscriber: newer-version PG,
~zero decode cost subset of tables, another system
- Streaming (physical) replication — the primary ships raw WAL bytes to replicas, which replay them continuously; the replica is a byte-for-byte identical copy of the entire cluster. — Analogy: restoring a binary disk snapshot, continuously — perfect fidelity, zero selectivity. — All-or-nothing and same-major-version-only (it’s literal page bytes), which is exactly why it cannot do a major-version upgrade — and that limitation is the setup for the zero-downtime chapter (ch16).
- Logical replication — the primary decodes the WAL back into row-level change events (“insert this row into
orders”) and publishes them; subscribers apply them as ordinary SQL. — This is a CDC stream — Debezium-into-Kafka energy — generated natively by Postgres from the same WAL. — Because subscribers apply meaning rather than bytes, they can be a different major version, take only some tables, or not be Postgres at all. — This is the mechanism behind zero-downtime major upgrades (the centerpiece of the zero-downtime chapter): stand up a new-version cluster, logically replicate until caught up, then switch traffic — the same expand/migrate/contract choreography you’d use to swap a payment provider without dropping transactions.
10. The lock queue: how a one-second migration takes the site down
The single most useful DBA fact for your interviews. The trap is not the lock — it’s the queue.
t=0 long analytics SELECT on orders [RUNNING, 40 min]
(holds ACCESS SHARE — the weakest lock)
t=10s ALTER TABLE orders ADD COLUMN risk_flag …
needs ACCESS EXCLUSIVE (conflicts with EVERYTHING,
even plain SELECTs) → must wait for the SELECT:
orders lock queue
┌───────────────────────────────┐
running → │ SELECT (analytics) │
waiting → │ ALTER TABLE ◄── would take 1s│
waiting → │ SELECT (app) ← blocked by │
waiting → │ INSERT (app) ← the WAITING │
waiting → │ SELECT (app) ← ALTER, not │
waiting → │ …every query… ← by the SELECT│
└───────────────────────────────┘
Locks queue FAIRLY: nobody may jump ahead of the
waiting ALTER. The whole app now waits on the
analytics query. Connections pile up. Site down.
- ACCESS EXCLUSIVE lock — the strongest table lock, required by most DDL (Data Definition Language —
ALTER TABLEand friends); it conflicts with every other use of the table, including reads. — It is a global read-and-write lock on the whole table: nothing touches it, not even a plain read, until the holder is done. — TheALTERitself is often metadata-only and takes a second — the lock, not the work, is the hazard. - The lock queue trap — Postgres grants locks in order: your DDL queues behind any long-running query, and because queueing is fair, every subsequent query queues behind your waiting DDL. A blocked one-second migration converts one slow analytics query into a full outage of the table. — It’s a head-of-line-blocking incident, the same shape as one stuck message freezing a FIFO queue — and you’ve likely felt this as “the deploy ran a migration and everything hung.” Now you know the mechanism. — The professional fix, verbatim for interviews:
SET lock_timeout = '2s';before DDL, so theALTERgives up rather than dam the queue, then retry in a loop at a quiet moment; plus the non-blocking variants —CREATE INDEX CONCURRENTLY, add columns without table rewrites,NOT VALIDconstraints validated later. The schema-evolution (ch14) and zero-downtime (ch16) chapters are applications of this one diagram.
11. Connection pooling: why pgbouncer exists
500 app connections pgbouncer Postgres
(each cheap to the app) ┌────────────┐ ┌────────────────┐
────────────────────────► │ multiplexer│───────►│ 25 backends │
────────────────────────► │ │───────►│ (each = a full │
────────────────────────► │ hands a │───────►│ OS PROCESS: │
… │ backend to │ │ MBs of memory,│
────────────────────────► │ whoever is │ │ fork cost, │
│ active NOW │ │ MVCC snapshot │
└────────────┘ │ bookkeeping) │
└────────────────┘
- Why Postgres connections are expensive — each connection is a forked operating-system process (not a thread, not a coroutine): megabytes of memory, real fork/teardown cost, and one more participant in shared bookkeeping (snapshots, locks) whose overhead grows with the crowd. — Node hands you sockets for near-free, so 500 idle connections feels normal; to Postgres, 500 processes is a genuine load before running a single query. — This is why every serious Postgres deployment fronts it with a pooler.
- pgbouncer — a small proxy that accepts thousands of cheap client connections and multiplexes them over a small pool of real Postgres connections. — You run this today: the
:5432on your multi-tenant GCP host is pgbouncer (500 client connections, 25 backends per database), with Postgres hidden on loopback:6432. — Two pooling modes, and the difference is exam material: - Session pooling — a client keeps one backend for its whole connection lifetime. — Like a dedicated phone line: safe, fully transparent, but a connected-and-idle client hogs a scarce backend. — Nothing breaks; little is saved.
- Transaction pooling — a client borrows a backend only for the duration of each transaction, then returns it; the next transaction may run on a different backend. — Like a stateless load balancer with no sticky sessions: massive multiplexing wins, but anything that assumes the same backend across transactions silently breaks. — The breakage list (know it cold): server-side prepared statements (
PREPARElives on backend A; your next transaction lands on B, which has never heard of it — the classic “prepared statement "s1" does not exist” error from ORMs), session state (SET,SET LOCALoutside the transaction, temp tables, session-scoped GUCs), advisory locks taken at session scope (the lock is held by a backend you no longer own — poisonous, since app-level advisory locking is a favorite pattern), andLISTEN/NOTIFY. — Trading and high-tenancy systems run transaction pooling for the multiplexing and design around the list; your own host is a working reference implementation to reason against when the databases chapter (ch15) discusses connection architecture.
What you can now read
- Event sourcing (ch13) — you now hold both halves: the matching-engine journal you built, and seeing that WAL is the same structure inside Postgres.
- Schema evolution (ch14) — the lock-queue diagram in §10 is the hazard that chapter’s every technique exists to avoid.
- Databases (ch15) — MVCC, VACUUM/bloat, WAL, and pooling modes are the assumed vocabulary; you have all four.
- Zero-downtime operations (ch16) — streaming-vs-logical replication (§9) is the entire mechanism; the chapter is choreography on top.
- Change management (ch17) and Lab: upgrade (ch18) — applications of §§9–10 with runbooks.
- TCP/UDP for trading (ch02) — the feed mechanics of §2 (multicast, sequence numbers, gaps, A/B arbitration) are the workload that chapter’s transport arguments are about.
- NIC internals (ch05) — §4’s cross-connects and §6’s wire-to-wire measurement are the context for why the NIC deserves its own chapter, and §3’s FIX session recovery is the state you’re protecting when hardware misbehaves.