TCP & UDP for Trading

Before you start. Two ideas from earlier carry this whole chapter: a socket (one network connection, handed to your code as a number — the thing net.createServer() or fetch() gives you, ch00b), and the kernel’s socket buffers (small queues the kernel holds your bytes in between the wire and your recv(), ch00b). That’s it. Everything else gets built up slowly below.

The one mental model: mail vs a phone call

There are two ways to move bytes across a network. Everything in this chapter is a consequence of this one choice.

 TCP  =  a phone call                 UDP  =  dropping postcards in a mailbox
 ─────────────────────                ──────────────────────────────────────
 • you dial, they pick up             • you write a card, drop it, walk away
   (a connection is established)       (no connection, no handshake)
 • words arrive in order              • cards may arrive out of order
 • nothing is lost — if they          • some cards get lost, and nobody
   didn't hear you, you repeat          tells you
 • but if the line crackles, you      • but one lost card never holds up
   both wait until it's clear           the next card

TCP — the reliable one. A connection is set up first (like dialing and waiting for “hello”), then every byte you send arrives, in order, exactly once. If a piece goes missing, TCP quietly re-sends it before giving you anything after it. You already trust TCP for everything: every HTTP request, every database query, every fetch() is TCP underneath.
UDP — the fire-and-forget one. You hand the kernel a small message (a datagram — one self-contained packet, like one postcard) and it’s flung at the destination. No connection, no ordering promise, no “did it arrive?” Some get lost. The upside: because there’s no ordering promise, one lost message never delays the next one.

Hold that last line. It’s the whole reason trading uses both.

The trading split, in one sentence

Prices come OUT of an exchange over UDP. Orders go IN over TCP.

Why the split? Because the two directions want opposite things when something goes wrong.

Prices: the exchange is firehosing book updates to hundreds of firms. If one update gets lost, you do not want everything behind it to freeze while the exchange re-sends the old one — by the time it arrives it’s stale anyway. You’d rather have the newest price now and patch the one gap separately. That’s UDP’s “one lost card doesn’t block the next” behaviour — exactly what you want.
Orders: you are sending “buy 100” and “cancel that”. Here, losing a message, or getting it out of order (a cancel landing before the order it was cancelling), is a disaster. You’d happily wait an extra millisecond to be sure it arrives correctly. That’s TCP.

Your payments instinct already knows this: a live price ticker is like an analytics event stream — drop one, show the next, who cares. An order is like a charge request — it must be exactly-once and correctly ordered, and you’ll wait to guarantee that.

Part 1 — Prices: the UDP side

Why one-lost-card-doesn’t-block-the-next matters so much

Concretely:

 TCP price feed (bad idea):    packet 4 is lost
   ...③ ④✗ ⑤ ⑥ ⑦...   → TCP holds ⑤⑥⑦ HOSTAGE until ④ is re-sent
                          you're now looking at a frozen, stale book

 UDP price feed (good idea):   packet 4 is lost
   ...③  ⑤ ⑥ ⑦...    → ⑤⑥⑦ arrive NOW; you notice ④ is missing
                          and go get it separately (next section)

With TCP, one lost packet stalls everything after it until the re-send completes — milliseconds of staleness on a feed where microseconds matter. This is called head-of-line blocking (the item at the front of the line holds up everyone behind it — like one stuck request blocking a queue). UDP has none of it: each message stands alone.

“But UDP loses messages — how do you not lose data?”

Fair question: the exchange numbers every message (1, 2, 3, 4, …). You watch the numbers. If you were expecting #1004 and #1005 shows up, you know you missed one — and now you can go get it.

 expected: 1004
 arrives:  1005   →  GAP. you missed 1004. two options to recover ↓

This is exactly the pattern you use in payments: webhooks are “at-least-once”, so you put a sequence number or cursor on each event, and if you notice a jump you call the “list events since X” endpoint to backfill. Same idea, different words.

The recovery options, cheapest first:

The backup feed — the exchange sends the same numbered messages twice, over two different network paths (call them feed A and feed B). If A drops #1004, B almost certainly has it. You listen to both. (More on this in a second — it’s clever.)
A “resend me #1004” request — a slower side channel where you ask for a specific missing range. Rate-limited, milliseconds.
The snapshot channel — a separate stream that broadcasts the entire current order book every few seconds. If you’re badly behind, you wait for the next full snapshot, throw away your stale book, rebuild from the snapshot, and resume. This is the exact same thing crypto venues do: the REST “order book snapshot” endpoint plus the WebSocket “diff” stream. You’ve built this consumer already — only the transport underneath is different.

While you have a gap and haven’t recovered yet, your book is stale — you stop quoting on it. A junior forgets that step; a senior says it unprompted.

The backup-feed trick (A/B feeds)

The exchange publishes every message twice, on two independent network paths. You subscribe to both and, for each message number, use whichever copy arrives first and ignore the second.

 feed A: ①  ②  ③  ④   ⑤
 feed B:  ①  ②  ✗  ④  ⑤        (B lost #3)
 you:    ①  ②  ③  ④  ⑤        (took #3 from A, everything else from whoever was first)
         └─ result: faster (min of two paths) AND survives loss on either path

Two wins at once: you get the faster of the two paths every time, and you only lose data if both paths drop the same message (rare). It’s the same move as sending a critical webhook through two providers and deduping by idempotency key — belt and suspenders.

The crypto reality is different — and this is your edge

Everything above (UDP, one-send-many-receivers, A/B feeds) is the traditional-finance world. Crypto venues mostly don’t do it — they send prices over a separate WebSocket connection to each client (TCP, one per customer). That changes the game, and knowing why is an interview point:

Traditional: one UDP broadcast, the network hardware copies it to everyone at the same instant. Fair and cheap no matter how many subscribers.
Crypto: the venue maintains a separate TCP/WebSocket connection per client. 10,000 clients = 10,000 sends per update. Now a slow client is the venue’s problem — its connection backs up, and the venue has to decide whether to buffer, skip, or disconnect it. (You’ll build exactly this in the venue chapters — it’s the “slow consumer problem”, ch25.)

That’s the whole reason crypto feeds feel different from tradfi feeds: broadcast vs. per-customer connections. You lived on the crypto side; the tradfi side is the mirror.

Part 2 — Orders: the TCP side

Orders go over TCP because correct and in-order beats fast. A late order is annoying; a lost or reordered order is a reconciliation incident. Order traffic is also low-volume compared to the price firehose, so TCP’s costs are easily affordable.

What rides on top of TCP is just a message format. You only need to recognize the names:

FIX — the old, universal one. Human-readable tag=value text (like a URL query string). Easy to read, slowish to parse.
Binary formats (Nasdaq’s “OUCH”, CME’s “iLink”) — fixed-layout binary, so reading a message is basically casting bytes to a struct. They exist purely to skip the text-parsing cost.
Your world — crypto venues use WebSocket/REST over TLS (i.e. TCP + encryption + a bit of framing). Structurally the same job as the tradfi session formats, plus encryption and JSON. When interviewing, translate out loud: “a venue’s WebSocket order channel is doing what OUCH-over-TCP does — a session with heartbeats and acks — just with TLS and JSON on top.”

The five TCP behaviours that actually bite

You do not need all of TCP. You need these five. Each gets a plain description, then the one-line fix.

1. The 40-millisecond stall (the famous one)

Two well-meaning “efficiency” features in TCP can lock together and freeze your small messages for ~40ms. Here’s the trap in plain terms:

TCP has a feature that says “this message is tiny — let me wait a moment in case more data is coming, so I can send it all together.” (Its name is Nagle’s algorithm.)
The receiving side has a feature that says “I just got data — let me wait a moment before acknowledging it, in case I’m about to send a reply I can piggyback the ack onto.” (Its name is delayed ACK.)

Now watch them deadlock: your side is waiting for an acknowledgement before sending the tiny order; their side is deliberately sitting on that acknowledgement. Both wait. The timer breaks the standoff after ~40ms.

 you:   "here's a 60-byte order" … (Nagle holds it, waiting for an ack)
 them:  (delayed-ack holds the ack, waiting for reply data to piggyback on)
        ⏳ … ~40ms … ⏳
 them:  timer fires → sends ack → your order finally goes. 40ms gone.

Fix: turn off the “wait to batch small messages” feature on every trading socket. In Rust it’s literally one line — stream.set_nodelay(true). The option’s name is TCP_NODELAY. There is never a reason to leave it on for trading. (This is the missing-database-index of network code: a one-line default nobody notices, costing 1000x. Lab I has you measure the 40ms yourself.)

Bonus rule that follows: build each logical message in one buffer and send it with one write(). Splitting a message into two writes (header, then body) re-invites the same class of problem.

2. Socket buffers (the kernel’s queues)

Each socket has a small kernel queue on each side — bytes wait there between your code and the wire. Two things to know:

Incoming prices (UDP): make the receive queue big. A burst of price updates in one busy microsecond can overflow a small queue, and overflow means silently dropped packets — which shows up later as mysterious feed gaps. Make it big, and monitor the overflow counter. This is the #1 real-world cause of “why did we gap?”
Outgoing orders (TCP): a huge send queue can hide a problem — your send() returns instantly while the data actually sits in the kernel aging. Some shops keep this queue small on purpose so backpressure is visible instead of hidden.

Analogy: it’s your job-queue depth. Too shallow and bursts overflow; too deep and you can’t see that you’re falling behind.

3. Congestion control — matters far, irrelevant near

TCP has logic that slows itself down when it thinks the network is congested. Whether this matters depends entirely on distance:

Same building as the exchange (tradfi colo, sub-millisecond): the network is private and never congested, so this logic never kicks in. Ignore it.
Across the world (crypto — Singapore to a venue in Tokyo, over the public internet): it matters a lot. A single lost packet makes TCP panic and throttle itself, and a stall can be hundreds of milliseconds. This is why serious crypto setups run several connections in parallel with failover, keep aggressive heartbeats, and place their servers in the same cloud region as the venue.

4. The idle-connection cold-start

One specific gotcha that makes a great interview answer: if an order connection sits idle for a bit and then you suddenly send a burst, that first burst can be weirdly slow. Reason: TCP “forgets” it had warmed up and resets itself to cautious-and-slow, exactly like a cold Lambda or a cold database connection.

Fix: keep the connection warm — send heartbeats so it’s never idle, and turn off the Linux setting that re-colds idle connections (tcp_slow_start_after_idle). “Keep it warm like a connection pool” is the whole idea.

5. Detecting a dead connection

TCP’s built-in “is the other side still there?” check defaults to two hours. Useless. In trading, a half-open connection — you think you’re connected and quoting, the venue thinks you’re gone — is how you end up with unhedged risk. So everyone relies on application-level heartbeats: FIX heartbeats, or WebSocket ping/pong, every 1–30 seconds with a hard timeout. If a few pings go unanswered, you assume dead and reconnect.

Part 3 — Crypto specifics (your interview home turf)

You lived here, so these are the points to make confidently.

Where the time actually goes on a crypto connection

Ordered biggest-to-smallest — reciting this order is the senior answer:

Distance (milliseconds). Light in fiber travels ~200km per millisecond, and nothing beats it. Singapore↔Tokyo is ~70ms round trip no matter how good your code is.
The venue’s own internal delay (milliseconds, worse during bursts) — their matching engine and gateways.
Reconnect handshakes (multiple round-trips) — only when a connection drops and has to re-establish TCP + encryption + WebSocket + re-auth. This is why you pre-warm backup connections.
JSON parsing (microseconds per message) — usually the biggest cost your own code pays, because most venues send JSON text, not binary. Serious shops use fast (SIMD) JSON parsers.
Your own stack (microseconds) — the lock-free, zero-allocation stuff from the rest of this book.

Here’s the real interview trap: when the venue is 70ms away, shaving 5µs off your own code is pointless for taker orders (you crossing the spread to hit a resting price). But it’s decisive for maker orders (you resting a quote and racing other bots to cancel/replace when the book moves) within the venue’s own region, where everyone’s a few microseconds apart. Knowing which race you’re in is the answer.

taker vs maker, since it’s load-bearing above: a taker hits an existing resting order (crosses the spread, pays the fee, gets filled now); a maker posts a resting quote and waits to be hit (earns the spread/rebate, risks being picked off). See ch00f.

Where crypto venues actually live

“Colo” in crypto means same cloud region as the venue:

Venue	Roughly lives in
Binance	AWS Tokyo
Bybit	AWS Singapore
Coinbase	AWS us-east-1 (Virginia)
Deribit	bare-metal, London

So cross-venue arbitrage (say Binance-in-Tokyo vs Coinbase-in-Virginia) has an irreducible ~150ms+ information gap between the two — the strategy has to absorb the delay that infrastructure can’t remove. Multi-region setups run a quoting engine in each venue’s region and reconcile global risk between them asynchronously.

The 60-second recap

Two ways to move bytes: TCP = a phone call (reliable, ordered, but a bad line makes you wait); UDP = postcards (some lost, out of order, but one lost card never blocks the next).
Prices go UDP, orders go TCP — because for prices you want “newest now, patch gaps later,” and for orders you want “correct and in-order even if slower.”
UDP doesn’t lose data in practice because every message is numbered; a jump in the numbers = a gap you go recover, from a backup feed / a resend request / a full snapshot. This is your payments webhook-with-a-cursor pattern.
A/B feeds = the same stream sent twice over two paths; take whichever’s first. Faster and loss-proof.
Crypto is different because it’s per-customer TCP/WebSocket instead of one UDP broadcast — which is why crypto has the “slow consumer” problem and tradfi doesn’t.
The 40ms stall is the one TCP bug to know cold: two batching features deadlock; the fix is TCP_NODELAY on every socket, always.
Distance is the budget everything lives inside: cross-region is ~70–240ms no matter what; know whether you’re in a cross-region taker race (milliseconds, your code barely matters) or an in-region maker race (microseconds, your code is everything).

Interviewer will ask

“Why is market data UDP but order entry TCP?” Prices: you want the newest update now, and one lost packet must not freeze everything behind it — so UDP, with message numbers and a snapshot channel to patch gaps. Orders: losing or reordering a cancel is a disaster, and order volume is low, so you pay for TCP’s reliability and ordering. One line: prices tolerate loss but not staleness; orders tolerate slowness but not loss.

“You see a gap in the message numbers — walk me through recovery.” First check the backup (B) feed — usually has it. Else request the missing range on the resend channel. Else join the snapshot channel, wait for a full book snapshot at or past the gap, rebuild, and resume — buffering newer updates meanwhile and discarding the ones the snapshot already covers. The whole time, the book is marked stale and I stop quoting on it. Same shape as recovering a crypto WebSocket book from the REST snapshot.

“What’s the 40-millisecond stall?” Two batching features deadlock: my side holds a tiny message waiting for an ack; their side delays the ack waiting for reply data to piggyback on. A timer breaks it after ~40ms. Fix: TCP_NODELAY on every trading socket, and one logical message per write(). I can measure it directly — Lab I does exactly that.

“What TCP tuning matters in colo vs cross-region crypto?” Colo: the LAN never congests and never loses packets, so the only enemies are TCP’s own batching timers and cold starts. Knobs: TCP_NODELAY, keep connections warm (pre-open, heartbeat, disable idle-cold-start), sane dead-peer detection. Cross-region: distance makes loss expensive — one lost packet stalls the stream for hundreds of ms of retransmit round trip — so the game is surviving loss, not shaving µs. Knobs: redundant parallel connections with failover, aggressive heartbeats for half-open detection, pre-warmed backups — and accept the speed-of-light floor by placing servers in the venue’s region.

“Where does the latency go on a crypto venue connection, in order?” Distance (ms) → the venue’s internal delay (ms) → reconnect handshakes (round-trips, only on drops) → JSON parsing (µs) → my own stack (µs). Optimize in that order. The exception: in-region maker races are decided at the µs tier, so there my stack is the whole game.

“Why do crypto feeds feel different from traditional exchange feeds?” Traditional = one UDP broadcast the network copies to everyone simultaneously (fair, scales for free). Crypto = a separate TCP/WebSocket to each client, so the venue does N sends per update and now has a slow-consumer problem — it must buffer, conflate, or drop for clients that can’t keep up. Different fairness and different failure modes fall right out of that one design difference.

The HFT Gap Closer