From an Express Handler to the Wire

This is the bridge chapter. Take the thing you know best — a request arriving at a handler — and ride it downward through every layer from the last two chapters, until we hit photons. At each level: what you always thought happened, what actually happens, what it costs, and what an HFT engineer does about it.

The full elevator shaft first. We’ll stop at every floor.

 LEVEL 1   app.post('/order', handler)          ← your mental model ends here
 LEVEL 2   event loop / epoll                   ← the runtime's secret life
 LEVEL 3   socket API + syscalls                ← the wall from the kernel chapter
 LEVEL 4   kernel TCP/IP stack                  ← sk_buffs, protocol layers
 LEVEL 5   driver + NIC descriptor rings        ← the shared queue
 LEVEL 6   PCIe + DMA                           ← the highway from the machine chapter
 LEVEL 7   the wire                             ← physics

A note before we descend: everything below Level 1 is symmetric. A request arriving climbs up 7→1; your response leaving rides down 1→7. We’ll narrate the receive direction because that’s where trading systems bleed.

Level 1 — your handler

   app.post('/order', async (req, res) => {
     const order = validate(req.body);        //  ← "my code starts here"
     await book.execute(order);
     res.json({ status: 'accepted' });        //  ← "my code ends here"
   });

What you always thought: requests arrive, my function runs, latency is my function’s fault. Which was true at your scale — when the handler does 2 ms of database work, the transport is rounding error.

What actually happens: by the time your handler’s first line runs, the request has already crossed six layers, been copied 2–3 times, triggered an interrupt, woken a process, and passed through a scheduler. Your function is the visible tenth of the iceberg.

What it costs: the sub-basement adds ~2–10 µs before line one of your code — invisible under a 2 ms handler, unacceptable when the whole job must finish in 5 µs.

What HFT does: inverts the ratio. The business logic (parse tick → update book → decide → emit order) is often only ~1 µs of real work, so the transport layers dominate, and that’s why this entire book exists. The handler stops being “where latency lives” and becomes the cheap part.

Level 2 — the event loop and epoll

        ┌────────────────── node / tokio runtime ─────────────────┐
        │                                                         │
        │   loop {                                                │
        │     events = epoll_wait(epfd, ...);   ← ONE syscall,    │
        │                 │                       sleeps until    │
        │                 │                       any socket fires│
        │     for ev in events {                                  │
        │        read the socket, parse HTTP,                     │
        │        invoke YOUR callback           ← level 1 lives   │
        │     }                                   inside this loop│
        │   }                                                     │
        └─────────────────────────────────────────────────────────┘

What you always thought: “Node is event-driven; libuv handles the sockets; callbacks fire when data arrives.” True — and you correctly never asked how a sleeping process finds out data arrived.

What actually happens: the runtime parks in epoll — a Linux facility where you register thousands of sockets once, then make a single epoll_wait syscall that blocks until any of them has data, returning a batch of ready ones. This is the entire trick behind “Node scales to 10k connections on one thread”: one blocked thread, one syscall, N sockets. (kqueue on macOS, IOCP on Windows — same idea.) The part nobody told you: blocks until means your process is descheduled — off the CPU entirely — and when a packet finally arrives, the kernel must mark the socket ready, find your process, and get the scheduler to put it back on a core.

What it costs: that wake-up-and-reschedule dance is ~1–5 µs, plus — remember the kernel chapter — your thread comes back to a possibly cold cache and possibly a different core. Cheap per event amortized across 10k idle connections; brutal as a fixed tax on the one connection you care about.

What HFT does: refuses to sleep. The hot thread never parks in epoll_wait — it spins, checking for data in a tight loop, burning a whole core at 100% forever so there is nothing to wake up. Scale-to-zero versus provisioned concurrency, taken to the physical limit. More on this trade at the end of the chapter.

Level 3 — the socket API: crossing the wall

   runtime calls: read(fd, buf, 65536)
        │
        ▼            ring 3 ──► ring 0 mode switch      (~50–100 ns)
   ═════╪═══════════ THE WALL ═══════════════════════
        ▼
   kernel: find socket for fd
           copy waiting bytes: kernel sk_buff ──► your buf   (~0.2–1 µs)
           free the sk_buff
        │
        ▼            ring 0 ──► ring 3
   returns: "here are 143 bytes"

What you always thought: nothing — socket.read() was a function like any other.

What actually happens: every socket operation is a syscall through the wall from the kernel chapter, and — the detail that matters — the data is copied: packet bytes already sitting in kernel memory (inside an sk_buff) are duplicated into your process’s buffer, because the wall means kernel memory and your memory are disjoint worlds.

What it costs: ~50–100 ns for the crossing plus ~0.2–1 µs for the copy and socket bookkeeping. Per packet — and market data is millions of packets.

What HFT does: attacks both halves. The copy → zero-copy techniques: arrange for packet buffers to be readable by your process directly, so you get a pointer to where the bytes already are instead of a duplicate (same instinct as sendfile, or streaming a request body instead of buffering it). The crossing → amortize it (io_uring: submit and reap I/O through queues in shared memory, syscalls optional) or delete it outright (kernel bypass, next levels).

Level 4 — the kernel TCP/IP stack

   sk_buff climbing the stack (receive direction):

   driver hands over raw frame
        │
        ▼
   ┌ Ethernet ┐  is this MAC address mine? strip header
        ▼
   ┌ IP ──────┐  is this IP mine? checksum, firewall/netfilter
        ▼        hooks, routing decision, reassembly if fragmented
   ┌ TCP ─────┐  which connection? (hash lookup) · in order?
        ▼        · update windows · schedule ACK · or:
   ┌ UDP ─────┐  which socket? checksum. done. (this is why
        ▼        market data ships over UDP)
   append to socket receive queue ──► wake epoll (level 2)

What you always thought: “TCP (Transmission Control Protocol) guarantees ordered, reliable delivery” — a property of the network, roughly like gravity.

What actually happens: TCP is not a thing the network does; it’s a large kernel program that runs per packet on your CPU. Reliability means the kernel keeps a copy of everything sent until acknowledged, tracks sequence numbers, detects gaps, retransmits, manages congestion windows — a full bookkeeping department, run in softirq context (kernel chapter), with state lookups and locks per packet. UDP (User Datagram Protocol) by contrast is barely a protocol: checksum, find socket, deliver — which is why exchanges ship market data over UDP multicast and let applications handle gaps (you’ll meet gap-fill and recovery channels in the market-data chapters; the reliability bookkeeping moves up a layer — it never disappears).

What it costs: ~1–2 µs per packet through the full TCP path; UDP a fraction of that.

What HFT does: UDP wherever the venue allows; where TCP is mandatory (order entry at many venues), a stripped userspace TCP stack — Onload and friends reimplement TCP outside the kernel with none of the general-purpose baggage — plus knobs like TCP_NODELAY (disable Nagle’s batching of small writes; batching is throughput’s friend and latency’s enemy — the same reason you flush a webhook immediately instead of coalescing).

Level 5 — driver and NIC rings

   ┌── NIC ──────────────┐        RX descriptor ring (RAM)
   │ frame arrives,      │        ┌────┬────┬────┬────┬────┐
   │ RSS hash picks a    │──DMA──►│full│full│full│free│free│
   │ queue               │        └──▲─┴────┴──▲─┴────┴────┘
   └─────────────────────┘   driver─┘          └─NIC's write cursor
                             harvests filled slots (in softirq/NAPI),
                             builds sk_buffs, re-posts free buffers

What you always thought: the driver shuttles frames between the NIC and the kernel — true, but you never had reason to watch how.

What actually happens: the descriptor ring from the kernel chapter, live. The driver pre-posts empty buffers; the NIC fills them and advances its cursor; NAPI polling harvests them in batches. One detail is new and important: RSS (Receive Side Scaling) — the NIC hashes each packet’s addresses/ports to pick among multiple RX rings, each ring owned by a different core. That’s consistent-hash sharding of inbound traffic across consumers, in silicon — one flow always lands on the same core (ordering preserved per flow), load spreads across cores.

What it costs: the harvest-and-rebuild machinery is part of the ~1–2 µs softirq bill from level 4; the dangerous cost is drops — ring overflows when draining lags a burst, and market data bursts precisely when it matters.

What HFT does: steering, then seizure. First steering: use RSS/flow-steering rules so the feed you care about lands on a dedicated ring whose interrupts target a dedicated core — traffic-shaped, isolated, like a priority queue with its own worker pool. Then seizure: kernel-bypass frameworks map the rings directly into your process — your spinning thread (level 2’s refusal to sleep) polls the descriptor ring itself, no driver, no sk_buff, no syscall. Levels 2 through 5 collapse into “read the next slot of a shared-memory SPSC queue,” which — note — is a thing you already know how to do.

Level 6 — PCIe and DMA

   NIC                          PCIe                          RAM
   ┌─────────┐   write burst: packet bytes    ┌──────────────────┐
   │ frame in │ ════════════════════════════► │ posted buffer    │
   │ SRAM     │   write: descriptor update    │ descriptor slot  │
   └─────────┘ ════════════════════════════► └──────────────────┘
                 (~500 ns – 1 µs for the round trip;
                  writes can land ahead of the doorbell — ordering rules
                  on this bus are their own dark art)

What you always thought: the network card is “attached to” the computer, prepositionally.

What actually happens: the NIC is a peer on the PCIe network (machine chapter) doing DMA writes into RAM — packet bytes first, then the descriptor status flip that announces them. The CPU is not involved; the first CPU-visible evidence of the packet is a changed word in RAM.

What it costs: ~500 ns–1 µs of PCIe transit — and here’s the sobering part: this is the floor. Once you’ve deleted the kernel (levels 3–5) and the sleep (level 2), PCIe transit is most of what remains — and unlike the rows above it, there is no software trick that deletes it.

What HFT does: shaves and relocates. Shaving: cache-coherent tricks so polling the descriptor is cheap, write-combining for the transmit direction, NUMA-local buffers (machine chapter) so DMA lands next to the consuming core. Relocating: if PCIe transit to the CPU is the floor, move the decision onto the card — FPGA NICs that parse the feed and fire the response from the device itself, never crossing PCIe at all. That’s the endgame chapter, and now you know exactly which cost it exists to delete.

Level 7 — the wire

   NIC serializes the frame:  100 bytes = 800 bits
   at 10 Gbps: one bit every 0.1 ns  →  80 ns to put the frame on the wire

   then propagation: light in fiber ≈ 5 ns per metre
      across the colo hall: ~50–200 ns
      Chicago → New York:   ~4,000,000 ns one way. no appeal.

What you always thought: the network is “fast” and its speed is someone else’s SLA (service-level agreement).

What actually happens: two separate physical costs. Serialization — payload size ÷ link speed; the time to spell out the bits. Propagation — distance ÷ two-thirds the speed of light (light travels slower in glass); the time for the first bit to arrive. Nothing negotiates with either.

What it costs: serialization: 80 ns per 100 bytes at 10 Gbps (halved at 25 Gbps — this is why link speed matters even at tiny message sizes). Propagation: 5 ns per metre, forever.

What HFT does: buys distance. Colocation — your server in the exchange’s own data center; cross-connects — the literal shortest fiber the facility will sell between your cage and the matching engine, priced per metre because metres are nanoseconds; microwave links between cities (air beats glass — light in air is ~50% faster than in fiber). It’s CDN edge logic driven to its physical conclusion: when you can’t make the machine faster, move the machine. And per-message: smaller encodings serialize faster — one reason binary formats like SBE beat text protocols (a message half as long hits the wire in half the time).

The bill, itemized

One 100-byte market-data packet, 10 Gbps link, well-configured stock Linux, process parked in epoll. Where the microseconds actually go:

That range — spanning good and unlucky runs — is the “kernel path costs ~2–10 µs per packet” figure Part I asserts on its first page. Now it’s not an assertion; it’s rows 5–9 of a table you can audit. And notice the shape of the fix: rows 5–9 are all software convention, not physics. Kernel bypass replaces them with “spinning thread reads the descriptor ring”: the same packet reaches your code in ~1–1.5 µs, i.e. rows 1–4 plus a ring read. Rows 1–4 are physics and PCIe — that’s the floor FPGAs exist to dig under.

epoll vs blocking recv vs busy-polling

Three ways for code to consume a socket — you’ve lived the first two without naming them. In event-loop terms:

 BLOCKING recv         one thread per connection; each thread sleeps in
                       recv() until its socket has data
                       = classic thread-per-request servers (pre-Node)
                       wake cost per event: ~1–5 µs · burns: a thread each

 EPOLL                 one thread sleeps in epoll_wait() on N sockets;
                       kernel wakes it with a batch of ready ones
                       = the Node/libuv/tokio model — YOUR model
                       wake cost per event: ~1–5 µs, amortized · burns: ~nothing

 BUSY-POLL             one PINNED thread, never sleeps:
                       loop { if (ring/socket has data) handle() }
                       = while(true) queue.tryPop() — no subscription,
                         no notification, no wake-up, ever
                       cost per event: ~0.05–0.5 µs · burns: 100% of a core

The progression is a single trade surfacing three times: notification is cheap when idle and expensive when it fires; polling is expensive always and cheap when it matters. Blocking recv and epoll are push — webhook-style — and pay the wake-up tax (scheduler + cold cache, the largest single row in the table above) on exactly the packet you care about. Busy-polling is pull at maximum aggression: the core is always on, always warm, already looking. Your world optimizes for efficiency at rest because idle servers are wasted money; HFT optimizes for readiness at the burst because a core held spinning in reserve is ammunition — the fight arrives with no warning and is over in microseconds. (Linux offers a halfway house, SO_BUSY_POLL — the kernel busy-polls the driver ring for you while you wait — but serious hot paths go all the way: pinned core, spinning loop, kernel evicted.)

What you can now read

• Part I in full, starting from its opening premise — the ~2–10 µs kernel-path figure is now row-by-row auditable, and each Part I chapter is a named attack on specific rows: kernel bypass (rows 5–9), zero-copy (row 9), busy-polling (row 8), NIC/IRQ tuning (rows 5–6).
• The kernel-bypass framework chapters (DPDK, Onload/ef_vi, AF_XDP) — you can now place each on the elevator shaft: which levels it deletes, which it keeps, who drains the ring.
• The market-data chapters — UDP multicast feeds, gap detection, recovery channels: level 4 explained why reliability moved out of the kernel and into your feed handler.
• The colo and time-sync chapters — level 7’s serialization/propagation split is the vocabulary they assume.
• The Jargon Decoder (next) — every term this part introduced, plus the rest of the book’s vocabulary, in one lookup table.