The Life of a Packet: Where Microseconds Go

Before you start. This chapter assumes NIC (Network Interface Card — the hardware that turns wire signals into bytes in your RAM, ch00a), PCIe (the bus connecting the NIC to the CPU/RAM, ch00a), DMA (the card writing straight into RAM without the CPU copying, ch00b), descriptor rings (the shared circular queues the NIC and driver hand buffers through, ch00b), interrupts / IRQ / softirq (how hardware gets the kernel’s attention, and the deferred half of handling it, ch00b), and syscalls / context switches (crossing the kernel↔userspace boundary and its cost, ch00b). If any are new, read those first — 20 minutes there saves an hour here. For the whole path in web-dev terms, ch00c traces an Express handler down to the wire.

You’ve built trading systems that respond in ~10 milliseconds. HFT systems respond in ~1–10 microseconds — a 1,000–10,000x difference. Almost none of that gap is closed by writing faster application code. It’s closed by understanding, and then eliminating, what the operating system does to every packet. This chapter is the anatomy lesson: every stage a packet crosses between the wire and your recv() call, and what each stage costs.

Hold one number in your head throughout: a competitive HFT tick-to-trade budget (market data in → order out) is under 5µs in software, under 1µs with FPGAs. The default Linux network stack spends 2–10µs just delivering a packet to userspace. The kernel path alone blows the entire budget. That’s the “why” behind everything in Part I.

The map

 RX (wire → your code)
 ─────────────────────
        wire
         │  serialization + propagation
         ▼
 ┌───────────────┐
 │  PHY / MAC    │  decode symbols, check FCS
 └──────┬────────┘
        │  DMA write over PCIe
        ▼
 ┌───────────────┐
 │ NIC RX ring   │  descriptor ring in host RAM, pre-posted buffers
 └──────┬────────┘
        │  MSI-X interrupt (or: NAPI already polling)
        ▼
 ┌───────────────┐
 │ IRQ handler   │  hardirq: ack, schedule NAPI, return
 └──────┬────────┘
        │  softirq (NET_RX_SOFTIRQ)
        ▼
 ┌───────────────┐
 │ NAPI poll     │  driver pulls descriptors, builds sk_buffs
 └──────┬────────┘
        ▼
 ┌───────────────┐
 │ IP layer      │  netfilter hooks, routing lookup, defrag
 └──────┬────────┘
        ▼
 ┌───────────────┐
 │ TCP / UDP     │  4-tuple demux, TCP state machine, ACK logic
 └──────┬────────┘
        ▼
 ┌───────────────┐
 │ socket queue  │  sk_receive_queue, memory accounting
 └──────┬────────┘
        │  wake up sleeping task (epoll / blocking recv)
        ▼
 ┌───────────────┐
 │ scheduler     │  context switch onto your thread
 └──────┬────────┘
        │  syscall: recvmsg / recvfrom
        ▼
 ┌───────────────┐
 │ copy_to_user  │  kernel buffer → your buffer
 └──────┬────────┘
        ▼
      your code

Every box is work. Every arrow is a potential queue. Let’s walk it with numbers.

Stage by stage, with costs

Numbers below are order-of-magnitude for a modern x86 server (3–4 GHz, 10/25GbE NIC). Exact values vary; the ratios are what you need to internalize.

1. The wire: serialization and propagation

Before the host sees anything, physics taxes you:

Serialization: putting bits on the wire. A 64-byte frame (+ preamble/IFG ≈ 84 bytes on wire) takes ~67ns at 10GbE, ~672ns at 1GbE, ~27ns at 25GbE. A 1500-byte frame at 10GbE: ~1.2µs. This is one reason HFT messages are tiny and why 25G/40G links matter even at low utilization — you pay serialization per hop.
Propagation: ~5ns per meter in fiber (light travels at ~2/3 c in glass — ~200km per millisecond). Inside a colo hall this is tens of ns; across an ocean it’s tens of ms (the TCP/UDP chapter).
Switches: a cut-through switch adds ~300–500ns port-to-port; a store-and-forward switch must buffer the whole frame first (add serialization time again). HFT-specialized L1 switches (Arista 7130, formerly Metamako) do ~5–50ns.

2. PHY/MAC and NIC processing

The NIC’s PHY decodes line symbols, the MAC validates the FCS (Frame Check Sequence — the CRC checksum trailing every frame; a bad one means the frame is silently dropped), and internal NIC logic parses headers to pick an RX queue (RSS — Receive Side Scaling, hashing flows across several queues so several cores can work in parallel; the NIC internals chapter). Budget ~300ns–1µs inside the NIC depending on the part. Low-latency NICs (Solarflare/AMD X2/X3, Mellanox/NVIDIA ConnectX) are engineered to keep this in the low hundreds of ns; commodity NICs and especially cloud virtual NICs are worse.

3. DMA to the RX ring

The driver has pre-posted descriptors — pointers to empty buffers in host RAM — into a circular descriptor ring. The NIC DMAs the frame into the next buffer over PCIe, then writes back the descriptor to mark it filled.

PCIe Gen3/4 posted write (“posted” = fire-and-forget, the card doesn’t wait for an acknowledgement): the data lands in RAM (or L3 directly, via Intel DDIO — Data Direct I/O) in ~300–900ns.
DDIO matters: without it, your first touch of the packet is a DRAM miss; with it, the payload is already in L3.

Up to here everything is hardware and unavoidable (bypass or not). Wire-to-RAM is roughly ~1µs. Everything after this point is what kernel bypass deletes.

4. Interrupt vs. NAPI

The NIC raises an MSI-X interrupt (Message Signaled Interrupts — the PCIe-native way for a card to poke a specific CPU core, ch00b) to say “descriptors ready.” Cost:

Interrupt delivery + hardirq entry (hardirq = the immediate, drop-everything half of interrupt handling, ch00b): ~1–2µs by the time your handler runs — pipeline flush, mode switch, vector dispatch. On a busy or power-managed core, worse.
Linux uses NAPI: the hardirq handler just disables further interrupts on that queue and schedules NET_RX_SOFTIRQ. A softirq is deferred kernel work run right after the interrupt with interrupts re-enabled — the kernel’s “bottom half”, conceptually a setImmediate for the expensive part of the job. The softirq then polls the ring, harvesting up to a budget (default 64 per NAPI instance, 300 globally per softirq round) of packets per pass. Under load, interrupts stay disabled and the kernel polls continuously — interrupt-per-packet would melt at 10G line rate (~14.8M packets/s at min frame size).
Softirq runs either on the tail of the hardirq or in ksoftirqd (a kernel thread) if load is high — and ksoftirqd is scheduled like any other thread, which adds scheduling jitter measured in tens of µs to ms in the worst case. This is a classic tail-latency source.

NAPI means the kernel already switches between interrupt-driven (idle, latency = IRQ cost) and polling (busy, latency = poll interval) modes. Kernel bypass just says: poll always, from userspace, on a dedicated core.

5. Driver + sk_buff allocation

The NAPI poll loop reads each filled descriptor and wraps the buffer in an sk_buff — the kernel’s ~200+ byte per-packet metadata object, roughly the framework Request object your web server allocates per HTTP request. Cost per packet: ~200–500ns for allocation (from the slab allocator — the kernel’s object pool, usually recycled per-CPU), initialization, and header parsing. GRO (Generic Receive Offload) may coalesce consecutive TCP segments into one super-skb here — great for throughput, adds latency (the kernel-tuning chapter).

6. IP layer and netfilter

ip_rcv(): sanity checks, then the netfilter hooks at PREROUTING and INPUT — the kernel’s packet-filtering framework, where iptables/nftables rules hang; think globally-registered Express middleware that every single request must walk, and even with empty rulesets the hook infrastructure costs. With conntrack loaded (connection tracking, the stateful table behind NAT and -m state rules), add a per-packet lookup: ~100ns–1µs and a lock hazard. Then a route lookup in the FIB (Forwarding Information Base, the kernel’s routing table — fast, tens of ns, cached) and possible defragmentation. Budget ~200ns–1µs. Rule of thumb from HFT ops: never load conntrack/iptables NAT on a trading box.

7. Transport: TCP/UDP demux and processing

UDP: hash the 4-tuple (source IP, source port, destination IP, destination port — the four fields that identify one flow), find the socket, charge the packet to the socket’s receive buffer accounting (SO_RCVBUF), append to sk_receive_queue. ~200–500ns.
TCP: much heavier — sequence number validation, reassembly ordering, ACK generation, RTT sampling, congestion window update, timestamp option processing. ~1–2µs per segment is typical. This is why market data is UDP (the TCP/UDP chapter).

8. Socket queue → wakeup → schedule

The packet is queued on the socket. If your thread is blocked in recv()/epoll_wait() (epoll — the readiness API that libuv/Node’s event loop sits on, ch00c), the kernel must wake it:

wake_up_interruptible → scheduler enqueue → IPI (Inter-Processor Interrupt — one core poking another) to the target CPU if it’s idle/elsewhere → context switch into your thread. Cost: ~1–5µs, and highly variable. If the core was in a deep C-state (a CPU sleep level — deeper sleep, slower wake, ch00a), add tens of µs for wakeup (the kernel-tuning chapter). If another thread was running, add a full context switch (~1–3µs direct cost) plus indirect cost: your cache and TLB state has been trampled.
This wakeup is usually the single largest and most variable RX cost. It’s why serious latency work starts with “never sleep”: busy-poll the socket and the wakeup disappears.

 blocked receiver                   spinning receiver
 ────────────────                   ─────────────────
 packet hits socket queue           packet hits socket queue
   → wake_up_interruptible            → next iteration of your
   → scheduler enqueue                  loop simply reads it
   → IPI to the target core
   → context switch in              ≈ tens of ns, ~zero variance
   → caches/TLB now cold              (cost: the core is 100% busy)
 ≈ 1–5µs, spiky (µs–ms tail)

9. Syscall + copy to userspace

Your recvfrom()/recvmsg():

Syscall entry/exit: ~50–100ns bare; ~100–250ns+ with Spectre/Meltdown mitigations (KPTI, retpolines, IBRS — CPU-vulnerability workarounds that make every kernel entry more expensive). Measure on your hardware — Lab I does exactly this.
copy_to_user (the kernel’s checked memcpy across the kernel/userspace boundary): from the sk_buff to your buffer. ~50–100ns for a 200-byte market data message (a few cachelines), ~200–400ns for a full 1500-byte frame. Small in isolation; at millions of packets/second it’s a core’s worth of memcpy.
Then sk_buff free, socket accounting, return.

The RX bill

Stage	Typical cost	Avoidable?
Serialization (64B @10G)	~70ns	No (physics)
Switch hop (cut-through)	~300–500ns	Buy better switch (~5ns L1)
NIC internal + DMA over PCIe	~500ns–1µs	No (both paths pay it)
IRQ + softirq dispatch	~1–3µs	Yes — poll instead
Driver + sk_buff	~200–500ns	Yes — bypass
IP + netfilter	~200ns–1µs	Yes — bypass
UDP processing	~200–500ns	Yes — bypass
Wakeup + context switch	~1–5µs (spiky)	Yes — busy-poll
Syscall + copy	~150–500ns	Yes — mapped rings
Kernel total (wire→app)	~2–10µs typical, ms-tail	This is the target

A tuned kernel-bypass path (DPDK, ef_vi — the two main bypass frameworks, covered in the bypass-landscape chapter) delivers wire-to-app in ~1–2µs, dominated by the unavoidable hardware stages — the gap the rest of Part I closes.

The TX path (briefly)

Roughly the mirror image: send() syscall → copy_from_user into an sk_buff → TCP/UDP header build (TCP: congestion/flow-control gating — your send can be queued, not sent) → IP + netfilter OUTPUT → the qdisc (queueing discipline — the kernel’s per-interface outbound packet scheduler, default fq_codel/pfifo_fast; another queue, another lock). From there, the driver’s ndo_start_xmit posts a descriptor to the TX ring and writes the doorbell — one PCIe register write telling the card “new descriptors are waiting,” literally ringing a bell (~100–300ns, posted). The NIC then DMAs the frame and serializes it. App-to-wire through the kernel: ~2–5µs. Bypass TX: ~700ns–1.5µs, mostly the PCIe doorbell + DMA fetch.

TX has its own trap: the qdisc layer and TX ring are shared, so an unrelated bulk transfer (log shipping!) can queue ahead of your order. HFT practice: dedicated NIC (or at minimum a dedicated TX queue via XPS — Transmit Packet Steering, per-CPU choice of TX queue, the kernel-tuning chapter) for order traffic, and never share it with anything.

Interrupt coalescing: the throughput–latency dial

NICs batch interrupts: “fire at most every N µs, or after M packets” (ethtool -c/-C, rx-usecs/rx-frames). Defaults often sit at tens of µs or “adaptive” — tuned for throughput per CPU cycle, not latency.

 rx-usecs = 50            rx-usecs = 0
 pkt ─┐                   pkt ── IRQ ── delivered   (+~2µs)
 pkt ─┤ (waiting...)      pkt ── IRQ ── delivered   (+~2µs)
 pkt ─┴── IRQ ── batch    ...
      └─ up to 50µs added    more IRQs, more CPU,
         to first packet     less latency

Throughput box: coalesce aggressively — fewer interrupts, better cache behavior, higher packets-per-core.
Latency box: rx-usecs 0 (or 1), adaptive moderation off. You accept a CPU-usage hit for determinism.
The mirror concept in bypass land: there are no interrupts at all; a core spins at 100% polling the ring. Coalescing is the kernel-world compromise; busy-polling is the refusal to compromise.

Why HFT cares: the three hidden killers

Beyond the per-stage bill, three systemic effects dominate tail latency — and in trading, the tail is where you get picked off:

Context switches. Direct cost ~1–3µs, but the bigger cost is cache/TLB pollution: post-switch, your hot path runs at DRAM speed (~100ns/miss × hundreds of misses) until caches re-warm. A 1µs hot path can take 10µs+ on its first post-switch execution. Hence: pinned threads, isolated cores, nothing else schedulable there.
Softirq interference. NET_RX_SOFTIRQ runs on whichever core took the interrupt. If that’s your trading core, your book-building thread gets preempted mid-update by someone else’s packets. Hence: IRQ affinity steering (the kernel-tuning chapter).
Cache pollution from the stack itself. Every kernel-processed packet drags sk_buffs, socket structs, netfilter tables, and kernel text through L1/L2 on your core. The stack doesn’t just take time — it evicts your data. Bypass paths keep the packet in a userspace-mapped ring your code touches directly, often still warm in L3 via DDIO.

The framing to carry into interviews: the kernel network stack is a general-purpose, multi-tenant, throughput-oriented system. It must be fair, secure, and correct for thousands of flows. You want a single-purpose, single-tenant, latency-oriented path for a handful of flows. Everything from kernel tuning to NIC internals (ch03–ch05) is either shrinking the general-purpose path (tuning) or replacing it (bypass).

Plain-English recap

If you remember nothing else from this chapter:

The kernel network path is a deep middleware stack that every packet must walk — like an Express app where a dozen globally-registered middlewares run on every request whether you need them or not. Each is cheap; the stack of them costs 2–10µs, which is the entire HFT budget.
The NIC and driver talk through a shared job queue (the descriptor ring): the driver pre-posts empty buffers like a worker pre-registering “I can take work here,” and the card DMAs packets into them without CPU involvement — a producer writing directly into your queue’s memory.
Interrupts vs. polling is webhooks vs. polling loops. An interrupt (webhook) is great when idle but costs µs of dispatch per event; a poll loop notices new work in tens of ns. NAPI is a consumer that starts on webhooks and switches to draining the queue when busy. Bypass just says: poll always.
The single biggest, spikiest cost is waking a sleeping thread (~1–5µs, sometimes tens of µs) — the cold-start problem. A blocked recv() is a Lambda that scaled to zero; a busy-polling core is a warm instance that answers instantly but bills 100% CPU.
Syscalls and copies are like JSON.parse per message: ~100–250ns each looks free, but at millions of packets/second it’s a whole core doing overhead instead of work.
Interrupt coalescing is webhook batching: deliver every 50µs in a batch and throughput improves — but the first event of a burst waits, and in trading the first event is the one that pays.
Tail latency comes from state you lost, not work you did: a context switch trashes your caches the way restarting a Node process throws away its JIT-warmed state — the next request runs slow even though “nothing changed.”

Interviewer will ask

“Walk me through what happens when a packet arrives, from the wire to your application.” Tell it as three acts. Act one — hardware gets it into RAM (~1µs, unavoidable): the PHY/MAC decode the frame and check the FCS, then the NIC DMAs it over PCIe into a buffer the driver pre-posted on the descriptor ring — with DDIO it lands already in L3. Act two — the kernel processes it: the NIC raises an MSI-X interrupt; the hardirq does almost nothing except schedule the NET_RX softirq; the softirq polls the ring under a budget — that’s NAPI, interrupt-driven when idle, polling when busy — and wraps each buffer in an sk_buff. Then the middleware stack: IP layer and netfilter hooks, transport demux on the 4-tuple, onto the socket’s receive queue. Each of those stages exists so the kernel can be correct and fair for thousands of flows it knows nothing about. Act three — the kernel hands it over: wake the sleeping task, context-switch onto it, and your recvmsg copies the bytes to userspace. That handover is the biggest, spikiest line on the bill — ~1–5µs by itself. Total: 2–10µs wire-to-app, which alone blows a sub-5µs tick-to-trade budget — and that’s the whole case for Part I.

“Where does most of the latency actually go?” Not in any single stage — in the transitions: interrupt dispatch (~1–3µs) and especially the scheduler wakeup (~1–5µs, spiky, worse with C-states). Per-stage protocol processing is hundreds of ns each. So the biggest wins are “never sleep, never switch”: busy-polling and core isolation, before any bypass.

“Why is polling better than interrupts for latency? Isn’t polling wasteful?” Interrupt cost is ~1–3µs of dispatch plus scheduling variance; a poll loop notices a new descriptor in ~tens of ns with zero variance. Yes, you burn a core at 100%. In HFT the core is worth far less than the microseconds — a core costs a few hundred dollars; a microsecond of edge is revenue. Also note Linux itself agrees: NAPI converges to polling under load, and SO_BUSY_POLL exists precisely to let sockets poll.

“What’s an sk_buff and why do bypass frameworks avoid it?” It’s the kernel’s per-packet Request object — the ~200+ byte metadata struct a web framework allocates for every HTTP request, here tracking buffer geometry, headers, protocol state, and refcounts. The kernel needs that generality because any packet might be any protocol for any socket. But generality is billed per packet: ~200–500ns to allocate, initialize, and free, plus the cachelines it drags through L1 on your hot core. A bypass path serves one app on one known flow, so it can replace the Request object with a pre-allocated, fixed-layout buffer — DPDK’s mbuf, ef_vi’s raw buffers — no allocation, no refcounts, no kernel bookkeeping. At 1M packets/s, that ~300ns of per-packet object overhead alone is ~30% of a core.

“What does interrupt coalescing do, and how would you set it on a trading box?” NIC delays interrupts to batch packets (rx-usecs/rx-frames). Default/adaptive settings add up to tens of µs to the first packet of a burst — and the first packet is the one that moves markets. On a latency box: ethtool -C eth0 adaptive-rx off rx-usecs 0 (or 1), accept the interrupt-rate cost, or better, make it moot by busy-polling.

“How much does a syscall cost, and why does it matter if it’s only ~100ns?” It’s the JSON.parse-per-message problem: the cost is small, but it’s charged per message, and rate multiplies it. A syscall is ~50–100ns bare, ~100–250ns with the Spectre/Meltdown mitigations on. Every received message costs at least one recvmsg — and a naive event loop pays epoll_wait too. So run the chapter’s arithmetic: 150ns × 1,000,000 packets/s = 150ms of CPU per second — 15% of a core spent just crossing the kernel boundary, before any copy or actual work. That’s why the endgame is a ring mapped into userspace, where receiving a packet involves no syscall at all.

“Your p50 is fine but p99.9 is 100x worse. Name likely causes in the stack.” p50 is the happy path through the map; p99.9 is the same path after something made it sleep, queue, or go cold — the generator for every suspect I’d name. Slept: the core was in a deep C-state and paid tens of µs waking, or the receiver was blocked and paid the full wakeup-plus-context-switch. Queued: softirq work got pushed to ksoftirqd, which is scheduled like any other thread, or the interrupt-coalescing timer held a lone packet waiting for a batch that never came. Went cold: a context switch trashed cache and TLB, so the first post-switch run of a 1µs hot path executes at DRAM speed — 10µs+. Tail debugging is hunting for whoever put a sleep, a queue, or a cache miss in the path; later chapters add more suspects to each verb (TCP retransmits in the TCP/UDP chapter, memory effects in Part I’s labs), and every knob in the kernel-tuning chapter is the removal of one of these.

The HFT Gap Closer