Lab I: Measuring the Stack You’re Bypassing

Before you start. This lab assumes syscalls and the vDSO (crossing into the kernel, and the kernel code mapped into your process that lets some calls avoid crossing at all, ch00b), TSC / rdtsc (the CPU’s cycle counter and the instruction that reads it, ch00e), percentiles, warmup, and benchmark hygiene (why p99.9 matters and means lie, ch00e), busy-polling vs. blocking (spin vs. sleep, in event-loop terms, ch00c), and the Nagle/delayed-ACK trap from chapter 2. If any are new, read those first — 20 minutes there saves an hour here.

The chapters before this lab (ch01–ch05) gave you numbers. Never quote a number you haven’t measured — this lab makes every headline claim reproducible on a stock Linux box: kernel RTT (round-trip time), the Nagle/delayed-ACK trap, syscall cost, the price of sleeping — and then two parts that were missing from this lab’s first edition: measuring what kernel tuning actually buys (Part F, the kernel-tuning chapter’s claims put on a scale) and running real kernel bypass (Part G, AF_XDP on a virtual wire). Parts A–D take under an hour; E–G are stretch goals worth a second session. Everything compiles with stable Rust. Run on Linux x86_64 (a cloud VM is fine — expect noisier tails, which is itself a lesson; the rdtsc part needs x86_64).

Run everything --release, and pin the process to quiet cores if you can (taskset pins a process to specific CPU cores — pinning matters because a thread that migrates mid-run lands on a core with cold caches, and that shows up as tail noise):

cargo new hft-lab1 && cd hft-lab1
# ... add files below ...
nproc                       # know your core count first
taskset -c 2,3 cargo run --release --bin udp_rtt    # 4+ cores
taskset -c 0,1 cargo run --release --bin udp_rtt    # 2-core VM: these are the cores you have

Know your box before you start. taskset -c 2,3 in the listings assumes 4+ cores — on a 2-vCPU cloud VM those cores don’t exist and taskset fails; substitute -c 0,1 (or -c 1 for single-core pins) throughout. Parts F and G want sudo and a few packages (sudo apt-get install -y stress-ng linux-cpupower; Part G’s extras are listed there). If your daily box is 2 vCPUs, Part F wants a throwaway 4-vCPU spot VM for an hour — separating “the load” from “the measured thread” needs cores to separate them onto.

Scaffold

Cargo.toml:

[package]
name = "hft-lab1"
version = "0.1.0"
edition = "2021"

[dependencies]
libc = "0.2"

# Part E (optional stretch) only; requires Linux 5.6+
[target.'cfg(target_os = "linux")'.dependencies]
io-uring = "0.7"
# Part G (optional stretch) only; see Part G's setup for system packages
xsk-rs = { version = "0.8", optional = true }

[features]
xdp = ["dep:xsk-rs"]

# Part G binary only builds when you ask for it — plain builds stay dependency-light
[[bin]]
name = "xdp_rx"
required-features = ["xdp"]

[profile.release]
opt-level = 3
lto = true
codegen-units = 1

src/lib.rs — one shared histogram reporter:

Two things in this file you won’t have met if you’ve only written safe Rust. First, a libc:: call. Picture Rust’s standard library the way you picture Node’s fs module: a friendly wrapper around the C functions the operating system actually speaks. When the wrapper doesn’t expose the thing you need, you go one floor down and call the C function yourself — libc::clock_gettime here is the same C function that sits under Node’s process.hrtime(). Second, an unsafe block. Rust’s compiler normally proves your memory access is sound before it lets you compile — but it can’t read C code, so for these calls unsafe is you signing the guarantee instead: “compiler, trust me on this one.” Both appear below, glossed where they land.

#![allow(unused)]
fn main() {
// src/lib.rs

// Sorts the samples and prints one row of percentiles (min/p50/p99/p99.9/max).
// Every part of the lab funnels its numbers through this.
pub fn report(name: &str, mut ns: Vec<u64>) {
    ns.sort_unstable();
    let n = ns.len();
    let pct = |p: f64| ns[((n as f64 * p) as usize).min(n - 1)];
    println!(
        "{:28} n={:<7} min={:>9} p50={:>9} p99={:>9} p99.9={:>9} max={:>9}",
        name, n, fmt(ns[0]), fmt(pct(0.50)), fmt(pct(0.99)),
        fmt(pct(0.999)), fmt(ns[n - 1])
    );
}

// Renders a nanosecond count as "850ns" / "12.3us" / "40.1ms" so columns stay readable.
fn fmt(ns: u64) -> String {
    if ns < 10_000 { format!("{}ns", ns) }
    else if ns < 10_000_000 { format!("{:.1}us", ns as f64 / 1e3) }
    else { format!("{:.1}ms", ns as f64 / 1e6) }
}

// Instant is a private stopwatch: each process starts it at its own zero, so
// a stamp from one process means nothing in another. CLOCK_MONOTONIC is one
// machine-wide clock — every process on the box reads the same one — which is
// why Parts F/G can stamp a packet in the sender process and subtract in the
// receiver process. (Returns nanoseconds.)
pub fn mono_ns() -> u64 {
    // An empty two-field form (whole seconds + leftover nanoseconds) that we hand
    // to the kernel to fill in. C functions return data by writing into memory
    // you provide — like passing an object for a callback to mutate.
    // (libc::timespec is that C struct.)
    let mut ts = libc::timespec { tv_sec: 0, tv_nsec: 0 };
    // `unsafe` = "compiler, trust me": Rust can't read C code, so it can't prove
    // clock_gettime writes only into our form and nothing else. We sign for it.
    unsafe { libc::clock_gettime(libc::CLOCK_MONOTONIC, &mut ts) };
    ts.tv_sec as u64 * 1_000_000_000 + ts.tv_nsec as u64
}
}

Percentiles, not averages: latency distributions are heavy-tailed, and the p99.9 column is where every kernel pathology from the packet-path chapter (ch01) shows up — wakeups, preemption, C-state exits, IRQs, page faults.

Part A — Baseline: UDP round-trip through the kernel

The whole journey first:

 THREAD A (userspace)  │            KERNEL            │  THREAD B (userspace)
 ── syscall boundary ──┤                              ├── syscall boundary ──
 1 a.send() ──────────►│ copy into B's socket queue   │
                       │ softirq: w1 wake B ─────────►│ (was parked in recv)
                       │ copy out on recv return ────►│ 2 b.recv() returns
                       │ copy into A's socket queue ◄─│─ 3 b.send()
 (parked in recv)      │ softirq: w2 wake A           │
 4 a.recv() returns ◄──│ copy out on recv return      │

 1 = a.send(), 3 = b.send() — syscalls: cross in, copy into the peer's queue
 2 = b.recv(), 4 = a.recv() — the syscalls each thread was parked inside
 w1, w2 = scheduler wakeups un-parking them → 4 syscalls + 2 wakeups per RTT

src/bin/udp_rtt.rs:

// Baseline: what does a full kernel round trip cost on loopback?
use std::net::UdpSocket;
use std::time::Instant;

const WARMUP: usize = 10_000;
const ITERS: usize = 100_000;

// Spawns an echo thread on socket b, then times WARMUP+ITERS blocking
// round trips from socket a and prints the percentile row.
fn main() {
    let a = UdpSocket::bind("127.0.0.1:0").unwrap();
    let b = UdpSocket::bind("127.0.0.1:0").unwrap();
    // connect() on UDP dials nothing — there's no handshake to perform. It just
    // saves the peer's address, like filling in a default "to:" field, so the
    // plain send()/recv() calls below don't need an address every time.
    a.connect(b.local_addr().unwrap()).unwrap();
    b.connect(a.local_addr().unwrap()).unwrap();

    // echo peer
    std::thread::spawn(move || {
        let mut buf = [0u8; 64];
        loop {
            let n = b.recv(&mut buf).unwrap();
            b.send(&buf[..n]).unwrap();
        }
    });

    let msg = [0u8; 32]; // tick-sized payload
    let mut buf = [0u8; 64];
    let mut samples = Vec::with_capacity(ITERS);
    for i in 0..WARMUP + ITERS {
        let t0 = Instant::now();
        a.send(&msg).unwrap();
        a.recv(&mut buf).unwrap();
        if i >= WARMUP {
            samples.push(t0.elapsed().as_nanos() as u64);
        }
    }
    hft_lab1::report("udp_rtt blocking loopback", samples);
}

One RTT = 4 syscalls + 2 loopback traversals + 2 scheduler wakeups (crossings 1–4, w1/w2 above). Loopback (127.0.0.1 — packets short-circuit inside the kernel and never touch a NIC) skips the NIC/DMA/IRQ hardware stages, so this measures the software stack — the part bypass deletes. Watch p50 vs p99.9 diverge; then re-run under load (stress-ng --cpu 4 elsewhere) and watch the tail explode — under load the echo thread queues up behind the stress-ng workers before the scheduler gets it back on a core, and that wait lands directly in your tail.

Part B — TCP_NODELAY A/B: catching Nagle in the act

The trap is a standoff between two timers. Time flows down:

 time  CLIENT userspace │ CLIENT KERNEL (Nagle)   ═ wire ═  SERVER KERNEL (delayed ACK)
  │     ── syscall boundary ──
  │    1 write 40B ────►│ nothing unACKed → out ──[40B]───► queued to app; server app
  │    2 write 60B ────►│ small + 40B still unACKed         reads 40/100 → no reply.
  │                     │ → Nagle HOLDS the 60B:            ACK owed for the 40B, but
  │                     │   "wait for the ACK"              the delayed-ACK timer HOLDS
  │     ~40ms           │        ▲                          it: "wait — I might piggy-
  │     deadlock        │        │  each side waits         back it on a reply"
  │     window          │        │  for the other           │
  ▼                     │        └──────[ACK]◄───────────── timer expires (~40ms)
       3 read returns ◄─│ ACK frees Nagle ──[60B]─────────► 100/100 → reply ─► client

 1 = c.write_all(&[1u8; 40]) — "header": sails through, no unACKed data yet
 2 = c.write_all(&[2u8; 60]) — "body": small write + unACKed data → Nagle queues it
 3 = c.read_exact(&mut resp) — completes only after the ~40ms timer breaks the tie

src/bin/tcp_nodelay.rs:

// The Nagle + delayed-ACK interaction (the TCP chapter), reproduced on demand.
// Run: cargo run --release --bin tcp_nodelay -- on
//      cargo run --release --bin tcp_nodelay -- off
use std::io::{Read, Write};
use std::net::{TcpListener, TcpStream};
use std::time::Instant;

const REQ: usize = 100;
const ITERS: usize = 200; // 40ms stalls make big runs slow

// Starts an echo server, then times ITERS request/response round trips where
// each request is deliberately split into two small writes — Nagle bait.
fn main() {
    let nodelay = std::env::args().nth(1).map(|s| s == "on").unwrap_or(true);

    let l = TcpListener::bind("127.0.0.1:0").unwrap();
    let addr = l.local_addr().unwrap();
    std::thread::spawn(move || {
        let (mut s, _) = l.accept().unwrap();
        s.set_nodelay(true).unwrap(); // server side kept sane
        let mut buf = [0u8; REQ];
        loop {
            if s.read_exact(&mut buf).is_err() { return; }
            s.write_all(&buf).unwrap(); // responds only after the FULL request
        }
    });

    let mut c = TcpStream::connect(addr).unwrap();
    c.set_nodelay(nodelay).unwrap();

    let mut resp = [0u8; REQ];
    let mut samples = Vec::with_capacity(ITERS);
    for _ in 0..ITERS {
        let t0 = Instant::now();
        // One logical request as TWO small writes: "header" then "body".
        c.write_all(&[1u8; 40]).unwrap();
        c.write_all(&[2u8; 60]).unwrap();
        c.read_exact(&mut resp).unwrap();
        samples.push(t0.elapsed().as_nanos() as u64);
    }
    hft_lab1::report(&format!("tcp rtt nodelay={nodelay}"), samples);
}

With nodelay=on the Nagle hold never happens and the standoff can’t form. Expect p50 to move from tens of µs to ~40ms — a 1000x regression from one missing setsockopt. Also internalize the second lesson: even with NODELAY, that’s two packets for one message — assemble one buffer, one write.

Part C — Syscall cost with rdtsc

The binary asks “what time is it?” three ways, and the only thing that differs is how far the question travels (journey 2 is the vDSO from ch00b):

               USERSPACE                      │   KERNEL
                             ── syscall boundary ──
 1 getpid ────────────────────────────────────┼──► run handler, come back
   ◄─────────────── ~100–250ns ───────────────┼────┘  (a full crossing)
 2 clock_gettime ──► ┌─────────────────────┐  │
   ◄──── ~20ns ───── │ vDSO clock page:    │◄─┼──── kernel refreshes the page
                     │ kernel data, mapped │  │     from its side
                     │ INSIDE your process │  │     (the call never crosses)
                     └─────────────────────┘  │
 3 rdtsc ── ~6–10ns ── one instruction: never leaves the core, let alone userspace

 1 = libc::syscall(SYS_getpid)             — a guaranteed kernel entry and exit
 2 = libc::clock_gettime(CLOCK_MONOTONIC)  — answered by the vDSO in userspace
 3 = _rdtsc()                              — reads the CPU's own counter register

src/bin/syscall_cost.rs (x86_64 only):

// What does crossing into the kernel cost, cycle-counted with rdtsc?
use std::time::Instant;

const N: u64 = 2_000_000;

// The odometer read. The TSC (timestamp counter) has been counting ticks since
// boot, and on modern CPUs it keeps a constant rate even when the core changes
// clock speed ("invariant TSC") — which upgrades it from rev counter to clock.
// (_rdtsc compiles to the single `rdtsc` instruction: ~6-10ns, no kernel involved.)
fn rdtsc() -> u64 {
    // `unsafe` = "compiler, trust me": this block does something Rust can't
    // check for you. Here it's harmless — we're just reading a counter the
    // CPU exposes.
    unsafe { core::arch::x86_64::_rdtsc() }
}

// Calibrating the odometer: nobody told us how fast it ticks, so we race it
// against ~200ms of wall clock and divide — ticks counted / seconds elapsed
// = the counter's rate in cycles per second (Hz).
fn tsc_hz() -> f64 {
    let t0 = Instant::now();
    let c0 = rdtsc();
    while t0.elapsed().as_millis() < 200 {}
    (rdtsc() - c0) as f64 / t0.elapsed().as_secs_f64()
}

// Warms up, runs f() N times inside one rdtsc bracket, prints avg cycles and ns/op.
fn bench(name: &str, hz: f64, mut f: impl FnMut()) {
    for _ in 0..10_000 { f(); } // warmup
    let c0 = rdtsc();
    for _ in 0..N { f(); }
    let cycles = (rdtsc() - c0) as f64 / N as f64;
    println!("{:22} {:>7.1} cycles  {:>7.1} ns/op", name, cycles, cycles / hz * 1e9);
}

// Calibrates the TSC, then cycle-counts three clocks: a real syscall,
// a vDSO call, and Rust's Instant::now.
fn main() {
    let hz = tsc_hz();
    println!("TSC ~{:.2} GHz", hz / 1e9);
    let mut ts = libc::timespec { tv_sec: 0, tv_nsec: 0 };

    // 1. A full crossing. Every kernel service has a number, and libc::syscall
    //    enters the kernel by that number with no library wrapper in between —
    //    a guaranteed entry and exit. (getpid as the probe: uncached by
    //    glibc since 2.25, so every call really crosses.)
    bench("getpid syscall", hz, || unsafe {
        libc::syscall(libc::SYS_getpid);
    });
    // 2. No crossing at all. Same C-function shape as a syscall, but the vDSO
    //    answers from a page the kernel keeps updated inside YOUR process —
    //    the question never crosses.
    bench("clock_gettime (vDSO)", hz, || unsafe {
        libc::clock_gettime(libc::CLOCK_MONOTONIC, &mut ts);
    });
    // 3. What your Rust code actually calls. black_box is a wall the optimizer
    //    can't see through: without it, the compiler notices nobody reads the
    //    Instant, deletes the loop, and you time an empty bracket.
    bench("Instant::now", hz, || {
        std::hint::black_box(Instant::now());
    });
}

Three lessons in one binary. (1) A genuine kernel entry costs ~100–250ns — how much depends on “mitigations”, the Spectre/Meltdown security patches that add extra work to every kernel entry; compare a box booted with mitigations=off to see their price. getpid is the probe precisely because glibc stopped caching it in 2.25 — every call genuinely crosses. (2) The vDSO means timestamping is not a syscall (~15–25ns), which is why you can afford to timestamp everything. (3) rdtsc itself (~6–10ns) is the only sane way to time nanosecond-scale operations — Instant::now inside the measured region would dominate it. One caveat rides along: the CPU can reorder rdtsc relative to nearby work; the clocks chapter (ch07) covers the fences.

Part D — The price of sleeping: blocking vs busy-poll vs spin

src/bin/busy_poll.rs:

The three receive modes first — all three answer the same question, how does a thread wait for a packet; they differ in who does the waiting and where. Blocking parks the thread: it tells the kernel “wake me when a packet arrives” and leaves the core entirely — free for other work — and when data lands the kernel must reschedule the thread and refill its caches. That wakeup is the microseconds this part measures. Busy-poll keeps the single blocking recv(), but on an empty queue the kernel itself polls the NIC driver’s ring for up to a bounded time before parking the thread — still one syscall; the waiting moves in-kernel, and any packet it finds is pulled through the stack right there instead of waiting for the interrupt path to deliver it. Spin makes the socket non-blocking and loops recv() from userspace: the waiting is your loop, one cheap syscall per check, and the thread never parks. The two delete different costs: spin deletes the wakeup; busy-poll deletes the interrupt-delivery leg while keeping parking as its fallback. They also compose — SO_BUSY_POLL on a non-blocking socket makes each spin-loop recv() poll the driver as well, deleting both costs at once; that combination on a pinned core is the strongest tuned-kernel receive short of bypass (inert on loopback like busy-poll alone, so measure it on the Part G veth or two hosts). Where each mode does its waiting:

            USERSPACE                │ syscall boundary │        KERNEL
 block   1 recv() ───────────────────┼─────────────────►│ empty → thread PARKED here
         (thread off the core)       │                  │ packet lands (softirq)
         3 recv() returns ◄──────────┼──── 2 WAKEUP ────│ reschedule, refill caches
                                     │  ▲ the expensive, spiky crossing — the tail
 busy-   1 recv() ───────────────────┼─────────────────►│ empty → kernel itself polls
 poll    2 recv() returns ◄──────────┼──────────────────│ the driver ring for ≤200µs
                                     │  one crossing; the waiting stays in-kernel
 spin    1 recv() → WouldBlock ◄────►│ cheap, immediate │ each call just peeks the
         2 recv() → WouldBlock ◄────►│ round trips      │ queue; the thread never
         n recv() → data ◄──────────►│                  │ parks, so nothing to wake

 block:    rx.recv() on a blocking socket — 2 is the µs-scale wakeup this part measures
 busypoll: set_busy_poll(&rx, 200), then the same rx.recv()
 spin:     rx.set_nonblocking(true); each WouldBlock loops via std::hint::spin_loop()

One new construct in this listing: SO_BUSY_POLL has no Rust wrapper, so we set it the C way, with setsockopt. A C API can’t see your types — C has no generics — so you hand it a raw pointer (“the data starts here”) and a byte count (“it runs this long”), and it takes your word for what’s there. That’s why the code below passes a pointer plus a size where Rust would normally pass a typed value.

// One-way latency into an IDLE receiver: blocked-and-woken vs spinning.
// Run: ... --bin busy_poll -- block | busypoll | spin
use std::net::UdpSocket;
use std::os::unix::io::AsRawFd;
use std::time::{Duration, Instant};

const ITERS: usize = 2_000;
const GAP: Duration = Duration::from_micros(500); // receiver idles between packets

// Tells the kernel: when a recv finds no data, keep checking the driver for up
// to `usec` microseconds before parking me. Warns (and continues) if refused.
fn set_busy_poll(s: &UdpSocket, usec: libc::c_int) {
    // `unsafe` = "compiler, trust me": Rust can't check a C function's paperwork.
    // We're vouching that the pointer and byte count below really describe one int.
    let r = unsafe {
        libc::setsockopt(
            // as_raw_fd(): underneath the Rust socket object sits a plain integer —
            // the number the kernel issued when the socket was opened (its
            // file descriptor). C APIs speak these numbers, not wrapper types.
            // SOL_SOCKET = "a socket-level option, not a TCP- or IP-level one".
            s.as_raw_fd(), libc::SOL_SOCKET, libc::SO_BUSY_POLL,
            // The pointer half — where the data starts:
            // &int → pointer-to-int → pointer-to-anything (void*). The casts
            // erase the type because C reads memory, not types...
            &usec as *const libc::c_int as *const libc::c_void,
            // ...and the length half — how many bytes sit there (socklen_t is
            // just C's name for "a length").
            std::mem::size_of::<libc::c_int>() as libc::socklen_t,
        )
    };
    if r != 0 {
        eprintln!("SO_BUSY_POLL failed: {} (older kernels want CAP_NET_ADMIN; try sudo)",
                  std::io::Error::last_os_error());
    }
}

// Sends a timestamped packet every 500µs from one thread and measures, in the
// chosen receive mode, how long each packet took to reach the receiver.
fn main() {
    let mode = std::env::args().nth(1).unwrap_or_else(|| "block".into());
    let rx = UdpSocket::bind("127.0.0.1:0").unwrap();
    let tx = UdpSocket::bind("127.0.0.1:0").unwrap();
    tx.connect(rx.local_addr().unwrap()).unwrap();

    match mode.as_str() {
        "busypoll" => set_busy_poll(&rx, 200),
        // Non-blocking recv never parks the thread. If the queue is empty it
        // returns instantly with WouldBlock — "nothing yet, ask again" — and
        // what happens next becomes OUR decision instead of the kernel's.
        "spin"     => rx.set_nonblocking(true).unwrap(),
        _          => {}
    }

    // Instant is Copy: each thread carries its own copy of the SAME stopwatch
    // start, so their elapsed() readings share one zero and can be subtracted.
    let epoch = Instant::now();
    let sender = std::thread::spawn(move || {
        for _ in 0..ITERS {
            std::thread::sleep(GAP);
            let t = epoch.elapsed().as_nanos() as u64;
            // to_le_bytes: the u64 as its 8 raw bytes, little-endian
            // (least-significant byte first) — our one-line wire format.
            tx.send(&t.to_le_bytes()).unwrap();
        }
    });

    let mut buf = [0u8; 8];
    let mut samples = Vec::with_capacity(ITERS);
    while samples.len() < ITERS {
        if mode == "spin" {
            loop {
                match rx.recv(&mut buf) {
                    Ok(_) => break,
                    // WouldBlock: "queue empty right now", delivered as an error
                    // value — not a failure, just the cue to ask again.
                    Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {
                        // We hold the core and ask "anything yet?" millions of
                        // times per second — polling instead of waiting to be
                        // woken. spin_loop() is the one courtesy: it tells the
                        // CPU "this is a poll loop", easing power. It never
                        // sleeps or yields.
                        std::hint::spin_loop();
                    }
                    Err(e) => panic!("{e}"),
                }
            }
        } else {
            rx.recv(&mut buf).unwrap();
        }
        let sent = u64::from_le_bytes(buf);
        samples.push(epoch.elapsed().as_nanos() as u64 - sent);
    }
    sender.join().unwrap();
    hft_lab1::report(&format!("one-way idle-rx [{mode}]"), samples);
}

Design notes, because they’re the transferable skill: the 500µs gap guarantees the receiver is idle when each packet lands — you’re measuring the wakeup path, which steady-throughput benchmarks hide. Both modes pay the same sender-side send() cost, so the delta between modes isolates the receive side. Honesty caveat you should repeat in interviews: on loopback there’s no NAPI driver to poll — NAPI being the kernel’s interrupt/poll hybrid for real NICs (ch00b) — so SO_BUSY_POLL is mostly inert here: expect busypoll ≈ block on localhost. The real kernel-busy-poll win only appears on a physical NIC; re-run with rx/tx split across two hosts, or across a veth pair (a virtual ethernet device pair), to see it. The spin mode is the one that shows the sleep tax on any box. Run block vs spin pinned to separate cores (taskset -c 2,3). And watch the units when you compare rows: report prints each value in its own unit — anything under 10µs comes out in ns — so a spin p50 of 4104ns is 4.1µs, beating a block p50 of 14.4us by ~3.5×, even though the digit string looks bigger.

Part E (stretch) — io_uring: fewer syscalls, same stack

The picture first. io_uring replaces the syscall-per-op pattern with two conveyor belts in memory both sides share: you place order forms on the submission queue (SQ) — “recv on this socket, into this buffer” — and the kernel drops finished-work receipts on the completion queue (CQ): “done, 8 bytes.”

 classic (Parts A–D): one crossing PER OP           N ops = N crossings
   USERSPACE                  │ syscall boundary │  KERNEL
   recv() ────────────────────┼─────────────────►│  do the op, come back
   recv() ────────────────────┼─────────────────►│  do the op, come back  ...

 io_uring: the rings straddle the boundary — shared memory both sides can touch
   USERSPACE                  │                  │  KERNEL
   1 push order form ─────► [ SQ ring — ON the boundary ] ◄─── kernel reads forms
   3 pop receipt ◄───────── [ CQ ring — ON the boundary ] ◄─── kernel writes receipts
   2 submit_and_wait ─────────┼─────────────────►│  drain SQ, run ops, fill CQ
                              │    N ops = 1 crossing  (SQPOLL: 0 — kernel polls SQ)

 1 = ring.submission().push(&e) — a plain memory write, no crossing
 2 = ring.submit_and_wait(1)    — the ONE syscall in the loop
 3 = ring.completion().next()   — a plain memory read, no crossing

src/bin/uring_recv.rs (Linux 5.6+):

// One submit_and_wait replaces the recv syscall-per-packet pattern.
use io_uring::{opcode, types, IoUring};
use std::net::UdpSocket;
use std::os::unix::io::AsRawFd;
use std::time::Instant;

// Sends 100k ticks to itself, receiving each via an io_uring submission
// instead of a recv() syscall, and reports the one-way latency distribution.
fn main() -> std::io::Result<()> {
    let rx = UdpSocket::bind("127.0.0.1:0")?;
    let tx = UdpSocket::bind("127.0.0.1:0")?;
    tx.connect(rx.local_addr()?)?;

    let mut ring = IoUring::new(8)?;
    let mut buf = [0u8; 64];
    let mut samples = Vec::with_capacity(100_000);

    for _ in 0..100_000 {
        // Fill out one order form for the submission belt: "recv on this fd,
        // into this buffer". user_data is your tag — it comes back stamped on
        // the receipt, so you can match receipts to forms when many are in flight.
        let e = opcode::Recv::new(types::Fd(rx.as_raw_fd()),
                                  buf.as_mut_ptr(), buf.len() as u32)
            .build().user_data(1);
        // `unsafe` = "compiler, trust me": the form carries a raw pointer to buf,
        // and Rust can't see when the kernel finishes writing there. We promise
        // not to move, reuse, or free buf until the receipt comes back.
        unsafe { ring.submission().push(&e).expect("sq full") };
        let t0 = Instant::now();
        tx.send(b"tick")?;
        ring.submit_and_wait(1)?; // ONE syscall: hand over the belt AND wait for a receipt
        let cqe = ring.completion().next().expect("cqe");
        assert!(cqe.result() > 0);
        samples.push(t0.elapsed().as_nanos() as u64);
    }
    hft_lab1::report("io_uring recv one-way", samples);
    Ok(())
}

As-is this shows parity, not victory — one op per submit means one syscall either way. The wins arrive when you go further (good exercises): multishot recv (opcode::RecvMulti + buffer rings — arm the receive once and completions keep flowing without resubmitting), batching N submissions per syscall, SQPOLL (a kernel thread polls your SQ — zero steady-state syscalls), and registered buffers (described to the kernel once, not re-validated per op). Remember the bypass-landscape framing (ch04): the packet still walks the whole kernel stack — io_uring economizes the doorway, not the hallway.

Part F — The tuning dividend: put the kernel-tuning chapter on a scale

Everything so far measured the stock kernel. The kernel-tuning chapter (ch03) claims its knobs buy you the tail — never quote that claim unmeasured either. No new Rust here: you re-run Part A and Part D’s binaries while turning ch03’s knobs one at a time, and watch which percentile moves.

First, discover what your box even allows — on a cloud VM, several knobs simply aren’t yours, and that observation is Part F’s first result (the NIC-internals chapter’s cloud lesson, ch05: the hypervisor owns the floor):

sudo apt-get install -y stress-ng linux-cpupower
sudo cpupower frequency-info 2>/dev/null | grep -iA1 governor   # governor visible? settable?
ls /dev/cpu_dma_latency 2>/dev/null || echo "no C-state control here (VM?)"
cat /sys/devices/system/cpu/cpu0/cpuidle/state*/name 2>/dev/null # which sleep states exist?

F1 — C-states and the governor (bare metal, or a VM that exposes them). Baseline: record Part D block mode’s p50/p99.9. Then cap sleep depth — holding /dev/cpu_dma_latency open with a zero written to it tells the kernel “no sleep state with more than 0µs wake latency” for as long as the file stays open (the file-as-lease mechanism from ch03):

sudo sh -c 'exec 3<>/dev/cpu_dma_latency; printf "\x00\x00\x00\x00" >&3; sleep infinity' &
CAP=$!
sudo cpupower frequency-set -g performance 2>/dev/null
taskset -c 2 cargo run --release --bin busy_poll -- block    # 2-core box: -c 1
kill $CAP    # releasing the file releases the C-state cap

Read the result like ch03 taught: p50 barely moves; p99.9 collapses. The median wakeup was already from a shallow state; the tail was the occasional deep-C6 exit (~40–130µs), and you just made deep sleep illegal. If block mode’s tail now approaches spin mode’s, you have measured exactly what spinning was buying you — and what a C-state cap buys instead, without burning the core.

F2 — Pinning and isolation under load (works on any box, including 2 vCPUs). The enemy here is the scheduler’s freedom to put the load where you are:

 run B — fence up (4+ cores shown; on 2 cores it's core 0 vs core 1):
   core 0        core 1     │ core boundary (taskset fence) │  core 2        core 3
   stress-ng     stress-ng  │  scheduler may not place      │  busy_poll      (idle)
   worker        worker     │  either side's threads        │  rx thread
   (--taskset 0,1)          │  across this line             │  (taskset -c 2)
 run A — no fence: same cores, boundary erased — load and receiver mix freely

A/B it:

# A: unpinned receiver, load everywhere — the scheduler mixes them freely
stress-ng --cpu $(nproc) --timeout 70 &
cargo run --release --bin busy_poll -- block

# B: same load, but fenced — load on core 0, receiver pinned to core 1
#    (4+ cores: stress on 0-1 with --taskset 0,1, lab pinned to 2)
stress-ng --cpu 1 --taskset 0 --timeout 70 &
taskset -c 1 cargo run --release --bin busy_poll -- block

Expected: run A’s p99.9 explodes (your receiver queues behind stress workers for whole scheduler timeslices — milliseconds); run B pulls the tail most of the way back to the quiet-box number, using nothing but placement. That is isolcpus in miniature: the boot flag makes this fencing permanent and kernel-enforced instead of per-command and advisory.

F3 — What loopback can’t show you. IRQ affinity, coalescing, GRO, flow steering — the NIC-side half of ch03 — are invisible here by construction: loopback has no NIC, no IRQs, no rings. Say that out loud in an interview when you present these numbers; the two-host version of this lab (real NIC, ethtool -C/-K, /proc/irq/*/smp_affinity) is where those knobs become measurable. A null result you can explain beats a positive one you can’t — same lesson as SO_BUSY_POLL in Part D.

Record everything in one table — stock / C-state-capped / pinned-under-load — per mode. The artifact you want at the end is one sentence with three numbers in it: “blocking receive went from p99.9 of X stock, to Y with sleep states capped, to Z when I fenced the load — the median never moved; tuning buys the tail.”

Part G (stretch) — Real bypass: AF_XDP on a virtual wire

io_uring economized the doorway. This part actually skips the hallway: an AF_XDP socket (ch04’s express chute) receiving raw frames into a UMEM your process owns — running against a veth pair, a virtual ethernet cable, so you can do it on any Linux box or VM without touching the interface your SSH session rides on. The A/B: the same UDP sender, received two ways — once through the kernel stack, once through the chute:

             ═══ wire (veth va) ═══   the same frame, two arms
   ARM A — kernel socket (veth_recv)  │  ARM B — AF_XDP (xdp_rx)
   KERNEL                             │  KERNEL (driver hook only)
   1 alloc sk_buff, copy frame in  ✂  │  1' XDP hook fires in the driver —
   2 walk the IP/UDP stack         ✂  │     before any sk_buff exists
   3 socket lookup, queue to rx    ✂  │  2' frame lands in a UMEM frame —
   4 wakeup: schedule the thread   ✂  │     YOUR memory, already mapped into
   ── syscall boundary ──             │     your process: nothing crosses
   5 recv() returns, copy to user     │  3' descriptor slip → RX ring
   USERSPACE                          │     (shared memory ON the boundary)
   6 payload handed to you,           │  USERSPACE
     already parsed by the kernel     │  4' harvest the slips off the ring
                                      │  5' read the frame in place
                                      │  6' parse eth/ip/udp yourself @42
   ✂ = stage arm B deletes            │  no sk_buff, no stack walk, no wakeup

 A: 1–4 happen behind the scenes; 5 = rx.recv(), the one call in veth_recv.rs
 B: 4' = rx_q.poll_and_consume, 5' = umem.data(d), 6' = the PAYLOAD_AT parse

Honesty up front: on veth there’s no real NIC, so AF_XDP runs in copy (“SKB”) mode — you get the full programming model (UMEM, fill/RX rings, frames-not-sockets, you-are-the-parser) and a real syscall/wakeup win, but not the DMA-into-your-memory zero-copy numbers a physical NIC gives. This is the flight simulator: every control is real, the physics are approximated.

Setup — the wire, and a room at the far end of it (a network namespace, so the kernel actually routes packets over the veth instead of short-circuiting via loopback):

# system packages Part G's build wants (bindgen + libxdp build chain):
sudo apt-get install -y clang llvm libelf-dev gcc make m4 pkg-config
# if the build still asks for libxdp explicitly: sudo apt-get install -y libxdp-dev

sudo ip link add va numrxqueues 1 numtxqueues 1 type veth peer name vb numrxqueues 1 numtxqueues 1
sudo ip netns add lab1
sudo ip link set vb netns lab1
sudo ip addr add 10.77.0.1/24 dev va && sudo ip link set va up
sudo ip netns exec lab1 ip addr add 10.77.0.2/24 dev vb
sudo ip netns exec lab1 ip link set vb up
sudo ip netns exec lab1 ip link set lo up
# Static ARP: once the XDP program owns va's queue it swallows EVERYTHING —
# including ARP requests — so the far side must not need to ask.
MAC_A=$(cat /sys/class/net/va/address)
sudo ip netns exec lab1 ip neigh replace 10.77.0.1 lladdr $MAC_A dev vb

src/bin/veth_send.rs — the constant across the A/B — a plain UDP sender stamping the cross-process monotonic clock:

// Run INSIDE the namespace: sudo ip netns exec lab1 ./target/release/veth_send
use std::net::UdpSocket;
use std::time::Duration;

// Sends 20k UDP packets across the veth, each carrying a mono_ns() timestamp
// the receiving process can diff against its own clock.
fn main() {
    let tx = UdpSocket::bind("10.77.0.2:0").unwrap();
    tx.connect("10.77.0.1:7777").unwrap();
    for _ in 0..20_000 {
        std::thread::sleep(Duration::from_micros(500)); // idle receiver, like Part D
        tx.send(&hft_lab1::mono_ns().to_le_bytes()).unwrap();
    }
}

src/bin/veth_recv.rs — arm A, the kernel-stack path:

// The control: same wire, same sender, ordinary blocking socket.
use std::net::UdpSocket;

// Receives 20k timestamped packets through the normal kernel stack and
// reports one-way latency (receive time minus the sender's stamp).
fn main() {
    let rx = UdpSocket::bind("10.77.0.1:7777").unwrap();
    let mut buf = [0u8; 8];
    let mut samples = Vec::with_capacity(20_000);
    while samples.len() < 20_000 {
        rx.recv(&mut buf).unwrap();
        let sent = u64::from_le_bytes(buf);   // reverse of to_le_bytes: 8 bytes -> u64
        // saturating_sub: clamp at 0 instead of wrapping if clocks disagree slightly.
        samples.push(hft_lab1::mono_ns().saturating_sub(sent));
    }
    hft_lab1::report("veth one-way, kernel socket", samples);
}

src/bin/xdp_rx.rs — arm B, the chute. Hold the whole mechanism as one picture before reading a line of it. You allocate one big slab of your own memory — the UMEM, think a single Buffer.alloc() done once at startup — and chop it into 4096 fixed-size frames: empty envelopes. Two shared queues connect you to the driver (each is a ring: a conveyor loop in memory both sides can see). On the fill ring you hand the driver your empty envelopes. When a packet arrives, the driver writes it straight into one of them and drops a slip in your tray — the RX ring. A slip (a descriptor) never carries the packet itself, only “envelope at offset N, M bytes used.”

         YOUR PROCESS                                DRIVER
     ┌───────────────────┐    fill ring          ┌─────────────┐
     │  UMEM: 4096       │ ──empty envelopes──►  │ packet in?  │
     │  fixed-size       │                       │ write it    │
     │  frames           │ ◄──slips: "envelope   │ into an     │
     │  (your memory)    │    N, M bytes"─────   │ envelope    │
     └───────────────────┘      RX ring          └─────────────┘
       read the payload in place, then return the
       envelope to the fill ring — the loop never allocates

The listing follows the hello_xdp example that ships with the xsk-rs crate (pinned at 0.8); if the API has drifted by the time you run this, the crate’s examples/ directory is the source of truth — the shape above is the lesson. Every unsafe in it means the same thing: the rings traffic in raw offsets into your UMEM, and it’s on you — not the compiler — to only hand over slips (descriptors) that really point at envelopes you own:

// Run as root: sudo ./target/release/xdp_rx va
// Build: cargo build --release --features xdp
use xsk_rs::{config::{SocketConfig, UmemConfig}, Socket, Umem};

const FRAMES: u32 = 4096;
const ITERS: usize = 20_000;
// eth(14) + ipv4(20) + udp(8): you own the protocol stack now — "parse UDP"
// is a pointer offset. This is ch04's "you rebuild the mailroom's services".
const PAYLOAD_AT: usize = 42;

// Attaches an AF_XDP socket to the interface, harvests 20k raw ethernet frames
// straight from the driver hook, parses the UDP payload itself, and reports latency.
fn main() {
    let iface = std::env::args().nth(1).unwrap_or_else(|| "va".into());

    // The envelope slab: one allocation, chopped into FRAMES envelopes, plus the
    // stack of slips (descs) that point into it. This memory is YOURS — packets
    // will land here without ever living in a kernel buffer.
    // (try_into().unwrap(): converts usize -> the exact integer type the API
    // wants, panicking only if it wouldn't fit — it always fits here.)
    let (umem, mut descs) =
        Umem::new(UmemConfig::default(), FRAMES.try_into().unwrap(), false).unwrap();

    // Socket on (interface, queue 0). fq is the fill ring from the diagram —
    // your empty-envelope belt. cq is its TX-side counterpart (the completion ring,
    // for sends; unused here). The ch04 ring pairs, live.
    let (_tx_q, mut rx_q, fq_and_cq) =
        Socket::new(SocketConfig::default(), &umem, &iface.parse().unwrap(), 0).unwrap();
    let (mut fq, _cq) = fq_and_cq.expect("fill/comp rings present when umem is unshared");

    // Hand the driver the entire stack of empty envelopes up front.
    // `unsafe` = "compiler, trust me": we vouch every slip points at a UMEM
    // envelope nobody else is using — ring math the compiler can't check.
    unsafe { fq.produce(&descs) };

    let mut samples = Vec::with_capacity(ITERS);
    while samples.len() < ITERS {
        // Check the tray (5ms timeout): take a batch of slips off the RX ring;
        // each one now points at an envelope the driver has filled.
        // (unsafe: descs must be scratch space we own for the slips to land in.)
        let n = unsafe { rx_q.poll_and_consume(&mut descs, 5).unwrap() };
        for d in &descs[..n] {
            // Open the envelope: a raw view into the UMEM at the slip's offset.
            // (unsafe: sound only because d is a slip the RX ring just handed
            // us — the compiler can't know that; we can.)
            let frame = unsafe { umem.data(d) };
            let bytes = frame.contents();
            if bytes.len() >= PAYLOAD_AT + 8 {
                let sent = u64::from_le_bytes(bytes[PAYLOAD_AT..PAYLOAD_AT + 8].try_into().unwrap());
                samples.push(hft_lab1::mono_ns().saturating_sub(sent));
            }
        }
        // Close the loop: the envelopes we just read go back on the fill ring,
        // empty again. This is the entire allocation story — there isn't one.
        unsafe { fq.produce(&descs[..n]) };
    }
    hft_lab1::report("veth one-way, AF_XDP (copy mode)", samples);
}

Run the A/B (build first: cargo build --release --features xdp):

# Arm A — kernel stack:
./target/release/veth_recv &
sudo ip netns exec lab1 ./target/release/veth_send
# Arm B — kill arm A first (the XDP program will steal its packets anyway):
sudo ./target/release/xdp_rx va &
sudo ip netns exec lab1 ./target/release/veth_send
# Teardown when done:
sudo ip netns del lab1 && sudo ip link del va 2>/dev/null

What to expect and how to read it: arm A lands near Part D’s block numbers (it is Part D over a virtual wire). Arm B typically lands in spin-mode territory with a flatter tail — it took the right-hand arm of the diagram, every ✂ stage gone. What you should narrate, though, is what your hands just did: posted empty frames to a fill ring, harvested raw ethernet off an RX ring, parsed UDP at byte 42 yourself, and recycled frames — that is the ch04 model executed, and it’s the same shape DPDK and ef_vi have. Copy mode on a veth is the mechanism without the magnitude; on a real NIC in zero-copy mode with a busy-polling core, this same code shape is the sub-2µs path — and you now know precisely which stages it deleted, because you measured them one at a time in Parts A–F.

Expected results

Reference: bare-metal-ish 3–4GHz x86_64, Linux 6.x, mitigations on, quiet cores. Cloud VMs: p50 similar-to-2x, tails 5–20x worse.

Measurement	Typical p50	Typical p99.9	What it proves
A: UDP RTT loopback (blocking)	8–25µs	30–200µs	Kernel software path alone ≫ a 5µs HFT budget (ch01)
B: TCP rtt, nodelay on	10–30µs	50–300µs	Healthy small-message TCP ≈ UDP + protocol overhead
B: TCP rtt, nodelay off	~40ms	~45ms+	Nagle × delayed-ACK = 1000x from one missing sockopt (ch02)
C: getpid syscall	100–250ns	—	Kernel entry cost; why per-packet syscalls add up (ch01)
C: clock_gettime vDSO	15–25ns	—	Timestamps are ~free → instrument everything
C: Instant::now	20–35ns	—	Rust’s clock = vDSO + small wrapper
D: one-way, `block`	4–15µs	20–100µs+	Wakeup + schedule dominates idle-receiver latency (ch01)
D: one-way, `spin`	1–4µs	5–20µs	Never sleeping removes the biggest, spikiest term (ch03)
D: one-way, `busypoll`	≈ block on loopback	—	SO_BUSY_POLL needs a real NAPI driver — knowing why is the point
E: io_uring single-op	≈ A one-way	—	io_uring ≠ bypass; value is batching/multishot/SQPOLL (ch04)
F1: D-block, C-states capped	≈ stock p50	collapses toward spin	Tuning buys the tail, not the median (ch03)
F2: D-block under load, unpinned	≈ stock p50	ms-scale	Scheduler timeslices land in your tail
F2: D-block under load, fenced	≈ stock p50	≈ quiet-box tail	Placement alone recovers it — `isolcpus` in miniature
G: veth one-way, kernel socket	≈ D block	≈ D block tails	Part D over a real (virtual) wire — the control arm
G: veth one-way, AF_XDP copy mode	≈ D spin	flatter	The chute skips skb/stack/socket/wakeup; mechanism real, magnitude needs a NIC (ch04)

If your numbers differ by 2–3x, fine — the ordering and the ratios (spin ≪ block; nodelay-off catastrophic; vDSO ≪ syscall) are the results. If ordering differs, debug with the kernel-tuning toolkit (ch03): governor, C-states (ch00a), pinning, noisy neighbors.

One chain from Part D is worth assembling in full before you narrate it: a blocked receiver’s core has nothing to run, so it idles, and an idle core sinks into a C-state — a hardware sleep level (ch00a) that gets cheaper to sit in and more expensive to wake from the deeper it goes. When the packet finally lands, you pay the whole chain in reverse — wake the core, exit the sleep state, reschedule the thread — and that is the 4–15µs (with ugly tails) the block row pays and the spin row deletes.

Narrating this in an interview

The lab’s real product is sentences you’ve earned. The shape that lands:

Claim → number → mechanism. “Blocking receive into an idle thread cost me ~8µs p50 with 50µs+ tails; spinning on the same socket was ~2µs and flat — that’s the scheduler wakeup and C-state exit, which is why hot paths busy-poll” beats any amount of recited theory, because it’s yours.
Show calibrated honesty. “I measured this on loopback, which skips the NIC/DMA/IRQ stages — so it’s a floor for the software stack, not a wire number. On hardware I’d expect X, and I’d verify with hardware timestamps (ch05).” Knowing what your benchmark doesn’t show is the senior tell; so is mentioning that SO_BUSY_POLL did nothing on loopback and exactly why.
Connect to decisions. “getpid cost ~150ns on my box; at one syscall per packet on a million-packet feed that’s 15% of a core before any work — that’s the case for recvmmsg (a single syscall that drains many queued packets at once) or io_uring batching at the gateway tier, and for kernel-bypass rings (ch04) on the true hot path.”
Tie to your production story. “My prod systems ran ~10ms budgets where venue RTT dominated, so tuned-kernel was the right call — but I’ve measured where the next 10µs live: wakeups, syscalls, Nagle-class footguns, and that’s the order I’d attack them before reaching for DPDK.”
Close with the ladder you climbed. “Stock kernel, then tuned — C-state cap and core fencing bought back the tail, median untouched — then AF_XDP, where I posted fill-ring frames and parsed UDP off raw ethernet myself. Copy mode on a veth, so I’ll claim the mechanism, not the zero-copy numbers — but I know exactly which stage each rung deleted, because I measured them separately.” That sentence is Part I of this book, compressed.

Plain-English recap

If you remember nothing else from this lab:

The software stack alone blows the HFT budget: a loopback UDP round trip — no wire, no NIC, pure kernel — costs 8–25µs p50. That’s like discovering your framework’s per-request overhead already exceeds the SLA before your handler runs. This is the packet-path chapter’s claim (ch01), now measured on your box.
One missing set_nodelay(true) costs 1000x: nodelay-off turns tens-of-µs round trips into ~40ms. It’s the missing-DB-index of sockets — invisible in code review, catastrophic in production, fixed with one line.
Kernel entries cost real money; clock reads are free: a genuine syscall is ~100–250ns, but clock_gettime/Instant::now is ~20ns because the vDSO keeps it in userspace — so timestamp everything, but batch your syscalls.
Sleeping is the tax: a blocked receiver pays 4–15µs (with ugly tails) just to be woken — the cold-start penalty. A spinning receiver gets 1–4µs, flat. Warm worker vs. scale-to-zero, measured.
A null result you can explain beats a positive one you can’t: SO_BUSY_POLL did nothing here because loopback has no driver ring to poll — knowing the mechanism behind the flat line is the difference between running benchmarks and understanding them.
io_uring at one op per submit is parity, not victory — the wins are in batching, multishot, and SQPOLL. It economizes the doorway; the packet still walks the hallway.
Tuning buys the tail, not the median — capping C-states left p50 alone and collapsed p99.9; fencing the load off your core undid a milliseconds-scale tail with placement alone. The kernel-tuning chapter’s whole thesis, now three numbers in your notebook.
Bypass is a programming model you have now used: fill ring, RX ring, frames in your own memory, UDP parsed at byte 42 by you. On a veth in copy mode it’s the flight simulator; on a real NIC in zero-copy it’s the sub-2µs path — same code shape.
Methodology is the transferable skill: warmup, 100k+ samples, percentiles never means, one variable per A/B, and a measuring clock cheap enough not to perturb the measurement.

Interviewer will ask

“How did you measure that? Walk me through the methodology.” Warmup iterations discarded (cold caches, page faults, slow-start effects); 100k+ samples; report full percentiles, never means; rdtsc for ns-scale sections with TSC calibrated against the monotonic clock; pinned cores; separate the thing measured from the measuring clock (vDSO reads at ~20ns so timestamping doesn’t perturb µs-scale results). And each A/B changes exactly one variable — nodelay flag, receive mode — on otherwise identical code.

“Why percentiles and not averages?” Latency is heavy-tailed: p50 of 8µs with 1% of samples at 100µs+ averages to a lie. Trading death lives in the tail — the slow response is correlated with the busy market moment when it costs most. p99.9/max are the engineering targets; the mean is marketing.

“Your loopback RTT was 15µs. What would wire-to-wire on real hardware add or remove?” Remove: nothing — loopback already charges the full software stack twice per round trip. Add: the physical path, per direction. Wire serialization is ~70ns to clock a 64-byte frame onto a 10G link. A cut-through switch hop is ~300–500ns — the switch starts forwarding once it has read the header. NIC internal processing plus PCIe DMA is ~1µs. The IRQ path, if I’m not polling, is another ~1–3µs. So hardware RTT lands around the loopback RTT plus roughly 2–4µs of physical path per direction — the software cost is unchanged, and the total stays consistent with the packet-path chapter’s (ch01) 2–10µs-per-direction kernel figure. And I’d verify it with NIC hardware timestamps, not app clocks.

“Why did busy-poll not beat blocking in your test?” Loopback has no NAPI context to poll — packets are delivered synchronously by the sender’s kernel path, so SO_BUSY_POLL’s driver-spin never engages and both modes reduce to queue-check semantics. On a physical NIC the blocking path eats IRQ + softirq + wakeup while busy-poll spins in the driver ring. Knowing the mechanism behind a null result is the difference between running benchmarks and understanding them.

“What’s wrong with timing a single operation with Instant::now before and after?” Clock read cost (~20–30ns) and its serialization effects swamp ns-scale operations; one sample tells you nothing about a distribution; the compiler may reorder or elide un-black-boxed work; and the first execution measures cold caches, not the operation. Hence: loops, warmup, rdtsc, black_box, percentiles.

“You ran AF_XDP. What did the kernel stop doing for you — and what did you have to start doing?” Stopped: allocating an sk_buff per packet, walking IP/UDP through the stack, the socket lookup and queue, and the wakeup — my frames landed straight in a UMEM my process owns. Started: everything the mailroom used to do — I posted empty frames to the fill ring myself, parsed ethernet/IP/UDP myself at fixed offsets, recycled frames back after reading, and static-ARP’d the far side because my XDP program was swallowing ARP along with everything else on that queue. That last one is the lesson in miniature: bypass takes the whole queue, not just your packets — the ch04 costs are real, and I hit one in a lab within the hour.

The HFT Gap Closer