Kernel Tuning Before Bypass

Before you start. This chapter assumes cores and thread pinning (a CPU is several independent cores; pinning ties a thread to one, ch00a), C-states (CPU sleep levels — deeper sleep saves power but costs tens of µs to wake, ch00a), IRQ / softirq (hardware interrupts and the deferred kernel work they schedule, ch00b), NUMA (each CPU socket has its own local RAM; the other socket’s RAM is slower, ch00a), TLB and hugepages (the address-translation cache, and bigger pages that relieve it, ch00b), and busy-polling (spinning in a loop instead of blocking, in event-loop terms, ch00c). If any are new, read those first — 20 minutes there saves an hour here.

Reaching for DPDK (the take-over-the-NIC bypass framework, next chapter’s subject) before tuning the kernel is a junior move, and interviewers probe for it. A tuned kernel stack gets you from “10–100µs with millisecond tails” to “single-digit µs with tens-of-µs tails” — using standard sockets, standard tooling, standard debuggability. And the tails are the prize: the tick that arrives mid-burst, exactly when the kernel is busiest, is usually the tick carrying the price move you wanted to trade — the bad tick is the valuable tick (the packet-path chapter). For most crypto trading, where venue RTT is 1–70ms, that’s the finish line. This chapter is what a senior does first, roughly in order of return on effort.

The unifying principle: the enemy is not average cost, it’s variance. Nearly every knob below removes a source of “sometimes the kernel does something else”: sleeps, migrations, interrupts, frequency changes, memory faults. You are converting a general-purpose timesharing OS into something that leaves N cores alone.

Here is the whole chapter in one picture — the same millisecond on a trading core, before and after:

 one millisecond on a trading core, before vs after tuning
 ─────────────────────────────────────────────────────────────────────
 BEFORE ──[tick ~2µs]──[IRQ: disk]──[RCU callback ~30µs]──[C6 doze]──▶
          scheduler's   random       kernel's deferred     core sleeps;
          1kHz beat     device IRQ   cleanup, any core     wake ~100µs
                            ▲
                            └─ your market-data tick lands HERE,
                               queued behind work it never asked for

 AFTER  ──────────── your pinned thread, spinning, alone ────────────▶
          isolcpus: no tasks  nohz_full: no tick  rcu_nocbs: no callbacks
          IRQs steered away   C-states capped: the core never dozes
          tick arrives ──▶ handled in ~1µs, every time

Every section below removes one of the BEFORE boxes.

1. Isolate the trading cores

Give your hot threads cores that the scheduler, timers, and kernel housekeeping leave alone. The first three knobs are kernel boot parameters (GRUB GRUB_CMDLINE_LINUX — settings baked into the boot line, so you can’t change them without editing the config and rebooting; the kernel’s equivalent of a redeploy):

isolcpus=2-7 nohz_full=2-7 rcu_nocbs=2-7

isolcpus: the scheduler will not place any task there unless explicitly affinitized (taskset/pthread_setaffinity_np). No random cron job, no kthread (kernel-owned background thread) load balancing onto your book-builder. Modern alternatives exist (cpusets/cset shield, systemd CPUAffinity=), but isolcpus is static, bulletproof, and still what most trading shops use.
nohz_full: stops the ~1000Hz scheduler tick on those cores while a single task runs — removes a periodic ~1–5µs interruption (plus cache damage) every millisecond. Requires a tickless-capable kernel (standard now). One task per core, or the tick comes back — the tick exists to timeslice between tasks, so the moment a second runnable task appears the kernel has to resume the tick.
rcu_nocbs: RCU (Read-Copy-Update) is a kernel synchronization scheme that defers its cleanup work — freeing old versions of shared data — into callbacks that run later, on whatever core is handy. Think of it as the kernel’s garbage collector: the pause lands wherever the collector happens to run, and without this flag “wherever” includes your trading core. rcu_nocbs moves those callbacks onto housekeeping cores. RCU callbacks are a classic “why did my core stall for 30µs” answer.
Pin your threads explicitly (in Rust: core_affinity crate or raw sched_setaffinity), one hot thread per isolated core, and pin the memory too (below, NUMA).

2. Kill power management on those cores

C-states (sleep depth) and frequency scaling are the biggest single source of tail latency on an untuned box:

Exiting C6 (the deepest common core sleep state) costs roughly 40–130µs on server parts. An idle-ish trading core that dozes between packets pays that on every wakeup. This alone can explain a p99 that’s 50x p50.
Frequency: powersave/schedutil governors (a governor is the kernel policy that decides each core’s clock speed) and idle cores mean your first µs of work runs at low clocks; ramp-up takes µs–ms.

# governor
cpupower frequency-set -g performance
# limit C-states (or boot params: intel_idle.max_cstate=1 processor.max_cstate=1)
cpupower idle-set -D 0          # disable all idle states deeper than C0/poll
# alternatively per-state: /sys/devices/system/cpu/cpuN/cpuidle/stateX/disable

Or hold /dev/cpu_dma_latency open with value 0 from your process. The open file works like a lease: while any process holds it, the kernel honors the latency cap; when the process exits, the file closes and the lease releases automatically — cleaner than boot params. This is what tuned (Red Hat’s tuning daemon, which applies named profiles like network-latency) does internally.
Turbo: opinions differ. Turbo is the CPU running cores above their base frequency when thermal and power headroom allow — and that headroom is a budget shared by the whole chip. A neighbor core grinding through heavy AVX-512 (wide vector instructions) draws so much power that the chip down-clocks other cores — including yours — to stay inside the budget. So turbo means higher peak clocks but variable clocks. Many shops disable turbo for determinism and fix all cores at base or a known all-core frequency. The senior answer is “we pin frequency and measure, because variance costs more than mean.”
If a core busy-polls (spins) 100% anyway, C-states never engage on it — one more argument for spinning.
SMIs (System Management Interrupts): the one interrupter no kernel knob controls — firmware pauses every core to run BIOS code (fan control, memory scrubbing) for tens of µs, invisibly to the OS. Count them with turbostat’s SMI column; the fix is in BIOS settings, not Linux.

3. Steer interrupts away, then steer the right flow in

Two-part game: (a) get all IRQs off trading cores; (b) get your NIC queue’s IRQ (if not busy-polling) onto the right core.

systemctl stop irqbalance && systemctl disable irqbalance   # it will fight you
# find the NIC's queue IRQs
grep eth0 /proc/interrupts
# pin each queue IRQ: the value is a bitmask of allowed CPUs (bit N = CPU N)
echo 2 > /proc/irq/123/smp_affinity        # 2 = 0b010 = CPU1  (queue 0 → CPU1)
echo 4 > /proc/irq/124/smp_affinity        # 4 = 0b100 = CPU2  (queue 1 → CPU2)
# default for everything else: housekeeping cores only
echo 3 > /proc/irq/default_smp_affinity    # 3 = 0b011 = CPUs 0+1

RSS (hardware): NIC hashes flows across queues; each queue has its own IRQ → set queue count with ethtool -L, pin as above. The NIC internals chapter covers steering a specific venue flow to a specific queue (ntuple/Flow Director) — the end state is “market data queue’s IRQ and consuming thread on the same core (or adjacent, sharing L3).”
RPS/RFS (software RSS): when the NIC can’t steer, the kernel does — the packet lands on whatever core took the interrupt, and the kernel then hands it to a different core to process. That hand-off is an extra inter-core hop, delivered by an IPI (inter-processor interrupt — one core poking another). RPS spreads packets across a chosen CPU set (/sys/class/net/eth0/queues/rx-*/rps_cpus). RFS goes one step further and steers each flow toward the core where its consuming app actually runs (net.core.rps_sock_flow_entries). Useful when the NIC has too few queues (cloud!); on bare metal with good NICs, prefer hardware steering and leave RPS off — hardware puts the packet on the right core to begin with, no hop.
XPS: /sys/class/net/eth0/queues/tx-*/xps_cpus — which TX queue each CPU uses. Each TX queue is guarded by a lock, so two cores sharing one queue serialize on it; give your order-send core its own TX queue and it never waits behind the logger.

4. Busy polling: stop sleeping

The single biggest kernel-path win (the packet-path chapter: wakeup = 1–5µs+, spiky). Three tiers:

Kernel busy poll: SO_BUSY_POLL (per-socket, µs value) or globally net.core.busy_poll=50 / net.core.busy_read=50. A blocking recv/poll will spin in the driver for up to that many µs — polling the NIC queue directly — before sleeping. Cuts the IRQ+softirq+wakeup chain out of the happy path. Typical gain: several µs off median, far more off tails. Cost: CPU on that core pegged during the window.
Userspace spin on nonblocking socket: loop on recv(MSG_DONTWAIT)/try_recv. Still pays a syscall per attempt (~100–250ns each) but never sleeps, never gets woken — simplest deterministic option, and what Lab I demonstrates.
epoll + busy poll hybrid: EPOLL_BUSY_POLL/epoll_pwait with busy_poll set — for many sockets on one thread.

The philosophical point for interviews: busy-waiting inverts the normal engineering instinct (“don’t spin, block”). In trading, blocking is the bug: sleeping costs wakeup latency, invites C-states, and cools your caches. Spin, and burn the core proudly.

5. NIC ring buffers, coalescing, offloads

ethtool -g eth0                      # current/max ring sizes
ethtool -G eth0 rx 4096 tx 4096      # bigger rings: burst-loss insurance
ethtool -c eth0                      # coalescing state
ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
ethtool -k eth0                      # offload state
ethtool -K eth0 gro off lro off tso off gso off

Rings (-G): bigger = fewer overflow drops during microbursts, slightly worse cache locality and (if a queue actually builds) more queued latency. On a latency box the queue should be near-empty anyway; max the rings and treat any drop (ethtool -S | grep -i drop) as an incident.
Coalescing (-C): the packet-path chapter. Zero it for latency; moot if busy-polling (no IRQs on the hot queue).
Offloads (-K): GRO/LRO batch received TCP segments before delivering — deliberately adds latency to save CPU; TSO/GSO (GSO = Generic Segmentation Offload, TSO’s software cousin) batch on send, meaning your data can sit while a super-segment forms and hardware slices it. On latency-critical interfaces: off. Keep checksum offload on (rx/tx csum) — it’s free in NIC hardware and never delays delivery. Nuance for interviews: GRO-off can hurt if the box also handles bulk TCP (each segment now costs full stack traversal) — hence the real pattern: separate interfaces for trading vs. bulk traffic, tuned oppositely.

6. Memory: hugepages and NUMA

Hugepages (ch00b): 2MB/1GB pages slash TLB pressure — a random walk over a multi-GB book with 4KB pages misses the TLB constantly, and each miss costs a page walk of ~tens of ns. Reserve at boot (hugepagesz=1G hugepages=8 or vm.nr_hugepages) and allocate via mmap(MAP_HUGETLB) or hugetlbfs. DPDK requires hugepages outright (the bypass chapter).
THP (Transparent Huge Pages): the kernel promoting your 4KB pages to 2MB ones automatically. A kernel daemon, khugepaged, rearranges your memory behind your back, and a page being moved is frozen mid-move — any thread that touches it stalls. Prefer explicit hugepages for the hot path, and set THP to madvise — promotion happens only where you explicitly opt in with madvise() — never always, on trading boxes.
Also: mlockall(MCL_CURRENT|MCL_FUTURE) — a page fault on the hot path is a µs–ms catastrophe; lock everything, pre-fault at startup, disable swap.
NUMA (ch00a): the NIC hangs off one socket’s PCIe root complex (where that socket’s PCIe lanes originate) — so packets DMA into that socket’s RAM. Cross-socket delivery adds ~100–200ns per access, plus variance from the UPI link (the socket-to-socket interconnect).
Keeping everything on the NIC’s node: find the node with cat /sys/class/net/eth0/device/numa_node, then keep the IRQ core, the busy-polling thread, and all packet-touching memory there — numactl --cpunodebind=0 --membind=0 ./engine, or per-allocation with libnuma. On dual-socket boxes, two placement defaults can silently cost double-digit percent latency: an interleave policy stripes pages across both sockets, and first-touch placement (Linux puts each page on the node of whichever core first writes it) means memory initialized by a thread on the wrong socket lives on the wrong side forever.

7. The checklist

Idempotent-ish setup block — annotate and adapt, don’t cargo-cult:

#!/usr/bin/env bash
# trading-node tuning — assumes: NIC=eth0 on NUMA node 0,
# housekeeping CPUs 0-1, trading CPUs 2-7 (isolcpus/nohz_full/rcu_nocbs set at boot)
set -euo pipefail

# power/clocks
cpupower frequency-set -g performance
cpupower idle-set -D 0 || true                     # cap C-states

# IRQs
systemctl stop irqbalance 2>/dev/null || true
echo 3 > /proc/irq/default_smp_affinity            # 3 = 0b011: others → CPUs 0-1
for irq in $(grep eth0 /proc/interrupts | cut -d: -f1); do
  echo 4 > /proc/irq/$irq/smp_affinity             # 4 = 0b100: NIC queues → CPU2 (near consumer)
done

# NIC
ethtool -G eth0 rx 4096 tx 4096
ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
ethtool -K eth0 gro off lro off tso off gso off    # keep csum offloads on

# sysctls
sysctl -w net.core.busy_poll=50 net.core.busy_read=50
sysctl -w net.core.rmem_max=134217728 net.core.wmem_max=134217728
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sysctl -w vm.swappiness=0
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

# verify
ethtool -S eth0 | grep -iE 'drop|err' ; nstat -az | grep -iE 'Rcvbuf|Drop' || true

Plus boot line: isolcpus=2-7 nohz_full=2-7 rcu_nocbs=2-7 intel_idle.max_cstate=1 mitigations=off hugepagesz=1G hugepages=8 — where mitigations=off (Spectre/Meltdown) buys back ~50–150ns per syscall and is defensible only on an isolated single-tenant box running fully trusted code; say the caveat out loud in interviews.

Then verify, don’t assume: perf stat (context-switches, migrations should be ~0 on hot cores), perf sched/bpftrace (tracing via small eBPF programs run safely inside the kernel) for stray wakeups, mpstat -P ALL (trading cores 100% user during busy-poll), ethtool -S and nstat for drops, and your own end-to-end histograms — every knob above should move a measured number or be reverted.

When tuned-kernel is enough — and when bypass pays

                     your latency  venue RTT     stack share
 crypto, taker, x-region   ~10µs    ~70-200ms    0.01%  ← tuning is already overkill
 crypto, maker, in-region  ~10µs    ~0.5-2ms     ~1-2%  ← tuned kernel: right answer
 crypto, top-tier in-AZ    ~5µs     ~100-500µs   ~5%    ← bypass starts to pay
 tradfi colo maker/arb     ~5µs     ~10-100µs    ~50%   ← bypass mandatory

Tuned kernel is enough when transit dominates: cross-region crypto (RTT ms+), or any strategy whose alpha horizon is ms+. You keep: normal sockets, tcpdump/eBPF observability, TLS libraries that just work, no dedicated-core tax, portability to cloud VMs where bypass is impossible or crippled. Most crypto trading, including profitable market making, falls in this band — a well-built tuned-kernel stack at ~5–20µs app latency is simply not the bottleneck when the venue’s own gateway jitter is hundreds of µs.
Bypass pays when you’re in the same building/AZ as the matching engine, competitors are at 1–5µs, and the race is per-event (queue position on cancel/replace, cross-venue arb in one metro). Then the kernel’s 2–10µs and its tails are the biggest remaining term, and the bypass chapter is your menu.
The interview frame: “I’d tune first, measure, and only bypass when the measured kernel share of my end-to-end budget justifies owning a userspace stack.” That sentence, with the table above behind it, is exactly the judgment call being tested.

Plain-English recap

If you remember nothing else from this chapter:

The enemy is variance, not the mean — same instinct as chasing p99 API latency instead of average response time. Every knob here removes one way the OS can “sometimes do something else” on your core.
Core isolation is a dedicated worker instance for your hot loop: isolcpus/nohz_full/rcu_nocbs give a thread a core where no cron job, no other pod, no kernel housekeeping ever gets scheduled — like moving batch jobs off the production database.
C-states are cold starts. A dozing core pays 40–130µs to wake, which is why p99 can be 50x p50 while p50 looks perfect. Disabling C-states (or spinning, which never lets the core sleep) is keeping the Lambda warm.
IRQ steering is traffic routing: all interrupts off the trading cores, your hot flow’s interrupt onto the core that consumes it. irqbalance is an autoscaler that will fight your manual placement — kill it.
Busy-polling inverts the async instinct. In Node you’d never while(true) tryRecv() — here, blocking is the bug: sleeping costs a wakeup, invites C-states, and cools your caches. Spin and burn the core; the core is cheaper than the microseconds.
NIC offloads and coalescing are batching: great for throughput (fewer interrupts, bigger chunks), bad for the first message of a burst. Latency interface: everything off. Bulk interface: everything on. Don’t share.
Memory: no surprises on the hot path — hugepages so address translation stays cached, mlockall so a page fault (a µs–ms stall, like a cache miss to disk) can’t happen mid-trade, and everything on the NIC’s NUMA node so packets don’t land in the far socket’s RAM.
Tune, then measure, then only maybe bypass: if venue RTT is milliseconds (most crypto), a tuned kernel at ~5–20µs is already not your bottleneck — spend the effort on placement and connections instead.

Interviewer will ask

“You get a fresh Linux box for a latency-sensitive service. First five things?” One principle generates the whole list: remove every way the OS sometimes does something else on my cores — scheduling, sleeping, interrupting, batching. (1) Scheduling: pin hot threads to isolated cores (isolcpus/cpusets) on the NIC’s NUMA node, so no other task can ever land there. (2) Sleeping: performance governor plus capped C-states, so a core never pays tens of µs waking up. (3) Interrupting: IRQ affinity — everything off the trading cores, the NIC queue’s IRQ next to its consumer, irqbalance dead so it can’t fight me. (4) Batching in the sockets: TCP_NODELAY and busy-poll, so no timer ever holds my message. (5) Batching in the NIC: coalescing to zero, GRO/LRO/TSO off, rings maxed. Then measure — histograms, perf, drop counters — because every knob must move a number or be reverted.

“What do isolcpus, nohz_full, and rcu_nocbs each actually do?” isolcpus: the scheduler won’t place unpinned tasks on those cores — because a random cron job or kthread landing there would preempt the hot loop and trash its caches. nohz_full: kills the periodic scheduler tick while one runnable task owns the core — because the tick only exists to timeslice between tasks, so with one task it’s a pointless ~1–5µs interruption a thousand times a second. rcu_nocbs: moves RCU callbacks — the kernel’s deferred garbage-collection, whose pause otherwise lands on whatever core is handy — onto housekeeping cores; RCU is the classic answer to “why did my core stall for 30µs.” Together: a core where only my pinned thread runs — and I steer the residual hardware IRQs away too.

“Why do C-states destroy tail latency, and what’s the tradeoff of disabling them?” Deep C-state exit costs tens of µs (C6 ~40–130µs) — an idle-ish core pays it per wakeup, so p99 explodes while p50 looks fine. Disabling burns power/heat and forfeits turbo headroom that C-states free up for other cores. On a spinning core it’s moot — the spin holds C0 — which is one more reason busy-poll designs are self-consistent.

“Explain SO_BUSY_POLL. How is it different from your app spinning on a nonblocking recv?” SO_BUSY_POLL makes the kernel spin in the device driver’s poll routine during a blocking receive — it polls the NIC queue itself, bypassing the IRQ→softirq→wakeup chain, falling back to sleep after the budget. App-level spin never sleeps but pays a full syscall per probe and only checks the socket queue (the packet still traversed softirq to get there). Busy poll shortcuts delivery; app spin only shortcuts the wakeup. They compose.

“When would you leave GRO on?” When the interface also carries throughput-oriented TCP (backups, log shipping, snapshots) — per-segment processing without GRO can saturate a core and cause drops, which is worse for latency than coalescing. Clean answer: separate interfaces — trading NIC with everything off, bulk NIC with offloads on. If forced to share, GRO on + accept the µs-level batching, or flow-steer trading traffic to queues you handle differently.

“Your p99.9 wire-to-app is 80µs on a tuned box, p50 is 4µs. Debug it.” Walk the BEFORE diagram from this chapter and ask: which box didn’t I remove? The C-state box first — a C6 wake is 40–130µs, which fits an 80µs tail perfectly — check cpupower idle-info and turbostat residency. The stray-task box: something landed on the core and brought the scheduler tick back with it — perf sched record, context-switch counters should be ~0. The RCU box: is rcu_nocbs actually on the boot line? The random-IRQ box: grep the core’s column in /proc/interrupts — anything counting up shouldn’t be. Two suspects live outside the diagram: a page fault mid-path (mlockall done? THP set to madvise, not always?) and the NIC’s coalescing timer holding a lone packet. And if every box checks clean, it’s the one interrupter no kernel knob controls: SMIs — firmware pausing every core for tens of µs, invisible to the OS; turbostat’s SMI column counts them, the fix is BIOS. Method throughout: bpftrace on scheduler and IRQ tracepoints around the bad samples, not guessing.

“Why might you NOT deploy all this on a crypto trading box in AWS?” Because the budget doesn’t warrant the ops cost and the platform blunts the tools: venue RTT is ms-scale so stack µs are noise for most flows; on VMs you don’t control C-states/SMIs, isolcpus is weaker under a hypervisor scheduler, and ENA/gVNIC limit queue/IRQ/coalescing control (the NIC internals chapter). I’d still do the cheap, robust subset — thread pinning, NODELAY, buffers, busy-poll, governor — and spend the effort on placement (same AZ as venue) and connection management, where the real ms live.

The HFT Gap Closer