NIC Internals

Before you start. This chapter assumes NIC / PHY / MAC (the network card and its wire-facing layers, ch00a), descriptor rings and RX queues (the shared circular queues packets arrive through, ch00b), MSI-X interrupts (how each queue pokes a CPU core, ch00b), DMA (the card writing straight into RAM, ch00b), NUMA and L3 cache (which socket’s RAM/cache the packet lands near, ch00a), and clocks / why timestamps are hard (ch00e). If any are new, read those first — 20 minutes there saves an hour here.

The NIC stopped being a dumb serializer twenty years ago. A modern server NIC is a packet-processing computer — a small router, not a modem: it hashes, classifies, steers, timestamps, checksums, segments, and virtualizes — all before the host sees a byte. The tuning and bypass chapters (ch03, ch04) told you to steer flows and trust hardware timestamps; this chapter is what’s actually happening in silicon, and what survives when you move to the cloud (where, as a crypto trader, you probably live).

RSS: hashing flows across queues

A single core can’t process 10–100G of packets, so NICs implement RSS (Receive Side Scaling): N independent RX queues, each with its own descriptor ring and MSI-X interrupt (ch00b), packets distributed by flow hash.

              ┌─ NIC ─────────────────────────────┐
 packet ──►   │ parse 5-tuple                     │
              │   │                               │
              │ Toeplitz hash(src_ip,dst_ip,      │
              │              src_port,dst_port)   │
              │   │  low bits → indirection table │
              │   ▼                               │
              │ [Q0][Q1][Q2][Q3]  rings + IRQs    │
              └───┬────┬────┬────┬────────────────┘
                  ▼    ▼    ▼    ▼
                 CPU2 CPU3 CPU4 CPU5   (one IRQ+consumer per core)

Mechanics worth knowing cold:

• The hash is usually Toeplitz over the 4/5-tuple (src IP, dst IP, src port, dst port, and optionally protocol — the fields that identify one flow) with a configurable secret key. The result’s low bits index an indirection table (128–512 entries) mapping hash→queue.
• The commands: ethtool -x eth0 shows key+table; ethtool -X rewrites it (e.g., weight queues unevenly, or exclude a queue reserved for your hot flow); ethtool -L sets queue count; ethtool -N eth0 rx-flow-hash udp4 sdfn picks which fields hash for UDP (sdfn = hash on src/dst IP and src/dst port; the letters are ethtool’s field codes, not initials).
• Same flow → same queue, always: per-flow ordering preserved, per-queue state stays core-local (no cross-CPU locking on TCP state).
• The trading caveat: RSS is a load balancer, and you don’t want your venue flow load-balanced — you want it placed. A hash doesn’t know that queue 3’s core also handles your logging flows. Hence flow steering.

Flow steering: pinning a venue’s flow to its queue

This is the routing rule for your VIP customer: “this exact flow → this exact queue, always” — no load balancer in the way. ethtool ntuple rules let you write exact-match classification rules that override RSS. (Vendors brand the same hardware differently: Intel calls it Flow Director; on Mellanox/NVIDIA ConnectX parts it’s flow tables.)

ethtool -K eth0 ntuple on
# CME-style: this multicast market data group:port → queue 0
ethtool -N eth0 flow-type udp4 dst-ip 224.0.31.1 dst-port 14310 action 0
# order-entry TCP session from the venue gateway → queue 1
ethtool -N eth0 flow-type tcp4 src-ip 203.0.113.10 src-port 40000 action 1
ethtool -n eth0            # list rules

The end-state design on a trading box:

• Queue 0 ← venue A market data only; IRQ pinned to (or busy-polled by) the core running venue A’s feed handler, same NUMA node, same L3 (the last-level cache shared between neighboring cores, ch00a).
• Queue 1 ← order-entry session(s). Queue N (the rest) ← RSS for everything else (ssh, monitoring), IRQs on housekeeping cores.
• Result: your hot flow’s packets land in a ring that exactly one core ever touches — no queue sharing with noise traffic, no cache-line bouncing, deterministic cache locality. This is the software-side prerequisite that makes both busy-polling (ch03) and per-queue bypass (ef_vi VIs, AF_XDP on a specific queue — ch04) surgical instead of whole-NIC.
• Related names to recognize, then skip: ATR (Intel’s Application Targeted Routing) and aRFS (accelerated RFS) — hardware that learns where to steer a flow instead of being told; for trading, explicit rules beat adaptive magic — determinism again.

Hardware timestamping: the difference between measuring and guessing

Every latency claim needs a timestamp, and where the timestamp is taken decides what you measured — the difference between reading the CDN edge log and starting a timer inside your route handler:

 wire ──► PHY/MAC ──► DMA ──► IRQ/poll ──► stack ──► recv() ──► app clock_gettime()
          ▲ HW timestamp                              ▲ SW timestamp (kernel rx)  ▲ app timestamp
          (PHC clock, ~ns)                            (µs later)                  (more µs later)

• Software timestamps (SO_TIMESTAMPNS, or your own clock_gettime after recv) include the entire kernel path + wakeup — µs of jitter that has nothing to do with the network. Fine for coarse work; useless for claims like “our wire-to-wire is 4µs.”

• Hardware timestamps: the NIC’s PHY/MAC stamps the packet against the PHC (PTP Hardware Clock — a clock on the NIC, exposed as /dev/ptp0). On RX, the stamp arrives beside each recvmsg as a cmsg — a control message, metadata the kernel attaches next to the payload, like response headers riding alongside a body. On TX, the stamp of your outgoing packet comes back on the socket error queue (MSG_ERRQUEUE) — a famously awkward API; you read your own echoes. ethtool -T eth0 shows what the NIC supports. The exact flags:

  // request hardware RX timestamps on this socket
  int f = SOF_TIMESTAMPING_RX_HARDWARE   // stamp packets in the NIC on receive
        | SOF_TIMESTAMPING_RAW_HARDWARE; // deliver the raw PHC value in the cmsg
  setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &f, sizeof(f));
  

• PTP (IEEE 1588) — Precision Time Protocol, NTP’s precise cousin (ch00e) — syncs the PHC to a grandmaster, the network’s reference clock, usually GPS-fed. The measurement itself is a four-timestamp exchange:

   master ──Sync────────▶ slave     t1 = master sends  (stamped in master's NIC)
                                    t2 = slave receives (stamped in slave's NIC)
   master ◀─Delay_Req──── slave     t3 = slave sends
                                    t4 = master receives (echoed back to slave)
  

  Four timestamps, two unknowns, so both fall out: clock offset = ((t2−t1)−(t4−t3))/2 and one-way path delay = ((t2−t1)+(t4−t3))/2. The stamps must be hardware stamps — a software stamp would add the kernel’s µs of jitter to a ns-scale measurement — and switch queuing would poison the path-delay estimate, which is why PTP-aware switches participate as boundary clocks (re-run the exchange per port) or transparent clocks (write their own queuing delay into the packet) so their queues don’t corrupt the math. Two daemons split the work. ptp4l runs this exchange and disciplines the NIC’s clock from the network — to discipline a clock is to continuously nudge it toward a reference (measure the offset, adjust the rate, repeat) rather than jumping it — achieving sub-µs, often tens-of-ns accuracy on a clean LAN. phc2sys then disciplines the system clock from the NIC’s clock. Trading venues and colos distribute PTP feeds (often GPS-disciplined); regulation (MiFID II RTS-25) requires 100µs-or-better clock sync for HFT firms — timestamping is compliance, not just engineering hygiene.

• Why you care beyond honesty: one-way latency measurement (feed handler A-to-B across hosts) is impossible without synced clocks; A/B feed arbitration analytics, venue SLA disputes, and tick-to-trade attribution (“of our 4.2µs, 0.9 was NIC-to-app”) all hang off hardware stamps. The interview-grade sentence: “software timestamps measure your kernel, hardware timestamps measure your network — never present the first as the second.”

• Top-tier shops go further: optical taps (a passive splitter spliced into the fiber — a wiretap that can’t perturb what it measures) + dedicated capture cards (or switch-based timestamping à la Arista 7130 + MetaWatch) stamping every packet at multiple points — latency measured by watching the wire, not by asking the servers.

Interrupt moderation, revisited at the hardware level

The packet-path chapter (ch01) gave the concept; the knobs live in the NIC:

• rx-usecs / tx-usecs: hold-off timers per queue (“wait up to N µs before raising the IRQ”); rx-frames: fire after N packets regardless. Adaptive modes retune these per-queue against traffic rate — throughput-friendly, latency-hostile (the algorithm optimizes packets-per-interrupt exactly when you want interrupt-per-packet).
• Trading-box setting: ethtool -C eth0 adaptive-rx off rx-usecs 0 rx-frames 1 on the hot queues — or, since ntuple gave you per-flow queues, the refined version: per-queue coalescing (ethtool --per-queue eth0 queue_mask 0x1 --coalesce rx-usecs 0) — zero moderation on the market-data queue, generous moderation on the noise queues so housekeeping cores take fewer interrupts. Best of both, and a detail that signals real hands-on-ness.
• If the hot queue is busy-polled or bypassed, its IRQ never fires and moderation is moot — the knob only matters for whatever still uses interrupts.

SR-IOV: slicing one NIC into many

SR-IOV is NIC multi-tenancy: one physical card carved into many virtual cards, each tenant getting its own. The physical NIC (PF, physical function) exposes multiple lightweight PCIe devices (VFs, virtual functions). Each VF gets its own rings, IRQs, and MAC address; an embedded switch on the NIC bridges them. A VF can be handed directly to a VM (PCI passthrough) or a container — DMA goes straight into the guest, hypervisor not in the datapath.

• Why it exists for you: it’s how you get bypass-grade access in shared environments. A VF passed into your VM/container can run DPDK or a PMD at near-bare-metal latency — add ~hundreds of ns for the embedded switch plus IOMMU translation. (The IOMMU is the memory-management unit for devices: it translates and polices the addresses a card may DMA into.) The alternative is the software-virtualized path: virtio, a NIC emulated in software, feeding through the hypervisor’s software switch (the vswitch) — which adds tens of µs and jitter.
• Costs/limits: VF count and features are constrained (fewer queues, limited filtering vs the PF), the embedded switch is another black box, live migration breaks, and the cloud providers you trade on don’t hand you raw VFs of their choosing: they expose their own paravirtual-ish devices built on this machinery. Which brings us to:

Cloud NICs: what AWS ENA and GCP gVNIC actually give you

Crypto reality: Binance ≈ AWS Tokyo, Bybit ≈ AWS Singapore, OKX ≈ Alibaba HK — so your “colo” is a VM near the venue, and your NIC is what the hypervisor says it is. ENA and gVNIC are the paravirtual NIC devices AWS and GCP respectively expose to guests. On AWS the platform underneath is Nitro — Amazon’s custom hypervisor plus hardware-offload cards; several rows in the table depend on it. Know precisely what you keep and lose vs. bare metal:

Two takeaways:

1. The cloud sets a latency floor you cannot tune through. Intra-AZ RTTs of tens of µs with occasional ms-tails come from the virtualization fabric, not your stack — so heroic guest-side tuning past a point buys nothing. The levers that do matter in cloud: instance family (Nitro-based, ENA Express/SRD where offered — SRD is AWS’s Scalable Reliable Datagram, a multi-path fabric protocol that cuts tail latency), cluster placement groups (same-rack-ish adjacency), same-AZ-as-venue placement (worth ~100µs–1ms vs cross-AZ — and venues’ AZs are community-known), biggest-instance-size for dedicated NIC queues and no CPU neighbors, and connection/session management (ch02). This is why crypto HFT is more about placement and protocol than stack — and being able to articulate that hierarchy is exactly the judgment interviews probe.
2. Timestamping honesty degrades in cloud. Without per-packet hardware stamps, your latency numbers include guest-kernel and hypervisor noise. Practical crypto approach: Nitro PTP plus chrony (the standard Linux NTP client) for decent absolute time (sub-100µs), measure distributions not single numbers, cross-check with venue-provided timestamps (exchange ts fields in feed messages — mind their own accuracy), and never quote “wire-to-wire” numbers you didn’t measure at the wire.

Plain-English recap

If you remember nothing else from this chapter:

• A modern NIC is a small router, not a modem: it parses, hashes, classifies, steers, and timestamps every packet in silicon before your OS sees a byte.
• RSS is a consistent-hash load balancer with sticky sessions: flows hash across N queues so N cores share the load, and one flow always lands on the same queue (ordering preserved, state stays core-local). But it’s statistical — and your venue feed shouldn’t be load-balanced, it should be placed.
• Flow steering (ntuple rules) is a routing rule for your VIP customer: “this exact multicast group → queue 0, always” — a dedicated queue, consumed by a dedicated core, instead of round-robin with the noise. It’s the prerequisite that makes busy-polling and per-queue bypass surgical.
• Hardware vs. software timestamps is measuring at the CDN edge vs. inside your handler: the NIC stamps at the wire against its own on-card clock (the PHC); software stamps include your whole kernel path and scheduling jitter. HW measures the network, SW measures your host — never present one as the other.
• PTP is NTP’s precise cousin: it disciplines NIC clocks across machines to sub-µs (NTP is ms-grade), which is what makes one-way, cross-host latency numbers meaningful at all — and regulators require it.
• SR-IOV is NIC multi-tenancy: one card carved into many virtual cards, each DMA-ing straight into its VM — the substrate cloud NICs are built on, except the provider keeps the knobs.
• In the cloud, placement beats tuning: ENA/gVNIC epoxy over most controls and the hypervisor sets a latency floor (tens of µs, ms tails) you cannot tune through. Spend effort on same-AZ placement, placement groups, instance choice, and the protocol layer — that’s what moves the milliseconds.

Interviewer will ask

“What is RSS and what problem does it solve? What’s the trading-specific problem with it?” First beat — the problem it solves: one core can’t process 10–100G of packets. So the NIC Toeplitz-hashes each flow’s tuple into an indirection table and spreads flows across N queues, one queue+IRQ per core. Same flow → same queue, always — so per-flow ordering holds and flow state stays core-local, no cross-CPU locking. Second beat — the trading problem: RSS is a load balancer, and a venue feed shouldn’t be load-balanced, it should be placed. The hash doesn’t know that queue 3’s core also handles my logging. So I override RSS with an ntuple/Flow Director rule for the hot flow — dedicated queue, dedicated core — and leave RSS to spread the noise.

“How would you pin a venue’s market data to one core, end to end?” ethtool -N ntuple rule matching the multicast group:port → dedicated queue; that queue’s IRQ affinitized to (or the queue busy-polled from) a core on the NIC’s NUMA node reserved via isolcpus; feed-handler thread pinned to that core; consumer of its output on a neighboring core sharing L3, connected by an SPSC ring (single-producer single-consumer lock-free queue). Then verify with ethtool -S per-queue counters and /proc/interrupts that only that queue moves when the feed bursts.

“Hardware vs software timestamps — what’s the difference and when does it matter?” Where the stamp is taken decides what you measured — it’s reading the CDN edge log versus starting a timer inside your route handler. Hardware: the NIC stamps at the PHY/MAC against its on-card PHC clock, ns-grade, so it measures the network; it needs PTP discipline to be comparable across hosts. Software: the kernel or app stamps only after the full stack and a scheduler wakeup — µs of jitter that measures your host, not the wire. So it matters whenever the claim reaches past your host: one-way numbers, A/B feed arbitration, network-vs-stack latency attribution, MiFID II clock-sync rules. One-liner: software timestamps measure your kernel, hardware timestamps measure your network — never present the first as the second. Hands-on tell: TX hardware stamps come back on the socket error queue (MSG_ERRQUEUE) — you read your own echoes.

“Explain PTP in two minutes.” NTP’s precise cousin, built on a four-timestamp exchange. The grandmaster (usually GPS-fed) sends a Sync message, hardware-stamped leaving its NIC (t1) and arriving at mine (t2); my side sends a Delay_Req back, stamped leaving (t3) and arriving (t4). Four timestamps, two unknowns, so both fall out: the asymmetric part is my clock offset, ((t2−t1)−(t4−t3))/2, and the symmetric part is the path delay, ((t2−t1)+(t4−t3))/2. The stamps must be hardware — a software stamp would inject the kernel’s µs of jitter into a ns-scale measurement — and switch queues would poison the path-delay estimate, which is why PTP-aware switches join in as boundary or transparent clocks and correct their own queuing delay. Operationally: ptp4l runs the exchange and disciplines the NIC’s PHC, phc2sys slews the system clock to the PHC. Result on a clean LAN: tens-of-ns to sub-µs sync — enough to make one-way latency numbers meaningful. Without hardware assist it degrades toward NTP’s milliseconds, which is why cloud time is the weak link.

“What’s SR-IOV and why does it matter for low-latency in virtualized environments?” One physical NIC exposes many virtual functions — real PCIe devices with own rings/IRQs — passed directly into VMs/containers so DMA skips the hypervisor’s software switch. It’s the difference between virtio-through-vswitch (tens of µs, jittery) and near-bare-metal (~µs + a few hundred ns for the embedded switch/IOMMU). It’s also the substrate cloud NICs are built on — except the provider keeps the controls, which is exactly why ENA/gVNIC feel like a NIC with most knobs epoxied over.

“You move a strategy from bare-metal colo to AWS next to a crypto venue. What changes in your networking approach?” The cloud sets a latency floor I cannot tune through — tens of µs with ms tails, made by the hypervisor and fabric, not my stack — so placement beats tuning. Three buckets follow. Placement: same AZ as the venue (worth ~100µs–1ms by itself), cluster placement group, Nitro/ENA-Express instance family, biggest size for dedicated queues and quiet neighbors. Stack and protocol: keep only the cheap robust tuning — pinning, NODELAY, busy-poll, buffers, max queue counts — and move the real effort up to the protocol layer: connection warmup, redundancy, reconnect storms, parsing. Hardware flow steering is gone; RPS/XPS — the kernel’s software steering knobs from the tuning chapter (ch03) — approximate the intent. Measurement honesty: ntuple rules and per-packet hardware stamps don’t exist here, so I measure distributions on PHC/chrony-disciplined clocks, cross-check venue timestamps, and say plainly that my numbers are host-inclusive.

“How do you verify your NIC config is actually doing what you think?” ethtool -S (per-queue packet/drop counters — burst the feed, watch exactly one queue increment), /proc/interrupts deltas per core, ethtool -x/-n to dump RSS table and ntuple rules, ethtool -T for timestamp capabilities, ethtool -c/-g/-k for coalescing/rings/offloads, and end-to-end: latency histogram before/after each change. Config that isn’t verified by a counter or a histogram is folklore.

Further reading

• Linux kernel docs: Documentation/networking/timestamping.rst — SO_TIMESTAMPING, PHC, MSG_ERRQUEUE semantics, with example code; the primary source for the timestamp APIs.
• linuxptp project: ptp4l(8) and phc2sys(8) man pages — the operational half of PTP; skim the IEEE 1588 overview in the ptp4l docs.
• Linux kernel docs: Documentation/networking/scaling.rst — RSS/RPS/RFS/XPS definitions straight from the maintainers; pair with ethtool(8) for -N/-X/-L/-C/--per-queue.
• Intel Ethernet Controller datasheets (e.g., X710/E810) Flow Director sections, and AMD Solarflare ef_vi docs — what steering hardware actually implements; heavier reading, high credibility yield.
• AWS amzn-drivers GitHub (ENA driver docs/README) and the AWS “PTP hardware clock on Nitro” documentation; GCP gVNIC docs — the authoritative statements of what cloud NICs do and don’t expose.

Where this goes next: Lab I: Measuring the Stack You’re Bypassing — the chapters behind you (ch01–ch05) handed you numbers; the lab makes you earn them: reproduce the kernel RTT, the Nagle trap, syscall cost, and the price of sleeping on your own machine, in Rust, in under an hour.