The Machine: What’s Actually in a Server

You’ve deployed to hundreds of servers without ever caring what’s inside the box. That was the correct level of abstraction for your work — a server was “CPU + RAM + network, Docker figures out the rest.” HFT (high-frequency trading) work happens below that abstraction, so this chapter builds the floor. One read of this and the words NIC, PCIe, NUMA, and cache line stop being noise.

Here is the whole machine. Everything in this chapter is a zoom-in on one part of this picture.

┌─────────────────────────────── THE SERVER ───────────────────────────────┐
│                                                                          │
│  ┌───────────────── CPU PACKAGE (one "socket") ─────────────────┐        │
│  │                                                              │        │
│  │  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐                 │        │
│  │  │ Core 0 │ │ Core 1 │ │ Core 2 │ │ Core 3 │  ... (8–64x)    │        │
│  │  │ L1  L2 │ │ L1  L2 │ │ L1  L2 │ │ L1  L2 │  private caches │        │
│  │  └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘                 │        │
│  │      └──────────┴────┬─────┴──────────┘                      │        │
│  │              ┌───────┴────────┐                              │        │
│  │              │  L3 cache      │  shared by all cores         │        │
│  │              │  (~32–256 MB)  │                              │        │
│  │              └───┬────────┬───┘                              │        │
│  │        ┌─────────┴──┐  ┌──┴──────────┐                       │        │
│  │        │  Memory    │  │  PCIe root  │                       │        │
│  │        │ controller │  │  complex    │                       │        │
│  └────────┴─────┬──────┴──┴──────┬──────┴───────────────────────┘        │
│                 │                │  PCIe lanes ("the highway")           │
│         ┌───────┴──────┐   ┌─────┴──────┬─────────────┐                  │
│         │     RAM      │   │            │             │                  │
│         │  (DIMM sticks│ ┌─┴────┐  ┌────┴────┐  ┌─────┴────┐             │
│         │   64–512 GB) │ │ NIC  │  │  NVMe   │  │  GPU /   │             │
│         └──────────────┘ │      │  │  SSD    │  │  FPGA    │             │
│                          └──┬───┘  └─────────┘  └──────────┘             │
│                             │                                            │
└─────────────────────────────┼────────────────────────────────────────────┘
                              │
                        ══════╪══════  the wire (fiber/copper to the switch)

Keep this map in your head. The core question of all low-latency work is: a market-data packet arrives on the wire at the bottom — how many of these boxes does it pass through, and how long does each one take, before your strategy code sees it?

The CPU: cores, threads, clocks

┌──────────────── ONE CORE ────────────────┐
│                                          │
│   Hyperthread 0        Hyperthread 1     │   ← two "logical CPUs"
│   (register set A)     (register set B)  │     the OS schedules onto
│        └──────┬───────────────┘          │
│        ┌──────┴────────────────┐         │
│        │  ONE set of execution │         │   ← but only one physical
│        │  units (ALUs, load/   │         │     engine underneath
│        │  store, branch pred.) │         │
│        └───────────────────────┘         │
│   L1i 32KB │ L1d 32KB │ L2 ~1MB          │   ← this core's private caches
└──────────────────────────────────────────┘

CPU core — one independent instruction-executing engine; a 32-core chip is genuinely 32 little computers sharing a memory system. A trading system cares because latency work is about giving one core one job and never letting anything else touch it.

Clock speed — how many instruction-steps a core takes per second; 3 GHz (gigahertz, billions of cycles per second) means one cycle every 0.33 nanoseconds. Trading systems care because every latency number in this book is secretly a cycle count: “100 ns” means “this core could have executed ~300 instructions in that time.” Adopt this human-scale conversion and keep it for the whole book: if one cycle were one second, 1 ns is ~3 seconds, and 100 ns is ~5 minutes.

Hyperthread (SMT, Simultaneous Multi-Threading) — one physical core pretending to be two logical CPUs by keeping two threads’ register state loaded and interleaving them whenever one stalls waiting on memory. Analogy: one Express server handling two requests “concurrently” — it’s not parallelism, it’s interleaving during I/O waits; throughput up, per-request latency jittery. A trading system cares because a hyperthread sibling competes for the same execution units and L1/L2 cache as your hot thread — so HFT boxes typically disable SMT or leave the sibling of every hot core empty. Free throughput is precisely what you don’t want; you want deterministic latency.

Pinning (CPU affinity) — telling the OS scheduler “this thread runs on core N and nowhere else,” via taskset or pthread_setaffinity_np. Physically it means: this core’s L1/L2 caches fill up with your data and stay warm, because no other thread is ever scheduled there to evict them. Analogy: sticky sessions. A load balancer that can bounce a user between app servers loses every server-local cache; pin the user to one server and its cache stays hot. Trading systems pin every latency-critical thread, always — an unpinned thread that migrates cores arrives at the new core with every cache cold, which is a multi-microsecond tax paid at a random moment.

The cache hierarchy: the latency ladder

        core issues a load: "give me the 8 bytes at address X"
                              │
              ┌───── hit? ────▼──────┐
              │  L1  ~1 ns   32 KB   │  per-core, checked first
              └───── miss ────┬──────┘
              ┌───── hit? ────▼──────┐
              │  L2  ~4 ns   ~1 MB   │  per-core
              └───── miss ────┬──────┘
              ┌───── hit? ────▼──────┐
              │  L3  ~15 ns  ~64 MB  │  shared by all cores
              └───── miss ────┬──────┘
              ┌───────────────▼──────┐
              │  RAM  ~80 ns  256 GB │  "main memory" — the actual DIMMs
              └──────────────────────┘

Cache hierarchy — a stack of progressively bigger, slower memories between the core and RAM (Random Access Memory), where every load checks L1, then L2, then L3, then finally RAM. Analogy: it is exactly your caching stack — in-process object cache → Redis → Postgres → S3 — except in silicon and managed automatically by the hardware. A trading system cares because the difference between “hot path fits in L1/L2” and “hot path chases pointers through RAM” is the difference between a 500 ns and a 5 µs tick-to-trade — same algorithm, same big-O.

The ladder, with the rest of the machine included and translated to human scale (1 cycle = 1 second):

Two lessons hiding in this table. First: RAM is not fast. RAM is your “database” — an 80 ns miss is a 4-minute stall in human scale, and code that misses cache on every access is doing the equivalent of one DB query per variable read. Second: the gap between RAM and network (~80 ns → ~50 µs) is three orders of magnitude, which is why the entire HFT game is “never leave the box, and inside the box, never leave the cache.”

Cache line — the unit in which memory actually moves: 64 bytes, always; ask for 1 byte and the hardware fetches the surrounding 64. Analogy: Postgres never reads you one column — it reads the whole 8 KB page containing the row, because the expensive part is the trip, not the payload. Same logic: the trip to RAM costs ~80 ns whether you take 1 byte or 64, so the hardware always takes 64. A trading system cares in two directions: (a) pack your hot struct so everything you touch per tick sits in one or two lines — one fetch instead of five (this is why order-book entries in HFT code are laid out by hand, like designing a table around its access pattern instead of normalizing it); (b) never let two cores write to different variables in the same line — the line ping-pongs between their caches, a pathology called false sharing you’ll meet in the concurrency chapters.

NUMA: the box is secretly two boxes

┌── Socket 0 ─────────────┐         ┌── Socket 1 ─────────────┐
│  cores 0–15             │ inter-  │  cores 16–31            │
│  L3 cache               │◄─socket─►  L3 cache               │
│  memory controller      │  link   │  memory controller      │
└──────┬──────────────────┘ (UPI)   └──────┬──────────────────┘
       │                                   │
   ┌───▼──────┐                        ┌───▼──────┐
   │ RAM bank │  local: ~80 ns         │ RAM bank │
   │ 128 GB   │  ← from socket 1:      │ 128 GB   │
   └──────────┘     ~130–200 ns        └──────────┘
       ▲
   ┌───┴───┐
   │  NIC  │   ← the NIC is also plugged into ONE socket's PCIe
   └───────┘

NUMA (Non-Uniform Memory Access) — on a two-socket server, each CPU package has its own RAM banks; a core reading its own socket’s RAM pays ~80 ns, but reading the other socket’s RAM crosses the inter-socket link and pays roughly double. Analogy: primary database plus a read replica in another region — same data model, same API, but one of them silently costs you an extra round trip, and the ORM won’t warn you. A trading system cares because the NIC hangs off one socket’s PCIe lanes: if your hot thread is pinned to a core on the other socket, every packet takes the cross-socket detour before your code sees it — a permanent, invisible ~50–100 ns tax on every single message. The standard HFT layout: find which socket owns the NIC (/sys/class/net/eth0/device/numa_node), pin the entire hot path — threads and their memory allocations — to that socket, and treat the other socket as a separate machine that runs logging and cron jobs.

PCIe: the network inside the box

   CPU package
  ┌───────────────────┐
  │  PCIe root complex│         "root complex" = the CPU-side
  └──┬─────┬───────┬──┘          terminus of all PCIe traffic
     │x16  │x8     │x4          ← lane counts: like link bandwidth tiers
  ┌──▼──┐ ┌▼────┐ ┌▼─────┐
  │ GPU │ │ NIC │ │ NVMe │
  └─────┘ └─────┘ └──────┘
   each lane ≈ 2–8 GB/s each way (gen 3→5); a x16 slot = 16 lanes
   one device round trip (CPU → device register → CPU): ~500 ns – 1 µs

PCIe (Peripheral Component Interconnect Express) — the point-to-point packet-switched network inside the box connecting the CPU/RAM complex to every plug-in card: NIC, NVMe (Non-Volatile Memory Express, i.e. modern SSDs), GPU, FPGA. It is literally a network — devices exchange addressed packets over serial links, with lanes bundled for bandwidth like link aggregation. Analogy: it’s the network between your app server and your database, except it lives inside the chassis and its round trip is ~500 ns–1 µs instead of ~500 µs. A trading system cares because PCIe is an unavoidable segment of the packet’s journey — the wire delivers bits to the NIC, but they only reach RAM by crossing PCIe — so on a well-tuned box PCIe becomes one of the largest remaining line items in the latency budget. In human scale: a 1 µs PCIe device round trip is ~50 minutes. This is also the door to the endgame you’ll meet later: FPGAs (Field-Programmable Gate Arrays) win partly by responding from the card itself, skipping the trip to the CPU entirely.

The NIC: a sidecar computer that owns the network

                    the wire (light in fiber / voltage on copper)
                                    │
┌────────────────── NIC ────────────▼────────────────────────────┐
│  ┌──────┐   analog signal ⇄ digital bits                       │
│  │ PHY  │   (clock recovery, encoding — pure electronics)      │
│  └──┬───┘                                                      │
│  ┌──▼───┐   bits ⇄ Ethernet frames                             │
│  │ MAC  │   (framing, addresses, CRC checksum, drop bad frames)│
│  └──┬───┘                                                      │
│  ┌──▼──────────────────────────────┐                           │
│  │ on-card processor + SRAM buffers│  queues, filtering,       │
│  │ + DMA engines                   │  timestamping, offloads   │
│  └──┬──────────────────────────────┘                           │
└─────┼──────────────────────────────────────────────────────────┘
      │ PCIe
      ▼
   host RAM   ← the NIC *writes packet bytes into RAM by itself*

NIC (Network Interface Card) — the hardware whose job is turning signals on a wire into bytes sitting in RAM, and back. A NIC is not a dumb port — it is a small independent computer with its own processor, own memory, and own direct access to your RAM, living on the PCIe bus. Analogy: a sidecar container that owns all network I/O for the pod — your app never touches the wire, it exchanges messages with the sidecar through shared memory; the sidecar has its own CPU budget, its own config, its own failure modes. A trading system cares because everything in Part I is really about your relationship with this device: how it signals you (interrupts vs polling), where it writes packets (kernel buffers vs your process’s memory — the entire “kernel bypass” story), and what work you delegate to it (checksums, timestamping, filtering). Premium NICs (Solarflare/Xilinx, Mellanox/NVIDIA, Exablaze) are HFT’s most-tuned components after the strategy itself.

PHY (physical layer transceiver — pronounced “fie”) — the analog electronics at the NIC’s edge that turn light pulses or voltage wiggles into clean digital bits: modulation, clock recovery, encoding. Analogy: the terminating proxy that turns raw TLS byte-noise into clean HTTP before your app sees anything — except this one terminates physics. Trading cares mostly at the extremes: PHY choices add fixed tens-of-nanoseconds, and exotic setups (hollow-core fiber, microwave links between exchanges) are competitions at this layer.

MAC (Media Access Control layer) — the digital logic one step up from the PHY that groups bits into Ethernet frames, reads source/destination hardware addresses, verifies the CRC (Cyclic Redundancy Check — a checksum), and silently drops corrupt frames. Analogy: body-parsing + signature-verification middleware — it turns a byte stream into discrete validated messages and rejects garbage before anything downstream runs. Trading cares because “the MAC” is where hardware timestamping hooks in (stamping a packet the instant it crosses this boundary — the most honest arrival time you can get), and because frames the MAC drops are invisible to software — you’ll learn to check NIC hardware counters when the sequence numbers in your feed have gaps.

C-states and frequency scaling: the CPU is trying to nap

   C0 ───────── running                    wake cost
   C1 ───────── halted, clocks gated        ~1 µs
   C3 ───────── core caches flushed         ~tens of µs      DEEPER
   C6 ───────── core powered off, state     ~100+ µs         SLEEP
                parked in shared cache
   plus: frequency scaling — an idle core drops from 3.5 GHz to 1.2 GHz
   and takes real time to spin back up when work arrives

C-states — numbered depths of CPU sleep (C0 = running, C6 = core effectively powered off) that idle cores drop into automatically to save power, each deeper level cheaper to sit in and more expensive to wake from. Analogy: serverless cold starts, in miniature and in silicon — scale-to-zero saves money and murders p99. Frequency scaling (P-states) is the same instinct on a running core: clock down when load looks light. A trading system cares because market data is bursty — the moment that matters most (a news spike after a quiet spell) is precisely the moment the governor has decided your core should be asleep and slow, adding tens of microseconds of wake-up latency to exactly the packet you care about. So HFT boxes disable C-states beyond C1, pin the clock at maximum (idle=poll, performance governor), and burn watts for determinism. It’s also half the reason hot loops busy-spin instead of blocking: a spinning core is a core that can never be caught napping. Provisioned concurrency, but for silicon.

The whole-machine latency map

Every number from this chapter on one picture. This is the terrain map for the rest of the book — later chapters are campaigns to shorten or eliminate specific arrows.

              ┌──────── CORE (3 GHz: 1 cycle = 0.33 ns) ────────┐
              │  register: 0 ns   │ branch mispredict: ~5 ns    │
              └───┬───────────────────────────────────┬─────────┘
          ~1 ns   │ L1                                │
          ~4 ns   │ L2                                │ cross-core /
         ~15 ns   │ L3   ◄───────────────────────────►│ false-sharing
                  ▼                                     ping-pong:
              ┌───────┐    cross-socket (NUMA):         ~40–100 ns
              │  RAM  │    +50–100 ns on top
              │ ~80ns │
              └───┬───┘
                  │ PCIe round trip: ~500 ns – 1 µs
              ┌───▼───┐
              │  NIC  │  wire→RAM via DMA: ~1 µs-ish
              └───┬───┘
                  │ serialization, 100B @ 10 Gbps: ~80 ns
            ══════╪══════ the wire
                  │ propagation: ~5 ns per metre of fiber
                  ▼
          same-rack switch hop: ~0.5–5 µs · cross-country: ~30 ms

   human scale (1 cycle = 1 s):  L1 = 3 s · RAM = 4 min · PCIe = 50 min
                                 SSD = 3.5 days · 1 ms of network = 5 weeks

What you can now read

You now have the vocabulary for the physical machine. This unlocks:

• The Kernel, Your Code, and the Wall Between Them (next) — what the software between the NIC and your code does with these components, and what it costs.
• Part I’s network path and kernel-bypass chapters — “the NIC DMAs into a ring on the NUMA node local to the pinned core” is now a sentence, not a hash of acronyms.
• Part I’s CPU isolation and tuning material — isolcpus, C-state and governor settings, and SMT policy are now just “stop the nap, stop the migration, stop the sibling.”
• The cache-aware data structures and false-sharing chapters — the latency ladder and the 64-byte cache line are their entire foundation.