Latency Methodology

Before you start — this chapter leans on a handful of primer ideas:

• Percentiles and why averages lie — p50/p99/p99.9 as the vocabulary of latency, and what a heavy tail is: ch00e
• HdrHistogram — the fixed-memory histogram that records billions of samples without keeping them: ch00e
• Coordinated omission — the load-testing bug where you stop measuring exactly when the system is worst: ch00e
• TSC / rdtsc stamps — where the raw timestamps come from (and the calibration discipline of the clocks chapter, ch07): ch00e
• The jitter suspects — page faults, IRQs, scheduler preemption and friends, the kernel-side causes of tail spikes: ch00b

Read those first — 20 minutes there saves an hour here.

You know how to build a fast system. This chapter is about how to know it’s fast — which is a different skill, and the one interviewers use to separate people who have operated latency-critical systems from people who have merely written them.

Distributions, not averages

An average latency is close to useless in trading, for two reasons:

1. Latency distributions are heavy-tailed. Your ~10ms system almost certainly has a p50 nowhere near 10ms; the mean is dragged around by a small number of huge outliers, and it tells you nothing about either the common case or the bad case.
2. The money is in the tail. In a competitive strategy, the distribution of your latency relative to competitors determines fill rates. One slow order during a volatile window isn’t a rounding error — it’s a fill you missed or adverse selection (getting filled at a stale price) you ate. A system with p50=5µs, max=50ms can lose more money than one with p50=8µs, max=100µs.

So you report and track: p50, p90, p99, p99.9, p99.99, max (percentiles — p99 is the value 99% of samples fall under; ch00e if the notation is new) — and you treat the right side of that list as more important than the left. p50 tells you about your architecture. p99.9 and max tell you about your discipline: allocator behavior, page faults, scheduler interference, GC-like stalls in dependencies. In trading interviews, saying “our p50 was X” and stopping is a red flag; saying “p50 X, p99.9 Y, max Z, and here’s what the max was caused by” is the credential.

Why max specifically, when statisticians hate it? Because at realistic sample counts it’s not noise. At 100k events/sec, a full trading day is ~2.3 billion samples; your p99.99 covers all but 230,000 of them. The max is the answer to “what is the worst thing my system actually did today,” and in trading that question has a dollar value.

One more piece of tail arithmetic, because it changes what “1% of requests” means: percentiles are per-request, but users — and strategies — experience sessions, runs of many requests in a row. A session of 100 requests dodges the p99 only if every single one does: probability 0.99¹⁰⁰ ≈ 37%. So ~63% of sessions eat at least one p99 event — “1% of requests” quietly becomes “most sessions.” The trading version is a burst: 500 correlated ticks in a volatile window almost certainly contain your p99.5, so the “rare” tail is effectively guaranteed during exactly the multi-message moments that matter.

HdrHistogram mechanics

Recording billions of samples means you can’t keep them all; you need a histogram. Naive linear buckets force a resolution/range tradeoff. HdrHistogram (High Dynamic Range histogram — ch00e), from Gil Tene, solves it with logarithmic bucketing with linear sub-buckets:

• You declare a range (say 1ns to 60s) and a precision in significant figures (usually 3).
• Values are bucketed so that relative error is bounded: 3 sig figs means any recorded value is within 0.1% of the true value. 1.000µs and 1.001µs land in different buckets; 10.000ms and 10.001ms don’t need to.
• Mechanically: the exponent range is covered by log2 buckets, each subdivided into 2^n linear sub-buckets; bucket index is found with a couple of shifts and a leading_zeros — recording is O(1), a handful of nanoseconds, no allocation after construction.
• Memory is fixed and small (tens of KB for ns→minutes at 3 sig figs), histograms are mergeable (per-thread histograms combined off-path), and percentile queries are cheap iterations.

Concretely, recording a 1,250ns sample: leading_zeros (a hardware instruction that finds the highest set bit of a number — which is its log2) puts 1,250 in the major bucket covering [1024, 2048); a shift of the remaining bits picks the linear sub-bucket inside that range; one counter at one array index is incremented. That’s the entire record path — no search, no allocation, one memory write.

#![allow(unused)]
fn main() {
use hdrhistogram::Histogram;

// 1ns to 60s range, 3 significant figures.
let mut h = Histogram::<u64>::new_with_bounds(1, 60_000_000_000, 3).unwrap();
h.record(1_250).unwrap(); // 1.25µs, in ns
println!(
    "p50={}ns p99={}ns p99.9={}ns max={}ns",
    h.value_at_quantile(0.50),
    h.value_at_quantile(0.99),
    h.value_at_quantile(0.999),
    h.max()
);
}

The shape of the whole pipeline, from a stamped event to a number on a dashboard:

  HOT PATH (per event, ~ns)          COLD PATH (per second, off the hot core)
  ─────────────────────────          ───────────────────────────────────────

  t0 = rdtsc()                        ┌──────────────┐
       …work…                         │ thread A hist│──┐
  t1 = rdtsc()                        ├──────────────┤  │   merge
       │                              │ thread B hist│──┼──────────►  merged
       │ cycles (u64)                 ├──────────────┤  │             histogram
       ▼                              │ thread C hist│──┘                │
  ┌─────────────────┐                 └──────────────┘                   │
  │ per-thread      │   record() is O(1): shift, clz,          query percentiles
  │ HdrHistogram    │   increment one bucket. Each hot                   │
  │ (no lock, no    │   thread owns one histogram — the                  ▼
  │  allocation)    │   A/B/C hists above ARE these.    p50 p99 p99.9 p99.99 max
  └─────────────────┘                                                    │
        no mutex — a shared histogram behind a lock                      ▼
        would itself become the jitter you're hunting             dashboard / SLO

The hdrhistogram crate is a faithful Rust port. Use one histogram per thread per stage, merge in the aggregator. Never share one behind a mutex (the observability chapter, ch11).

Coordinated omission: the flagship failure

This idea has a name because Gil Tene spent years yelling about it: coordinated omission (ch00e has the gentle version) is when your load generator conspires with the system under test to not measure during the worst moments — precisely the samples you care about.

The worked example

You want to test at a constant 10,000 requests/sec, one request every 100µs. You write the obvious closed loop:

loop {
    t0 = now();
    send(); wait_for_response();
    record(now() - t0);
    sleep_until_next_interval();
}

The system runs happily at 50µs per response. Then it stalls for 1 second — page fault storm, whatever. What does your histogram show?

• One sample of ~1 second.
• Then the loop resumes and records 50µs samples again.

But you intended to send 10,000 requests during that second. A real open-world client population (orders arriving from the market) doesn’t politely stop arriving because you stalled. The request that would have arrived 100µs into the stall would have waited ~999.9ms. The one at 200µs, ~999.8ms. And so on: 10,000 samples ranging uniformly from ~0 to ~1s should be in the histogram. Instead there is one bad sample — the other 9,999 were simply never taken, because the load generator was blocked, coordinating with the stall.

Run the numbers for a 100-second test: 1,000,000 intended samples. Honest accounting puts ~10,000 samples (1%) spread uniformly across 0–1s — the entire top percentile is that stall block, so the honest p99.5 sits near 500ms and the p99.9 near 900ms. The coordinated-omission version shows ~50µs at both marks, plus one weird max. The reported tail is wrong by four orders of magnitude — 500ms against 50µs is 10,000×. This is not a subtle statistical quibble; it is the difference between “our system is fine” and “our system dropped the ball for a full second of market activity.”

The correction: intended send time

Account for when each request should have been sent:

• Schedule request i at intended[i] = start + i * interval.
• Record response_time = completion - intended[i], not completion - actual_send. The time a request spent queued behind your own blocked load generator is real latency a real client would have seen.

HdrHistogram also offers post-hoc correction — record_correct(value, expected_interval) back-fills the missing samples by synthesizing the linearly decreasing series — but intended-time accounting at the source is strictly better; use record_correct only when you can’t fix the generator.

#![allow(unused)]
fn main() {
use hdrhistogram::Histogram;
use std::time::{Duration, Instant};

let interval = Duration::from_micros(100); // 10k/s intended rate
let mut h = Histogram::<u64>::new_with_bounds(1, 60_000_000_000, 3).unwrap();
let start = Instant::now();

for i in 0u64.. {
    let intended = start + interval * (i as u32);
    // Open-loop: wait until the *scheduled* send time — never later because
    // a previous response was slow.
    while Instant::now() < intended { std::hint::spin_loop(); }

    do_request_and_wait(); // the system under test

    // Latency measured from *intended* send time: queuing delay caused by
    // our own backlog is charged to the system, as a real client would see.
    h.record(intended.elapsed().as_nanos() as u64).unwrap();
    if start.elapsed() > Duration::from_secs(30) { break; }
}
}

(A fully open-loop harness sends from a paced thread regardless of outstanding responses and matches completions asynchronously; the code above is the minimal single-threaded version that still gets the accounting right.)

Open vs closed loop, and the throughput–latency curve

• Closed loop: N virtual clients, each sends, waits, sends again. Arrival rate adapts to system speed. Models: a fixed pool of synchronous callers. Inherently prone to coordinated omission.
• Open loop: arrivals come from an external schedule (Poisson — randomly spaced arrivals, like independent customers walking in — or fixed-rate), regardless of completions. Models: the market. Market data does not slow down because you’re busy — trading systems must be tested open-loop.

The deliverable of a load test is not a number, it’s a curve: sweep offered rate, plot p50/p99/p99.9 vs throughput. Every queueing system shows the same shape: flat latency at low utilization, then a knee where queueing delay explodes as you approach capacity (queueing theory: delay ∝ 1/(1−utilization)). Picture a checkout line at 95% utilization: there is no slack left to absorb a clump of arrivals, so the queue — and the wait — explodes. Find the knee, then state capacity as “we run at X, the p99.9 knee is at 4X.” A single “we handle 1M msgs/sec” claim without the latency curve is meaningless — anything can “handle” any rate if you let the queue grow.

The jitter-source checklist

When the tail is worse than the median by more than ~10×, walk this list. Each item is a distinct mechanism with a distinct signature and fix. Most of the vocabulary is already yours: page faults, TLB, IRQ/softirq, and scheduler preemption are ch00b’s kernel mechanisms; NUMA and hyperthreads are ch00a’s hardware topology; IPC is instructions-per-cycle (ch00e). The genuinely new terms: SMIs (System Management Interrupts — firmware-level interrupts the OS literally cannot see), an arena (a preallocated block you carve allocations out of and free all at once), khugepaged (the daemon that merges regular pages into hugepages in the background — it can freeze a page mid-move, right under your hot path), SCHED_FIFO (the run-until-yield real-time scheduler class), and an AVX license transition (the CPU briefly downclocking while its wide-vector units power up).

The interview version: don’t recite the list; explain that each has a measurable signature (the profiling chapter, ch09, shows how to catch scheduler noise with ftrace and the rest with perf) and that you eliminate them by measurement, not superstition.

Measuring under realistic load

Steady-state numbers lie. Market load is violently non-stationary:

• Open/close auctions: message rates 10–100× the daily median in the first and last minutes. If you sized queues and measured latency at median load, the open is where you find out.
• News/econ prints: near-instantaneous bursts — thousands of ticks in a millisecond across correlated symbols. This is also when your strategy most wants to trade, so tail latency during bursts is the only tail latency that matters.
• Quiet periods: paradoxically dangerous — caches cool, pages get reclaimed, frequencies drop, branch predictors (the CPU’s learned guesses about which way your ifs go) decay and retrain. The first message after a lull is often your worst message. (Countermeasure: keep the path warm with synthetic traffic / cache-warming dummy work.)

Methodology consequences:

1. Replay captured market data with original timestamps, including the worst bursts you’ve recorded — not a Poisson generator at average rate.
2. Report percentiles conditioned on load regime: p99.9 during burst windows vs overall. A system can have a beautiful overall p99.9 that is entirely composed of quiet periods.
3. Test the burst after the lull — the cold-start-into-burst transition is the realistic worst case, and steady-state harnesses never exercise it.

Plain-English recap

• You already do the first half of this in Datadog. Tracking p95/p99 per endpoint instead of averages is standard APM practice; trading just extends the discipline to p99.99 and max, because one slow “request” is a missed fill with a dollar sign, not one grumpy user.
• Coordinated omission is the synthetic-monitor blind spot. Your checkout service freezes for 30 seconds; your health checker (one synchronous loop) logs one slow check and resumes. The 3,000 customers who would have arrived during the freeze were never measured. That’s exactly what a closed-loop load generator does to a latency test.
• The intended-time fix is “measure from enqueue, not from dequeue”. If you time webhook processing from when the worker picked the job up, queue wait is invisible — and queue wait is precisely what the user experienced. Measuring from the scheduled send time charges your own backlog to the system, like measuring payment latency from the user’s click.
• Open loop vs closed loop is Black Friday vs a polite single customer. Real traffic doesn’t slow down because you’re struggling; a closed-loop tester does, and thereby flatters you. Markets are Black Friday all day.
• The throughput–latency knee is connection-pool saturation. A pgbouncer pool that’s fine at 60% utilization goes vertical near 100% — same 1/(1−u) queueing math. Capacity is the rate at the tail knee, not the biggest number the box survived.
• Conditioning on load regime = “p99 during the flash sale”. An overall p99 dominated by 3am quiet hours is flattering fiction; the only tail that matters is the one during the moments you actually need to perform — auctions, news bursts, your Black Friday.

Interviewer will ask

Q: Explain coordinated omission like I’m a skeptical SRE. A: If your load generator waits for each response before sending the next, then during a stall it stops sampling — exactly when latency is worst. One 1s stall at an intended 10k/s costs you 10,000 samples that should have recorded up to 1s of wait; you record one. Your tail percentiles come out wrong by four orders of magnitude — the chapter’s worked example reports ~50µs where the honest p99.5 is ~500ms. Fix: schedule sends at intended times and measure from the intended time, or use HdrHistogram’s record_correct as a patch.

Q: Why do you care about max latency? Any statistician will call it noise. A: At 100k events/sec a day is billions of samples — the max is a real event that really happened to a real order, and in trading one slow order has a direct cost: a missed fill or adverse selection. Also, maxes recur: today’s unexplained max is tomorrow’s p99.9. I track max per interval and I want a causal story for every spike.

Q: Open vs closed loop — which do you use and why? A: Open loop for anything market-shaped: the market’s arrival rate doesn’t adapt to my system’s speed, so a closed-loop test both understates latency (coordinated omission) and overstates capacity. Closed loop only models fixed pools of synchronous callers, which almost nothing in trading is.

Q: How does HdrHistogram get huge range and fine precision in fixed memory? A: By bounding relative error, not absolute — and the chapter’s 1,250ns sample is the whole trick in one record. leading_zeros finds the highest set bit, which is the log2, so 1,250 lands in the major bucket covering [1024, 2048); a shift of the remaining bits picks the linear sub-bucket; one counter at one index increments. Sub-buckets are sized so every value stays within the configured significant figures — 0.1% at 3 sig figs — which is why 1.000µs and 1.001µs get separate buckets while 10.000ms and 10.001ms don’t need to. That’s O(1) record — shift, clz, increment — no allocation, mergeable, tens of KB for the whole ns-to-minutes range.

Q: Your p99 is fine but customers complain. What’s going on? A: Percentiles are per-request, but a customer experiences a session — the chapter’s tail arithmetic. A session of 100 requests dodges the p99 only if all 100 do, and 0.99¹⁰⁰ ≈ 37% — so ~63% of sessions eat at least one p99 event. “1% of requests” is therefore “most customers,” and the complaints are exactly what the math predicts. The trading version: a 500-tick burst almost certainly contains the p99.5, during precisely the moments the strategy trades. And before defending the p99 itself, I’d check it wasn’t measured with coordinated omission — a flattering p99 is this chapter’s other classic lie.

Q: How do you find the capacity of a service? A: Sweep offered rate open-loop, plot p99/p99.9 vs throughput, find the knee where queueing delay diverges. Capacity is the rate at the knee with an SLO on the tail, not the max rate the box survived. Then re-run with recorded burst traffic because the knee under bursty arrivals is lower than under smooth arrivals.

Q: Steady-state p99.9 is 8µs. What number do you tell the desk? A: Neither that nor anything single. I’d give the tail conditioned on regime: p99.9 during open/close and news-burst windows from replayed captures, plus the worst-case cold-into-burst number. Steady-state tails are the flattering, irrelevant case — the tail during bursts is when the strategy actually trades.

Further reading

• Gil Tene, “How NOT to Measure Latency” (talk, many recordings) — coordinated omission, percentile fallacies, the service-time vs response-time distinction.
• HdrHistogram documentation and the original Java repo’s design notes (hdrhistogram.org); the Rust hdrhistogram crate docs mirror the API.
• Brendan Gregg, Systems Performance (2nd ed.) — methodology chapters (USE method, workload characterization, latency analysis).
• The wrk2 README — the canonical short explanation of constant-throughput, corrected-latency load generation.
• Neil Gunther / standard queueing-theory treatments of utilization vs response time (any presentation of M/M/1 and the 1/(1−ρ) blow-up) for the knee.

Where this goes next: you now know what to measure and how to avoid lying to yourself — Chapter 9 is the toolbox that answers where the time went and who caused the spike: perf counters, flamegraphs, ftrace, and a worked tail-regression diagnosis.