Microbenchmarking Without Fooling Yourself

Before you start — this chapter leans on a handful of primer ideas:

• Caches, cache lines, and coherence — why the same queue costs 4ns on one thread and 40ns across two cores: ch00a
• Cores, hyperthreads (SMT), and sockets — the topology words the contended benchmarks depend on: ch00a
• False sharing — two “independent” variables on one cache line taxing each other: ch00a
• Cycles vs nanoseconds, and PMU counters — why comparisons are done in cycles: ch00e
• rdtsc and frequency scaling — the clocks chapter’s discipline (ch07), which this chapter assumes: ch00e

Read those first — 20 minutes there saves an hour here.

Microbenchmarks are the easiest measurements to produce and the easiest to be wrong about. The failure modes are systematic — the compiler deletes your work, the CPU warms into an unrepresentative state, your “uncontended” queue bench never exercises coherence traffic — and each has a specific countermeasure. This chapter is those countermeasures, plus the calibration table that lets you judge whether a number is even plausible.

Criterion: the baseline harness

Criterion (the de-facto Rust benchmarking crate) is the standard because it does statistics you’d otherwise skip: warmup, many samples, outlier classification, bootstrap confidence intervals (“bootstrap” = resampling your own measurements many times over to estimate how confident to be in the result), and regression comparison against the saved baseline.

# Cargo.toml
[dev-dependencies]
criterion = "0.5"

[[bench]]
name = "spsc"
harness = false

[profile.bench]
debug = true          # so perf can symbolize the bench binary too

#![allow(unused)]
fn main() {
// benches/spsc.rs — skeleton
use criterion::{criterion_group, criterion_main, Criterion};
use std::hint::black_box;

fn bench_push_pop(c: &mut Criterion) {
    let mut g = c.benchmark_group("spsc");
    g.bench_function("uncontended_push_pop", |b| {
        let (mut tx, mut rx) = spsc::channel::<u64>(1024);
        b.iter(|| {
            tx.push(black_box(42u64)).unwrap();
            black_box(rx.pop().unwrap());
        });
    });
    g.finish();
}
criterion_group!(benches, bench_push_pop);
criterion_main!(benches);
}

Useful discipline: cargo bench -- --save-baseline main before a change, --baseline main after — Criterion reports the delta with confidence intervals, which kills “it looks maybe 2% faster” conversations.

black_box and dead-code elimination

The optimizer’s job is to delete work whose result is unused — the pass is called dead-code elimination (DCE); your benchmark’s job is to do work. Left unresolved, that conflict yields benches that measure an empty loop at 0.3ns/iter — a number that should trigger immediate suspicion (that’s one cycle; almost nothing real is one cycle).

std::hint::black_box is an identity function the optimizer must treat as opaque:

• Wrap inputs so the compiler can’t constant-fold the computation across iterations: compute(black_box(x)).
• Wrap outputs so the result is “used”: black_box(compute(x)).
• It is a hint, not a guarantee, but on current rustc it reliably forces the value to materialize (typically to a register/stack slot) without adding a memory fence — that is, without smuggling in an extra CPU-ordering instruction whose own cost would pollute the measurement.

Three more traps, all with the same shape — the loop you wrote is not the loop that ran:

• Loop-invariant hoisting. If every iteration computes the same thing, the compiler computes it once, outside the loop, and your timed loop is a no-op. Feed it varying inputs instead:

  #![allow(unused)]
  fn main() {
  b.iter(|| lookup(&table, 42));               // hoisted: computed once, timed never
  b.iter_batched(|| rng.gen(),                 // fresh input per iteration
      |k| lookup(&table, k), BatchSize::SmallInput);
  }

• Bounds checks as the measurement. Indexing inside the bench loop can make the bounds check — not your function — the dominant cost, a cost the real call site may never pay:

  #![allow(unused)]
  fn main() {
  b.iter(|| { for i in 0..n { sum += data[i]; } });   // measures bounds checks
  b.iter(|| { for x in &data { sum += x; } });        // measures the loop body
  }

• iter_batched setup pollution. For a tiny measured function, the per-batch machinery around it becomes a real fraction of the number — pick the batch size deliberately rather than accepting a default:

  #![allow(unused)]
  fn main() {
  b.iter_batched(setup, tiny_op, BatchSize::LargeInput);  // batch overhead drowns tiny_op
  b.iter_batched(setup, tiny_op, BatchSize::SmallInput);  // sized for small routines
  }

Warmup, frequency, and thermals

The first iterations run cold: icache misses (the icache is the instruction cache — the L1 that holds code rather than data, ch00a), branch predictor untrained, and — biggest — the CPU may be at idle frequency. Criterion’s warmup (default 3s) handles training; it does not control the platform:

• Governor: performance, not schedutil/powersave, or your first benchmark runs at 1.2GHz and your comparison across runs tracks the governor’s mood.
• Turbo: turbo frequency depends on how many cores are active and thermal headroom — a single-threaded bench turbos higher than the same code will run in your 8-thread production process. For comparable numbers either disable turbo (no_turbo=1) or at least know your bench frequency (turbostat).
• Thermals: a 5-minute bench suite on a small box slowly clocks down; benchmark A (run first, cold package) beats benchmark B (run second, hot) for no code reason. Randomize/interleave order or fix frequency.
• Report cycles when comparing algorithms. Nanoseconds confound your code with the frequency circus; cycles (rdtsc, or perf’s cycle counter) isolate the code. Nanoseconds are for budgets; cycles are for comparisons.
• Laptops (and especially MacBooks) are for writing benches, not for believing them. Numbers you’ll quote come from the pinned, governed, isolated Linux target.

Benchmarking lock-free structures: contention is the benchmark

Here is the mistake that invalidates most published queue benchmarks: an SPSC/MPMC (single-producer single-consumer / multi-producer multi-consumer queue) structure has two completely different performance regimes, and a single-threaded bench only ever measures the first:

1. Uncontended: producer and consumer never race; every access hits L1 (ch00a); you’re measuring instruction count and store-buffer behavior — the store buffer is the core’s small private outbox: writes queue up there and drain to the cache a moment later. Good SPSC: a few ns/op.
2. Contended (cross-core): head/tail lines bounce between cores; you’re measuring the coherence protocol (the MESI machinery that keeps caches consistent — ch00a). Same code: 20–100+ns/op, and throughput depends on which cores you picked.

Both are real workloads (a queue drained in bursts runs mostly-uncontended; a saturated pipeline runs contended). So you write both harnesses and report both:

#![allow(unused)]
fn main() {
// Harness 1: uncontended — same thread, alternating push/pop (as above).
// Measures pure instruction cost. Expect single-digit ns.

// Harness 2: contended — two pinned threads, sustained streaming.
fn bench_contended(c: &mut Criterion) {
    c.bench_function("spsc/contended_throughput_1M", |b| {
        b.iter_custom(|iters| {
            let (mut tx, mut rx) = spsc::channel::<u64>(1024);
            let n = iters.max(1_000);
            let consumer = std::thread::spawn(move || {
                core_affinity::set_for_current(core_affinity::CoreId { id: 4 });
                let t0 = std::time::Instant::now();
                for _ in 0..n {
                    loop { if let Some(v) = rx.pop() { black_box(v); break; }
                           std::hint::spin_loop(); }
                }
                t0.elapsed()
            });
            core_affinity::set_for_current(core_affinity::CoreId { id: 2 });
            for i in 0..n {
                while tx.push(i).is_err() { std::hint::spin_loop(); }
            }
            consumer.join().unwrap() // elapsed / iters = ns per transfer
        });
    });
}
}

Notes on that harness:

• iter_custom because Criterion’s default timing loop can’t span two threads; you time the whole stream and divide.
• Pin both threads, explicitly, in the bench (core_affinity crate). Unpinned, the scheduler sometimes lands both threads on SMT siblings — two logical CPUs on one physical core (ch00a) — which share L1/L2 and look suspiciously fast, and sometimes across sockets (suspiciously slow), and your bench has multi-modal results that track nothing in your code. Choose the same topology production uses: same-socket, different physical cores, and say so in the results.
• Report pairs: same-core-SMT / cross-core / cross-socket are three different (interesting) numbers. Cross-socket can be 3–5× cross-core — the cache line has to travel over the physical link between the two CPU chips, not just within one.
• Batch effects: benching push of 1 item repeatedly is different from streaming. A good SPSC keeps a private cached copy of the other side’s index — the producer remembers where it last saw the consumer’s tail — and only re-reads the shared index when that cached copy says the queue might be full or empty. Streaming amortizes that expensive re-read over many items; single-item ping-pong forces one per item and measures the worst case. Know which one your harness exercises.

Measuring allocation

“Zero-alloc hot path” is a claim; measure it:

• dhat (dhat crate): heap profiling with allocation-site stacks. Run the replay under dhat::Profiler, assert the hot phase allocates nothing. Slow, thorough — a CI job, not a hot-path tool.
• Counting allocator: a GlobalAlloc wrapper that increments an atomic. Cheap enough to keep in test builds, and it turns “zero allocations after warmup” into a unit test:

#![allow(unused)]
fn main() {
use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};

pub static ALLOCS: AtomicU64 = AtomicU64::new(0);

pub struct Counting;
unsafe impl GlobalAlloc for Counting {
    unsafe fn alloc(&self, l: Layout) -> *mut u8 {
        ALLOCS.fetch_add(1, Relaxed);
        System.alloc(l)
    }
    unsafe fn dealloc(&self, p: *mut u8, l: Layout) { System.dealloc(p, l) }
}

#[global_allocator]
static A: Counting = Counting;

// In the test: warm up, snapshot ALLOCS, run 1M events, assert delta == 0.
}

The assertion version is the valuable one: it catches the intern who adds a format! to the hot path six months from now, mechanically.

Calibration: theoretical limits your numbers must respect

A number without context can’t be judged. One term before the table: an atomic RMW is a read-modify-write — e.g. fetch_add; on x86, a lock-prefixed instruction like lock xadd. Memorize the ladder (typical modern x86 server, order-of-magnitude):

Use it in both directions. Your contended SPSC does a transfer in 25ns? Plausible — that’s one-and-change line transfers, believable for a cached-index design. It does a transfer in 3ns cross-core? Not plausible — a single cross-core line transfer costs more than that; your harness isn’t crossing cores (SMT siblings, or the consumer is batching in a way you didn’t intend, or DCE ate the work). Implausibly good numbers are bugs in the bench far more often than breakthroughs in the code.

The trap: microbench won, system regressed

You optimized a function; Criterion says −30%; you ship; the replay harness says end-to-end p99.9 got worse. This is common, and the mechanisms are worth knowing cold:

• icache/code-size pressure: the “faster” version is 4× more code (unrolling, inlining, a LUT — lookup table). Alone in a microbench loop, it’s icache-resident and wins. In the real system it evicts other hot code; total icache misses rise; the system loses. Microbenches systematically favor code bloat because the bench binary has no competing working set.
• Inlining changes: your edit pushed a function past the inlining threshold — callers across the crate now make real calls; or the reverse, and register pressure spilled somewhere else. (x86-64 has only ~16 general-purpose registers; inline too much code into one function and the compiler runs out, “spilling” variables to stack memory — and every spilled access is a memory access.) The diff you benched is not the diff that ran.
• D-cache working set: a 64KB lookup table beats computation in isolation, and then evicts the order book from L2 in production.
• Branch predictor training: the microbench’s input distribution trains the predictor to near-perfection; production’s distribution doesn’t.

The rule: a microbenchmark is evidence about a mechanism, never a verdict about the system. The verdict comes from the macro replay harness — captured market data, full pipeline, HdrHistograms, A/B against baseline (the measurement lab, ch12, builds exactly this). The microbench tells you why the macro result moved; only the macro result tells you whether to ship. Institutionalize it: no perf PR merges on Criterion output alone.

Plain-English recap

• black_box is the jsperf lesson. A JIT (or LLVM) will happily delete a loop whose result nobody reads, and your “benchmark” measures an empty loop. Wrapping inputs and outputs in black_box is how you force the work to be real — and a sub-nanosecond result means you forgot.
• Criterion baselines are perf snapshot tests. --save-baseline before, --baseline after, and the delta comes with confidence intervals — the same mechanical “did this PR regress it?” gate you’d want in CI, killing “looks maybe 2% faster” debates.
• Warmup and thermals: never trust a benchmark from a laptop on battery. Cold CPU at idle frequency vs warm CPU at turbo is the cold-Lambda vs warm-Lambda problem; if you don’t pin the governor and know the frequency, you’re benchmarking the thermostat, not the code.
• Contended vs uncontended is hot-row contention in Postgres. The same UPDATE costs wildly different amounts uncontended vs when every transaction hammers one row. Queues are identical: single-threaded cost and cross-core cost are two different numbers, and a benchmark must state which one it measured — and which cores it used.
• The counting allocator is the N+1-query assertion. Like a test asserting an endpoint issues exactly 3 SQL queries, asserting “zero allocations per million events” turns a performance claim into a mechanical regression gate that catches next quarter’s accidental format!.
• Microbench won, system lost = bundle-size thinking. Inlining and lookup tables win in isolation the way a heavyweight dependency “wins” one page — then the bigger footprint evicts everything else and the whole app slows. The end-to-end replay harness is your E2E suite: it, not the unit-level number, decides whether to ship.
• The calibration table is your plausibility linter. You already know a network round trip to Postgres can’t take 10µs, so a test claiming it is broken. Same instinct, new ladder: a cross-core transfer can’t cost 3ns, so a bench claiming it isn’t actually crossing cores.

Interviewer will ask

Q: What does black_box actually do, and when do you need it? A: It’s an optimizer-opaque identity — the compiler must assume the value is read and produced arbitrarily, so it can’t dead-code-eliminate the computation or constant-fold across iterations. Wrap benchmark inputs and outputs. It’s needed whenever the measured work’s result doesn’t otherwise escape. Sanity check: a sub-nanosecond result usually means I forgot it.

Q: Your SPSC benches at 4ns/op. Ship it? A: First question: which regime? 4ns is believable single-threaded/uncontended — that’s an L1-resident instruction-cost measurement. Cross-core it’s below the price of one cache-line transfer, so I’d suspect the harness: threads not actually on separate physical cores, or batching hiding the coherence cost. I want both numbers, from pinned threads, with the topology stated.

Q: Why pin threads inside a benchmark? A: Because cross-core cost depends on topology: SMT siblings share L1/L2 and look fast; cross-socket pays interconnect and looks slow; the scheduler picks differently each run, so unpinned benches are multi-modal noise. I pin to the production topology and state it with the result.

Q: How do you verify a zero-allocation claim? A: Mechanically, two ways: dhat in CI for allocation sites, and a counting GlobalAlloc wrapper with a test that runs the warmed hot path for a million events and asserts the allocation counter delta is zero. Claims that aren’t asserted regress silently.

Q: Microbench improved 30%, system p99.9 regressed. Explain. A: Most likely icache: the faster variant is bigger, wins alone, and evicts other hot code in the full binary. Or the change moved inlining decisions so the production call sites compile differently than the benched one. Diagnosis: perf stat icache/frontend-stall counters on the full system before/after, and the macro replay harness as arbiter. This is why microbenchmarks are evidence, not verdicts.

Q: What would make you distrust a Criterion result of “−3%, p < 0.05”? A: The delta is smaller than what the frequency circus alone can cause — a governor swing from idle clocks to turbo moves nanosecond numbers by tens of percent, and a package that warmed up between the baseline run and the candidate run moves them too. Criterion’s p-value only covers sampling noise within a run; it can’t see that the two runs happened at different clock speeds. So before believing a small delta I remove frequency from the experiment: compare cycles, not nanoseconds; fix the frequency — performance governor, turbo off; and interleave baseline and candidate runs so thermal drift hits both equally. If the −3% survives that, it’s real; if it doesn’t, I was benchmarking the thermostat, not the code.

Q: Roughly what does an uncontended atomic fetch_add cost? Contended? A: Uncontended with the line in L1: ~15–25 cycles — a lock-prefixed RMW. (Ancient x86 froze the whole memory bus for every lock instruction; modern parts just hold on to the one cache line, which is why the uncontended case is this cheap.) Contended: the line ping-pongs, each RMW waits ~40–100+ cycles for line ownership and total throughput collapses to line-transfer rate — which is why per-thread counters aggregated off-path beat a shared counter (the observability chapter, ch11).

Further reading

• Criterion.rs user guide — statistics model, iter_custom/iter_batched, baselines.
• Brendan Gregg, Systems Performance (2nd ed.) — benchmarking chapter (“benchmarking sins” checklist).
• Denis Bakhvalov, Performance Analysis and Tuning on Modern CPUs — measurement bias, frequency management, counter-based bench validation.
• Agner Fog’s optimization manuals (instruction tables + microarchitecture guide) — the source for instruction/atomic cost intuition.
• Paul McKenney, Is Parallel Programming Hard, And, If So, What Can You Do About It? — counting and cache-coherence cost chapters behind the contended-vs- uncontended distinction.

Where this goes next: benchmarks live in the lab — Chapter 11 answers how to keep measurement running inside the production hot path, always on, for a cost you can state and defend.