Question Bank: Performance

25 questions, easy to brutal. These rounds test one meta-skill: whether your numbers are real — measured correctly, decomposed into stages that add up, and defended against the classic traps (coordinated omission above all — it gets two questions in this bank because it lurks in every measurement loop). Answers are 3–8 sentences, first person, written to be spoken.

1. What is tick-to-trade latency, precisely? Wire-to-wire: first bit of the triggering market-data packet arriving at my NIC to first bit of the resulting order leaving it — because that’s the only definition the market grades me on. Anything measured app-in to app-out silently excludes the network stack and serialization, which can dominate. Internally I decompose it into stages, but the headline number must be wire-to-wire with hardware timestamps or a capture device, or it’s a partial truth.

2. Decompose a tick-to-trade path and put rough numbers on each stage. Colo-class decomposition: NIC ingest + delivery to userspace (kernel path ~1–5µs, bypass ~sub-µs), feed decode (tens of ns for binary, fixed layout), book update (tens of ns, cache-resident), strategy decision (ns to µs depending on model), risk checks (tens of ns if in-process tables), order encode + TX (sub-µs bypass). Software total: single-digit µs kernel-stack, ~1µs tuned bypass, hundreds of ns for the elite, and FPGAs take the decode-to-order core to ~100ns-class. My crypto version has the same shape with a different floor: JSON decode alone can be 1–5µs, and the venue RTT of 1–50ms dominates everything — which is why I profile the host but place the strategy geographically.

3. What is rdtsc and why do we use it for timing? The x86 time-stamp counter: a per-core counter read in ~20–30 cycles, versus ~20–30ns+ for clock_gettime even via vDSO — so it’s the only clock cheap enough to sprinkle through a hot path. Modern CPUs give you invariant/constant TSC (ticks at a fixed rate regardless of frequency scaling) and synchronized-across-cores on sane systems, but you must verify both (CPU flags, and beware VMs/migration). Convert ticks to ns with a calibrated ratio, and remember what it isn’t: not wall clock, not comparable across hosts — for cross-host claims you’re back to PTP.

4. rdtsc vs rdtscp vs fences — when does it matter? rdtsc can execute out of order relative to the code you’re timing — the CPU may hoist it above or sink it below your measured region, corrupting nanosecond-scale measurements. rdtscp waits for prior instructions to retire (and returns a core ID); pairing with lfence/serializing instructions gives stricter ordering at higher cost. For coarse timing (µs+) it’s noise; for timing a 40ns book update it’s the difference between a measurement and a fiction. My rule: rdtscp or lfence+rdtsc at region boundaries when the region is <1µs, and always sanity-check by timing an empty region to know the measurement’s own floor.

5. The classic filter: what is coordinated omission? The measurement bug where your load generator waits for each response before sending the next request, so when the system stalls, you stop measuring during the stall — the periods of worst behavior generate the fewest samples. A 1-second hiccup under intended 1k req/s should contribute ~1000 terrible samples; a coordinating harness records one. Result: tail percentiles that look beautiful and are fabricated. Fix: schedule sends on the intended timeline and measure from intended send time, so a stall’s queueing delay lands in every affected sample — this is Gil Tene’s core critique and HdrHistogram’s correction mode exists precisely for it.

6. Where does coordinated omission hide outside benchmarks? Everywhere a measurement is taken only when the system is healthy enough to take it: an event loop that timestamps work when it dequeues rather than when work arrived (its own stall vanishes), metrics sampled by the same thread that stalls, replay harnesses that feed the next event only after the previous is processed (a real feed would have queued), and health checks that can’t run during the incident they exist to catch. My defense in prod: timestamp at ingress (ideally NIC), compute latency as completion-minus-arrival, and alert on absence of measurements as loudly as on bad ones.

7. Mean, p50, p99, p99.9, max — which do you engineer for and why? For trading: the tail, because losses correlate with exactly the moments that produce tails — volatile markets generate both the most opportunity and the most load, so the p99.9 during a burst is the latency of your most important trades, not your rarest. The mean is nearly useless (dominated by quiet-market samples); p50 tells you about the common path; p99.9 and max tell you what happens when it matters. I also look at conditional percentiles — latency during the top decile of message rate — which is the number that actually matters.

8. What is IPC and how do you interpret it on a hot path? Instructions per cycle from perf counters — a proxy for how much the core stalls. Modern cores can retire 4–6 µops/cycle (µops are the micro-steps instructions decode into; that 4–6 is the hardware ceiling, while typical real code achieves 0.5–4 IPC — the number from the ch00d table); hot-path code at IPC 0.5 is stall-bound (usually cache misses, branch mispredicts, or dependency chains), while IPC 3+ means you’re compute-dense. But interpret with care: high IPC isn’t the goal — low time is; you can inflate IPC with useless instructions, and a spin-wait loop shows gorgeous IPC while doing nothing. I use IPC as a triage signal: low IPC → go look at cache-miss and branch-miss counters; high IPC but slow → you’re simply executing too many instructions, go look at the algorithm.

9. How do you diagnose false sharing, and what does the fix look like? Symptom: a multithreaded path slows down when threads increase, with perf showing heavy cache-line contention on lines no one logically shares — two threads writing distinct variables that happen to sit in the same 64-byte line, ping-ponging it between cores in Modified state. Diagnosis: perf c2c is purpose-built (shows contended lines and the offsets within them); or bisect by padding suspects and re-measuring. Fix: pad/align to 64 bytes (#[repr(align(64))] wrappers in Rust, e.g. around per-thread counters or the head/tail indices of an SPSC queue — the classic case), or restructure so each thread writes its own line and aggregation reads occasionally. The interview flourish: consecutive AtomicU64 counters in an array, one per thread, is the textbook self-inflicted version.

10. Your p50 improved but p99.9 got worse after an optimization. Hypotheses? The optimization moved work from “always” to “sometimes”: (1) added a cache — hits are faster (p50), misses now include fill cost or invalidation stalls (tail); (2) batching/deferral — common case cheaper, deferred work now lands in bursts (tail); (3) memory layout changed and a rare path now misses where it didn’t (or allocation moved: fast path pool-hits, slow path now takes the allocator lock); (4) code got bigger through inlining — pressure on the i-cache, the cache that holds the code itself, hurts the rarely-executed paths most, because big code evicts them first; (5) lock-free retry loops — cheap uncontended, unbounded under contention; (6) measurement itself changed (fixed a coordinated-omission bug and the tail was always there). My actual procedure: capture the tail samples’ context (what stage, what core, what concurrent load) rather than guessing — the histogram tells you that, only tracing tells you why.

11. How do you benchmark a lock-free queue properly? Wrong way: tight loop of enqueue/dequeue on one thread — measures nothing but cache-hot happy path. Right way: real thread topology (producer and consumer on the pinned cores production will use, because cross-core costs are what you’re there to measure), realistic arrival process (Poisson or recorded arrivals at target rate — not back-to-back, which is coordinated omission again), measure per-item latency from intended-enqueue-time to dequeue-completion into HdrHistogram, and sweep occupancy: an SPSC ring behaves completely differently near-empty (producer and consumer share the same cache lines) vs half-full vs near-full (backpressure path). Report percentiles at each rate, include the saturation knee (the arrival rate where latency turns sharply upward), and run long enough to catch periodic interference. Also benchmark the failure path — what full-queue does to the producer — because that’s the path that fires during the burst you built the queue for.

12. What observability can you afford in the hot path, and what’s the budget? Rule: the hot path may record, never emit — no syscalls, no locks, no allocation, no formatting. Affordable: rdtsc stamps into a preallocated per-thread ring buffer (tens of ns), counters in thread-local cache-line-padded slots, latency histogram increments (HdrHistogram-style array bump, ~ns). A separate core drains rings, aggregates, formats, ships. Budget it explicitly: if the path is 1µs, I’ll spend ≤5% — say 50ns — on instrumentation, which buys ~2–4 timestamps and a few counter bumps; choose stage boundaries accordingly. And never make it toggleable in a way that changes layout/branching between “observed” and “unobserved” builds, or your measurements describe a different program — always-on cheap beats sometimes-on detailed.

13. ‘How do you KNOW your 10ms number is real?’ Chain of custody for the measurement: where exactly are the two timestamps taken (NIC hardware vs app), are the clocks the same clock (same host? PTP/NTP quality if not — cross-host µs claims need PTP), does the harness coordinate-omit (send on intended schedule?), is the distribution reported or just a mean, what’s the sample count at the percentile quoted (a p99.9 needs ≥ tens of thousands of samples to mean anything), and was it measured under production-shaped load including bursts. My prod version: wire-adjacent timestamps where possible, ingress-stamped otherwise, HdrHistogram, conditional-on-load percentiles, and an external check — venue ack timestamps or a capture box — to catch my own instrument lying. If someone’s number lacks that chain, I don’t argue with it; I ask where the timestamps live and the number usually corrects itself.

14. Walk me through a p99.9 regression investigation, start to finish. First, verify it’s real: same measurement path, same load shape, sample counts adequate, and bracket it in time — “started with Tuesday’s deploy” is half the diagnosis (correlate with deploys/config/data-shape changes; my event-sourced setup lets me replay the same day through both builds, which isolates code from market regime in one step — that replay-diff is genuinely my favorite tool). If replay reproduces it: profile the two builds on identical input, diff flamegraphs, done. If replay is clean, it’s environmental: check the tail samples’ metadata for clustering (one core? one venue? periodic — smells like a timer/housekeeping; correlated with GC/allocator/page faults?), check host changes (kernel, IRQ affinity, CPU governor, neighbor VMs in cloud). Fix, then prove it with the same histogram under the same load, and add the regression as a permanent gate in CI replay.

15. What do cache misses actually cost, and how does that shape hot-path design? Rough modern numbers: L1 ~4–5 cycles (~1ns), L2 ~12–14, L3 ~40–60, DRAM ~60–100ns+ — so one DRAM miss costs as much as ~100 well-fed instructions, and a pointer-chasing structure that misses 5 times per event has spent half a microsecond doing nothing. Design consequences: arrays over linked structures, hot fields packed into the first cache line of a struct, index-based arenas instead of pointer soup, per-core data to avoid coherence traffic, and prefetch-friendly (predictable-stride) access. The book-update path is the canonical example: a flat sorted vector or array-indexed price ladder beats a node-based tree not because of big-O but because of misses — and perf’s cache-miss counters, not intuition, arbitrate.

16. Branch mispredicts: when do they matter and what do you do? ~15–20 cycles per miss, which matters when it’s per-message on a multi-million-message path. The usual sinners: data-dependent branches with near-50/50 outcomes (buy/sell, message-type dispatch over unpredictable sequences), virtual/indirect calls with mixed targets, and error-checking chains. Remedies, in order: make branches predictable (sort/partition work so the same path repeats), replace with branchless selects/arithmetic where cheap (cmov — but measure; branchless can be slower when the branch predicts well), turn indirect dispatch into direct code per stream (per-venue monomorphized parsers — which I do in spirit by giving each venue its own decode path), and keep the rare path out of line (#[cold], unlikely hints) so it doesn’t pollute the i-cache. perf’s branch-miss counters per stage tell you whether this is your problem before you contort the code.

17. What does the allocator do to your latency, and what’s the discipline? malloc/free are locks, syscalls (sometimes), page faults (sometimes), and cache misses (always eventually) — any of which is a tail event; plus allocation-heavy steady state fragments and drifts over hours, which is deadly for an always-on engine. Discipline: no allocation on the hot path after warmup — preallocated pools/arenas for orders and messages, fixed-capacity rings for queues, Vec::with_capacity and reuse, and object recycling with generation counters (a reuse counter that catches stale references to a recycled slot). Enforce it, don’t intend it: a counting global allocator in debug/CI that panics on hot-path alloc (Rust makes this a 20-line GlobalAlloc wrapper), plus page-fault counters in prod. And the crypto-specific sin I’ve actually fixed: JSON parsing allocating per message — an arena, or simd-json-style tape parsing (parse into one flat pre-allocated array instead of allocating objects), turns the dominant cost into a bounded one.

18. NUMA: when does it bite a trading box? Two-socket boxes have local and remote memory: remote misses cost ~1.5–2x local, and worse, the NIC DMAs into one node — if your consumer thread runs on the other socket, every packet starts with remote-memory reads. Bites: after a naive restart when the scheduler places threads differently than last time (mysterious “it’s slower since the reboot”), when the IRQ/queue core and the processing core straddle sockets, and when a memory pool was faulted in from the wrong node (first-touch policy — allocate and touch from the thread that will use it). Discipline: single-socket for the hot path if at all possible; otherwise pin NIC, IRQs, memory, and threads to one node (numactl, libnuma) and verify with numastat/perf. Cloud instances mostly hide this from me, though — it’s a colo-class concern I know the theory and tooling for, not one I’ve fought in prod.

19. Huge pages: why and what’s the catch? 4KB pages mean a large hot heap thrashes the TLB — every TLB miss is a page-walk (multiple memory accesses); 2MB/1GB pages cut TLB entries needed by ~512x, removing a whole class of tail events for big books and history buffers. Use explicit hugepages (hugetlbfs/mmap flags) for the known-large arenas. The catch: transparent huge pages (THP) can be worse than nothing — khugepaged compacts and splits pages in the background, causing exactly the multi-ms stalls you were avoiding, so the standard prescription is THP off (or madvise-only) and explicit allocation where you want it. It’s cheap to check (/proc/meminfo, TLB-miss counters) and one of the highest signal-to-effort host tunings.

20. How do you keep a rarely-taken path fast — the ‘cold branch that matters’ problem? The risk-limit breach, the kill-switch check, the error path: taken once a month, must be fast that once, and meanwhile must not slow the common path. Techniques: keep the check itself trivially cheap and always-exercised (a compare against a cached limit — cold data is fixable: keep the limit on the hot cache line even when the branch is never taken), move the handling out of line (#[cold]/outlined function) so i-cache stays clean, and — the underrated one — exercise the path artificially: synthetic events in production quiet hours or in replay that take the branch, so its code and data aren’t ice-cold (and its correctness isn’t unverified) when the real trigger fires. This mirrors kill-switch drills: cold paths rot, and rot is both a latency and a correctness bug.

21. Interpreting a flamegraph of a hot loop: what are the traps? Sampling profilers lie about hot loops in specific ways: cheap-but-frequent leaf functions get inflated or vanish depending on inlining (profile with debug info and check inlined frames); skid — samples attribute to instructions near, not at, the cost (use precise events, e.g. PEBS/:pp); off-CPU time is invisible entirely (a lock wait or page fault won’t show — pair with off-CPU analysis or scheduler tracing); and a flat 2% across many frames can be one cause (cache misses everywhere) that call-stack aggregation hides — event-based profiles (cache-miss-triggered sampling) regroup it. My rule: flamegraph for where, counters for why, and for tail hunting neither is enough — you need per-event tracing of the slow instances specifically, because the p99.9 samples are 0.1% of a sampled profile by construction.

22. Measure the latency of the measurement: what’s your instrumentation’s own cost and jitter? Before trusting any stage decomposition, time an empty region with the same instrumentation: back-to-back rdtsc pairs give you the floor (~20–40 cycles) and, more importantly, its distribution — occasional 1000-cycle outliers in an empty region mean SMIs, hypervisor exits, or preemption polluting all your measurements, and no amount of averaging removes a contaminant you haven’t characterized. Same discipline as instrument calibration in a lab. I also keep instrumentation always-on (Q12) precisely so its cost is a constant included in every number rather than a Heisenberg term that appears only while debugging.

23. Brutal: your engine is ‘fast’ in benchmarks but the fill quality says you’re slow to the market. Reconcile. The benchmark measures my code; fill quality measures my code plus everything I didn’t benchmark: queueing before my ingress timestamp (NIC rings, socket buffers — measure wire-to-ingress explicitly), the path after TX (my order gateway, TLS, venue-side queueing), clock error making me think I react faster than I do, and — the subtle one — selection: I only get fills when I’m late (adverse selection — when I’m fast the order I raced for is gone or I’m queued behind no one and the fill is bad news). So: instrument wire-to-wire with capture-grade timestamps to close the internal gaps, compare my order’s venue arrival time against the triggering tick’s venue publish time (venue timestamps bracket the full path including geography), and analyze fill quality conditioned on my measured latency — if fills are bad even when measurably fast, the problem is strategy, not speed. This question is really “do you know benchmarks aren’t the market.”

24. Brutal: design a latency regression gate for CI that doesn’t cry wolf. Naive gates (“p99 must be < X”) flake because CI hosts are noisy and tails need huge samples. Design: run on a dedicated, pinned, isolated benchmark host (never shared CI runners) with fixed frequency governor; feed recorded production input via the replay harness at production arrival times (determinism makes input identical across runs); compare candidate vs baseline on the same host, same run session, interleaved A/B/A/B to cancel drift; gate on distribution comparison (difference at p50/p99 beyond noise bands); require N consecutive failures before red. Two details make the gate honest. The noise bands come from A/A runs — you must first measure the harness’s own run-to-run variance before judging a candidate against it. And a Mann-Whitney-style check beats a point threshold: a rank-based test asking “are these two latency distributions actually different,” robust to outliers. And keep two gates: a tight statistical one that warns, a loose absolute one (“p99 doubled”) that blocks — warn-noise is tolerable, block-noise kills the culture of trusting the gate.

25. Brutal: ‘convince me your whole performance culture isn’t just benchmark theater.’ Three receipts. One: production numbers, not benchmark numbers, are the system of record — always-on histograms per stage per venue, conditional on load, with an external cross-check (venue timestamps / capture) so the instrument can’t grade itself. Two: the loop closes — every latency incident becomes a replayable regression test (the event log makes the exact day reproducible), and CI gates on replayed production input, so improvements are proven on the traffic that mattered, not on synthetic loops. Three: honesty artifacts — we track the known lies (coordinated omission audits of every harness, instrumentation floor characterized, sample counts printed next to every percentile) and we publish internally the numbers that got worse, because a perf culture that only produces improvements is theater by definition. Then the personal version: my p99.9 claims come with “measured at these two points, this clock chain, this sample count” attached, and I’d rather report an ugly true number than a pretty partial one — which, in this seat, is the entire job.

Further reading

• Gil Tene, “How NOT to Measure Latency” (talk, widely available) and the HdrHistogram documentation — coordinated omission from the source; watch the talk twice.
• Brendan Gregg, Systems Performance (2nd ed.) — ch. 6 (CPUs), ch. 13 (perf), off-CPU analysis; plus his flamegraph and perf c2c writeups online.
• Ulrich Drepper, “What Every Programmer Should Know About Memory” — the cache/NUMA/TLB numbers behind Q15/18/19, still the canonical text.
• Intel Software Developer’s Manual / Agner Fog’s optimization manuals — rdtsc semantics, branch and cache costs with real numbers (Q3/4/16).
• perf documentation and the kernel’s perf c2c article — false-sharing diagnosis as a tool workflow, not folklore (Q9).
• Kleppmann, DDIA ch. 1 — percentiles and tail latency framing (the SLO-flavored basics, useful vocabulary for Q7/24).