Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Linux Profiling

Before you start — this chapter leans on a handful of primer ideas:

  • Sampling vs counting profilers, PMU counters, and IPC — the two ways perf watches a program and the numbers it emits: ch00e
  • Flamegraphs — what the axes mean (width = samples, x ≠ time): ch00e
  • Caches, cache lines, and coherence — L1/L2/LLC and why a line “bouncing” between cores is expensive: ch00a
  • Kernel vs userspace, context switches, IRQs/softirq — the scheduler machinery this chapter hunts for on isolated cores: ch00b
  • Pages and the TLB — what a dTLB miss even is: ch00b

Read those first — 20 minutes there saves an hour here.

[Linux] — everything here assumes a Linux box with perf and root or perf_event_paranoid configured. This is the chapter that turns “it got slower” into a diagnosis.

perf fundamentals: counting vs sampling

perf fronts the kernel’s perf_event subsystem, which exposes two distinct modes you should never conflate:

  • Counting (perf stat): program the PMU (Performance Monitoring Unit — the CPU’s built-in event-counting hardware, ch00e) counters, run the workload, read totals at the end. Near-zero overhead, exact totals, no attribution — you learn what happened, not where.
  • Sampling (perf record): take a sample every N events — cycles by default. Each sample records the IP (instruction pointer — the address of the instruction the CPU was executing at that instant, “where in the code you were”) plus an optional call stack. Statistical attribution to code. Overhead is real but controllable via frequency. One catch: the sample can land a few instructions after the real culprit — that drift is called skid. Appending :p/:pp to an event asks the CPU’s hardware assist (Intel’s is called PEBS) to pin the sample exactly.

Workflow discipline: count first, sample second. perf stat tells you which resource is the problem; perf record on the corresponding event tells you where.

perf stat literacy

perf stat -d -- ./router --replay ticks.bin
# or attach to a running pinned process for 10s:
perf stat -p $(pidof router) -e cycles,instructions,branches,branch-misses,\
cache-references,cache-misses,LLC-loads,LLC-load-misses,dTLB-load-misses -- sleep 10

What the numbers mean on a modern x86 core (4–6 wide issue — the core can start 4–6 instructions per clock tick, so IPC around 4 is the practical ceiling):

  • IPC (instructions per cycle): the single most information-dense number.
    • IPC ≈ 0.5 or below: the core is stalled — almost always memory: cache misses, or atomics ping-ponging cache lines. Or (check separately) it’s not stalled at all but spinning: a polite poll loop executes the pause instruction, which deliberately does nothing for many cycles — the loop is waiting by design, so IPC craters with nothing actually wrong.
    • IPC ≈ 1–2: typical mixed code; nothing screaming.
    • IPC ≈ 3+: compute-dense, well-fed pipeline — L1-resident data, predictable branches. Your hot loop should look like this; if your SPSC drain loop shows IPC 0.4, the queue’s cache behavior is the story.
  • branch-misses / branches: >2–3% in a hot loop is worth attention. The CPU is an assembly line that speculatively runs ahead: it starts executing instructions past a branch before it knows which way the branch actually goes. Guess wrong and it throws away everything already on the line and restarts — each miss is ~15–20 cycles of that pipeline flush. Trading hot paths with unpredictable data-dependent branches (order type dispatch) are classic offenders — fix with branchless forms or sorting work by type.
  • cache-misses / cache-references, LLC-load-misses (LLC = last-level cache, the big L3 shared by all cores on the socket — ch00a): “memory bound” looks like: low IPC + high LLC misses + high memory-stall counters (PMU events that count the cycles the core spent waiting on memory; Intel’s catch-all is named cycle_activity.stalls_mem_any). Every LLC miss is a trip to DRAM: ~60–100ns, i.e. ~200–400 cycles — one miss costs more than an entire well-tuned queue operation.
  • dTLB-load-misses (the TLB caches virtual→physical address translations; a miss means a page-table walk — ch00b): elevated with large scattered working sets → hugepages conversation.

Counter multiplexing trap: the PMU has ~4–8 programmable counters; ask for more events and perf time-slices them and scales the results (see the [xx.x%] annotation). For precise work, run multiple passes with few events each.

perf record / report and flamegraphs

# Sample on-CPU cycles with call graphs, 99Hz to avoid lockstep with timers:
perf record -F 99 -g -p $(pidof router) -- sleep 30
perf report            # TUI; use --no-children to see self time

(Why 99Hz and not a round 100: the kernel’s own timers fire at round rates. Sample at exactly the same rate and every sample lands at the same phase of the timer cycle, showing you the same instant over and over. An odd rate drifts relative to the timers and sweeps the whole range.)

Call-graph capture has two modes and the choice matters for Rust:

  • Frame pointers (-g = --call-graph fp): each function keeps one register pointing at its caller’s stack frame, so the live call stack forms a linked list — the profiler just follows the chain from the sampled IP back to main. Cheap, reliable if frames exist — but compilers omit the frame pointer exactly to free that register for real work, and Rust/LLVM omits it by default in release. Fix in .cargo/config.toml or RUSTFLAGS: -C force-frame-pointers=yes. Cost is ~1% (one register); every serious low-latency shop just leaves it on in production builds precisely so perf works when it matters.
  • DWARF (--call-graph dwarf): when there’s no frame-pointer chain to follow, perf snapshots a chunk of raw stack memory with every sample (8KB by default) and reconstructs the call chain offline — “unwinding” — using DWARF, the standard debug-info format, whose tables describe each function’s frame layout. Works without frame pointers but is heavy, can truncate deep stacks, and slows recording. Use when you can’t rebuild.

Flamegraphs (Brendan Gregg’s stackcollapse-perf.pl | flamegraph.pl, or cargo flamegraph which wraps the whole pipeline):

cargo flamegraph --bin router -- --replay ticks.bin
# or from an existing perf.data:
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

Reading (ch00e walks the axes slowly): x-axis is alphabetical, not time; width = fraction of samples; you’re looking for wide plateaus (where cycles go) and unexpectedly-present frames (why is memmove 8% of my hot path?). Remember it shows on-CPU time only.

Rust-specific perf hygiene

  • [profile.release] debug = true (or debug = "line-tables-only"): keeps DWARF so perf maps addresses to source lines. Zero runtime cost — it only fattens the binary.
  • Symbols: the compiler encodes each function’s module path and generic parameters into one flat, linker-safe string (“mangling” — my_crate::foo becomes something like _ZN8my_crate3foo17h…); perf demangles these reasonably well, and rustfilt cleans up the rest.
  • Inline noise: release Rust inlines aggressively, so samples attribute to the caller into which code was inlined. perf report --inline (with debug info) expands inlined frames. When a flamegraph shows a fat frame for a function that “does nothing,” it’s usually full of inlined callees.
  • Iterator chains compile to loops that attribute strangely; profile with optimizations on always (a debug-build profile is fiction), and use #[inline(never)] temporarily to force attribution boundaries when you need to isolate a suspect.

perf c2c: catching false sharing

perf c2c (cache-to-cache) samples loads and stores with PEBS (the precise-sampling hardware from earlier) and looks for one specific event. The picture: core A writes a cache line, so A’s cache now holds the only current copy (MESI’s “Modified” state — false sharing and MESI are unpacked in ch00a). When core B then loads from that line, the data has to come out of A’s cache, not from RAM — each such load counts as one HITM (“hit in another core’s modified line”). A few are normal; thousands mean the line is commuting between cores — bouncing — which is exactly the false-sharing signature.

perf c2c record -p $(pidof router) -- sleep 10
perf c2c report --stats     # then the full report

Worked interpretation of the report:

  1. Top section lists cache lines sorted by HITM count. A line with thousands of Rmt/Lcl HITM events (Rmt = the other core was on the remote socket, Lcl = the local one) is contended.
  2. For each hot line, the per-offset breakdown shows which byte offsets within the 64-byte line are touched, by which code (symbol+source line), from which CPUs.
  3. True sharing: all accesses hit the same offset — that’s a genuinely shared variable (e.g., both threads on one atomic head index). Fix = algorithmic.
  4. False sharing: different offsets on the same line — e.g., offset 0x00 written by CPU 2 (producer’s head) and offset 0x08 read by CPU 14 (consumer’s cached tail copy). Fix = pad to 64B with #[repr(align(64))] wrappers — and on Intel pad to 128B: the adjacent-line prefetcher speculatively pulls in the neighboring 64B line alongside every fetch, so two variables 64B apart can still end up colliding.

The classic trading-system find: two per-thread counters declared adjacently in a struct, “independent” but sharing a line, silently taxing both threads ~100ns per increment pair. c2c is how you prove it rather than pad everything superstitiously.

ftrace: the scheduler-silence check

For an isolated hot core, the invariant is brutal and testable: no scheduler activity on that core, ever. ftrace (the kernel’s built-in tracer — no install, you drive it through /sys/kernel/tracing) tracepoints verify it:

cd /sys/kernel/tracing
echo 0 > tracing_on
echo > trace
echo sched_switch sched_wakeup > set_event
echo 1 > tracing_on;  sleep 60;  echo 0 > tracing_on
grep 'CPU:3\|cpu=3' trace     # your isolated core — this should print NOTHING

Expected result on a properly isolated core (isolated via the isolcpus boot flag or its runtime cousin, cpusets — plus IRQ affinity + nohz_full): silence, or a single switch-in of your pinned thread. Anything else — kernel background threads like ksoftirqd, kworker, or migration, or a timer tick — is a named intruder with a named fix. This check takes two minutes and settles arguments that otherwise run for days. Two cheap corroborations: perf stat’s context-switches counter must read zero over the window, and a before/after diff of that core’s column in /proc/interrupts — the kernel’s per-core interrupt scoreboard — must show no counts moving beyond any IRQ you deliberately routed there. (perf sched record / perf sched timehist gives the same data with per-event wakeup latencies when you need detail.)

Off-CPU analysis

Flamegraphs show where you burn cycles; they’re blind to where you wait. Off-CPU analysis (ch00e — profiling the time a thread spends not running) attributes blocked time (locks, page faults, I/O, involuntary preemption) to stacks. Options:

  • perf sched timehist — per-wakeup scheduling latencies.
  • offcputime from bcc/bpftrace — toolkits for running small programs inside the kernel (bpftrace gets its proper introduction in the next section); it sums blocked time by stack, kernel+user, cheaply in BPF. Renderable as an off-CPU flamegraph.

A hot path that should never block makes this a null-check: any off-CPU stack for the hot thread other than your intended park/poll site is a bug report writing itself.

bpftrace one-liners worth memorizing

bpftrace is a one-liner language over eBPF — small verified programs the kernel runs at probe points, so you can ask production questions without patching anything (ch00b).

# Syscall latency histogram for one process (should be EMPTY for a hot thread):
bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == 1234/ { @t[tid] = nsecs; }
  tracepoint:raw_syscalls:sys_exit /@t[tid]/
  { @lat = hist(nsecs - @t[tid]); delete(@t[tid]); }'

# Who is sending my hot thread signals / waking it?
bpftrace -e 'tracepoint:sched:sched_wakeup /args->pid == 1234/
  { @wakers[comm, kstack] = count(); }'

# Page faults on the hot process after warmup (want: zero):
bpftrace -e 'software:page-faults /pid == 1234/ { @[ustack] = count(); }'

The pattern: for a well-behaved hot thread, most of these tools should return nothing, and “instrumented silence” is exactly the evidence you bring to a review.

Worked session: “router p99.9 regressed 3µs — walk the diagnosis”

The setup: order router, pinned to isolated core 3, historically p99.9 = 9µs. After Tuesday’s deploy: 12µs. p50 unchanged at 4.1µs. Walk it:

1. Frame the symptom. p50 flat + tail worse = not a straight-line code slowdown; something episodic. Prior: new allocation, new fault, new interference, or new contention.

2. Cheap wide net first — counting, attached to prod replica under replay:

perf stat -e cycles,instructions,cache-misses,LLC-load-misses,page-faults,\
context-switches,dTLB-load-misses -p $(pidof router) -- sleep 30

Result: IPC 2.1 → 2.0 (noise), page-faults 0 (good — mlockall, the call that locks every page of the process into RAM so none can fault, is holding), context-switches: 0 before, 41 now. On an isolated core that number must be 0. Tail regression + nonzero context switches ≈ found the mechanism; now find the actor.

3. Name the intruder with ftrace/perf sched:

perf sched record -C 3 -- sleep 30 && perf sched timehist -C 3

Result: rdkafka-metrics thread scheduling onto core 3 every ~750ms, 2–8µs each. The deploy added a metrics client whose background thread inherited the process’s CPU mask before main() pinned the hot thread — nothing repinned the spawned thread, and the cpuset allowed it.

4. Corroborate against the latency data. Pull the event ring (the in-process log of per-event timestamps the hot path records — the observability chapter, ch11, builds it) for the p99.9 outliers: outlier timestamps line up with sched_switch events on core 3 at ~750ms cadence. Mechanism, actor, and correlation all agree — this is the bar for “diagnosed,” not “the flamegraph looked different.”

5. Fix and verify. Spawn ancillary threads with an explicit non-isolated affinity (or move pinning before any thread spawns); re-run the 60s ftrace silence check (clean); re-run the replay: p99.9 back to 9.1µs. Attach the before/after histograms and the sched trace to the postmortem.

Total wall time: ~an hour, and no code was read until step 3 named a thread. That’s the shape of counter-driven diagnosis: symptom → resource → actor → correlation → fix → re-verify.

Plain-English recap

  • perf stat vs perf record is dashboard vs profiler. Counting is your Datadog metrics view — CPU%, DB time, error rate — it names the resource that’s wrong. Sampling is the profiler flame view — it names the code. Same discipline you use today: look at the dashboard before opening the profiler.
  • A flamegraph is the Chrome DevTools flame chart with one crucial difference: the x-axis is not time. Stacks are merged and sorted, so width = “share of all samples,” and you read it by hunting wide plateaus, not left-to-right.
  • IPC is work-per-tick, like rows-per-second per connection. Low IPC means the CPU is mostly waiting — usually on memory — the way a worker with low throughput is usually blocked on I/O, not short of CPU.
  • False sharing is two services updating unrelated columns of the same DB row. Each write invalidates the other side’s cached copy, and both pay for a conflict that exists only because of physical layout. perf c2c is the tool that shows you the row and the columns.
  • The ftrace silence check is asserting your container runs nothing else. For an isolated core the correct trace output is empty — any line is a named intruder. Instrumented silence is evidence, like a clean Sentry release.
  • Off-CPU analysis is the “waiting” spans in an APM trace. A CPU profiler only sees running code; time blocked on locks, faults, or the scheduler is invisible to it — exactly like DB-wait time that never shows in a CPU profile.
  • The worked session is a Datadog-first incident review. Symptom → counters → actor → correlate with the latency spikes → fix → re-verify, and no code gets read until the mechanism has a name. You already work this way; here the counters are just closer to the metal.

Interviewer will ask

Q: IPC is 0.5 on your hot thread. What are your hypotheses and next steps? A: Stalled or spinning, so first split those: a polite poll loop is mostly pause instructions, and low IPC while waiting is by design — check whether the loop was actually doing work. If it’s genuinely stalled, it’s memory-bound until proven otherwise, and I count before I sample: LLC-load-misses and the memory-stall counters name the resource, then perf record -e cycles:pp names the loads. The final fork is three diseases, each with its own counter signature: high LLC misses with a large working set means capacity — the cache is simply too small; HITM lines in perf c2c mean coherence — cores fighting over shared lines; elevated dTLB-load-misses means TLB — and starts the hugepages conversation.

Q: Frame pointers or DWARF unwinding for production profiling? A: Frame pointers, compiled in always (-C force-frame-pointers=yes). ~1% cost for the ability to profile any incident live with cheap, reliable stacks. DWARF unwinding copies stack per sample — heavy, truncates, and I don’t want to be rebuilding binaries during an incident.

Q: How do you distinguish true from false sharing in perf c2c output? A: Look at the per-offset breakdown of the hot line: same offset hammered by multiple CPUs = true sharing (fix the algorithm); different offsets on one line = false sharing (pad/realign, 64B minimum, 128B on Intel because the spatial prefetcher pulls line pairs).

Q: Prove to me a core is actually isolated. A: Enable sched_switch/sched_wakeup tracepoints filtered to that CPU for a minute of production traffic — output must be empty. Plus: context-switches counter zero in perf stat, /proc/interrupts deltas zero for that core, and the timer tick confirmed off via nohz_full. Silence in the trace is the proof; anything else names the intruder.

Q: Your flamegraph looks identical before and after a tail regression. Why? A: Flamegraphs are on-CPU and dominated by the common case; a p99.9 event is 1 in 1000 samples — invisible. Tails need targeted tools: off-CPU analysis for blocking, sched tracing for preemption, the in-process event ring for outlier timestamps to correlate against. Sampling profilers answer “where do cycles go,” not “what happened at 14:31:07.”

Q: What does perf stat cost the target? And perf record? A: They’re the chapter’s two modes, and each cost follows from its mechanism. Counting programs the PMU’s hardware registers and reads totals at the end — the hardware counts whether or not you look, so perf stat is effectively free and safe in production. Sampling pays an interrupt per sample, so its cost scales with rate and with what each sample carries: 99Hz with frame pointers is negligible; 10kHz with DWARF copying 8KB of stack per sample very much isn’t. So the workflow discipline doubles as the safety rule: count first, sample second — and on live systems keep record frequencies low and windows short.

Q: Why do Rust release-build profiles attribute time to the “wrong” function? A: Aggressive inlining — callee cycles land in the caller frame. Keep debug = true in the release profile so DWARF inline info exists, use perf report --inline, and when isolating a suspect, #[inline(never)] it temporarily to force a real frame boundary.

Further reading

  • Brendan Gregg, Systems Performance (2nd ed.) — chapters on CPUs, perf, and methodology; and his website’s perf examples, flamegraph, and off-CPU analysis pages.
  • The perf wiki (perf.wiki.kernel.org) — canonical reference for events, call-graph modes, and perf c2c.
  • Joe Mario’s Red Hat blog write-up on perf c2c — the worked false-sharing interpretation this section compresses.
  • Brendan Gregg, BPF Performance Tools — offcputime, syscall tracing, and the bpftrace idioms above.
  • Denis Bakhvalov, Performance Analysis and Tuning on Modern CPUs (free book) — PMU literacy, top-down analysis, skid/PEBS details.

Where this goes next: profiling finds where time goes in a running system — Chapter 10 is about measuring a single function honestly: Criterion, black_box, contention regimes, and why a microbenchmark win can still be a system-level loss.