How Computers Measure Themselves

This chapter is the vocabulary and mental model for everything in Part II — clocks through the measurement lab. The performance chapters assume you already know what a cycle is, what perf reads, why an average latency number is worthless, and what a flamegraph’s x-axis means. None of that is hard — it’s just never explained. Here it is, explained.

A recurring device in this chapter: the cycle-as-second scale you adopted in the machine chapter — one cycle ≈ 0.33 nanoseconds at ~3 GHz, and we pretend one cycle takes one second:

  real time          cycles          human scale
  ─────────          ──────          ───────────
  1 ns               ~3              3 seconds
  100 ns             ~300            5 minutes
  1 µs               ~3,000          50 minutes
  10 µs              ~30,000         8 hours
  1 ms               ~3,000,000      5 weeks
  35 ms (SG→Tokyo)   ~105,000,000    3.3 years

Keep this table in your head. When the profiling chapter (ch09) says a syscall costs ~100 nanoseconds before it does any useful work, that’s the CPU taking a five-minute coffee break in the middle of your hot path.

1. How a computer knows what time it is

  ┌──────────────┐    ticks     ┌──────────────────────┐
  │   crystal    │ ───────────► │  counters built on   │
  │  oscillator  │  (a fixed    │  the tick stream     │
  │  (a quartz   │   frequency, │                      │
  │   tuning     │   e.g. some  │  ┌────────────────┐  │
  │   fork)      │   MHz base   │  │ TSC: cycles    │  │
  └──────────────┘   clock)     │  │ since boot     │  │
                                │  └────────────────┘  │
                                │  ┌────────────────┐  │
                                │  │ wall clock:    │  │
                                │  │ TSC + offset + │  │
                                │  │ NTP correction │  │
                                │  └────────────────┘  │
                                └──────────────────────┘

There is no magical “time” inside a computer. There is a crystal oscillator — a sliver of quartz that vibrates at a fixed frequency when you run current through it, exactly like the crystal in a quartz watch — and everything else is counting those vibrations. Every clock your code has ever read is an integer counter plus arithmetic.

• TSC (Time Stamp Counter) — a 64-bit register inside the CPU that increments once per cycle since boot. — It’s the odometer of the CPU: it only goes up, it doesn’t know what year it is, and it’s absurdly cheap to read. — The instruction that reads it, rdtsc, costs roughly 20 cycles — 20 seconds on our human scale, versus the five-minute coffee break of a syscall. — Trading systems timestamp with the TSC because when your whole budget is a few microseconds, you cannot spend hundreds of nanoseconds asking what time it is; the latency-methodology and observability chapters (ch08, ch11) assume TSC timestamping as the default.

• Monotonic clock vs wall clock — a monotonic clock only moves forward and measures durations; a wall clock tells you the calendar time and can jump backwards or forwards when it’s corrected. — This is exactly performance.now() vs Date.now() in the browser/Node: you already know never to compute a duration from two Date.now() calls, because NTP (below) can step the clock mid-measurement and give you a negative latency. — Same rule, one layer down: durations come from the TSC, timestamps for the audit log come from the wall clock, and confusing the two produces impossible latency numbers.

• vDSO (virtual Dynamic Shared Object) — a small page of kernel code mapped into every process so that “syscalls” like clock_gettime run as ordinary function calls, no kernel transition. — When you call Date.now() in Node or Instant::now() in Rust, you end up in the vDSO, which reads the TSC and applies the kernel’s calibration (cycles → nanoseconds, plus the NTP-corrected offset). No mode switch, ~20–30 ns. — This matters because it means timestamping can be cheap — and because profiling output in the profiling chapter (ch09) will show [vdso] frames and you should know they’re clock reads, not a bug.

So the stack is: quartz vibrates → CPU counts vibrations in the TSC → kernel calibrates TSC-ticks-per-nanosecond → vDSO exposes that as clock_gettime → your language runtime wraps that as Instant::now() / Date.now(). Every timestamp you’ve ever logged was this pipeline.

2. Why clocks disagree across machines

   machine A                          machine B
   ┌─────────────┐                    ┌─────────────┐
   │ crystal:    │                    │ crystal:    │
   │ 3.000000 GHz│                    │ 2.999994 GHz│   ← manufacturing spread +
   │ (nominal)   │                    │ (2 ppm slow)│     temperature
   └──────┬──────┘                    └──────┬──────┘
          │  drift apart ~2 µs every second │
          ▼                                 ▼
   "event at 09:00:00.000100"        "event at 09:00:00.002300"
                    │                        │
                    └── which happened first? unknowable ──┘

Two crystals are never identical. Drift is measured in ppm (parts per million) — a 10 ppm crystal gains or loses 10 µs every second, which is ~1 second per day. Temperature changes the vibration frequency, so drift isn’t even constant. Left alone, two servers’ clocks walk away from each other indefinitely. Three technologies fight this, at three price points:

• NTP (Network Time Protocol) — the standard internet protocol where your machine periodically asks time servers what time it is and slews its clock toward the answer. — It’s the default on every Linux box you’ve ever deployed; it’s what keeps your GCP VMs roughly on time. — Accuracy: milliseconds, sometimes tens of milliseconds, because the correction rides over ordinary software networking and assumes the path is symmetric. — For trading, NTP is not measurement-grade: a millisecond of clock error is a thousand times larger than the latencies you’re trying to measure.

• PTP (Precision Time Protocol) — a time-sync protocol where the network hardware itself stamps sync packets at the moment they touch the wire, removing all the software-stack noise from the measurement. — Think of the difference between measuring an API’s latency from your app code (includes your event-loop lag, GC pauses, scheduler noise) versus having the load balancer stamp packets at ingress: PTP is the load-balancer version. — Accuracy: sub-microsecond, often tens of nanoseconds with good hardware. — Exchanges and regulators (MiFID II in Europe, for one) require trading timestamps synced at this grade; the clocks chapter (ch07) covers deploying it.

• PHC (PTP Hardware Clock) — a real clock that lives on the network card itself, which PTP disciplines directly. — The NIC keeps its own clock instead of asking the CPU. — This is what makes hardware packet timestamping possible: the NIC stamps a packet with its clock at wire-touch time, and the clocks and latency-methodology chapters (ch07, ch08) lean on this for honest one-way latency numbers.

Why you care, in payments terms: you have debugged an incident by lining up your logs against Stripe’s dashboard timestamps, and you know the pain — if the two providers’ servers disagree by even a second, you cannot order the events; you can’t tell whether the webhook arrived before or after your retry fired. Now shrink that to trading: “our tick-to-trade is 200 µs” computed from a timestamp on machine A minus a timestamp on machine B is pure fiction if those machines’ clocks differ by 2 ms — the clock error is 10× the thing being measured. Cross-machine latency numbers are only as good as the clock sync underneath them, which is why an entire chapter about clocks (ch07) exists before any chapter about measuring latency.

3. What a profiler actually does

  SAMPLING                                INSTRUMENTATION
  ─────────                               ───────────────
  timer fires 999×/sec                    every function wrapped:
       │                                  fn foo() {
       ▼                                    let t = now();   ← added
  ┌───────────────┐                         ...real work...
  │ interrupt!    │                         record(now()-t); ← added
  │ what's on the │                       }
  │ stack RIGHT   │
  │ NOW? write it │                       exact counts & times,
  │ down. resume. │                       but the measuring
  └───────────────┘                       changes the measured
  statistical picture,
  ~zero distortion

• Sampling profiler — a profiler that interrupts the program N times per second (say 999 Hz), records the current call stack, and resumes; after a few seconds, the statistics of those snapshots tell you where time goes. — This is exactly what the Chrome DevTools Performance tab does when you hit record, and exactly what Datadog’s continuous profiler does to your Node services in production: nobody rewrote your code, they just photographed the stack a few thousand times. — If parse_order appears in 40% of samples, parse_order consumes ~40% of CPU time. It’s polling, not tracing — statistically true, individually meaningless.

• Instrumentation — modifying the code (manually or automatically) so every function entry/exit records a timestamp. — This is you sprinkling console.time() / console.timeEnd(), or an APM agent monkey-patching every HTTP and database call. — You get exact call counts and per-call durations… but the recording itself costs time, and for functions that run in nanoseconds, the measurement can cost more than the function. Heisenberg with a stopwatch.

Why sampling wins in production, and especially in trading: the observer effect. Wrapping a 50 ns function with two clock reads (~40 ns) doubles its cost and — worse — changes inlining, code layout, and cache behavior, so you’re now profiling a different program. A sampling profiler perturbs the target a few microseconds per second, statistically invisible. The rule the profiling chapter (ch09) assumes: sample to find where time goes; instrument only the specific boundaries you’ve already decided to care about (and the observability chapter (ch11) shows how trading systems instrument those boundaries cheaply). You already live by this rule — you reach for the APM flame view first and add custom spans second — this is the same discipline with better vocabulary.

4. The PMU: the APM agent baked into the silicon

  your code runs ──►  CPU core
                      ┌──────────────────────────────────────┐
                      │  execution units                     │
                      │                                      │
                      │  PMU (a few hardware counters):      │
                      │   ┌────────────────────────────┐     │
                      │   │ cycles          1,203,441  │     │
                      │   │ instructions      601,882  │     │
                      │   │ cache-misses       14,207  │     │
                      │   │ branch-misses       3,190  │     │
                      │   └────────────────────────────┘     │
                      └──────────────────────────────────────┘
                                    ▲
                                    │  read by `perf stat`

• PMU (Performance Monitoring Unit) — a small block of circuitry inside every CPU core containing a handful of programmable hardware performance counters: registers you can point at an event type (“count every cache miss”) and read later, at zero cost to the running code. — It is a Datadog agent implemented in silicon: always on, free to run, and it answers “what was my program actually doing?” one layer below any software. — perf stat ./your-program is just: program the PMU counters, run the program, print the counters. The profiling and microbenchmark chapters (ch09, ch10) use this constantly.

• IPC (Instructions Per Cycle) — instructions retired divided by cycles elapsed: how much work the CPU completed per tick. — A modern core can retire 4–6 instructions per cycle when everything flows; think of it as throughput utilization for the silicon. — Rough field guide: IPC ≈ 0.5 means the core spent most cycles stalled, almost always waiting on memory (a cache miss is a trip to RAM: ~100 ns, ~300 cycles — a five-minute wait on our human scale, during which the core does nothing). IPC ≈ 3 means the pipeline is humming. — This is precisely the diagnosis you already make with APM: your endpoint is slow, the trace shows the handler spent 40% of wall time awaiting Postgres — the CPU wasn’t busy, it was waiting. IPC is that same busy-vs-waiting split, one layer down: not “my process waits on the DB” but “my instructions wait on RAM.” Trading systems obsess over it because the fix differs completely — low IPC means restructure your data for cache locality (the microbenchmarks chapter, ch10), high IPC with slow results means you’re doing too much work.

One habit to build now: when a hot loop is slow, your first question is no longer “what is it doing?” (profiler) but “is it computing or waiting?” (perf stat, look at IPC and cache misses). Two tools, two different questions.

5. Flamegraphs: you already read these

  x-axis: fraction of samples (NOT time order — alphabetical merge!)
  y-axis: stack depth (who called whom)

  ┌──────────────────────────────────────────────────────┐
  │                      main  100%                      │  ← +7% self
  ├───────────────────────────────┬──────────────────────┤
  │      on_market_data  62%      │   send_order  31%    │
  ├───────────────┬───────────────┼──────────────────────┤
  │ parse_msg 40% │ update_book   │   encode_fix  29%    │
  ├───────────────┤     22%       ├──────────────────────┤
  │ memcpy    35% │               │   checksum    12%    │
  └───────────────┴───────────────┴──────────────────────┘
       ▲
       width of "memcpy" = it was on-stack in 35% of samples

• Flamegraph — a visualization that merges thousands of sampled stacks into one picture: each box is a function, its width is the fraction of samples it appeared in (i.e., its share of CPU time), and boxes stack vertically by caller→callee. — You read these already: the React Profiler’s flame chart and the Chrome Performance tab are the same drawing. The one trap for people coming from Chrome: in Chrome’s timeline view, left-to-right is time order. In a classic flamegraph it is not — siblings are sorted alphabetically and merged, so left-to-right means nothing. Width is everything. — Diagnosis is the same skill you use on React renders: wide box you didn’t expect = the surprise cost; wide flat-topped box (nothing above it) = the leaf actually burning CPU; tall narrow towers = deep call chains that cost little. In the picture above, the actionable fact is memcpy at 35% — a third of the CPU budget is copying bytes, which is why the microbenchmarks chapter’s zero-copy discussion (ch10) exists.

The profiling chapter (ch09) generates these from perf record; the reading skill transfers unchanged from your React profiler experience.

6. Percentiles, and why averages lie about latency

Ten payment authorizations, milliseconds:

  20, 20, 21, 21, 22, 22, 23, 23, 24, 804

  mean   = 100 ms   ← describes NONE of the ten requests
  median = 22 ms    ← describes the experience
  max    = 804 ms   ← describes the incident

The mean says “typical request: 100 ms.” No request took anything like 100 ms — nine customers had a snappy 22 ms experience and one sat through 804 ms and possibly abandoned checkout. Latency distributions are skewed: there’s a floor (physics) but no ceiling (a GC pause, a lock, a page fault can stretch one request arbitrarily). The mean is dragged by the tail while describing nobody. This is why latency work speaks percentiles:

• Percentile — the value below which that fraction of samples fall: sort all samples ascending; p50 is the middle one, p99 is the value 99% of samples beat, p99.9 the value 99.9% beat. — In the dataset above p50 = 22 ms, p99 ≈ 804 ms. — You already run this instinct in payments: median auth time is a vanity metric; p99.9 is what actually trips a customer’s checkout timeout and loses the sale.

The web-vs-trading difference, and it changes everything about Part II’s measurement chapters: in a web system, one slow request is one mildly grumpy user — you manage p99 and shrug at max. In trading, one slow order is real money: the market moved while your order was in flight, and you got filled at a worse price (or an arbitrageur got there first — the industry phrase is adverse selection). The tail isn’t a quality metric, it’s a P&L line item. So trading latency reporting is p50 / p99 / p99.9 / p99.99 / max, and the max is read first, not last. When the latency-methodology chapter (ch08) spends pages on tail methodology, this is why.

7. Histograms: how you store a billion latencies

  naive: keep every sample            histogram: keep bucket counts
  ┌──────────────────────┐            ┌───────────────────────────┐
  │ 1,000,000,000 × 8B   │            │  bucket        count      │
  │ = 8 GB, growing,     │            │  [1.0–1.1 µs)  114,882    │
  │ must sort to get     │            │  [1.1–1.2 µs)  903,415    │
  │ percentiles          │            │  ...                      │
  └──────────────────────┘            │  [95–100 ms)   1          │
                                      │  fixed memory, percentile │
                                      │  = walk buckets, O(1)-ish │
                                      └───────────────────────────┘

• Histogram (as a latency data structure) — instead of storing every sample, pre-define value ranges (buckets) and store one counter per bucket; recording a sample is bucket[index]++, and any percentile is recovered by walking the buckets until you’ve passed the right fraction of the total count. — This is exactly what Prometheus histogram metrics and Datadog distribution metrics do internally — you’ve been consuming bucketed percentiles every time you read a dashboard; now you know why the p99 line looks slightly quantized. — Recording is nanoseconds and allocation-free, which matters when the recording happens on the hot path (the observability chapter, ch11).

• HdrHistogram (High Dynamic Range Histogram) — the standard implementation (Gil Tene’s), which spaces buckets logarithmically: fine-grained buckets at small values, coarser at large ones, maintaining a fixed relative precision (e.g. every recorded value within 0.1% of truth) across values from nanoseconds to minutes — six orders of magnitude — in a fixed few-hundred-KB footprint. — Same idea as logarithmic axes on your Grafana charts: equal ratios get equal resolution, because the difference between 1 µs and 2 µs matters as much as between 1 ms and 2 ms. — Every serious latency toolchain (and the methodology of ch08) speaks HdrHistogram natively; when you see .hgrm files or “hiccup charts,” this is what’s underneath.

8. Coordinated omission: the interview filter

This one is worth over-learning — it’s a classic interview question in latency-sensitive shops precisely because it separates people who have thought about measurement from people who have run ab once. Learn it well enough to teach it back — here it comes.

The setup. Your load generator is closed-loop: send a request, wait for the response, then send the next. Target rate: one request per millisecond. The system under test hums along at 100 µs per response… then stalls completely for 100 ms (GC pause, lock, whatever). Ten seconds of test, 10,000 intended requests.

  intended:  req every 1 ms, no matter what
  ────────────────────────────────────────────────────────────►
  t=0        t=4000ms                        t=4100ms
  │ ││ ││ ││ │╳  ← stall begins              │ ││ ││ │
             │                               │
  what a closed-loop generator does:         │
             sends 1 request at t=4000,      │
             BLOCKS waiting for it,          │
             gets response at t=4100,        │
             records ONE sample: 100 ms      │
             ...resumes as if nothing happened

  what a real client population would have experienced:
             req sent t=4000 → waited 100 ms
             req sent t=4001 → waited  99 ms
             req sent t=4002 → waited  98 ms
             ... 100 requests, waits 100,99,98,…,1 ms ...
             req sent t=4099 → waited   1 ms

The generator coordinated with the system it was measuring: the moment the system got slow, the generator politely stopped sending. The stall — during which 100 requests should have been sent and would have queued — produced one bad sample instead of one hundred. The measurement omitted precisely the data from the period being measured. Hence: coordinated omission — the systematic under-counting of bad samples that happens when a blocked load generator (or any measurement loop) stops generating during the very stalls it exists to detect.

The numbers. Normal latency 0.1 ms; one 100 ms stall in a 10-second run at 1 req/ms:

Read the p99.9 row twice. The naive report says “p99.9 = 0.1 ms” — three-nines excellence — for a system that went completely dark for a tenth of a second. The corrected report says p99.9 ≈ 90 ms, which is the truth: during that window, a real order (or a real checkout) sent at any point in the stall would have waited up to 100 ms. The max was never wrong — another reason trading reads max first — but every percentile between p50 and max was fabricated by the measurement methodology.

You have seen the production version of this bug: a webhook consumer falls over, your dashboard averages only the requests that completed, and the graph looks fine while a queue of unsent retries piles up out of frame. Same disease: measuring only what the sick system allowed to happen.

The fixes (detailed in the latency-methodology chapter, ch08): open-loop load generation — send on schedule from an independent timeline whether or not responses came back; or correct after the fact — when a sample exceeds the intended send interval, synthesize the missing samples (100 ms observed at 1 ms intervals ⇒ also record 99, 98, … 1 ms). HdrHistogram ships this correction built-in (recordValueWithExpectedInterval), which is not a coincidence — same author, same war.

The one-sentence version for interviews: “Coordinated omission is when a closed-loop load generator stops sending during a stall, so a 100 ms freeze that should contribute a hundred bad samples contributes one, and every percentile below max becomes fiction; you fix it with open-loop generation or expected-interval correction.”

9. Jitter: variance is the enemy, not slowness

  low jitter (good even if slower):     high jitter (worse even if "faster on average"):

  µs                                    µs
  12 ┤                                  12 ┤        ╷            ╷
  10 ┤                                  10 ┤        │            │
   8 ┤────────────────────              8 ┤        │       ╷    │
   6 ┤                                   6 ┤   ╷    │   ╷   │    │
   4 ┤                                   4 ┤───┴────┴───┴───┴────┴──
   2 ┤                                   2 ┤
     └────────────────────                 └─────────────────────────
      every order: 8 µs                     usually 4 µs… sometimes 12
      you can promise 8 µs                  you can promise nothing

• Jitter — variance in latency: the spread between your typical and your worst, as opposed to the typical itself. — It’s the difference between an API that always answers in 80 ms and one that usually answers in 40 ms but sometimes takes 400 ms — every retry policy, timeout, and SLA you’ve ever written was really about jitter, not speed. — Trading systems will happily trade a slower consistent path for a faster jittery one, because strategy decisions are priced assuming an execution latency; the orders that miss that assumption are the ones that lose money. Much of the kernel-tuning chapter (ch03) and the measurement chapters (ch10–ch12) is jitter hunting, not speed hunting.

The standard suspects, each expanded in later chapters — memorize the lineup, because “what causes latency jitter on Linux?” is another interview staple:

The trading-systems countermeasures — pre-allocate everything, pre-fault memory, pin threads to isolated cores, steer interrupts elsewhere, lock the frequency — are all “remove a suspect from the lineup,” and they’re exactly what the kernel-tuning (ch03) and microbenchmark-hygiene (ch10) chapters do one suspect at a time.

What you can now read

• Clocks & time sync (ch07) — you know what the TSC, NTP/PTP/PHC, and monotonic-vs-wall distinctions are; that chapter is now deployment detail, not new concepts.
• Latency measurement methodology (ch08) — you can read percentile tables, HdrHistogram output, and coordinated-omission corrections without stopping.
• Linux profiling (ch09) — perf stat is “read the PMU,” perf record is “sampling profiler,” flamegraphs are React profiler charts; the chapter is now tooling walkthrough.
• Microbenchmarks (ch10) — IPC, cache misses, jitter suspects, and observer effect are the entire vocabulary of that chapter.
• Hot-path observability (ch11) — you know why recording is histograms-not-samples and why instrumentation must be nanosecond-cheap.
• Lab: measurement (ch12) — the lab has you produce every artifact this chapter described: a flamegraph, a perf stat reading, an HdrHistogram, and a coordinated-omission demonstration.