Clocks: TSC, PTP, and Lying Timestamps

Before you start — this chapter leans on a handful of primer ideas:

• TSC / rdtsc — the CPU’s built-in cycle counter and the instruction that reads it, the rawest clock you have: ch00e
• Crystals and clock drift — why every clock is a vibrating crystal that runs slightly fast or slow, and what NTP/PTP do about it: ch00e
• Kernel vs userspace and syscalls — why “calling the kernel for the time” used to be expensive and how the vDSO fixed it: ch00b
• The NIC and the PHY — the network card hardware that can stamp packets the instant they hit the wire: ch00a
• Tick-to-trade — the market-data-in to order-out latency this chapter teaches you to decompose: ch00f

Read those first — 20 minutes there saves an hour here.

You have a ~10ms system — the web-stack trading setup you’ve run in production, where venue RTT dominated — and you want to make claims about microseconds. Every one of those claims rests on a clock, and most engineers have never audited theirs. This chapter is about knowing exactly what your timestamps mean, what they cost to take, and when they lie.

The clock hierarchy on one box

Your machine does not have “a clock.” It has one raw counter, then layers of kernel arithmetic on top of it, then the NIC’s own watch off to the side. The raw counter is the TSC (the CPU’s cycle counter, read by the rdtsc instruction with no kernel involved — ch00e): it counts ticks since reset, costs a few nanoseconds to read, and has no idea what time it is. The kernel takes that counter and, per clock ID, applies a scale and an offset to turn ticks into nanoseconds — that’s the whole clock_gettime family. And the NIC keeps a physical clock of its own, so a packet can be stamped the instant it touches the wire rather than whenever your software got around to noticing it.

What separates the kernel’s clocks is adjustment policy. A clock is disciplined when a daemon keeps nudging it so it tracks some reference — like a thermostat, forever measuring the error and correcting toward the target. The nudge comes in two flavors: slewing (gently stretch or shrink the length of the second until the clock catches up — time never jumps) and stepping (yank the hands straight to the right time — a discontinuous jump that can even go backwards). CLOCK_MONOTONIC_RAW is the undisciplined hardware rate; CLOCK_MONOTONIC is slewed but never stepped; CLOCK_REALTIME can be both.

Every kernel clock here is built on the TSC on a modern x86 Linux box (clocksource=tsc). clock_gettime is the kernel reading the TSC in the vDSO and applying a scale/offset. When you call rdtsc yourself, you are just cutting out the middleman — and losing the calibration the kernel maintains.

The whole hierarchy as a summary table, cheapest/rawest to most expensive/most-meaningful:

Remaining terms in that table, in one clause each: vDSO (the kernel page mapped into your process that keeps clock reads out of the kernel — ch00b); NTP (network time protocol — sync over the ordinary network, millisecond-class accuracy — ch00e); “ordered” / lfence (a fence instruction that stops the CPU reordering the read relative to your work — the serialization section below); PHY/MAC (the NIC’s physical-layer and link-layer hardware, the last silicon a packet touches before the wire — ch00a).

rdtsc: what you must know before using it

Invariant TSC

Old CPUs ticked the TSC at the current core frequency, so it stopped in sleep states and changed speed with turbo. Every CPU you will trade on since roughly ~2008 (Intel Nehalem) has an invariant TSC: it ticks at a fixed frequency (the “TSC frequency”, near the base clock, e.g. 2.994 GHz on a “3.0 GHz” part) regardless of P-states, C-states, or turbo (the CPU’s frequency-scaling and sleep states — ch00a).

Verify, don’t assume:

• CPUID leaf 0x80000007, EDX bit 8 = invariant TSC (cpuid is the x86 instruction that reports the CPU’s features; a “leaf” is just which page of answers you ask for).
• Linux: grep -o 'constant_tsc\|nonstop_tsc' /proc/cpuinfo | sort -u — you want both.
• cat /sys/devices/system/clocksource/clocksource0/current_clocksource should say tsc. If it says hpet or acpi_pm, the kernel demoted the TSC because it observed it misbehaving — investigate before trusting any timing on that box.

Frequency: never use /proc/cpuinfo MHz

The “cpu MHz” field is the current core frequency, which turbos and idles all over the place. The TSC frequency is a different, fixed number. Get it from:

• dmesg | grep 'tsc:' — kernel prints the refined calibration, e.g. tsc: Refined TSC clocksource calibration: 2994.374 MHz.
• CPUID leaves 0x15/0x16 (crystal clock ratio) on Skylake+.
• Or calibrate it yourself against CLOCK_MONOTONIC_RAW (code below) — this is what you should do anyway, because it makes your code robust and is a one-time startup cost.

Serialization: rdtsc is not ordered

rdtsc is just another instruction to the out-of-order engine (modern CPUs execute instructions in whatever order keeps the pipeline busy, not program order — ch00a). The CPU is free to execute it before the work you’re trying to time has finished, or hoist work from after it to before it. For coarse pipeline stamps (microseconds apart) this doesn’t matter — a few nanoseconds of skid is noise. For microbenchmarks of 20-cycle operations it destroys the measurement.

The modern discipline (Intel’s recommendation since the Paoloni whitepaper era) uses lfence (a “load fence” instruction — a barrier that stops the CPU reordering instructions across it; the out-of-order engine is ch00a’s territory):

start:  lfence; rdtsc          ; lfence stops earlier insns' results arriving late
end:    rdtscp                 ; waits for all prior insns to retire (fully
                               ; finish and commit their results)...
        lfence                 ; ...and lfence stops later insns starting early

rdtscp is only partially serializing — it waits for prior instructions but does not fence subsequent ones, hence the trailing lfence. The old recipe — issuing a cpuid instruction as the barrier, because it fully serializes the pipeline — works but cpuid costs hundreds of cycles and has variable latency; use lfence.

In Rust:

#![allow(unused)]
fn main() {
#[cfg(target_arch = "x86_64")]
#[inline(always)]
pub fn rdtsc_ordered() -> u64 {
    use core::arch::x86_64::{__rdtscp, _mm_lfence, _rdtsc};
    unsafe {
        _mm_lfence();
        let t = _rdtsc();
        _mm_lfence();
        t
    }
}
}

For pipeline stamps taken microseconds apart, plain _rdtsc() without fences is fine and is what you want in the hot path — 2–4ns, no pipeline drain.

Per-core sync caveats

On a single modern socket, cores’ TSCs are synchronized at reset and the hardware keeps them together; the kernel checks this at boot (tsc: Synchronized across N CPUs) and via IA32_TSC_ADJUST (a per-core register recording any offset software applied to that core’s TSC — nonzero means someone shifted it). Practical rules:

• Same socket, pinned threads: comparing TSC values across cores is fine to within a few cycles. This is what makes cross-thread pipeline stamping work.
• Multi-socket: usually still synchronized (same reset signal), but verify; NUMA-era (ch00a — multiple CPU sockets, each with its own local memory) horror stories exist. Run the kernel’s check, or measure a ping-pong round trip and confirm one-way ≈ RTT/2 in cycles both directions — RTT/2 is legitimate here because a same-box ping-pong is symmetric by construction, unlike the cross-network case skewered later in this chapter.
• VMs: all bets off. Live migration rewrites TSC offsets; some hypervisors trap rdtsc (intercept the instruction and emulate it in software — slowly, which is the whole reason VM timing numbers are worthless). If you’re timing inside a VM, you’re characterizing the hypervisor.
• Unpinned threads: a thread migrating mid-measurement between synchronized cores is fine; between unsynchronized sockets it’s not. One more reason you pin.

clock_gettime and the vDSO

clock_gettime(CLOCK_MONOTONIC) does not make a syscall on any kernel you’ll run: the vDSO maps a page with the TSC scale/offset into your process and the “call” is a userspace function that does rdtsc, multiply, shift. ~20–30ns. Verify with strace (the Linux tool that prints every syscall a process makes): you should see no clock_gettime syscalls in steady state. (Trap: CLOCK_MONOTONIC_RAW wasn’t in the vDSO until around Linux 4.16 — it fell back to a real syscall — so on an ancient kernel that “cheap” call is 100ns+ and, being a kernel entry, a moment where the scheduler may take your core away.)

Distinctions that matter:

• CLOCK_REALTIME: wall time. NTP can step it backwards. Never compute a duration from it. Its only job is correlating with the outside world.
• CLOCK_MONOTONIC: never steps backwards, but NTP slews its rate (stretches or shrinks the second by up to 500ppm) to chase true time. Fine for timeouts; a subtle lie for precision measurement — your “1ms” might be 0.9995ms.
• CLOCK_MONOTONIC_RAW: the undisciplined hardware rate. This is what you calibrate the TSC against, because both are lies in the same direction. Mechanically: the TSC and CLOCK_MONOTONIC_RAW are derived from the same physical crystal on the board, so if that crystal runs 30ppm fast, both run 30ppm fast together and the ratio between them stays fixed. A calibration against CLOCK_MONOTONIC would chase NTP’s slew instead.

So is gettimeofday in a hot path a sin? Less than folklore says — it’s a ~25ns vDSO call now, not a 1µs syscall. But the standard in trading systems is raw rdtsc in the hot path anyway: it’s 5–10× cheaper, it’s immune to NTP slew, and cycles are the natural unit when you’re also reading performance counters. Take cycles hot, convert to nanoseconds cold.

Calibrating cycles → nanoseconds in Rust

#![allow(unused)]
fn main() {
use std::time::Instant;

/// TSC ticks per nanosecond, measured at startup. Do this once, on a pinned
/// thread, and sanity-check against dmesg's "Refined TSC" value.
pub fn calibrate_tsc_ghz() -> f64 {
    let mut best = f64::MAX;
    for _ in 0..5 {
        let t0 = Instant::now();
        let c0 = rdtsc_ordered();
        // Long enough to swamp the ~20ns measurement edges: 50ms.
        while t0.elapsed().as_millis() < 50 {
            std::hint::spin_loop();
        }
        let c1 = rdtsc_ordered();
        let ns = t0.elapsed().as_nanos() as f64;
        let ghz = (c1 - c0) as f64 / ns;
        // Take the minimum-noise (most consistent) sample.
        if ghz < best { best = ghz; }
    }
    best
}
// Usage: ns = cycles as f64 / ghz. Store 1.0/ghz and multiply in the cold path.
}

Two production notes: (1) do the division off the hot path — stamp raw cycles, convert during aggregation; (2) recheck the calibration periodically in long-running processes and alarm if it moves — a shifting apparent TSC rate means the crystal feeding your reference clock has heat-shifted (crystals speed up and slow down with temperature — ch00e) or there’s a clocksource problem.

Cross-machine time: where the real lies live

Everything above was one box. The moment your latency claim spans two machines — “exchange gateway to our server in 40µs” — you need both clocks to agree, and this is where most published numbers are fiction.

NTP: ±milliseconds, and it won’t tell you

NTP over a LAN under good conditions gets you within tens to hundreds of microseconds; over anything congested or asymmetric, single-digit milliseconds of error is routine — and NTP happily reports itself “synchronized” the whole time. If your one-way latency claim is 50µs and your clock error budget is ±1ms, your measurement is 100% noise. Any cross-machine latency figure derived from NTP-disciplined CLOCK_REALTIME deserves exactly zero trust below the millisecond.

The RTT/2 fallacy

The tempting dodge: measure round trip with one clock, divide by two. This assumes the path is symmetric. In trading infrastructure it reliably isn’t: different fiber routes in each direction, asymmetric queuing (your order enters a busy gateway, the ack returns on an idle path), different switch hop counts, NIC send vs receive path costs. Asymmetries of 2:1 are common. RTT is a real, useful number — quote it as RTT. One-way numbers require synchronized clocks, full stop.

PTP: tens of nanoseconds

IEEE 1588 (PTP) with hardware timestamping gets machines within tens to hundreds of nanoseconds of each other:

• The NIC’s PHY stamps sync packets on the wire (removing OS jitter from the sync loop entirely), maintaining a PHC (PTP Hardware Clock — an actual clock that lives on the NIC itself, separate from the system clock — ch00e).
• ptp4l disciplines the PHC to the grandmaster — the one reference clock in the network that everyone chases; phc2sys disciplines the system clock to the PHC. Boundary/transparent clocks in the switches correct for queuing delay on the sync path itself.
• Software-timestamped PTP is a halfway house: ~µs-tens-of-µs accuracy. Better than NTP, not good enough to decompose a 10µs path.

The whole sync chain on one screen:

 GRANDMASTER ──► switch ──────────────► NIC's PHC ──────────► system clock
 (the reference   (boundary/transparent   (clock chip on        (what clock_gettime
  everyone         clock: corrects for      the NIC)              reads)
  chases)          its own queuing delay)
                          └── ptp4l disciplines ──┘  └── phc2sys disciplines ──┘

Colos serving exchanges run PTP infrastructure precisely because clients demand defensible one-way numbers. If you’re asked “how would you verify a vendor’s claimed one-way latency” — the answer is PTP-disciplined hardware timestamps at both ends, or you refuse to state one-way numbers and quote RTT.

NIC hardware timestamps

Independent of PTP, the NIC can stamp your traffic: enable SO_TIMESTAMPING with SOF_TIMESTAMPING_RX_HARDWARE / TX_HARDWARE (config via ethtool -T to check capability, hwtstamp_config to enable) and each packet arrives with the PHC time it hit the wire, delivered in the socket’s error queue / control messages (a side channel on the socket where the kernel attaches per-packet metadata — despite the name, nothing has gone wrong). Kernel-bypass stacks (Onload, ef_vi, DPDK) surface the same hardware stamps directly. This is the only timestamp that is not polluted by interrupt latency, softirq scheduling (the kernel’s deferred packet-processing work — ch00b), or your process getting around to calling recv.

Timestamping discipline for a trading pipeline

The four stamps that decompose tick-to-trade, and the clock each uses:

The three deltas: (1→2) is your network stack + wakeup cost — this is the delta where kernel vs bypass shows up, and the one most people have never measured. (2→3) is your code — the only part your Rust hot path controls. (3→4) is the send-side stack. End-to-end (1→4) is the number you quote; the decomposition is the number you debug. To mix PHC stamps with rdtsc stamps you need the PHC↔TSC relationship, which phc2sys maintains (or sample both at effectively one instant yourself — read clock A, read clock B, read A again, and pair B with the midpoint of the two A readings — and keep the offset).

Anyone who quotes tick-to-trade without saying which two of these points they measured between is quoting an incomparable number. “1.2µs tick-to-trade” measured 2→3 is a completely different claim from 1→4.

Plain-English recap

• The clock hierarchy is like timestamp columns in a payments system. A payment has a created_at from the client, one from your API server, one from the database, and one from the PSP — four different clocks meaning four different things. rdtsc, CLOCK_MONOTONIC, CLOCK_REALTIME, and the NIC stamp are the same idea; the sin is reading one column and thinking it means another.
• rdtsc vs CLOCK_REALTIME is performance.now() vs Date.now(). One is a raw monotonic tick counter that’s cheap and never jumps; the other is wall time that NTP can yank around. You’d never compute a duration from Date.now() across a DST change — CLOCK_REALTIME durations are the same bug at microsecond scale.
• NTP error swamping your measurement is a reconciliation problem. It’s like diffing your ledger against a PSP settlement report where each side stamped events with its own clock, off by an unknown couple of seconds: you cannot order events across the two systems, no matter how precise each timestamp looks. PTP is both sides agreeing to a shared, audited clock before anyone compares timestamps.
• The RTT/2 fallacy is “webhook delivery time = API round trip ÷ 2”. Your request went out over one path and the webhook came back over a completely different one (different queues, different retries). Halving the round trip assumes symmetry that isn’t there.
• NIC hardware timestamps are the PSP’s own received_at. Stamped at the front door, not when your worker finally pulled the job off the queue. Every software timestamp includes “how long until my process got around to it”; the wire stamp doesn’t — that’s why it’s ground truth.
• The four tick-to-trade stamps are spans in an APM trace. Wire-in → thread has it → decision made → wire-out is exactly a Datadog trace of gateway → worker → handler → response. Quoting “latency” without saying which two spans you measured between is as meaningless in trading as it is in an APM dashboard.

Interviewer will ask

Q: Why can’t you measure a 50µs latency with SystemTime::now()? A: SystemTime is Rust’s Date.now() — it reads CLOCK_REALTIME, the wall clock NTP is allowed to yank around. NTP can slew or even step that clock mid-measurement, so end−start isn’t a duration — it’s two wall-time readings with an adjustment of unknown size hiding between them. And that adjustment is millisecond-class, while the thing being measured is 50µs — the error bar is bigger than the measurement. So single-box durations come from the TSC or CLOCK_MONOTONIC_RAW, the clocks nothing yanks; cross-machine one-way numbers need PTP or they’re fiction.

Q: What must be true before you trust raw rdtsc as a clock? A: Each condition rules out one specific way the counter lies. It must tick at a fixed rate through turbo and sleep — invariant TSC — checked via constant_tsc and nonstop_tsc in /proc/cpuinfo (CPUID leaf 0x80000007 if you want the hardware’s own word). The kernel must still trust it too: if current_clocksource says hpet instead of tsc, the kernel caught the TSC misbehaving and demoted it. I need its real frequency, not a guess — dmesg’s refined calibration or my own against CLOCK_MONOTONIC_RAW, never /proc/cpuinfo’s MHz field, which is the wandering core frequency. My threads must be pinned, because cross-socket TSC agreement is verify-not-assume. And on a VM none of this holds — live migration rewrites TSC offsets — so there I don’t time with rdtsc at all.

Q: When do you fence rdtsc, and when is it a waste? A: rdtsc is unordered — the out-of-order engine can hoist it before the work under test finishes. For a microbenchmark of a 20-cycle operation that skid is the whole measurement, so: lfence; rdtsc at the start, rdtscp; lfence at the end (rdtscp only waits for prior instructions, hence the trailing lfence). For pipeline stamps microseconds apart, a few ns of skid is noise and the fences’ pipeline drain costs more than the error — plain _rdtsc() in the hot path, fenced variants in benches.

Q: Your two servers are NTP-synced. Can you quote “gateway to server in 40µs”? A: No. NTP’s actual error on a LAN is tens to hundreds of µs, and it reports “synchronized” regardless — a 40µs claim with a ±500µs clock is 100% noise. I either quote the round trip measured on one clock, as RTT, or I get both ends onto PTP with hardware timestamping — PHC-stamped at the PHY, ptp4l to the grandmaster — which brings inter-machine error to tens of nanoseconds and makes a one-way number defensible.

Q: Why not just measure RTT and divide by two? A: Because the path isn’t symmetric and in trading infrastructure it reliably isn’t: different fiber routes each way, asymmetric queuing (busy ingress gateway, idle return path), different hop counts. 2:1 asymmetries are common, so RTT/2 can be off by half the RTT. RTT is a real number — I quote it as RTT. One-way numbers require synchronized clocks, full stop.

Q: Hardware vs software timestamps — what does the NIC stamp buy you? A: The NIC stamp is the PSP’s own received_at — stamped at the front door, not when my worker finally pulled the job off the queue. A software stamp is taken whenever my code ran, so it silently includes interrupt latency, softirq scheduling, and scheduler mood. The NIC’s PHY stamps the packet against the PHC the instant it hits the wire, before any of that can pollute it. So the delta between the wire stamp and my first software stamp is my network-stack-plus-wakeup cost — the number that justifies (or kills) a kernel-bypass project. Enable it with SO_TIMESTAMPING; bypass stacks surface the same stamps directly.

Q: Decompose tick-to-trade for me. Which clocks, which stamps? A: It’s an APM trace with four spans, so: four stamps, three deltas, then endpoint honesty. The stamps: t1 wire-in on the NIC’s hardware clock (PHC), t2 packet-in-hand and t3 decision-made on rdtsc, t4 wire-out on the NIC’s TX stamp. The deltas each blame one owner: 1→2 is network stack plus wakeup, 2→3 is my code, 3→4 is the send-side stack — and mixing PHC with rdtsc stamps needs the PHC↔TSC offset that phc2sys maintains. Endpoint honesty: quote 1→4, debug with the deltas, and never state a tick-to-trade figure without naming its two endpoints — “1.2µs” measured 2→3 differs from 1→4 by the entire network stack.

Q: I don’t believe your 8µs number. Convince me. A: A number is the output of a measurement, so I defend the measurement — and a measurement is trusted through exactly three things: its clock, its endpoints, and its reproducibility. The clock: raw TSC with invariant-TSC verified, calibration cross-checked against dmesg’s refined value, kernel clocksource still tsc. The endpoints: I say which two of the four stamps, and they’re wire-referenced — NIC hardware stamps, not when my process got around to it. The reproducibility: the whole distribution, worst case included — next chapter’s machinery — under replayed production bursts, plus the recipe: same capture, same pinned cores, run it yourself. Show all three and 8µs stops being a claim and becomes a result; the strongest move is handing over the capture file so the skeptic reproduces it.

Further reading

• Intel SDM, Volume 3, the Time-Stamp Counter section (invariant TSC, IA32_TSC_ADJUST) — plus Gabriele Paoloni’s Intel whitepaper “How to Benchmark Code Execution Times”, the source of the lfence/rdtscp discipline.
• IEEE 1588 / PTP overviews — the linuxptp project’s documentation (linuxptp.org) is the practical entry point; ptp4l(8) and phc2sys(8) man pages for the actual sync chain.
• clock_gettime(2), vdso(7), and time(7) man pages — the authoritative word on what each clockid means and which calls avoid the syscall.
• The Linux kernel’s timestamping documentation (Documentation/networking/timestamping.rst) — SO_TIMESTAMPING, hardware RX/TX stamps, and PHC plumbing.
• Gil Tene, “How NOT to Measure Latency” — the methodology talk the next chapter builds on; watch it before making any latency claim in an interview.

Where this goes next: you can now take honest timestamps — Chapter 8 is about turning millions of them into honest statistics: percentiles, HdrHistograms, and the coordinated-omission trap that invalidates most published latency numbers.