Question Bank: Networking

25 questions, ordered easy to brutal. Model answers are written in your voice: first-person where your 20+ crypto venue integrations give you standing, and explicitly honest (“I haven’t run X in prod; here’s how I’d approach it”) for colo/bypass tech you haven’t touched. Interviewers reward that candor heavily — the fastest way to fail these rounds is to bluff kernel-bypass war stories you don’t have. Practice answering out loud; 3–8 sentences each, no more.

1. What actually happens, end to end, when a market-data packet arrives at your NIC? NIC DMAs the frame into a ring-buffer descriptor in RAM, raises an interrupt (or is polled), the driver hands it to the kernel network stack — IP reassembly, TCP/UDP demux, socket buffer copy — then the epoll/recv wakes my thread, possibly on another core, and I copy into userspace and parse. Each stage costs: interrupt + context switch (µs-scale), at least one kernel-to-user copy, cache misses from waking on a cold core. That whole tour is why the kernel path floors out around single-digit microseconds and why bypass exists: map the NIC rings into userspace and poll, skipping interrupt, syscall, and copy.

2. What does TCP_NODELAY do and when do you set it? It disables Nagle’s algorithm, which delays small writes hoping to coalesce them into fewer segments. Nagle is throughput logic; an order message is a small write whose latency is all that matters, so every order-path and market-data socket I own gets NODELAY set at connect. In my crypto stack that’s the first thing checked when someone reports “orders are slow at low volume” — Nagle plus a quiet connection produces exactly that signature.

3. Explain the Nagle + delayed-ACK interaction. Nagle holds a second small segment until the first is ACKed; delayed ACK holds the ACK up to ~40–200ms hoping to piggyback on data. Put a Nagle sender opposite a delayed-ACK receiver with a write-write-read pattern (the story from ch02: send a header, send a body, wait for the reply) and each waits on the other: the sender won’t send segment two un-ACKed, the receiver won’t ACK segment one until its delayed-ACK timer expires. Result: latency spikes of tens of ms that appear only for certain message patterns. Fix on your side with NODELAY; the failure mode is worth knowing because you often can’t fix the venue’s side.

4. Why UDP multicast for market data in tradfi, and what does it cost you? One send reaches all subscribers with no per-receiver fan-out cost, no receiver can slow the sender (no backpressure), and switches replicate in hardware — it’s the only economical way to feed hundreds of subscribers the same firehose at the same time. The cost is that UDP guarantees nothing: loss, reordering, no retransmit, so reliability becomes the application’s problem — sequence numbers on every packet and a recovery story. Crypto flipped this: venues serve fan-out over per-client TCP/WebSocket because clients are on the open internet, which is why my world’s failure modes are slow-consumer disconnects rather than multicast gaps.

5. How do you detect and recover a gap on a multicast feed? Every packet carries a sequence number; a jump from n to n+k means k−1 lost. Recovery layers, in order of preference: arbitrate the A/B feed (venues transmit each packet on two lanes — most “gaps” are filled by the twin), then a retransmission request service for small gaps, then a snapshot/recovery channel to rebuild book state when you’re too far behind, buffering live increments to splice after the snapshot. I haven’t run exchange multicast in prod, but the splice logic is identical to what I do daily on crypto: REST/WS book snapshot plus buffered deltas, apply deltas with seq > snapshot seq. The design question underneath is always “how stale before you stop trading” — gap handling must set a poisoned flag the strategy respects.

6. Busy-polling vs interrupts — the actual trade. Interrupts let the core sleep and cost you wakeup latency and jitter (µs-scale, scheduler-dependent) per event; busy-polling burns a core at 100% spinning on the ring/socket and gets you consistent sub-µs reaction. Trading hot paths busy-poll on pinned, isolated cores because the jitter matters more than the electricity. Two caveats: you need core isolation (isolcpus/nohz_full) or the spinner gets preempted anyway; and for the 18 quiet venues out of 20, parking them on a shared epoll thread is correct — burn cores only on the venues that move P&L.

7. What is kernel bypass, concretely — and name the flavors. Userspace owns the NIC: the device’s TX/RX rings are mapped into the process, which polls and builds/parses frames itself — no interrupts, no syscalls, no kernel copies. Flavors: DPDK (take over the whole NIC, bring your own TCP stack), Solarflare/Xilinx ef_vi and Onload (Onload is the gentle one — LD_PRELOAD sockets API acceleration, no code change), AF_XDP (kernel-sanctioned middle path), and full FPGA NICs beyond that. I haven’t operated these in prod — my venues are internet-distant, where bypass is rounding error — but I’d start any colo build with Onload precisely because it keeps the sockets API while I benchmark whether the extra step to ef_vi/DPDK pays.

8. When is DPDK the wrong choice? When the network isn’t your bottleneck: my crypto reality is 1–50ms internet RTTs to venues, so saving 3µs of kernel stack is noise — engineering effort belongs in strategy and venue-quirk handling. Also wrong when you need the kernel’s TCP stack maturity (DPDK means bringing your own TCP — a giant liability against internet endpoints), when ops maturity is low (you lose tcpdump/netstat tooling and standard debugging), and when Onload gets you 80% of the win for 5% of the cost. It’s a decision tree ordered by RTT scale and team capacity, not a technology preference.

9. Design flow steering for a host handling 20 venues. Goal: hot venues get dedicated resources, the long tail shares. NIC-level: RSS to spread flows across queues, then explicit flow-steering rules (ntuple/Flow Director) pinning each hot venue’s 5-tuple to a dedicated queue whose IRQ (or busy-poll thread) is pinned to a dedicated core co-located with that venue’s parser — packet arrives on the core that will process it, no cross-core handoff. Long-tail venues share a couple of queues and an epoll thread. My version of this in the crypto world is the same shape one layer up: one socket-owner thread per hot venue pinned to a core, quiet venues multiplexed, because with TCP/WS the “flow” is the connection. Include the failure mode: steering rules silently vanish on NIC reset/driver reload, so audit them continuously.

10. Your feed handler sees the book crossed (bid ≥ ask). What do you do? Treat it as a data-integrity alarm, not an arb: mark the book poisoned, stop strategies consuming it, and resync (snapshot + delta splice). Causes I’ve actually hit across venues: missed/dropped delta, venue sending updates out of order across channels, snapshot/delta seq misalignment, and occasionally the venue itself briefly crossed during an auction or outage. The design point is that every consumer must respect a validity flag — a book that silently serves stale or crossed state to a strategy is how you buy the top of a spike. Uncrossing heuristics (“just trust the newer side”) are how people lose money quietly.

11. TCP retransmission timeout just fired on your order session. What did it cost and what do you do? TCP notices loss two ways: a retransmission timer that starts pessimistic at ~200ms, or seeing the same ACK repeated — which needs traffic to keep flowing. (This RTO is TCP’s retransmission timeout — no relation to the recovery-time objective of ch13.) On a busy session fast-retransmit (3 dup ACKs) saves you, but a quiet session generates no dup ACKs, so a lost packet waits out the full timer — a single lost order message eats a fifth of a second, and my order sessions are exactly that: quiet. Mitigations: keep the session warm (heartbeats also serve as loss detectors), tune what’s tunable, monitor per-socket retransmit counters (ss -i) and alert, and at the app layer treat an un-acked order past a deadline as state-unknown — never assume delivered or not delivered; reconcile. This “order in flight, session degraded” ambiguity is the worst state in trading networking and deserves an explicit state machine.

12. How would you timestamp packets honestly, and where do people lie to themselves? Hardware timestamps at the NIC PHY (SO_TIMESTAMPING with a PTP-disciplined clock) are ground truth for wire arrival; software timestamps at recv include kernel and scheduling noise and drift unless NTP/PTP is tight. People lie by: timestamping in the app and calling it wire time (hides the whole stack), comparing timestamps across unsynchronized hosts (clock skew swamps µs claims), and quoting venue-provided timestamps as if the venue’s clock were their own. My honest version: I run software timestamps disciplined by NTP because at internet RTTs that’s proportionate, and I’d move to PTP + NIC hardware stamps the day I’m in a colo claiming single-digit-µs numbers — a µs claim is only as good as the clock chain behind it, and in tradfi that chain is a regulatory artifact (RTS 25 flavor).

13. Cloud vs colo: what latency do you actually get, and what’s different in kind, not just degree? Colo gives you stable single-digit-µs to the matching engine and, critically, low variance. Cloud gives crypto reality: 1–50ms RTTs to venues, and — the difference in kind — variance you don’t control: multi-tenant hosts, virtualized NICs, transient reroutes, neighbor noise; your p50 might be fine while p99.9 is 10x. What I do about it, from prod experience: pick regions per venue (AWS Tokyo for the Tokyo-hosted venues), measure continuously per venue (RTT histograms, not averages), run redundant paths/instances and race them, and accept that my edge cannot be pure speed — it’s being fast enough consistently plus being smarter on venue microstructure. Also worth saying: some crypto venues now sell colo-ish proximity (same-AZ placement, private links), so the two worlds are converging.

14. Design a feed handler for a venue with a known-lossy feed. Assume loss is normal, not exceptional. Structure: seq-checked ingest → gap detector → recovery ladder (A/B arbitration if the venue offers dual feeds, retransmit channel for small gaps, snapshot resync for large ones) → book builder that tracks a validity state (LIVE / GAPPED / RESYNCING) exposed to every consumer → buffered-delta splice on recovery. Key design decisions to state: bounded buffer for live deltas during resync (and what to do on overflow: re-snapshot), staleness deadline after which the book is poisoned regardless, per-gap metrics because gap rate is a venue-health signal, and idempotent delta application so overlapped splice regions are harmless. This is my daily bread in crypto — every WS venue is “lossy” via disconnects — so I’d narrate it with a real venue’s snapshot+delta protocol.

15. What’s a speed-of-light budget and why do interviewers ask it? Light in fiber does ~200km/ms (c/1.5), so geography sets hard floors: NY–Chicago ~4ms one-way in fiber, ~3.9ms line-of-sight — which is why microwave networks exist and shave ~25%; Tokyo–Singapore ~27ms+; nothing you do in software beats geography. They ask it to see if you sanity-check systems against physics: if someone claims 2ms Tokyo–London, you should reject it on arithmetic alone. I use the same arithmetic in crypto for venue-region placement and for cross-venue arb feasibility — if venue A is in Tokyo and B in Virginia, the ~75ms one-way defines the whole strategy class.

16. Why can a single TCP connection never keep up with a multicast feed during a burst, and what’s the crypto equivalent? TCP has backpressure: the receiver advertises how much buffer it has left (the receive window), and at zero the sender must stop — so during a market-data burst the venue’s TCP fan-out either buffers (adding latency that compounds) or disconnects slow clients. Multicast has no such coupling; slow receivers just lose packets and recover themselves, protecting the fast ones. The crypto equivalent I’ve lived: venues detect slow WS consumers by send-buffer depth and kill the connection mid-burst — precisely when you least want a resync — so the defense is to drain the socket unconditionally fast (dedicated reader thread that never blocks, drop-to-conflation internally if the strategy can’t keep up — conflation is defined in the next question) and make disconnect-resync cheap because it will happen at the worst time.

17. Explain conflation, and when it’s correct vs a bug. Conflation replaces queued undelivered updates with the latest state — you deliver “the current book,” not every intermediate tick. Correct when the consumer only cares about current state and is slower than the feed: a human UI, a slow risk recalculation, a monitoring dashboard. A bug when the consumer needs the sequence: trade prints for volume signals, or a strategy whose logic depends on order-of-events. Design pattern: conflate per-key (per price level / per symbol) in the queue between feed thread and slow consumers, never on the hot strategy path, and make it explicit in the interface — a consumer must know whether its stream is conflated. Several crypto venues conflate server-side at low tiers; knowing which streams are conflated per venue is exactly the kind of quirk table my 20-venue layer maintains.

18. What does the OS scheduler do to your tail latency, and how do you stop it? Any runnable thread can be preempted, migrated cross-core (cold caches), or delayed by timer ticks, RCU callbacks (deferred kernel cleanup work that runs on your core), and kernel housekeeping — each event is rare but shows up precisely at p99.9+. Containment: pin threads (taskset/pthread_affinity), isolate cores (isolcpus, nohz_full, rcu_nocbs) so the kernel schedules nothing else there, keep IRQs steered away from hot cores, disable deep C-states/turbo transitions for consistency, and busy-poll instead of sleeping so the scheduler has no decision to make. I’ve applied the cheap half of this list (pinning, IRQ steering) on cloud boxes to visible p99.9 effect; full isolation I’ve only lab-tested — cloud hypervisors cap how far it goes, which is itself a point worth making.

19. Two feeds for the same venue disagree. Which do you trust, and how do you even know? First, “disagree” needs machinery: run both through identical normalizers and diff top-of-book with a tolerance window — transient disagreement within propagation delay is normal, persistent disagreement is an incident. Trust hierarchy: prefer the feed whose seq/heartbeat integrity is currently clean; if both are clean but differ persistently, one is stale — check heartbeat ages and last-update timestamps; if you can’t resolve, poison the book and resync both. This shows up constantly in crypto (WS vs REST snapshots are eventually consistent with each other), so my rule from prod: never mix fields from two sources into one book state without a sequencing story — pick a primary, use the secondary for validation and failover.

20. Walk me through diagnosing ‘orders are slow to one venue since Tuesday.’ Bisect the path with data. App-side first: per-venue order-path histograms (I keep them) — did submit→ack change at p50 or only tail? p50 shift smells like route/venue; tail-only smells like my host or bursts. Then network: RTT to venue endpoint over time (ping/TCP handshake times), traceroute vs a saved baseline for route changes, ss -i for retransmits and cwnd (the congestion window — how much TCP currently dares send) on the session, socket-buffer depths for local queueing. Then the venue: status page, other participants, a canary order from a second region — my prod experience is that a plurality of these are venue-side or ISP reroutes, which is exactly why the per-venue baseline histograms exist: without Tuesday’s baseline you can’t even ask the question crisply.

21. Why is the last mile inside your own host often worse than the WAN hop, in crypto? Because 30ms of internet RTT hides sins: a GC pause, a lock, a cross-core handoff, a syslog write on the hot path — each 100µs–10ms — are invisible at p50 but stack the tail, and unlike the WAN they’re yours to fix. I’ve found more latency in JSON parsing and allocator behavior than in routing. So the priority order inverts vs tradfi: profile the process before blaming the network — the WAN sets the floor, the host sets the tail.

22. How would you build the network side of a new colo deployment, never having run one? (Honesty test.) I’d say exactly that, then show the plan: engage the exchange’s connectivity docs and a peer/vendor early (cross-connect specs, feed bandwidth, A/B multicast details); provision Solarflare-class NICs and start with Onload for sockets-API acceleration, benchmarking ef_vi as step two; PTP time infrastructure from day one because every latency claim depends on it; capture appliance or port mirror for ground-truth wire timestamps; burst-size the multicast feeds from spec, not averages (opening auction dictates buffers); and drills for gap recovery before go-live. Frame it as: the concepts — seq/gap/recovery, pinning, steering, honest timestamps — I already operate daily in crypto; what’s new is the specific tooling, and I de-risk that with vendor references and measurement rather than bravado.

23. Multicast joins, IGMP, and what goes wrong at 3am. Picture pub-sub implemented by the network hardware: a receiver’s IGMP join is a subscription message, and the switch is the broker — it snoops the joins to learn which ports want which groups, and it forgets a subscription unless a querier periodically refreshes membership. Failure modes that page people: IGMP snooping misconfigured → either flooding every port (bandwidth collapse) or pruning your port (silent feed death); a membership timeout after a querier hiccup → feed just stops with no error on your socket — from userspace it’s indistinguishable from a quiet market, which is why heartbeat-expected alarms on every feed are non-negotiable; and after switch maintenance, joins that never re-establish. I haven’t run this in prod — it’s the multicast cousin of what I do monitor: a WS connection that’s open but silent — same alarm design (“expected message rate violated”), different layer.

24. Design market-data distribution inside your own shop: 1 feed handler, 30 internal consumers. Don’t let 30 consumers each parse the venue feed — normalize once, distribute internally. Options ladder: shared-memory ring buffer (single-writer, seqlock’d or Disruptor-style — a seqlock means the writer bumps a version counter around each write and readers retry if it changed mid-read) for same-host hot consumers — nanoseconds, no syscalls, slow consumers just fall behind by sequence and know it; Aeron/multicast for cross-host fan-out with the same loss-is-the-consumer’s-problem discipline; TCP/IPC only for the slow tier, with conflation. Key properties to state: single writer per stream, sequence numbers end-to-end (consumers detect their own gaps), no backpressure from consumer to feed handler ever — a slow risk dashboard must not be able to slow the strategy path. This is the internal mirror of Q4/Q16, and interviewers ask it to see if you apply venue-side lessons to your own architecture.

25. Brutal one: your p99.9 tick-to-trade doubled but every network metric you have is flat. Argue that it’s still the network. Flat sampled metrics don’t exonerate the network: microbursts overflow NIC/switch queues in microseconds while 1-second counters average them away — check ring-buffer drop counters and switch telemetry, not rates; retransmits on a different flow can head-of-line-block a shared queue — both flows land in the same NIC RX ring; IRQ steering may have quietly reset after a driver update, landing packets on the wrong core (the “network” is fine, the delivery locality isn’t); and if timestamps come from the app, a clock or scheduling change masquerades as network time. The meta-answer interviewers want: name where each metric is measured, find the unmeasured gaps between measurement points, and instrument the gap — hardware timestamps at the NIC vs app timestamps bracket the stack and settle the argument. Then concede the prior: having chased this in cloud prod, the culprit was usually my host, and I’d say so while still checking the drop counters first — they’re cheaper to read than a profile.

The HFT Gap Closer

Question Bank: Networking

Further reading

Keyboard shortcuts

The HFT Gap Closer

Question Bank: Networking

Further reading