Question Bank: Venue & Broker Design

25 questions, ordered easy to brutal, covering Part V (ch23–ch27). Model answers in your voice: first-person and concrete where your CLOB, market-data pipeline, and SOR at Crypto.com give you real standing; explicitly framed (“the productization layer I’d add is…”) where multi-client broker or venue-operator experience is the new territory. Being candid about which is which is what wins these rounds — you built a matching engine, a 20-venue data pipeline, and a router for one desk, and that’s more than most candidates walk in with. Practice aloud; 3–8 sentences each.

1. What does the sequencer do, and why does everything need a total order? The sequencer receives every input — orders, cancels, timer ticks, reference-data changes — assigns each a monotonically increasing sequence number, and publishes the sequenced stream; everything downstream (matching engine, risk, drop copy, standby) is a deterministic function of that stream. Total order is what makes determinism possible: if two consumers could see events in different orders, they’d compute different books, and replication, recovery, replay-testing, and audit all break simultaneously. It also is the fairness ruling — “who was first” has exactly one answer, the sequence number. I built this shape at Crypto.com: single-writer, event-sourced matching engine with a hot standby consuming the same log; the standby works precisely because the log is totally ordered.

2. Untangle OMS, EMS, and SOR. OMS owns the what: the parent order as a durable business object plus client accounts, buying power, and allocations — the system of record, like Stripe’s PaymentIntent. EMS owns the how: working the parent over time — algo choice, slicing, urgency — the retry/orchestration layer. SOR owns the where: for one child right now, which venue, given fees, books, latency, and fill probability — acquirer selection. In practice EMS and SOR blur into one engine; what I built was an SOR with EMS behaviors and a thin single-tenant OMS, and the multi-client OMS is the layer I’d build fresh — event-sourced, because allocations and best-execution evidence both demand deterministic replay.

3. What is drop copy and why not just use your execution reports? Drop copy is a real-time duplicate of your fills and order events on a separate session, feeding risk and clearing independently of the trading session — a CDC stream off the venue’s ledger. Independence of the watcher from the watched is what earns the separate session: if risk builds positions from the trading gateway’s own view of its acks, one bug corrupts trading and risk together; drop copy gives risk a venue-authoritative stream the trading code never touches. It’s also the practical feed for the real-time aggregate exposure monitoring 15c3-5 expects. In my recon design it’s the live leg of a triple loop: internal book vs drop copy continuously, vs venue statements at cutoffs, vs custodian on settlement.

4. What is cancel-on-disconnect and when is it wrong? CoD auto-cancels a session’s resting orders when its connection drops — the default safety rail, because a dead strategy leaving stale quotes in the market gets picked off. It’s wrong in two directions: triggering on a network blip while the strategy is healthy mass-cancels your queue position, which is expensive for a market maker; and broker-side, killing a client’s 6-hour TWAP because their monitoring session dropped confuses session with order ownership — the OMS owns the parent, not the client’s TCP connection. So it’s per-session configuration with tuned heartbeat timeouts, and the platform mirror is an explicit per-client disconnect policy: CoD for takers, keep-working for parked algos.

5. List the pre-trade risk checks and their latency budgets. Price collar against a reference band, max order size and notional, position-plus-open-orders limit, credit/buying power, per-session rate limit, self-match prevention, duplicate detection on client order ID. Venue-side each is a compare or an atomic read on in-memory, cache-line-padded counters — call it 10–100ns per check, a few hundred nanoseconds total, small against a 5–50µs gateway path, which is the arithmetic that shuts down “skip checks for latency.” Placement: venue runs them in the gateway before the sequencer so rejects never consume a sequence number; broker runs them in the OMS before the SOR so a breach never reaches a venue. Two of these I’ve built in payments under different names: the client-order-ID check is an idempotency key, and the credit check is an auth hold with reserve/release/convert semantics.

6. Self-match prevention: what are the options and who wants which? At the touch (the best bid and ask — where the next trade prints), if the aggressor and the rester resolve to the same firm or SMP group, don’t print — apply policy: cancel-newest (aggressor dies; market makers re-quoting want this because the rester keeps queue position), cancel-oldest (rester dies, aggressor trades on; sweepers — aggressive orders eating through multiple price levels — want their intent to survive), or cancel-both. It’s a per-session flag because both preferences are legitimate. Venues do it mechanically rather than adjudicating intent afterwards because self-matches print volume indistinguishable from wash trading. Cost is a tens-of-ns ownership check, since order structs already carry owner IDs.

7. Why is cross-session ordering decided only at the sequencer, and what does that mean for gateways? Because the gateways are parallel: two orders entering different gateways have no meaningful “first” until something imposes one, and any attempt to decide order at the gateway tier — timestamps from different NICs, queue positions on different boxes — manufactures a fake ordering from unsynchronized clocks. So the contract is: gateways validate, normalize, and forward as fast as they can; arrival at the sequencer is the ordering event; fairness engineering means making the gateway-to-sequencer path uniform (same hops, same budget per session) so no session buys an edge from topology. The venue’s job isn’t zero latency, it’s equal latency — fairness as variance control, not speed.

8. Why can multicast be fair when TCP fanout cannot? Multicast hands one packet to the switch and the switch replicates in hardware — every subscriber’s copy leaves at effectively the same instant, and no receiver’s slowness backpressures the sender or delays anyone else. TCP fanout is N serialized sends: someone is first and someone is last on every update, the ordering is an implementation accident that becomes a de-facto tiering, and a slow receiver’s closed window forces the sender to buffer or disconnect. That’s why tradfi feeds are multicast with loss handled by the receiver, and why crypto — stuck with TCP/WS across the internet — substitutes randomized send order, tiered products, and slow-consumer kills instead of true simultaneity. I’ve lived the receiving end of the crypto version across 20+ venues; building the sending side means choosing which unfairness you can defend.

9. How do you handle a slow consumer in a WebSocket fanout tier? Never let it backpressure the publisher — that’s the one inviolable rule, because a slow risk dashboard must not slow the feed. Detect via send-buffer depth or lag from head-of-stream; respond in escalation: conflate per-key so the consumer gets current state instead of every tick (correct for UIs and dashboards, a bug for anything sequence-dependent), then drop to snapshot-on-reconnect, then disconnect with a resume token. Make the contract explicit — the consumer knows whether its stream is conflated and can detect its own gaps via sequence numbers. I’ve been on the receiving end of venues that kill slow WS consumers mid-burst, precisely when resync is most expensive, so I’d also invest in making reconnect-and-splice cheap: snapshot plus buffered deltas applied by sequence.

10. Design a market-data feed: snapshot + incremental, and the consumer contract. Publish an incremental stream where every update carries the sequence number of the book state it produces, plus a periodic (or on-demand) snapshot stamped with the sequence it reflects. Consumer contract: subscribe to increments first, buffer; fetch snapshot; discard buffered increments with seq ≤ snapshot seq; apply the rest; a gap in increment seq means you’re broken — re-snapshot, and expose a validity state (LIVE/GAPPED/RESYNCING) to every downstream consumer. Publisher-side obligations: snapshots must be consistent cuts at an exact sequence (generated from a replayer of the sequenced log, not from a racing read of live state), and increment retention must cover the slowest tolerated consumer. I’ve implemented the consumer half against 20+ venues and can enumerate which venues get this wrong and how; building the publisher half is the same contract from the other side.

11. How do you shard a venue by symbol, and what about the hot symbol? Partition instruments across matching-engine shards — each shard its own sequencer+engine, totally ordered within itself, no ordering guarantee across shards; that’s valid because price-time priority is per-book. The catch: sharding gives you nothing for the hot symbol, because one book is inherently a single total order — BTC-perp at an open is one shard’s problem no matter how many shards exist. Hot-symbol answers are vertical and structural: make the single-writer path faster (the engine is small; the win is usually upstream in the gateway/risk tier, which does parallelize), move non-matching work off the shard, and accept that cross-shard products (margin across books, self-match across books) now need either an aggregation tier or asynchronous enforcement. If someone proposes splitting one book across shards, the interview answer is: you’ve just reinvented the fairness problem with extra steps.

12. How do you fail over a sequencer without losing acked orders? Define the invariant first: an ack means the event is durably sequenced, so the ack must only be sent after the event is replicated to the standby (or a quorum) — ack-after-replication, not ack-after-local-write. Then failover is: fence the old primary so it can’t append (epoch numbers on every log entry; consumers reject stale epochs), standby verifies it holds the log through sequence N, takes over at N+1, and re-establishes sessions; clients reconcile via exchange-provided order-status queries. The unacked in-flight window is the client’s problem by contract — which is why client order IDs and idempotent resubmit exist. Options ladder: manual standby with fencing (what I effectively ran — primary/standby with log shipping), or Raft via something like Aeron Cluster, where every input must reach a majority of nodes before it’s acked — so each order pays that quorum round-trip — in exchange for principled automatic failover. The trap answer to avoid: “ping timed out so standby promotes itself” — that’s split-brain, and fencing is exactly what prevents it.

13. One venue gives you 100 orders/sec. N clients share your platform. Design the budget. Per-client sub-budgets inside each venue budget — weighted fair queuing by tier or contract, unused capacity redistributed — because one client’s algo burst must not starve another’s flow. Two hard rules: cancels never queue behind new orders, since a starved cancel is a risk event, not a UX issue; and throttling is surfaced to the client as an explicit signal rather than silent queuing, because silently delaying an order changes its execution price and that’s a best-execution problem. It’s API-gateway per-tenant rate limiting where the upstream quota is hard and external. I ran the single-tenant version — per-venue budgets in my adapters — and the fair-share scheduler across clients is the productization layer.

14. Walk me through the double-fill race on re-route. Child on venue A isn’t filling; SOR re-routes: cancel to A, new child to B; A’s fill was already through its sequencer when the cancel arrived, so both fill and the client is over-bought. Prevention is cancel-ack discipline: don’t send B’s child until A confirms the cancel with final cumulative quantity — one venue RTT per re-route, and worth it. That’s the payments idempotency rule wearing trading clothes: never retry until the first attempt’s outcome is known, because “probably failed” is how double-charges happen. If an algo genuinely can’t wait, cap the new child assuming worst-case fill on A and run an error account for over-fills with explicit client-disclosure policy — optimism becomes opt-in and accounted for, never the default.

15. What does client segregation mean as engineering, not policy? One client’s flow is alpha, so the design must make leakage structurally hard, not contractually forbidden. Concretely: no shared mutable state whose observable behavior reveals another client’s activity — a shared queue where A’s burst delays B’s acks is a side channel, so per-client queues fair-scheduled into shared stages; per-client authorization on every read path including support dashboards, with access audited; and if the firm trades principal, the firm’s own desk is the hardest wall — separate services and credentials, because “the router operator sees everyone’s flow” is the FTX/Alameda lesson. It’s multi-tenant SaaS isolation with an adversarial twist: the leak isn’t embarrassment, it’s directly monetizable against the victim.

16. What evidence does best execution require, and what’s TCA? Best execution is the obligation to get the client the best available result across price, cost, speed, and likelihood — and the architectural forcing function is provability: every routing decision logged with the market snapshot it saw — per-venue books, fees, health scores, cost-model arithmetic — so “why venue B at 14:32:07” has a replayable answer. An event-sourced SOR gets this nearly free; I logged decisions at Crypto.com for debugging and venue scoring, and the upgrade is retention and tamper-evidence. TCA — transaction cost analysis — is the after-the-fact measurement: slippage vs arrival price, vs VWAP for scheduled algos, and per-venue scorecards (fill rates, effective spread, post-fill markout as a toxicity signal). It’s simultaneously the router’s feedback loop and the client-facing proof — the auth-rate dashboard a PSP shows merchants to justify its routing.

17. How does “the venue is a counterparty” change a crypto router? Pre-funding means balances sit on each exchange, so venue failure is a credit loss, not an outage — FTX is the case study for why best-price-only routing is broken. So the venue score carries slow-timescale risk inputs next to fast microstructure ones: withdrawal-latency trends (withdrawals quietly slowing is the canonical early warning), concentration caps on the share of assets per venue, jurisdiction and licensing posture. It also spawns a treasury subsystem — rebalancing inventory across venues against withdrawal fees and on-chain confirmation times, the multi-currency prefunding problem from payments. And the stablecoin legs are real instruments: BTC-USDT vs BTC-USD carry a basis, and routers that hardcoded a stablecoin at $1.00 learned about it in March 2023, when USDC broke to $0.87.

18. Sketch a liquidation engine. Why mark price, not last price? Margin and liquidations key off a mark price — an index over multiple external spot venues, smoothed — because keying off your own last trade lets an attacker print one small trade in a thin book and cascade-liquidate everyone; mark-price design is manipulation resistance, and it forces index rules: several constituents, published weights, outlier rejection, staleness eviction. The waterfall: margin call → partial liquidation (reduce, don’t nuke) → full liquidation via rate-limited orders into the book → insurance fund absorbs closes worse than bankruptcy price → ADL against profitable opposing positions as the published last resort. The engine itself is a trading system with the same disciplines: mark-price ticks as sequenced events, deterministic decisions, and its own kill switch, because a runaway liquidator is the worst self-inflicted incident a perps venue can have. I know this stack from the consumer side across venues; specifying it operator-side is the flip this book’s Part V is about.

19. How would you detect wash trading on your venue? As stream jobs on the sequenced log — the deterministic log is what makes surveillance tractable, because “what did the book look like when this order arrived” has an exact, replayable answer. Wash detection: entity-resolve accounts into beneficial owners (shared funding sources, withdrawal addresses — a graph problem in crypto), then flag self-crossing rates, volume with near-zero net position change, and tight round-trips. Spoofing/layering, its sibling: order-to-trade ratios, cancel-latency distributions — spoofers cancel fast when approached — and size posted opposite subsequent aggression. Flags go to human review with a book-replay evidence bundle, which is why a venue that can’t replay its book can’t really do surveillance. Architecturally it’s payments transaction monitoring — velocity rules on a ledger feeding a case queue — with different features, and SMP upstream removes the innocent cases first.

20. Give me 15c3-5 and RTS 6 in one line each, plus what each forces you to build. 15c3-5, the US market-access rule: a broker giving clients market access must run pre-trade financial and regulatory checks under the broker’s own control — no naked access — which forces the OMS risk gate before the SOR, non-disableable per client, plus real-time aggregate exposure monitoring, which drop copy feeds. RTS 6, MiFID II’s algo-controls standard: firms running algos must have kill functionality, pre-trade limits, real-time monitoring, and an annual self-assessment — which turns the kill-switch taxonomy into an audited artifact with named owners and test evidence. I’d volunteer the adjacent one: MiFID’s clock-sync rules (RTS 25) make the PTP chain regulatory evidence. My frame in the room: I’m not a lawyer, but I design assuming the event log will be read by a regulator — cheap if you’re event-sourced from day one, impossible to retrofit.

21. Why must mass-cancel be the fastest path in the system? Because it’s used exactly when the market is moving against someone and every millisecond of cancel latency is money — a mass-cancel that walks orders one-by-one through the normal pipeline arrives after the damage. So it’s a first-class sequenced operation: one message cancels by owner/symbol/scope, orders are indexed by owner so cancel-all is O(orders owned), and queue capacity is pre-reserved so the cancel can’t be backpressured by the very flood it’s stopping. Layered above it sits the kill-switch taxonomy — per-session, per-symbol/venue, global — pre-authorized, drilled (RTS 6 audits this), with an out-of-band path so pulling it doesn’t require the sick system to cooperate. It’s the payments “pause payouts” button: pre-wired, permissioned, logged, and the postmortem always asks why it took N minutes to press.

22. Two clients’ orders compete for the same liquidity on your platform. Who wins? Whoever the written policy says — the indefensible answer is not having one, because the disadvantaged client’s lawyer will ask. My default: strict time priority of parent arrival at the platform, simple and incentive-compatible (no client gains by gaming when or how they submit), mirroring what the venue itself does; where the platform aggregates parents into shared children, fills allocate back pro-rata by a documented formula. The engineering requirement is that allocation is deterministic and replayable — same fills in, same split out — computed in the event-sourced fold, and logged with the same evidence discipline as routing decisions. And it’s a product decision, not just an engineering one: I’d want compliance in that design review, because the allocation policy is a commitment the firm makes to every client simultaneously.

23. Design a crypto exchange from scratch. First five boxes on the whiteboard. One: gateways — sessions, authn, normalization, pre-trade risk, per-session rate limits, cancel-on-disconnect; parallel and stateless-ish. Two: the sequencer — single writer assigning the total order, ack-after-replication, the fairness and determinism anchor. Three: the matching engine — a deterministic fold over the sequenced log, price-time priority, in-memory book, sharded by symbol with the hot-symbol caveat. Four: the market-data publisher — incremental feed with sequence numbers plus consistent snapshots, fanout tier that slow consumers cannot backpressure. Five: the post-trade spine — drop copy, positions/ledger, recon, and (for perps) the margin/liquidation engine with mark-price index. Then I’d say out loud: boxes two and three I have actually built as one desk’s engine — deterministic single-writer, event-sourced, hot standby — and the venue version is the same skeleton with fairness, surveillance, and counterparty obligations bolted on where my one-desk version could ignore them.

24. Your venue’s p99.9 ack latency doubled at market open. Walk me through it. First, localize with the timestamps the pipeline already carries: gateway-in, sequencer-in, engine-out — the doubling lives in one segment, and p99.9-only (p50 flat) means queueing, not a uniformly slower path. Open-specific suspects in order: burst arrival overflowing a gateway queue (check depth high-watermarks and per-session arrival histograms — often one participant’s algo went aggressive overnight), a risk-check hitting a cold or contended path (a limit table that grew, a false-sharing regression on the counters), GC/allocator or page-cache effects from the overnight batch still settling, and the market-data fanout stealing cycles from a shared core if isolation regressed after a deploy. I’d also diff against yesterday’s open — same percentile, same minute — because “doubled” only means something against a baseline, which is why per-segment histograms exist (ch08). And I’d volunteer the uncomfortable part: at crypto’s latency scales the culprit was usually my own host, and I’d say so while checking queue depths first because they’re cheaper to read than a profile.

25. A client says your router gave them a bad fill. Prove it didn’t — or find out it did. Pull the decision record: the SOR is event-sourced, so that child’s routing decision exists with the full input snapshot — every venue’s book, fees, health scores, and the cost-model arithmetic at decision time — and I replay it, which either reproduces the choice or exposes a divergence. If the decision was right on its inputs, TCA frames the outcome: slippage vs arrival for that parent, the venue’s scorecard that day, and the counterfactual cost of the alternatives from the same snapshot — sometimes the answer is “the market moved during your order; here’s the tape.” If the inputs were wrong — stale feed, mis-scored venue — the same log shows exactly that, and now it’s an incident with a fix and possibly a client remedy, which is a better outcome than winning the argument. This is why the evidence discipline exists before the dispute does: you cannot retrofit the snapshot. It’s the chargeback-dispute flow from payments — the merchant who kept the AVS response and the signed receipt wins; the one who didn’t, pays.

The HFT Gap Closer

Question Bank: Venue & Broker Design

Further reading

Keyboard shortcuts

The HFT Gap Closer

Question Bank: Venue & Broker Design

Further reading