You Are the Broker: SOR Across Venues
Before you start. This chapter leans on:
- Exchange/venue anatomy — gateways, sequencer, matching engine — because the broker is a client of N of them: ch23
- Order gateways and sessions — the venue-side view of the connections your adapters maintain: ch24
- Market-data publishing — snapshot+incremental feeds, because your normalized-data layer consumes them: ch25
- Event sourcing and the determinism contract — the OMS/SOR here is an event-sourced system with compliance obligations: ch13
- Trading vocabulary (parent/child orders, TIF, tick size, TWAP/VWAP, and the rest of the dialect) — the decoder’s “Orders and execution” table: ch00d
Read those first — this chapter assumes all of them.
You already built a smart order router: cost-model routing across 20+ venues, ~10ms decision on CLOB legs (a leg is the per-venue piece of one trade), venue scoring, one desk consuming it. This chapter takes that exact system and asks the question a Talos-style firm will ask you in the first ten minutes: what changes when the SOR is a product serving N external clients instead of an internal tool serving one desk? The answer is not “add auth and a billing table.” Almost every hard property of the system — risk, fairness, state, compliance — multiplies in a specific, nameable way, and interviewers at execution-platform firms are testing whether you can name the multiplications.
The one-sentence framing to open with: a broker/execution platform is a multi-tenant SaaS whose tenants’ requests compete for the same scarce external resources (liquidity, venue rate limits) and whose every decision must later be defensible to the tenant and to a regulator. You have built the single-tenant version. This chapter is the delta.
The whole machine
Client A ─┐
Client B ─┤ FIX / WS / REST ┌───────────────────────────────┐
Client C ─┼─► [client gateways] ──► │ OMS │
... │ authn, sessions, │ parent orders, client accts, │
Client N ─┘ per-client rate │ per-client risk & buying │
limits, CoD │ power, allocation policy │
└──────────────┬────────────────┘
│ approved parents
┌──────────────▼────────────────┐
│ SOR / algo engine │
│ slicing, cost model, │
│ venue scoring, child orders │
└──────────────┬────────────────┘
│ child orders
┌────────────────────────────────▼───────────────────────┐
│ venue adapters × 20+ │
│ symbology map, order-semantics normalization, │
│ per-venue sessions, heartbeats, shared rate budget │
└───┬──────────┬──────────┬─────────────┬────────────────┘
▼ ▼ ▼ ▼
[venue 1] [venue 2] [venue 3] ... [venue 20+]
│ │ │ │
└──────────┴────┬─────┴─────────────┘
│ fills, acks, rejects
┌───────────────────▼───────────────────┐
│ normalized market data (your pipeline) │──► SOR cost model
└───────────────────┬───────────────────┘
│
[fills → allocation → client reporting → ledger → TCA (§below)]
The top half is new (multi-client OMS). The middle is your existing SOR. The bottom is your existing adapter + market-data layer, now with multi-tenant complications. The fills pipeline at the bottom grows from “update my position” to “allocate, report, invoice, and prove.”
OMS vs EMS vs SOR: untangling the triad
The industry uses these three acronyms with maximal sloppiness. Untangle them once and you sound native.
client intent ──► OMS ──► EMS ──► SOR ──► venues
what how where
- OMS — Order Management System — owns the what: the parent order as a durable business object — client, account, instrument, side, quantity, limit, instructions — plus client accounts, positions, buying power, and post-fill allocations. Analogy from your world: Stripe’s PaymentIntent — the durable object representing “customer wants to pay $50,” which survives retries, partial captures, and whatever routing happens underneath it. Why it matters: the OMS is the system of record; when everything else disagrees, the OMS’s ledger is what you reconcile against.
- EMS — Execution Management System — owns the how: working the parent over time — algo selection, slicing schedule, urgency. The algo names, in plain English: TWAP drips the order out evenly over N hours; VWAP drips it out proportional to when the market usually trades; POV never lets you be more than X% of current volume; liquidity-seeking hunts for size wherever it appears. Analogy: the retry/orchestration logic in a PSP that decides when and in what sizes to attempt captures — not what the customer owes, not which acquirer.
- SOR — Smart Order Router — owns the where: for one child slice, right now, which venue(s), at what price, given fees, latency, fill probability, and current books. Analogy: the routing step that picks an acquirer for a single authorization based on cost, auth-rate history, and health.
In practice the EMS/SOR boundary blurs (an aggressive multi-venue sweep is slicing and routing in one motion), and vendors sell combined “OEMS” products. The clean claim for interviews: you built an SOR with some EMS behavior (slicing, cost-model timing) and a thin single-tenant OMS (your desk’s position and risk state); productizing means building the OMS out into a multi-tenant system of record.
What multiplies at N clients
1. Risk moves in front of routing
One desk: the risk limits were your own, checked wherever convenient, and a breach hurt only you. N clients: per-client pre-trade risk — buying power, position limits, notional caps (notional = quantity × price, the dollar size of the order), price collars (defined in ch27), duplicate-order detection — must run before the SOR ever sees the parent, per client, with per-client configuration, and with the results logged. In the US this is literally law (SEC Rule 15c3-5, ch27): a broker providing market access must apply pre-trade financial and regulatory checks; “the client says they’re good for it” is not a control. Latency budget: these checks sit on the client’s critical path, so they get the same treatment as a venue’s risk gate — microseconds, in-memory limit counters, no database on the hot path, async persistence.
Buying power — the client’s remaining capacity for new exposure: cash/collateral, minus positions, minus the reserved exposure of working orders. The subtlety that trips people: open child orders reserve buying power the moment they’re sent and release it on cancel or fill — a hold/capture lifecycle. Analogy: card auth holds — an authorization reserves funds; capture settles; void/expiry releases — and the classic payments bugs (double-release, a hold leaked forever after a lost webhook) have exact counterparts in order-state reconciliation.
2. Client segregation is an engineering requirement, not a policy document
One client’s order flow is alpha — information you can trade profitably on; knowing A is buying big is a trading signal. To work an order is to execute it slice by slice; a thin book is one with little resting size, where a big order moves the price. If client B can infer that client A is working a large buy in a thin book — from timing, from shared-queue backpressure, from a support dashboard, from a log line — you have leaked information that B can trade against. Information barriers — controls preventing one client’s trading information from reaching another — sound like a compliance concept; in an execution platform they are concrete engineering:
- No shared mutable state whose observable behavior reveals another client’s flow. A shared unbounded queue where A’s burst delays B’s acks is a side channel. Per-client queues, fair-scheduled into shared downstream stages.
- Per-client authorization on every read path — reporting APIs, dashboards, support tooling. A support engineer’s “all open orders” view is itself an information-barrier surface, and access to it is audited.
- The platform’s own trading desk (if any) is the hardest wall. If the firm also trades principal, “the SOR operator sees everyone’s flow” is the FTX/Alameda lesson. Say this in interviews: the architecture must make leakage structurally hard — separate services, separate credentials, audited access — not just contractually forbidden.
Analogy: multi-tenant SaaS tenancy isolation, except a “data leak” here isn’t PII embarrassment — it’s directly monetizable against the victim, so the threat model is adversarial and partly internal.
3. Fair allocation is a product decision you must document
Two clients want the same liquidity: A and B both send buys in the same instrument, and venue X has one resting offer big enough for only one of them. Who gets it? At one desk the question doesn’t exist. At N clients it needs a written allocation policy — the documented rule for how competing orders share access to liquidity and how fills on aggregated child orders split back to parents. Common answers: strict time priority of parent arrival at the platform; pro-rata splits when the platform batches several parents into one child (proportional to size: A wants 10, B wants 4, the fill is 7 → A gets 5, B gets 2); never-aggregate (each parent gets its own children; venue queue position decides). Each is defensible; having no documented answer is not, because the disadvantaged client’s lawyer will ask. Analogy: a PSP splitting a partial settlement across merchants needs a deterministic, documented rule — ad-hoc splits are how you fail an audit. Engineering consequence: allocation must be deterministic and replayable — same fills in, same allocation out — your event-sourcing contract (ch13) applied to a new deterministic aggregation.
4. Pricing and fees enter the cost model — with a conflict of interest attached
Per-client fee schedules: some clients pay cost-plus (venue fees passed through plus an itemized platform fee), others all-in (one bundled rate; the platform keeps or eats the venue-fee difference). Why an engineer cares: under all-in pricing the platform has an incentive to route to cheap venues even when a pricier venue is better for the client — precisely the conflict best-execution rules (defined below) police. Your cost model now needs two outputs per candidate route — cost-to-client and cost-to-platform — and the routing decision must optimize the client’s number. Log both; the gap between them is exactly what a regulator asks about.
The venue adapter layer: the crown jewels
This is your daily craft, so own this section. Talos’s public positioning amounts to “we normalized dozens of venues so you don’t have to” — the adapter layer is the moat, because every venue integration is months of quirk-discovery a competitor must repeat.
What normalization actually means, from someone who has done 20+ of these:
-
Instrument master — the subsystem mapping every venue’s symbology into one canonical instrument space:
BTC-USDTvsBTCUSDTvsXBTUSDvs a numeric instrument ID; spot vs perp vs dated future as different canonical instruments even when a venue reuses a ticker; tick size (smallest price step), lot size (smallest quantity step), min notional (smallest dollar size accepted), contract multiplier (how many units one contract represents) per venue per instrument. This is a real service with its own storage, update pipeline (venues list and delist constantly), and versioning — a stale instrument master sends orders at the wrong tick and gets rejects, or worse, gets accepted at an unintended price scale. Analogy: currency-exponent reference data in payments — boring, and the source of the worst incidents when wrong (the “amount in cents vs units” bug class). -
Order-semantics normalization — the same order flag means different things at different venues:
Term What the client means How venues differ postOnlyrest in the book only, never trade on arrival venue A rejects the order if it would cross; venue B silently re-prices it to sit passive TIF (IOC / FOK) how long the order may live — IOC: fill what you can now, cancel the rest; FOK: fill all of it now or nothing flavors vary; some venues have no true FOK, so the adapter emulates it or refuses Iceberg show only a small visible slice of a big resting order support varies — native, absent, or emulated by the adapter STP flags self-trade prevention: never match against my own resting orders per-venue enums with different cancel-newest / cancel-oldest / cancel-both semantics The adapter exposes a canonical order-type set and, per venue, maps, emulates, or explicitly refuses. Silent approximation is the sin: the client asked for FOK semantics and got something else.
-
Per-venue health: heartbeats, sequence-gap counters, ack-latency histograms, reject rates by reason — feeding the venue score the SOR consumes. You built this; say so, with numbers.
-
The new multi-tenant problem — shared rate budgets. Venue X allows the platform 100 orders/sec total. That budget is now shared by N clients, and one client’s algo burst can starve everyone else’s cancels — and a starved cancel is a risk event, not an inconvenience. Design: per-client sub-budgets inside each venue budget (weighted fair queuing — each client gets a guaranteed share of the 100/sec, and unused share is redistributed; cancels strictly prioritized over new orders — cancels must never queue behind entries), burst allowances, and throttling surfaced to the client as an explicit signal rather than silent queuing. Analogy: an API gateway doing per-tenant rate limiting inside a global upstream quota — except the upstream quota is a hard external constraint and “just queue it” changes execution prices.
Where the HFT skillset pays on the broker side — and where it doesn’t
You came to this book for the HFT toolbox: kernel tuning, lock-free queues, µs measurement. The broker seat uses all of it — but the payoff is asymmetric, and knowing where it pays is itself the senior skill. Start the way the kernel-tuning chapter (ch03) taught you: with the budget table, not with the coolest tool.
| Segment of a broker’s order path | Typical cost | Can engineering shrink it? |
|---|---|---|
| WAN hop to an internet crypto venue | ~1–70ms | No — buy placement (region, colo), don’t code |
| Venue’s own processing | ~ms | No — it’s their machine |
| Your adapter/gateway path (sign, session, submit) | ~1–50ms | Yes — usually the biggest controllable term |
| Your SOR decision loop | µs–ms | Yes — the classic hot-path skillset |
| Your feed ingestion + normalization | µs–ms | Yes — same |
The rule is the one from the kernel-tuning chapter (ch03): optimize the biggest controllable term first. For an internet-venue broker that’s almost never the kernel — it’s warm sessions, pre-computed auth, and not blocking the decision loop. The µs disciplines still transfer wholesale; they just aim at different segments.
Why speed converts to money for a broker — three concrete mechanisms:
- A stale book is a lying cost model. The SOR splits parents using the normalized books. If one venue’s feed is seconds stale (silent WS death, missed reconnect), you route into prices that no longer exist — rejects, slippage, re-plans. Fast, health-checked market data is routing correctness. Hence: staleness stamps on every update, per-venue feed-lag tracking, and a hard rule — a venue you can’t currently see is a venue you don’t route to.
- Slow orders eat adverse selection. Between the routing decision and the child’s arrival at the venue, the market moves — and it moves against you more often than chance, because the counterparties who rest orders are watching too. Every ms shaved off the adapter path is slippage not paid. This is tick-to-trade discipline with the finish line moved: decision-to-venue-ack, measured per venue with the ack-RTT EWMA feeding straight back into the cost model as a latency handicap.
- Slow terminal-state detection freezes capital and blocks re-plans. The double-fill race (below) forces the discipline that unfilled size can only move once a child is terminal — filled, canceled, or rejected, with nothing in flight that could still execute. The faster you process acks and fills, the faster parents complete and the less reserved buying power sits idle.
And the build order the whole book has been teaching: measure first, change behavior second. Feed staleness stamps, ack-RTT EWMAs, and TCA logging change nothing about routing — they build the scoreboard. Only when the scoreboard exists do you let it drive behavior (health demotion, latency penalties, re-route policy), because otherwise you cannot tell whether any of the fast-path work paid. That’s the lesson of the latency-methodology chapter (ch08) wearing a broker suit.
Anticipating the crypto-reality section below: with internet venues, a ~10ms decision loop is genuinely adequate, and the edge is venue knowledge — quirks, health, credit — not nanoseconds. The HFT skillset’s biggest broker-side dividend isn’t raw speed; it’s the discipline — measure everything, never block the hot path, make every state transition explicit — applied to a system whose scoreboard is TCA instead of tick-to-trade.
The synthetic book: one market view, published to N clients
Your adapters maintain a normalized book per venue. The SOR reads them. The next product step — and a favorite interview design question — is publishing a consolidated (synthetic) book: one merged view of all N venues, streamed to clients. You become, from the client’s perspective, the venue — which means the market-data-publishing chapter (ch25) now applies to you, plus some broker-specific honesty rules.
Hold the pipeline as one straight line first:
venue A ws ─► adapter A ─► ΔA ─┐
venue B ws ─► adapter B ─► ΔB ─┼─► one queue ─► MERGER ─► Delta{mseq,…} ─► PUBLISHER ─► clients
venue N ws ─► adapter N ─► ΔN ─┘ (per symbol) (one thread) │
│ └─► journal (evidence)
└─► merged book, read by the SOR
Left to right: each adapter turns its venue’s feed into normalized book changes (ΔA = “on venue A, bid level 10001 now has 400”). All changes for a symbol funnel into one queue. One merger thread drains it, updates the merged book, and — the step most descriptions skip — emits deltas as a by-product of applying changes. The publisher fans those deltas out. That’s the whole machine.
Where a delta actually comes from
You never receive a delta for the merged book — you manufacture it, in the merger, as the diff your own update caused. Your mini-market lab already does exactly this on the venue side: MatchOut carries the book deltas that a match produced (ch28, Step 2). The broker version is the same move one level up.
Trace one update end to end. State before: venue A shows 250 at bid 10001, venue B shows 150 at bid 10001. The merged level is therefore 10001 → {A: 250, B: 150}, total 400, and that total is what clients currently see.
- Venue A’s websocket delivers: “bid 10001 now 400” (A’s own book-delta format, whatever it is).
- Adapter A normalizes it — canonical symbol, integer ticks — updates its local book A replica, and pushes one message to the merger’s queue:
BookChange { venue: A, side: Bid, px: 10001, qty: 400 }. - The merger applies it to the merged level:
{A: 250→400, B: 150}. Total was 400, is now 550. The total changed, so the merger emits:Delta { mseq: 8813, side: Bid, px: 10001, total: 550, by_venue: {A: 400, B: 150} }— and increments mseq. - If the total had not changed (say A revised 250→250 metadata, or the change only touched a depth tier you don’t publish), no delta is emitted. The merged book moved; the published view didn’t; clients hear nothing.
Two design choices hiding in step 3, both worth saying in an interview: deltas carry the absolute new total (“level now has 550”), not the increment (“+150”) — absolute levels are idempotent, so a client that somehow applies one twice is still correct, and conflation (below) becomes trivial. And mseq is assigned inside the single merger thread, which is what makes the stream gap-detectable: a client holding mseq 8813 that receives 8815 knows it missed one.
The structs, in the lab’s style:
#![allow(unused)]
fn main() {
struct BookChange { venue: VenueId, side: Side, px: Px, qty: Qty } // adapter → merger
struct MergedLevel { total: Qty, by_venue: SmallMap<VenueId, Qty> } // merger state, per px
struct Delta { // merger → publisher → clients; also the journal record
mseq: u64,
side: Side, px: Px,
total: Qty, // absolute: "this level now has"
by_venue: SmallMap<VenueId, Qty>, // attribution tier only
}
struct Snapshot { mseq: u64, bids: Vec<(Px, MergedLevel)>, asks: Vec<(Px, MergedLevel)> }
}
And the merger loop is a fold, nothing more:
#![allow(unused)]
fn main() {
loop {
let ch = rx.recv(); // one queue in, one thread
if stale(ch.venue) { book.evict(ch.venue); emit_evict_deltas(); continue; }
let lvl = book.level_mut(ch.side, ch.px);
let old_total = lvl.total;
lvl.by_venue.insert(ch.venue, ch.qty); // qty 0 removes the venue's contribution
lvl.total = lvl.by_venue.values().sum();
if lvl.total != old_total { // published view changed?
let d = Delta { mseq: next_mseq(), side: ch.side, px: ch.px,
total: lvl.total, by_venue: lvl.by_venue.clone() };
journal.append(&d); // evidence first
pub_tx.send(d); // then fan out
}
}
}
How publishing actually works
The publisher owns one bounded outbound queue per client (never shared — one slow client must not delay another, and observable backpressure is an information-barrier leak, per the segregation section above). The loop:
- Fast client: every
Deltais pushed to its queue; the socket writer drains it in mseq order. The client appliesbook[px] = total, deleting the level when total is 0. That’s the entire client-side algorithm — a consequence of absolute-total deltas. - Slow client (queue full): switch that client to conflation mode. Stop queueing every delta; instead keep a per-client dirty set of price levels touched since it last kept up. When its socket drains, walk the dirty set and send one delta per level with the current total — the flickers in between are gone, the end state is identical. This is why absolute totals matter: conflating relative increments would require summing them; conflating absolute levels is “just send the latest.”
- Trades are never conflated — trades are facts (ch25’s rule). If a client can’t keep up with the trade stream either, disconnect it; it re-enters through the snapshot door like any late joiner.
- Snapshot service: every K deltas (or T ms) the publisher serializes the merged book as
Snapshot { mseq, … }. A connecting client gets: latest snapshot, then every delta withmseq > snapshot.mseq, then the live stream. Gap detected mid-stream → client re-requests a snapshot. Exactly the late-joiner contract you demand from venues, now offered by you. - Entitlements are filters on the way into each client’s queue: depth tier (L1 only? ten levels? full ladder), update-rate tier (real-time vs 100ms-conflated — the conflation machinery doubles as the product knob), and whether
by_venueattribution is included or stripped. ch25’s L1/L2 product ladder, yours to sell.
“But ch25 rebuilds the feed from the sequencer — we don’t have one”
Half right, and the half matters. You have no sequencer of the market — reality already happened N times, at N venues, each with its own sequence space, and the arrival order of their updates at your doorstep is a race with no true answer (the lab’s Step 7 nondeterminism). But look at the merger loop above: one thread, draining one queue, stamping mseq on each emitted delta. That is a sequencer — of your published view, not of the market. The venue’s seq answers “what order did the market happen in”; your mseq answers “what order did we show the world to our clients in” — and the second question is precisely the one best-execution evidence needs.
The distinction rewrites recovery too. The venue replays its log because the log is reality. Your merged book is derived state — always reconstructible from upstream — so recovery is re-derivation, not replay: adapters re-snapshot from their venues (the gap-fill machinery they already have), the merger rebuilds and bumps an epoch, the publisher emits a fresh snapshot, every client bootstraps from it. The journal the merger writes (journal.append above, before fanout) exists for a different job: evidence. “What did clients see at 14:32:07” = binary-search the journal by time, replay deltas since the prior snapshot — same query shape as the venue’s explain tool (ch23), answering display instead of matching. The venue event-sources because its log is the truth; you journal because you must prove what you displayed.
Merge honesty rules
Each merged level keeps per-venue attribution — 10001 → {A: 400, B: 150} — because the SOR routes on it and sophisticated clients pay for it. Beyond that, four decisions make the book “synthetic” rather than merely summed:
- Crossed books are real. Venue A’s bid at 10002 above venue B’s ask at 10001 happens legitimately — latency skew, or fees that make the “arbitrage” unprofitable. Publish it as-is and flag it; “de-crossing” the view means publishing prices nobody can trade.
- Raw prices, not fee-adjusted. B’s 10001 plus a 20 bps taker fee can cost more than A’s 10003 at zero. You could publish an effective-price book — but fees are per-client-tier, so that book is different for every client. Standard answer: publish the raw consolidated book to everyone; the fee arithmetic lives where it already lives, in the SOR’s cost model.
- Staleness eviction. A venue whose feed has gone stale leaves the merge entirely — otherwise you are publishing phantom liquidity, levels that stopped existing seconds ago. Same rule the router follows: a venue you can’t currently see is a venue whose liquidity you don’t show.
- Self-exclusion. Your own resting child orders sit on those venues. The naive merge shows your clients your own orders as market depth — flag or subtract broker-own liquidity, or client B can be routed into crossing with client A’s order through the venue, which is the self-match problem wearing market-data clothes.
And one thing to teach clients (it will come up in their TCA reviews): ghost liquidity. The same market maker quotes on all N venues; the merged book shows several times the real depth; sweep every level at once and the maker pulls the other venues the moment the first fill prints. Consolidated depth is an upper bound, not a promise — which is precisely why post-fill markout lives in the venue scorecard.
The one-source-of-truth rule. The published book must be derived from the same merged state the SOR reads. If clients see book X while the router routed on book Y, every best-execution conversation becomes unwinnable. One merged state, one sequence — and that state snapshot is exactly what the decision log (below) records, so “what did the market look like at 14:32:07” is one lookup, serving client support, TCA, and the regulator alike.
The two-sided order state machine
One desk: one state machine per order — you versus the venue. Platform: every parent order has a client-facing state (what you’ve told the client: NEW → ACKED → PARTIALLY_FILLED → FILLED / CANCELED) and a set of venue-facing states (one per live child per venue), evolving asynchronously. The parent state is a fold: an aggregation over child states plus OMS decisions.
client view: parent: BUY 100 BTC [ACKED, filled 37.5]
▲
aggregation │ allocation
│
platform view: child 1 ──► venue A FILLED 20
child 2 ──► venue B PARTIAL 12.5, working 12.5
child 3 ──► venue C CANCEL_PENDING (re-route in flight)
child 4 ──► venue D NEW_PENDING (sent, no ack yet)
Reconciliation between the two sides is continuous, not end-of-day: every venue execution report updates a child; every child update recomputes the parent; and a periodic sweep compares platform-believed child state against venue-reported open orders (venues expose order-status queries for exactly this). Ambiguous states — order sent, no ack, session dropped — get the payments treatment you know cold: the order is state-unknown, never assumed dead, and must be resolved by query or cancel-with-confirmation before its reserved quantity is released.
The double-fill race
The canonical platform-SOR failure, and a guaranteed interview question. Sequence: the child on venue A isn’t filling → the SOR re-routes → cancel to A, new child to B → A’s fill arrives after the cancel was sent (the cancel lost the race to a match already through A’s sequencer). Both A and B fill: the client bought more than they asked for.
- Prevention: cancel-ack discipline. Never send the replacement child until venue A confirms the cancel and reports final filled quantity (a proper cancel-ack carries cumulative fill). Cost: one venue round-trip added to every re-route — 10–100ms on internet crypto venues. This is the correct default, and it is exactly the idempotency discipline from payments: don’t retry the charge until the first attempt’s outcome is known, because “probably failed” is how double-charges happen.
- Mitigation when speed forces optimism: if an aggressive algo routes to B before A’s cancel-ack, cap B’s child at parent-remaining assuming A fully fills, and run an over-fill handler: excess lands in a platform error account, gets traded out, with explicit policy (and disclosure) on whether a client ever wears an over-fill. The error account is a real subsystem with its own P&L and audit — the reconciliation suspense account of your payments world.
Say the payments version out loud in interviews — “this is the double-charge problem, except the retry moves the market” — it lands.
Best execution: your event-sourcing instinct, now legally mandated
Best execution — the broker’s obligation to take all sufficient (MiFID II, the EU regime) or reasonable (FINRA, the US regime) steps to obtain the best possible result for the client, weighing price, cost, speed, likelihood of execution and settlement, and size; both regimes are toured in ch27. The architectural forcing function: you must be able to prove, after the fact, that each routing decision served the client. So every SOR decision is logged with the market snapshot it saw — per-venue books, venue scores, cost-model inputs and outputs, at decision time — because “we routed to venue B” is only defensible as “and here is what every venue’s book looked like at that moment, and here is the arithmetic.” You built decision logging for debugging and venue scoring; here it is the compliance artifact. An event-sourced SOR gets this nearly free: the decision log is the log.
TCA — Transaction Cost Analysis — measuring execution quality after the fact — is both your internal feedback loop and a client-facing product:
- Slippage vs arrival: average fill price vs mid-market (the midpoint between best bid and best ask) at parent arrival — the headline number (“your 100 BTC buy cost 4.2 bps vs arrival”).
- vs VWAP / participation benchmarks for scheduled algos.
- Venue scorecards: per-venue fill rates, effective spread, reject rates, post-fill markout (does the price move against you right after fills there?). Markout is a toxicity signal: toxic flow is counterparties who only trade with you when you’re about to lose. You already compute venue scores for routing; TCA is the same data productized into client reports.
Analogy: the auth-rate and cost dashboards a PSP shows merchants to justify its acquirer routing — same data, same “we route in your interest, here’s proof” purpose.
Failure modes the platform must survive
- Venue down mid-parent. Children on the dead venue go state-unknown. Playbook: mark the venue unroutable (score → 0); do not release the unknown children’s reserved quantity; keep working the parent elsewhere only up to remaining-minus-unknown; reconcile on venue recovery (order-status queries or the venue’s recovery feed reveal what happened in the dark). The conservative quantity arithmetic is the whole game — optimism here is the double-fill race at venue scale.
- Client disconnect with live children. Per-client policy, configured up front: cancel-on-disconnect (safe default for takers), or keep-working — an algo running a 6-hour TWAP shouldn’t die because the client’s monitoring session dropped; the OMS owns the parent, not the client’s TCP connection. The session-vs-order-ownership distinction is the design point; ch24 covers the venue-side mirror.
- The platform’s own kill switches, layered: per-client (their breach → halt their flow, cancel their children), per-venue (venue misbehaving → stop routing, mass-cancel there), global (platform incident → everything stops, mass-cancel everywhere). Each layer gets its own drilled big red button, and the mass-cancel path must be the fastest path in the system (ch27).
Crypto spice: the venue is also a counterparty
In tradfi, the broker’s venue risk is mostly operational — central clearing means settlement risk sits with a clearinghouse, which legally becomes the buyer to every seller (and seller to every buyer), so a member’s default is the clearinghouse’s problem, not yours. In crypto, the venue holds your assets: pre-funding means the platform (or its clients) keeps balances on each exchange, and an exchange failure is a credit loss, not an outage. FTX made it concrete: routing 100% of flow to the venue with the best prices was catastrophic when that venue was also insolvent.
Engineering consequences:
- The venue score includes credit and counterparty terms, not just latency and fees: withdrawal-latency monitoring (withdrawals quietly slowing is the canonical early-warning signal), proof-of-reserves posture, jurisdiction, and hard concentration caps (“never more than X% of platform assets on one venue”). Your venue scorer grows slow-timescale risk inputs alongside the fast microstructure ones.
- Treasury/rebalancing as a first-class subsystem: moving balances between venues so routable inventory sits where the flow is, netting against withdrawal fees and on-chain confirmation times (minutes, sometimes hours). This is the multi-currency treasury problem from payments — prefund the local rails where the volume is, sweep to safety otherwise.
- Stablecoin/fiat legs:
BTC-USDon one venue andBTC-USDTon another are different instruments with an FX-like basis (a persistent price gap, like a currency pair that never quite sits at 1.00); the SOR either keeps them as distinct books or explicitly models the USDT/USD leg. Pretending stablecoin = USD is a routing bug with a case study: in the March 2023 USDC depeg, routers that hardcoded $1.00 “arbitraged” themselves into depegged inventory. - 24/7, no close: no end-of-day window for reconciliation, upgrades, or resets. Every maintenance operation is a live operation (ch16); reconciliation is continuous; “we’ll fix it after the close” is not in the vocabulary.
Plain-English recap
- A broker/execution platform is your one-desk SOR wrapped in a multi-tenant OMS: OMS = the durable what (PaymentIntent), EMS = the how (retry/orchestration), SOR = the where (acquirer selection).
- Going from 1 desk to N clients multiplies four things: risk moves before routing and becomes per-client and (in the US) legally mandatory; client flow must be segregated like adversarial tenant data; competing clients need a documented, deterministic allocation policy; and fees enter the cost model carrying a conflict of interest that best-ex rules police.
- The venue adapter layer — instrument master, order-semantics normalization, health scoring — is the product moat; its new multi-tenant problem is fairly sharing each venue’s rate limit across clients, with cancels always winning.
- The HFT skillset transfers to the broker seat with the finish line moved: speed pays through routing correctness (never route on a stale book), less adverse selection (decision-to-ack, EWMA-scored per venue), and faster re-plans (terminal-state discipline) — measured first, behavior-changing second, with TCA as the scoreboard instead of tick-to-trade.
- Every parent order is two state machines — client-facing and venue-facing — continuously reconciled; the double-fill race on re-route is the double-charge problem, and cancel-ack discipline is its idempotency key.
- Best execution turns decision logging into a legal obligation: every routing decision stored with the market snapshot it saw; TCA (slippage vs arrival, venue scorecards) is the client-facing proof.
- In crypto the venue is also a counterparty: withdrawal monitoring, concentration caps, and treasury rebalancing belong in the router’s venue score, next to latency and fees.
Interviewer will ask
Q1: “You built an SOR for one desk. What actually changes when it serves external clients?” I’d name the multiplications rather than list features. Risk checks move in front of the router and become per-client and mandatory — buying power with hold/release semantics on working orders, which is card-auth-hold logic I’ve built before. Client segregation becomes an engineering property: no shared state whose timing or backpressure leaks one client’s flow to another. Allocation between competing clients needs a documented, deterministic, replayable policy — at one desk that question doesn’t exist. And every routing decision becomes evidence: logged with the market snapshot it saw, because best execution means proving the route served the client. My existing SOR — cost model, venue scoring, 20+ adapters — is the engine; the productization is the OMS shell and the fairness-and-evidence layer around it.
Q2: “Untangle OMS, EMS, and SOR.” OMS owns the what: the parent order as a durable business object, plus accounts, buying power, allocations — the system of record, like a PaymentIntent in Stripe. EMS owns the how: working the parent over time — algo choice, slicing schedule, urgency. SOR owns the where: for one child right now, which venue, given fees, books, and fill probability — the acquirer-selection step. In practice EMS and SOR blur into one engine, and what I built was an SOR with EMS behaviors and a thin single-tenant OMS. The multi-client OMS is the part I’d build fresh, and I’d build it event-sourced, because allocation determinism and best-ex evidence both demand exact replay.
Q3: “Venue X gives you 100 orders/sec total. Client A’s algo wants all of it. Go.” A doesn’t get all of it, and here’s the second-by-second version. Say A’s contracted share is 40 of the 100. A’s algo bursts 100 orders in one second: the first 40 pass. If B and C are quiet this second, their unused share redistributes, so A might actually get 85–90 — but never the last slice, because a floor stays reserved so that B’s next order doesn’t have to wait behind A’s burst. Whatever A sends beyond its share comes straight back rejected with an explicit “throttled by platform” reason — not silently queued. Why not queue it? A queued order executes later at a different price; silently changing a client’s execution price is a best-execution violation, but an explicit reject lets A’s algo make its own choice: re-pace, or route the flow to another venue. Now the moment that matters: mid-burst, B sends a cancel. The cancel does not join any line — cancels preempt new orders, always, because a cancel that queues behind A’s entries means B is locked into market risk they’re trying to exit. That’s a risk event, not a fairness event. So the picture is API-gateway per-tenant rate limiting with three trading-specific amendments: the upstream quota is hard (the venue enforces it), cancels preempt, and throttling is explicit. I ran per-venue rate budgets for one desk; the per-client scheduler that decides whose order passes this second is the genuinely new layer.
Q4: “Walk me through the double-fill race and your answer to it.” Child on venue A, not filling; I re-route: cancel to A, new child to B. A’s fill was already through its sequencer when my cancel arrived, so both venues fill and the client is over-bought. Prevention is cancel-ack discipline: don’t send B’s child until A confirms the cancel with final cumulative quantity, paying one venue round-trip per re-route. Same idempotency rule as payment retries — never retry until the first attempt’s outcome is known, because “probably failed” is how double-charges happen. And that round trip is exactly where the speed work pays: the faster I turn A’s cancel-ack into a terminal state, the sooner the parent’s reserved quantity is free to move — the cost buys certainty, and the ack-path speed buys the cost back. Where an aggressive algo can’t wait, I cap the new child assuming worst-case fill on A and run an error account for over-fills, with explicit policy on whether a client ever wears one. Prevention is the default; optimism is opt-in and accounted for.
Q5: “A client claims a fill wasn’t best execution. Prove them wrong — or right.” The proof exists because I decided to log for it on day one: the SOR is event-sourced and every routing decision carries its full input snapshot — per-venue books, fees, health scores, cost-model output — keyed by decision time. So “why venue B at 14:32:07” gets a replayable answer: here’s what every venue showed, here’s the arithmetic, B won on all-in client cost including expected slippage. TCA closes the loop: that parent’s slippage vs arrival, and B’s scorecard showing the fill wasn’t an outlier. It cuts both ways: the replay can show the router was wrong — and that’s what the scoreboard exists for. The staleness stamp on every book update and the per-venue ack-RTT EWMAs sit inside the decision snapshot too, so the same log that defends a good route exposes a stale feed or a bad score when that’s the true story — and then it’s how I fix it. I built this logging at Crypto.com for debugging; the upgrade is treating it as a compliance artifact with retention and tamper-evidence.
Q6: “How is routing crypto different from routing equities?” Three structural differences. First, the venue is a counterparty: pre-funded balances mean venue failure is a credit loss, so my venue score carries slow risk signals — withdrawal-latency trends, concentration caps — next to the fast microstructure ones; FTX is why “best price” can’t be the only axis. Second, the instrument space is messier: BTC-USD and BTC-USDT are different instruments with a real basis, so the router keeps them as separate books or explicitly models the stablecoin leg — hardcoding a stablecoin at $1.00 is a bug with a 2023 case study, the USDC depeg. Third, 24/7 with no close: reconciliation, treasury rebalancing, and deploys are continuous live operations. The compensation is that crypto venues are internet-distant, so my ~10ms routing budget was genuinely adequate — the edge was venue knowledge, not nanoseconds.
Q7: “Two clients’ buys compete for one resting offer. Who gets it, and how do you defend that?” Whatever the answer, it must be written, deterministic, and replayable — the indefensible position is having no rule. My default: strict time priority of parent arrival at the platform — simple, incentive-compatible, and it mirrors what the venue itself does. If the platform aggregates parents into shared children, fills allocate back pro-rata by a documented formula computed inside the event-sourced fold, so the same fills always produce the same allocation. Allocation decisions get the same evidence logging as routing decisions, because the disadvantaged client in a fast market is exactly who audits you. And I’d want product and compliance in that design review — the policy is a business commitment, not an engineering preference.
Q8: “A venue goes dark with your child orders live on it. Next 60 seconds?” Immediately: venue marked unroutable, its children flip to state-unknown, and — the key move — their reserved quantities are not released. The parent keeps working on other venues only up to remaining-minus-unknown, so the worst case (every dark order filled) cannot over-fill the client. Affected clients get the honest state: quantity X unconfirmed on venue Y. Then reconciliation on reconnect: order-status queries or the venue’s recovery feed resolve the dark window, and reserved quantities settle into fills or releases. The discipline, from having lived venue outages across 20+ integrations: unknown quantity is treated as filled for risk purposes and as nothing for revenue purposes — conservatism points in exactly one direction.
Q9: “You come from HFT infrastructure. Where does that skillset actually pay at a broker — and where doesn’t it?” I’d start where the kernel-tuning chapter (ch03) taught me to start: with the budget table, not the tool. Cost each segment of a child order’s path: the WAN hop to an internet venue is 1–70ms, and you buy that down with placement — region, colo — you don’t code it away; the venue’s own processing is theirs; my adapter path — sign, session, submit — is 1–50ms and entirely mine. So the rule is optimize the biggest controllable term, and for an internet-venue broker that’s almost never the kernel — it’s warm sessions, pre-computed auth, and a decision loop that never blocks. That’s where the milliseconds live. Speed then converts to money through three specific mechanisms: a stale book is a lying cost model, so fast health-checked market data is routing correctness; every ms on the adapter path is adverse selection, because the market moves against you between decision and arrival; and slow terminal-state detection freezes reserved buying power and blocks re-plans. But the build order is measure first — staleness stamps, ack-RTT EWMAs, TCA logging change no routing behavior at all; they build the scoreboard — and only once the scoreboard exists does it drive health demotion and latency penalties, because otherwise I can’t tell whether any of the fast-path work paid. With internet venues my ~10ms loop was genuinely adequate, and the edge was venue knowledge — quirks, health, credit. What the HFT seat really hands the broker seat is the discipline — measure everything, never block the hot path, make every state transition explicit — pointed at a scoreboard that reads TCA instead of tick-to-trade.
Further reading
- SEC Rule 15c3-5 (Market Access Rule) — the adopting release (SEC Release 34-63241) is readable and is the canonical statement of “pre-trade risk at the broker is mandatory.”
- MiFID II best-execution materials — ESMA’s best-execution Q&As and the (now-retired) RTS 27/28 reporting regime, for what “prove your routing” means operationally in the EU.
- FIX Trading Community specifications — ExecutionReport, order-state, and allocation message semantics; reading the FIX order-state model is the fastest way to internalize the two-sided state machine.
- Talos engineering blog and product documentation — the closest public description of the multi-client execution-platform architecture this chapter describes.
- Larry Harris, Trading and Exchanges — the chapters on brokers and order routing; old, but the conceptual frame for best execution and agency conflicts is unchanged.
Where this goes next: every arrow in this chapter’s diagram crosses a risk check — Chapter 27 specifies the control plane both venue and broker must carry: pre-trade budgets, kill switches, drop copy, surveillance, and the regulators forcing all of it.