Gateways, Sessions & Fairness at the Edge

Before you start. This chapter assumes the sequencer-centric exchange layout (gateways → sequencer → deterministic consumers, ch23), TCP session behavior (connections, retransmits, what a dropped connection means, ch02), the FIX / OUCH / ITCH jargon (ch00d), timestamping and clock discipline (ch07), and latency budgeting by component (ch08). If any are new, read those first.

For years you were on the outside of this door: twenty venues, twenty API-key schemes, twenty flavors of rate limit, twenty ways to get disconnected at the worst moment. This chapter is the view from inside the door. The order gateway is the venue’s edge tier — the only component that talks to untrusted parties — and its job description is a contradiction: be maximally paranoid (auth, risk, rate limits, abuse defense) while adding minimal and equal latency for everyone. Everything interesting about gateway design falls out of that tension.

The gateway’s job, end to end

                ONE ORDER'S PATH THROUGH THE GATEWAY  (budget: single-digit µs total)

 client ──TCP──►┌──────────────────────────────────────────────────────────────┐
                │ ① ingress timestamp        (HW tstamp at NIC where serious)  │
                │ ② session layer            (auth'd? seq numbers ok? HB ok?)  │
                │ ③ protocol translation     (FIX 4.4 / OUCH / WS-JSON         │
                │                             → internal fixed binary msg)     │
                │ ④ pre-trade risk, in-line:                                   │
                │      • fat-finger limits    (qty/notional caps)      ~100ns  │
                │      • price bands          (limit px vs reference)  ~100ns  │
                │      • self-match check tag (attach STP group id;    ~50ns   │
                │        STP = self-match prevention, §below)                  │
                │      • credit/position      (per-account exposure)   ~200ns  │
                │ ⑤ rate limiter             (token bucket per session) ~50ns  │
                │ ⑥ stamp session id + gateway id + ingress time,              │
                │    forward to SEQUENCER                                      │
                └──────────────────────────────────────────────────────────────┘
                          │                                    ▲
                          ▼                                    │
                     [ sequencer ]                    acks/fills routed back
                                                      to the owning session

Your mental model from the SaaS world is right, and one adjustment away: this is an API gateway + auth middleware + rate limiter in front of a monolith — except the “monolith” is one sequenced pipe for the whole market, and the p99 budget for the entire middleware stack is microseconds, not milliseconds. Every check in box ④ has an explicit per-check latency budget, the way you’d budget a Postgres query — but the budgets are in nanoseconds, which forces the implementation style: no allocation, no locks, no syscalls, all limits pre-loaded into gateway-local memory and updated out-of-band.

Walk the boxes:

② Session auth. In FIX land: a Logon message opens a session identified by (SenderCompID, TargetCompID), authenticated by credentials and increasingly by TLS or network identity (in colo, “you’re on port 7 of my switch” is part of auth). In crypto land: API key + HMAC signature per request or per connection — the scheme you implemented twenty times as a client. Either way the gateway resolves connection → account → risk profile once at logon and caches it, because re-resolving per order costs a lookup you can’t afford.

③ Protocol translation. Clients speak FIX (verbose, tag=value, self-describing), OUCH (NASDAQ’s lean fixed-width binary order protocol — the whole spec is a few pages), or WS-JSON (crypto). The internal world downstream speaks one fixed-layout binary format, because the sequencer and engines must never pay parsing costs (ch00c). The gateway performs the last parse. Translation cost is why serious venues offer a native binary protocol next to FIX and why their FIX gateway is documented as slower — a fact your SOR exploited when it chose which protocol to send on.

④ Pre-trade risk, in-line, before the sequencer. This ordering is a design decision that earns its place three times over. First, a rejected order must never consume a sequence number or touch a book — the sequenced log is the market’s official history (ch23), and garbage doesn’t belong in the record. Second, the sequencer is the one single-threaded resource the whole market funnels through, so every cycle it spends stamping a message that was never eligible is capacity stolen from legitimate flow — precisely during the bursts when that capacity is scarce. Third, a reject decided at the edge turns around inside the gateway in microseconds, with no round trip through the core and back — the client gets a fast, explicit answer from the one tier that’s cheap to scale. The checks:

Fat-finger limits — max order quantity/notional per message: the “did a human or a bug add three zeros” check, the per-transaction card limit of the PSP edge. It matters because one fat order can walk an entire book before anyone reacts.
Price bands — reject limit prices further than X% from a reference price — the venue-side companion to circuit breakers.
Credit / position limits — per-account max exposure, maintained gateway-locally with async reconciliation — the check that turns “client went insolvent” from a venue loss into a rejected order. (In tradfi this is regulatory: SEC 15c3-5, the “market access rule,” legally requires pre-trade risk checks on every order before it reaches the market. Sponsored access is a member lending its exchange access to a client; doing that without checks — “naked access” — was banned in 2010.)
Self-match tagging — attach the account’s STP group so the engine can act on it (below).

⑤ Rate limiting. A token bucket per session — plain English: you accrue N message-credits per second up to a burst cap; each message spends one — you spent years on the receiving end of exactly this, engineering your order flow to stay under venues’ buckets. From this side, its purpose is crisp: the sequencer is a shared funnel, and one runaway client must not be able to consume the market’s headroom. Same reason your multi-tenant Postgres has per-tenant connection caps: protect the shared resource, make blast radius per-tenant.

Sequence-number bookkeeping per session. FIX sessions number every message in both directions (MsgSeqNum), and both sides track expected-next. This is not the sequencer’s global number — it’s per-session, and it exists for gap detection and recovery on that session (see the state machine below). Your idempotency-key discipline from payments is the same instinct: both sides must agree on exactly which messages were exchanged, and replays must be detectable.

The protocol zoo at the front door

The gateway speaks whatever its clients speak, and the choice each client makes is itself informative:

	FIX	OUCH-style native binary	WS-JSON (crypto)
Encoding	tag=value ASCII (`35=D\|55=AAPL\|...`)	fixed-width binary, ~10 msg types	JSON over WebSocket
Typical msg size	200–400 bytes	30–50 bytes	300–800 bytes
Gateway parse cost	~0.5–2 µs	~50–200 ns	~1–5 µs (JSON + TLS)
Session recovery	full resend machinery (below)	thin — relies on daily reset + drop copy	ad-hoc: REST resync
Who uses it	institutions, brokers, anything cross-venue	latency-sensitive prop flow	everyone in crypto

The pattern to articulate in an interview: FIX is the venue’s compatibility API, native binary is its performance API, and serious venues ship both — same resource, two contracts, exactly like a REST API next to a gRPC endpoint in your world.

FIX won institutional trading not on merit of encoding (the encoding is famously awful) but on network effects: every OMS (order management system — an institution’s system of record for orders), every broker, every compliance system already speaks it. It’s the SWIFT of trading connectivity. That network effect is also why Talos — a vendor whose product is exactly this layer, an institutional FIX face over twenty bespoke crypto WS APIs — exists as a company, and why Talos is the shape of your interview target: the wrapper itself is a product.

Your SOR chose the native protocol wherever a venue offered one; now you know the gateway-side reason it was faster: you were skipping the expensive parse row of this table.

Identity is a hierarchy, not a flat key. The session that connects is the bottom of a tree the gateway must understand, because risk limits attach at every level:

   FIRM (member)                    ── firm-wide credit/exposure cap
     ├─ ACCOUNT A (prop desk)       ── per-account position & notional limits
     │    ├─ session 1 (colo, OUCH)      ── per-session rate, COD config
     │    └─ session 2 (backup, FIX)
     └─ ACCOUNT B (client of the member, sponsored access)
          └─ session 3                   ── 15c3-5 checks applied by the
                                            SPONSOR's limits, at YOUR gateway

Multi-tenant SaaS shape again — org → project → API key, quotas at each tier, exactly your Postgres tenant hierarchy — with one venue-specific twist: limit checks roll up (session 1’s order must pass session, account, and firm caps), and the firm-level check is the one that can’t be sharded away, since two sessions on two different gateways draw on one shared credit number.

Here’s how that shared number gets checked without a cross-gateway lock. Say the firm’s cap is $10M and its sessions land on three gateways. Each gateway holds a $2M lease carved out of the cap and admits orders against its lease locally — a nanosecond memory check, no network hop. The remaining $4M stays with the credit service, which reconciles actual usage continuously and re-sizes leases out-of-band, shrinking them as the firm approaches its cap. The trade you’re making: a small, bounded over-admission risk (leases can momentarily add up to more than what’s truly left) instead of a synchronous cross-gateway round trip on every order. The trade has a limit, though: as the firm approaches its cap the leases shrink, and eventually the possible over-admission is no longer small next to what’s genuinely left — so near the cap, large orders fall back to a synchronous check against the shared number, paying the round trip only when the error would matter. Same pattern as distributed rate limiting across your API fleet: local buckets, eventual global truth, bounded error.

When the bucket runs dry: reject, don’t queue

A detail you experienced from the client side and now get to decide as the operator: what happens to the message that exceeds the rate limit?

   token bucket empty:
     Option A: QUEUE the message until tokens accrue     ◄── tempting, wrong
     Option B: REJECT immediately with an explicit error ◄── correct

Queueing feels polite — it’s what your HTTP-world rate limiters often did — but it’s wrong here for a reason specific to trading: a delayed order is a different order. An instruction priced against the book at T, silently executed against the book at T+50ms, is not what the client sent — you’ve converted their limit order into a worse one without telling them.

Rejection with an explicit, machine-readable throttle error returns the decision to the only party who has current context: the client, whose strategy can re-price, re-route (your SOR did exactly this — venue throttled → next venue in the table), or drop the intent.

The corollary you lived: venues that silently queued under load were the ones whose ack latencies went bimodal and whose fills came back mysteriously stale. As the operator, the rule is: the gateway may be a filter, never a buffer. The same logic is why pre-trade risk rejects synchronously instead of parking orders for review — in this domain, fast explicit failure is a feature, and every queue you add between client and sequencer is latency variance you’ll have to explain.

One refinement venues actually ship: separate buckets (or reserved headroom) for cancels. Throttling a client’s new orders protects the market; throttling their cancels traps them in positions during exactly the volatile moments when buckets empty — which is dangerous for them and, if the venue extends credit, for you. “Cancels always get through” (or get a much deeper bucket) is a common and defensible asymmetry, and a good interview flourish because it shows you’re reasoning about the risk semantics of message types, not treating throughput as undifferentiated load.

A worked latency budget

Numbers make the design concrete. A respectable software gateway (no FPGA), door-to-sequencer:

Stage	Budget	Notes
NIC → userspace (kernel bypass, ch04)	~1–2 µs	or ~4–8 µs through the kernel stack (ch01)
Ingress timestamp + session lookup	~50–100 ns	pre-resolved at logon; one predictable read
Protocol decode (OUCH/binary)	~50–200 ns	FIX tag=value parse: ~0.5–2 µs — the price of verbosity
Risk-check chain (④, all checks)	~0.5–1 µs	each check ~50–200 ns, in-cache, branch-predictable
Token bucket + stamp + enqueue to sequencer link	~100–200 ns
Total, gateway ingress → sequencer inbound	~2–4 µs	FIX clients pay ~1–2 µs more than binary clients

Human scale: the entire paranoid middleware stack — auth, parse, five risk checks, rate limit — fits in the time one accidental syscall would cost (ch00b). That’s the discipline: the budget doesn’t survive a single allocation, lock contention, or cache miss on a cold limits table, which is why limits are pre-loaded, structures are fixed-layout, and the hot loop runs pinned on an isolated core with the ch03 treatment. And it’s why hardware-assisted venues (FPGA risk checks at the NIC) can quote sub-microsecond gateways: box ④ is embarrassingly parallel fixed-function logic — each check is independent of the others, and what runs branch-predictable in software unrolls to branch-free logic in hardware — the most FPGA-shaped workload in the whole venue.

Fairness at the edge

Every gateway adds latency. So the fairness rules are:

   client A ──► gateway 1 ──┐
   client B ──► gateway 1 ──┤        RULE 1: within a session/gateway,
   client C ──► gateway 2 ──┼──► sequencer      arrival order preserved
   client D ──► gateway 3 ──┘        RULE 2: ACROSS sessions, order is
                                     decided ONLY at the sequencer
        gateways must be interchangeable:
        same hardware, same code, same path length to the sequencer

Per-session ordering is guaranteed by the gateway (it’s a TCP stream; keeping order is free).
Cross-session ordering is decided only at the sequencer (ch23). The gateway must not create ordering — it must merely not distort the race. Hence: gateways are interchangeable (identical hardware, identical code path, no “premium gateway” unless it’s a disclosed product), and gateway→sequencer links are latency-equalized.
The famous tradfi expression of this: equalized cable lengths in colo. Exchanges like NYSE literally provision the same optical fiber length from every colo cage to the matching-engine access switch — a client one rack closer gets the same propagation delay as one across the hall. It sounds like theater until you remember Part 0’s numbers: light in fiber covers ~1m in 5ns, cages differ by tens of meters, and 100ns is a winnable race margin. Know this as lore; interviewers use it to test whether you grasp that fairness is enforced physically, not just in software.

The gateway’s fairness obligations, stated as invariants: never reorder within a session; never add client-dependent latency (no per-client code paths on the hot path); timestamp at ingress so distortions are measurable. That last one is your ch08 discipline turned into a compliance artifact — venues monitor per-gateway latency distributions and investigate skew, because skew is a fairness incident, not a perf bug.

Self-match prevention and cancel-on-disconnect

Two features you used as a client, now seen from the operator’s side:

Self-match prevention (STP) stops one firm’s buy from trading against the same firm’s sell. Why clients want it: a firm running many independent strategies (or your SOR racing your own maker flow) can accidentally cross with itself, and in regulated markets self-trades look like wash trading (fake volume, potentially market manipulation) — STP shields the client from that compliance exposure.

The venue has its own stake: printed volume that’s really one firm trading with itself is garbage data, and a tape polluted with self-crosses undermines the venue’s core product. Preventing them protects the venue’s credibility, not just the client’s.

Mechanics: orders carry an STP group ID, tagged at the gateway (④). When the engine detects that an aggressing order would match a resting order in the same group, policy applies — cancel-newest (aggressor dies), cancel-oldest (resting order dies), or cancel-both. Note the split of labor: the gateway tags (it has the account context), the engine enforces (only it sees both sides of the prospective match). A gateway can’t do STP alone — it never sees the book.

Cancel-on-disconnect (COD): when your session drops, the venue pulls all your resting orders. You configured this on every venue that offered it, because a maker with dead connectivity and live quotes is a sitting duck. From the venue side it’s equally self-interested: orders whose owner can’t manage them are stale liquidity that will print bad trades, generate disputes, and — for the venue extending credit — grow unmanageable exposure.

Design detail that matters: COD must trigger on effective death, not just TCP FIN — which is what heartbeats are for (below), and why heartbeat intervals (commonly 1–30s, negotiated at logon) are effectively the COD detection latency. The gateway turns “session declared dead” into a burst of cancel messages injected into the sequencer like any other flow — which itself is a burst-capacity line item: one big maker disconnecting can be 50k cancels in one shot.

Abuse: quote stuffing and messaging economics

The gateway is also the venue’s abuse-defense tier. The canonical attack is quote stuffing: flooding the market with orders and cancels you never intend to trade, to congest competitors’ feed processing or the venue itself. Think of an application-layer DoS that is fully authenticated and well-formed — a logged-in API client hammering a legitimate endpoint. That’s exactly what makes it hard: classic DoS defenses (drop unauthenticated junk at the edge) don’t apply, because every message is from a paying member.

Venues respond on two axes:

Technical: the per-session token buckets (⑤); per-account aggregate limits across sessions; port-level throughput caps in colo.
Economic: order-to-trade ratio policies and messaging fees. Your messages are free until your ratio of orders to actual trades crosses a threshold (venues set thresholds anywhere from tens to ~500:1 depending on the program); after that, each excess message costs money. It’s API pricing tiers: the free tier is sized for legitimate use, and abuse gets priced out rather than blocked. When your abusers are authenticated paying customers, pricing is a better throttle than blocking, because it scales pressure smoothly and doesn’t require you to adjudicate intent.

Scale shape: a wide, shallow fleet

        thousands of sessions × modest per-session rate

  5,000 sessions ──► [ gw1 ] [ gw2 ] ... [ gwN ]  ──► one sequencer
        │                 (sticky: a session lives
        │                  entirely on one gateway)
        └── p99 per-gateway budget: single-digit µs, flat under load

The gateway tier’s load profile is the opposite of the sequencer’s: wide and shallow. Thousands of concurrent sessions, each individually modest (a big market maker might run 1–10k msgs/s per session; most sessions do far less), summing to the market’s total. So it scales horizontally like any stateless-ish edge tier.

The one big caveat: sessions are sticky. A FIX session’s state (auth, sequence numbers, in-flight orders, heartbeat timers) lives on one gateway; you can’t round-robin messages of one session across the fleet. Losing a gateway = bouncing its sessions = every affected client runs recovery (below) and COD fires for those who configured it. This is your multi-tenant SaaS scaling story — horizontal fleet, per-tenant stickiness, blast radius = one box’s tenants — with the twist that “failover” is client-visible by design, because the protocol makes session death explicit rather than hiding it behind a retry.

Session state machines: FIX mechanics for a WebSocket native

You know this dance from the other side — twenty venues’ worth of reconnect/re-auth/resubscribe logic. FIX formalizes it:

            ┌────────┐   Logon(seq, HB interval)    ┌─────────────┐
   TCP up ─►│ CONNECT│ ────────────────────────────►│ ESTABLISHED │◄─┐
            └────────┘      (auth + seq negotiate)  └──────┬──────┘  │
                                                           │         │ Heartbeat
                     no msg for HB interval ───────────────┤         │ every N sec
                                                           ▼         │ (both ways)
                                                   ┌──────────────┐  │
                                                   │ TestRequest  │──┘ reply in time
                                                   │  sent (ping) │
                                                   └──────┬───────┘
                                        no reply          │
                                                          ▼
            ┌────────┐    Logout / force close    ┌──────────────┐
            │ CLOSED │◄───────────────────────────│ DECLARED DEAD│──► trigger COD
            └────────┘                            └──────────────┘

Mapping to your WebSocket world: Logon = connect + auth handshake; Heartbeat = ping/pong frames; TestRequest = “I haven’t heard from you, prove you’re alive” (an explicit, in-protocol ping with a required correlated reply — WS ping/pong made mandatory and audited); Logout = graceful close. The part with no clean WS analogue is recovery:

Resend Request — when a session re-establishes mid-day, both sides compare sequence numbers. If the client logs on saying “my next inbound should be 8,001” but the gateway already sent through 8,240, the client issues a Resend Request for 8,001–8,240 and the gateway replays those application messages from its per-session outbound store. What this means for you as the operator:

Every gateway keeps a durable-enough per-session outbound message store for the day — a real storage/retention design item, not a nicety.
Replayed execution reports are flagged (PossDup) so the client’s idempotency layer — and this is literally your payments idempotency-key pattern, down to the semantics — can distinguish “new fill” from “redelivery of a fill you may have seen.”
The dangerous asymmetry: a reconnecting client mostly wants your messages resent (did I get filled while dark?); for orders the venue missed from the client, resending is usually wrong — you don’t want 90-second-old order instructions entering today’s market, so convention is to gap-fill them with sequence resets rather than replay them. Stale intents die; facts get redelivered. Get this backwards as an operator and you inject a client’s dead orders into the book — the kind of incident that ends up in a venue’s disciplinary notices.

Compare crypto: most venues punt — on reconnect you re-auth, re-subscribe, and poll REST for open orders and recent fills to resync. Same problem, coarser tool. Having implemented the coarse version twenty times, you can explain precisely why the FIX version exists: it turns “resync after disconnect” from a bespoke racy dance into a protocol-guaranteed replay with explicit dup marking.

What the gateway operator watches

Running a gateway fleet, your observability (ch11) centers on a short list — worth having ready because “what would you monitor?” is a standard interview close:

Per-gateway ingress→sequencer latency distributions, compared across gateways. Absolute p99 matters; skew between gateways matters more — skew is a fairness incident (some clients’ door is slower than others’), and sophisticated clients will find it before you do. Alert on divergence, not just degradation.
Reject rates by reason code, per account. A spike in fat-finger rejects on one account = their bug (call them before they call you — the venue-side version of a PSP flagging a merchant’s retry storm). A spike across many accounts = your bug — a bad reference price feeding the price bands is the classic: suddenly every legitimate order is “outside the band,” and you’ve effectively halted the market from the edge. Reference-price staleness needs its own alarm.
Token-bucket saturation per session — who is riding their limit, which is both a capacity-planning signal and a sales lead (“your flow has outgrown your tier”).
Heartbeat/TestRequest statistics and COD firings — a cluster of sessions going dark together is a network incident on your side, not twenty coincidences; auto-correlating “who died together” by gateway/switch/path is the fastest triage signal you have.
Sequence-gap and resend-request counts per session — a client constantly gap-filling has a flaky path or broken engine; either way it becomes your support ticket and, in the disputes above, your evidence.

Plain-English recap

FIX is the compatibility API, native binary is the performance API — REST next to gRPC — and the identity model under both is a multi-tenant hierarchy (firm → account → session) with limits that roll up, enforced via local leases against shared caps rather than cross-gateway locks.
The gateway is an API gateway + auth middleware + rate limiter in front of a monolith — except the monolith is one sequenced pipe, and the whole ingress stack — auth, decode, five risk checks, rate limit — fits a 2–4 µs budget, roughly the cost of one stray syscall. Hence: no locks, no allocation, no syscalls, no cold cache lines, limits cached in local memory.
Pre-trade risk runs before the sequencer so a rejected order never consumes a sequence number — and in US markets, pre-trade checks are literally the law (15c3-5), not a best practice.
Rate limits you suffered as a client are, from this side, per-tenant caps protecting a shared funnel — same logic as connection caps on your multi-tenant Postgres.
Fairness at the edge = gateways are interchangeable and add equal latency; ordering across clients is created only at the sequencer. Tradfi enforces this down to equalized fiber lengths in the colo hall.
Self-match prevention: gateway tags, engine enforces — because only the engine sees both sides of a prospective match. Cancel-on-disconnect: heartbeat interval is your detection latency, and one dead maker means a 50k-cancel burst.
Abusers here are authenticated paying customers, so the best throttle is pricing (messaging fees on high order-to-trade ratios) layered on top of token buckets.
When the rate limit trips: reject explicitly, never queue — a delayed order is a different order, and silent buffering converts clients’ instructions into worse ones. Cancels get a deeper bucket, because trapping a client in a position helps no one.
FIX session recovery is your payments idempotency discipline as a wire protocol: redeliver facts (fills, flagged PossDup), never replay stale intents (old orders).
The operator’s dashboards watch fairness, not just health: cross-gateway latency skew, reject-reason spikes (one account = their bug; all accounts = your reference price), and sessions dying together (your network, not coincidence).

Interviewer will ask

“You consumed 20 venues’ gateways. Now design one. What’s on the hot path and what’s not?” “Hot path, in order: ingress timestamp, session lookup (pre-resolved at logon, one cache-friendly read), protocol decode into our internal binary, pre-trade risk as a chain of pure in-memory checks — fat finger, price band, STP tag, credit — each budgeted at ~50–200ns, token bucket, forward to sequencer. Off the hot path: auth resolution (logon-time), limit updates (pushed out-of-band from a risk service, gateway just reads local memory), the per-session outbound store writes (async, but must be durable enough to serve resend requests), and all observability via the ch11 patterns. The design rule I’d carry from the client side: anything that can vary per-client must not vary in latency per-client, or I’ve built an unfair gateway.”

“Why must risk checks run before the sequencer rather than in the matching engine?” “Picture the funnel from ch23: everything the market does squeezes through one sequenced pipe, and the gateway is the door in front of it. Three reasons the risk gate sits at the door. The sequenced log is the market’s official, replayable history, so a rejected order must never consume a sequence number — garbage doesn’t get a place in the record. The sequencer is the scarcest single-threaded resource in the building, so cycles it spends stamping ineligible messages are capacity stolen from legitimate flow — exactly during the bursts when everyone’s fighting for it. And a reject decided at the edge turns around in microseconds inside the gateway, with no round trip through the core — a fast explicit answer from the tier that’s cheap to scale. The one check that can’t live at the edge is anything needing book state — the actual self-match decision at match time — which is why STP is split: gateway tags the group ID, engine enforces. I saw both halves as a client: instant rejects from the door, STP cancels only at the moment a match would have printed.”

“A big market maker’s session drops at 14:30 on CPI day. Walk me through what your venue does.” “Heartbeat/TestRequest declares the session dead — worst case one heartbeat interval plus the test-request timeout, so if they negotiated 5s heartbeats, up to ~10s of dark time. If they’ve armed cancel-on-disconnect, the gateway injects cancels for all their resting orders into the sequencer — potentially tens of thousands of messages, which my burst budget must include, on a day that’s already 10–100×. Their per-session outbound store retains every execution report from the dark window. When they reconnect: logon, sequence-number comparison reveals the gap, they issue a Resend Request, I replay their fills flagged PossDup so their idempotency layer dedupes. Having been the client in exactly this scenario, the thing I’d obsess over as the operator is that replayed fills are complete and correctly flagged — a missing fill during a disconnect is how clients end up unknowingly short on a volatile day, and it becomes the venue’s dispute.”

“How do you stop quote stuffing without hurting legitimate market makers?” “The hard part is the chapter’s framing: the attacker is an authenticated paying member sending well-formed messages, so I can’t drop them at the edge like anonymous DoS junk. So the defense layers. Token buckets per session and per account are the technical ceiling, sized so legitimate maker behavior never touches them. Then economics does the discrimination that intent-guessing can’t, and the discriminator falls out of the order-to-trade ratio. A legitimate maker cancels constantly because they’re re-quoting — many hold quoting obligations, contractual commitments to keep two-sided quotes posted, so every price move forces a cancel/replace — but that churn resolves into actual trades, so their ratio stays inside the venue’s program thresholds. A stuffer’s messages exist only to congest; almost none ever trade, so their ratio goes pathological. Hence the policy: order-to-trade thresholds with messaging fees above them — pricing applies pressure smoothly, scales with the abuse, and never requires me to adjudicate intent in real time. I lived under these regimes as a client and they shaped our SOR’s cancel discipline — the venue’s fee schedule is an API that trains client behavior.”

“Why do exchanges equalize cable lengths in colo? Isn’t nanosecond fairness theater?” “It’s not theater once you do Part-0 arithmetic: light in fiber does ~1m per 5ns, colo cages differ by tens of meters, so unequalized runs hand some firms a 100–200ns structural edge — and races at the top of book are won by less. More than that, fairness is the product: firms pay colo fees precisely for a credible level playing field, so the venue’s job is to push all systematic advantage out of the infrastructure and let firms compete on their own stack. Software mirror of the same principle: interchangeable gateways, no per-client hot-path branches, ingress timestamps so any skew is measurable and fixable.”

“Why offer FIX at all if your native binary protocol is 10× cheaper to parse?” “Because FIX is the compatibility surface, not the performance surface — every institutional OMS, broker bridge, and compliance stack already speaks it, so a venue without FIX has no institutional on-ramp. You run both, like REST next to gRPC: FIX gateway documented as the slower path, native binary for flow that cares. The subtle operator obligation is honesty about the gap — clients choose protocols based on your published gateway latencies, and my SOR did exactly that arithmetic per venue. The Talos-shaped observation: the FIX-to-many-crypto-APIs translation layer is valuable enough that it’s an entire product category, which tells you how sticky the FIX network effect is.”

“Two sessions of one firm sit on two different gateways sharing one firm-level credit limit. How do you check it without a cross-gateway lock?” “Local reservations against the shared number: each gateway holds a lease — say 20% of remaining firm credit — admits orders against its lease locally at nanosecond cost, and reconciles with the credit service out-of-band, shrinking leases as utilization climbs. It’s distributed rate limiting across an API fleet: local buckets, eventual global truth, bounded over-admission error. The design conversation is about the tolerance band — how much over-limit exposure is acceptable for how many microseconds. And the scheme degrades gracefully at its edge: as the leases shrink near the cap, the possible over-admission stops being small next to what’s genuinely left, so large orders fall back to a synchronous check against the shared cap — you pay the round trip only when the error would matter. What you never do is put a cross-gateway round trip on every order’s hot path to make the error zero.”

“Your gateway fleet — what happens when one gateway host dies?” “Sessions are sticky, so that host’s sessions hard-drop; blast radius is its tenant list, like losing one shard of a multi-tenant fleet. Clients see explicit session death — which is correct in this domain; unlike a web LB I must not transparently fail over mid-session, because session state includes sequence numbers and in-flight order context that another box doesn’t have, and pretending continuity risks silent gaps. So: COD fires for opted-in sessions, clients reconnect to surviving gateways via their connection lists, sequence negotiation + resend recovers the facts. My job is making that path boring: per-session outbound stores replicated off-box so resend works after host loss, and capacity headroom so N−1 gateways absorb the re-logon stampede.”

The HFT Gap Closer