Change Management

Before you start — this chapter leans on a handful of primer ideas:

Determinism and replay — the contract that makes “re-run production history through the candidate” possible at all: chapter 13

Shadow deploys and mechanical gates — the deploy patterns this process wraps: chapter 16

Config as code and N-1 compatibility — why config gets the full pipeline: chapter 14

Market/venue vocabulary (fills, quotes, resting orders, adverse selection): ch00f

Read those first — 20 minutes there saves an hour here.

The previous four chapters (ch13–ch16) gave you the mechanisms: determinism, versioned schemas, tiered storage, cutover patterns. This chapter is the process wrapper — how changes get from a branch to production money without betting the firm, and what happens when one goes wrong anyway. Interviewers probe this because most trading blowups are change-management failures wearing a technology mask, and the genre’s founding cautionary tale is worth knowing cold:

The Knight Capital story (August 2012). Knight deployed new order-routing code to 7 of its 8 servers — the 8th was missed, and it still carried dead test code called “Power Peg,” wired to a feature flag whose name the new release had reused. When the flag flipped on, the dead code woke up on that one server and started buying — and kept buying for 45 minutes, because nobody could tell which system was doing it. Roughly $440M gone; the firm was effectively dead within a week. Not a clever bug — a partial deploy plus a repurposed flag.

Replay-based regression: the backbone

Your event log makes possible a test most industries can’t have: run actual production history through the candidate build and compare what it would have done against what production did.

The nightly loop. Every night, CI replays a library of recorded prod days through the current release candidate: yesterday (regime freshness), plus a curated set of hard days — the flash-crash day, the venue-outage day, the day the feed gapped, the highest-volume day, the day that triggered last quarter’s incident. Curating that library is real work and worth mentioning: hard days are your most valuable test assets, and every incident post-mortem (below) contributes its day to the library.

Decision-diff gating. The output isn’t pass/fail on state hash — that only works for refactors. For real changes you diff decisions (orders sent, cancels, prices, sizes) and classify: expected diffs (the change’s intent — the new cancel logic should diff on exactly the cancel path; predict the category and rough magnitude before running) versus unexpected diffs (anything else — automatic gate failure, no human override culture).

The gate itself is mechanical: zero unexpected diffs; expected diffs within predicted bounds; plus invariant checks that must hold diff-or-not — position limits never breached in replay, no self-crosses (your own buy order filling your own sell order), risk checks fire where they should.

Replay’s limit: it assumes the market would have behaved identically despite your different orders — fine for regression (“did I change what I didn’t mean to change”), invalid for strategy evaluation. That’s backtesting with impact models (models of how the market would have reacted to your orders), a different discipline — don’t conflate them in an interview.

Determinism tests in CI are the substrate (the determinism audit of ch13): dual-process replay with hash comparison on every PR touching the engine; golden-file decode tests (a golden file is a checked-in expected-output fixture — snapshot tests, in Jest terms) on every PR touching codecs; and the N-1 compatibility replay (old binary reads new log) on every release build. Cheap, fast, and each one guards a contract the deploy patterns depend on.

Canary with real money

After replay and shadow (ch16) comes the only test that includes market impact: trading real money, made survivable by hard-capped limits. Canary scope: the new build runs one strategy, or one venue, or a slice of flow, with independent enforced caps — order size, open-order count, gross/net notional (the total dollar value at stake), max loss. A sane opening posture is ~1% of normal notional. The caps live in the risk layer, not the strategy config — the canary must not be able to mis-config its own cage; enforcement belongs to a component the change didn’t touch. Graduation is stepwise (1% → 5% → 25% → 100%) with mechanical gates between steps — same-hour baselines for fill rates, rejects, latency, P&L-vs-expectation — and any gate failure returns to zero, not to the previous step. Time-box each step: a canary that lingers at 5% for three weeks is a decision nobody made.

Feature flags vs. binary deploys in hot paths

The web-industry default — runtime feature flags everywhere, deploy dark, flip flags — needs modification in a latency engine, and interviewers use this to test whether you cargo-cult practices across domains.

The case against runtime flags in the hot path: every flag costs a branch, a load from flag storage (cache traffic if it’s shared/atomic state), and combinatorial state (2^N flag combinations, of which you tested maybe three; Knight again — the repurposed flag). The branch cost is worse than it looks: the CPU’s branch predictor guesses each branch’s direction in advance and pays a pipeline flush when it guesses wrong, so a flag that almost never flips is exactly the branch it mispredicts the one time it matters. Worst of all, a runtime-flippable flag means the system’s behavior can change without a deploy, without CI, without replay-regression — you’ve built a bypass around your entire verification pipeline.

The pattern that survives: config-at-startup. Behavior toggles are read once at process start into immutable config; changing one = restart = a deploy, going through the same gates (config is code — ch14). If a branch is truly hot-path-critical, resolve it at startup via monomorphization — generics/const-generics compiling the chosen variant to straight-line code; think build-time tree-shaking: the untaken variant doesn’t exist in the shipped binary, so there is no branch left to mispredict. The lighter alternative: dispatch chosen once at init, not per-message. What remains legitimately runtime-mutable is a small, enumerated set of operational controls: kill switches, limit values, throttles — things that must move faster than a deploy in an emergency. That’s not a feature-flag system; that’s the risk-control plane, next section, and the two must not blur: features go through deploys, stopping goes through switches.

Kill switches: taxonomy and drills

The inverted priority of trading infrastructure: you must be able to stop faster and more reliably than you can do anything else. Taxonomy, in expanding blast radius:

Per-strategy: stop quoting/taking for one strategy; optionally cancel its resting orders. First resort; desk-level authority; used weekly in normal life.
Per-venue: halt all activity on one venue (venue misbehaving, feed suspect, session flapping) — cancel opens there, block new sends. Your 20-venue world uses this constantly.
Per-symbol / per-account: finer scopes as the risk model demands.
Global (“the big red button”): stop all order flow firm-wide, cancel everything cancellable. Anyone on the desk can pull it; nobody needs permission; un-pulling requires seniority and a checklist. The asymmetry is the design: cheap to trigger, expensive to reset, because false-positive stops cost basis points (hundredths of a percent) while false-negative non-stops cost the firm.
Flat-position button: one level beyond stop — stop and actively liquidate to flat. This one is dangerous — market orders into a dislocated market realize the loss at the worst price. Concretely: the price just gapped 5% down and the bid side of the book is empty (a liquidity vacuum); “flatten now” sells into nothing and locks in the worst print of the day. So it’s tiered: passive-flatten with a timeout, then aggressive. But it must exist and be rehearsed, because “we couldn’t get flat” is how bad hours become mortal days.

Engineering requirements:

Minimal-dependency path — a kill switch that traverses the whole stack dies with the stack; it should be enforceable at the gateway/risk edge even if the engine is wedged.
Persisted state, survives restart — a rebooting engine must come up stopped if the switch was pulled.
Every pull logged — who, when, why.
Drills — the part that separates real shops: pull each tier on a schedule against production (in a quiet window, with the desk warned), measure time-to-stopped and time-to-flat, and treat a failed drill as a P1. An untested kill switch is a decorative button.

One more reason to build all this properly: MiFID II (the EU’s markets regulation) literally requires the kill capability and evidence that it works (below).

Config as code + staged rollout

Restating the config discipline of ch14 as process, because config changes outnumber code changes and cause a disproportionate share of incidents:

Config lives in git.
Schema-validated and semantically linted in CI (limits positive, venues have credentials, referenced strategies exist).
Deployed as versioned artifacts through the same pipeline stages as binaries — replay-regression where behavior-affecting (a new risk limit changes decisions; replay it).
Canary scope first, then staged rollout, with instant rollback to the previous artifact.
Hand-edits on hosts are findable (drift detection) and treated as incidents even when harmless.

The one-sentence interview version: “we deploy config with exactly the ceremony of code, because the system can’t tell the difference — and neither can the P&L.”

Incident discipline

When it goes wrong anyway:

First minutes — stop the bleeding, in order: appropriate-scope kill switch (smallest that plausibly contains it; global if unsure — the asymmetry rule), assess exposure (positions, open orders, venue state — this is why recon tooling must work during chaos, not just nightly), flatten if risk demands (the tiered flat button), then stabilize and only then debug. Roles matter: one incident commander, one person on comms, hands-on-keyboard separated from decision-making. Say the discipline plainly: no debugging while bleeding — the binary/rollback decision tree (ch16) comes after the position is safe.

Post-incident replay forensics — your structural advantage. Because every input is in the log, the incident is exactly reproducible: replay the day into the incident window, single-step the decisions, test the “what if the fix had been live” counterfactual by replaying through the patched build, and confirm the fix kills the failure without collateral diffs. No “couldn’t reproduce,” no log-archaeology guesswork — the event log converts post-mortems from forensics into re-execution. Then the loop closes: the incident’s day joins the nightly replay library, the post-mortem is blameless-but-specific (mechanism, not villain), and every action item is a gate, test, or drill — not a “be more careful.”

Audit and compliance trail (regulated-venue flavor)

Name-drop depth only — enough to signal you know this world exists and maps onto machinery you already have. EU MiFID II’s RTS 6 (algorithmic-trading systems requirements — SEC 15c3-5, the “market access rule,” is the US cousin) requires, roughly:

Pre-trade controls on every order — price collars (reject any order priced absurdly far from the current market), max order size, max notional, repeated-order throttles — hard-coded in the flow, evidenced.
Kill functionality — the switch taxonomy above, mandated, with proof it works.
Annual self-assessment and stress testing of algo systems.
Testing before deployment, including non-live environments — your replay/shadow/canary pipeline is the evidence pack.
Real-time monitoring with alerting.
Record-keeping — orders, quotes, decisions, time-synchronized to UTC within regulated tolerances (RTS 25 clock-sync flavor), retained for years.

The interview move: your event-sourced architecture makes most of this nearly free — the sequenced log with sequencer timestamps is the record-keeping and the reproduction evidence; the kill drills and canary gates are the self-assessment artifacts. Firms without the log retrofit compliance as a bolt-on; you get it as a projection. That contrast, stated calmly, is a senior answer.

Plain-English recap

Replay regression is record-replay testing with production traffic. Imagine replaying yesterday’s actual webhook stream through a candidate build and diffing every side effect against what production really did. The hard-day library is your incident-fixture collection — every outage contributes its day.
Decision-diff classification is snapshot-test discipline. Expected diffs are the snapshots you meant to update (predicted category and rough count, in advance); any other diff fails CI, no human-override culture. The gate is mechanical, by design.
Canary with hard caps is processing 1% of live payments with a spend limit the new code can’t touch. The caps live in the risk layer — a component the change didn’t modify — because the canary must not be able to misconfigure its own cage. Graduation is stepwise, and any failure goes back to zero.
The feature-flag argument is about your verification pipeline, not flags. A runtime-flippable flag changes production behavior with no deploy, no CI, no replay-regression — a bypass around every gate you built (that’s the Knight Capital shape). Config-at-startup turns every behavior change back into a deploy. The legitimate runtime-mutable set is the circuit-breaker plane: kill switches, limits, throttles.
Kill switches are your PSP “pause payouts” button. Cheap to trip, expensive to reset, scoped by blast radius (strategy → venue → global), enforceable at the edge even when the core is wedged, and drilled — an untested kill switch is a decorative button.
Incident discipline is a SEV process with money on fire. Smallest sufficient kill switch first, exposure check, flatten if needed, IC/comms/hands roles — no debugging while bleeding. Then the structural advantage: the log makes every incident exactly reproducible, so the postmortem is re-execution, not archaeology, and the incident’s day joins the regression library.
The compliance section is audit-trail requirements, payments-style. Like PCI/SOC2 evidence, RTS 6 wants records, controls, and proof of testing — and an event-sourced system emits all of it as a byproduct of the log.

Interviewer will ask

Q1: “How do you test a change to the matching/strategy path before it sees money?” State the principle that generates the ladder: each rung adds a reality the previous one can’t see. CI tests — determinism dual-replay, golden files, N-1 replay — see only the code’s own contracts. Replay-regression adds your own history: recorded prod days plus the hard-day library, gated on classified decision-diffs (zero unexpected, expected within predicted bounds). Shadow adds today’s regime — live inputs your recorded days don’t contain. Canary adds your own market impact — the one thing no offline test can show — under risk-layer-enforced ~1%-notional caps with stepwise mechanical graduation. You climb because each rung answers a question the rung below structurally cannot. Then the caveat that marks seniority: replay is regression, not strategy evaluation — impact isn’t modeled, so P&L claims come from the canary, not the replay.

Q2: “Feature flags in a low-latency system — yes or no?” Runtime flags in the hot path: no — branch and cache cost, combinatorial untested states, and a behavior-change channel that bypasses replay-regression entirely; cite Knight’s repurposed flag as the canonical disaster. Config-at-startup instead, with startup-time monomorphization for hot branches (the tree-shaking move from above: the untaken branch doesn’t exist in the binary) — and every behavior change becomes a deploy through the gates. Carve-out: the runtime-mutable set is the enumerated risk-control plane — kill switches, limits, throttles — which is deliberately not a feature system.

Q3: “Design the kill-switch system.” Three parts: taxonomy, engineering, drills. Taxonomy is scoped by blast radius — strategy, venue, symbol, global — plus the tiered flat-position button (passive first, then aggressive, because market-ordering into a dislocated book locks in the worst print), with the authority asymmetry: cheap to pull, expensive to reset. Engineering: enforcement at the minimal-dependency edge so it works when the engine is wedged; persisted state so a restart comes up stopped; every pull logged. Drills: pull each tier on a schedule against production, measure time-to-stopped and time-to-flat, and treat a failed drill as a P1. Close with: regulators mandate this anyway (RTS 6 kill functionality), so build it once, properly.

Q4: “A bad change made it to prod and is losing money. Walk me through your first ten minutes.” Minute 0: kill switch, sized to what I actually know — if I can name the strategy, its switch; if all I know is “we’re bleeding,” global, because the expensive mistake is scoping the kill by optimism. New flow stops, resting orders cancel, and the loss rate is now bounded by open positions instead of by a runaway algo. Minutes 1–3: exposure assessment with the recon tooling — what positions do we actually hold versus what the bad change believed — which is where drop-copy recon earns its keep, because the sick system’s own view is a suspect witness. Minutes 3–5: flatten if risk demands, per the tiered policy — passive first, aggressive only if the market is moving against the position, because market-ordering into a dislocated book locks in the worst print. In parallel the roles split: one incident commander, one on comms to desk and compliance, one pair of hands on the system — the same person doing all three is how a bad ten minutes becomes a bad hour. Minutes 5–10: only now the rollback-vs-forward-fix tree, read from the runbook written before the deploy — almost always rollback, and the criteria were pre-committed precisely so nobody reasons under adrenaline. What never happens inside these ten minutes: debugging. No debugging while bleeding. Afterward: exact replay reproduction of the failure, counterfactual validation of the fix, and the incident day goes into the regression library so this exact mistake can never ship twice.

Q5: “What does ‘config as code’ mean concretely in your world?” Rationale first: config changes outnumber code changes, get fewer reviewing eyes than binaries do, and the engine can’t tell the difference — so config earns the full ceremony. Concretely: git plus schema validation and semantic lint in CI; versioned artifacts through the same replay/canary/staged pipeline as binaries when behavior-affecting; rollback to the previous artifact; host-drift detection, with hand-edits treated as incidents; and hot-reloadable config that touches the deterministic path enters as log events (ch13).

Q6: “How would you catch a change that’s subtly wrong — not crashing, just worse?” Layered nets for the quiet failures: decision-diff replay catches “different where it shouldn’t be”; shadow catches regime-dependent divergence replay can’t; canary gates on same-hour baselines (fill rate, reject rate, adverse selection, latency percentiles) catch “statistically worse”; and invariant monitors (self-cross, limit proximity, quote-to-trade ratios) catch “categorically wrong.” What none of them catch: a change that’s wrong only via market impact at full size gets past all of it until graduation steps expose it — which is why graduation is stepwise with returns-to-zero.

Q7: “What do regulators actually require of algo trading systems?” RTS 6 flavor (15c3-5 in the US): evidenced pre-trade controls (collars, size, notional, throttles), mandated and tested kill functionality, deployment testing and annual self-assessment, real-time monitoring, and UTC-synchronized order/decision record-keeping with multi-year retention. Then the architecture point: an event-sourced engine produces the records and reproduction evidence as a byproduct — compliance as projection of the log, not a bolt-on — and the drills/canary artifacts double as the self-assessment pack.

The HFT Gap Closer