Schema & Protocol Evolution

Before you start — this chapter leans on a handful of primer ideas:

The event log and the replay contract — why old bytes must stay decodable forever: chapter 13

FIX and venue protocols — the session/message world your schemas talk to: ch00f

Feed handlers and normalization — the per-venue adapter layer this chapter uses as its isolation boundary: ch00f

Struct layout and byte offsets — why “fixed-offset” formats are fast and why inserting a field mid-struct corrupts everything: ch00a

Read those first — 20 minutes there saves an hour here.

Your event log lives for years. Your binaries live for weeks. Your venues change their protocols whenever they feel like it. The gap between those lifetimes is schema evolution, and it’s the single most-probed “can this person run a system, not just build one” topic in infrastructure interviews. The discipline is small and rigid: never mutate a published schema; only add; version everything; translate at the boundary.

The prime directive: never mutate v1

Once an event with schema v1 has been written to a log that outlives the current binary, v1 is frozen forever. Not deprecated-then-changed. Frozen. A byte layout, field meaning, and unit convention that some reader will need to understand in three years.

What “mutate” covers, because people rationalize all of these:

Changing a field’s type (u32 qty → u64 qty) in place.
Changing a field’s meaning or units (price in cents → price in ticks) without changing anything structural — the worst kind, because nothing crashes, numbers are just silently wrong.
Renaming with reuse — deleting foo and adding a different foo.
Reusing a numeric field id/tag of a deleted field. Protobuf identifies fields on the wire by number, not name (mechanics in the wire-formats section below), so reusing a dead number makes years-old bytes silently decode as the new field. This is protobuf’s cardinal sin; reserved exists for it.
Changing enum variant discriminants (the discriminant is the integer the variant is stored as — 0=buy, 1=sell) or reordering variants in a format where the discriminant is the wire value.

What’s allowed, format permitting: adding new optional fields, adding new event types, adding enum variants (if readers tolerate unknowns), widening semantics in a way old readers can safely ignore.

Everything else is a new version: OrderPlacedV2 alongside OrderPlacedV1, both decodable forever, with an upcaster bridging them.

Upcasters: translate at read time, once, at the boundary

An upcaster is a pure function v_old -> v_new applied when reading old events, so that everything past the deserialization boundary sees only the newest version. The state machine (the event-sourcing fold — ch13) handles exactly one version: current. All version sprawl is quarantined in the codec layer.

Rules:

Upcasters are pure and total. For every valid v1 event, a defined v2 result. New fields get explicit, documented defaults — and the default must reproduce old behavior (“source: Unknown behaves exactly as v1 did”), otherwise you’ve broken the replay contract (ch13).
Chain them. v1→v2→v3, not v1→v3 direct. N versions cost N-1 upcasters, not N². Each is written once, when the new version ships, while the semantics are fresh.
Upcast on read, never rewrite the log. The log is immutable and often a compliance artifact; rewriting it destroys the audit trail and risks corruption. (A deliberate offline “re-encode the archive to vCurrent” migration is a separate, rare operation — and you keep the original.)
Test with golden files. Committed binary fixtures of real v1/v2 bytes; CI decodes them with today’s code and asserts equality with expected structs. This catches “someone touched the old decoder” — the mutation you swore wouldn’t happen.

Rust: enum-versioned events + upcasting

#![allow(unused)]
fn main() {
// codec layer — the ONLY place old versions exist
#[derive(Clone, Debug, PartialEq)]
pub struct OrderPlacedV1 { pub id: u64, pub side: Side, pub price: i64, pub qty: u64 }

#[derive(Clone, Debug, PartialEq)]
pub struct OrderPlacedV2 {
    pub id: u64, pub side: Side, pub price: i64, pub qty: u64,
    pub source: Source,          // NEW in v2
}

pub enum WireEvent {             // what the log actually contains
    OrderPlacedV1(OrderPlacedV1),
    OrderPlacedV2(OrderPlacedV2),
    // every version ever written, forever
}

impl From<OrderPlacedV1> for OrderPlacedV2 {
    fn from(v1: OrderPlacedV1) -> Self {
        OrderPlacedV2 {
            id: v1.id, side: v1.side, price: v1.price, qty: v1.qty,
            source: Source::Unknown,   // default MUST reproduce v1 behavior
        }
    }
}

/// Boundary function: the engine only ever sees `Event` (== current versions).
pub fn upcast(w: WireEvent) -> Event {
    match w {
        WireEvent::OrderPlacedV1(v1) => Event::OrderPlaced(v1.into()),
        WireEvent::OrderPlacedV2(v2) => Event::OrderPlaced(v2),
    }
}
}

On disk, each record carries a header the decoder dispatches on:

#![allow(unused)]
fn main() {
#[repr(C)]
pub struct RecordHeader {
    pub len: u32,        // payload length
    pub event_type: u16, // OrderPlaced = 1, OrderCancelled = 2, ...
    pub version: u16,    // schema version of THIS record
    pub seq: u64,
    pub ts_ns: u64,      // sequencer-assigned time (event-sourcing chapter!)
    pub crc32c: u32,
}
}

(event_type, version) pairs are append-only registry entries. Keep the registry in one file with a comment per entry: date shipped, what changed, default rules. That file is your schema history — interviewers respond well to “we kept a single append-only registry, code-reviewed like an API.”

Wire formats and their evolution rules

You’ll be asked to compare these. Every format picks a point on the triangle of decode cost, evolution flexibility, and self-description. Know each format’s specific evolution mechanics — the exact add/remove/reorder rules — not a general impression of which is faster.

SBE (Simple Binary Encoding)

The binary standard from the FIX Trading Community (the standards body behind the FIX protocol — ch00f); the tradfi latency-tier default. Fixed-offset fields — decode is pointer-cast plus field reads, zero allocation, effectively memory-bandwidth speed.

Evolution model: extension, not flexibility. Messages declare a schema version; new fields are appended after existing fixed fields — never inserted, because every field sits at a fixed byte offset and inserting shifts all the later ones:

 v1 layout:           [id @0][price @8][qty @16]
 v2 by APPEND:        [id @0][price @8][qty @16][source @24]   old offsets still valid
 v2 by INSERT:        [id @0][source @8][price @16][qty @24]   every later offset shifted
                              ^^^^^^^^^^  old readers read source as price, price as qty — garbage

Old readers, told the message is a newer version, simply don’t read past what they know (“extension” semantics); new readers of old messages must apply defaults for missing trailing fields. Variable-length data goes at the end.

What you can’t do: remove or retype fields in place, insert mid-message. The pragmatic pattern is pre-allocated reserved fields — pad the layout at design time, claim padding later without changing size or offsets:

#![allow(unused)]
fn main() {
#[repr(C, packed)]
pub struct OrderPlacedWire {
    pub id: u64,
    pub price: i64,        // ticks (units documented AT the field)
    pub qty: u64,          // lots
    pub side: u8,          // 0=buy 1=sell
    pub source: u8,        // v2: was _reserved[0]; 0 = Unknown = v1 behavior
    pub _reserved: [u8; 6],// claim bytes here in future versions; zeroed on write
}
// Size and every offset are frozen. Zero means "not set / old default" by
// convention, so a v1 writer's zeroed padding IS a valid v2 message.
}

That “zero = legacy default” convention is what makes reserved-field evolution safe: v1 writers produce valid v2 messages for free.

Protobuf

The general-purpose default. Tag-length-value: every field carries its numeric id; decoders skip ids they don’t know (unknown-field pass-through). Proto3 also retains unknown fields on re-serialize, which matters for proxies: a middlebox that decodes a message and forwards it re-serialized no longer silently strips the fields it didn’t understand.

Evolution rules: never reuse or renumber a field id (reserved 5; after deletion); adding fields is free; several type changes are wire-compatible (int32↔int64 with truncation caveats), most aren’t; required was removed from the language because it made evolution brittle — everything is optional, and application-level defaults do the work.

The cost is decode speed. Integers are varints — a variable-length encoding where small numbers use fewer bytes — so parsing is field-by-field with allocations, not a pointer cast. Fine for control plane, config, and cold path; too slow and too allocation-happy for a market-data hot path.

FlatBuffers / Cap’n Proto

Zero-copy access like SBE, with a vtable/pointer layer that buys protobuf-like evolution — a vtable here is a small per-message lookup table mapping field → byte offset, consulted on every read. Fields located via these offset tables can be added and unknown ones skipped without fixed-position rigidity. The tax: indirection on every access (vtable lookup vs. SBE’s compile-time offset), bigger messages, and alignment discipline. A defensible middle choice for internal buses; in practice tradfi picked SBE (with FIX heritage) and most crypto shops picked JSON-because-the-venue-did plus an internal binary format.

JSON

Your venues’ reality: crypto exchange WebSocket feeds are JSON. Evolution is trivially flexible (add keys; readers ignore unknowns) and totally undisciplined (nothing stops a venue renaming a key, changing a number to a string — Binance famously sends quantities as strings — or changing units; you find out in production). Parse cost is brutal: hundreds of ns to µs per message, allocation-heavy. The professional stance: JSON is a boundary format you normalize out of immediately (see feed handlers below); simd-json-style parsers and arena allocation if the boundary itself is hot.

Comparison one-liner for interviews: “SBE when both ends are mine and latency is the product; protobuf when evolution across many teams matters more than nanoseconds; FlatBuffers when I want both and accept indirection; JSON when the counterparty chose it for me — and then I normalize it away at the edge.”

Rolling upgrades: the N/N+1 compatibility guarantee

You have many services on the internal bus (your internal message stream between components) — feed handlers, engine, risk, gateways, drop-copy — and you deploy them one at a time (ch16). During any deploy window, versions N and N+1 coexist on the same streams. The rule that makes this safe:

Every message schema change must be compatible in both directions across one version step. N+1 readers accept N’s messages (upcast/defaults); N readers accept N+1’s messages (unknown-field skip / extension semantics). You never need N and N+2 live simultaneously because deploys are serialized — but note that the log makes the guarantee stronger than it looks: the current reader must handle every version ever written to retained logs, not just N. Live traffic needs N/N+1; replay needs the current reader to decode arbitrarily old versions — N back to the start of the archive — via the upcaster chain.

The rollout choreography for a breaking-ish change (new required-by-logic field):

Ship readers first. Deploy code that understands v2 everywhere, still writing v1.
Flip writers. Once all readers speak v2, writers start emitting v2 (config/startup flag, not code deploy, ideally).
Deprecation window. v1 write-capability is removed after readers have been v2-aware for a defined period (and logs containing v1 remain decodable forever regardless).

Reader-before-writer ordering is what keeps this safe; almost every “mysterious deserialization error during deploy” post-mortem is that order violated. For deprecation windows, be concrete: internal services, one or two release cycles; anything persisted, effectively never (decoder lives as long as the archive).

Venue protocol upgrades: the normalize layer as isolation boundary

This is your daily reality with 20+ venue integrations, so own it in interviews. A venue announces: “on March 15 we migrate to WS API v5; new auth flow; qty field renamed; new message for liquidations; old API sunset in 90 days.” You do not get a vote.

The architecture that makes this survivable: per-venue feed handlers that normalize to your internal schema at the edge. The venue’s protocol exists only inside its handler. Everything downstream — book builders, strategies, the engine — consumes your internal events (this chapter’s and ch13’s discipline applies to those, and you control their evolution). A venue migration is then a change to exactly one process, deployable venue-by-venue (rolling-by-venue — ch16), testable in isolation.

Operational playbook for a venue migration:

Capture first. Record raw v5 traffic (they usually run new API in parallel before sunset). Raw capture — bytes with timestamps, before parsing — is your regression fixture and your dispute evidence.
Build the v5 handler as a new module/binary, not edits to v4 in place — you’ll run both during transition.
Shadow it: v5 handler consumes live, normalized output diffed against the v4 handler’s output in real time. Diffs reveal the semantic changes the changelog didn’t mention (different snapshot depth, different trade-side convention, timestamps now in µs not ms — units again).
Cut over per venue with instant fallback to v4 while it still exists.
Keep the v4 decoder as long as you retain v4-era raw captures.

The normalized schema itself needs the same evolution discipline: when a venue exposes something new you actually want (a liquidation flag), that’s an additive internal-schema change rolled out readers-first — venue chaos on the outside, boring N/N+1 on the inside. That sentence is the interview answer.

Config schema evolution: same discipline, smaller egos

Config is code-adjacent input to a deterministic system, so it gets the same treatment, and interviewers increasingly probe it because config changes cause more incidents than code changes.

Version the config schema. A schema_version field in the file; the loader upcasts old configs exactly like old events (defaults must preserve behavior).
Config as code: files in git, reviewed, CI-validated (parse + semantic lint: “limit must be > 0”, “every enabled venue has credentials”), deployed as artifacts with a rollback path — never hand-edited on hosts.
Renames are additive: accept both keys for a window, warn on old, remove later. A binary that crashes on an old config file during rollback has broken N-1 compatibility exactly as badly as a message schema would (the rollback discipline of ch16 depends on this).
If config affects the deterministic fold, config changes are events in the log (ch13). Startup-loaded config that never changes mid-session is exempt; anything hot-reloaded is not.

Plain-English recap

“Never mutate v1” is Stripe API versioning. A published API version is frozen forever; changes ship as new versions, and old clients keep working indefinitely. Your event log makes every past writer an “old client” you can never break — same contract, enforced by your own archives.
Upcasters are Stripe’s version-transform layers. Old-shaped requests pass through a chain of pure transforms (v1→v2→v3) so core code only ever sees the latest shape. Version sprawl is quarantined at the boundary — exactly like keeping all the legacy-request shims in the API gateway, never in the domain logic.
Golden files are snapshot tests for bytes. Committed fixtures of real old payloads, decoded in CI and compared to expected structs — the same reflex as Jest snapshots, protecting decoders someone will otherwise “clean up.”
Readers-first is the expand–contract deploy you already do with Postgres. Add the nullable column and deploy code that tolerates it before anything writes it; drop the old column only after nothing reads it. Flipping writers before readers is the same outage in both worlds.
N/N+1 on the bus is rolling deploys with queued webhooks. During a deploy, old and new pods coexist and messages written by either must be readable by both. And the retained log stretches the rule: today’s reader must decode every version ever archived, like a webhook consumer that might receive a replay of events from 2019.
A venue migration is a PSP API migration. When Stripe or Adyen announce a breaking change, only your per-PSP adapter changes; the rest of the system sees your internal, stable schema. Shadow-diffing the new handler against the old is running both integrations in parallel and reconciling their outputs before cutover — the anti-corruption layer (the adapter layer that keeps an external system’s weirdness out of your core) earning its keep.

Interviewer will ask

Q1: “How do you evolve an event schema when the log is retained for years?” Open from the Stripe picture: a published API version is frozen forever, and the log makes every past writer an old client you can never break. The log outlives every binary, so bytes written today must still decode in three years. Therefore v1 freezes the moment it’s written, and every change is a new type — OrderPlacedV2 beside OrderPlacedV1, both decodable forever. Upcasters quarantine the sprawl at the read boundary: pure v1→v2→v3 translations with behavior-preserving defaults, so the engine only ever sees the current version. Golden files keep the freeze honest: committed real v1 bytes decoded in CI, so “someone cleaned up the old decoder” fails a test instead of corrupting replay. Land: never mutate, only add, translate at the boundary — and the log is never rewritten. The registry detail: (event_type, version) pairs live in one append-only, code-reviewed file.

Q2: “Protobuf vs. SBE — when and why?” One fork decides everything: how does a reader find a field — compile-time offset, or per-field tag on the wire? SBE picks offsets: decode is a pointer cast plus field reads, memory-bandwidth fast. But frozen offsets mean evolution is append-only — new fields at the end, or claim pre-reserved padding — because inserting mid-message shifts every later offset into garbage. Protobuf picks tags: every field carries its id, so readers skip unknown fields and cross-team evolution is easy — but you pay varint decode and field-by-field parsing, too slow and allocation-happy for the hot path. Land: SBE when both ends are mine and latency is the product; protobuf for control plane and cold path. Volunteer each choice’s signature failure: reusing a dead protobuf field id, so years-old bytes silently decode as the new field (reserved exists for this); and inserting a field mid-SBE-message.

Q3: “You’re deploying a change that adds a field consumers need. Walk me through the rollout.” Start from the asymmetry that dictates the order: a new reader given old bytes is safe — the missing field takes a default; an old reader given new bytes is not — it can’t use a field it never learned. So readers ship first: deploy v2-aware readers everywhere while everything still writes v1. Only when reader coverage is total do writers flip, via config rather than another deploy. Then a deprecation window before v1 write capability is removed — while the v1 read path lives as long as any log contains v1. Name the guarantee — N/N+1 both directions on live traffic, N back to forever on replay — and the classic failure: flipping writers before readers, the root of almost every “mysterious deserialization error during deploy” post-mortem.

Q4: “A venue announces a breaking API change with 90 days notice. What do you do?” Same shape as a PSP API migration: when Stripe announces a breaking change, only your Stripe adapter changes — here, the venue’s protocol lives only in that venue’s feed handler; build the new handler alongside the old, capture raw traffic, shadow-diff normalized output against the current handler to catch undocumented semantic changes (units, side conventions, snapshot behavior), cut over per-venue with fallback. Downstream sees zero change unless we choose an additive internal-schema update. This is the question where your 20-venue experience should carry the answer — have one real migration story ready.

Q5: “What breaks if you just add a field to a #[repr(C)] struct you write to the log?” Every prior record now decodes at wrong offsets — silent corruption, not a crash, because the bytes still “parse.” Fixed-layout formats require either explicit versioning in the record header with per-version decoders, or pre-allocated reserved space with zero-as-default so old records remain valid new-version records. This is the trap question checking you’ve actually done binary logging.

Q6: “How do old readers handle fields they don’t understand — compare formats.” This question reduces to one requirement: skipping a field you don’t understand means knowing where it ends. Protobuf builds that in — tag-length-value framing tells the reader how many bytes to skip — and proto3 even retains skipped fields on re-serialize, so proxies stop stripping them. SBE has no per-field framing, so old readers can’t skip; instead the extension model — read only your known prefix, safe because new fields only ever append — which needs the version in the header. FlatBuffers: the vtable lookup misses, and the reader gets a default. JSON: keys are self-describing, so unknown keys are simply ignored. Land the design lesson: unknown-field tolerance is what makes reader/writer skew survivable, so a hand-rolled format must design it in — a version field plus length-prefixed records, letting readers skip whole records they can’t parse.

Q7: “Does config get the same treatment as code?” Yes, and say why with force: config changes cause more trading incidents than code changes because they skip the pipeline. Config as code in git, schema-versioned, CI-validated, deployed and rolled back like binaries, N-1 compatible (old binary must load new-ish config during rollback), and hot-reloaded config that affects the deterministic path enters as log events.

Q8: “How would you version snapshots?” Trap check — people version events and forget snapshots. A snapshot is one giant event, so in principle it needs the same versioned header and upcasters. The pragmatic shortcut is two steps: the upgraded binary reads the previous snapshot once — one version step, so N-1 compatibility suffices — then immediately writes a fresh snapshot in the new format. After that, the only cross-version read that can ever happen is a rollback inside the bake window. And the escape hatch if that read fails: event-level replay from an older snapshot — the log is the safety net under the shortcut.

The HFT Gap Closer