The contract nobody wrote down
Four bugs in one month, in four different subsystems, that turned out to be one defect.
In July, SGLang's Apple Silicon backend produced four bugs that had nothing in common. A TypeError at startup. Prefix caching that silently stored a fraction of what it should. A segmentation fault behind an unusual flag pair. A predicate that returned the right answer for the wrong reason.
Different symptoms, subsystems, and people finding them, weeks apart. Each was fixed as an unrelated defect because each looked like one, when, in fact, they were all one defect.
The optimization
A decode loop asks the scheduler what to run next, prepares the batch, launches the forward, waits, samples a token and repeats.
On a datacenter GPU, the forward dominates and the scheduler round trip is noise. On Apple Silicon, the launch overhead is a large fraction of a decode step, which makes the round trip itself the bottleneck. So the MLX backend added chained decode: having launched step N, the loop builds step N+1 from the pending result and launches it without having to return to the scheduler. Steps link to each other rather than back through a coordinator. The measured gain was roughly 150 tokens per second, which on a laptop is the difference between usable and not.
What rode along
Scheduler.run_batch does two unrelated things. It runs the torch forward and it performs the per step bookkeeping that the rest of the framework depends on: a timestamp for the watchdog, a slot reservation recorded in the position to slot map, three length counters the memory pressure logic reads and a relay buffer holding the previous step's sampled tokens.
None of it is expensive. The expensive part was the round trip, and the cheap part happened to live inside it.
Chained decode skipped the round trip and everything the round trip carried and nothing anywhere said what it was.
Four idioms
Loud. The step timing field was never stamped, so the first real request reached a subtraction with None and the scheduler died before serving anything.
Silent. Chained steps never reserved a KV slot, so the position to slot table was never written for those tokens. The code that copies MLX KV into the shared pool reads the table to find its destination and unwritten cells read as zero, which is the reserved padding slot. Measured on hardware: a 220 token generation committed 94 and a follow up request sharing the text hit the prefix cache for 94 of 321 tokens. Every multi turn conversation reprefills the entire assistant turn.
Conditional. A batch mixing prefill and decode gathers the decode side's tokens from a relay buffer only the torch path publishes. The buffer is allocated without initialization, so the gather pulled uninitialized memory into a tensor concatenation and Metal segfaulted. It required two non default flags together, which is why it took time for anyone to hit it.
Latent. A predicate asking whether work is in flight tested a flag, meaning the torch overlap loop is active, which is, by definition, false when the MLX loop is running. It still returned correct answers because other conditions in the same expression happened to catch the busy case. Nothing enforced the coupling. The consumers were memory release and cache flush.
Nothing connects a timestamp, a slot table, a token relay, and a boolean, except that one function call used to maintain all four.
Why they looked unrelated
Each failure was legible on its own terms and received a fix from whoever found it. Nothing in the codebase said these were instances. There was no list of what a decode step must leave behind, so no way to check a replacement loop against it and no way to know how many more were waiting.
The second reason is more interesting. Every test in the area constructed its object by hand: allocate a scheduler or runner without running its initializer, then set the attributes the function under test happens to read. Three suites by three authors all did this and it makes sense because the real object is expensive to build.
However, a test that supplies the state it asserts on cannot detect missing state. The failures were all missing state. The tests were not inadequate; they were structurally incapable of catching this class, and they reported green while being so. The one place this became visible was when I added a field to a conditional and an unrelated test broke because its fixture had set only the flags the old version read.
Auditing does not fix this either, as the audits are written against the same hand built object.
Enumerating instead of waiting
Walk one decode step through the standard path. Record every write to state that outlives the step, meaning anything set on a request, a batch, a pool, or the scheduler itself. For each, find who reads it. Then walk the same step through the replacement loop and check.
Eighteen fields. Most were either already maintained on both paths or torch specific, in which case a non torch backend owes nothing. What remained was five clauses:
- Every emitted position has a recorded slot.
- Length counters advance by the tokens actually emitted.
- Per step identity and timing are stamped.
- Cross iteration relays are published or provably unread on the path.
- Predicates asking whether work is in flight test any loop, not one specific loop.
Four had been discovered by crashing into them. The fifth had not failed yet, and would not have failed for a while because what masks it is how the current loop happens to be written rather than anything enforced.
Clause 4 is worth reading twice. Contracts specifying mechanism rather than obligation make backends implement unnecessary things, so the clause does not say publish the relay. MLX has no use for it. It says publish it or show nobody reads it on your paths, which is a weaker obligation MLX fails only because one mixed batch path does read it.
What this generalizes to
An implicit contract survives exactly as long as nobody writes a second implementation. The moment another exists, whatever the first path maintained incidentally becomes an unenforced assumption, and every consumer of that state becomes a landmine with a different fuse.
The failure mode is recognizable: an optimization removes a coordination step, but the step was also doing bookkeeping nobody listed. The optimization is usually correct on its own terms; the bookkeeping is usually cheap. The bundling is the defect.
Write the contract down as an enumerated list, each clause naming the reader that depends on it because a clause with no reader is not an invariant and a reader you cannot name means you do not understand the clause.
Enforce it with a conformance test, driving both paths through the same workload, not with review because a review of a replacement loop requires holding the entire implicit contract in your head and nobody does that reliably twice.
Be suspicious of any test that builds its subject by hand. It will pass. It may be structurally unable to fail.
Where it stands
The contract is now an RFC against the project, with each clause citing the code that writes it and the code that reads it. Three of the five clauses have fixes in flight from three contributors; the fifth has a small fix of mine. It is the only clause found by enumeration rather than by a user hitting it.
I do not know whether the enumeration is complete. It covers one decode step; prefill, request admission, and request release are unwalked; the auxiliary state that hybrid models carry is a separate lifecycle with its own recent bug, but what I do know is the number is no longer unbounded and the next person writing a replacement loop has something to check themselves against.