The Serving Layer Is Where the 10× Hides

Hardware is the visible cost and the serving stack is the invisible one. Batching policy, scheduling, cache management, speculative decoding, and disaggregated prefill each contribute a multiple, and they compose.

Continuous batching with a well-tuned admission policy is the single largest lever for throughput-bound workloads. Prefill/decode disaggregation is the largest lever once latency SLOs get tight, because it stops long prefills from stalling every decode in the batch.

Speculative decoding pays off exactly where the draft model's acceptance rate is high — structured output, code, templated responses — and is close to neutral elsewhere. Measure acceptance per workload, not per model.

None of this requires new silicon. It requires treating inference as a scheduling problem with a cost objective, which is the practical center of our optimization work.