Quantization Without Regret

A quantized model that loses one point of average benchmark score has usually not lost one point uniformly. Long-context retrieval, multi-step arithmetic, and low-resource languages degrade first and hardest.

Our acceptance process evaluates quantization per capability, with separate thresholds for each. A recipe that is fine for summarization can be unacceptable for structured extraction on the same deployment.

Mixed-precision schemes — keeping attention projections and a handful of sensitive layers at higher precision — recover most of the loss at a small memory cost, and are almost always worth the added complexity.

The rule we operate by: never ship a quantization without a per-capability regression report, and never accept an aggregate score as evidence of safety.