The KV Cache Is a P&L Line

Agentic workloads re-send enormous shared prefixes: system prompts, tool schemas, retrieved documents, conversation history. Without prefix caching, every step pays full prefill cost for context that has not changed.

Cache-aware routing changes the objective. The cheapest replica is not the least loaded one; it is the one that already holds your prefix. Once scheduling accounts for cache locality, effective throughput on agent traffic improves severalfold on identical hardware.

The second-order effects matter too. Longer cache residency means prompt design becomes a cost decision: stable prefixes are cheap, and rotating anything early in the prompt invalidates everything after it.

We now treat cache hit rate as a first-class production metric, tracked next to latency and error rate rather than buried in infrastructure dashboards.