PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

PEEK is a joint scheduling and eviction framework that exploits prefix-sharing structure in the pending request queue—a signal no existing engine surfaces. An incremental radix tree over waiting requests plus a low-overhead dual-walk against the engine’s prefix cache drives cluster-aware admission (cLPM), queue-aware eviction that protects blocks ancestral to queued demand, and a multi-lane stride scheduler that bounds starvation. On SGLang and vLLM across five workloads up to 4xH100, PEEK delivers up to 3.0x/2.6x cache hit, 7.9x/7.1x TTFT, 6.7x/5.5x E2E, and 3.6x/4.5x throughput over each engine’s strongest baseline.