Artificial Intelligence
Prompt Caching on Flash: Achieving Memory-Class Latency for LLM Inference
Large Language Model (LLM) inference increasingly faces latency and cost challenges as model sizes and user demand grow. Prompt caching reduces redundant computation by reusing previously computed Key-Value (KV) data, but the reusable KV working set in online services can rapidly exceed the capacity of GPU memory and host DRAM. As a result, recent LLM serving systems increasingly adopt hierarchical KV cache architectures spanning HBM, DRAM, and SSD/storage tiers. This work focuses on optimizing the SSD-hit retrieval path within such multi-tier prompt cache systems. Under prompt caching with high cache hit rates, preparing KV data becomes dominated by their movement rather than computation, so achieving low latency depends on how effectively interconnect bandwidth is utilized. We enhance this path by shaping I/O behavior to activate SSD internal parallelism and by exploiting external parallelism through multiple NVMe devices. With these optimizations, SSD-backed prompt caching delivers Time-To-First-Token (TTFT) latency comparable to DRAM-backed prompt caching while providing far greater capacity at lower cost. We further analyze SSD endurance under realistic LLM serving workloads and quantify the capacity and Drive Writes Per Day (DWPD) requirements for sustainable deployment. Our results show that scaling from one to four SSDs reduces TTFT by 42% through external parallelism, while internal parallelism optimizations provide an additional 45% reduction—together enabling SSD-backed prompt caching to approach memory-class latency. These findings demonstrate that, with endurance-aware system design, modern SSDs can serve as a practical, low cost, high performance foundation for scalable prompt caching in LLM serving systems.