Efficient Long-Context LLM Inference With Sparse Attention Over HBM-HBF Tiered Memory
The rapid growth in context length for LLM inference is increasingly making KV cache capacity a major bottleneck for GPU memory systems. HBF has recently emerged as a promising solution to this problem by providing much larger capacity than HBM, motivating HBF-based KV cache offloading. However, simply using HBF for KV cache offloading yields only limited throughput improvement because HBF bandwidth does not scale with its capacity; its capacity advantage is offset by the substantial attention-latency overhead of accessing an expanded KV-cache footprint over limited HBF bandwidth. We propose SparseTier, a novel HBF-based KV cache offloading scheme. To mitigate the latency overhead of accessing large KV caches from HBF, SparseTier combines HBF with emerging sparse attention and applies data-aware placement, keeping latency-critical data in HBM while storing capacity-intensive KV caches in HBF. Our evaluation shows that SparseTier improves long-context LLM inference throughput by up to 6.73x over HBM-only systems.