Blog

Stay updated with our latest news and announcements.

Insights

[Summary] Efficient Long-Context LLM Inference With Sparse Attention Over HBM-HBF Tiered Memory

Jaeyong LeeIEEE Computer Architecture Letters, 2026

Efficient Long-Context LLM Inference with Sparse Attention over HBM-HBF Tiered Memory

The growing adoption of long-context Large Language Models (LLMs) is placing increasing pressure on GPU memory capacity. As context lengths extend to hundreds of thousands or even millions of tokens, the Key-Value (KV) cache becomes a major memory bottleneck, limiting the number of requests that can be processed concurrently.

High-Bandwidth Flash (HBF) has recently emerged as a promising solution, offering substantially larger memory capacity than High-Bandwidth Memory (HBM). However, simply offloading KV caches to HBF provides limited throughput improvement. Although HBF enables larger batch sizes, its relatively limited bandwidth leads to significant attention latency, offsetting the benefits of increased memory capacity.

This work proposes SparseTier, an HBM-HBF tiered memory scheme that combines sparse attention with data-aware memory placement to enable high-throughput long-context LLM inference.

Key Concepts and System Design

SparseTier addresses the limitations of existing HBF-based KV cache offloading through two key optimizations.

1. Combining Sparse Attention with HBF

Unlike conventional dense attention, which accesses the KV caches of all previous tokens, sparse attention retrieves only a small subset of relevant tokens (Top-K) during each decoding step. This significantly reduces KV cache access traffic while preserving the full KV cache in memory. Sparse attention is therefore particularly well suited to HBF, which offers large memory capacity but relatively limited bandwidth.

2. Data-Aware Placement Across HBM and HBF

Sparse attention introduces two distinct data structures with different memory requirements: index vectors and KV caches.

  • Index vectors → HBM: Index vectors must be densely accessed to identify the Top-K relevant tokens, making their access latency critical to inference performance.
  • KV caches → HBF: KV caches require substantially larger storage capacity but are accessed sparsely, making them suitable for high-capacity HBF.

By placing each data structure in the memory tier that best matches its characteristics, SparseTier exploits HBM's high bandwidth for latency-critical indexing and HBF's large capacity for KV cache storage.


Figure 1: Overview of SparseTier's HBM-HBF tiered memory organization.

 

Performance Evaluation

SparseTier is evaluated using LLMServingSim with MiniMax-M3, a frontier-class LLM employing sparse attention. The evaluation considers two long-context workloads with input/output sequence lengths of 128K/16K and 512K/32K tokens. The results demonstrate substantial improvements over conventional HBM-only GPU systems.

  • Larger Batch Sizes: SparseTier supports up to 16× larger batch sizes by expanding the memory capacity available for KV caches.
  • Throughput Improvement: SparseTier achieves 6.25× and 6.73× higher inference throughput than HBM-only systems for the 128K/16K and 512K/32K workloads, respectively.

 

Figure 2: Inference throughput comparison between SparseTier and HBM-only GPU systems.

 

Conclusion

SparseTier demonstrates the potential of combining sparse attention with HBM-HBF tiered memory for efficient long-context LLM inference. By reducing KV cache traffic through sparse attention and strategically placing data across HBM and HBF, SparseTier effectively translates HBF's large memory capacity into higher inference throughput. The results highlight how memory hierarchy design can complement emerging LLM attention mechanisms, providing a promising direction for next-generation memory systems supporting long-context AI workloads.


DOI : https://doi.org/10.1109/LCA.2026.3737915 



#High-Bandwidth-Flash#Long-Context-LLM-Inference#Data-Aware-Placement

Popular Insights

Previous Next List