Blog

Stay updated with our latest news and announcements.

Insights

[Summary] CANNON: A CXL-Based Near-Memory Processing Architecture for Approximate Nearest Neighbor Search on Real Hardware

Kangkyu ParkDAC 2026

I services like RAG need to find the vectors most similar to a query — this is approximate nearest neighbor (ANN) search. Vector collections now reach billions of entries, and graph-based ANN search is the popular choice because it gives good accuracy at low latency. The catch is that graph search jumps around memory randomly, which gets slow and hard to maintain at large scale.

This work presents CANNON, a system that runs ANN search inside a CXL memory expander using near-memory processing (NMP) logic. The device does all the work — walking the graph, computing distances, and picking the top-k results — while the host only sends the query and reads back the answer. CANNON was built and tested on real CXL hardware, turning CXL memory from a passive storage pool into a computing tier.

Problem and Design

Two problems motivate CANNON. First, big datasets do not fit in one machine: with a 320 GB working set, a GPU with 80 GB memory runs out of memory, and a CPU with 256 GB DRAM has to fall back to swap or SSD. Second, just adding capacity does not help enough: SSD reads pull in whole pages for small data (read amplification), and plain CXL memory expansion still needs a slow host–device round trip for every graph hop. In Figure 1, the expander connects to the host over CXL.io and CXL.mem; inside, NMP logic on an AXI interconnect drives multiple memory controllers and DRAM channels, and the ANN circuitry sits alongside the CXL controller (verified in the FPGA placement view):


Figure 1: CANNON design — host, CXL endpoint, NMP logic, DRAM channels, and ANN logic on the FPGA.

  • 64-value striping: each 64-element FP32 sub-vector is split across four DRAM channels, so all 64 values reach the distance pipeline at once.
  • Fine-grained pipeline: four stages — prep, distance calculation, select, writeback — handle many candidate nodes at the same time, reducing idle time and raising throughput.
  • Asynchronous hashing: visited-node checks run in parallel with speculative vector loads and distance math, so hash latency is hidden and the pipeline does not stall.

Evaluation

CANNON was tested on SIFT, MS-Turing-ANNS, and Wiki-all scaled up to 500 million vectors, against CPU search (HNSW), GPU search (CuVS/CAGRA), and CXL memory expansion alone (CXL-MEM). When data fits in local memory, CANNON matches the CPU; as data grows, its latency stays flat.

  • Latency: from 100M to 500M vectors, CANNON holds about 0.1 ms per query, while the CPU drops into swap and slows to around 100 ms — nearly 1,000x worse.
  • Vs. GPU: the GPU fails with OOM once the index exceeds its memory; CANNON keeps running and is up to 843.87x faster than the CPU in these memory-bound cases.
  • Vs. plain CXL: cutting the per-hop host–device round trips makes CANNON about 3x faster than the CXL-MEM baseline.



Figure 2: Query latency vs. dataset size — CPU/GPU/CANNON, CPU-HNSW/CPU-CAGRA, and CXL-MEM/CANNON.

 

Conclusion

Large-scale graph ANN search no longer fits in CPU or GPU local memory. CANNON moves the whole search pipeline into logic next to the memory, makes CXL memory an active compute tier, and on real CXL hardware scales smoothly with up to 843x lower latency than the CPU.


DOI: https://doi.org/10.1145/3770743.3803905 



#CANNON#HNSW#CXL-memory-expander

Popular Insights

Previous Next List