[Summary] Toward Deployable CXL-PNM: The CMM-Ax Prototype and Software Stack
Background and Goal
Data-centric workloads such as RAG vector search, analytics, graph processing, and ML are increasingly limited by memory access rather than compute (the memory wall). CXL offers a standard path to near-memory computing on commodity servers, but prior CXL-PNM designs either rely on fixed-function logic with limited programmability or remain lab-scale prototypes and simulations, without validated software-stack integration or multi-tenant operation. Extending an earlier logic-only accelerator that confirmed the feasibility of near-data execution (Sim et al., IEEE CAL 2023), this paper builds and evaluates CMM-Ax, a hybrid CXL-PNM with device-side programmability, together with a production-oriented software stack on a real FPGA device, from a single node up to a cluster. As a standard CXL Type-3 device, CMM-Ax requires no host modifications such as changes to the CPU memory controller.
Hardware Architecture (Fig. 1)
We classify CXL-PNM designs as logic-only, hybrid, or core-only and adopt the hybrid design to combine fixed-function efficiency with general-purpose flexibility. Offload is limited to bandwidth-bound, streaming-friendly operators with strong data reduction (filter, aggregation, top-K), so that host–device transaction costs are amortized. The Primitive Function Layer (PFL) implements only domain-agnostic primitives such as MAC, ACC, and CMP in hardware; on-chip ARM cores parse descriptors, build micro-op graphs, and dispatch them to the PFL through decoupled FIFOs. Branch-heavy steps, such as composing distance calculations and sorting/merging top-K results, run in firmware, so new operators require no logic re-synthesis.
|
|
|
|
Fig. 1. CMM-Ax hybrid architecture and FPGA add-in card |
Fig. 2. Software stack with three integration paths |
The host writes descriptors to CXL memory and submits their pointers via MMIO; control stays on CXL.io while data moves over CXL.mem, minimizing round trips. For address translation, the driver resolves host virtual addresses to HPAs and builds a scatter-gather list, which device hardware maps to DPAs while merging non-contiguous fragments into long burst streams. Because translation state is kept per command, no coherence-managed translation cache is needed on the device, and applications can pass unmodified CPU pointers (via HMSDK malloc/mmap). The prototype is an FH3/4L add-in card with a Xilinx Versal VP1502 (PFL: FP32 32 GFLOPS @ 250 MHz) and 512 GB of 4-channel DDR4-2400 (76.8 GB/s peak), operating as a CXL 2.0 Type-3 device over PCIe Gen4 ×8.
Software Stack (Fig. 2)
A user-space runtime (SQ/CQ, async streams, multi-process access, zero-copy buffers) and a kernel driver (IRQ/poll notification, VA→PA scatter-gather, CXL.io/.mem resource control) underpin three integration paths, with host memory provisioned through HMSDK (dynamic) or DAX (static) in an address space shared with the device. First, custom FAISS Index subclasses (CMMAxIndexFlat/IVFFlat) let existing train()/add()/search() calls run on CMM-Ax by changing only the index type (cosine similarity only, for now).
Second, the programming model compiles __CMMAX__-annotated functions in a single C/C++ source in two passes, producing an x86 host binary with runtime stubs and an ARM shared object for the device. At run time, the .so is loaded onto the device and invoked through descriptors carrying the function name and serialized arguments; firmware resolves the symbol and executes it via the PFL library, whose APIs map one-to-one to hardware primitives, so new operators ship as a new .so without firmware changes. Third, a Kubernetes device plugin partitions each card into vCMM-Ax slices (a CXL.mem aperture with offset and size, plus a compute entitlement) exposed as extended resources; per-pod opaque keys, with the real device file hidden, let the user-space runtime enforce tenant isolation without kernel modifications.
Evaluation
|
Fig. 3. Case A: CPU+CMM vs. CMM-Ax |
Fig. 4. Case B: scaling with CMM-Ax cards (hatched = projected) |
- Exact kNN semantic search. Measurements use an in-house Telco image corpus (FP32, 2048-D, 16.37 M rows, 134 GB) on the initial logic-only CMM-Ax, averaged over three runs. With the same host, index, and recall (27 GB set), replacing a compute-free CXL memory module (CMM) with CMM-Ax raises throughput from 14.0 to 63.6 GB/s (4.54×) and lowers node power from 389 to 318 W, cutting energy per query by 5.56× (≈82%) in terms of TPW (Case A). The authors attribute this to PNM eliminating host round trips in the distance/scan loops, which a capacity-only CMM cannot reduce. When cards are added to a CPU+DIMM server with data split 4:1 by bandwidth (307.2:76.8 GB/s), throughput rises to 1.28× with one card and 1.52× with two, and is projected at 2.04× with four—comparable to two CPU-only servers at about 30% lower power (629 vs. 894 W) (Case B).
|
Fig. 5. ANN IVF ηth and TIR99 |
Fig. 6. K8s cluster memory and QPS (hatched = projected) |
- ANN IVF microbenchmark. Using cuVS Bench in throughput mode on Zilliz-Cohere-10M (1024-D, 40.96 GB), nlist/nprobe were swept per device and batch, and the highest QPS at recall ≥ 0.90 was compared. Throughput efficiency ηth divides measured QPS by an ideal QPS derived from effective bandwidth (74% of peak) and bytes accessed per query; tails are characterized by TIR99 (p99/p50). IVF is memory-bound (81% distance operations, operational intensity < 0.5), so absolute QPS follows bandwidth: at batch 1/10/100, the H100 GPU reaches 485/1,603/4,146, the SPR CPU 127/193/682, and CMM-Ax 63/104/104 (two cards: 50/141/149). At batch 1 and 10, however, CMM-Ax attains ηth of 37.2% and 35.4%, well above the CPU (12.0%, 16.4%) and GPU (7.4%, 23.2%), because near-memory scans drive long DRAM bursts even for a single query; the GPU reaches 63.6% only at batch 100, and TIR99 stays mostly below 2 on all devices. Profiling shows insufficient MLP on the CPU, which at batch 10 spends only 26.81% in user mode with 24.19% memory utilization despite 73.65% CPU utilization, and poor locality on the GPU, whose L1/L2 hit ratios stay at or below 12.7% and SM throughput at or below 5.2% even at 91.7% SM activity. The ~1.4× gain from two cards is attributed not to a DRAM limit but to sub-index broadcast/merge and cross-card imbalance.
- Kubernetes cluster replacement. Under the same filtered-kNN workload (Locust open-loop; 1.64 M × 1024-D working set from a 625 GB corpus), shrinking a four-server ICX cluster (4.0 TB, 71.1 QPS) to two SPR servers drops QPS to 73%, but adding four CMM-Ax cards restores 90% at the same 4.0 TB, halving the server count. Assuming DDR5-4800 CMM-Ax, four cards are projected at 111% and eight at 154% (6.0 TB), i.e., about 1.5× in both capacity and QPS. Tail latency is out of scope.
Conclusion and Limitations
The paper shows on a real FPGA device and software stack that executing data-movement-dominated kernels near CXL memory is architecturally sound and scales at the system level; since energy gains appear even within an FPGA power envelope, an ASIC implementation should leave more headroom. Measured throughput, however, reaches only 60–70% of the hardware data-path peak under conservative defaults, the software is prototype-quality (robustness, failure isolation, security, and observability remain open), and tail latency is not evaluated. Future work includes hybrid/core-only ASIC proofs of concept with billion-scale validation, multi-card studies beyond four cards per node, latency-aware multi-tenant scheduling, and disaggregated PNM over CXL 3.x fabrics.
DOI: https://doi.org/10.1109/TC.2026.3654850









