NUMA inference: CPU memory bandwidth and server pinning
Learn how NUMA inference limits large language model throughput on multi-socket CPU servers and why pinning engines to local memory maximizes bandwidth.
In this article6

What NUMA means for an inference server
Non-Uniform Memory Access (NUMA) is the design in which a server's cores and memory are partitioned into nodes, each with its own memory controller. A core reads its own node's memory quickly and any other node's memory slowly, because the request crosses an inter-socket or inter-die link. On a multi-socket CPU server running large language model (LLM) inference, this divides the machine's memory bandwidth into local and remote pools, and the difference between them sets the token rate.
The recurring error on this topic is reading a single-socket figure as a system figure. A two-socket server does not add its two sockets' bandwidth into one pool: a core on socket 1 reaching into socket 0's memory gets a fraction of the local rate.
Local versus remote memory bandwidth
The size of the penalty is measurable, not marginal. On a four-node Arm test machine with 48 cores and 6 DDR4 channels per node, the ArcLight paper (arXiv, retrieved 25 September 2026) measured 102 GB/s for a node reading its own memory and 22 to 26 GB/s for the same node reading any other node's memory[1]. An Intel community forum measurement (retrieved 25 September 2026) on a two-node Xeon E5-2620 v3 reported 44 GB/s local against 6 GB/s remote[2]. The inter-socket link, not the DIMM, is the bottleneck in both cases.
Memory and interconnect figures for the seven providers, side by side[3][4][5][6][7][8][9]. A "–" means the page lists no value for that cell.
| Provider | Memory interface per socket | Interconnect architecture | Source page (retrieved 2026-09-23) |
|---|---|---|---|
| Ampere Computing | 12 channels DDR5-5600, 1.5 TB addressable | Coherent mesh with distributed snoop filtering | AmpereOne M product brief |
| Arm | – | Coherent Mesh Network (CMN), the interconnect behind Neoverse-based server CPUs | Arm Neoverse CMN-700 interconnect page |
| Fujitsu | 32 GiB HBM2 per package | 48 computing cores and 4 assistant cores per package, connected by the Tofu interconnect | Fujitsu A64FX specification announcement, 22 August 2018 |
| IBM | – | On-processor AI accelerator on Telum II, paired with the Spyre Accelerator for IBM Z and LinuxONE | IBM Telum II and Spyre announcement, 26 August 2024 |
| SiFive | – | RISC-V core IP (Intelligence X300 series, building on the X280) | SiFive Intelligence X300 series page |
| SiPearl | 4 DDR5 interfaces supporting 2 DIMMs per channel, plus 4 stacks of on-package HBM | 80 Arm Neoverse V1 cores with 104 lanes PCIe Gen5; no published socket-to-socket interconnect figure | SiPearl Rhea1 product page |
| Ventana Micro Systems | – | RISC-V CPU chiplets (Veyron family) connected over a standard chiplet interconnect; memory channel count not stated on the announcement page | Ventana Veyron V2 announcement, 7 November 2023 |
For context on what one DDR5 channel contributes: a single DDR5-4800 DIMM carries 38.4 GB/s, against 25.6 GB/s for DDR4-3200, per ServeTheHome's generation-by-generation guide (retrieved 25 September 2026)[10]. Multiplying a per-channel figure by a socket's channel count gives the local pool that one socket can draw on.
How serving engines are pinned in practice
Pinning means binding a serving engine's worker threads to specific cores and its weight tensors to the memory on those cores' own node, so the operating system cannot migrate a thread to a different socket mid-generation. Without pinning, a thread that moves to a remote socket keeps reading weights that stayed behind, and every token step pays the remote-access penalty measured above.
The ArcLight paper documents why default behaviour falls short. llama.cpp exposes a --numa distribute option that spreads threads evenly across nodes, but it does not bind tensors to nodes, so under a first-touch allocation most weight partitions end up serving remote activation reads on every general matrix-multiply step after the first[1]. ArcLight instead allocates a separate buffer in each node's local memory and binds tensors to nodes explicitly, and measures up to 46% higher inference throughput than llama.cpp on many-core platforms as a result[1].
- Identify the node topology (numactl --hardware or lscpu) and record core counts and memory channels per node.
- Pin worker threads to the cores of one node, using numactl, taskset or the serving engine's own affinity flags.
- Allocate model weights and KV cache from that same node's local memory, so tensor placement matches thread placement.
- Repeat per node for tensor-parallel workers, rather than letting one process span all nodes.
- Measure tokens per second against the unpinned baseline and record the configuration alongside the figure.
Where NUMA effects dominate throughput
NUMA effects dominate wherever the workload is memory-bandwidth-bound rather than compute-bound. LLM token generation is the clearest case: during decode, every token step streams the model weights from memory, so the token rate tracks the bandwidth of the pool the weights sit in. A reader comparing a CPU deployment against a GPU one should weigh the same local-versus-remote question on both sides, as covered in the CPU versus GPU comparison for AI inference; accelerator specifications sit in the overview of the world's strongest GPU. Batch or offline inference, where latency is not critical, tolerates the remote-access penalty better than latency-constrained serving, where a single cross-socket read stalls the response.
MLPerf Inference: Datacenter results (retrieved 25 September 2026) publish the processor, accelerator and count for each submitted system, which makes them a checkable basis for comparison across providers[11]. Where a provider publishes no such configuration alongside a performance claim, the claim is not reproducible and should be treated as unverified rather than estimated.
Measuring your own NUMA penalty
The steps above give the procedure; the check that makes the result usable in a plan is a local-versus-remote pair on the same machine, each with its date and configuration recorded. A reader deciding which host silicon to plan around can then compare that measured pair against the per-socket figures in the table and the guidance in the best processor for inference roundup, which covers the same local-pool question from the hardware side.
Sources
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Processors →AMD EPYC for AI inference
Cores, DDR5 channels and memory bandwidth per socket and per system for EPYC 9004, 9005 and 9006, with measured LLM results and what AMD's claims assume.

Server memory bandwidth: how to read and cite the numbers
Server memory bandwidth per socket and per system, peak versus measured STREAM results, and published figures by vendor, each with its date.

Memory allocation on AI servers: weights, KV cache and host RAM
How AI server memory splits between weights, KV cache, activations and host RAM, with dated figures on capacity, bandwidth, price and wafers.
