ProcessorsMemory bandwidth

NUMA inference: CPU memory bandwidth and server pinning

Learn how NUMA inference limits large language model throughput on multi-socket CPU servers and why pinning engines to local memory maximizes bandwidth.

, AI Researcher5 min read
In this article6
  1. What NUMA means for an inference server
  2. Local versus remote memory bandwidth
  3. How serving engines are pinned in practice
  4. Where NUMA effects dominate throughput
  5. Measuring your own NUMA penalty
  6. Sources
A schematic diagram showing a multi-socket server motherboard with CPUs and memory banks, highlighting the latency differences between local memory channels and cross-socket interconnects.
A schematic diagram showing a multi-socket server motherboard with CPUs and memory banks, highlighting the latency differences between local memory channels and cross-socket interconnects.

What NUMA means for an inference server

Non-Uniform Memory Access (NUMA) is the design in which a server's cores and memory are partitioned into nodes, each with its own memory controller. A core reads its own node's memory quickly and any other node's memory slowly, because the request crosses an inter-socket or inter-die link. On a multi-socket CPU server running large language model (LLM) inference, this divides the machine's memory bandwidth into local and remote pools, and the difference between them sets the token rate.

The recurring error on this topic is reading a single-socket figure as a system figure. A two-socket server does not add its two sockets' bandwidth into one pool: a core on socket 1 reaching into socket 0's memory gets a fraction of the local rate.

Local versus remote memory bandwidth

The size of the penalty is measurable, not marginal. On a four-node Arm test machine with 48 cores and 6 DDR4 channels per node, the ArcLight paper (arXiv, retrieved 25 September 2026) measured 102 GB/s for a node reading its own memory and 22 to 26 GB/s for the same node reading any other node's memory[1]. An Intel community forum measurement (retrieved 25 September 2026) on a two-node Xeon E5-2620 v3 reported 44 GB/s local against 6 GB/s remote[2]. The inter-socket link, not the DIMM, is the bottleneck in both cases.

Memory and interconnect figures for the seven providers, side by side[3][4][5][6][7][8][9]. A "–" means the page lists no value for that cell.

ProviderMemory interface per socketInterconnect architectureSource page (retrieved 2026-09-23)
Ampere Computing12 channels DDR5-5600, 1.5 TB addressableCoherent mesh with distributed snoop filteringAmpereOne M product brief
Arm–Coherent Mesh Network (CMN), the interconnect behind Neoverse-based server CPUsArm Neoverse CMN-700 interconnect page
Fujitsu32 GiB HBM2 per package48 computing cores and 4 assistant cores per package, connected by the Tofu interconnectFujitsu A64FX specification announcement, 22 August 2018
IBM–On-processor AI accelerator on Telum II, paired with the Spyre Accelerator for IBM Z and LinuxONEIBM Telum II and Spyre announcement, 26 August 2024
SiFive–RISC-V core IP (Intelligence X300 series, building on the X280)SiFive Intelligence X300 series page
SiPearl4 DDR5 interfaces supporting 2 DIMMs per channel, plus 4 stacks of on-package HBM80 Arm Neoverse V1 cores with 104 lanes PCIe Gen5; no published socket-to-socket interconnect figureSiPearl Rhea1 product page
Ventana Micro Systems–RISC-V CPU chiplets (Veyron family) connected over a standard chiplet interconnect; memory channel count not stated on the announcement pageVentana Veyron V2 announcement, 7 November 2023

For context on what one DDR5 channel contributes: a single DDR5-4800 DIMM carries 38.4 GB/s, against 25.6 GB/s for DDR4-3200, per ServeTheHome's generation-by-generation guide (retrieved 25 September 2026)[10]. Multiplying a per-channel figure by a socket's channel count gives the local pool that one socket can draw on.

How serving engines are pinned in practice

Pinning means binding a serving engine's worker threads to specific cores and its weight tensors to the memory on those cores' own node, so the operating system cannot migrate a thread to a different socket mid-generation. Without pinning, a thread that moves to a remote socket keeps reading weights that stayed behind, and every token step pays the remote-access penalty measured above.

The ArcLight paper documents why default behaviour falls short. llama.cpp exposes a --numa distribute option that spreads threads evenly across nodes, but it does not bind tensors to nodes, so under a first-touch allocation most weight partitions end up serving remote activation reads on every general matrix-multiply step after the first[1]. ArcLight instead allocates a separate buffer in each node's local memory and binds tensors to nodes explicitly, and measures up to 46% higher inference throughput than llama.cpp on many-core platforms as a result[1].

  1. Identify the node topology (numactl --hardware or lscpu) and record core counts and memory channels per node.
  2. Pin worker threads to the cores of one node, using numactl, taskset or the serving engine's own affinity flags.
  3. Allocate model weights and KV cache from that same node's local memory, so tensor placement matches thread placement.
  4. Repeat per node for tensor-parallel workers, rather than letting one process span all nodes.
  5. Measure tokens per second against the unpinned baseline and record the configuration alongside the figure.

Where NUMA effects dominate throughput

NUMA effects dominate wherever the workload is memory-bandwidth-bound rather than compute-bound. LLM token generation is the clearest case: during decode, every token step streams the model weights from memory, so the token rate tracks the bandwidth of the pool the weights sit in. A reader comparing a CPU deployment against a GPU one should weigh the same local-versus-remote question on both sides, as covered in the CPU versus GPU comparison for AI inference; accelerator specifications sit in the overview of the world's strongest GPU. Batch or offline inference, where latency is not critical, tolerates the remote-access penalty better than latency-constrained serving, where a single cross-socket read stalls the response.

MLPerf Inference: Datacenter results (retrieved 25 September 2026) publish the processor, accelerator and count for each submitted system, which makes them a checkable basis for comparison across providers[11]. Where a provider publishes no such configuration alongside a performance claim, the claim is not reproducible and should be treated as unverified rather than estimated.

Measuring your own NUMA penalty

The steps above give the procedure; the check that makes the result usable in a plan is a local-versus-remote pair on the same machine, each with its date and configuration recorded. A reader deciding which host silicon to plan around can then compare that measured pair against the per-socket figures in the table and the guidance in the best processor for inference roundup, which covers the same local-pool question from the hardware side.

Sources

  1. arxiv.org
  2. community.intel.com
  3. amperecomputing.com
  4. arm.com
  5. fujitsu.com
  6. ibm.com
  7. sifive.com
  8. sipearl.com
  9. prnewswire.com
  10. servethehome.com
  11. mlcommons.org

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Processors →

AMD EPYC for AI inference

Cores, DDR5 channels and memory bandwidth per socket and per system for EPYC 9004, 9005 and 9006, with measured LLM results and what AMD's claims assume.

An opened server tray with CPU sockets, memory modules and cabling on a workbench in a hardware lab.