ProcessorsServer lines

Arm CPUs for AI inference: what published results show

Arm CPUs for AI inference: where Arm silicon is deployed, the bandwidth and benchmark results, and when Arm is the defensible choice.

, AI Researcher5 min read
In this article6
  1. Where Arm server CPUs are actually deployed
  2. Bandwidth, power and published results
  3. Cloud instances and in-house silicon
  4. Where the software stack still lags
  5. When Arm is the defensible choice
  6. Sources
Diagram of a many-core Arm server CPU socket with cores grouped into NUMA nodes and DDR5 memory channels, illustrating memory bandwidth for AI inference.
Diagram of a many-core Arm server CPU socket with cores grouped into NUMA nodes and DDR5 memory channels, illustrating memory bandwidth for AI inference.

Where Arm server CPUs are actually deployed

Arm silicon reaches AI inference through two routes. The first is hyperscaler in-house design: AWS's Graviton5 puts 192 Arm cores on a single socket, is in technology preview with selected customers, and backs M9g instances that are in preview, with C9g and R9g instances expected in 2026 (The Next Platform, 5 December 2025)[1]. The second is the licensed platform: Arm's own infrastructure pages name the Neoverse CSS N2, V3 and N4 compute subsystems and the Arm AGI CPU, described as Arm's first production silicon (arm.com, retrieved 23 September 2026)[2].

Where a CPU rather than a GPU should carry a workload is a separate architectural question; the design differences behind that split are compared in CPU vs GPU for AI inference: Architecture and performance. The table below covers the companies that design or ship Arm server CPUs.

ProviderArchitecturePublished figure relevant to inferenceSource pageRetrieved
Ampere ComputingAmpereOne and AmpereOne M: 96 to 192 Arm cores, 8- or 12-channel DDR5AmpereOne M is described as optimized for volume AI inference; the product page gives no benchmark figureProcessors page2 October 2026
ArmNeoverse CSS N2, V3 and N4 compute subsystems; Arm AGI CPU, described as Arm's first production siliconNo inference figure on the page readComputing infrastructure page23 September 2026
AWSGraviton5: 192 Arm cores on one socket, twelve DDR5 controllers806.4 GB/sec memory bandwidth per socket at DDR5-8400The Next Platform, 5 December 202523 September 2026
Google CloudAxion, behind the C4A and N4A instancesNo inference figure on the page readArm computing infrastructure page23 September 2026
MicrosoftCobalt 100 and Cobalt 200, run in AzureNo inference figure on the page readArm computing infrastructure page23 September 2026
NVIDIAGrace CPU Superchip: two Grace CPUs, 144 Arm Neoverse V2 cores, joined by NVLink-C2C at 900 GB/sUp to 1 TB/s LPDDR5X memory bandwidth per two-CPU SuperchipGrace CPU page2 October 2026
SiPearlRhea1: 80 Arm Neoverse V1 cores, two 256-bit SVE units per core, 4 HBM stacks plus DDR5Positioned for HPC and AI inference; no inference figure on the page readRhea1 page2 October 2026

Each row is read from the vendor's own page or, where noted, from Arm's or The Next Platform's; a page that gives no inference figure is marked as such.

Bandwidth, power and published results

The measurable case for current Arm sockets rests on memory arithmetic, and the recurring error on this topic is a single-socket figure read as a system figure. Four published data points frame it:

  • Per-DIMM bandwidth: DDR5-4800 delivers 38.4 GB/s per DIMM against 25.6 GB/s for DDR4-3200, about 50% more (ServeTheHome guide, published February 2023, retrieved 23 September 2026)[6].
  • Socket-level arithmetic: Graviton5 carries twelve DDR5 controllers and supports DDR5-8400, giving 806.4 GB/sec for a single socket, against the 1,075.2 GB/sec that a dual-socket Graviton4 system aggregates (The Next Platform, 5 December 2025, retrieved 23 September 2026)[1].
  • Comparable instrument: MLPerf Inference: Datacenter, whose Closed division compares hardware platforms and software frameworks like for like, and whose power figures are measured at the wall (MLCommons, V6.1, retrieved 23 September 2026)[7].
  • Published software result: ArcLight, a many-core CPU inference architecture, reports up to 46% higher inference throughput than llama.cpp on many-core CPU platforms (arXiv, May 2026)[8].

The error to avoid is concrete: 806.4 GB/sec is one Graviton5 socket, while 1,075.2 GB/sec is two Graviton4 sockets pooled through NUMA clustering. A document that quotes the first against the second compares a socket with a system, and the deficit disappears once the basis is stated.

Cloud instances and in-house silicon

The two routes reach a buyer through different doors. Cloud instances put Arm silicon behind a vendor's API: AWS's Graviton5 backs M9g instances in preview, with C9g and R9g instances expected in 2026 (The Next Platform, 5 December 2025)[1], while Google Cloud offers Axion-based C4A and N4A instances and Microsoft runs Cobalt 100 and Cobalt 200 processors in Azure (arm.com, retrieved 23 September 2026)[2]. In-house silicon is the other door: a provider licenses the Neoverse platform or builds its own CPU around it, as AWS, Google and Microsoft have each done. For a buyer, the practical difference is who owns the tuning: a cloud instance arrives already matched to the silicon, while in-house silicon puts the integration work on the operator.

Where the software stack still lags

The software limit is architectural. Many-core servers group cores and memory into NUMA nodes, and accessing memory on a remote node costs significantly more than local access. Mainstream CPU frameworks such as llama.cpp largely overlook this cross-NUMA memory access wall, which is why ArcLight was built NUMA-aware from the ground up, with NUMA-aware memory and thread management, and open-sourced for arbitrary CPU devices (arXiv, May 2026)[8].

When Arm is the defensible choice

For workload sizing, a vendor's own published guidance is usable if labelled as such: providers publish the model-size and latency bands their CPUs and accelerators each cover, and a buyer can read those bands off a provider's own page with its retrieval date. That guidance is the vendor's own, not an independent finding, and the same caution applies to any provider page.

  1. Match model size and latency band to silicon before comparing vendors or prices.
  2. Check whether a bandwidth figure is per socket or per system before carrying it into a decision.
  3. Prefer MLPerf Closed-division results, with power measured at the wall, to vendor marketing figures.
  4. Treat hosting jurisdiction as a separate axis from silicon choice; the EU residency question is covered in A practical guide to data sovereignty for AI inference in the EU: residency, the CLOUD Act, and how to reduce exposure.

On those criteria, Arm is the defensible choice where the workload fits the CPU band, the bandwidth figure is quoted on a stated socket or system basis, and the performance claim traces to a dated benchmark rather than a vendor page. The broader host-processor decision is worked through in Best processor for inference.

Sources for this article were last verified on 23 September 2026, with a next review due by 23 March 2027.

Sources

  1. nextplatform.com
  2. arm.com
  3. amperecomputing.com
  4. nvidia.com
  5. sipearl.com
  6. servethehome.com
  7. mlcommons.org
  8. arxiv.org

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Processors →

AMD EPYC for AI inference

Cores, DDR5 channels and memory bandwidth per socket and per system for EPYC 9004, 9005 and 9006, with measured LLM results and what AMD's claims assume.

An opened server tray with CPU sockets, memory modules and cabling on a workbench in a hardware lab.