Arm CPUs for AI inference: what published results show
Arm CPUs for AI inference: where Arm silicon is deployed, the bandwidth and benchmark results, and when Arm is the defensible choice.
In this article6

Where Arm server CPUs are actually deployed
Arm silicon reaches AI inference through two routes. The first is hyperscaler in-house design: AWS's Graviton5 puts 192 Arm cores on a single socket, is in technology preview with selected customers, and backs M9g instances that are in preview, with C9g and R9g instances expected in 2026 (The Next Platform, 5 December 2025)[1]. The second is the licensed platform: Arm's own infrastructure pages name the Neoverse CSS N2, V3 and N4 compute subsystems and the Arm AGI CPU, described as Arm's first production silicon (arm.com, retrieved 23 September 2026)[2].
Where a CPU rather than a GPU should carry a workload is a separate architectural question; the design differences behind that split are compared in CPU vs GPU for AI inference: Architecture and performance. The table below covers the companies that design or ship Arm server CPUs.
| Provider | Architecture | Published figure relevant to inference | Source page | Retrieved |
|---|---|---|---|---|
| Ampere Computing | AmpereOne and AmpereOne M: 96 to 192 Arm cores, 8- or 12-channel DDR5 | AmpereOne M is described as optimized for volume AI inference; the product page gives no benchmark figure | Processors page | 2 October 2026 |
| Arm | Neoverse CSS N2, V3 and N4 compute subsystems; Arm AGI CPU, described as Arm's first production silicon | No inference figure on the page read | Computing infrastructure page | 23 September 2026 |
| AWS | Graviton5: 192 Arm cores on one socket, twelve DDR5 controllers | 806.4 GB/sec memory bandwidth per socket at DDR5-8400 | The Next Platform, 5 December 2025 | 23 September 2026 |
| Google Cloud | Axion, behind the C4A and N4A instances | No inference figure on the page read | Arm computing infrastructure page | 23 September 2026 |
| Microsoft | Cobalt 100 and Cobalt 200, run in Azure | No inference figure on the page read | Arm computing infrastructure page | 23 September 2026 |
| NVIDIA | Grace CPU Superchip: two Grace CPUs, 144 Arm Neoverse V2 cores, joined by NVLink-C2C at 900 GB/s | Up to 1 TB/s LPDDR5X memory bandwidth per two-CPU Superchip | Grace CPU page | 2 October 2026 |
| SiPearl | Rhea1: 80 Arm Neoverse V1 cores, two 256-bit SVE units per core, 4 HBM stacks plus DDR5 | Positioned for HPC and AI inference; no inference figure on the page read | Rhea1 page | 2 October 2026 |
Each row is read from the vendor's own page or, where noted, from Arm's or The Next Platform's; a page that gives no inference figure is marked as such.
Bandwidth, power and published results
The measurable case for current Arm sockets rests on memory arithmetic, and the recurring error on this topic is a single-socket figure read as a system figure. Four published data points frame it:
- Per-DIMM bandwidth: DDR5-4800 delivers 38.4 GB/s per DIMM against 25.6 GB/s for DDR4-3200, about 50% more (ServeTheHome guide, published February 2023, retrieved 23 September 2026)[6].
- Socket-level arithmetic: Graviton5 carries twelve DDR5 controllers and supports DDR5-8400, giving 806.4 GB/sec for a single socket, against the 1,075.2 GB/sec that a dual-socket Graviton4 system aggregates (The Next Platform, 5 December 2025, retrieved 23 September 2026)[1].
- Comparable instrument: MLPerf Inference: Datacenter, whose Closed division compares hardware platforms and software frameworks like for like, and whose power figures are measured at the wall (MLCommons, V6.1, retrieved 23 September 2026)[7].
- Published software result: ArcLight, a many-core CPU inference architecture, reports up to 46% higher inference throughput than llama.cpp on many-core CPU platforms (arXiv, May 2026)[8].
The error to avoid is concrete: 806.4 GB/sec is one Graviton5 socket, while 1,075.2 GB/sec is two Graviton4 sockets pooled through NUMA clustering. A document that quotes the first against the second compares a socket with a system, and the deficit disappears once the basis is stated.
Cloud instances and in-house silicon
The two routes reach a buyer through different doors. Cloud instances put Arm silicon behind a vendor's API: AWS's Graviton5 backs M9g instances in preview, with C9g and R9g instances expected in 2026 (The Next Platform, 5 December 2025)[1], while Google Cloud offers Axion-based C4A and N4A instances and Microsoft runs Cobalt 100 and Cobalt 200 processors in Azure (arm.com, retrieved 23 September 2026)[2]. In-house silicon is the other door: a provider licenses the Neoverse platform or builds its own CPU around it, as AWS, Google and Microsoft have each done. For a buyer, the practical difference is who owns the tuning: a cloud instance arrives already matched to the silicon, while in-house silicon puts the integration work on the operator.
Where the software stack still lags
The software limit is architectural. Many-core servers group cores and memory into NUMA nodes, and accessing memory on a remote node costs significantly more than local access. Mainstream CPU frameworks such as llama.cpp largely overlook this cross-NUMA memory access wall, which is why ArcLight was built NUMA-aware from the ground up, with NUMA-aware memory and thread management, and open-sourced for arbitrary CPU devices (arXiv, May 2026)[8].
When Arm is the defensible choice
For workload sizing, a vendor's own published guidance is usable if labelled as such: providers publish the model-size and latency bands their CPUs and accelerators each cover, and a buyer can read those bands off a provider's own page with its retrieval date. That guidance is the vendor's own, not an independent finding, and the same caution applies to any provider page.
- Match model size and latency band to silicon before comparing vendors or prices.
- Check whether a bandwidth figure is per socket or per system before carrying it into a decision.
- Prefer MLPerf Closed-division results, with power measured at the wall, to vendor marketing figures.
- Treat hosting jurisdiction as a separate axis from silicon choice; the EU residency question is covered in A practical guide to data sovereignty for AI inference in the EU: residency, the CLOUD Act, and how to reduce exposure.
On those criteria, Arm is the defensible choice where the workload fits the CPU band, the bandwidth figure is quoted on a stated socket or system basis, and the performance claim traces to a dated benchmark rather than a vendor page. The broader host-processor decision is worked through in Best processor for inference.
Sources for this article were last verified on 23 September 2026, with a next review due by 23 March 2027.
Sources
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Processors →AMD EPYC for AI inference
Cores, DDR5 channels and memory bandwidth per socket and per system for EPYC 9004, 9005 and 9006, with measured LLM results and what AMD's claims assume.

Where to run Kimi K3 in Europe, and what it costs
Compare published Kimi K3 token prices across European inference providers, including cached rates, and learn what rate cards do not cover.

Inference engines compared: what eight serving engines are for
vLLM, TensorRT-LLM, TGI, SGLang and four more inference engines compared on hardware, licence and maturity, with every benchmark's batch size named.
