AMD EPYC for AI inference
Cores, DDR5 channels and memory bandwidth per socket and per system for EPYC 9004, 9005 and 9006, with measured LLM results and what AMD's claims assume.
In this article7

Where EPYC fits in an AI server
AMD EPYC appears in AI inference in two roles, and each rewards a different specification. In the first, EPYC is the host CPU of a GPU server: it runs the operating system and the serving stack and moves data to the accelerators. In the second, EPYC runs the model itself, with no accelerator. AMD sells the 5th Gen EPYC 9005 family for both: it names CPU-only servers for language models of 13 billion parameters and below, and high-frequency parts as host CPUs for GPU systems.[1]
The two roles split along one line. A host CPU is judged on clock speed, PCIe lanes and how fast it feeds the GPUs. A CPU that runs the model is judged on memory bandwidth, because generating each token reads the model's weights from memory again. The general trade-off between the two kinds of silicon is covered in CPU vs GPU for AI inference.
Three EPYC generations are in the market in November 2026. EPYC 9004 is the 4th Gen family, EPYC 9005 ("Turin") the 5th, and EPYC 9006 ("Venice") the 6th. AMD announced general availability of the EPYC 9006 SP7 processors on 10 November 2026, with server platforms shipping from OEMs and cloud instances from Microsoft Azure and OCI announced as coming soon.[2] SP7 is the high-end socket; the 9006 SP8, 9006X and 9006 LP families follow later.[3]
Cores, channels and published inference results
The figure that decides CPU inference is theoretical memory bandwidth: memory channels × transfer rate in MT/s (million transfers per second) × 8 bytes. One channel of DDR5-4800 delivers 38.4 GB/s.[4] A socket is one processor package; a two-socket server holds two, each with its own memory.
| EPYC generation | Max cores per socket | DDR5 channels per socket | Memory speed | Theoretical bandwidth per socket | Theoretical bandwidth, two-socket system |
|---|---|---|---|---|---|
| EPYC 9004 (4th Gen) | 128 | 12 | DDR5-4800 | 460.8 GB/s | 921.6 GB/s |
| EPYC 9005 (5th Gen) | 192 | 12 | DDR5-6000 reference, DDR5-6400 on enabled platforms | 576 GB/s at 6000, 614.4 GB/s at 6400 | 1,152 GB/s at 6000, 1,228.8 GB/s at 6400 |
| EPYC 9006 SP7 (6th Gen) | 256 | 16 | MRDIMM-12800 | 1,638.4 GB/s | 3,276.8 GB/s |
The 9004 and 9005 rows follow Supermicro's CPU table, which lists 12 channels, 4800 and 6400 MT/s and 460.8 and 614.4 GB/s per socket.[5] AMD supports DDR5-6400 on the 9005 family only for targeted customers and configurations, with OEM enablement and a BIOS update, and its own test systems run "DDR5-6400 (at 6000 MT/s)".[1] The 9006 row is AMD's: 16 channels of MRDIMM-12800, a multiplexed registered memory module, for up to 1,638.4 GB/s of theoretical bandwidth per socket, and up to 256 cores and 512 threads on the EPYC 9996.[3] The two-socket column is the per-socket figure doubled, nothing more.
Measured bandwidth sits below the table. In heise's test of EPYC 9005 systems, a stream-triad-like measurement reached just under 900 GB/s on a pair of processors with memory run at 6000 MT/s, AMD's reference specification.[6] That is about 78 % of the 1,152 GB/s the two sockets offer on paper.

Published inference results are fewer than published bandwidth figures. The comparable public suite is MLPerf Inference Datacenter, which sets compliance rules and quality targets per scenario.[7] The most direct CPU-only data point is a llama.cpp report from 8 February 2025: on a dual EPYC 9175F system with 16 DDR5-6400 modules, Llama-3.1-70B-Instruct at 16-bit precision generated 2.4 tokens per second, and 4.31 tokens per second once the model was loaded with matching NUMA settings.[8]
Host duty versus standalone CPU inference
Standalone CPU inference is a bandwidth calculation first. During decode, the phase in which a model writes its answer token by token, every token reads the full set of weights. The upper limit is usable bandwidth divided by the bytes read per token.
The llama.cpp result shows the arithmetic. Sixteen DDR5-6400 modules give 16 × 51.2 GB/s = 819.2 GB/s across both sockets. A 70B model at 16 bits holds about 140 GB of weights, so the ceiling is close to 5.8 tokens per second. The measured 4.31 tokens per second is about three quarters of it.[8] For a model of 8 billion parameters stored at 8 bits, about 8 GB per token, half of heise's measured pair (about 450 GB/s on one socket) puts the single-stream ceiling near 56 tokens per second. That is the range in which AMD positions CPU-only servers.[1]

The second socket is not free bandwidth. NUMA (non-uniform memory access) means each socket reaches its own memory faster than the other socket's; the llama.cpp result went from 2.4 to 4.31 tokens per second on the same hardware only after the model was spread across both memory pools. Placement is covered in NUMA inference and server pinning.
Host duty asks for different parts. In the 9005 family, AMD offers the high-frequency EPYC 9575F as a host CPU for GPU servers.[1] The 6th Gen family reaches up to 5.3 GHz with 96 cores for host-node roles, with PCIe Gen 6.[3] For buyers who rent rather than build, AWS M8a instances run on 5th Gen EPYC with up to 192 vCPUs and 768 GiB of memory and list ML inference among their uses.[9]
What AMD's own benchmarks assume
AMD's inference figures are AMD's measurements and projections, and each one rests on a configuration worth reading before it goes into a plan.
- CPU-only throughput. "Up to 2x inference throughput" compares servers with two EPYC 9965 processors against the previous generation; it is a two-socket figure.[1]
- Host CPU. An 8-GPU server with the EPYC 9575F delivered up to 13 % faster time to first token and 6.6 % higher overall inference throughput than the same server with Intel Xeon 6960P CPUs, in AMD's internal testing across 8 models and 4 use cases.[1]
- 6th Gen. "Up to 1.8x the LLM output throughput of previous generation EPYC processors" is a projection for one workload: Qwen3-30B-A3B in vLLM with CPU offload to one NVIDIA B200 GPU.[2]
- Memory speed. AMD's 9005 test systems list DDR5-6400 modules run at 6000 MT/s, so the 614.4 GB/s figure is not the configuration behind its results.[1]
AMD's own page on inference with EPYC 9005 frames the choice the same way: match model size and latency requirements to the hardware. None of the figures above is an independent benchmark.
Specifying EPYC against the alternatives
A specification starts from the role. For CPU-only inference, fill every memory channel, count bandwidth per socket and plan NUMA placement before counting cores. For a GPU host, weigh clock speed and PCIe generation over core count. In both cases, read every published figure for its socket count and memory speed.
The alternatives compete on the same terms. AMD's own footnote lists Intel Xeon 6 and Arm AGI processors at 12 channels of MRDIMM-8800, 844.8 GB/s of theoretical bandwidth per socket, against 16 channels on EPYC 9006.[3] That is AMD's comparison, not a measurement of either rival. The evidence for each one is in Intel Xeon for AI inference and Arm CPUs for AI inference, and the wider choice in best processor for inference.
Price is published only at the top of the 6th Gen range: AMD lists the 256-core EPYC 9996 at USD 14,904 in 1,000-unit quantities, as of 5 November 2026.[3] Before a server is ordered, the one setting that most often decides CPU inference speed is memory placement across sockets, covered in the guide to NUMA pinning for inference servers.
Frequently asked questions
How much memory bandwidth does an AMD EPYC 9005 have?
An EPYC 9005 socket has 12 DDR5 channels: 576 GB/s of theoretical bandwidth at the reference DDR5-6000 and 614.4 GB/s at DDR5-6400 on enabled platforms. A two-socket server doubles that on paper; heise measured just under 900 GB/s on a pair at DDR5-6000.
Can an AMD EPYC server run an LLM without a GPU?
Yes, for smaller models. AMD names language models of 13 billion parameters and below for CPU-only EPYC 9005 servers. Speed is limited by memory bandwidth: a 70B model at 16 bits reached 4.31 tokens per second on a dual EPYC 9175F system in llama.cpp.
What changes with EPYC 9006 Venice for inference?
EPYC 9006 SP7 raises memory channels per socket from 12 to 16 and supports MRDIMM-12800, for up to 1,638.4 GB/s of theoretical bandwidth per socket, and offers up to 256 cores. It became generally available on 10 November 2026.
Sources
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Processors →Server memory bandwidth: how to read and cite the numbers
Server memory bandwidth per socket and per system, peak versus measured STREAM results, and published figures by vendor, each with its date.

Do you need GPU or RAM for LLM inference? The bandwidth answer
GPU memory vs system RAM for LLMs: what each tier holds, why PCIe offloading slows decoding, and how HBM and DDR5 contract prices compare.

Arm CPUs for AI inference: what published results show
Arm CPUs for AI inference: where Arm silicon is deployed, the bandwidth and benchmark results, and when Arm is the defensible choice.
