How to build an AI cluster: the fabric numbers that matter
How to build an AI cluster: the dated fabric and accelerator numbers, the ten-vendor field, and which decisions are expensive to reverse.
In this article8

What an AI cluster has to deliver
The workload sets the specification. A 2024 survey of distributed LLM training on arXiv records the anchor figure: LLaMA-3 pre-training took about 54 days on 16,000 H100-80GB GPUs on Meta's production cluster[1]. The same survey frames the requirements as SER: scalability to tens of thousands of accelerators, efficiency measured as Model FLOPs Utilization, and reliability, because training runs last weeks to months and must survive hardware, network and software failures without losing progress. That makes failure recovery a design input, not an afterthought.
The fabric a cluster must carry is published in NVIDIA's HGX AI Factory reference architecture. It separates four roles: an RDMA-based GPU compute East/West fabric in a rail-optimized leaf-spine topology; a converged North/South network for compute, storage, in-band management and user traffic, with two 400 GbE ports per node for redundancy; storage connectivity; and low-cost 1 Gb/s out-of-band management for baseboard controllers and switches. The measurement basis is stated explicitly: the architecture's 4-node scalable unit carries 64 × 400 Gb/s connections, an aggregate 25.6 Tb/s of compute-fabric bandwidth, retrieved 8 September 2026[2]. How the bandwidth is consumed inside a node and across nodes is covered in how GPUs communicate in a training cluster and in GPU communication limits.
- Scalability: clusters of tens of thousands of accelerators, connected so model parallelism can spread across them (arXiv survey, 2024).
- Efficiency: high Model FLOPs Utilization, which requires minimizing communication overhead, not just fast silicon.
- Reliability: weeks-to-months runs that detect and recover from failures without losing progress.
Power, cooling, network and accelerator choice
Balancing power, cooling, network topology and accelerator choice is critical to cluster design: the layout of the compute nodes is constrained by how they are connected to each other.
Specifying the cluster, step by step
Specify the scale-up domain first, then the scale-out fabric. Scale-up means accelerator-to-accelerator memory links inside a pod; scale-out means the network between pods. The published specifications, each with its date: UALink 200G 1.0 defines 200G per lane for up to 1,024 accelerators in an AI computing pod (UALink Consortium specification page, retrieved 8 September 2026)[3]. NVLink 5 with NVSwitch 4 can in theory span 576 GPUs in one shared-memory pod, but commercial support ships only up to 72 GPUs in the DGX B200 NVL72 (The Next Platform, 30 May 2024)[4]. Ultra Ethernet 1.0, published in 2025, reworks RDMA as a native Ethernet application running at 400 Gbps today with 800 Gbps near-term, designed for scaling over 1 million endpoints (Arista, July 2025)[5].
Vendors at each layer of the fabric
The table below names the providers active at each layer of the fabric stack, from switch silicon to optics. Cells without a dated public source read "not published". Detailed fabric-requirement criteria for sizing each layer sit in cluster networking requirements.
| Provider | Layer | Published specification involvement | Dated evidence |
|---|---|---|---|
| Arista Networks | Ethernet switching | Founding member, Ultra Ethernet Consortium | UEC 1.0 specification published; RDMA at 400 Gbps (blog, 16 Jul 2025) |
| Broadcom | Switch silicon | UALink promoter group; xGMI/Infinity Fabric on future PCIe switches | Promoter group member (Business Wire, 30 May 2024) |
| Marvell Technology | Custom AI silicon and optics | not published | not published |
| Alcatel Submarine Networks | Submarine cable systems | not published | not published |
| Ayar Labs | Optical I/O chiplets | not published | not published |
| Ciena | Optical networking | not published | not published |
| Cisco | Ethernet switching | UALink promoter group | Promoter group member (Business Wire, 30 May 2024) |
| Coherent | Optical components | not published | not published |
| Credo Technology | Connectivity silicon | not published | not published |
| Juniper Networks | Ethernet switching | not published | not published |
The recurring error on this topic: a specification revision or roadmap announcement is quoted as if it were a shipping product. Every figure in the table carries its unit, its date and its source page, and the distinction between specification and product is stated explicitly each time.
The decisions that are expensive to reverse
Three decisions dominate cost and are hard to reverse: scale-up domain size, fabric topology and switch radix. The RailX paper (arXiv, 2025) prices them from published sale prices: a GB200 NVL72 cabinet sold for about $3M against $2.16M for its 36 separate GB200 Superchips, so the scale-up network is roughly 30% of node cost, and a three-tier fat-tree adds further scale-out cost per node[7]. As a published research result, not a forecast, RailX interconnects 200K chips at 1.8 TB/s bandwidth for about $1.3B, at under 10% of fat-tree cost per unit of All-Reduce bandwidth[7].
Validating the design against published benchmarks
Validation before sign-off: check every claimed figure against the published specification revision, lane rate and switch radix on a dated page, and state the measurement basis (published revisions, lane rates, switch radix) in the briefing. No forecast is stated as a measurement in this article, and each figure above belongs to the named provider or instrument and the date given. This article's facts were last verified on 8 September 2026, and its next scheduled review is 8 March 2027.
Frequently asked questions
How many GPUs does an AI cluster need?
It depends on the workload, and the published reference point is large: pre-training LLaMA-3 took about 54 days on 16,000 H100-80GB GPUs on Meta's production cluster, per a 2024 arXiv survey. Fine-tuning and inference clusters are typically far smaller; the accelerator count should be derived from the model size and training-time budget, not from a vendor deck.
What is the difference between scale-up and scale-out networks in an AI cluster?
Scale-up (East/West) links accelerators into a shared-memory domain inside a pod. Scale-out links pods and nodes across the datacentre, today over Ethernet or InfiniBand at 400 Gbps. They are separate purchases with separate vendors.
Is Ultra Ethernet shipping yet?
The Ultra Ethernet Consortium published its 1.0 specification in June 2025, with over 100 member companies involved (Arista, 16 Jul 2025). The specification defines the Ultra Ethernet Transport with packet spraying, selective retransmission and new congestion control on standard Ethernet. Hardware implementations follow the spec; treat any product claim as shipping only when a dated product page says so.
How much of an AI cluster's cost is the network?
Published research gives order-of-magnitude shares. The RailX paper (arXiv, 2025) notes a GB200 NVL72 cabinet at about $3M against $2.16M for its 36 Superchips, putting the scale-up network near 30% of node cost, and estimates a three-tier fat-tree scale-out fabric adds further substantial cost. The same paper prices a circuit-switched rail design at under 10% of fat-tree cost per unit of All-Reduce bandwidth.
Which vendors supply AI cluster networking?
Suppliers span switching, silicon and optics: Arista Networks, Broadcom, Marvell Technology, Alcatel Submarine Networks, Ayar Labs, Ciena, Cisco, Coherent, Credo Technology and Juniper Networks. Check each vendor claim against a dated product page.
What is the most common mistake when specifying an AI cluster?
Quoting roadmap silicon as if it were shipping. Specifications exist on paper before products exist in racks. Every figure in a specification should carry its source page, its date and its status: specification revision, announced product, or shipping product.
Sources
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Networking →Datacenter water consumption: WUE and published figures
Datacenter water consumption explained: the WUE formula, withdrawal vs consumption, which providers publish figures, and the dated numbers a plan can defend.

Inference vs training: how the compute split is actually shifting
The inference vs training compute split: which publishers measure it, on what basis, and the one dated figure that survives a plan review.

The 70/20/10 and 30% rules for AI budgets: what they claim
What the 70/20/10 and 30% rules for AI budgets actually say, who published them, and which dated spending figures can replace them in a plan.
