NetworkingFabric

How to build an AI cluster: the fabric numbers that matter

How to build an AI cluster: the dated fabric and accelerator numbers, the ten-vendor field, and which decisions are expensive to reverse.

, AI Researcher6 min read
In this article8
  1. What an AI cluster has to deliver
  2. Power, cooling, network and accelerator choice
  3. Specifying the cluster, step by step
  4. Vendors at each layer of the fabric
  5. The decisions that are expensive to reverse
  6. Validating the design against published benchmarks
  7. Frequently asked questions
  8. Sources
Abstract diagram of an AI training cluster: racks of GPU nodes connected by a leaf-spine network fabric, with separate scale-up links inside racks and scale-out links between them, rendered in a sober technical illustration style.
Abstract diagram of an AI training cluster: racks of GPU nodes connected by a leaf-spine network fabric, with separate scale-up links inside racks and scale-out links between them, rendered in a sober technical illustration style.

What an AI cluster has to deliver

The workload sets the specification. A 2024 survey of distributed LLM training on arXiv records the anchor figure: LLaMA-3 pre-training took about 54 days on 16,000 H100-80GB GPUs on Meta's production cluster[1]. The same survey frames the requirements as SER: scalability to tens of thousands of accelerators, efficiency measured as Model FLOPs Utilization, and reliability, because training runs last weeks to months and must survive hardware, network and software failures without losing progress. That makes failure recovery a design input, not an afterthought.

The fabric a cluster must carry is published in NVIDIA's HGX AI Factory reference architecture. It separates four roles: an RDMA-based GPU compute East/West fabric in a rail-optimized leaf-spine topology; a converged North/South network for compute, storage, in-band management and user traffic, with two 400 GbE ports per node for redundancy; storage connectivity; and low-cost 1 Gb/s out-of-band management for baseboard controllers and switches. The measurement basis is stated explicitly: the architecture's 4-node scalable unit carries 64 × 400 Gb/s connections, an aggregate 25.6 Tb/s of compute-fabric bandwidth, retrieved 8 September 2026[2]. How the bandwidth is consumed inside a node and across nodes is covered in how GPUs communicate in a training cluster and in GPU communication limits.

  • Scalability: clusters of tens of thousands of accelerators, connected so model parallelism can spread across them (arXiv survey, 2024).
  • Efficiency: high Model FLOPs Utilization, which requires minimizing communication overhead, not just fast silicon.
  • Reliability: weeks-to-months runs that detect and recover from failures without losing progress.

Power, cooling, network and accelerator choice

Balancing power, cooling, network topology and accelerator choice is critical to cluster design: the layout of the compute nodes is constrained by how they are connected to each other.

Specifying the cluster, step by step

Specify the scale-up domain first, then the scale-out fabric. Scale-up means accelerator-to-accelerator memory links inside a pod; scale-out means the network between pods. The published specifications, each with its date: UALink 200G 1.0 defines 200G per lane for up to 1,024 accelerators in an AI computing pod (UALink Consortium specification page, retrieved 8 September 2026)[3]. NVLink 5 with NVSwitch 4 can in theory span 576 GPUs in one shared-memory pod, but commercial support ships only up to 72 GPUs in the DGX B200 NVL72 (The Next Platform, 30 May 2024)[4]. Ultra Ethernet 1.0, published in 2025, reworks RDMA as a native Ethernet application running at 400 Gbps today with 800 Gbps near-term, designed for scaling over 1 million endpoints (Arista, July 2025)[5].

Vendors at each layer of the fabric

The table below names the providers active at each layer of the fabric stack, from switch silicon to optics. Cells without a dated public source read "not published". Detailed fabric-requirement criteria for sizing each layer sit in cluster networking requirements.

ProviderLayerPublished specification involvementDated evidence
Arista NetworksEthernet switchingFounding member, Ultra Ethernet ConsortiumUEC 1.0 specification published; RDMA at 400 Gbps (blog, 16 Jul 2025)
BroadcomSwitch siliconUALink promoter group; xGMI/Infinity Fabric on future PCIe switchesPromoter group member (Business Wire, 30 May 2024)
Marvell TechnologyCustom AI silicon and opticsnot publishednot published
Alcatel Submarine NetworksSubmarine cable systemsnot publishednot published
Ayar LabsOptical I/O chipletsnot publishednot published
CienaOptical networkingnot publishednot published
CiscoEthernet switchingUALink promoter groupPromoter group member (Business Wire, 30 May 2024)
CoherentOptical componentsnot publishednot published
Credo TechnologyConnectivity siliconnot publishednot published
Juniper NetworksEthernet switchingnot publishednot published

The recurring error on this topic: a specification revision or roadmap announcement is quoted as if it were a shipping product. Every figure in the table carries its unit, its date and its source page, and the distinction between specification and product is stated explicitly each time.

The decisions that are expensive to reverse

Three decisions dominate cost and are hard to reverse: scale-up domain size, fabric topology and switch radix. The RailX paper (arXiv, 2025) prices them from published sale prices: a GB200 NVL72 cabinet sold for about $3M against $2.16M for its 36 separate GB200 Superchips, so the scale-up network is roughly 30% of node cost, and a three-tier fat-tree adds further scale-out cost per node[7]. As a published research result, not a forecast, RailX interconnects 200K chips at 1.8 TB/s bandwidth for about $1.3B, at under 10% of fat-tree cost per unit of All-Reduce bandwidth[7].

Validating the design against published benchmarks

Validation before sign-off: check every claimed figure against the published specification revision, lane rate and switch radix on a dated page, and state the measurement basis (published revisions, lane rates, switch radix) in the briefing. No forecast is stated as a measurement in this article, and each figure above belongs to the named provider or instrument and the date given. This article's facts were last verified on 8 September 2026, and its next scheduled review is 8 March 2027.

Frequently asked questions

How many GPUs does an AI cluster need?

It depends on the workload, and the published reference point is large: pre-training LLaMA-3 took about 54 days on 16,000 H100-80GB GPUs on Meta's production cluster, per a 2024 arXiv survey. Fine-tuning and inference clusters are typically far smaller; the accelerator count should be derived from the model size and training-time budget, not from a vendor deck.

What is the difference between scale-up and scale-out networks in an AI cluster?

Scale-up (East/West) links accelerators into a shared-memory domain inside a pod. Scale-out links pods and nodes across the datacentre, today over Ethernet or InfiniBand at 400 Gbps. They are separate purchases with separate vendors.

Is Ultra Ethernet shipping yet?

The Ultra Ethernet Consortium published its 1.0 specification in June 2025, with over 100 member companies involved (Arista, 16 Jul 2025). The specification defines the Ultra Ethernet Transport with packet spraying, selective retransmission and new congestion control on standard Ethernet. Hardware implementations follow the spec; treat any product claim as shipping only when a dated product page says so.

How much of an AI cluster's cost is the network?

Published research gives order-of-magnitude shares. The RailX paper (arXiv, 2025) notes a GB200 NVL72 cabinet at about $3M against $2.16M for its 36 Superchips, putting the scale-up network near 30% of node cost, and estimates a three-tier fat-tree scale-out fabric adds further substantial cost. The same paper prices a circuit-switched rail design at under 10% of fat-tree cost per unit of All-Reduce bandwidth.

Which vendors supply AI cluster networking?

Suppliers span switching, silicon and optics: Arista Networks, Broadcom, Marvell Technology, Alcatel Submarine Networks, Ayar Labs, Ciena, Cisco, Coherent, Credo Technology and Juniper Networks. Check each vendor claim against a dated product page.

What is the most common mistake when specifying an AI cluster?

Quoting roadmap silicon as if it were shipping. Specifications exist on paper before products exist in racks. Every figure in a specification should carry its source page, its date and its status: specification revision, announced product, or shipping product.

Sources

  1. arxiv.org
  2. docs.nvidia.com
  3. ualinkconsortium.org
  4. nextplatform.com
  5. blogs.arista.com
  6. businesswire.com
  7. arxiv.org

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Networking →