AI Training Hardware: Cluster Networking Requirements
Compare InfiniBand and Ethernet networking hardware for AI training clusters, including throughput, switch topologies, and emerging UALink standards.
In this article9
- AI Training Hardware: Cluster Networking Requirements
- Why cluster networking limits AI training scale
- What hardware is needed for AI training
- InfiniBand capabilities for multi-node clusters
- Ethernet evolution and the Ultra Ethernet standard
- Accelerator communication with the UALink standard
- Evaluating InfiniBand vs Ethernet: Network comparison criteria
- Frequently asked questions
- Sources

AI Training Hardware: Cluster Networking Requirements
Why cluster networking limits AI training scale
When scaling deep learning workloads beyond a single server node, the primary hardware bottleneck transitions from GPU computing capacity to the network fabric interconnect. Inside a standard 8-way accelerator chassis, high-speed board-level interconnects handle peer-to-peer memory transfers. When training state-of-the-art transformer models across multi-node infrastructure, thousands of discrete processors must synchronize gradient parameters during every backward pass[1]. Without sufficient inter-node bandwidth and low path latency, high-performance data center GPUs sit idle while waiting for network operations to finish.
Distributed model training relies on collective communication primitives such as All-Reduce, All-Gather, and Reduce-Scatter to update weights across distributed compute nodes[1]. As model sizes reach hundreds of billions of parameters, the payload generated during each synchronization phase scales proportionally. A 175-billion parameter model in 16-bit precision requires exchanging approximately 350 gigabytes of gradient data per iteration. If the network fabric cannot process these transfers at line rate, communication time dominates compute time, leading to a severe drop in scaling efficiency.
- Collective Operation Overhead: Synchronizing gradients across nodes creates synchronized traffic spikes that saturate link capacity[1].
- Tail Latency Sensitivity: The overall duration of an All-Reduce phase is bounded by the slowest single network path in the cluster.
- Hardware Underutilization: Inadequate fabric bandwidth reduces Model Flops Utilization (MFU), increasing total GPU operational hours.
What hardware is needed for AI training
Artificial intelligence model training has shifted data centre compute requirements from general-purpose CPUs toward highly parallel GPU architectures. Training foundation models requires executing billions of dense matrix multiplications simultaneously across deep neural network layers. Standard server CPUs, optimized for sequential execution and low-latency single-threaded instruction processing, lack the memory throughput and parallel compute cores necessary to process modern model weights efficiently.
For context, Meta's Llama 3 405B training run took place on a cluster of over 16,000 Nvidia H100 GPUs (Meta says 'up to 16K'), with published reliability data covering a 54-day snapshot of pre-training[3]. At this scale, the aggregate memory across accelerator memory chips and system RAM exceeds hundreds of terabytes, requiring continuous synchronization across all compute units.
- Model Weight Footprint: A 405-billion parameter model in 16-bit floating point precision (FP16) requires 810 GB of memory just to store unquantized model weights before allocating optimizer states.
- Optimizer State Scaling: Standard AdamW optimizers maintain 12 bytes of state per parameter, adding 4.86 TB of required memory footprint across the cluster.
- Activation Memory: Long context windows generate multi-terabyte intermediate activation tensors during forward and backward passes, which must be held in accelerator memory or recomputed.
To prevent memory starvation, enterprise hardware uses High-Bandwidth Memory (HBM) stacked directly on the accelerator substrate. The NVIDIA H200 GPU, for example, integrates 141 GB of HBM3e memory delivering 4.8 TB/s of aggregate memory bandwidth[5]. In an eight-way HGX server configuration, this architecture provides 1.1 TB of aggregate high-bandwidth memory, allowing engineering teams to keep larger model partitions resident in local VRAM without resorting to off-chip swapping.
InfiniBand capabilities for multi-node clusters
InfiniBand has long served as the dominant interconnect technology for high-performance computing and large-scale AI clusters[2]. Purpose-built for low-latency, high-throughput environments, InfiniBand employs a credit-based flow control mechanism that guarantees lossless packet delivery at the hardware level. This design prevents packet drops, eliminating the latency penalties associated with retransmission protocols.
The core architectural advantage of InfiniBand lies in its native support for Remote Direct Memory Access (RDMA). RDMA enables network adapters to transfer data directly between GPU memories across nodes without involving operating system kernels or host CPU cycles[2]. In Next Data Rate (NDR) InfiniBand deployments operating at 400 Gb/s per port, end-to-end switch latency drops to approximately 1 microsecond. This predictable latency profile makes InfiniBand ideal for tightly coupled training tasks.
| Specification Metric | InfiniBand HDR | InfiniBand NDR |
|---|---|---|
| Port Bandwidth | 200 Gb/s | 400 Gb/s |
| Switch Latency | Approximately 1.3 microseconds | Approximately 1.0 microsecond |
| Fabric Management | Centralized Subnet Manager | Centralized Subnet Manager |
| Flow Control Protocol | Hardware Credit-Based | Hardware Credit-Based |
By bypassing the host networking stack and providing deterministic packet delivery, InfiniBand minimizes communication tail latency across multi-node clusters network fabrics. This reliability ensures that distributed training jobs maintain high GPU compute efficiency across thousands of connected chips.
Ethernet evolution and the Ultra Ethernet standard
Traditional Ethernet was originally designed for general-purpose enterprise networks where packet drop handling and asynchronous traffic flow are standard. When applied to AI training clusters via Remote Direct Memory Access over Converged Ethernet (RoCE), standard Ethernet requires complex Priority Flow Control (PFC) configurations to avoid packet loss. Under heavy, synchronized AI training traffic, PFC can trigger congestion spreading and deadlocks, degrading total cluster throughput.
To address these structural limitations, industry leaders formed the Ultra Ethernet Consortium (UEC) to build an open, high-performance protocol stack optimized for AI and high-performance computing[4]. The UEC 1.0.3 specification introduces multipath transport, flexible packet ordering, and advanced congestion management algorithms that adjust transmission rates in real time without relying on coarse pause frames[4].
- Packet-Granular Spraying: Transmits individual packets across all available network paths simultaneously to eliminate link hotspotting[4].
- Selective In-Network Retransmission: Recovers missing packets individually without dropping or resending an entire frame sequence[4].
- Modern Congestion Control: Fast-reacting algorithms measure precise queuing delays to dynamically throttle flow rates before buffers overflow[4].
- Ubiquitous Ecosystem Integration: Maintains physical Ethernet compatibility while delivering transport performance tailored for distributed AI[4].
Accelerator communication with the UALink standard
While InfiniBand excels in scale-out network topologies connecting discrete server nodes, specialized scale-up protocols govern communication between accelerators inside the server chassis. To establish an open standard for scale-up links, industry leaders formed the UALink Consortium, whose 200G 1.0 specification defines a low-latency, high-bandwidth interconnect for communication between accelerators and switches in AI computing pods[8].
- High Lane Bandwidth: Supports 200 Gb/s physical lane rates for high-throughput memory sharing[6].
- Scale-Up Memory Pooling: Enables low-latency direct addressing across up to 1,024 accelerator chips in a single logical pod[6].
- Open Governance: A multi-vendor industry consortium board guides specification development[6].
Open specifications like UALink, developed by a consortium whose board includes Alibaba, AMD, Apple, Astera Labs, AWS, Cisco, Google, HPE, Intel, Meta, Microsoft and Synopsys[9], target accelerator-to-accelerator communication as an alternative to proprietary scale-up links. That lets hardware architects decouple internal accelerator topologies from single-vendor chip sets, expanding supply chain options for custom AI server node construction.
Evaluating InfiniBand vs Ethernet: Network comparison criteria
InfiniBand in its NDR (Quantum-2) generation provides dedicated per-port throughput over OSFP connectors, utilizing cut-through forwarding and hardware-based remote direct memory access (RDMA) to achieve ultra-low sub-microsecond latencies. Conversely, the Ultra Ethernet Consortium released its 1.0 specification in June 2025 to deliver an open, interoperable Ethernet communication stack specifically optimized for AI and high-performance computing at scale[7]. Evaluating Ethernet vs InfiniBand requires balancing open multi-vendor software standards against predictable fabric latencies.
When comparing Ethernet vs InfiniBand, infrastructure decision-makers evaluate raw bandwidth, tail latency, ecosystem interoperability, and long-term operating costs. While InfiniBand historically dominated top-tier AI training sites, market research from 650 Group projects that 91% of AI workloads will run on Ethernet fabrics by 2029[10].
| Criterion | NVIDIA Quantum-2 InfiniBand NDR | Ultra Ethernet Consortium (UEC) Standard |
|---|---|---|
| Peak Port Throughput | 400 Gb/s per port (NDR) | Up to 800 Gb/s / 1.6 Tb/s roadmaps |
| Primary Fabric Protocol | InfiniBand Architecture (IBTA) | Enhanced UEC Ethernet transport stack |
| Congestion Control | In-Network Computing (SHARP) | Advanced multi-pathing and packet trimming |
| Standards Governance | Proprietary / Industry Ecosystem | Open Consortium (UEC Specification 1.0.3) |
Frequently asked questions
What hardware is required for AI training?
Training large AI models requires a distributed cluster of AI accelerators, such as GPUs, connected by a high-bandwidth network fabric. Meta's Llama 3 405B run, for example, used a cluster of over 16,000 Nvidia H100 GPUs (Meta says 'up to 16K') working in parallel.
How do GPUs connect in an AI training cluster?
GPUs in a cluster connect via scale-out networks, typically using InfiniBand or specialized Ethernet. These fabrics manage the constant exchange of gradients and activations across thousands of nodes during heavily network-bound all-reduce operations.
What is Ultra Ethernet?
The Ultra Ethernet Consortium develops standards to optimize Ethernet for AI and HPC networking. Its Ultra Ethernet Transport introduces a clean-slate RDMA transport, packet spraying across multiple paths, and congestion control aimed at reducing tail latency in demanding AI workloads.
What is the UALink Consortium?
The Ultra Accelerator Link Consortium develops open industry interconnect standards for AI accelerator-to-accelerator communication. Its 200G 1.0 specification enables 200G per lane scale-up connectivity for up to 1,024 accelerators inside an AI computing pod.
Is InfiniBand or Ethernet better for AI training?
InfiniBand traditionally offers the lowest latency and lossless transmission for AI training. However, analysts project that 91% of AI workloads will run on Ethernet by 2029 as emerging standards close the performance gap and lower deployment complexity.
Sources
- When Scaling Fails: Network and Fabric Effects on Distributed GPU Training Performance arxiv.org
- Meta's huge 16,384 NVIDIA H100 AI GPU cluster: HBM3 memory crashed half of Llama 3 training tweaktown.com
- ultraethernet.org
- NVIDIA H200 GPU nvidia.com
- ualinkconsortium.org
- Ultra Ethernet Has Arrived: One Network to Rule Them All? hpcwire.com
- businesswire.com
- InfiniBand vs Ethernet - Why Ethernet fits AI Networking needs drivenets.com
Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score
Related reading
All Networking →Ethernet closes in on InfiniBand in the AI fabric race
Compare AI training network fabrics: evaluate InfiniBand NDR against emerging Ultra Ethernet Consortium standards for large-scale GPU clusters.

Is latency important for AI training: Network bottlenecks
Explore how tail latency acts as the primary bottleneck in distributed AI training and compare InfiniBand with tuned Ethernet for GPU networking.

The InfiniBand premium, priced out
Compare InfiniBand vs Ethernet for AI training cost and performance. Evaluate latency, effective bandwidth and when to choose RoCEv2 for your GPU cluster.
