DatacentersLatency regions

Is latency important for AI training: Network bottlenecks

Explore how tail latency acts as the primary bottleneck in distributed AI training and compare InfiniBand with tuned Ethernet for GPU networking.

, AI Researcher9 min read
In this article8
  1. The synchronization bottleneck in distributed AI training
  2. Synchronization amplification and GPU starvation
  3. InfiniBand versus Ethernet: The latency gap
  4. The role of Ultra Ethernet and advanced switching
  5. Configuring RoCE for lossless transport
  6. Decision criteria for the InfiniBand premium
  7. Frequently asked questions
  8. Sources
A dense optical patch panel with yellow fibre jumpers at an internet exchange
An optical patch panel at AMS-IX. In distributed training the fabric, not the accelerator, is usually what waits. Photo: Fabienne Serriere, CC BY-SA 3.0, via Wikimedia Commons

The synchronization bottleneck in distributed AI training

Distributed AI training splits compute jobs across hundreds or thousands of accelerator nodes in data-parallel and tensor-parallel configurations. The dominant primitive for synchronizing model weights and gradient updates across those devices is the All-Reduce collective operation.

Under the standard Bulk Synchronous Parallel (BSP) framework, every training iteration consists of a compute phase followed by a communication phase. Accelerators cannot begin the subsequent forward pass until all nodes have finished exchanging and aggregating their local gradients through All-Reduce. Research on distributed collective communication demonstrates that the execution time of synchronous All-Reduce operations is gated by the arrival of the slowest participating node or network packet[1]. Consequently, overall training cluster throughput is determined not by mean network latency, but by P99 and P99.9 tail latency across the fabric.

The mechanics of the All-Reduce barrier

Collective communication libraries implement All-Reduce through multi-stage schedules such as Ring All-Reduce or Recursive Halving and Doubling, partitioning gradient tensors into chunks that pass along network rings or tree topologies between adjacent nodes.

  1. Local forward and backward passes compute gradients across local accelerator memory.
  2. ReduceScatter phase splits gradient tensors into equal partitions and exchanges them across the cluster, leaving each node with a fully reduced slice.
  3. AllGather phase synchronizes the aggregated slices back to every node, reconstructing the complete updated model state.
  4. Synchronization barrier unblocks the compute engine once all packets arrive, allowing the next iteration to commence.

Because the synchronization barrier is binary, a single delayed packet caused by fabric congestion, transient buffer buildup, or routing skew stalls the entire collective operation. When thousands of nodes synchronize simultaneously, packet delay variation creates an immediate computing stoppage.

Synchronization amplification and GPU starvation

As compute clusters expand to thousands of accelerators, network delay characteristics scale non-linearly due to synchronization amplification. Across eight or sixteen nodes, packet arrival times stay relatively uniform; across thousands of endpoints, the probability that at least one packet hits queueing delay or transient path congestion approaches certainty on every training step.

When the slowest packet delays the collective communication barrier, all compute engines across the synchronization group are forced to wait. Broadcom's analysis of AI network design reports Meta's measurement that, on average, 30% of elapsed time in AI and machine learning workloads is spent waiting for the network[2]. This condition, known as GPU starvation, directly reduces effective hardware compute efficiency.

Financial and operational impact of compute idle time

GPU starvation carries direct economic and energetic costs for data center operators. High-density training clusters populated by top AI accelerators draw substantial baseline power even while idling at the synchronization barrier. Idling compute hardware consumes energy without advancing model convergence, prolonging training schedules and increasing overall operational expenditures.

Cluster Size (GPUs)Dominant Traffic Dynamic
64 to 256Intra-rack, low-hop communication
512 to 2,048Multi-tier leaf-spine fabric
4,096 to 24,000+Multi-layer multi-tier fabric with heavy incast

Mitigating synchronization amplification requires fabrics with deterministic packet delivery, minimal buffer queuing, and fast congestion response at every switch layer.

InfiniBand versus Ethernet: The latency gap

Distributed High Performance Computing (HPC) and AI training clusters have historically relied on InfiniBand, which combines cut-through switching, hardware credit-based flow control, and Remote Direct Memory Access (RDMA) primitives that bypass kernel overhead. Published fabric comparisons put InfiniBand NDR small-message latency at approximately 1 microsecond.[3]

The emergence of RDMA over Converged Ethernet version 2 (RoCEv2) has bridged the historical latency divide. By encapsulating InfiniBand transport packets inside standard UDP/IP Ethernet frames, RoCEv2 enables hardware-offloaded zero-copy memory transfers over standard data center switches. When properly tuned for lossless operation, modern RoCEv2 deployments achieve small-message latencies between 1.5 and 2.5 microseconds[3], delivering 85% to 95% of InfiniBand training throughput at significantly lower port infrastructure costs[4]. This makes evaluating Ethernet versus InfiniBand an essential trade-off for infrastructure procurement.

Architectural MetricInfiniBand (NDR 400G)RoCEv2 Ethernet (400G / 800G)
Small-Message LatencyApproximately 1 microsecond (NDR)1.5 to 2.5 microseconds
Effective Training ThroughputBaseline (100%)85% to 95% of baseline
Flow Control ArchitectureNative credit-based flow controlPriority Flow Control (PFC) and ECN
Routing MechanismFabric-managed deterministic routingECMP, flowlet switching, or adaptive routing
Ecosystem SourcingProprietary single-vendor hardwareOpen multi-vendor switch and optics supply

InfiniBand keeps a slight raw latency advantage for tightly coupled communication patterns; Ethernet answers with multi-vendor supply chains and integration with existing enterprise network monitoring.

The role of Ultra Ethernet and advanced switching

The Ultra Ethernet Consortium (UEC) released its 1.0 specification in June 2025, defining an Ethernet stack tailored to AI and HPC workloads[5]. Its architecture modernizes transport above the physical Ethernet layer, removing the structural weaknesses of legacy TCP/IP and conventional RoCE.

High-performance switch silicon, such as the Tomahawk and Jericho processor families developed by Broadcom, incorporates hardware telemetry and dynamic load balancing to eliminate tail latency spikes. Rather than relying on static Equal-Cost Multi-Path (ECMP) hashing, which frequently assigns multiple large elephant flows to the same uplink, advanced switch silicon employs adaptive routing and fine-grained packet spraying.

Ultra Ethernet Transport innovations

The Ultra Ethernet Transport (UET) specification replaces rigid packet-ordering constraints with flexible delivery models designed for machine learning collectives[6]. Because All-Reduce and All-to-All operations require only the complete arrival of bulk data buffers rather than sequential packet ordering, UET enables multi-path packet spraying directly across all available fabric links without requiring large destination re-order buffers.

  • Multi-path packet spraying: Flows are distributed across all active fabric links at the packet level, eliminating switch hash collisions and maximizing bisection bandwidth.
  • Reliable Unordered Delivery (RUD): The default bulk transport mode, in which each packet carries full destination addressing and is written out of order directly into the destination buffer, removing receive-side reordering overhead[6].
  • Advanced in-network telemetry: Switches truncate congested packets and forward header telemetry to endpoints, accelerating rate throttling adjustments.
  • Selective acknowledgments: Senders retransmit only lost packets rather than executing heavyweight Go-Back-N window retransmissions.

These capabilities reduce tail latency variance, bringing Ethernet determinism close to dedicated HPC interconnects.

Configuring RoCE for lossless transport

Standard Ethernet operates on a best-effort delivery model, dropping packets whenever switch port buffers saturate. Because RDMA primitives lack the native retransmission mechanisms found in TCP, an unhandled packet drop triggers a severe Go-Back-N recovery sequence that stalls GPU communication for milliseconds. Operating RoCEv2 reliably therefore requires lossless Ethernet transport across every switch and NIC.

Lossless RoCEv2 depends on the coordinated implementation of Priority Flow Control (PFC, IEEE 802.1Qbb) and Explicit Congestion Notification (ECN) managed by Data Center Quantized Congestion Notification (DCQCN). PFC operates as a link-level safety mechanism, enabling a downstream switch to transmit pause frames on a dedicated traffic priority class when buffer thresholds are crossed, preventing drops while allowing background traffic to flow unimpeded.

Mitigating PFC deadlocks and buffer exhaustion

If PFC is configured without precise congestion signaling, pause frames can cascade backward through the switch topology, generating fabric-wide PFC deadlocks and pause storms that freeze training traffic. Production deployments prevent this condition by coupling PFC with proactive ECN marking.

  • ECN minimum and maximum thresholds: Configured on switch egress queues to mark packets with Congestion Experienced (CE) flags before buffer capacity reaches PFC trigger levels.
  • DCQCN rate adjustments: Destination endpoints generate Congestion Notification Packets (CNPs) upon receiving CE flags, instructing sending NICs to reduce injection rates smoothly.
  • Deep buffer allocations: Deploying leaf and spine switches with enough shared packet buffer to absorb synchronized All-Reduce incast bursts.
  • PFC watchdog timers: Automatic detection and recovery routines that disable flow control on misbehaving queues to prevent cluster deadlock.

Implementing these configurations establishes stable transport behavior aligned with rigorous cluster networking requirements, minimizing tail latency during heavy collective synchronization. This tuning burden is also why some training teams never touch the fabric at all: managed cluster products absorb it.

Decision criteria for the InfiniBand premium

Selecting between InfiniBand and RoCEv2 Ethernet means weighing workload communication profiles against capital cost. Network latency investments pay off only when communication overhead is a material fraction of total iteration time.

When communication overhead accounts for less than 20% of iteration time, the training job is compute-bound[3]. In these scenarios, further reductions in fabric latency produce diminishing returns on total job completion time. Conversely, when large model architectures require frequent inter-node tensor parallel communication or extensive pipeline bubbles, communication overhead rises, increasing sensitivity to fabric latency.

Infrastructure evaluation matrix

The trade-offs resolve differently by cluster size and workload profile.

Cluster Scale & Workload ProfileRecommended Network FabricKey Economic & Technical Drivers
Up to 512 GPUs, standard data parallelismRoCEv2 EthernetLow communication overhead; standard Ethernet hardware minimizes capital cost and leverages existing networking staff.
Mid-scale clusters, mixed parallelismRoCEv2 Ethernet or InfiniBandTuned RoCEv2 provides 85% to 95% of InfiniBand throughput at significantly lower cost.
2,048+ GPUs, frontier model trainingInfiniBand (or optimized UEC Ethernet)Extreme scale amplifies tail latency; InfiniBand justifies its cost premium when training time directly dictates commercial release schedules.
Multi-tenant enterprise AI clusterRoCEv2 EthernetFull support for VXLAN EVPN overlays and converged storage workloads across shared data center infrastructure.

For the majority of enterprise deployments scaling below 2,048 GPUs, properly tuned RoCEv2 Ethernet offers the optimal balance of throughput, cost efficiency, and infrastructure flexibility, while InfiniBand remains the dedicated choice for extreme-scale foundation model development where minimizing job completion time outweighs capital premiums.

Frequently asked questions

Why is tail latency critical for AI training clusters?

In distributed AI training, GPUs rely on collective communication operations like All-Reduce to synchronize gradients. Because the training loop cannot proceed to the next iteration until every single GPU has reported back, the entire cluster must wait for the slowest packet in the network. This makes tail latency the primary factor determining job completion time.

How much compute time is lost to network bottlenecks in AI training?

Unoptimized networks can cause severe synchronization amplification, Broadcom, citing Meta's measurement, reports that on average 30% of elapsed time in AI and machine learning workloads is spent waiting for the network. This starvation occurs when the compute elements finish their tasks but are forced to wait for delayed network packets to arrive before beginning the next forward pass.

What is the latency difference between InfiniBand and Ethernet for AI clusters?

InfiniBand typically delivers small-message latency of around 1 microsecond. Modern Ethernet, when properly configured with RoCE (RDMA over Converged Ethernet) and advanced congestion control, can achieve latency between 1.5 and 2.5 microseconds. This allows tuned Ethernet to deliver 85% to 95% of the training throughput of InfiniBand.

What is the Ultra Ethernet Consortium (UEC)?

The Ultra Ethernet Consortium is an industry body specifying Ethernet standards optimized specifically for AI and HPC workloads. Its specifications define standardized congestion signaling, transport protocols, and telemetry designed to handle the massive burst traffic patterns generated by GPU-to-GPU gradient synchronization.

When should an enterprise pay the premium for InfiniBand?

InfiniBand is generally recommended for very large deployments, typically exceeding 2048 GPUs, where small latency improvements compound significantly. It is also justified if workload profiling shows that communication overhead accounts for more than 20% of the total training time, meaning the system is severely network-bound rather than compute-bound.

How does adaptive routing improve RoCE performance?

Standard Ethernet routing can cause specific network links to become congested while others remain idle. Adaptive routing monitors per-path congestion in real-time and dynamically reroutes traffic around hotspots within microseconds. This eliminates head-of-line blocking and ensures consistent packet delivery times, which is critical for minimizing tail latency in AI fabrics.

Sources

  1. Efficient AllReduce with Stragglers arxiv.org
  2. Optimizing the network for AI workloads broadcom.com
  3. InfiniBand vs Ethernet for AI Clusters: GPU Networks vitextech.com
  4. RoCE vs InfiniBand for AI Data Center Networking: What Network Engineers Need to Know in 2026 firstpasslab.com
  5. UEC Launches Spec 1.0 Transforming Ethernet for AI and HPC at Scale linuxfoundation.org
  6. Ultra Ethernet’s Design Principles and Architectural Innovations arxiv.org

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Datacenters →

The InfiniBand premium, priced out

Compare InfiniBand vs Ethernet for AI training cost and performance. Evaluate latency, effective bandwidth and when to choose RoCEv2 for your GPU cluster.

A switch rack at the DE-CIX internet exchange in Frankfurt