AcceleratorsAvailability

GPU failure rate: data center statistics and node reliability

Discover the real-world failure rate of data center GPUs based on published cluster studies, from H100 hardware interruptions to the impact on training MTTF.

, AI Researcher8 min read
In this article6
  1. How often data center GPUs actually fail
  2. Published failure and interruption rates
  3. What operators report from large clusters
  4. Why vendor RMA data stays unpublished
  5. Sizing spare capacity and job checkpoints
  6. Sources
A large-scale data center server rack with close-up focus on a row of high-performance GPUs, some indicating hardware fault status.
A large-scale data center server rack with close-up focus on a row of high-performance GPUs, some indicating hardware fault status.

How often data center GPUs actually fail

No vendor publishes a failure rate for its data center GPUs, so the defensible figures come from operator telemetry rather than marketing claims. The largest published dataset is Meta's 11-month study of two research clusters totalling 24,000 NVIDIA A100 GPUs, covering more than 150 million GPU hours and 4 million jobs, published on arXiv on 29 October 2024.[1] On that data, hardware-attributed failures affected fewer than 1% of jobs but about 19% of GPU runtime, because a single failed component can idle thousands of GPUs in a gang-scheduled job.[1]

Three metrics carry the answer, and each answers a different question. Annualised failure rate (AFR) is the share of installed units that fail in a year. Mean time to failure (MTTF) is the average runtime a job of a given size achieves before a component failure interrupts it. Effective Training Time Ratio (ETTR) is the share of wallclock time a job spends training productively rather than restarting or re-running from its last checkpoint. The same hardware produces very different MTTF numbers depending on job size, which is why the figure belongs in a plan as a function of scale, not as a single constant.

Job size (NVIDIA A100, Meta research clusters)Mean time to failure, measured or projected
8 GPUs47.7 days
1,024 GPUs7.9 hours
16,384 GPUs1.8 hours (projected)
131,072 GPUs0.23 hours (projected)

The scaling law is the operative finding. Meta's fitted model shows hardware reliability shrinking inversely with the number of GPUs, with consistent trends from 32 GPUs upward, so a 16,384-GPU job is projected to meet a failure roughly every 1.8 hours. That is a model projection; the Llama 3 run below measured roughly one unexpected failure every three hours at the same scale. The measured failure rate itself is not constant either: the study records roughly 2.5 failures per node-day in normal periods, spiking to about 17.5 per node-day, an order-of-magnitude swing that any single annualised number conceals.[1] The A100 and H100 silicon behind these studies has since been succeeded by the H200 and Blackwell parts, but the failure behaviour of memory, PCIe and interconnect components scales the same way as cluster size grows for H100 successors.

Published failure and interruption rates

The clearest published record for a single training run is the Llama 3 405B pre-training report, published on arXiv on 23 July 2024.[2] Over a 54-day snapshot on up to 16,384 H100 GPUs, the cluster recorded 466 job interruptions: 47 planned, for maintenance and configuration changes, and 419 unexpected, an average of one unexpected failure roughly every three hours.[2] More than half of the unexpected interruptions were caused by issues with faulty GPUs or their onboard HBM3 memory, and despite this the run maintained more than 90% effective training time.[3]

Component category (54-day Llama 3 405B run)Interruptions
Faulty GPU148
GPU HBM3 memory72
Network switch or cable35
Unplanned host maintenance32
GPU SRAM memory19
GPU system processor17
CPU2

The measurement basis is observed failure counts over a dated window, not a vendor projection, and the two must not be confused. GPU throughput figures from specification sheets and benchmark lineage, such as the FlashAttention work on fast, memory-efficient exact attention, describe what hardware does when it is healthy; interruption tables describe what happens to it under sustained load. Independent coverage of the same Meta report summarised the finding the same way: more than half of the 419 unexpected interruptions were caused by issues with faulty GPUs or their onboard HBM3 memory, an average of one failure every three hours across the 16,384-GPU cluster.[3]

What operators report from large clusters

The symptom pattern is consistent across the studies that publish data. User-level errors such as out-of-memory conditions are frequent but contained, because a failed small job costs only its own runtime. The interruptions that destroy progress at scale are hardware-attributed, and three component families dominate: GPU memory, the PCIe host bus, and the interconnect fabric that ties the GPUs together.

  • GPU memory: HBM3 errors produced 72 of the 419 unexpected Llama 3 interruptions (17.2%), with a further 19 from GPU SRAM.[3]
  • PCIe bus errors: on Meta's RSC-1 cluster, 43% of PCI errors co-occurred with XID 79, the code for a GPU falling off the bus, and 63% did on RSC-2, indicating the two signals usually describe the same fault.[1]
  • Interconnect faults: on the Delta system, NVLink errors on A100 nodes frequently preceded job failures, while the H100 nodes recorded none in their measurement period, a generational difference the study attributes to the interconnect design.
  • Memory recovery failure: the study concludes that the GPU memory error-recovery mechanisms on H100 GPUs are insufficient to handle the increased memory capacity, indicating the spare-row mechanism strains as HBM capacity grows.[4]

The longest-running comparative dataset, 2.5 years and 11.7 million GPU-hours on Delta's A100 and H100 nodes, puts numbers on the generational trend. H100 GPU memory resilience is worse than A100 GPU memory, with 3.2x lower per-GPU mean time between errors for memory errors, and the memory error-recovery mechanisms on H100 GPUs are insufficient to handle the increased memory capacity, though recovery mechanisms still contain the large majority of uncorrectable memory errors on H100.[4] GPU errors on both A100 and H100 GPUs frequently result in job failures due to the lack of robust recovery mechanisms at the application level, which is why large jobs feel failures that small jobs never notice.[4]

Why vendor RMA data stays unpublished

A return merchandise authorisation (RMA) is the process by which a failed unit is sent back for repair or replacement, and its volume is the closest thing to a true component defect rate. Neither GPU manufacturers nor hyperscalers publish raw RMA counts. The figures vendors do publish, quarterly data-centre revenue and specification sheets, are structured so that reliability cannot be derived from them, and no accelerator vendor's public pages state a failure rate, an MTTF or an RMA rate for its data centre products.

The comparison below covers the accelerator vendors a buyer on this question would shortlist, from AMD and Axelera AI through to Huawei and NVIDIA, the same field covered in our comparisons of the strongest data center GPUs and of the GPUs that succeed the H100.

ProviderPublished failure rate or RMA dataPublished MTTF or MTBE figure
AMD––
Axelera AI––
Cerebras Systems––
Etched––
FuriosaAI––
Graphcore––
Groq––
Huawei––
Intel––
NVIDIA––

A dash means the provider's own published pages state no value for that cell as of the 23 September 2026 check. In the absence of vendor data, the industry reads service life from supercomputer and research-cluster telemetry. As noted in a September 2026 infrastructure analysis, those measurements point to a useful life of one to three years under continuous high utilisation, driven by thermal and electrical stress, which is far shorter than the five-to-six-year depreciation periods most operators book.[5] The observable market signal that replaces missing RMA data is rental pricing: according to the SemiAnalysis GPU Pricing Index in early 2026, one-year H100 contract rates rose from $1.70 per GPU-hour in October 2025 to $2.35 by March 2026, a rise of almost 40%, with on-demand capacity sold out across GPU types.[6][7]

Sizing spare capacity and job checkpoints

Because MTTF falls in inverse proportion to job size, failure frequency is an operational design input, not an afterthought. The Delta study projects that holding 99.9% job-level availability on a 608-GPU, one-month training run requires 5% overprovisioning, 31 additional GPUs beyond the original 608, and that faster recovery to five minutes would cut the requirement to 2%, a 2.5-times reduction. Per-node availability of roughly 99.3% to 99.4% is the measured baseline that overprovisioning has to absorb.

  • Checkpoint frequently: each GPU's model state ran from 1 MB to 4 GB on the Llama 3 run, written with pause time minimised, so an interruption costs minutes of re-training rather than days.
  • Diagnose fast: Meta's NCCL flight recorder captures collective metadata in a ring buffer so hangs are localised quickly, and of 419 unexpected interruptions only three required significant manual intervention.
  • Hold spare capacity: budget the 5% overprovisioning that measured failure rates imply, and note that rental markets are tight, with on-demand capacity sold out across GPU types in early 2026.[8]
  • Contract for impairment, not just capacity: AWS states that where it detects impairment affecting a reserved ultraserver capacity block it will notify the customer but generally will not terminate instances, so the replacement path sits with the buyer.[9]
  • Compare contract structures: short-term cluster rental, committed-use pricing at Google Cloud, and reserved blocks each shift spare-capacity risk between buyer and provider.

For a reader sizing a market or a budget, the defensible position is three-part. State the interruption figure with its job size, window and date: 419 unexpected failures in 54 days on 16,384 H100s, reported July 2024. Separate observed from projected: the 1.8-hour MTTF at 16,384 GPUs is a model fit, not a measurement, and the Delta study's 99.3% node availability is a measurement. Then price the mitigation: spare capacity, checkpoint storage and recovery tooling are the line items that convert a failure rate into an ETTR above 90%, and short-term cluster rental from providers such as Lambda (whose September 2026 pricing page lists 1-Click Clusters running from 16 to 2,000-plus GPUs) prices the alternative to owning that spare capacity.[10]

Sources

  1. arxiv.org
  2. ar5iv.labs.arxiv.org
  3. datacenterdynamics.com
  4. arxiv.org
  5. optinest.de
  6. semianalysis.com
  7. gpu-index.semianalysis.com
  8. newsletter.semianalysis.com
  9. aws.amazon.com
  10. lambda.ai

Rankings and scores on Optinest cannot be bought — no commercial relationship changes how a provider is assessed. How we score

Related reading

All Accelerators →