At cluster scale, the data centre stops being infrastructure and becomes part of the system you are debugging
There is a level of scale at which distributed training stops behaving like software running on hardware and starts behaving like a single physical machine with tens of thousands of moving parts. Most of the engineering attention goes to the parts you can see in the stack: the parallelism strategy, the collective communication library, the checkpointing cadence. The part that is missing from that picture is the building, and at the frontier the building has become one of the terms in the equation that decides whether your run finishes on schedule or spends its life recovering from faults.
This is not an argument that facilities are nice to have. It is an argument that above a certain cluster size, the physical environment, its cooling stability, its power behaviour and its interconnect, sets a ceiling on effective training throughput that no amount of code can lift. The evidence for this is now in the open literature, and it is worth walking through it as an engineer would, from the failure data outward.
Reliability does not scale the way intuition says
The single most important fact about large-scale training reliability is that mean time to failure falls roughly linearly with the number of GPUs, because a synchronous job fails when any participating device does.
The numbers are documented. Meta’s account of training Llama 3 gives a mean time to failure of 7.9 hours for a 1,024-GPU job, dropping to 2.7 hours at 16,384 GPUs. Over the 54-day 405B pre-training run on that larger cluster, the team logged 466 interruptions, of which 419 were unexpected, and 78% of the unexpected ones were traced to hardware, with GPUs and their HBM memory the largest single categories. That is an unplanned interruption roughly every three hours. Extrapolate the same per-device rate and the picture at the next scale is stark: researchers modelling a 100,000-GPU cluster estimate a failure roughly every 30 minutes. xAI’s Colossus, at around 100,000 H100s, is precisely the regime where that estimate applies.
The consequence is that at the frontier, a meaningful fraction of wall-clock time is spent not training but detecting faults, restarting from checkpoints and re-converging. Epoch AI’s analysis of the Llama 3 run estimates around 2.1% of training time went on checkpointing and recovery, with optimal checkpoints every few minutes. That figure is manageable, and it is manageable precisely because the failure interval, while short, still comfortably exceeds the checkpoint write time. Push the failure rate up, whether through scale or through an environment that degrades hardware faster, and that comfortable relationship narrows. Every additional increment of failure rate is a direct tax on the fraction of your cluster’s life spent moving the model forward.
None of this is a facility problem in isolation. But the dominant failure categories, GPUs and HBM, are thermally sensitive components under sustained high power, and the rate at which they fail is not independent of the environment they run in. That is the first place the building enters the loop.
The interconnect is where the building’s fingerprints are clearest
If the GPU failure data is suggestive, the network failure data is close to conclusive, and it is the part of the story an AI engineer is least likely to have priced in.
A large training cluster is stitched together with optical transceivers, thousands upon thousands of them, one at each end of every link between a NIC and a leaf switch and onward through the fabric. SemiAnalysis, modelling a 100,000-GPU cluster, made the point that lands hardest here: even if every individual NIC-to-switch link had a mean time to failure of five years, the sheer number of transceivers means the expected time to the first link failure on a brand new, fully working cluster is about 26 minutes. Because AllReduce requires every GPU to participate, a single failed link can stall the entire job. Without fault recovery through memory reconstruction, a cluster of that size would spend more time restarting than advancing.
Optical transceivers are among the highest failure-rate components in the entire system, and this is where cooling stops being an abstraction. Manufacturers are explicit that higher operating temperatures raise transceiver failure rates, that inconsistent cooling airflow causes unpredictable failures, and that thermal cycling, the repeated heating and cooling that comes with a load swinging between phases, physically wears the optics faster. A facility that holds a stable, uniform thermal environment is not offering a comfort feature. It is directly suppressing the failure rate of the components most likely to halt your run.
The failure statistics bear out how large this category is. Production reports compiled in the recent fault-tolerance literature attribute 15.8% of all training failures at Alibaba to network issues, split between NICs and optics, and 8.3% of failures in one set of Azure InfiniBand clusters to network causes. These are not edge cases. They are a standing fraction of everything that goes wrong, and their rate moves with the thermal quality of the hall.
Power behaviour, and the third path into the loop
The third mechanism is power, and it is distinctive because AI training abuses the electrical system in a way conventional workloads never did.
A synchronous cluster draws power in unison. Thousands of GPUs move through compute and communication phases together, so the facility-level load swings hard and fast. Research from NVIDIA, Microsoft and OpenAI documents racks moving from around 30% of full draw to 100% and back within milliseconds. Google’s engineering team, writing about power fluctuations in ML infrastructure, reported swings of roughly 15 megawatts against around 1.5 for the non-AI workloads they were used to, and noted the underlying cause: conventional data centres host many uncorrelated workloads, whereas synchronous training does not. Meta hit the same wall during Llama 3, where coordinated power changes across the cluster stressed the limits of the data centre’s power delivery.
For the engineer, this matters in two ways that connect back to the two mechanisms above. The swings themselves stress power delivery and, through thermal cycling, the hardware whose failure rate you are trying to keep down. And the facility’s ability to absorb the swings determines whether you are allowed to run flat out at all, or whether ramp limits and power caps are imposed that show up as lost throughput. The mitigations are arriving in the hardware: NVIDIA’s GB300 NVL72 adds capacitor-based energy storage in the power shelf and a startup power cap, cutting grid-facing peak by up to 30%, but the facility still has to be built to work with them rather than against them.
What this changes about how you choose where to run
Pull the three mechanisms together, and the conclusion is not that liquid cooling makes your model train faster in any direct sense. The honest evidence on raw throughput is modest: a controlled 2025 benchmark of identical liquid-cooled and air-cooled H100 nodes found a 17% throughput gap only under synthetic maximum-load stress, shrinking to a fraction of a percent on real fine-tuning workloads, while the reliable, repeatable difference was in power draw, roughly a sixth lower on the liquid node for identical work.
The throughput case for a well-built facility is indirect, and it runs entirely through reliability. A stable thermal environment suppresses the failure rate of the GPUs, the memory and above all the optics that dominate cluster-scale interruptions. A lower failure rate means a longer effective interval between restarts, which means a higher fraction of wall-clock time spent training. At 1,000 GPUs this is a rounding error. At 16,000 it is real. At 100,000 it is the difference between a cluster that trains and a cluster that mostly recovers. The facility does not speed up the arithmetic inside a step. It changes how many steps you complete before something breaks.
That reframes the specification questions worth asking of any provider. Not what the rated density is, but what sustained GPU and optic temperatures actually are under load, and how uniform they are across the hall. Not whether there is liquid cooling, but whether the thermal environment is stable enough to hold transceiver failure rates down over a run measured in weeks. Not the headline power figure, but whether the facility can absorb synchronous swings without imposing the ramp limits that cap your throughput. These are the questions a purpose-built high-density colocation facility can answer with measurements, and a converted air-cooled hall generally cannot.
Gavin Lester is COO at Carbon-Z, a UK colocation provider running high-density and liquid-cooled data centres for AI and HPC workloads.
