Let’s start with an amazing fact: despite massive investment in AI infrastructure, most AI clusters operate at just 50–70% average GPU utilization, even during peak periods, highlighting an efficiency gap between theoretical performance and real-world AI infrastructure.
When people talk about AI infrastructure, the conversation almost always starts and ends with GPUs. Which GPU generation? How many TFLOPS? How much HBM? While compute is undeniably critical, focusing on GPUs alone is one of the most common and expensive mistakes organizations make when building AI clusters.
In reality, AI performance is the result of an entire system working in harmony. Compute, memory, storage, software, scheduling, and networking, all determine how fast you reach results. And when AI workloads scale beyond a single node, the component that matters most is usually not the one you spent the most money on.
Let’s break it down.
The Building Blocks of an AI Cluster
GPUs
GPUs are the engines of AI training and inference. Their raw compute capability, memory bandwidth, and support for reduced precision (FP16, BF16, FP8) directly affect how fast individual operations execute. Well-optimized kernels, fused operations, and efficient use of tensor cores can dramatically reduce step time.
But here’s the catch: GPUs only deliver value when they are busy. An idle GPU, waiting for data or synchronization is underused or even wasted silicon.
Memory (On-GPU and Host)
Large models and batches push GPU memory to its limits. When memory is constrained, developers rely on techniques like activation checkpointing, gradient accumulation, or model parallelism. These may work, but they introduce recomputation and communication overheads that ripple through the system.
Host memory and NUMA placement also matter. Poor CPU–GPU affinity or insufficient memory bandwidth can throttle data movement and kernel launches, quietly slowing everything down.
Storage
AI training is data hungry. If storage can’t keep up, GPUs stall waiting for input. Dataset format, sharding strategy, caching, and prefetching all influence how efficiently data flows from disk to GPU.
Many teams discover too late that “fast GPUs + slow data” equals slow training. Storage bottlenecks don’t show up in spec sheets, but they show up clearly in GPU utilization graphs.
Software Stack
Frameworks, compilers, libraries, and drivers determine how effectively hardware is used. Kernel fusion, graph compilation, collective communication libraries, and runtime scheduling can make the difference between theoretical peak performance and real-world results.
The same hardware can deliver wildly different performance depending on software maturity and configuration.
Scheduling and Operations
Queue time, job placement, preemption, and checkpointing overhead all affect job completion time (JCT). A perfectly optimized training job doesn’t help much if it waits in a queue or restarts repeatedly due to poor scheduling decisions.
The Part Everyone Underestimates: Networking
Once training scales beyond a single node, networking becomes the dominant factor in performance.
Distributed training is fundamentally a communication problem. Gradients must be synchronized. Activations and parameters must move between GPUs. In modern large-scale training, these transfers happen every step. If the network can’t keep up, GPUs wait, and waiting GPUs are the most expensive kind of inefficiency.
Why Networking Matters So Much for AI Training
- Synchronous training amplifies delays
In data-parallel training, all GPUs must wait for the slowest gradient synchronization. A small network delay is multiplied across thousands of steps.
- Communication grows with scale
As you add GPUs, compute scales linearly, but communication often does not. Poor networking leads to diminishing returns and terrible scaling efficiency.
- Collectives are latency and bandwidth sensitive
Operations like all-reduce, all-gather, and all-to-all stress the network with large, frequent transfers. Both latency and sustained bandwidth matter.
- The network defines the system
At scale, your AI cluster behaves less like “many GPUs” and more like a single, distributed machine whose performance is bounded by how fast data moves between its parts.
This is why AI grade fabric has become such a critical topic. Ethernet is no longer just a general-purpose interconnect, it is increasingly the fabric that determines whether AI clusters scale efficiently or collapse under their own communication overhead.
Optimizing Networking for AI Training over Ethernet
Making Ethernet work well for AI is not about raw link speed alone. It’s about designing the network as a first-class part of the AI system.
Design for Lossless, Predictable Transport
AI collectives are extremely sensitive to packet loss and congestion. Congestion control, traffic load balancing, and lossless or near-lossless behavior are essential to avoid retransmissions that harm performance predictability.
Match Topology to Training Patterns
All-to-all and all-reduce traffic patterns place very different demands on the fabric. A topology optimized for changing traffic paterns, with minimal tuning, is critical for multi-node training efficiency.
Smart Bandwidth Scale
As GPU compute increases, network bandwidth must scale with it. Under-provisioned networks quickly become the bottleneck, no matter how fast the accelerators are. However, simply matching an 800 Gbps GPU interface with an 800 Gbps network link is not sufficient to ensure smooth operation. Collective communication patterns introduce additional challenges such as incast, and traffic bursts that require intelligent network technologies purpose-built for AI workloads.
Optimize for Collective Operations
Ethernet-based AI fabrics must be tuned specifically for collective communication. This includes minimizing hop count, balancing paths, and ensuring consistent latency across flows.
Treat the Network as Part of the Training Stack
Networking can’t be an afterthought. Placement, scheduling, and software libraries must all be topology-aware to avoid stranding jobs on slow paths or congested links.
The Real Lesson
GPUs are essential, but they are not the system.
In modern AI clusters, networking is often the deciding factor between impressive scaling and disappointing results. You can buy the fastest accelerators on the market, but without an optimized Ethernet fabric underneath them, you will never see their full potential.
The most important part of your AI cluster isn’t what you think. It’s the part that keeps all those expensive GPUs working together as one.
Bio:
Sani Ronen
Director of AI Networking, DriveNets
Sani serves as a director of AI Networking at DriveNets. With over 25 years’ experience in communications systems and semiconductors, Sani brings expertise in AI, networking, wireless communications, and autonomous driving accumulated at a variety of organizations including Arbe, Microchip, Microsemi, Radwin and Siemens. He holds a BSc in Electrical & Computer Engineering from Ben-Gurion University and a Masters in Entrepreneurship and Innovation from Swinburne University of Technology.
