The conversation around AI infrastructure is obsessed with securing compute. But with industry-wide utilization sitting below 40%, the real race is not about buying more GPUs. It is about fixing the massive visibility gap in production.
Which GPU. How many. Which cloud. What price per hour. These are the wrong questions.
Most GPU clusters running AI inference today operate at 35-40% utilization. Uptime Intelligence found that even well-optimized models reach only 35-45% of the compute performance the silicon can deliver, and note the numbers are likely worse for inference. Anyscale independently confirms production AI workloads often achieve well below 50% sustained utilization, even under load. It is a consistent pattern observed across clusters of every size, in every major cloud. The hardware is there. The money is being spent. Less than half the compute is being used.
It is a visibility problem. MLOPs teams cannot optimize what they can’t see.
The Unit Economics
Take a 1,000-GPU cluster running NVIDIA GB200 NVL72 instances on CoreWeave, listed at $42.00 per instance per hour with 4 GPUs per instance, or $10.50 per GPU per hour. That cluster costs approximately $7.7M per month. At 35-40% utilization, somewhere between $4.6M and $6.9M of that spend is going to idle cycles every single month.
Now consider a conservative scenario: continuous production profiling and optimization recovers 40% of that wasted capacity. On a 1,000-GPU cluster, that means 1,000 GPUs now deliver the equivalent output of roughly 1,400 GPUs. The expansion you would have provisioned to meet demand becomes unnecessary.
- Monthly saving: approximately $3M
- Annual saving: approximately $36M
- Unit economics: 40% more requests served per GPU at the same infrastructure cost
These are conservative numbers. The actual ceiling is higher. But even at 40% recovery, the return on investing in production visibility rewrites the unit economics of the entire operation.
The instinctive response when performance degrades is to provision more hardware. It is also the most expensive response. Each provisioning cycle embeds the inefficiency more deeply: more GPUs running at the same utilization rate, more spend, the same underlying problem. Over a two or three-year infrastructure cycle, that compounds into a cost base that is structurally higher than it needs to be.
The teams that understand this are not asking “how do we get more GPUs.” They are asking “why are we not using the ones we have.”
What Is Actually Causing the Waste
The bottlenecks driving GPU underutilization are rarely obvious. They are not configuration errors or architectural mistakes. They are emergent behaviors that only appear under production conditions: real input distributions, sustained throughput, memory pressure, and interactions between components that do not exist in a lab environment.
A denoising loop re-entering the Python interpreter on every step, throttling CPU dispatch and leaving the GPU idle between kernel launches. NCCL synchronization lags accumulating invisibly across nodes. Memory fragmentation compounding across hours of sustained load. These are not the kinds of problems that show up in a five-minute profiling window during development. They emerge in production, under real traffic, at scale.
The tools most engineers reach for: PyTorch Profiler, NVIDIA Nsight Systems, DCGM metrics in Grafana, were all designed for development environments. PyTorch Profiler loses context when execution crosses from Python into native code, which is exactly where modern inference frameworks like SGLang operate. Nsight Systems captures windows of five minutes or less; production regressions are frequently intermittent and load-dependent. Nsight Compute serializes GPU execution to collect metrics, introducing overhead that can mask the very conditions it is trying to observe. DCGM tells you utilization is low. It does not tell you why.
The gap between “something is wrong” and “here is exactly what is wrong, traced to the instruction” is where most teams are flying blind.
What Production Visibility Actually Requires
Effective production visibility has four requirements that development tooling does not meet.
Always-on data collection. The evidence must exist when a regression occurs, not require manual capture after the fact. Regressions are not scheduled.
Zero instrumentation. Profiling in production must not require code changes, recompilation, or debug symbols. The system being observed must be the system that runs in production.
Full-stack correlation. Engineers must be able to follow the execution path from a Python function through native code, CUDA kernel launches, and GPU instruction execution. Bottlenecks routinely cross these boundaries.
Fleet-wide scope. Bottlenecks that originate in one node and manifest across the cluster must be visible as a single coherent picture. Per-node, per-process tooling misses the interaction effects that drive the largest inefficiencies.
When you have all four, something else becomes possible: autonomous optimization. The profiling data contains enough context for an AI agent to identify the root cause, generate a fix, and raise it as a pull request without requiring a specialist in the loop on every regression. This is what I call Profile-Guided AI Optimization: using continuous, fleet-wide profiling data as the input for agentic engineering workflows.
The Competitive Dimension
The economic argument is straightforward. But there is a second-order effect that matters more over time.
AI teams that can iterate faster, because their infrastructure is efficient, observable, and self-correcting, compound their advantage. The gap between a team running at 40% utilization and one running at 80% is not just a cost difference. It is a difference in how many experiments they can run, how quickly they can move from training to deployment, and how much of their budget is available for model development versus infrastructure remediation.
Every month a performance bottleneck goes unresolved on a 1,000-GPU cluster, between $4.6M and $6.9M in capacity is being paid for and not used. That is a direct consequence of the 35-40% utilization rates the industry already accepts as normal. It does not have to be normal.
The cheapest GPU is the one already running efficiently. Right now, for most teams, that is not the GPU they have.
Author
Israel Ogbole is Co-Founder and CEO of Zymtrace, an AI infrastructure optimization platform. Previously, he was Principal Product Manager at Elastic, where he led Universal Profiling to GA and led the contribution of the eBPF CPU profiler to the OpenTelemetry project. Before that, he held product roles at Cisco/AppDynamics and worked at Microsoft following the Yammer acquisition.
