The AI Infrastructure Bottleneck No One Is Solving

Every infrastructure roadmap for generative AI reads the same way: more GPUs, more high-bandwidth memory, faster interconnects, better token throughput. These investments matter, but specialization is only currently targeting one half of the inference pipeline. The other half, the stage where a model actually reads the prompt before it writes a word, still has no purpose-built hardware in production. 

This stage is called prefill, and it is where a large language model processes every token of the input, whether that is a short chat message or a 200-page document, before generating its first response token. The IEA reports that electricity use from AI-focused data centers surged 50% in 2025 alone. Prefill has become a significant share of that total: at 128,000-token contexts it can account for the majority of the compute in a single request, depending on output length: a long prompt answered briefly is almost all prefill, while an extended reasoning trace shifts weight back to decode. Infrastructure leaders who keep treating prefill as an afterthought are going to feel that cost first. 

Two workloads wearing one uniform 

Every inference request is really two different jobs stitched together. Prefill ingests the full prompt in one parallel pass, which makes it compute-bound – the system is limited by how many matrix operations it can push through per second, and attention cost rises with the square of the input length. Decode generates the response one token at a time, reading an ever-growing cache of previous tokens from memory at each step, which makes it memory-bandwidth-bound. 

Researchers at UC San Diego and Peking University formalized this distinction in DistServe, showing that co-locating both phases on the same hardware forces a serving system to compromise between time-to-first-token and time-per-output-token, because the two goals pull on the same resources. Separating the phases into independent pools, an approach now used in systems such as Mooncake, is no longer a research curiosity – it is standard practice in production inference stacks at scale. 

The specialization is happening on one side only 

Splitting the two workloads into separate pools solves a scheduling problem. It does not solve a hardware problem, because most disaggregated pools still run prefill and decode on the same type of chip, just organized differently. This chip was designed as a compromise between compute throughput and memory bandwidth, and hence neither phase gets the balance it actually needs.  

The good news is that there are now processors designed specifically for decode, built around on-chip memory, deterministic latency and small-batch efficiency. These are commercially available and running at production scale.  

Prefill is, however, still served by general-purpose GPUs originally built to handle training, prefill and decode all at once. NVIDIA has conceded the architectural point – Rubin CPX was announced in September 2025 as a prefill-specific part – but it has been pulled from and added back to the roadmap since, and nothing purpose-built for prefill is in a production rack today. 

The gap between what is needed and the hardware available is widening because the volume of prefill work is growing faster than the industry’s attention to it. Longer documents, retrieval-augmented pipelines that stuff more context into every call, and agentic workflows that chain multiple reasoning steps together all multiply the number of tokens a system must prefill before it generates anything useful. The IEA has flagged energy-intensive, always-on AI agents as a specific driver of rising data center demand, on top of an electricity base that is already straining grid connection queues in every major market. 

Caching is used to help with this problem – prefix and context caching are real and effective, and any operator serving a fixed system prompt to many users may already be using them. They help with the part of the input that repeats, however they do not help with the novel long document or the freshly retrieved context, which is precisely the traffic growing fastest. Agentic loops cache better than they first appear, since each step shares a monotonically growing prefix; the cost there is the fresh context every step pulls in, not the trajectory itself. So although caching reduces the amount of prefill workload for some applications, it does not fix the hardware efficiency. 

There is a gap in how any of this is measured: although major inference APIs already price input tokens separately from output tokens, most public postmortems on inference cost still report a single blended number per request, rather than breaking out what prefill cost separately from decode. This makes it hard for any individual team to know whether their own bottleneck is even in the phase they assume it is. The telemetry mostly exists already – vLLM and SGLang both emit per-phase metrics – so this is a reporting convention gap rather than a tooling one, and closing it is the precondition for a more optimized hardware flow.  

A framework for deciding when to specialize 

Recognizing the problem is the easy part. The harder question for anyone running or architecting inference infrastructure is when, and how, to act on it. The following sequence is a practical starting point, independent of any specific vendor or chip architecture. 

Profile the workload before you architect for it. Break down your inference traffic by phase and measure what share of latency and cost each one actually consumes. Track time-to-first-token separately from time-per-output-token, and tag requests by input length so you can see how the split shifts as prompts grow. A customer support bot answering short questions has a very different prefill-to-decode ratio than a system summarizing long contracts or running multi-step agents, and without this baseline, any specialization decision is a guess dressed up as a strategy. 

Separate the scheduling problem from the hardware problem. If your inference stack still co-locates prefill and decode on the same nodes, disaggregating them logically, using an orchestration layer such as vLLM, SGLang, or a similar serving framework, is the first move and it does not require new hardware. This step alone recovers throughput lost to phase interference and gives you clean visibility into how each pool behaves independently. 

Model the total cost of ownership at the breakeven point, not the sticker price. The comparison that matters is energy and throughput per dollar at your actual prefill volume, not list price per chip. Build the model using your own workload profile from step one, and be explicit about the context-length and request-volume thresholds where the economics flip in favor of specialization. Power and cooling capacity should be part of that model too, since a rack that draws less power for the same throughput frees up capacity. A data center is built with a fixed maximum power capacity, so the question is not what a rack costs but how much usable compute you can extract from the watts available. 

Pilot the smallest viable slice. Route a single, well-characterized workload, such as one API endpoint or one internal application, through a dedicated prefill pool before touching the rest of the fleet. This limits blast radius, produces real operating data faster than a lab benchmark, and gives your team hands-on experience with a second hardware profile in production. Set success criteria before the pilot starts, covering cost per request, latency, and operational overhead, so the decision to expand or abandon the approach is based on evidence rather than momentum. 

Build the operational muscle for heterogeneous hardware before you need it at scale. Running two different hardware families in production means two sets of monitoring, two capacity-planning models, and a procurement process that can evaluate a non-GPU accelerator on its technical merits rather than defaulting to what is already on the approved vendor list. Teams that build this muscle during a small pilot move faster when the economics justify a wider rollout, while teams that wait inevitably end up learning it under production pressure. 

Score any new hardware against workload fit, not marketing claims. Whatever technology you evaluate for prefill, whether that is a different accelerator generation, a purpose-built ASIC, or an emerging compute paradigm, judge it against the same criteria you would apply to any infrastructure investment: does it fit in your existing power and cooling envelope, does it support the numerical precision your models actually need, does it integrate with your current ML frameworks, and does its supply chain carry acceptable risk?  

Sequencing matters more than speed 

None of these steps require ripping out an existing stack. Disaggregating prefill and decode logically, then evaluating specialized hardware for the compute-bound half only after the workload data justifies it, is a lower-risk path than betting an entire infrastructure refresh on a single architecture choice. It also produces a defensible business case, because every claim is backed by your own measured workload rather than a vendor’s reference benchmark. 

The organizations most exposed to the prefill bottleneck are the ones with the longest context windows and the most agentic traffic, which happen to be the workloads growing fastest. Waiting for the market to converge on a single answer is itself a decision, and it is an expensive one measured in both power and dollars. 

The cost of waiting 

Capital expenditure among the largest technology companies is running at close to $800 billion in 2026 across the five largest hyperscalers, up roughly 65% on 2025, largely to build out AI infrastructure. A meaningful share of that spend is going toward general-purpose accelerators asked to do a job they were never optimized for. None of that is an argument for a specific chip or a specific vendor. It is an argument for treating prefill as a first-class infrastructure decision rather than an inherited default. 

The industry has spent the last two years optimizing decode, and it shows: memory bandwidth, KV cache management, and speculative decoding have all matured quickly under sustained attention. Prefill has not had the same scrutiny, even though it now accounts for a growing share of both the latency users feel and the compute that operators can extract from a fixed power budget. The framework above will not tell any single organization exactly which hardware to buy, because that answer depends on a workload profile only that organization has. 

What it should do is move the conversation from whether prefill deserves dedicated infrastructure to when, at what volume, and under what evidence. The infrastructure leaders who start applying that discipline now, rather than waiting for the market to hand them a consensus answer, are the ones who will not be caught paying for a gap everyone could see coming.