Within just two years following the launch of OpenAI’s ChatGPT in November 2022, we have seen an explosion of AI tool deployments in a wide range of industries. The success of these tools is anchored on the efficiency and processing power of AI systems that understand and generate human language, also known as Large Language Models or LLMs.
From tech giants like Google and Meta to startups like Anthropic and Mistral, and now DeepSeek, everyone is vying for a piece of the LLM pie. While the open-source movement fosters a sense of democratization, the reality is far more nuanced. The true value of LLMs lies not in the readily available models themselves, but in their application using proprietary data. This creates a self-perpetuating cycle where large enterprises, with their vast data troves and deep pockets will have a significant advantage.
This advantage is amplified by the economics of LLM development. The price tag for training these sophisticated models is projected to reach a staggering $100 billion by 2027, driven largely by hunger for high-performance GPUs.
Generative AI requires MASSIVE infrastructure to train a state-of-the-art model. These models are evolving towards dedicated AI chips and optimized software on top of it to maximize performance from the hardware. Also, AR/VR requires low-latency setup to provide real time rendering and sensory integration. This is forcing data centers to have high speed interconnects to ensure smooth immersion experience for the users. Personalized recommendations mean a higher data processing burden in order to run machine learning models that analyze user behavior and context in real time.
This dynamic has catapulted companies like NVIDIA, the leading provider of AI chips, to a position of immense power within the burgeoning LLM market. This strategic advantage is further solidified by NVIDIA’s recent foray into developing its own LLM competitor. Consider this – With infrastructure accounting for over 60% of LLM development costs, and NVIDIA commanding a 75% gross margin on its hardware, they have the potential to undercut competitors by a staggering 45% simply by adjusting their pricing strategy.
Even with AWS introducing new Trainium 2 chips and aiming to be 30-40% cheaper, NVIDIA’s H100 and A100 GPUs remain the industry standard. This pattern echoes the consolidation seen in cloud computing with AWS, Microsoft, and Google..
However, the ultimate winners in this evolving landscape will undoubtedly be cash-rich large enterprise businesses across industries. Their advantage stems from three key factors. Firstly, data dominance: these behemoths sit on mountains of proprietary information, providing unparalleled training grounds for AI models. Secondly, established channels: existing distribution networks, strong customer relationships, and robust sales forces offer a significant leg up. Third, the increasing interoperability of LLMs benefits these giants most.
As switching between LLMs becomes simpler, larger companies can seamlessly leverage the most advanced AI tools. How is all of this reshaping data center infrastructure? In terms of the evolution for AI powered consumer applications and data center demand, watch out for:
Hardware – Shift from traditional CPU‑centric designs to hybrid architectures that combine CPUs, GPUs, and other accelerators geared toward AI workloads. In addition to LLMs, consider personalized recommendations use cases – it imposes a higher data processing burden to run machine learning models that analyze user behavior and context in real time. One might think that there is too much unstructured data in silos in organization which could be making it difficult for companies to synthesize personal recommendations for their customers.
Not really! LLMs have taught everyone that the excuse of insurmountable data no longer holds water. Customers are demanding personalized recommendations and it is getting infused into all applications.Such complex workloads involving real-time analysis of user behavior, are far better handled by the parallel processing capabilities of GPUs and accelerators than by traditional CPUs.
Energy requirement – There are innovations based on nuclear energy that are focused on Small Modular Reactors (SMRs) that are making nuclear solutions a good option considering the amount of power that AI based workloads are going to need in the future. The energy footprint of large language models is rapidly increasing. Some estimates suggest that in the next 5 years, an individual LLM model can consume up to 1.5 gigawatts of power, a level comparable to a substantial city and sufficient to power over a million households. This immense energy demand highlights a critical challenge: the power consumption of AI models is set to grow even further.
Lower Latency – While the first mile was the ability to ingest, create and generate documents including pictures, videos and voice. Now, Augmented Reality (AR) and Virtual Reality (VR) are revolutionizing the interaction between digital and physical spaces. These immersive technologies demand ultra-low latency. To meet this need, and to enhance reliability and scalability, distributed data centers are becoming essential and boosting system resilience significantly.
Data center infrastructure is most certainly changing. Organizations should consider use cases and contexts where they could leverage Small Language Models (SLM) instead. If an SLM can adequately perform a task, lower resource usage, and power will translate into high-cost effectiveness. In the case of data privacy, it offers several advantages over LLMs and fine-tune them for specific verticals. Using a Lord of the Rings analogy – do we need one ring to rule them all, or should we craft rings tailored for each distinct realm?
