The LLM market has rapidly accelerated in its technological development, marking a new era of innovation thanks to generative AI’s ability to leverage unthinkable amounts of data to produce advanced insights. Within this, foundational artificial intelligence (AI) models are evolving rapidly; there are more coming to the market each month, and in the next year or so, the sheer volume of these models will explode. As AI tools become ubiquitous, more people are leveraging the technology to confront complex challenges more effectively than they were ever able to in the past.
However, we are at an inflection point. Large language models (LLMs) as they are today have already consumed the entirety of the world’s usable training data, leading some models to be trained on synthetic data that they generated themselves. Despite this, we still haven’t reached AI’s final form. In order to progress AI innovation at scale despite a lack of organic training data, we will inevitably see a splitting of the market. There will be a transition to an era in which more intelligent systems, not necessarily larger models, sit at the forefront.
Not every use case requires the full power of an LLM. Niche use cases, like fixing a new model of airplane in a remote location, for example, don’t require the full extent of machine knowledge, but rather targeted, tailored instructions, images, videos, and perhaps language translation. For this use case, a Small Language Model (SLM) accessible from a mobile phone with no service would be more appropriate than an LLM with an encyclopedic knowledge of the history of British Airways.
This is just one example of how the demand for generative AI to fit specific emerging use cases is shifting the LLM market we have all become familiar with, into more of an ‘xLM’ market. Models will be large, small, portable, hybrid, remote, and domain-specific, and use cases will grow more diverse with varying price, security, and latency sensitivity levels.
The data engineering challenge
Models with enhanced reasoning abilities are key to the burgeoning xLM market, with progress seen in the likes of OpenAI’s o3. This development, which allows models to be smarter but reliant on less data, is the quickest route to achieving artificial general intelligence (AGI). Yet, a transformation of the data infrastructure that underpins these models is needed for them to succeed. This presents challenges.
The primary struggle is creating architectures that manage structured and unstructured data types, streaming data, and real-time updates. Developing model types requires flexibility in data consumption while upholding strict governance and security standards. This balance is achieved by managing two distinct strands of uniquely managed data. Training data has to be curated, tracked, and aligned with evolving data governance policies. In contrast, in-the-moment data must be configured with robustness, latency, cost, and compliance. Adaptability is critical in data pipeline design as language models will be applied in the near future in ways that don’t make sense now; rigidity will result in being forced to replatform down the line.
Real-time data and the engineering burden
xLM evolution could potentially put significant pressure on data engineering resources, primarily when an organisation relies on static batch data uploads and batch fine-tuning. Frequent uploads and meticulous attention to data accuracy demands a team of experts with specialist skills, frequently leading to cost and resource barriers. The growing demand for applications with to-the-moment accuracy exacerbates this challenge. However, there is a light at the end of the tunnel.
Easing the burden with real-time AI systems
Real-time data pipelines are an area of AI innovation that is developing quickly. Live data feeds offer the dual benefit of improving model accuracy by facilitating continuous learning and unlearning while easing the burden on data engineering teams. Shifting away from static pipelines in favour of ones that use hybrid systems to combine batch processing with live data connectors or API-based feeds removes the time-consuming task of plumbing the data pipeline with integrated, transformed data for engineers.
Organisations and data engineering teams should assume that most future systems will be ‘real-time-ish’ and design their systems appropriately in preparation for this. Of course, to implement real-time systems, data infrastructure needs to be robust, and historically, this would have put a heavy burden on data engineering resources. So, a more modern strategy is essential.
Data pipelines that automatically integrate, transform and deliver data to the xLM without the need for continual manual input should be designed from the beginning. Emerging tools and advanced data infrastructures allow these feeds to be up and running within hours without the need for lengthy evaluation and training cycles, reducing pressure on data engineers. These tools also make it possible to experiment with more flexibility and encourage organisations to think ahead and select tools that will seamlessly adapt to future applications and use cases.
By adopting frameworks that enable automation and intelligent data management, businesses will build the groundwork for more intelligent models. This will help engineering teams move away from repetitive, time-consuming manual tasks and allow them to focus on testing the waters with new models, approach challenges with creativity and help decision-makers unlock further AI-driven efficiency.
Embracing evolution
The splintering of the LLM market into an xLM market is the next stage of AI’s development. Adopting the next generation of intelligence models and future-proofing the data infrastructure that underpins them will enable businesses to circumvent current roadblocks, fuel a wider range of use cases and create an environment of innovation that will accelerate industries.
