There is a conversation that keeps happening in enterprise boardrooms. Leadership has invested in AI. The data science team has built models that perform well in the lab. The business case is compelling. And then the initiative stalls somewhere between proof of concept and production.
When teams dig into why, they usually find the same culprit. It is not the model. It is not the business logic. It is the data pipeline.
After 17 years building data platforms across Media, Banking, and Healthcare, I have seen this pattern play out in every sector I have worked in. Organizations invest heavily in AI capability while treating the data infrastructure underneath as a secondary concern. The result is sophisticated AI running on foundations that were not designed to support it.
The data engineering problem is the AI problem. Most organizations have not recognized this yet.
Why Data Pipelines Are the AI Bottleneck
When I began working in data engineering, the primary concern was reporting. Data teams built pipelines to move information from source systems into data warehouses where analysts could query it. The latency was measured in hours or days, and that was acceptable because the business decisions being informed by the data were measured in weeks.
AI changes the latency requirement fundamentally. A model that predicts customer churn needs current behavioral data to be accurate. A fraud detection system needs near-real-time transaction feeds to be effective. An inventory optimization model needs supply chain data that reflects what is actually happening, not what happened last night.
Traditional batch ETL pipelines were not designed for this. When I design near-real-time data solutions today using Databricks and AWS Kinesis, the architecture is fundamentally different from the batch pipelines that still power most enterprise data warehouses. Data arrives continuously rather than in overnight windows. Processing logic runs constantly rather than on a schedule. Monitoring and error recovery must operate continuously rather than between batch runs.
This architectural shift is necessary for AI to work, but it introduces complexity that organizations consistently underestimate. The engineering discipline required to build and operate a streaming data platform is different from the discipline required to manage a batch data warehouse. The teams that treat it as a simple technology upgrade rather than an architectural transformation end up with systems that look modern on paper but fail in production in ways that are difficult to diagnose and expensive to fix.
The Data Quality Problem AI Makes Worse
There is a phrase I use often with teams I train on AI in data warehousing projects: AI does not solve your data quality problems. It exposes them faster.
This is not a subtle point. A business intelligence report that aggregates from a dirty data source produces inaccurate results slowly. A model trained on that same dirty data produces inaccurate predictions at scale and propagates the errors through every decision it influences. The damage happens faster and is harder to reverse.
Data governance design is one of the areas where I have seen organizations make the most consistent mistakes when preparing for AI. Traditional data governance focuses on access control, security, and compliance. These remain important. But AI adds a new dimension: model data lineage. Understanding where the training data came from, how it was transformed, and what quality checks it passed before being used to train a model is essential for trusting the model’s outputs and debugging failures.
In banking and healthcare, sectors where I have led data warehousing teams on complex enterprise-level initiatives, the regulatory requirements for data lineage are becoming more stringent precisely because regulators are recognizing that AI decisions need an auditable data foundation. An AI system that approves or denies a loan needs to be traceable back to the data that influenced that decision. That traceability has to be built into the data platform architecture, not retrofitted after the fact.
Building scalable ETL pipelines that incorporate governance by design, rather than governance as an afterthought, is one of the highest-leverage investments an organization can make before scaling AI. The teams that have done this work can audit their models. The ones that have not cannot.
Feature Stores and the Consistency Problem Nobody Talks About
One of the less glamorous but more consequential infrastructure challenges in production AI is maintaining consistency between how features are computed during model training and how they are computed during inference.
In a typical enterprise scenario, a data science team builds a model using historical data. They write feature engineering code that transforms raw data into model inputs. That code works in the research environment. Then the model goes to production, and a different team has to implement the same feature transformations in a production data pipeline, usually with different tools, different data access patterns, and different latency requirements.
What sounds like a straightforward engineering problem becomes a source of systematic model degradation. The model was trained on features computed one way. It is receiving features computed a different way. The predictions are worse, and nobody can explain why because the two implementations look similar enough on the surface.
I have built table design, ETL frameworks, and pipeline orchestration systems that address this consistency problem by design. The solution is a feature store: a centralized repository that defines how each feature is computed, stores the definitions in one place, and serves those definitions consistently to both the training environment and the inference environment. This is not a novel concept. But it is not yet standard practice in most enterprise data environments, and the inconsistency it prevents is a major source of production AI failures.
What Sector Experience Teaches You
Working across Media, Banking, and Healthcare teaches you something that no amount of theoretical data engineering does: different sectors have fundamentally different data cultures, and AI adoption has to account for that.
In Media, I have seen organizations that have invested heavily in data infrastructure, where the engineering culture supports continuous data processing and the business stakeholders understand the value of real-time data. These organizations are generally further along in adopting AI because the data engineering foundations are already in place.
In Banking, I have encountered organizations where data quality and governance are mature but where the regulatory environment creates constraints on how data can be used for AI that require careful architectural planning. The AI strategy has to account for compliance requirements from the beginning, not as an add-on to a data science project.
In Healthcare, I have found that the data challenges are often the most severe: fragmented data sources, inconsistent standards, and legacy systems that were never designed to support the data integration requirements that AI demands. The engineering work required to prepare healthcare data for AI is more substantial than in other sectors, and organizations that underestimate this struggle for years.
The common thread across all three sectors is that AI success is preceded by data engineering success. The organizations that have done the foundational work of building reliable, governed, well-orchestrated data platforms are the ones that can operationalize AI. The ones that have not are the ones running perpetual pilots.
What Engineering Leaders Should Focus On
If I were advising data engineering leaders on where to focus over the next several years, I would start with five concrete priorities.
Conduct a data pipeline readiness assessment before committing to AI roadmaps. The most common mistake I see is organizations building AI strategies without first understanding whether their data pipelines can support AI workloads. If your pipelines run on daily batch cycles, if data quality metrics are not tracked systematically, if lineage is not documented, those are the problems to solve before scaling AI. The AI roadmap will be more realistic and more achievable once the data infrastructure is ready for it.
Build streaming capability where AI use cases require it. Not every data pipeline needs to be streaming, but every AI use case with real-time requirements needs infrastructure that can deliver data with sub-minute latency. Databricks and AWS Kinesis are two of the more mature platforms for this, and the engineering patterns for building reliable streaming pipelines are well established. The investment is real, but it is smaller than the cost of operating AI on infrastructure that was not designed for it.
Implement data quality monitoring as a production discipline. Treat your data pipelines like production software: with automated testing, continuous monitoring, and incident response processes that trigger when quality degrades. Data quality metrics should be visible alongside pipeline health metrics, and data quality incidents should be treated with the same urgency as system outages.
Invest in a feature store as a core data platform component. The consistency problem between training and inference environments is real and consequential. Building a centralized feature store is the most reliable way to prevent it. This is an investment that pays dividends across every model that goes into production.
Build AI literacy across the data engineering team, not just among specialists. The teams that have conducted training sessions on applying AI in data warehousing projects understand this well. When a data engineer understands how a model consumes features, they make better upstream design decisions. When a business analyst understands what data quality means for model accuracy, they take data governance more seriously. Cross-functional AI literacy multiplies the effectiveness of every specialist on the team.
The Foundation Determines the Ceiling
The most common question I get from organizations exploring AI is which model they should use. The more important question is whether their data infrastructure can support the model they choose.
The models available today are sophisticated enough to extract value from well-structured data. The limiting factor is not model availability. It is data pipeline readiness. Organizations that invest in the foundational work of building reliable, governed, real-time capable data platforms will be the ones that translate AI potential into business results.
The data engineering problem is the AI problem. Solving it requires the same rigor, investment, and architectural discipline that organizations apply to any mission-critical system. The difference is that the payoff is not just operational efficiency. It is a competitive advantage that compounds over time.
The organizations that get this right will have a structural advantage that is difficult to replicate. The ones that continue to treat data infrastructure as a secondary concern will keep running pilots while others run production.
