Poor data quality costs organizations an average of $12.9 million per year. Inconsistency across siloed systems is the problem data teams consistently rank as the hardest to fix. And yet most companies continue to treat data quality as a post-hoc cleanup exercise, something you investigate after a dashboard looks wrong or a model starts drifting.
I lead the architecture of a unified customer knowledge platform for the largest local commerce marketplace in the United States, a system that serves tens of millions of consumers, hundreds of thousands of merchants, and millions of delivery partners. The platform processes billions of API calls per day across dozens of integrated services, powering real-time personalization, fraud prevention, marketing, advertising, and machine learning workflows. Every promotion a user sees, every fraud check that runs before an order, every eligibility decision for a subscription benefit all of it depends on attribute data flowing through this platform in real time.
The single hardest engineering problem we faced was not throughput, latency, or scale. It was trust. How do you make dozens of independent engineering teams each with their own priorities, timelines, and definitions of correctness confidently depend on the same customer attributes for high-stakes, real-time decisions?
The answer was not better pipelines. It was enforceable data contracts.
The Fragmentation Problem That Scale Exposes
At a small scale, data quality is a personal relationship. The analyst who built a dashboard knows exactly where the numbers come from. The engineer who owns a pipeline can explain its quirks. When something breaks, the right person is a desk away.
That model does not survive past a few teams. Once you have multiple engineering organizations reading from and writing to the same attribute store, the informal trust network collapses. A growth team defines an active user one way. A fraud team defines it another.
A marketing team builds a third version. Each definition is reasonable in isolation. Together, they pull the same customer in conflicting directions.
The deeper problem is that these teams do not distrust each other — they distrust the data. And when downstream teams cannot trust the data, they do the rational thing: they build their own copy. They fork the pipeline, add their own validation, maintain their own transformations.
Now you have N versions of the same attribute, drifting further apart with every deployment. The cost is not just compute. It is the compounding complexity of debugging incidents where two teams looking at the same logical entity see different values and neither can explain why.
The global data quality market is projected to reach $4.3 billion by 2030, driven precisely by this pattern. Organizations are spending more on tools to fix a problem that is fundamentally architectural: data quality is treated as an afterthought when it needs to be designed into the platform from the start.
Building Trust Into the Pipeline
When I designed the data quality framework for our platform, I made a deliberate choice: quality enforcement could not be a separate system that teams opted into. It had to be part of the pipeline itself an automatic property of any attribute that flowed through the platform.
We built three layers that work together as a continuous trust mechanism:
Pre-merge validation. Before any attribute update is merged into the source-of-truth table, it passes through configurable validation checks. Schema conformance, nullability constraints, value range boundaries, referential integrity the rules are defined at the attribute level and enforced at write time. If an update fails validation, it is quarantined and alerted. It does not pollute the trusted dataset.
Post-merge anomaly detection. Validation catches known failure modes. But the most dangerous data quality issues are not schema violations, they are silent drifts. A distribution shifts. A null rate creeps up.
A downstream model starts seeing patterns that do not match training. We built automated anomaly detection that runs after merge, watching for statistical deviations across every attribute. When it detects something unusual, it alerts before a human would think to look.
Automated profiling. Every attribute in the platform is continuously profiled row counts, distinct values, null ratios, distribution statistics so there is always a baseline of normal behavior. Profiling runs as a background process, building the signal that both validation thresholds and anomaly detection rely on.
These three layers do not eliminate the need for human judgment. But they eliminate the need for human vigilance. A data engineer does not need to remember to check whether yesterday’s batch looked healthy. The platform enforces that check automatically, every time, for every attribute.
The result is that quality shifts from a reactive discipline “something broke, go find it” to an architectural guarantee. Teams downstream of the platform do not need to verify the data before using it. The verification is a property of the pipeline itself.
Data Contracts: From Informal Trust to Enforceable SLOs
Validation and anomaly detection tell you whether data is correct. But they do not tell downstream teams what they can expect and expectation is what engineering trust is really about.
A fraud prevention system does not just need correct data. It needs data that arrives within seconds of a transaction, with freshness guarantees, at defined throughput levels. A personalization model does not just need accurate attributes. It needs them to be consistently available during peak traffic hours, with bounded latency.
These are operational requirements, not quality requirements. And in most organizations, they are negotiated informally with a Slack message, a meeting, a shared document that nobody updates. The guarantees exist only as long as the people who made them remember them.
We solved this by codifying data contracts into a tiering framework with enforceable service-level objectives. Every attribute in the platform is classified into a tier. Each tier defines:
- Freshness SLO: How quickly updates must propagate from source to serving layer
- Availability SLO: Minimum uptime for the attribute’s read path
- Validation requirements: Which pre-merge checks must pass before the attribute is published
- Anomaly detection coverage: Which post-merge statistical monitors are active
- Support obligations: Incident response time, escalation path, owner contact
High-tier attributes the ones feeding fraud, eligibility, and customer-facing personalization carry the strongest guarantees. Lower-tier attributes experimental or exploratory carry lighter requirements. The tier is not a judgment of importance. It is a contract about what downstream teams can depend on.
This changes the consumption model fundamentally. Instead of every downstream team negotiating trust informally with every upstream team, they inherit a defined reliability posture based on the attribute’s tier. A fraud engineer building a new model does not need to ask, “Can I trust this attribute?” The answer is documented, measured, and enforced by the platform.
Why This Pattern Generalizes Beyond One Platform
The data contract and tiering pattern is not specific to customer attribute platforms. It applies anywhere that multiple teams produce and consume shared data assets:
In financial services, a risk-scoring model consuming transaction attributes from a central data platform needs the same kinds of guarantees of freshness, accuracy, and availability that a fraud detection system needs in a commerce marketplace. The tiering framework maps directly: high-tier for risk and compliance attributes, lower tiers for exploratory analytics.
In healthcare, patient attributes flowing from electronic health records to downstream analytics, clinical decision support, and billing systems must carry defined reliability guarantees. A billing system and a clinical trial matching system may consume the same patient attribute but require different freshness and accuracy SLOs. Data contracts make those differences explicit.
In advertising technology, audience attributes powering real-time bidding systems need sub-second propagation guarantees that reporting attributes do not. The tiering framework lets the platform offer both without forcing every consumer into the same reliability profile.
The common thread is that as data platforms grow from serving a single team to serving the entire organization, the informal trust models that worked at small scale break. Teams need explicit, enforceable, machine-readable contracts not because they distrust each other, but because trust at scale requires structure.
What Practitioners Should Do Differently
If you are building or operating a shared data platform, here are three changes I recommend:
First, make quality enforcement part of the pipeline, not a separate system. If your data quality checks require a human to run them, they will not run consistently. Build pre-merge validation, automated profiling, and anomaly detection into the write path so that every attribute is verified before it reaches consumers. The quality guarantee should be an automatic property of the platform, not a manual discipline.
Second, classify every asset into a tier with defined SLOs. Not every attribute needs five-nines availability. Not every attribute needs sub-second freshness. But every consumer needs to know what tier an attribute belongs to and what guarantees that tier provides.
Define the tiers, document the SLOs, and measure compliance. If an attribute consistently misses its tier’s SLOs, either fix the pipeline or reclassify the attribute but stop pretending that all data is equally reliable.
Third, treat data contracts as living documents enforced by the platform, not written agreements filed in a wiki. A contract that nobody reads is not a contract. Build tooling that lets consumers inspect an attribute’s tier, SLO compliance history, and validation status programmatically. Make the contract discoverable at the point of consumption when an engineer is writing a query or configuring a pipeline, they should see the guarantees attached to each attribute without leaving their workflow.
The most honest thing you can tell a downstream team is not “this data is reliable.” It is “this data is guaranteed to meet these specific standards, and here is how we measure it.” That level of specificity is what earns engineering trust at scale. And it is the only foundation on which real-time, high-stakes data products can be built with confidence.
Data contracts are not bureaucracy. They are the architecture of trust. And at the scale where incorrect data propagates into thousands of automated decisions before anyone notices, trust is not a nice-to-have. That is the whole point.
