Many companies embarking on AI projects will need to bring in an array of external data feeds into the AI systems they are building. Their data teams will need to extract data from these feeds, turn that data into intelligence, score and benchmark the outputs, and use the combined output as the basis of valued added services for suppliers, customers and/or partners alike.
GaiaLens has run more than a dozen AI pilots for a range of large enterprises in financial services, automotive, utilities and other sectors during recent years, and thought that it was worth putting together this piece which lays out some best practice in the vital area of managing data feed integrations to ensure the success of AI pilots.
Data Purpose
In terms of validating early in an AI pilot that the identified external data is capable of supporting the intended outcome, it’s important to use partial data sets and simplified logic to test the validity of the data.
The objective at this early stage is to assess correlation rather than determine absolute data accuracy. In other words, we need to answer the question: ‘do changes in the data move scores in ways our client’s domain experts expect?’. If relevant correlation signals are absent at this early stage, scaling will not fix it!
It’s also important at this stage to avoid a natural tendency to over-collect data. Each data feed must justify its inclusion based on three key criteria:
- Decision impact
- Uniqueness of signal, and
- Operational cost.
If removing a feed does not materially change outputs or decisions, then it does not belong in the core data feed pipeline.
Data Source Identification
The types of data feeds you select needs to be hardwired to the agreed business objective and to the solution you are trying to create. For example, GaiaLens built its own ESG scoring and anti-greenwashing AI-based solution for asset managers and financial institutions which drew on a mixture of third party Application Programing Interfaces (APIs), structured batch data sets, regulatory disclosures, event-driven updates (including annual reports), and ‘derived’ data sets created by our data team through contextual enrichment and data normalisation.
Data Quality, Data Cleaning & Normalisation
Data normalisation work is the disciplined process of making different data points comparable, stable and safe to combine so that like can be compared with like; scale does not distort importance; noise doesn’t overwhelm signal and outputs remain interpretable and explainable.
Assessing the reliability, latency and volatility of external data sources is also vital work for our data engineers. Reliability is measured by historical uptime and schema stability; latency by delivery consistency; and volatility by how often values change unexpectedly.
Avoid Silent Substitution
Real-time pipelines need to be set up to prioritise resilience and allow for graceful degradation. This means that when a real-time data feed becomes unavailable; latency increases beyond pre-agreed tolerance levels; data quality drops below acceptable thresholds; or schema or semantic changes are detected; the system needs to be set up to reduce capability deliberately. It must be designed to preserve correctness over completeness and avoid corrupting ground truth or scores.
Near-real time pipelines focus on checkpointing and replay. For example, in a scenario where processing failure occurs midway through processing the data. The system establishes at which point the near-real time pipeline crashed, the last successful checkpoint and the number of that record.
On restart, the state is restored from that checkpoint and data from that point is set up to be re-read. Outputs are recomputed. By doing this, no data is lost, double counting is avoided and a deterministic recovery is enabled.
Batch pipelines, by contrast, emphasise validation and reconciliation. Bear in mind batch pipelines’ primary design objective is correctness rather than immediacy. Batch processing is typically used where data feeds define records of truth, support financial, regulatory or reporting outcomes. They must be complete and internally consistent. It must also be possible to prove this data quality.
During pilots, GaiaLens applies checks for data feed completeness, freshness and logical consistency. Missing or stale data is explicitly flagged. Anomalies are isolated and investigated offline before they influence scores.
Silent substitution must be avoided as it quickly compromises AI systems and is difficult to detect and recover from – partly because it tends to produce outputs that look plausible. However, it will corrupt scoring and ground truth quietly without anyone finding out until it is too late. In short, silent substitution can invalidate months of results.
Establishing ‘Ground Truth’
Defining and agreeing ground truth with business and domain experts is a governance and design exercise first, and a technical exercise second. The objective is not to find a philosophically ‘perfect’ truth but to establish a shared, testable, auditable reference reality that the organisation agrees to treat as correct for a specific decision, at a specific point in time.
Ground truth is meaningless unless it is anchored to a business decision or outcome. It’s important to answer the following questions as part of ground truth establishment:
- What decision does this AI output influence?
- What would a human expert decide if they had perfect information?
- How would we judge whether that decision was ‘right’ later?
Examples:
“Was this supplier genuinely high risk at the time of taking them on?”
“Was this transaction actually fraudulent?”
“Were reported emissions materially accurate?”
This sort of framing prevents ground truth becoming abstract or academic.
It is also important to have a senior named individual owning that ground truth. That person must know what evidence underpins it. He or she must also have a procedure for resolving disagreements around the ground truth and assessing whether it is still accurate. Ground truths must all be reviewed regularly. Assess whether your ground truth(s) are explainable to a regulator, auditor or standards body. If they sound too vague this is a good indication they are not precise enough.
Scoring, weighting & value attribution
The key, particularly with trust-based domains like ESG scoring, is to be able to drill down into factors which contribute to the scoring. We call this ‘explainability’. By contrast, ‘black box’ systems which spit out uncheckable scores are simply not good enough for most AI applications. Your AI system must be fully explainable and transparent, especially when underpinning regulatory reporting.
It is also important to be able to measure confidence, uncertainty level or margin of error in scores. Confidence ranges based on data coverage are only credible if they are grounded in measurable coverage indicators. GaiaLens draws on up to 10 typical data coverage dimensions which are built into our transparency scores to measure confidence level as follows:
- Completeness
- Percentage of required inputs present
- Freshness
- Age of data relative to expected update cadence
- Breadth
- Number of contributing data sources vs expected number of sources
- Continuity
- Gaps in time series or event streams
- Representativeness
- Whether data covers all relevant segments.
Model Interaction & Data Exposure
When data is being compared with proprietary datasets or LLM outputs, it is important to expose the minimum amount of your data to the models. Sensitive or proprietary data must be isolated and referenced indirectly where possible. It is also important to tightly control retrieval scope for prompts. GaiaLens RAG (retrieval-augmented generation) pipeline ensures real-time parsing and validation to deliver assured data reliability. Any model outputs must be fully traceable back to source data for checking later.
We test ‘hallucination risks’ where external and proprietary data intersects. We routinely query outlying edge cases and monitor unsupported assertions, particularly where external and proprietary data intersect.
It is important that every factual claim must connect to a retrieved document, record, or metric. If evidence is clearly missing, the AI system must respond with uncertainty. The systems we develop are constrained to retrieved content. Free-form synthesis is restricted or explicitly labelled as ‘inference’. Citations or references are mandatory for factual outputs.
Data Governance & Security
GaiaLens uses Role-based Access Control (RBAC) for ingestion, storage and model layers to define what actions a role may perform.
For example, a:
Data Engineer may define features, but not deploy models
Analyst may query outputs, but not raw data
Operator may run inference, but not change prompts
Auditor can read logs only.
It is also possible to control access to different types of data outputs, again based on role. So,
Analysts might be set up to see aggregate scores
Executives can see summaries
Regulators see full explanations behind findings.
Operational resilience & scalability
Our data team routinely monitors data feed health for our clients – assessing data quality degradation and scoring anomalies in real-time. It is important that pipelines remain modular and are horizontally scalable – allowing capacity to grow without redesign. A horizontally scalable pipeline must tolerate increased load. Yet scalability is meaningless unless load is fully defined.
Load typically increases via:
- Volume – more records, events or files
- Velocity – higher arrival rates
- Cardinality – more keys, entities or dimensions
- Complexity – more joins, enrichments or features
- Tenancy – more users, clients, or models.
Data sets, models and scoring logic are all independently versioned to enable controlled and fully traceable evolution. Data feeds are actively monitored by our teams via automated alerts.
Systems must be designed to notify users immediately if a feed fails or begins degrading. Downstream validation checks assess the impact on data quality and coverage. This ensures that any degradation is clearly identified and addressed in a timely manner, rather than silently propagated.
Organisational Enablement
Handing back AI systems which we have designed and developed for clients is sensitive work. In-house teams need multiple skills to ensure the health and effectiveness of AI systems going forward.
They need mature data engineering, domain expertise, governance and operational monitoring capabilities, not just data science skills. In terms of governance structures, clear ownership of data sources, scoring logic and change controls needs to be supported by cross-functional oversight.
Test, Learn, Test before Scaling
Finally, it is important to consider and mitigate the biggest data feed integration risk which might derail an AI pilot in the vital first 90 days of a project. The biggest risks, in my view, are that the team over-estimates the data quality the system is generating from Day 1, while underestimating schema volatility. Be ready to monitor and tweak systems to keep them on track. To this end, phased delivery reduces risk, builds trust and allows learning and hardening of systems before scaling.
