Businesses need to know what will happen if they change something. Will investing in stronger customer service reduce cancellations? Will a new pricing strategy increase sales?
Simulation can help retailers, banks and subscription businesses explore these scenarios before committing time and money. Companies such as Netflix, Walmart and Uber already use causal inference to inform business decisions, but building these capabilities in-house requires specialist expertise. The value of a simulation ultimately depends on how well it reflects the factors driving real-world outcomes.
Consider a subscription business using AI to predict which customers are likely to cancel. Its analysis shows that customers who submit more support tickets are more likely to leave. The business might conclude that improving customer service will help retain them. However, those customers may have already decided to leave, or contact support because of technical faults or billing problems. Acting on the wrong explanation could mean investing in a solution that doesn’t address the real problem.
Explaining a model is not explaining reality
When an AI model makes a prediction, analysts often need to understand how it reached that result. Tools such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic Explanations) help explain individual predictions by showing which factors influenced the result and how. SHAP estimates each factor’s contribution to a prediction, while LIME builds a simpler, more interpretable approximation of the model around a specific case. For example, an analyst investigating why a customer was flagged as likely to cancel their subscription could use these tools to examine how support tickets, account history, and other information influenced the model’s assessment.
These tools help explain a model’s behavior, but they cannot establish whether support tickets caused the customer to leave or simply appeared alongside other reasons for canceling.
A 2018 study, Sanity Checks for Saliency Maps found that some techniques for explaining predictions made by complex AI systems produced similar explanations even when the systems or the data used to train them changed. The research examined visual explanation methods rather than SHAP and LIME specifically, but raised a wider concern: an explanation can look convincing without accurately reflecting how a system reaches its conclusions.
For businesses, knowing which information influenced a prediction is not the same as knowing what drives the outcome.
Historical data can hide the real problem
AI models learn from historical data, which reflects customer behavior, past business decisions, and how information was collected.
In the above example, high support ticket volumes could indicate poor service. They could also reflect customers who have already decided to leave, technical faults or a difficult cancellation process. Other customers might leave without contacting support at all.
A model can identify support tickets as a predictor of cancellations without distinguishing between causes and symptoms. Reducing recorded support requests might make the model predict fewer cancellations, without addressing the problems driving customers away. Fixing a technical fault or billing error could have a much greater impact.
The same issue affects lending, retail and marketing. A lender’s historical data reflects previous lending decisions as well as borrowers’ repayment behavior. A retailer’s sales figures reflect demand, pricing, stock availability and marketing.
If businesses mistake these patterns for explanations, they risk making poor decisions, from investing in ineffective interventions to overlooking growth opportunities.
A simpler model is not necessarily a better explanation
Businesses sometimes turn to regression analysis, a statistical method that estimates how an outcome relates to one or more factors. Its results can be easier to communicate than those of a complex AI model, but a clear statistical relationship does not automatically establish cause and effect.
Consider a company measuring advertising’s impact on sales. Advertising attracts people to its website, some of whom make purchases. If analysts account for website traffic when measuring advertising’s effect, they may remove part of the impact they are trying to measure.
The factors included in an analysis, and the relationships between them, can change its conclusions. As Miguel Hernán and James Robins explain in Causal Inference: What If, answering questions about cause and effect requires careful assumptions and study design.
AI models and statistical methods can identify patterns without establishing cause. Making a model easier to understand does not automatically make its conclusions more reliable.
Building on prediction to understand the impact of business decisions
A retailer might use AI to predict which customers are likely to stop buying. That forecast helps identify customers who need attention, but it does not show whether a discount, better delivery, or a revised loyalty program will persuade them to stay.
A discount might retain a price-sensitive customer but waste money on someone who would have purchased anyway. Better delivery might address the real source of dissatisfaction, while a loyalty offer might have little effect.
Businesses need to estimate what would happen under different decisions. Causal analysis helps assess the likely effects of interventions, while simulation can help compare potential outcomes under different assumptions.
A 2020 study, Feature Relevance Quantification in Explainable AI: A Causal Problem, examined the assumptions involved in measuring feature relevance using Shapley values, highlighting the importance of understanding the relationships assumed between variables.
Causal diagrams can help analysts map those relationships and identify which factors matter when estimating an effect. Analysts can then assess their assumptions against existing evidence, expert knowledge, and, where possible, real-world experiments.
In one anonymized business example we have seen, regression analysis suggested that a new digital sales channel was taking revenue away from a traditional channel. Causal analysis instead indicated that wider market trends were driving the traditional channel’s decline, with customers shifting towards digital as a result. Restricting the digital channel could therefore have sacrificed growth without solving the underlying problem.
Causal analysis is not infallible. Its conclusions depend on data quality, assumptions and whether important factors have been overlooked. Its advantage is that it makes those assumptions explicit, allowing decision makers to challenge them.
Trust requires more than transparency
As AI influences decisions about credit, healthcare and other consequential areas, businesses need to understand whether its outputs provide a sound basis for action.
The EU AI Act, an important international benchmark for AI regulation, sets requirements for certain high-risk AI systems, including provisions on transparency, documentation, and human oversight. The obligations depend on the system and its intended use.
For example, a lender may need to investigate whether a factor influencing an automated credit decision is appropriate or acts as a proxy for another characteristic. A chart showing which factors influenced a prediction cannot answer every question about the relationships behind it.
There is a commercial cost to getting this wrong. Executives may disregard useful AI insights or act on misleading interpretations. A model can perform well against a technical benchmark while contributing little to better business decisions.
Start with the decision, not the model
As AI becomes more capable and businesses use it to inform increasingly complex decisions, the distinction between predicting outcomes and understanding them becomes more important. Retailers deciding how to price products, banks assessing lending decisions and subscription companies trying to retain customers all need to know what their analysis can reliably tell them.
For forecasting demand, identifying unusual transactions or prioritizing cases for review, accurate predictions may be enough. Tools such as SHAP and LIME can help analysts understand which information influenced those predictions.
For decisions about changing prices, increasing marketing spend or redesigning a service, businesses need evidence about the likely consequences. This may require causal analysis, controlled experiments or simulation to compare possible outcomes.
Teams should be clear about the limits of their evidence. A prediction does not prove cause and effect, and a persuasive chart cannot make up for missing evidence.
SHAP and LIME remain useful tools. The mistake is expecting them, on their own, to prove why something happened or what will happen if a business intervenes.
AI will earn greater trust when businesses distinguish between understanding a model’s predictions and understanding the world those predictions describe. The real test is whether the analysis helps decision makers compare options and make better informed choices.
About the author
Dr Ayman Elhalwagy is Co-Founder of Rootcause.ai, a simulation and decision intelligence platform that helps enterprises model scenarios, test decisions, and forecast outcomes using advanced mathematics and business data. He holds a PhD in Computer Engineering from Brunel University London, where he researched deep learning for machinery condition monitoring, and he received the 2023 Brunel Research Impact Award.
