Today is a new world — and it will be a different world when we wake up tomorrow. The pace of AI is accelerating and changing everything.
Much of human success as a biological species can be attributed to the fact that we can transform the world around us in a way that suits us and conforms to our needs. This includes creating complex tools to tackle tasks that extend beyond our human capabilities. Recent advances in machine learning seem to be no different. But like never before, tools can now make certain decisions for humans.
So now we face a new challenge: what if we don’t understand how an algorithm arrives at a given decision? In the past, technologies were largely transparent, allowing for easier and more definitive fine-tuning upon close inspection. However, as AI increasingly challenges human intelligence, it is crucial to clearly understand the factors that contribute to a decision to effectively assess its accuracy.
What is explainable AI – and why is it important?
For about a decade now, explainable AI (XAI) has been a key concept, promoting the idea to develop algorithms that are able to provide explanations that we can understand. This aligns with the previously mentioned human desire to transform the world in a way that conforms to our needs: If we do not understand something, we don’t trust it – and if we want to trust decisions that algorithms are making, we should create algorithms that can provide explanations.
Not all automated decisioning systems actually require XAI. Prominent examples that we are all familiar with are autocorrect systems: we only need the algorithm to be able to make accurate predictions, and far less important to understand why and how the algorithm arrived at the suggestion to correct the text.
However, other applications, like automated decisioning about mortgage loans, are happening in a highly regulated space. Here, explainability is a critical factor: If customers are denied mortgages, this has significant implications for their lives, and they have the right to know why.
Another reason why XAI plays a crucial role is that any algorithm can make mistakes and, if this happens – in particular, for high-stakes decisions, the question of responsibility and accountability arises. One example is hallucinations in large language models, where the algorithm produces nonsensical text. For instance, factual information that is entirely wrong. In many modern systems, we as humans are integrated in the decision-making, reflecting the so-called Human-In-The-Loop (HITL) concept, and XAI can equip us with the tools necessary to keep the AI in check.
At the same time, there is an ongoing debate about how HITL practices can be made meaningful, and whether XAI can provide accurate and actionable insights. While HITL practices are needed to consider effective methods of human oversight and intervention, both of these challenges benefit from interpretable AI.
Often used interchangeably with transparency and explainability, interpretability specifically refers to the ability for humans to understand the cause-and-effect relationships within a system, achieved through a clear understanding of its inner workings. This approach ensures that explainability is based on a deeper understanding of an AI system, rather than relying on correlated assumptions.
Why is explainability challenging?
The difficulty in creating explainable algorithms is deeply rooted in the complexity of the mathematics necessary to cope with rich data sets and to produce accurate and reliable results. One aspect is the number of dimensions. As humans, we can easily understand one-dimensional objects (e.g., lines), two-dimensional objects (e.g., surfaces), and three-dimensional objects (e.g., a cone).
Our brain, however, as powerful as it is, is not designed to easily imagine four-, five-, or even higher-dimensional spaces. For example, take a basic data set in credit risk together with a model that predicts default probabilities. Typical input features include annual income, interest rate, debt-to-income ratio, FICO score and more, where each feature corresponds to one dimension.
Often, data sets have tens or even hundreds of features, operating – mathematically speaking – in spaces with tens or hundreds of dimensions. When looking for explanations, we tend to strive for some reduction of this complexity to lower dimensions, for example, by asking “what were the three major factors that contributed to this decision?” While there are excellent algorithms to answer this question, the answer itself needs to be understood without painting the full, and inherently complex, picture.
This idea is very intuitive in sports when asking in a similar way the question “which players contributed the most to winning the game?” While we can certainly try to find an answer (and we acknowledge that the answer might be valuable), it is obvious that the complete answer to this question will certainly not capture everything about the team and the game.
A second layer of mathematical difficulty for creating simple explanations stems from the fact that the relationship between the output of the model and the input features can be complicated. When we think of the relationship between two variables, we often think of linear behavior, reflecting proportional relationships. For instance, the longer I drive at a certain speed, the larger the distance from my point of origin. In reality, however, many relationships are more complex and often nonlinear.
An example for such a relationship from our daily life would be how the flourishing of a plant depends on the amount of water given. Initially, we observe that watering the plant more will make it thrive, but with time, we realize that too much water is detrimental. If we graph this relationship, it will resemble a parabola with a maximum, denoting the point with the ideal amount of water given for the plant.
Combining high dimensionality with nonlinearity often leads to highly complex relationships between the features of a data set and the output variable that we are trying to model, and this makes explainability such a challenging task. Certain modeling approaches, like neural networks, are able to capture these relationships very efficiently, leading to high accuracy of their predictions. However, they belong to the class of so-called black-box algorithms that do not lend themselves to providing explanations easily.
What are the solutions?
Modern-day algorithms are getting more and more complex, capturing more dimensions and non-linearities. They cannot (and should not) be made interpretable unless those are involved in high-stakes decision-making. An attempt to ask LLM to explain its decisions will only create more confusion.
But how can we reconcile this with the growing need for responsible decision-making? Interpretable algorithms of the past, such as scorecards and rules-based decisioning systems, stand no chance of addressing the challenges and the complexity of the tasks faced by businesses and individuals today. Fortunately, nowadays, our technology around AI has matured to help us tackle this difficult problem.
First, agentic architecture gives us some understanding of how this problem should be addressed. For instance, a genAI-driven virtual assistant is not a single-point-of-failure architecture, where an LLM is responsible for interpreting the request, processing it and generating the response. A robust application can use LLM (or a number of LLMs specialising in different fields of expertise) for interpreting the request and generating response candidates.
But a system of weights and balances controls the actual content of the response and eventually, decision making. Those weights and balances can utilize interpretable algorithms, such as scorecards, rule-based inference, and even the most advanced probabilistic logic rules. Yet as with all other algorithms, those are also subject to human supervision.
Second, if we need to work with a black-box algorithm, there are a variety of tools (explainers) available that can still help to provide explanations. Often, explainers efficiently zoom into the data and extract the importance of one or a group of various features around a certain data point. While they might not explain relationships valid for the entire data set, their results can be highly valuable to understand individual decisions.
Most consequentially, we need to be aware of what AI can and cannot do, especially for high-stakes decisions where it is increasingly necessary to keep humans in the loop.
