What is Explainability?

For any Data Scientist working on a business use case, it becomes crucial to be able to explain their Machine Learning models leading to a decision in order to convince the business teams of the relevance of the model and to possibly be able to use this model as an aid to the decision.


The Data Scientist is often caught in a dilemma between building a simple model (such as a logistic regression) that she will be able to explain simply but whose accuracy is not always satisfactory, and a more complex model (such as Deep Learning) which will be efficient but difficult to explain.

There are now different methods and explainability tools that allow Data Scientists to take advantage of the latest machine learning techniques while being able to explain their models, and thus helping business leaders to trust them.


The different tools and articles in the field of explainability use concepts defined in a disparate manner across different projects.

Récemment, l’article Recently, the article “Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI”provided a synthesis around the following concepts:


Interpretability defines making the operation of models understandable to humans without requiring a particular (technical) prerequisite around Data Science.


Explainability defines being able to explain predictions resulting from a model from a more technical point of view to a human.


A model is considered transparent if it is understandable on its own from simple explanations.

The main types of so-called “transparent” classification models are:

Logistic regressions: this is a type of model that allows one (or more) target categories to be predicted based on a combination of input variables. Provided that the number of variables is reasonable and that the variables are understandable. It is possible to describe and even to graphically represent the operation of these models, making them easily usable and understandable.

Decision trees: decision trees are models based on sequences of nodes corresponding to binary conditions on several hierarchical levels leading to a sheet corresponding to the result of the prediction. As long as you use a reasonable number of variables and nodes, it is possible to represent and explain the different steps resulting in the decision of the model.

K nearest neighbors: The K nearest neighbors algorithm predicts the class of an input element by examining the class of its known K nearest neighbors. For example for K = 5, the predicted class for a new element will be the majority class among the classes of the 5 closest neighbors of this element.

These transparent models are therefore simple models which are easily explained and therefore commonly used in practice but do not always provide precise decisions.

The following figure, from the “Principles and Practice of Explainable Machine Learning” article, also provides examples of the main transparent models:

This image has an empty alt attribute; its file name is qHOidBit1PDreQL09V4HVkv0v6hkiOiDFLM0V73df047KzXK73rKIwXcGSQVZAoHLJcn89wNkKAAnPq4QaYuTuY-zrnMWZORJrbNT3RmrXs7B4OsPvDO764LTGQHB1C-ymDQt9uifuw_rkxaHw

On the other hand, interpretability and explainability make it possible to make more complex models understandable and therefore to obtain models that perform better.

Main explainability approaches and methods

Global approach

A first family of methods aims to provide a global explanation of the functioning of the model independently of the input data. Among the best known methods, we can cite the “Features Importance” or the “Partial Dependence Plot” which are regularly used as a first approach but only provide a limited explainability of the model.

Local approach

In contrast, local approaches provide a precise explanation associated with a particular element to be predicted. The 2 local methods currently most used are “LIME” and “SHAP” (which appeared in 2016 and 2017 respectively) and provide an explanation by indicating the variables which contributed the most to the prediction for this element.

The following figure shows an example of an explanation provided by SHAP to explain the probability of survival of a passenger on the Titanic based on their characteristics. The probability of the individual (here 0.81) is mainly explained by the fact that he is female (sex_male = 0) and does not travel in third class (pclass_3 = 0).

This image has an empty alt attribute; its file name is jT2-3nRhqwmopjho7YIDvgvl_moXlcx1uo5IKAwlGUAUZuZzcUMLyz6kuaxgWXVxn1tk69iL-exSU0OPJJxbVKbchJgxB5WfJHOEdt-GY0t6vLAYFxG219qSRHOO5S__ohaoIAodyzGjiGGURQ

For further

These are the main concepts around the explainability of Machine Learning.

To learn more, you can read the following:

“Principles and Practice of Explainable Machine Learning” is an article offering a broad overview of explainability methods as well as recommendations on the most useful methods in different use cases.

SHAP: The implementation of the SHAP method is available on Github and offers a fairly easy to use API.


[ExplainableConcepts] [ExplainableConcepts]: “Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities, and Challenges toward Responsible AI”; Arrietaa et al. https://arxiv.org/pdf/1910.10045.pdf https://arxiv.org/pdf/1910.10045.pdf