Explainable AI refers to strategies and procedures used in the application of artificial intelligence (AI) that allow human specialists to understand the solution’s findings. To ensure that explanation methods are correct, they must be systematically reviewed and compared. In contrast to achieving quantitative explanation, in this article, we will discuss Quantus, a Python library that evaluates a convolutional neural network’s working, predictions and explanation of parameters. Below is the list of major points that will be discussed in this article.
Table of contents
- The explainable AI
- The limitation of current methods
- How does Quantus enhance the explanation?
- Explaining the CNN with MNIST data
Let’s first discuss Explainable AI.
The explainable AI
Explainable artificial intelligence (XAI) refers to a set of processes and strategies that enable humans to comprehend and trust the results of machine learning algorithms. “Explainable AI” refers to the ability to define an AI model, its predicted impact, and potential biases.
It contributes to the definition of model correctness, fairness, transparency, and outcomes in AI-powered decision-making. The ability of an organization to generate trust and confidence is critical when deploying AI models. AI explainability also helps to implement a responsible AI development strategy.
Explainable AI is analogous to “showing your work” in a math problem. All AI decision-making and machine learning processes do not take place in a black box – it is a transparent service designed to be dissected and understood by human practitioners. In order to add ‘explanation’ to the output, input/output mapping is essential.
Understanding how an AI-enabled system has resulted in a specific output has numerous advantages. Explainability can assist developers in ensuring that the system is performing as expected, it may be required to meet regulatory standards, or it may be critical in allowing those affected by a decision to challenge or change the outcome.
The limitation of current methods
Despite a lot of interest and action in the subject of XAI, evaluating explainable approaches is still a topic that hasn’t been solved. Because explaining, unlike traditional Machine Learning (ML), requires no ground-truth data, there is no commonly agreed definition of what constitutes a right explanation, and even less so, which properties an explanation should satisfy.
Because of the lack of standardized assessment techniques in XAI, diverse parameterizations, preprocessing, and normalizations are frequently used, each producing different or even contradictory results, making evaluation outcomes difficult to comprehend and compare.
For these reasons, we frequently rely on a qualitative assessment of explanation methods, such as believing that humans know what an accurate explanation would (or should look like, often ignoring the function of the explained model in the explanation process). The notion that people can recognize a correct explanation, on the other hand, is rarely valid.
How Quantus enhance the explanation
In order to address this problem, Anna Hedström et al. created Quantus, a versatile and comprehensive toolbox that collects, organizes, and explains a wide range of assessment metrics recommended for explanation techniques.
Quantus delivers a set of more than 25 reference metrics for evaluating explanations of ML forecasts to its intended users ML and XAI practitioners. Furthermore, it provides detailed instructions on how to apply these measurements, as well as advice on potential dangers. We can see some examples of analysis that can be done with Quantus in the diagram below.
For any type of CNN problem, as demonstrated in the above picture, section-a where a simple qualitative comparison of XAI approaches is frequently insufficient to determine whether the gradient-based method is favoured, such as Saliency, Integrated Gradients, Gradient Shape, or FusionGrad. We can get more information about how the methods compare with Quantus by doing a holistic quantification on several evaluation criteria (shown in section b) and running a sensitivity analysis on how a single parameter, such as the pixel replacement strategy of a faithfulness test, affects the ranking of explanation methods (shown in section c).
Explaining the CNN with MNIST data
Here in this section, we’ll utilize Quantus on MNIST handwritten digits dataset to explain how a standard CNN works. Here we generate explanations for saliency and integrated gradients map qualitatively (in terms of visualizations) and quantitatively (in terms of scores) for a set of the data.
Here I’m replicating the official starter guide and will explain in the best possible manner. The CNN model used here is called LeNet5 which is a basic CNN model with all essential layers. It was one of the first convolutional neural networks to be developed and it aided in the advancement of deep learning. Now to move further we need to install and import all the dependencies as below.
# installing dependencies
! pip install quantus
! pip install captum
! git clone https://github.com/understandable-machine-intelligence-lab/Quantus.git
import quantus
import torch
import torchvision
from torchvision import transforms
# load the model from repo
from Quantus.quantus.helpers.models import LeNet
import captum
from captum.attr import *
import pandas as pd
import numpy as np
# Enable GPU.
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
Now, let’s load the model and dataset with a data loader.
# load the model
model = LeNet()
model.load_state_dict(torch.load("Quantus/tutorials/assets/mnist"))
# load the data
transformer = transforms.Compose([transforms.ToTensor()])
train_set = torchvision.datasets.MNIST(root='./sample_data', train=True, transform=transformer, download=True)
test_set = torchvision.datasets.MNIST(root='./sample_data', train=False, transform=transformer, download=True)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=128, shuffle=True, pin_memory=True)
test_loader = torch.utils.data.DataLoader(test_set, batch_size=200, pin_memory=True)
# Load a batch of inputs and outputs to use for evaluation.
x_batch, y_batch = iter(test_loader).next()
x_batch, y_batch = x_batch.to(device), y_batch.to(device)
Then, using the Quantus library, we generate qualitative explanations for some test set samples that we want to evaluate.
# Generate normalized Saliency and Integrated Gradients attributions of the first batch of the test set. a_batch_saliency = quantus.normalise_by_negative(Saliency(model).attribute(inputs=x_batch, target=y_batch, abs=True).sum(axis=1).cpu().numpy()) a_batch_intgrad = quantus.normalise_by_negative(IntegratedGradients(model).attribute(inputs=x_batch, target=y_batch, baselines=torch.zeros_like(x_batch)).sum(axis=1).cpu().numpy()) # Save x_batch and y_batch as NumPy arrays that will be used to call metric instances. x_batch, y_batch = x_batch.cpu().numpy(), y_batch.cpu().numpy()
The following is a visualization of attributions for a given model and input-output pairs. Which is nothing but the saliency map of the digits and gradient map (feature importance map) given by the model.
Because we lack ground truth of what the explanations should look like, it is difficult to make inferences about the explainable evidence that we encounter, the qualitative features of the Saliency and Integrated Gradients explanations may appear to be quite uninterpretable.
We can utilize Quantus to quantitatively analyze the explanation. As a first step, we might want to see how sensitive the explanations are to relatively little changes. We can test the Max-Sensitivity of our explanations by evaluating them against a range of quantitative criteria. This metric examines how the explanations change the most when subjected to minor disturbances.
# Define params for evaluation.
params_eval = {
"nr_samples": 10,
"perturb_radius": 0.1,
"norm_numerator": quantus.fro_norm,
"norm_denominator": quantus.fro_norm,
"perturb_func": quantus.uniform_sampling,
"similarity_func": quantus.difference,
"disable_warnings": True}
Get the scores for saliency and integrated gradients map as follows.
scores_saliency = quantus.MaxSensitivity(**params_eval)(model=model,
x_batch=x_batch,
y_batch=y_batch,
a_batch=a_batch_saliency,
**{"explain_func": quantus.explain, "method": "Saliency", "device": device, "img_size": 28, "normalise": False, "abs": False})
scores_intgrad = quantus.MaxSensitivity(**params_eval)(model=model,
x_batch=x_batch,
y_batch=y_batch,
a_batch=a_batch_intgrad,
**{"explain_func": quantus.explain, "method": "IntegratedGradients", "device": device, "img_size": 28, "normalise": False, "abs": False})
Here are the scores,
Now when the max-Sensitivity scores for the Saliency and Integrated Gradients explanations are compared, we may infer that Saliency is less resilient (scores 0.74 +-0.21std) in this experimental scenario than Integrated Gradients (scores 0.28 +-0.07std).
For details on code and implementations, please refer to this notebook.
Final words
Through this article, we have discussed what explainable AI is. Particularly we have seen Quantus, a Python-based toolkit that collects, organizes, and discusses a wide range of proposed evaluation criteria for explanation approaches. The library is intended to aid in the automation of the XAI quantification process by providing quick, easy-to-understand, and comprehensive summaries of the quality of the offered explanations.
