From Black Box To Glass Box: This New Explainable AI Algorithm Lifts The Veil On ML Models

The research team included U of T Engineering graduate Mahesh Sudhakar and master’s candidate Sam Sattarzadeh and researchers led by Jongseong Jang at LG AI Research, Canada.

Recently, researchers from the University of Toronto and LG AI Research developed an Explainable AI algorithm (XAI) to help identify and eliminate defects in display screens. Researchers claimed the algorithm outperformed the existing approaches and benchmarks. 

Last year, LG invested in the University of Toronto’s artificial intelligence research to focus on building business-to-business applications. The university has been doing extensive research in deep learning. Meanwhile, LG has been leading the efforts in artificial intelligence with its AI research centre in Toronto and LG Science park, the global research and development arm of the LG group of companies. 

While the LG and U of T team have tapped AI to improve the functionality of LG products such as TV and smartphones, it’s the first time they are venturing into explainable AI territory.

The research team included U of T Engineering graduate Mahesh Sudhakar and master’s candidate Sam Sattarzadeh and researchers led by Jongseong Jang at LG AI Research, Canada. 


The new XAI algorithm addresses the black-box problem in machine learning models. Researchers said, in a black-box model, training data could be provided in the form of millions of labelled images. The algorithm learns from this data and eventually associates certain input features with certain output features. The machine decides for itself which aspect of the image to pay attention to. 

For industries that rely on sensitive data like healthcare, law, and insurance, such black-box models may present challenges. Any actions based on biased information may lead to severe consequences. For instance, machine learning models may determine a patient has a 90 percent chance of having a tumour. The margin for error on such a critical diagnosis is low. And the black-box nature of the model means doctors have no clue how the algorithm arrived at the diagnosis, which can have serious consequences.

Researchers, therefore, proposed a glass-box approach that makes the decision-making transparent. They run the XAI algorithms with traditional algorithms to audit the validity and the level of training performance. It also allows carrying out tasks such as debugging and finding training efficiencies. 


Researchers used two methodologies to develop XAI algorithms — backpropagation and perturbations. 

For instance, in backpropagation-based methods, only the local attributions are represented, making them unable to measure global sensitivity. This drawback is addressed by image perturbation techniques used in recent works such as RISE and Score-CAM. 

Perturbation methods sacrifice speed for accuracy and involve changing data inputs and tracking the corresponding outputs to determine the necessary compensation. However, researchers said feedforwarding several perturbed images in these works make them very slow. Explanation maps produced by CAM-based methods suffer from a lack of spatial resolution as they are formed by combining the feature maps in the last convolutional layer of CNNs, which lack spatial information on the captured attributions.

“In this work, we delve deeper into providing a solution for interpreting CNN-based models by analysing multiple layers of the network. Our solution concentrates on mutual utilisation of features represented inside a CNN in different semantic levels, achieving class discriminability and spatial resolution simultaneously,” the researchers stated. 

Borrowing productive ideas from different approaches, the researchers formulated a four-phase explanation method. In the first three phases, information extracted from multiple layers of the CNN is represented in their accompanying visualisation maps. These maps are then combined via a fusion module to form a unique explanation map in the last phase. They also used techniques such as block-wise feature explanation, feature map selection and attribute-based input sampling to get results. 

Researchers hope XAI algorithms could be used in areas where it’s critical to understand how machine learning makes its decision. The algorithm has a lot of application potential in product features, manufacturing, supply chain, etc. 

Read the complete paper here. 

Download our Mobile App

Srishti Deoras
Srishti currently works as Associate Editor at Analytics India Magazine. When not covering the analytics news, editing and writing articles, she could be found reading or capturing thoughts into pictures.

Subscribe to our newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day.
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Our Recent Stories

Our Upcoming Events

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

6 IDEs Built for Rust

Rust IDEs aid efficient code development by offering features like code completion, syntax highlighting, linting, debugging tools, and code refactoring

Can OpenAI Save SoftBank? 

After a tumultuous investment spree with significant losses, will SoftBank’s plans to invest in OpenAI and other AI companies provide the boost it needs?

Oracle’s Grand Multicloud Gamble

“Cloud Should be Open,” says Larry at Oracle CloudWorld 2023, Las Vegas, recollecting his discussions with Microsoft chief Satya Nadella last week.