From Black Box To Glass Box: This New Explainable AI Algorithm Lifts The Veil On ML Models

The research team included U of T Engineering graduate Mahesh Sudhakar and master’s candidate Sam Sattarzadeh and researchers led by Jongseong Jang at LG AI Research, Canada.

Recently, researchers from the University of Toronto and LG AI Research developed an Explainable AI algorithm (XAI) to help identify and eliminate defects in display screens. Researchers claimed the algorithm outperformed the existing approaches and benchmarks. 

Last year, LG invested in the University of Toronto’s artificial intelligence research to focus on building business-to-business applications. The university has been doing extensive research in deep learning. Meanwhile, LG has been leading the efforts in artificial intelligence with its AI research centre in Toronto and LG Science park, the global research and development arm of the LG group of companies. 

While the LG and U of T team have tapped AI to improve the functionality of LG products such as TV and smartphones, it’s the first time they are venturing into explainable AI territory.


Sign up for your weekly dose of what's up in emerging technology.

The research team included U of T Engineering graduate Mahesh Sudhakar and master’s candidate Sam Sattarzadeh and researchers led by Jongseong Jang at LG AI Research, Canada. 


The new XAI algorithm addresses the black-box problem in machine learning models. Researchers said, in a black-box model, training data could be provided in the form of millions of labelled images. The algorithm learns from this data and eventually associates certain input features with certain output features. The machine decides for itself which aspect of the image to pay attention to. 

Download our Mobile App

For industries that rely on sensitive data like healthcare, law, and insurance, such black-box models may present challenges. Any actions based on biased information may lead to severe consequences. For instance, machine learning models may determine a patient has a 90 percent chance of having a tumour. The margin for error on such a critical diagnosis is low. And the black-box nature of the model means doctors have no clue how the algorithm arrived at the diagnosis, which can have serious consequences.

Researchers, therefore, proposed a glass-box approach that makes the decision-making transparent. They run the XAI algorithms with traditional algorithms to audit the validity and the level of training performance. It also allows carrying out tasks such as debugging and finding training efficiencies. 


Researchers used two methodologies to develop XAI algorithms — backpropagation and perturbations. 

For instance, in backpropagation-based methods, only the local attributions are represented, making them unable to measure global sensitivity. This drawback is addressed by image perturbation techniques used in recent works such as RISE and Score-CAM. 

Perturbation methods sacrifice speed for accuracy and involve changing data inputs and tracking the corresponding outputs to determine the necessary compensation. However, researchers said feedforwarding several perturbed images in these works make them very slow. Explanation maps produced by CAM-based methods suffer from a lack of spatial resolution as they are formed by combining the feature maps in the last convolutional layer of CNNs, which lack spatial information on the captured attributions.

“In this work, we delve deeper into providing a solution for interpreting CNN-based models by analysing multiple layers of the network. Our solution concentrates on mutual utilisation of features represented inside a CNN in different semantic levels, achieving class discriminability and spatial resolution simultaneously,” the researchers stated. 

Borrowing productive ideas from different approaches, the researchers formulated a four-phase explanation method. In the first three phases, information extracted from multiple layers of the CNN is represented in their accompanying visualisation maps. These maps are then combined via a fusion module to form a unique explanation map in the last phase. They also used techniques such as block-wise feature explanation, feature map selection and attribute-based input sampling to get results. 

Researchers hope XAI algorithms could be used in areas where it’s critical to understand how machine learning makes its decision. The algorithm has a lot of application potential in product features, manufacturing, supply chain, etc. 

Read the complete paper here. 

More Great AIM Stories

Srishti Deoras
Srishti currently works as Associate Editor at Analytics India Magazine. When not covering the analytics news, editing and writing articles, she could be found reading or capturing thoughts into pictures.

AIM Upcoming Events

Regular Passes expire on 3rd Mar

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Early Bird Passes expire on 17th Feb

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, Virtual
Deep Learning DevCon 2023
27 May, 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

What went wrong with Meta?

Many users have opted out of Facebook and other applications tracking their activities now that they must explicitly ask for permission.