Facebook Gives Away Captum For Free: Its Model Interpretability Library For PyTorch

Facebook AI released Captum 0.4 with new functionality for model understanding.

Concept-based interpretability tools assist artificial intelligence researchers and engineers in designing, developing and debugging AI models. Additionally, these tools help understand the working of AI models, helping businesses assess if the models deliver accurate results and reflect their values. One such interpretability tool is Facebook’s Captum. 

Captum is a powerful and flexible interpretability library for PyTorch, which makes algorithms for interpretability readily available for access to the PyTorch community. Captum supports model interpretability across modalities — vision, text; additionally allowing researchers to add new algorithms and benchmark their work against existing algorithms available in the library. Finally, it offers tools to help developers uncover vulnerabilities using metrics and adversarial attacks. 

Recently, Facebook released its latest version of Captum — Captum 0.4, which has new functionality for model understanding. Facebook AI has added tools to evaluate model robustness, improvements to its existing attribution models, and new attribution methods in the latest version. 

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Removal of Statistical Biases 

Captum 0.4 adds testing with Concept Activation Vectors (TCAV) that allows researchers and engineers to assess how different user-defined concepts affect a model’s prediction. It can also be used for checking algorithmic and label bias, which might be embedded in networks. 

Additionally, TCAV’s capabilities expand beyond currently available attribution methods, enabling researchers to quantify the importance of the various inputs and quantify the impact of concepts like gender and race on a model’s prediction. 

Captum 4.0 comes with generically implemented TCAV, allowing users to define custom concepts with example inputs for different modalities — vision and text. 

Source: Facebook AI

The graphs above showcase visualised distributions of TCAV scores for the sensitivity analysis model implemented in a Captum tutorial. As a data set, Facebook AI researchers have used movie ratings with positive sentiment. The graphs visualise TCAV scores for positive adjectives concepts along with five sets of neutral terms concepts. The positive adjectives concept is more important than for both convolutional layers across all the five neutral concept sets, indicating the importance of positive adjectives in predicting positive sentiment. 

Robust AI Models 

Deep learning techniques are often vulnerable to adversarial inputs that, in turn, can fool the AI model and be imperceptible to humans. Captum 0.4 comes with tooling to support the improved understanding of the limitations and vulnerabilities of a model. As a result, the AI system will react to unforeseen issues and make necessary changes to avoid harming or otherwise negatively affecting people. 

Captum 0.4 also comes with tools to understand the robustness of the model, including implementations of adversarial attacks and robustness metrics to evaluate the impact of different attacks or perturbations on a model. The robustness metrics included in its latest version are: 

  • Attack Comparator: It allows users to quantify input perturbation’s impact. This includes text augmentation, and torchvision transforms. It also helps quantify adversarial attacks on a model and compare the impact of the various attacks.  
  • Minimal Perturbation: It identifies the minimum perturbation required to cause a model to misclassify the perturbed input. 

This new tooling enables developers to understand potential model vulnerabilities better and analyse counterfactual examples to comprehend a model’s decision boundary better. 

Source: Facebook AI

Relevance Propagation & Attribution

Facebook AI has implemented a new attribution algorithm, in collaboration with Technische Universitat Berlin, to offer a new perspective for explaining model predictions. Captum 0.4 adds both LRP and a layer-attribution variant — layer LRP. 

The Layer-wise Relevance Propagation (LRP) algorithm is based on a backward propagation mechanism applied sequentially to all layers of the model. The model output score represents the initial relevance which is then decomposed into values for each neuron of the underlying layers. 

Additionally, in Captum 0.4, the Facebook AI team has added tutorials, improvements and bug fixes to the existing attribution methods. 

Debolina Biswas
After diving deep into the Indian startup ecosystem, Debolina is now a Technology Journalist. When not writing, she is found reading or playing with paint brushes and palette knives. She can be reached at debolina.biswas@analyticsindiamag.com

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox