Model Explainability Validates AI For Safety-Critical Systems, Says Prashant Rao, MathWorks India

Model Explainability Validates AI For Safety-Critical Systems, Says Prashant Rao, MathWorks India
Image © Model Explainability Validates AI For Safety-Critical Systems, Says Prashant Rao, MathWorks India


The world has changed significantly since the industry tracked the AI trends for 2020, a year ago. While some industries have cut back on AI investment in the near term in the present scenario, others have increased their investment above what they had forecasted, offsetting the loss. Many are even using this time to invest in upskilling through remote learning, with AI-themed courses, which is currently among the most sought after by the engineering and scientific community. And the goal of these companies is to make sure that they are relevant, primed and ready to take on more AI projects in 2021. 

To understand the industry better and the key AI trends of 2021, we got in touch with analytics visionary, Prashant Rao, the senior manager at MathWorks. While talking about the massive changes the world has witnessed amid pandemic, Rao, spoke about how AI has massively penetrated various disciplines, across industry and academia. Further, he has also identified model explainability as one of the critical aspects that the industry will witness in the coming year.


Sign up for your weekly dose of what's up in emerging technology.

Edited excerpt — 

What are your thoughts on the prevalence of artificial intelligence within engineering and science disciplines, across industry and academia? Is AI aligning the direction of engineering, computer science, data science and technology as a whole?

The first tool engineers and scientists will be looking out for developing innovative solutions in solving problems and building applications in 2021, is artificial intelligence. It will be used to completely replace or augment traditional techniques in mathematics, physics, and engineering disciplines such as controls and signal processing. 

Further, artificial intelligence will continue to help bring answers in order to previously unsolved problems and enhance existing solutions. Reduced order-modelling is one such example where artificial intelligence has replaced large high-fidelity and computationally expensive models with a much more efficient solution. 

Academia too, will see an increase in the amount of research where AI is embedded in engineering and scientific applications. This trend can be foreseen because the next generation of engineers and scientists are now learning to use artificial intelligence, thanks to the rise of “AI + X” courses, where AI is integrated into traditional modules such as AI plus Control Theory.

Alongside, engineers will continue to work with data scientists using AI models to enhance existing applications or discover new innovative solutions to the projects they’re working on. However, creating a successful AI-based system is more than just developing a model. It requires model lifecycle management, including training, deploying, monitoring, and updating the model for the system in which it resides. To do this efficiently, these processes need to be automated, robust, and well maintained. 

In 2021, engineers will augment their workflows to include development best practices, such as model and data versioning, and production pipelines with information technology. These processes will be required to support the AI-enabled systems that will operate in real-world environments over years and even decades.

What is MathWorks’ vision for making AI tools more accessible to the community of India?

The emphasis is on increasing the accessibility of MathWorks products to the community of engineers and scientists spanning across academia, government, and industry. Our Campus-Wide License offering for academic institutions enables unlimited access to our products to students and faculty in an institution. Other key players in the ecosystem are accelerators and startups. We have partnered with several accelerators so that startups hosted within these accelerators can have access to the products at no cost for the first year. 

Additionally, we work with startup companies to offer them licenses at a discounted rate. It is also important that we support members of the community to adopt these necessary tools. Thus, in addition to many demos, tutorials and onramp resources available online, we engage directly with the customers to understand their problem statements and help guide them toward realising their solutions.

How is model explainability helping companies to gain the confidence to use AI within safety-critical systems?

Artificial intelligence has long been considered a black-box approach to modelling systems because the way it operates is largely unknown. As more explainability methods are produced by researchers and more software vendor tools include them, industry practitioners will more readily adopt AI innovations within their workflows. 

Even engineers and scientists are beginning to understand why a model is making certain decisions and the limits at which a model can operate safely. For this, they are running experiments to explain how a model operates in various scenarios and uses visualisations to understand the inner workings of a model when it doesn’t behave as it should. Understanding this complex mechanism drives innovation in the verification and validation of artificial intelligence for safety-critical systems. Further, automotive, aerospace, and medical standards committees, such as EUROCAE and the FDA, should continuously work on the levels needed for these certifications.

How is MathWorks enhancing rapid control prototyping and hardware-in-the-loop testing capabilities with QNX Neutrino Realtime Operating System?

Earlier this year, MathWorks announced that Simulink Real-Time had been enabled with the QNX Neutrino RTOS, a multi-process 64-bit POSIX-compliant real-time operating system from BlackBerry. This allows us to offer engineers enhanced rapid control prototyping and hardware-in-the-loop (HIL) testing with Model-Based Design. This multi-process 64-bit operating system is widely deployed in life and safety-critical systems for vehicles, medical devices, industrial controls, rail, robotics, aerospace and defence. 

The update builds on the existing Simulink Real-Time and Speedgoat integration enables engineers and developers to extend their Simulink models with I/O driver blocks. This also allows engineers to automatically build real-time applications, develop instrumentation, and carry out automated runs on a target computer. 

With this, the engineers can easily use a real-time simulation of a virtual system to replace a physical system in order to reduce testing costs. These physical systems can be a vehicle, aircraft, or a robot. The QNX Neutrino RTOS enables new workflows, making real-time computing problems easier to solve, particularly when multiple tasks compete for a system’s resources.

Alongside the hardware for Simulink Real-Time and Speedgoat are designed to collaborate in order to build real-time systems for various environments like desktop, lab, or field.

What are your views on increased reliance on simulation and testing with 3D and realistic scenarios? How’s MathWorks participating?

An important step toward formal verification and validation of AI for safety-critical systems is to test every possible scenario in which the system will operate. For self-driving cars, this step is currently performed physically through road testing with the aid of a human driver. Physical testing drastically limits the variety of scenarios and increases the time it takes to capture all the critical edge cases. 

In 2021, engineers will look to leverage recent advances in software tools with a 3D simulation that eliminates physical testing. They will integrate their AI models with traditional methods such as Model-Based Design for modelling physical systems and then perform automated testing against various simulated 3D scenarios.

At MathWorks, we provide a comprehensive platform for solving AI challenges based on decades of supporting complex engineering projects. Not only do we empower engineers with varying AI experience by helping them build better AI datasets, tackle integration challenges and reduce risk, but also help in continuously testing AI models in a system-wide context. Each year, MathWorks unveils new products, major updates, capabilities and features in a twice-a-year cadence – for the past several years, with a focus on artificial intelligence. We expect to continue maintaining this focus based on our learnings from customers, how we see the market and the applications of AI across evolving industries. 

More AI models are getting deployed on more low power, low cost embedded devices. What are your thoughts on it? Also, what is MathWorks’ plan for the future?

The options to incorporate AI into more edge-based systems are increasing, and engineers are taking advantage of expanded hardware support for more low-cost, low-powered devices including FPGAs (Field Programmable Gate Arrays), ECUs (Electronic control unit), and MCUs (microcontroller unit). As a matter of fact, software vendors enable this innovation by extending capabilities to non-chip experts who can apply advanced techniques once reserved for embedded systems engineers. 

Techniques such as quantisation and pruning, which reduces the size of the deployed model, and efficient pre-trained models available in the deep learning community will enable efficient deployment of artificial intelligence and will allow broader adoption of AI-based systems in 2021.

With over 35 years of helping engineers and scientists, our focus, at MathWorks, remains on offering them the tools and technologies they need to do their best work. Our purpose has always been to change the world by accelerating discovery, innovation, development, and learning in engineering and science.

More Great AIM Stories

Sejuti Das
Sejuti currently works as Associate Editor at Analytics India Magazine (AIM). Reach out at

Our Upcoming Events

Conference, in-person (Bangalore)
MachineCon 2022
24th Jun

Conference, Virtual
Deep Learning DevCon 2022
30th Jul

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM