IBM’s Research team at Haifa built FreaAI to detect flaws in machine learning models by automatically scrutinising human-interpretable chunks of data to predict when algorithms are working and when they are not.
The researchers addressed the challenge of finding under-performing data slices to validate the ML performance of ML solutions. They developed automated slicing heuristics and implemented them in FreaAI, such that the resulting slices are correct, statistically significant and explainable.
“It might be acceptable in some cases to have an incorrect answer from time to time, but it’s vital that we are able to understand and control the extent of a mistake and the circumstances under which it could occur,” the IBM blog said.
For example, take a health insurance firm. Here, FreaAI could be used to uncover the error rate in insurance approval or rejection and fix the issue. Going forward, FreaAI will be able to automatically discover inaccuracies in an AI model and suggest course corrections.
Image Credits: IBM
IGNITE is an IBM platform that pools automated testing services. The Big Blue incorporated FreaAI to IGNITE, along with additional testing technologies from their collaborators in the India Research Lab, to provide IBM clients with Full AI Cycle Testing (FACT) capabilities.
Today, machine learning models are the key drivers of business decisions. And, these ML models must be updated over time to neuter ‘Model Drift’ risks. Model drift can be classified into two broad categories.
- Concept drift occurs when the statistical properties of the target variable changes leading to a broken model.
- Data drift happens when the statistical properties of the predictors change.
Testing has traditionally been an exercise in optimising detection based strategies for correcting business processes. Machine learning and neural networks offer obvious advantages in software testing compared to traditional testing:
- Traditional testing tools can test and provide results automatically, but they still require human oversight.
- Traditional test automation technologies can’t identify which tests to run without human supervision; thus, they wind up performing every test or a predetermined set of tests.
- AI in test automation tools can intelligently pick which tests to run and then trigger them automatically based on test status, recent code modifications, available data and associated code metrics.