It’s essential to understand the inherent uncertainties machine learning models carry to ensure fairness, build trust, and improve decision-making. Despite being an important factor, uncertainty is often overlooked in the context of machine learning-assisted decision making.

To this end, IBM released the Uncertainty Quantification 360 (UQ360) open-source toolkit to provide developers and data scientists with a guideline/process to quantify, evaluate, improve, and communicate the uncertainty of machine learning models. The AI toolkit was introduced at the recent IBM Data & AI Digital Developer Conference.

With a guideline in place, as in the case of UQ360, developers will be able to estimate the uncertainty in ML model prediction and evaluate them and, if needed, improve their quality. It also helps in effectively communicating these uncertainties to other stakeholders. It currently provides 11 UQ algorithms.

What is UQ360

Uncertainty can emerge from multiple factors:

Aleatoric uncertainty: It is also called statistical uncertainty. In this case, two examples with the same profiles may give different outcomes every time they are measured. The underlying cause behind this kind of uncertainty is usually the noise associated with data. 

Epistemic uncertainty: It is also referred to as systematic uncertainty. Epistemic uncertainty happens due to ambiguity with the model. This could be due to unclear mapping function, inaccurate measurements, or different functions explaining a given set of training data.

Choosing a UQ estimation method depends on several factors such as the underlying model, type of machine learning task, data characteristics, and the end objective. It is possible that a chosen UQ method does not produce high-quality estimates and could mislead users. In critical situations such as medicine and finance, such lapses can create significant risks. It is therefore important for model developers to evaluate and improve the quality of UQ before deploying an AI system.

This is where UQ360 can help. IBM’s UQ360 is a set of algorithms and taxonomy to quantify uncertainty. It also gives guidelines on improving uncertainty quantification (UQ). Uncertainty quantification exposes the limits of a machine learning model and points at the potential weak links. Such high-quality uncertainty estimates and open communication around this topic can greatly improve and benefit human-AI collaboration.

UQ360 provides a set of metrics to measure the quality of uncertainties produced by different algorithms. These metrics include classification metric, regression metric, and uncertainty characteristic curve. It also contains a set of techniques for improving the quality of the estimated uncertainties.

The UQ360 Python package contains UQ algorithms that help users choose styles, such as descriptions or visualisations, to communicate UQ estimates. It also includes several tutorials and demonstrations depicting how to use UQ across the AI lifecycle.

Wrapping up

IBM said UQ information could be used in high-stakes applications such as medicine, security, and finance to prevent excessive reliance on AI systems and to enable better decision making. 

“We have developed UQ360 to disseminate the latest research and educational materials for producing and applying uncertainty quantification in an AI lifecycle. This is a growing area and we have developed this toolkit with extensibility in mind,” the team said. Further, IBM has urged contributions from the developer’s community to add to the UQ capabilities and explore UQ’s connection to other factors of Trustworthy AI–fairness, robustness, factsheets, and explainability.