IBM has recently announced the commercialisation of its AI FactSheets, which were first introduced in 2018. In an official release, the company wrote that it plans to “commercialise key automated documentation capabilities from IBM Research’s AI FactSheets methodology into Watson Studio in Cloud Pak for Data throughout 2021”.

This fact sheet will provide businesses with a framework to define how AI is to be used, to measure a model’s performance, and to generate reports on internal and external transparency.

Can such an AI fact sheet from IBM become an industry standard?

What Is IBM FactSheet & Why Is It Required?

Some of the most important factors to consider while achieving trust in AI are — fairness, safety, explainability, reliability, and accountability. Apart from these factors, it must be accompanied by having a parameter against which the models are measured. A lot of this could be attributed to the increasing usage of AI models and services, even in high-stakes situations such as financial risk assessments, medical diagnosis, talent acquisition, policing, and governance. 

First proposed in 2018, the authors of IBM’s AI FactSheets argued that every AI service developer and the provider must release a Supplier’s Declaration of Conformity in order to assure transparency and trust. As a matter of fact, such a factsheet was akin to nutrition labels that one may find on food items or information sheets for appliances. “Standardising and publicising this information is key to building trust in AI services across the industry,” IBM had then noted.

Along with this, the fact sheets include initial suggestions, information on system operation, algorithms used, testing set-up, performance benchmarks, and fairness and robustness checks.

With the recent announcement from IBM, this AI FactSheet tool will complement IBM Cloud Pak for Data. Notably, IBM has also added new capabilities for Cloud Pak for Data to provide a foundation for AI to run on any cloud while providing enhanced governance and security. This will ensure federated learning to facilitate model training on distributed datasets while assuring security.

AI FactSheets Offer:

• The scope for policy creation which defines what information is collected on models, who can use the model and for what purpose, and the way it should operate.
• The AI FactSheets are designed to help in automatically capturing the model facts as detailed in the FactSheet template throughout the AI lifecycle.
• These FactSheets can offer knowledge on the AI model in multiple formats depending on the preferences of the user and external audience.

Some of the questions from IBM’s FactSheets include:

• Whether the dataset being used for training has a datasheet or data statement?
• What are the biases policies against which the dataset was checked for any possible bias?
• What is the target user for the explanation — machine learning expert, domain expert, consumer, or regulator
• What is the testing methodology followed?
• Robustness policies followed to ensure the model’s protection against possible adversarial attack.
• Expected behaviours if input deviates from the training/testing data
• How is the overall workflow to AI service tracked?

Further, to evaluate the quality of the fact sheet, IBM has also developed a set of quality dimensions which are based on prior research in software development and communication. 

A few of the dimensions:

Credit: IBM

Can It Become An Industry Standard?

To deploy and scale AI, enterprises must invest their trust in their models and the business outcomes of the entire AI lifecycle. IBM says that the new announcement of commercialising the AI FactSheets on Watson will provide easy access to clients of their comprehensive portfolio of AI governance solutions, which in turn will increase transparency, manage risks, and establish trust on AI models.

It is true that apart from such a FactSheet, there is no standard of explaining or establishing the capabilities of an AI model. It forces the customers to make an uninformed decision and go with the common consensus. So naturally, a standardised fact sheet on the lines of IBM FactSheet will help indirectly addressing the issues, as also suggested by industry experts.