Microsoft Research and Harvard University recently came together to release a research paper that demonstrated how an end-to-end learning strategy can be used to optimise the performance of a human and a machine team, by considering the distinct abilities of people and machines. According to the researchers, the objective was to focus machine learning on problem instances that are difficult for humans, while at the same time, recognising instances that are difficult for machines, and thus, seeking human input on them.

With the breakthroughs of artificial intelligence and machine learning, some debatable topics that concern researchers and developers around the globe are – Will AI replace humans? Will AI create an unemployment and job crisis in the future? 

Instances, like when a Japanese man got married to an AI voice assistant, human customer support replaced by chatbots, etc. have left a deep mark in such conversations. However, scientists and researchers believe that these innovations are meant to make humans more productive and efficient. 

Behind The Combined Approach

The researchers introduced methods for optimising team performance, where machines take on some parts of the task, and the rest of the tasks are taken on by humans. The researchers stated,

“We develop approaches that are aimed to train machine learning models to complement the strengths of human beings while accounting for the cost of querying an expert.”

They added, “While the work of the human and machine team can take several forms, here, we focus mainly on situations where a machine takes on the tasks to decide which instances require human inputs, and then fusing both machine and human judgments.”

To accomplish this method, a family of approaches has been proposed to train a machine learning system for human-machine complementarity. In the above figure, the run-time system combines machine predictions with human input. During training, logged human responses have been used to simulate queries to a human. 

Both discriminative and decision-theoretic approaches have been taken into account to optimise the model performance. The baseline approach of this model is first to construct a machine learning model, which predicts the solution to a given task, and then construct a strategy for deciding when to query the human.

Contributions Of This Research

According to the researchers, there are four main contributions as mentioned below:

• Firstly, the researchers proposed a family of approaches to train a machine learning system for human-machine complementarity.
• The researchers demonstrated the benefits of optimising for team performance in human-machine teams for two real-world domains of societal importance; these are scientific discovery, which is a galaxy classification task; and medical diagnosis, which is the detection of breast cancer metastasis.
• The researchers pursued experimental insights about when and how complementarity-focused training provides benefits.
• The researchers analysed how the methods distribute instances to human and machine, and how these allocations reflect differences in relative capabilities. 

Wrapping Up

According to the researchers, this is the first approach to optimise human and artificial intelligence teams by jointly training machine learning systems together with policies for allocating tasks to human experts versus machines; and it has outperformed the individual performance of machines and human beings. 

It studies how machine learning systems can be optimised to complement humans via the use of discriminative and decision-theoretic modelling methodologies. The methods presented are aimed at optimising the expected value of human-machine teamwork by responding to the shortcomings of ML systems, as well as the capabilities and blind spots of humans. 

The researchers stated that the “errors made by the ML model change under joint training, as the model places more emphasis on instances that are difficult for humans. Through joint training, human and machine errors become different in structured ways that can be leveraged by the methods to improve team performance.”

For future work, the researchers stated that there are several opportunities for studying additional aspects of human-machine complementarity across different settings. Some of the directions include optimisation of team performance when interactions between humans and machines extend beyond querying people for answers. This includes settings with more complex, interleaved interactions and different levels of the human initiative as well as machine autonomy. 

Read the paper here.