Scientists have now developed a new AI system that converts brain activity into text could result in transforming communication for people who can’t speak or type. The AI-based machine works by using brain implants to track what neurons do when someone is speaking something. According to experts, the system could eventually aid communication for patients who are unable to speak or type, such as those with locked-in syndrome.

Dr Joseph Makin, a co-author of the research from the University of California, San Francisco, said, “We are not there yet, but we think this could be the basis of a speech prosthesis.”

How does the system work?

Explaining the process, the research stated that they recruited some participants with electrode array implants in their brain so that the researchers can monitor their epileptic seizures. Once these participants were read aloud from 50 set sentences multiple times, the team tracks the activity in their brain, also known as the neural activity. Then the data was then fed into a machine-learning algorithm that converted the brain activity data for each spoken sentence into a string of numbers, which was then fed into the next part of the system which turned it into a sequence of words.

According to the researchers, at first, the system was not perfect. Some of its mistakes were — sentences like “Those musicians harmonise marvellously” was decoded as “The spinach was a famous singer”, and sentences like “A roll of wire lay near the wall” became “Will robin wear a yellow lily”. However, gradually the team found the accuracy far higher than previous approaches. 

The researchers further added that the accuracy of the system varied from person to person. For one participant, the accuracy was just 3% of each sentence on average needed correcting, which was technically higher than the word error rate of 5% for professional human transcribers. However, fortunately, the algorithm only handles a small number of sentences.

Researchers also realised that training the algorithm on one participant’s data will mean that less training data was needed from the final user, which is going to make training less difficult for patients.

When asked an expert, Dr Christian Herff from Maastricht University, stated that the research was exciting because the intelligent machine used less than 40 minutes to train the data for each participant, and a limited collection of sentences, rather than the millions of hours typically needed. He said, “By doing so they achieve levels of accuracy that haven’t been achieved so far.” However, he also noted the AI-based system was not ready for many severely disabled patients as the machine relies on the brain activity when spoken out loud by the people.