Researchers Use A.I. to Decode Words From Brain Scans

Researchers have created a non-invasive brain decoder that can translate stories heard by participants into a string of text, based on their MRI scans. The team says this technology could one day aid communication in people who are mentally conscious but physically unable to speak, such as stroke patients. 

“For a noninvasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences,” Alex Huth, a professor of neuroscience and computer science at the University of Texas at Austin, says in a statement. “We’re getting the model to decode continuous language for extended periods of time with complicated ideas.”

Huth and his colleagues collected hours of data from three participants listening to the podcasts “Modern Love” and “The Moth Radio Hour” while connected to a functional MRI (fMRI) scanner, which recorded the participants’ blood oxygen levels in parts of their brains. This data was used to train an A.I. model to match specific brain activity patterns with strings of words, the team reports in a new study published Monday in Nature Neuroscience. The decoder used GPT-1, an earlier version of the technology that powers ChatGPT. 

When the participants listened to new stories, the tool couldn’t spit back the exact words verbatim, but it could convey the gist. For example, a participant hearing “I don’t have my driver’s license yet” had their thoughts translated to: “She has not even started to learn to drive yet.” (The model is still “uniquely bad” at pronouns, Huth tells HealthDay News’ Dennis Thompson.)

In another test, the participants didn’t listen to anything—they just imagined saying words, and the tool could decipher their general thoughts. The researchers also had participants watch muted animated short films. Though the model was trained only on spoken words, it could still generate a language description of what was happening.

“I was more surprised by the video than the imagined speech,” Huth tells Scientific American’s Allison Parshall. “I think we are decoding something that is deeper than language.” 

Other methods used to decode brain activity usually involve surgically placing sensors on a person’s brain. While this is faster and more accurate, it is also much more invasive than using fMRIs and carries risks of infection, bleeding and seizures. fMRI data can be easily collected in healthy patients without exposing them to these risks, Mitchell Elkind, chief clinical science officer of the American Heart Association and a neurologist at Columbia University, tells HealthDay.

Reading minds raises a multitude of ethical questions about brain privacy, but the researchers say their tool doesn’t work without willing cooperation. They tested whether the decoding could be consciously resisted if the participants simply thought of something else, and they found that it could. And decoders that had not been trained on only one subject’s personal data “performed barely above chance” per the study. In other words, the A.I. won’t be able to interpret your thoughts unless it has first studied hours-worth of your brain scans.

“I don’t think that this is going to be used to violate people’s privacy,” Francisco Pereira, a scientist at the National Institute of Mental Health who has worked on brain decoders for decades and was not involved in the study, tells STAT News’ Brittany Trang. “Or if it is, and people are in a situation where they can compel you to be in the scanner for 16 hours and somehow get you to think about things, you’re already in a pretty compromised situation.”

Still, as technology advances, future versions could be created that don’t require patient cooperation, write the authors. And even if decoders are inaccurate, they could be “misinterpreted for malicious purposes,” per the study. Experts say raising awareness of these risks and enacting policies to protect mental privacy is critical. 

“This is all about the user having a new way of communicating, a new tool that is totally in their control,” David Moses, a researcher of automatic speech recognition at the University of California, San Francisco, who was not involved in the study, tells NPR’s Jon Hamilton. “That is the goal, and we have to make sure that stays the goal.”

Margaret Osborne | | READ MORE

Margaret Osborne is a freelance journalist based in the southwestern U.S. Her work has appeared in the Sag Harbor Express and has aired on WSHU Public Radio.