This A.I. Used Brain Scans to Recreate Images People Saw

Though it sounds like something out of a science fiction novel, researchers have successfully trained an artificial intelligence system to recreate images people have looked at based on their brain scans. The A.I. generated pictures of objects including a teddy bear, clock tower and airplane after participants viewed similar images.

“The accuracy of this new method is impressive,” says Iris Groen, a neuroscientist at the University of Amsterdam who was not involved in the research, to Science’s Kamal Nahas.

While this brain-scan-to-image A.I. technology is far from ready for public use, researchers say it could someday prove useful for understanding what’s happening inside people’s minds. Once scientists refine the concept a bit more, doctors may eventually be able to use it to help people, such as those suffering from paralysis, to communicate. It might also help neuroscientists interpret dreams or even understand how other species perceive the world around them.

The researchers from Osaka University in Japan are among the ranks of scientists using A.I. to make sense of human brain scans. Their approach to this, however, is the first to use the text-to-image generator Stable Diffusion, which came on the fast-growing A.I. scene in August 2022. Their model is also much simpler, requiring only thousands, instead of millions, of parameters, or values learned during training.

The team shared more details in a new paper, which has not been peer-reviewed, published on the preprint server bioRxiv. They also plan to present their findings at an upcoming computer vision conference, according to Science.

So, how does it all work? Typically, a user inputs a word or phrase that Stable Diffusion—or other similar technologies, such as DALL-E 2 and Midjourney—transforms into an image. This process works because the A.I. technologies have studied lots of existing images and their accompanying text captions—over time, this training allows the technology to identify patterns, which it can then recreate based on a prompt.

The researchers took this training one step further, by teaching an A.I. model to link functional magnetic resonance imaging (fMRI) data with images. More specifically, the researchers used the fMRI scans of four participants who had looked at 10,000 different images of people, landscapes and objects as part of an earlier, unrelated study. They also trained a second A.I. model to link brain activity in fMRI data with text descriptions of the pictures the study participants looked at.

Together, these two models allowed Stable Diffusion to turn fMRI data into relatively accurate imitations of images that were not part of the A.I. training set. Based on the brain scans, the first model could recreate the perspective and layout that the participant had seen, but its generated images were of cloudy and nonspecific figures. But then the second model kicked in, and it could recognize what object people were looking at by using the text descriptions from the training images. So, if it received a brain scan that resembled one from its training marked as a person viewing an airplane, it would put an airplane into the generated image, following the perspective from the first model. The technology achieved roughly 80 percent accuracy.

Indeed, the recreated images look eerily similar to the originals, albeit with some noticeable differences. The A.I.-generated version of a locomotive, for example, is shrouded in a murky gray fog, rather than showing the cheery, bright blue skies of the actual image. And the A.I.’s depiction of a clock tower looks more like an abstract work of art than an actual photograph of one.

The technology shows promise, but it still has some limitations. It can only recreate images of objects included in its training material. And, since the A.I. processed the brain activity of just four people, broadening it to include others would require training the model on each new individual’s brain scans—an expensive and time-consuming process. As such, the technology isn’t likely to become widely accessible to the public—at least in its current form.

“This is not practical for daily use at all,” says Sikun Lin, a computer scientist at the University of California Santa Barbara who was not involved with the project, to New Scientist’s Carissa Wong.

Zooming out, people have broader concerns around A.I. technologies in general. Are they stealing from human artists or violating copyright laws? Will they make police more biased against certain groups of people, contribute to misinformation or invade our privacy? Engineers and ethicists are still grappling with these and many other questions, and these discussions likely will continue for the foreseeable future, even as scientists come up with novel—and potentially beneficial—ways of using A.I.

“When it comes to very powerful technologies—and obviously A.I. is going to be one of the most powerful ever—we need to be careful,” said Demis Hassabis, CEO of the A.I. research laboratory DeepMind, to Time magazine’s Billy Perrigo last year. “It’s like experimentalists, many of whom don’t realize they’re holding dangerous material.”

Sarah Kuta | READ MORE

Sarah Kuta is a writer and editor based in Longmont, Colorado. She covers history, science, travel, food and beverage, sustainability, economics and other topics.