New Study Calls the Reliability of Brain Scan Research Into Question

When functional magnetic resonance imaging (fMRI) was introduced in the late 1990s, it drew raves for its ability to show brain activity—and concerns that it might be the modern equivalent of phrenology. Now, that debate could spring to life again with revelations that the popular imaging technology could have been flawed for years. As Kate Lunau writes for Motherboard, new research suggests that software used to analyze fMRI results could invalidate up to 40,000 brain activity studies.

Science never operates in a vacuum—reproducing results over and over again is central to research. But a new paper published in the journal PNAS calls the reproducibility of fMRI studies into question, Lunau writes. The analysis examined resting-state fMRI data from 499 healthy people. Researchers split the people into groups and used three statistical packages commonly used to analyze fMRI data to conduct three million comparisons. 

Since the data used were of people whose brains were not particularly active, so they should not have shown any significant trends of neural activity. Researchers expected to find false positives—that is, results showing that people’s brains were not at rest—about five percent of the time. But that’s where the expected results broke down: Rather than showing a five percent chance of finding a false positive, the analysis revealed a 70 percent chance. 

A bug in one of the software programs used to analyze fMRIs seems to have been at least partly to blame. When the researchers reported their findings to software manufacturers, writes Lunau, they responded with their own analyses and, in one case, code changes. But the study calls into question decades of research that relies on fMRI studies that used the flawed code.

“It is not feasible to redo 40,000 fMRI studies, and lamentable archiving and data-sharing practices mean most could not be reanalyzed either,” the team writes. 

The study has caused a stir among scientists who rely on fMRI. But how bad is the problem? Not as bad as you might think, says Discover’s “Neuroskeptic.” The commentator points out that the problem only applies to one statistical package and that up to 70 percent of studies containing at least one false positive does not mean that 70 percent of studies are, in fact, invalid or false. Further, writes Neuroskeptic, the problem only affects a small percentage of brain studies—those that deal with brain activation. 

Regardless, the study is likely to play into a bigger debate in the field of science and the brain: reproducibility. Since a gigantic international effort called the ability of psychology studies to be reproduced last year, the debate about how to make research more reliable and reproducible has heated up. (The study in question was controversial and continues to be debated, especially among the psychology community.) In May, the journal Science published the results of a study of 1,500 researchers on reproducibility. Over 70 percent reported they had tried and failed to reproduce others’ research, and more than 60 percent listed selective reporting and pressure to publish as reasons studies that are not reproducible are published. More than half of the respondents (52 percent) called reproducibility “a significant crisis” in science—unsurprising, given that scientists have trouble agreeing what the word even means.

Don’t despair, though: As Monya Baker writes for Slate, recent reproducibility kerfuffles are likely good for science and spur additional visibility and funding for more reliable results. "Taken together,” writes Baker, attempts to make work more reproducible "...could stop researchers from blithely following up on work that cannot be reproduced or charging down paths others have charted as dead ends." Studies like the one that calls fMRI brain activation results into question are sobering, but even as they potentially unseat years of research, they may push science into a more reliable future. 

Erin Blakemore | | READ MORE

Erin Blakemore is a Boulder, Colorado-based journalist. Her work has appeared in publications like The Washington Post, TIME, mental_floss, Popular Science and JSTOR Daily. Learn more at erinblakemore.com.