Loss-of-Confidence Project Aims to Foster Culture of Self-Correction in the Scientific Record
Psychologists can submit a statement on how they lost confidence in one of their own findings to help end the stigma around admitting errors
/https://tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/e4/86/e48655b7-2c44-47e0-b553-72db89a60cc5/psychology.jpg)
In the last few years, the “science” part of social science has been put to the test. A 2015 study found that out of 100 well-known psychology experiments, only 39 were successfully reproduced. A study from last summer found that only 13 of 21 psychology studies published in the well-respected journals Science and Nature were reproducible. Cornerstones of the discipline like the “marshmallow test” and the concept of “ego depletion” are casualties of the ongoing “reproducibility crisis.”
That’s where the Loss-of-Confidence Project comes in. Brian Resnick at Vox reports the project, led by Julia Rohrer, a psychologist at the Max Planck Institute for Human Development, wants to be a refuge for psychologists to declare they no longer back the conclusions of their own research.
To that end, researchers can fill out a form detailing why they no longer trust their conclusions on the project’s website until January 31, at which point the loss of confidence statements will be published together. By coming out together, Rohrer hopes the project will destigmatize the practice of self-correction. While interest in the idea is high, perhaps tellingly, the project's first round, which was open from December 2017 to July 2018, only received six loss-of-confidence statements, which, the team writes, "might be taken to suggest an important discrepancy between researchers’ ideals and their actual behavior."
“I do think it’s a cultural issue that people are not willing to admit mistakes,” Rohrer tells Resnick. “Our broader goal is to gently nudge the whole scientific system and psychology toward a different culture.
The hope is getting to a place where admitting a mistake becomes routine, not the death-knell of a scientist's career. “People will defend their scientific claims until their death,” she tells Dalmeet Singh Chawla at Undark. “As scientists, we should be aware that people are often wrong.”
For now, the project is only open to studies published in the field of psychology. As the team explains in an FAQ: "The main reason behind this decision is we want to keep the project manageable and lack the expertise to properly evaluate the eligibility of statements from other fields. Of course, if you want to start a similar project for a different field, you are more than welcome to do so."
Importantly, the loss of confidence statement can only be submitted by someone "willing to take primary responsibility for any methodological or theoretical problems that have caused you to lose confidence in your findings since their publication." Rohrer and her collaborators, who include Tal Yarkoni of the University of Texas at Austin and Christopher Chabris, at the Geisinger Health System in Pennsylvania, do not want the project to become a whistleblower system or a way to criticize other researchers. Instead, they say, it’s about taking personal responsibility.
Psychologist Rebecca Willén, who is part of the Loss-of-Confidence project, and is now disclosing her own loss-of-confidence statements on her website, says this kind of system is better in the long run. “Take the chance of disclosing now,” she tells Chawla. “Once this phase is over, it’s more likely that retroactive disclosures will result in negative consequences for your career.”
More technical fixes could also help improve the state of research—Chabris, one of the project leaders, suggests that researchers should pre-register their study designs before embarking on an experiment. That would make it more difficult for a team to change methods mid-stream or cherry pick results. It also makes reproducing an experiment easier. “[That] sort of builds humility into the structure of the scientific enterprise,” Chabris says. “We’re not all-knowing and all-seeing and perfect at our jobs, so we put [the data] out there for other people to check out, to improve upon it, come up with new ideas from and so on.”
Chris Havergal at Times Higher Education reports that publishers too, could help foster a new culture of self-correction. By offering other options beyond the dreaded “retraction,” which means an erroneous paper has been “unpublished” by a journal, they could allow “voluntary withdrawal” or other terms that let the researchers correct themselves.