If it feels like you can find a study to back up any harebrained idea these days, you actually might be right, a new study says. On Thursday we, as humans, arrived at the zenith of irony when a study of studies was published and found—you guessed it—many studies were totally full of it, and overstated their findings. To be fair, the Reproducibility Project investigation, led by a University of Virginia psychologist did not set out to debunk every study in every discipline and only dealt with psychological-based studies published in three leading journals. The critique focused on social science research, so this does not necessarily disprove either of the studies you just read saying seven cups of coffee a day will, and will not, make you live longer.
From these scholarly publications, a group of researchers took 100 published studies and did the experiments again for themselves, in collaboration with the original authors, and found a whopping 60 didn’t hold up. The new findings didn’t necessarily contradict the previous findings, but in the replication, often using more subjects, the researchers found the results were not statistically significant, a measure of whether an outcome likely occurred by chance or not.
Here are some examples of watered down findings from the New York Times:
The new analysis focused on studies published in three of psychology’s top journals… The studies included research into topics like mate preference, decision-making, word memory and willpower. Among the studies that did not hold up was one on free will. It found that participants who read a passage arguing that their behavior is predetermined were more likely than those who had not read the passage to cheat on a subsequent test. Another was on the effect of physical distance on emotional distance. Volunteers asked to plot two points that were far apart on graph paper later reported weaker emotional attachment to family members, compared with subjects who had graphed points close together. A third was on mate preference. Attached women were more likely to rate the attractiveness of single men highly when they were highly fertile, compared with when they were less so. In the reproduced studies, researchers found weaker effects for all three experiments.
“The report appears at a time when the number of retractions of published papers is rising sharply in wide variety of disciplines,” the Times notes. “Scientists have pointed to a hypercompetitive culture across science that favors novel, sexy results and provides little incentive for researchers to replicate the findings of others, or for journals to publish studies that fail to find a splashy result.”