Don't Let The Replication Crisis Fool You

Half of social science studies don’t replicate. That has become one of the most widely cited facts about modern research. It has also become one of the most misunderstood.

“You can’t trust studies,” people say, “because they don’t replicate.” But how do we know they don’t replicate? Because of studies.

A new report in Nature discusses the results of a seven-year project called Systematizing Confidence in Open Research and Evidence, or Score, which aimed to evaluate the replicability of 3,900 social science papers. Researchers who reran these studies could only reproduce about half of the results. This echoes previous endeavors over the past decade, which found that in fields such as psychology and cancer biology, less than half of replication attempts matched the original results.

The replication problem is real. But this discovery shows that the scientific process, even with its many flaws, can correct itself. The alternative to relying on research is intuition. This works for the challenges of everyday life. It’s less useful when it comes to big questions, identifying abstract patterns or making predictions about complex systems.

Skepticism about replication isn’t the only way people avoid engaging with research. Consider hindsight bias, or the tendency to believe we already knew something once we heard it. It shows up in a familiar way. You describe a research finding. Someone replies, “Did we really need a study for that?” The implication is that researchers waste time studying things that are obvious.

But “obvious” is a slippery idea. It often means little more than this: I can come up with a story that fits the result after I’ve heard it.

Sociologist Samuel Stouffer and his team studied soldier morale in World War II. If he’d discovered that men from rural backgrounds were usually in better spirits than men from cities, it would sound reasonable. Rural men in the 1940s were used to harder conditions. Army life probably felt less stressful to them. Why do we need a study to tell us something so obvious?

But that wasn’t Stouffer’s finding. In fact, he found the opposite. City men adjusted better. That, too, sounds reasonable. City men were accustomed to crowded conditions and frequent contact with strangers. Of course they would be better suited to military service.

Either result feels obvious after the fact. And that’s the problem. When any outcome can be explained away as something we already knew, the word “obvious” loses meaning. It becomes a way to guard the belief that we understand more than we actually do.

There is a better way to think about this. Scientific research is a tool. Like any tool, it can be misused. Poor methods, small samples and selective reporting can produce unreliable results. That is why reforms such as larger sample sizes, making datasets publicly available and preregistration—or specifying hypotheses and methods in advance—have gained traction among social scientists in recent years.

Abandoning trust in research leaves us with our intuitions. And our intuitions are prone to systematic error. We are good at explaining things after they happen. We are much worse at predicting them in advance. Research, even when it fails, forces us to confront that gap.

---

This article was originally published in The Wall Street Journal under the title “Social Science Has a Replication Problem.”