Should We Trust Social Science Research?
The latest on the replication crisis
In Case You Missed It…
As a reader of The Nature-Nurture-Nietzsche Newsletter, you’ve no doubt heard of the replication crisis: the finding that only around half of studies in the social sciences replicate. Well, here’s a question for you: Is that finding itself replicable?
A massive new study by Andrew Tyner and colleagues, published in the journal Nature, addresses that thorny question, and comes to a strong conclusion: Yes.
The researchers attempted to replicate 274 findings drawn from 164 papers published between 2009 and 2018, spanning 54 journals across the social and behavioral sciences. The replication studies were well-powered, with a greater than 99% chance of detecting the original effect if the effect were real. The teams used the original materials wherever possible, and the replication plans were peer-reviewed in advance. In short, this was about as careful and systematic a replication effort as you could hope for.
What did the researchers find? The headline result is that only around 55% of the original results held up. Not only that, but among those that did replicate, the effect sizes were notably smaller in the replications - roughly half the size of the originals.
Different fields showed somewhat different replication rates. As the graph below shows, education was in first place, with economics bringing up the rear. My own field of psychology fell somewhere in the middle of the pack. That’s not much of a boast, though, given that there’s only about a 50% chance that any given psychological finding will replicate.
On the one hand, this is somewhat depressing news. On the other, we shouldn’t forget that some findings in psychology are as solid as anything else in science. As I argue in the essay linked below, not everything you were taught in intro psych was false!
You can access the Tyner paper here.
You can request a free copy here.
In a few weeks, I’ll be publishing a piece on the Top 10 most replicable findings from behavior genetics. Stay tuned!
Follow me on Twitter/X for more psychology, evolution, and science.
How You Can Support the Newsletter
This post was free to read for all. If you like what I’m doing with The Nature-Nurture-Nietzsche Newsletter, and want to support my work, there are several ways you can do it.
Like and Restack: Click the buttons at the top or bottom of the page to boost the post’s visibility on Substack.
Share: Send the post to friends or share it on social media.
Upgrade to Paid: A paid subscription gets you:
Full access to all new posts and the archive
Full access to exclusive excerpts from my forthcoming book, A Billion Years of Sex Differences
Full access to exclusive content such as my “12 Things Everyone Should Know” posts, Linkfests, and other regular features
The ability to post comments and engage with the N3 Newsletter community
If you could do any of the above, I’d be hugely grateful. The support of readers like you helps keep this newsletter going and growing.
Thanks!
Steve