Survival of the sloppiest

A Q&A with Paul Smaldino on the natural selection of bad science

Survival of the sloppiest
Researchers pass down methods to their students, who use what they inherited to compete on the academic job market and become the next generation of researchers. Set up the incentives wrong, and this can lead to the evolution of bad science — even if nobody is actively trying to gain the system. Image: Family Tree of the House of Habsburg by Aegidius Sadeler II (1629)

Preview · Audio edition
audio-thumbnail
Survival of the sloppiest
0:00
/0:00
You're hearing a short preview. Paying members get the full audio edition of this article.
Upgrade to listen

Researchers have relatives they aren't related to: their academic "family tree."

Training to become a researcher is much more like being a medieval apprentice than it is like going to school. At the PhD level — when aspiring researchers actually learn to do research — students train with a senior researcher, their PhD supervisor, in a lab group of typically no more than a dozen or so people. It's a pretty intimate setup that's more like a family than a university course. More than your field or your university, the research culture, norms, network, and methods of your supervisors' lab group largely determine your own research culture, norms, network, and methods — and thereby your chances of a future academic career. Researchers who set their students up for success boast many academic "children" and "grandchildren," and scientists can trace their "academic lineage" back to venerable "academic grandfathers" and grandmothers. In Germany, your PhD supervisor might be called your Doktorvater or Doktormutter. Literally, doctor father or doctor mother.

Research labs obviously do not literally reproduce in the biological sense. But students do inherit and iterate on the scientific methods and norms of their advisors and use what they've inherited to compete to become the researchers of the future. If you assume some natural variation in methods and competition between students to become the student-advising lab leaders of the future, you have everything you need for an evolutionary process: variation, inheritance, and selection. And if selection in the scientific job market tends to reward methods with certain characteristics, then those methods should come to dominate over time.

You might see where this is going. If the scientific job market selects for methods that correlate with incorrect results — say, producing a lot of flashy, surprising new findings rather than let-down null results — might it end up selecting for bad science?

Paul Smaldino, a computational social scientist at the University of California, Merced, has been modeling science as an evolutionary process for a decade now. In 2016, he and Richard McElreath published a paper titled "The natural selection of bad science" that presents a simple model of how sloppy science gets selected, and Paul and his colleagues have been elaborating it on since then. The findings are both concerning and encouraging, depending on how hopeful you are about being able to change the incentive structures that shape the scientific information environment.

I learned about Paul's work earlier this year when he gave a talk at the Complexity Science Hub in Vienna about how journals favoring flashy, novel results can lead wrong conclusions to accumulate in the scientific literature — assuming flashy, novel results are more likely to be wrong than boring ones that mostly just confirm what we already know. Bad outcome, no conspiracy required. (I suspect a similar process operates in science media, but that's a topic for another day.)

As a science journalist, and as a person just generally obsessed with incentives and how they shape information environments, my ears immediately perked up. I wasn't able to chat with Paul much while he was in Vienna, but he was kind enough to hop on a call with me for a Q&A about the evolution of bad (and good) science earlier this year. We discuss the replication crisis, psychic undergraduates, why scientists in some fields might actually be rewarded for being bad at statistics, and why there's reason to be hopeful about making research better.

Our conversation was edited for length and clarity.