Why Research Goes In the Wrong Direction | Psychology Today

As a biomedical science reporter at NPR, Richard Harris has long known not to take new discoveries at face value. Researchers are bound to delve further into a subject, expanding or complicating scientists’ current knowledge of it. And sometimes, new studies are just plain wrong.

In his new book, Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions (out April 4th), Harris discusses some of the flaws behind misleading research—things like blunt measurement tools, contaminated cell lines, and badly designed experiments. Sloppy studies aren’t caused by one core problem, but unrealistic expectations from journals, a lack of funding, and few available jobs can incentivize scientists to cut corners or even leave them ill-equipped to run an experiment. Ultimately, bad research is detrimental to taxpayers, scientific progress, and patients waiting for a cure.

Rigor Mortis isn’t all gloom and doom. Harris’s book also introduces people and organizations who are encouraging better research. Research mistakes can be prevented, he says, if everyone in the biomedical community is more careful. —Lara McCaffrey

You’ve reported on a wide range of science topics, but you now mostly focus on biomedical science. How did that become your main beat?
I'd been covering climate change and related topics for quite a while, and in 2014, NPR said "We'd like you to switch focus a little bit." So I said "OK!" I actually spent that first year trying to understand how things had changed in biomedicine since I'd last covered it around the year 2000, when I covered the sequencing of the human genome. I came to realize that there were a lot of issues—particularly around funding shortages—that were driving undesirable behaviors in biomedical research, and that ultimately led to my book.

Why did you want to dedicate an entire book to the flaws in the biomedical research process?
I wanted to narrow the topic—presumably there are similar issues, although not necessarily so extreme, in other parts of science. But there has been a fair amount written in the literature pointing out the flaws in biomedicine. The community itself had started to focus its attention on the issues. Then I realized it was a good time to look at what was going on and to also to tap into people who were thinking about how to make it better.

When did you start to realize there was a large amount of questionable research out there?
When I started to look at some of the data that had accumulated about biomedical research, I started to recognize that a fair amount of it deserves skeptical treatment. As journalists, we tend to be attracted to the latest, greatest new finding as opposed to the sixth or eighth or tenth paper on the same subject. So it's a conundrum for journalists, because we obviously want to have the latest greatest, but on the other hand, we don't always do a really great job with circling back to say "Hey remember that study I told you about three years ago? Well, guess what? It turns out not to have panned out."

Was there a specific event or study that brought faulty research to light for you?
One of the studies that really caught my eye was a study by C. Glenn Begley at (the biopharmaceutical company) Amgen. He looked at 53 cancer studies and decided to replicate them. He could only replicate six of them—that’s 11 percent. But there's no question when you talk to people throughout the world of biomedicine that many people say, “Oh yeah, there's lots and lots of stuff in the literature that's wrong.” It's a normal process in science that you get an idea, you publish it, and as it gets refined, you realize whether you're right or wrong. But false leads are actually avoidable. Of course we should expect science to have wrong answers, but we also should hope that scientists take a little extra effort to try to reduce that as much as they can.

You quote social scientist Brian Martinson, who says questionable research is caused by “too many scientists competing for too few dollars, and too many postdocs competing for too few faculty positions. Everything else is symptoms of those two problems.” Do you agree, or do you think bad research is caused by more than that?
I think it's more than that. I think that Martinson is right—that is a core problem and it drives a lot of it. But I've talked to a lot of other scientists who said that's not the whole story. And let's recall that science was far from perfect even 20 years ago, when grant money was easier to find and jobs were easier to land. Sometimes people aren't very well trained and so they end up using the wrong statistical tasks. Or, sometimes, people have too much faith in the ingredients that they're using and they don't stop to really ask, "Do I really know what the implications are of using this antibody or this cell line?"

Your book mentions that sometimes journals don’t double check the validity of studies they choose to publish. An editor at the Journal of Proteome Research, Joshua LaBaer, has stricter publishing guidelines. Should others follow his lead? What protocols should they adopt?
I think science journals should ask the same question that LaBaer asks: Is it a complete study? His rule is basically: Don’t just tell me that gene x is associated with disease y. Those associations are often red herrings. Show me that it’s biologically possible. And I think that's a very good step. There are thousands of journals out there and they all operate under different standards and have different audiences and so on. So I don't think there's any one-size-fits-all rule that you can apply to science journals to make sure they do a better job. But I think it helps to be aware of a lot of the potential pitfalls and make sure that the reviewers and the editors are alert to those and ask the hard questions.