RACHEL MARTIN, HOST:
Not all science is created equal. More than 6,000 scientific papers that are related to the coronavirus have been posted online without peer review. Some are poor in quality. Others have vital information. A number of them are posted by little known scientists at unfamiliar institutions. NPR science correspondent Richard Harris reports on how you go about separating the good science from the bad.
RICHARD HARRIS, BYLINE: The beauty of science is that the facts are supposed to speak for themselves.
THEO BLOOM: The ideal world, we would simply read the paper, look at the data and not be influenced about where it came from or who it came from.
HARRIS: But Theo Bloom, the executive editor of the medical journal BMJ, knows we don't live in an ideal world. We are deluged with information. So people necessarily turn to shortcuts to help them sort through it all. One shortcut is to look for a familiar name or at least a trusted institution in the list of authors.
BLOOM: As human beings, I think we default to thinking, well, you know, how do I know this? Where does it come from? Who's telling me? Do I believe them?
HARRIS: But leaning on that has a downside.
BLOOM: The shortcuts we use tend to propagate what in this country we would call an old boys network.
HARRIS: This favors biases over fresh ideas. Bloom is involved in one of the major online repositories for unpublished papers called Medrxiv. And she realizes that the sheer quantity of new information, especially from unknown labs and unknown scientists around the world, is a huge challenge. Jonathan Kimmelman, a professor of biomedical ethics at McGill University, confronts this as he struggles to learn from the coronavirus literature.
JONATHAN KIMMELMAN: It takes a large investment of attention and effort to really dig deeply into a manuscript to scrutinize the methods, the claims, the relationship between the methods and the claims.
HARRIS: He first asks himself a basic question. Can I trust what I'm reading here?
KIMMELMAN: Knowing where a researcher is or who a researcher is can be part of establishing that trust. But I do think it harbors some dangers.
HARRIS: He could be dismissing important work. After all, for coronavirus, a lot of important research comes out of labs he's never heard of, produced by people he doesn't know. So Kimmelman tries to look for signals of quality in the papers themselves. He recalls one paper out of China that touted the benefits of hydroxychloroquine, a malaria drug that had been promoted as a possible treatment for COVID-19.
KIMMELMAN: This was pretty quickly taken up by The New York Times. And a number of different experts had fairly positive statements to say about the clinical trial.
HARRIS: Kimmelman looked to see whether the results aligned with a research plan that the scientists had earlier posted in a government registry. He quickly found major inconsistencies.
KIMMELMAN: So those struck me as a lot of major red flags. It probably took me maybe somewhere between 15 minutes and 30 minutes to come to the conclusion that this paper wasn't worth the time of day.
HARRIS: Sure enough, the promise of hydroxychloroquine as a COVID-19 treatment eventually crumbled. Lauren Ancel Meyers, a scientist at the University of Texas in Austin, says some of the important early information flowing from unfamiliar scientists in China got immediate recognition. Those papers were co-authored by prominent researchers in Hong Kong and Britain.
LAUREN ANCEL MEYERS: I imagine that some of the reaching out and the bridges that were built were not just to get credibility, but really to bring the brightest minds to help think through the data and what was going on.
HARRIS: Meyers herself has collaborated with researchers in China thanks, in no small part, to a postdoctoral researcher in her lab who is from Hong Kong.
MEYERS: Working very closely with someone from China has been incredibly invaluable to just getting basic understanding of the situation, but also building bridges to researchers and to data that are coming out of China.
HARRIS: There is no substitute for really knowing. The data and Theo Bloom at the BMJ says there is a danger from relying too heavily on surrogates, such as big names or big name journals, when evaluating a new finding.
BLOOM: There are retractions, falsifications from great journals, great institutions, Nobel laureates and so on. So it behooves us all to try and move away from relying on who we recognize as good.
HARRIS: That takes self-awareness and an investment of time.
Richard Harris, NPR News. Transcript provided by NPR, Copyright NPR.
