SOME people describe it as 鈥渉avoc鈥�, others as 鈥渁 recipe for disaster鈥�. Not the effect of the coronavirus on healthcare or the economy, but on something even more fundamental to defeating it: science.
Since the pandemic began, thousands of studies related to it have been published. 鈥淭he research community has mobilised in the face of the pandemic in an unprecedented way,鈥� says John Inglis at academic publisher Cold Spring Harbor Laboratory Press in New York.
But in the race to understand the coronavirus, and amid the cacophony of political messages, inexpert journalists and viral social media messages, a parallel pandemic has emerged 鈥� one of rumours, unverified claims and malicious falsehoods. The World Health Organization has described this confusion as an .
Advertisement
In particular, the role of preprint servers has been raising alarm. These are online repositories of preliminary findings that haven鈥檛 yet been independently reviewed. They were invented because of dissatisfaction with the conventional peer-review model, and to take advantage of new opportunities afforded by the internet.
Free-flowing information
This alternative system of academic publishing has increased in importance and credibility in recent years. It means findings can be shared widely much faster 鈥� a useful tool in an unprecedented health crisis. But the pandemic has also exposed the practice鈥檚 weakness: anyone can publish anything, with little or no quality control.
Preprint servers enable information to 鈥渇low directly from people who are making scientific claims to users who don鈥檛 have the savvy to evaluate those claims鈥�, says Jonathan Kimmelman, a biomedical ethicist at McGill University in Canada.
鈥淲hen you mix the science with all that social and media reverberation, you get an explosive mix鈥�
Life science research was slower to adopt preprint servers than the physical sciences, in part because the research often has health implications, says John Ioannidis at Stanford University in California. But in 2013, Inglis and his colleagues created a biosciences server, bioRxiv, and last year they set up another for the health sciences, medRxiv.
In medRxiv鈥檚 first eight months, people posted 1100 papers to it. Then the pandemic hit. A further 3700 have been added in the two months since, mostly about the SARS-CoV-2 virus and the covid-19 disease that it causes. A combined medRxiv/bioRxiv site dedicated to the virus contains more than 2700 articles.
The protocol of a preprint server is that scientists post their preliminary findings for others in their field to comment on and criticise. The paper is typically then revised and submitted to a journal, or retracted. About 70 per cent of preprints eventually get published in conventional journals, says Inglis.
Under normal circumstances, this can greatly improve the research process, says Stuart Ritchie at King鈥檚 College London, author of the forthcoming book Science Fictions: Exposing fraud, bias, negligence and hype in science. 鈥淚n general, I think preprints are a brilliant innovation, speeding up science, allowing open discussion of findings and letting people see others鈥� criticisms.鈥�
But these aren鈥檛 normal circumstances. All of a sudden, says Ritchie, people who wouldn鈥檛 normally be interested in biomedical preprints, and don鈥檛 necessarily understand or care about their limitations, have started reading and sharing them. That includes politicians, policy-makers, journalists, bloggers, social media influencers, armchair pandemic warriors, political agitators and conspiracy theorists. 鈥淲hen you mix the science with all that social and media reverberation, you get an explosive mix, and that creates havoc,鈥� says Ioannidis.
Another problem is that people with little or no biomedical knowledge are posting and commenting on preprints. 鈥淵ou have lots of preprints that are by people who are not properly trained in science at all, and others from people who work in scientific fields that are completely different,鈥� says Ioannidis. 鈥淵ou have lots of people who have suddenly become epidemiologists overnight. It鈥檚 a recipe for disaster.鈥�
There are, of course, benefits to the rapid dissemination of data and hypotheses. When the coronavirus first emerged, preprint servers hosted valuable insights into preliminary data from Wuhan and the surrounding area in China that helped us begin to understand the virus. 鈥淵ou can鈥檛 wait six months, which is the typical time for a scientific paper from submission to publication,鈥� says Ioannidis. 鈥淭hat is double the time of the entire epidemic wave.鈥�
鈥淲e have learned an extraordinary amount extraordinarily quickly about the pandemic and the infectious agent,鈥� says Inglis. 鈥淧reprint servers were there for [researchers] to use for sharing the latest research almost immediately, at no cost, with minimal obstacles, and in an entirely open way.鈥�
But some think the rapid dissemination of hastily done research can be too risky. 鈥淯nder certain circumstances, a little bit of information is worse than no information at all,鈥� says Kimmelman. 鈥淚 would say these are circumstances under which that would apply.鈥�
鈥淕ame-changing鈥� drugs
The much-touted antimalaria drug hydroxychloroquine is a good example of the system going badly wrong. A preprint about the drug鈥檚 efficacy against covid-19 in a small clinical trial appeared on 20 March (medRxiv, ). The trial was poorly conducted, says Alfred Kim at Washington University School of Medicine in St Louis, Missouri, who wrote a critique of it in the Annals of Internal Medicine (). Among other issues, the trial had a sample size of just 20 people (see 鈥�How to sniff out the good science studies from the bad鈥�).
A second preprint by different researchers detailing methodological flaws in the trial appeared three days later (Zenodo, ).
鈥淎ny medical study with fewer than 50 participants should be treated as highly tentative鈥�
Nonetheless, says Kim, the trial鈥檚 findings were picked up and amplified by the press, social media and many government and institutional leaders, including US president Donald Trump, who famously called the drug a 鈥済ame changer鈥�. Public interest exploded.
It got worse. On 30 March, a preprint appeared on medRxiv reporting the results of another small clinical trial on 62 patients in a hospital in Wuhan with mild-to-moderate covid-19 (). It found that those given the drug recovered faster.
The the day after. The article, written by a specialist science reporter, was careful to say that the study was small and not peer-reviewed, and that more research was needed, but it did feature boosterish expert quotes. 鈥淭he physicians interviewed in the story made it sound like it was a credible report,鈥� says Kimmelman.
But this study was beset with methodological problems, says Kimmelman. 鈥淭here was striking dissimilarity between what they said they were going to do in that study and what was actually reported.鈥� A diligent peer reviewer might have picked this up, he says, but somebody who isn鈥檛 an expert in the methodology of clinical trials has little chance of doing so.
This shows just how difficult it is for even skilled journalists to pick up pretty glaring errors in research reports, says Kimmelman, who adds that even trained doctors are rarely equipped to do so.
These infodemic failures have real-world consequences. Hospital doctors started giving hydroxychloroquine to covid-19 patients, and some people began self-medicating with it. There were shortages of the drug for people with rheumatoid arthritis who really need it, and scarce scientific resources pivoted to research the drug when they could perhaps have been spent better elsewhere.
The blame for such confusion can鈥檛 all be placed on the preprint servers. The dedicated covid-19 preprint site features a prominent disclaimer, reminding visitors that the reports posted there shouldn鈥檛 guide clinical practice or health-related behaviour, or be reported in the media as established information.
Fixing the problem
Preprint servers aren鈥檛 the font of all bad knowledge. Peer-reviewed journals have also been .
We also know that the process of academic publishing has many issues, including a bias towards positive results. Peer review is no guarantee that a study鈥檚 findings will stand the test of time and be successfully replicated.
But given the benefits of preprint servers, what can be done to reduce their negatives? Inglis says the science community is already taking action to rapidly vet the content of preprints. These efforts include dedicated projects to provide informal peer review and expert commentary at Mount Sinai Hospital in New York, and at the University of Cambridge. A consortium of journal publishers is exploring how to speed up the peer review of preprint papers without compromising on quality.
Ritchie suggests that preprints should be electronically watermarked with a disclaimer to avoid any doubt about the provisional status of the research.
And while the medRxiv site says all manuscripts undergo a basic screening process for non-scientific content and material that might pose a health risk, arguably more could be done to vet research before it is posted to servers around the world.
But scientists should shoulder some of the blame, says Ritchie. One problem is the volume of low-quality studies that researchers are posting.
Many scientists are unwilling to discuss their research with journalists until it has been peer reviewed. This doesn鈥檛 necessarily help, because journalists may report on preprint studies anyway. Some of those researchers who do speak to the media could be clearer about the preliminary nature and limitations of their work, says Ritchie.
鈥淚f we want good research and effective healthcare, we need to get on top of this problem鈥�
Another issue is experts in one field turning their hand to another. In March, for example, an electrical engineer and a cardiologist posted a preprint estimating that the UK could experience just 5700 covid-19 deaths (medRxiv, ). Several UK newspapers gave the estimate prominent coverage. The UK鈥檚 confirmed death toll currently stands at over 28,000.
Kimmelman believes there is a wider societal issue. 鈥淚 think this is part of a much broader problem of how information flows in contemporary societies, particularly around expertise. We鈥檝e seen parallel issues in politics and democracy 鈥� fake news, false claims, etc.,鈥� he says. 鈥淚f we want an efficient research enterprise and an effective healthcare system, we need to get on top of it.鈥�
Health Check newsletter
Get a monthly dose of health analysis in your inbox
