In these times of fake news, it鈥檚 good to know that there鈥檚 still one source we can rely on: the scientific community. Wielding rigorous standards of evidence, researchers can be counted on to give us trustworthy insights amid a sea of nonsense.
Yet this, too, is fake news. For decades, scientists have been using flawed methods for turning raw data into insight about, say, the effectiveness of a new medical therapy or method of teaching. As a result, the research literature is awash with findings that are nothing more than meaningless flukes. No less shocking is the fact that researchers have been repeatedly warned about the problem, to no effect.
This week, the American Statistical Association (ASA) hopes to change that. It is hosting a conference . But what has this scientific crisis got to do with statistics?
Advertisement
Last year, the ASA stated , these are suspected of playing a key role in the replication crisis in science, in which startling claims collapse when other researchers try to confirm them.
Taking the p
The focus of the ASA鈥檚 concern lies in the calculated probability, or p-value, of experimental results. After crunching data on, say, the fitness of people before and after some new form of exercise, researchers calculate the p-value of the difference in performance.
If the outcome leads to a p-value below 5 per cent, the findings are conventionally said to be 鈥渟tatistically significant鈥� and worth taking seriously. That鈥檚 because the p-value is widely thought to be the chances of the findings being a fluke. As such, those with a p-value of 5 per cent seem 95 per cent likely to be the result of a genuine effect.
But that鈥檚 not what a p-value means. It鈥檚 actually the chances of getting at least as impressive a finding as that seen, assuming fluke was the real cause. Which might sound the same, but is subtly 鈥� and dangerously 鈥� different.
Because it is based on the assumption that fluke is the real cause of the effect, a p-value can鈥檛 just be flipped around to give the chances of that assumption being true.
Statistical headache
Confused? Then try this medical analogy. You鈥檙e a doctor with a patient complaining of recurrent headaches. You know there鈥檚 a 60 per cent chance of getting these headaches 鈥� if there is a brain tumour.
So does that mean the chances your patient has a tumour are also 60 per cent? Hardly: it鈥檚 clearly a mistake to simply flip the problem around and assume the chances of having headaches if you have a tumour are the same as those of having a tumour if you鈥檙e having headaches.
Yet that鈥檚 exactly the kind of blunder researchers make with p-values, leading them to see 鈥渟ignificance鈥� in meaningless flukes, like careless doctors confusing symptoms with causes.
The ASA was right to go public with its concerns last year, and to call on researchers to adopt more reliable techniques for extracting insights from data. Now it is asking statisticians to come up with alternatives.
Civil war
But there鈥檚 a problem. There鈥檚 no consensus on what these alternatives should be. For decades there has been internecine civil war between statisticians who think the p-value approach just needs tweaking, and those seeking radical reform.
In the run-up to this week鈥檚 meeting, one group of leading statisticians . It鈥檚 as simplistic and beset with unintended consequences 鈥� such as discouraging follow-up studies, which will need to be much larger and more expensive if they鈥檙e to meet the tougher standard.
A host of alternatives, including outright bans on p-values and claims of 鈥渟ignificance鈥�, will be discussed at the conference, but if history is any guide, they too will be torn apart by one faction or another.
It is time for statisticians to bury their hatchets and reach a pragmatic consensus. Unless they do, researchers will simply stick with the flawed methods they鈥檝e always used 鈥� and leave the rest of us struggling to tell the fake breakthroughs from the real ones.
Read more: Why so much science research is flawed 鈥� and what to do about it;听Science isn鈥檛 as solid as it should be 鈥� but science can fix it
