Sound decisions should look at all the scientific evidence, whether we are considering healthcare, policy, business strategy or research priorities. That often means putting together many studies, often with conflicting results and with many biases, to try to get a better idea of which way they point overall.
How do we do this? Or if someone does it for us, how can we make sure that they will not be biased and will not pile more errors upon ?
With data and scientific literature burgeoning, attempts to combine many existing studies to try to get a clearer overarching picture, known as systematic review and meta-analysis, .
Advertisement
Unfortunately more is not necessarily better. For example, most systematic reviews and meta-analyses in biomedicine are faulty beyond repair, being based on flawed, highly conflicted or useless data.
鈥淒one properly, meta-analyses can be more than harbingers of bad news about the state of research evidence鈥�
Conclusions often depend more on what the reviewers believe: for example, meta-analyses of the brain tumour risk posed by cellphones , even though they looked at the same available data. Given that we increasingly rely on such analyses to guide decisions, this is a problem.
Clinical medicine is one of the more prolific generators of meta-analyses, with tens of thousands of published examples. There are more reviews published each year than new randomised clinical trials. There鈥檚 also tons of redundancy, for example over 200 meta-analyses of antidepressants for depression, and dozens looking at some newer treatments for rheumatoid arthritis.
But this proliferation doesn鈥檛 afflict all fields of study. In some fields meta-analyses are still rare 鈥� in economics there are fewer than 200 done to date. More would be welcome, but only if done well.
At the heart of the problem is that much of this work puts together fragments of selectively published information from studies that are poorly executed, or tainted by conflicts of interest, or focused on unimportant or misleading questions. What is really needed is meta-analysis to probe the problems in the primary data.
Reality check
If we are to avoid heaping error on error, in many cases systematic reviews and meta-analyses may just have to conclude that the evidence is dubious, rather than simply churning out even more statistically significant, yet spurious, summary results. Meta-analyses are far better used this way 鈥� as a reality check.
However, done properly, meta-analyses can be more than harbingers of bad news about the state of research evidence.
With a strong push to improve transparency, openness and sharing, more pressure to make full data routinely available, and registered protocols made public before studies even begin, there are new opportunities for better use of these techniques.
With transparency, meta-analysis can be part of primary research. Instead of trying to fix something after it is broken, let study design and data be optimised from the outset to ensure meta-analysis actually works.
Used this way, new research, teamwork and meta-analysis become amalgamated into a common paradigm. This is already working nicely in several fields, such as genetics or high energy physics. One example is the amassing of particle collision data at the CERN laboratory near Geneva, Switzerland, from many experiments in order to tease out a reliable result.
If we can apply similar principles to other fields such as medicine or psychology, we can get back on the path to reliable evidence.
