New Scientist Archive I Selected Article	Page 1 of 2
NewScientist Archive I
NEwSCEf~TlSTCOM NEWSC~ENTtST JOBS I NEW SCIENTIST GRADUATE NlEW SC!ENTISTARCHIVE I NEWSCIENTIST SUDSC

Selected Article
…………………………………………………………………………………………………………………………

Modify search I Back to results

				Magazine section: Comment and Analysis
SEARCH
HELP	
	The numbers don’t add up	
	New Scientist  vol 177 issue 2385 -08 March 2003, page 28

_________________	Blind faith in statistical recipes is good for churning out papers, but how often does it put promising new lines of research at risk, asks Robert Matthews

YOU’D be a fool to believe everything in the papers. Made-up facts, half-baked analysis, hand-picked conclusions- yes, scientific papers can seem no more reliable than the tabloid press.

In the past month alone, concern over fraud in medical research papers has prompted the retraction of one paper in The New England Journal of Medicine and disciplinary action against the lead author of another, published in the British Medical Journal. In both cases the whistle-blowers were co-authors of the papers who became concerned about the reliability of the research and asked that the papers be retracted.

Some will see this as evidence that the scientific community can police itself. But what if the authors have no idea that what they have done is questionable or misleading? What if they routinely use unreliable methods, with top research journals cheerfully publishing results based on them?

That is the stark prospect highlighted by the epidemic of flawed statistical analysis plaguing today’s research literature. From medical “breakthroughs” that prove to be mirages to grand conclusions drawn from tiny samples, the journals are awash with unreliable findings based on faulty statistics.

The threat this poses to the credibility of science has been pointed out repeatedly over the past 40 years. Yet the warnings have gone unheeded. Take the case of that sine qua non of research, the “statistically significant” result. As long ago as 1963, statisticians at the University of Michigan issued a warning that the methods used to gauge the statistical significance of findings were “startlingly prone” to exaggerate real significance.

At the heart of these methods is the so-called P-value. By time-honoured tradition, P-values below 0.05 are deemed “statistically significant”, as they seem to imply that there is just a 1 in 20 chance of the results being flukes. In fact, because P-values take no account of the inherent plausibility of the hypothesis being tested, it is in fact four times as likely that the result is a fluke. In other words, a substantial proportion of “statistically significant” findings are meaningless flukes - scientific fool’s gold that lures others to waste time and money in attempts to replicate findings.

Then there’s the flip side: the rejection of potentially important ideas because a small study fails to confirm them. Such studies may be quick and cheap, but they also lack statistical power  that is, they run a high risk of failing to detect real effects.

As with statistical “significance”, experts have been warning about the dangers of underpowered studies for over 40 years, but you would never know it to look at the current medical literature. A recent analysis of trials of treatments ranging from antidepressants to plastic surgery has shown that many are still far too small to detect clinically worthwhile effects.

That hasn’t stopped researchers making grand pronouncements off the back of negative results. Last month, the Journal of the Royal Society of Medicine published the results of a study comparing a homeopathic arnica extract, a treatment for post-operative pain, with a placebo. Though the study involved just 62 patients, its failure to find any evidence of benefit led the researchers to describe the treatment’s supposed efficacy as a “myth”. They may well be correct, given that homeopathic treatments have no obvious scientific basis. Yet the trial was so small that it stood barely a 1 in 3 chance of confirming the efficacy of many tried and trusted conventional treatments for post-operative pain.

Small studies are not wholly useless: when taken together in a metaanalysis they can produce scientifically reliable insights. Cynics point out, however, that small studies are also perfect for researchers keen to churn out lots of papers with minimal effort and cost. Certainly, over interpreting negative results from such trials risks killing off interest in promising new lines of research. One recent study found that of 25 clinical trials in orthopaedic medicine reporting negative results, not one had the statistical power to detect a potentially worthwhile benefit. A similar study of surgical trials found that the authors of fewer than a third of papers reporting negative results bothered to check if their trial was even big enough to detect any benefit.

The editors of some scientific journals have recognized the threat posed by faulty statistical analysis and have tried to clamp down. Many statisticians suspect the real issue is a lack of awareness among working scientists about the limitations of statistical methods. Some have called for professional bodies such as the Royal Statistical Society in Britain and the American Statistical Association to speak out about the routine abuse of statistics in scientific research.

So far, however, their silence has been deafening. Until they join with other scientists to take this issue seriously, the rest of us can surely be excused for dismissing the next headline-grabbing “breakthrough” by repeating that old saw about lies, damned lies -and statistics.

Robert Matthews
Robert Matthews is visiting reader at the department of information engineering at Aston University in Birmingham

Modify search I Back to results

Contact Us FAQ • Media Information Disclaimer • Terms and Conditions • Privacy Policy © Copyright Reed Business Information Ltd.