Humans are generally unable to handle uncertainty rationally. Finding significance in random stimuli and interpreting random phenomena as mostly dangerous give a survival advantage that can explain some of the anxiety and superstition in today's society. In the history of science, uncertainty has not always been accepted. For example, questioning the church's scientific statements about the earth being the centre of the solar system rendered Galileo a death sentence in 1633. During the following age of enlightenment, scientific societies started to publish research findings in scientific journals but these were mainly what we today describe as expert opinions and subjective case reports without proper recognition of uncertainty.

The authority-based approach to science was gradually replaced by an evidence-based approach during the 20th century. Uncertain research findings, accompanied by objective and reproducible quantification of the uncertainty, were published. Statistics made its entry into medical science, with major effects on health care. By the end of the century it was no longer possible to get market approval for a drug by just presenting a letter from a medical professor having tested the drug on 3 patients and found it effective. Instead, randomized trials with pre-specified endpoints and statistically significant outcomes were required.

Without a genuine education in probability theory and inference theory, statistical significance is, however, a difficult concept, easy to misunderstand. The widespread use of statistical significance tests in the exponentially growing number of research reports has also lead to a caricature of hypothesis testing with presentation of results as either "significant" or "not significant", the former ("p<0.05") believed to indicate practical importance and the latter ("NS") to represent evidence of equivalence. As much as this may seem to be objective, practically useful, and generally accepted, the two interpretations are fundamentally flawed. Statistical significance says nothing about practical importance and statistical non-significance nothing about equivalence. Finding such evidence requires much more than simple statistical tests.

In spite of many available user-friendly statistical computer packages, the uncertainty of the findings presented in scientific publications today are grossly misleading. As a consequence, medical research suffers from a monumental reproducibility crisis. It is obvious that there is no way out of this mess other than to improve the evaluation and presentation of the findings' uncertainty. Statistical science will, as reflected by the growing number of statisticians reviewing grant applications and manuscripts, play an even more important role in future medical research.

However, all stakeholders will not benefit from methodological improvements, and the research environment rewards, at least in the short run, spectacular news better than efforts to find truth. Spin and exaggeration have become the standard. Statistical reviewers and editors have a hard job.

More from Where the rubber meets the road
All posts