Where the rubber meets the road

Humans are generally unable to handle uncertainty rationally. Finding significance in random stimuli and interpreting random phenomena as mostly dangerous give a survival advantage that can explain some of the anxiety and superstition in today's society. In the history of science, uncertainty has not always been accepted. For example, questioning the church's scientific statements about the earth being the centre of the solar system rendered Galileo a death sentence in 1633. During the following...

This quote by the American historian Daniel Boorstin is becoming increasingly relevant also in medical research. Nowadays, it seems to be much more common than one or two decades ago that authors confuse cohort studies with case-control studies, cumulative incidence with incidence density, hazard ratios with odds ratios, etc., etc. It is paradoxical that the increased accessibility of information with Internet (and the easiness with which the definition of methodological terms can be checked) co...

While statistical significance often is mistaken as an indication of practical importance or scientific relevance, an even greater mistake is to believe that statistical non-significance indicates equivalence or "no difference". It doesn't. Statistical non-significance reflects uncertainty, which perhaps can be considered as an indication of a too small sample size. ...

All medical scientific publications do not present evidence based research. Many, if not all, hypothesis presentations, non-systematic reviews, and case reports are authority based rather than evidence based. Also such publications may have a role to play for the progress of science. It should, however, always be made clear to the reader whether the author's ambition has been to present a personal opinion or an objective and reproducible research finding. From an editorial point of view, it may ...

A manuscript that is entirely based on assumptions presents a hypothesis. Manuscripts written for presentation of empirical findings must be based on data. As Edward Demings said, "In god we trust, all others must bring data". However, in order to analyse data, assumptions must be made. When presenting the analyses its results, the author's presentation must clearly distinguish between observations, assumptions and analysis outcomes. Confusing assumptions and outcomes is not a good thing. ...

Modern medical research claims to be objective and reproducible. The reproducibility has, however, recently been questioned. One explanation for this could be that while the findings may appear to be based on sound objective research, they actually just represent subjective opinion. In contrast to well-performed clinical trials many laboratory studies, including those based on statistically correct methods, do not have a well-defined pre-specified study design linking the investigated study hy...

Statistical significance has nothing to do with practical importance or scientific relevance; statistical significance reflects sampling uncertainty. Moreover, the number of statistically significant findings that can be expected in a study is related to the study design, not least sample size, number of statistical tests peformed, and strategy used for addressing multiplicity issues. Successfull investigators develop the design of their experiments in a way that enables detection of practical...

Using correct terminology is important for avoiding misunderstandings. For example, the terms univatiate and multivariate are often misunderstood. The terms refer to the type of probability distribution a model is based on. A univariate statistical model is based on a univariate probability distribution, i.e. it has one outcome variable, and a multivariate analysis is based on a multivariate probability distribution, i.e. the model has multiple outcome variables. An ANOVA model, for example, is ...

Many authors are confused about whether the purpose of writing a research report is to describe or to generalise. Some authors also seem to believe that p-values and confidence intervals are descriptive measure that must be used to describe the importance of what has been observed in a studied group of subjects. This is not the case; p-values and confidence intervals describe generalisation uncertainty. However, the question is more complicated than this. Generalising with the help of p-values a...

Statistical models are in observational clinical research primarily used for developing algorithms for individual prediction and for estimating average effects of treatments or of exposure to hazardous agents, and, which is confusing for many authors, these two modelling purposes require different methodological approaches. While the best prediction model is the model that predicts best (whether or not the parameter estimates are biased is irrelevant) and this is evaluated using the area under...

The new version of Stata (release 16) includes LASSO regression. This is excellent because LASSO is one of the better methods for developing prediction and classification models. However, like stepwise regression, it is unfit for producing parameter estimates with adjustment for confounding bias. The adjustment must be based on considerations regarding cause-effect relations (i.e. confounders must be included in, and mediators and colliders excluded from, the statistical model used for the estim...

Some manuscripts are based on a detailed description of a series of patients and have a conclusion restricted to what has been observed. This is what could be expected of a case-series report. However, the same patients could also have been considered a random sample drawn from and representing a greater population of patients, perhaps including future ones. In this case, the findings cannot be directly generalised to the greater population because of sampling uncertainty. When attempting to des...

A study hypothesis can usually not be evaluated by a statistical test of a single null hypothesis. The study may be based on comparisons of more than two groups and of more than one endpoint. Several, perhaps hundreds of statistical tests can then be found in a manuscript, and when multiple null hypotheses are tested, the false positive risk increases with the number of tested hypotheses. In confirmatory studies the significance level may need to be corrected for the multiplicity. One often used...

It is a common belief that a statistically significant finding always is practically important and that statistical nonsignificance is a good indication of "no difference". This belief is a major mistake. Statistical significance is a measure of uncertainty, not of importance; practical importance has to be shown by other means than p-values. Equivalence and non-inferiority can only be statistically tested when an equivalence or non-inferiority margin, specifying the practical importance, has be...

When evaluating the effect of a treatment, it may be tempting to perform the treatment on a group of subjects that have scored extremely on some measurement, and then measure the subjects again after the treatment. The difference in the measured values would provide a good estimate of the treatment effect. Or wouldn't it? The answer is, that if measurement errors and accidental variation affect the measurements randomly, more subjects will be included with too high values than with too low, an...

Given that only three quartiles are defined, the middle one also known as the median (see The International Statistical Institute. The Oxford Dictionary of Statistical Terms. Oxford University Press, New York 2003), it is surprisingly common to see results presented with four quartiles. The explanation is, of course, that the term is misunderstood. The misunderstanding is actually so common that the Merriam-Webster dictionary states that the four quartiles are the same as the four quarts defined...

Statistical reviewing is performed for the benefit of the reader. The main purpose is to make sure that the limitations imposed upon a study's findings by the authors' data collection, study design, and statistical analysis are clearly presented to the reader. Spending several hours on writing rebuttal letters in order to avoid addressing problematic issues in the manuscript is a bad idea. ...

The common claim that a study "proves" or "demonstrates" the correctness of a study hypothesis usually reveals the authors' readiness to exagerate the importance of their findings. The words "indicates" or "suggests" are aften more appropriate, and their usage may give the reader a better impression of the authors' judgement. ...

During the last 25 years I have reviewed over 6000 research reports submitted to more than 75 different scientific journals, most of them medical. Some of these reports have presented reliable and groundbreakning research results, but far too many have just represented a serious waste of resources that could have been used to find better medical treatments, reduce suffering, and prolong life. This experience has changed my life; trying to identify empirical support for presented findings but fi...

Randomised trials and other clinical studies are performed to learn about the effects of a drug or treatment among all patients, not least future ones and not just the ones included in the study. Generalisation from a small sample to an infinite population can, however, not be made without uncertainty, and the magnitude of this uncertainty is often presented in terms of p-values. These values depend on sample size and variability and have, in themselves, nothing to do with clinical relevance or ...