Poor quality statistical reporting in the biomedical literature is not uncommon. Here is another example by Cirio et al. (2016). The study itself is well planed, executed and reported. The aim was to assess whether heated and humidified high flow gases delivered through nasal cannula (HFNC) improve exercise performance in severe chronic obstructive pulmonary disease (COPD) patients. It all started when I saw their Fig.1. Here is my attempt to reproduce it

Figure 1: Effect of the HFNC on exercise capacity compared to a control condition. Tlim = exercise duration.

In total there are 12 patients tested twice; once under the control test and once under the HFNC test. The outcome of interest is the endurance time (Tlim; y axis). This is practically how long each test lasted. The authors hypothesized that HFNC would improve exercise performance, that is the test would last longer. This was the case since Tlim increased for all subjects under the HFNC test (see figure 1). Moreover, this increase reached statistical significance (p-value = 0.015) - ready to publish! Looking at the plot I was pondered about the “outlying” patient (red dot). His/her Tlim increased by a whooping 400 seconds! This is huge compared to the other patients. Then I wondered how would the results change if we excluded him/her from the analysis? And here is where the problems start.

There is no way from the text to figure out which test was used to produce the p-value of 0.015. Is is a paired t-test or a Wilcoxon test? (they mention both in the statistical analysis section). So it is impossible to evaluate and/or try to reproduce the results.

Having abandoned the idea of being able to reproduce the analysis I started thinking about reporting guidelines, hence the title of this post. I thought the journal must have guidelines for reporting statistical analysis. No, it does not and unfortunately, most of the biomedical journals don’t have such guidelines even though 40 years ago O’Fallon and colleges recommended that “Standards governing the content and format of statistical aspects should be developed to guide authors in the preparation of manuscripts” (O’Fallon et al. 1978). Since then many have repeated the message. A few sporadic attempts are usually editorials such as Cummings and Rivara (2003), Curran-Everett and Benos (2004) and Arifin et al. (2016).

Recently, Lang and Altman (2013) published a comprehensive set of statistical reporting guidelines suitable for medical journals - the SAMPL guidelines. “The SAMPL guidelines are designed to be included in a journal’s Instructions for Authors”. So the journals just need to refer to them! As there are many general reporting guidelines based on the study design as such CONSORT, STROBE, PRISMA etc (see http://www.equator-network.org/) that authors in many journals must adhere to, I believe the SAMPL guidelines is a big step forward on reporting statistics. The only journal (that I know of) that suggests the use of the SAMPL guidelines is the British Journal of Dermatology (Hollestein and Nijsten 2015). (I’ll keep adding to this list).

Now that the guidelines exist, let’s make use of them.