Authors reply to allegations published under a pseudonym on the PubPeer website, relating to the paper 'BCG-induced non-specific effects on heterologous infectious disease in Ugandan neonates: an investigator-blind, randomised controlled trial'.
Concerns about this paper have been raised anonymously to the heads of our institutions and posted under a pseudonym online. The lead institution for the study, The London School of Hygiene & Tropical Medicine, has conducted a formal investigation and has not upheld any of the allegations. As these concerns were shared in a public forum, the authors wish to publicly respond to the issues raised:
1. Issue raised relating to the timeline of perceived changes in outcome measures. The authors recognise this may cause concern given only a small explanatory paragraph was included in the published paper. A full explanation is as follows:
- The clinical outcome described in the published paper was included as a primary outcome in the first approved version of the protocol in 2013, and remained as such in all subsequent approved, amended versions.
- When publishing the trial paper and registering the trial with ISRCTN this clinical outcome was accidentally listed as a secondary outcome. This was the result of a misunderstanding/human error. The trial protocol in use remained unchanged, with the clinical outcome as described in the paper remaining as a primary outcome.
- When publishing the results, this error was recognised, highlighted, and fully discussed with the Lancet Infectious Diseases editorial team (first submission in early June 2020). They advised that the approved protocol is always taken as the source document and that it would be best to report the results as co-primary outcomes with an erratum to the protocol paper and ISRCTN documentation and an explanatory paragraph in the published paper. This was accomplished, with submission to Trials and ISRCTN by 25 June 2020.
Once recognised, the team acted promptly to resolve this error, were transparent throughout and acted in good faith.
2. Issue relating to concerns regarding the possibility of type 1 error given the number of outcomes investigated. The authors accept that this was a complex study design, aimed at maximising scientific output from a trial given that trials in such settings and populations are logistically challenging. However:
- The statistical analysis of the clinical and immunological results for the published paper followed the plan in the approved protocol, which all reviewers had access to during the review process.
- The modest sample size and risk of type 1 error is highlighted as a limitation of the study in the discussion.
- We appropriately reported that almost all the immunological outcomes were null findings, rather than over-interpreting results of borderline significance, and outlined at some length in the discussion that the study had not identified a biological mechanism to explain the observed differences in clinical outcomes associated with BCG at birth.
We believe we reported the results in a fair, open and balanced way.
3. Suggestion that we erroneously reference three other randomised controlled trials and a meta-analysis of those trials as proof of the non-specific effects of BCG.
We described the results of these trials in a similar manner to how they were described in the large review of the non-specific effects of BCG conducted independently by the SAGE group for WHO (https://doi.org/10.1136/bmj.i5170. We did not claim the results as proof of the non-specific effects of BCG in general and were clear in both the Research-in-Context box and introduction that there are limitations to prior work – hence the importance of our independent and rigorous trial.
4. Suggestion that we overinflated the novelty of our study in the discussion section and claimed it to be "the first trial to prospectively assess the effect of BCG on all-cause infectious disease morbidity" omitting reference to the Danish Calmette Study. However:
- The complete sentence referred to reads 'This trial is the first to prospectively assess the effect of BCG on all-cause infectious disease morbidity, rather than relying on retrospective analysis of all-cause mortality or hospital admission rates.' This sentence is included in the paper to highlight that our study had rigorous clinician assessment of all participants when unwell, rather than relying on hospital admission data as in the Danish Calmette Study.
- The null results of the Danish Calmette Study are discussed and referenced in paragraph 8 of the discussion (reference 29).
- The Research-In-Context box also discusses the null results of the Danish Calmette Trial (paragraph 2 of the box). Although referencing is not permitted in this box, the search strategy is clearly provided and the null results of the RCTs are highlighted.
We believe we represented the results of our study appropriately within the context of all the available research at the time.
As authors and scientists, we absolutely agree with the importance of being able to highlight concerns with published research and having a free and frank debate. We were therefore disappointed that this was not done openly through an exchange of letters with the journal, and that those making these claims did so anonymously. We understand that the research surrounding the non-specific effects of vaccinations is particularly polarised. The fact that there are differing results in the field means that we need to work harder to understand the true nature and importance of any non-specific effects of vaccines, but the existence of such differing results does not imply concerns with the conduct or reporting of our study. Our aim throughout this study was to produce rigorous, independent research to the best of our abilities. We believe we achieved this aim. Should BCG provide additional non-specific protection to young infants, it is important that this is taken into consideration as new TB vaccines are developed, so that any beneficial effects of BCG vaccination are not lost and the benefits to the most vulnerable in our society are maximised.
Dr Sarah Prentice, Professor Hazel Dockrell, Professor Alison Elliott, Professor Emily Webb and Dr Stephen Cose on behalf of the Delayed BCG Study Team