A controversial study into Victoria’s mask mandates has been dinged by the scientific journal that published it after widespread backlash.
A major scientific journal has officially acknowledged “serious concerns” about the findings of a study into Victoria’s mask mandates, more than 18 months after the paper was torn to shreds as “crap” by experts.
The journal Plos One on Friday slapped an “expression of concern” on the Burnet Institute’s controversial July 2021 paper, which purported to show that mandating masks was the “single most important control measure” that turned around Victoria’s second Covid wave in 2020 “almost overnight”.
In October 2021, experts told news.com.au that the supposed “world-first” study was riddled with basic errors and should never have been published in a major journal, blasting the paper as “crap”, “extremely lightweight” and “very, very low quality”.
The Burnet Institute – an influential public health body that advised the Victorian government on its Covid response – stood by its findings and described the scientists’ comments to news.com.au as “disappointing and unprofessional”.
But Plos One has now acknowledged flaws in the study with a lengthy notice appended to the paper. An expression of concern – one step above a correction but short of a retraction – is used to “alert readers of serious concerns about published work”.
“After publication of this article, readers raised a number of concerns, including about the methodology, the limitations of the study design, and whether the conclusions are fully supported,” Plos One editors wrote in the notice.
“The Plos One editors consulted with two members of the editorial board and a statistical advisor who advised that the study design is associated with a number of weaknesses that are discussed in the article, and which are unavoidable because of ethical issues that would be associated with a randomised controlled trial in the context of a pandemic, but that there were also additional weaknesses.”
For the study, the Burnet Institute looked at photos from The Age newspaper to conclude that mask usage rose from 43 per cent to 97 per cent after the July mandate came into effect.
Medical researcher Dr Kyle Sheldrick told news.com.au in 2021 that it was “hard to think of a worse methodology to answer this question than just looking at which photos are collected by a metropolitan newspaper”.
“I was staggered to see this was published by a major journal,” he said. “If a student came to me and said, I’m going to compare these two sets of photos and draw some conclusion about whether a policy worked, you would send them away to think about it.”
Another researcher, an eminent Australian clinician and scientist who spoke on the condition of anonymity, was equally scathing. “I agree, it’s crap,” he said. “It’s extremely lightweight. I think it’s a totally feeble article. It doesn’t have a rigorous methodology and it is weak in its scientific inference.”
Concerns were also raised about other aspects of the paper, including that it singled out the mask mandate as the key reason for the fall in Covid transmission out of all of Victoria’s sweeping lockdown measures.
Plos One’s editors acknowledged the three main issues raised with the paper – the photographic observation of mask usage, self-reported mask wearing survey data, and unmeasured confounding variables such as curfews, movement restrictions and closures.
On the photos, Plos One’s editors noted it was “unclear whether such images are representative of mask usage in the population”.
They also noted the Survey of Covid-19 Responses to Understand Behaviour (SCRUB) sample size was “small and may not be representative of the population”.
In response, the Burnet Institute’s authors indicated that “the sample sizes were not inappropriately small for the purpose, noting that the high-powered interrupted time series study found a highly statistically significant change in the growth rate of the epidemic that coincided with the mask mandate”.
Plos One’s editors also noted that “the study design could not exclude the possibility of contributions from unmeasured confounding variables, including the implementation of a curfew and movement restrictions on 2nd August 2020 and closure of childcare facilities, schools, and non-essential businesses on 5th August 2020”.
“According to the cumulative expert input received by the Plos One editors, the results of the published study contribute to the field of mask evaluation research, provided results are not overinterpreted and limitations are acknowledged,” they wrote.
“The Plos One editors felt that the conclusions, including those that imply causation, a direct correlation between Covid-19 cases and mask mandates, and the ability of masks for controlling epidemics, were not suitably tempered in light of the limitations of the study design. The Plos One editors issue this expression of concern to inform readers about the above considerations regarding study design and interpretation of the results.”
The Burnet Institute has been contacted for comment.
It comes after a major Cochrane Review – considered the gold standard of evidence-based medicine – earlier this year assessed 78 high-quality scientific studies, and found “wearing masks in the community probably makes little or no difference” when comparing masking with non-masking to prevent Covid.
The bombshell findings proved controversial and sparked a war of words between Karla Soares-Weiser, editor-in-chief of the Cochrane Library, and the study’s lead author Tom Jefferson.
“Many commentators have claimed that a recently-updated Cochrane Review shows that ‘masks don’t work’, which is an inaccurate and misleading interpretation,” Soares-Weiser said in a statement in March.
“[The study’s] wording was open to misinterpretation, for which we apologise. While scientific evidence is never immune to misinterpretation, we take responsibility for not making the wording clearer from the outset.”
Jefferson subsequently hit back at suggestions the apology from Soares-Weiser meant the study had been retracted.
“It was upsetting,” he told journalist Maryanne Demasi. “Cochrane has thrown its own researchers under the bus again. The apology issued by Cochrane is from Soares-Weiser, not from the authors of the review.”
Jefferson claimed Soares-Weiser, in response to media pressure, had “gone outside the normal channels and made decisions without any consultation with the authors of the review”.
“It is unacceptable,” he said. “We are the copyright holders of the review, so we decide what goes in or out of the review. We do not change our reviews on the basis of what the media wants.”