Covid-19 studies based on flawed Surgisphere data force medical journals to review processes

Some of the world’s leading medical journals are reviewing their processes after they were forced to retract studies based on flawed data.

None of the peer reviewers who examined a questionable study on the impact of blood pressure medications on Covid-19 saw the raw data behind the findings before it was approved for publication in world-renowned medical journal, the New England Journal of Medicine.

The study was based on a massive dataset supposedly gathered from hospitals worldwide by a US company called Surgisphere, but a Guardian investigation has since revealed the database to be seriously flawed. The revelation, combined with concerns highlighted by scientists worldwide about the data, prompted the journal to retract the study. The Lancet, another leading medical journal, also published a study based on the Surgisphere database.

The Lancet study found the anti-malarial drug hydroxychloroquine was associated with a higher risk of death and heart complications in Covid-19 patients, a finding which prompted the World Health Organisation and research institutions worldwide to place studies of the drug on hold. That study was also revealed to be problematic and was retracted by the Lancet earlier in June. However, there is still no strong evidence that hydroxychloroquine is effective in the prevention and treatment of Covid-19, and health authorities have warned against the drug being given to Covid-19 patients outside clinical trials given its toxicity profile and side effects.

Related: Unreliable data: how doubt snowballed over Covid-19 drug research that swept the world

The publication and retraction of the studies in renowned medical journals has reignited concerns in the research community about the rigour of peer review. Peer review is where scientists evaluate the quality of other scientists’ work to identify any issues before it is published in industry journals. This process is designed to prevent weak studies and their findings from being published by journals, which is important because what appears in leading medical journals often changes health and medical guidelines for patients.

Many peer-reviewed journals require that their authors be prepared to share their raw, unprocessed data with peer reviewers, and also to make the de-identified raw data available to scientists more generally so that it can be both used and scrutinised. The International Committee of Medical Journal Editors statement on accountability says: “As editors, we strongly oppose contractual agreements that deny investigators the right to examine the data independently.”

The Lancet and New England Journal of Medicine are said to have among the most stringent peer review processes of the scientific journals.

A spokeswoman for the New England Journal of Medicine told the Guardian that the Surgisphere paper was reviewed by four external experts and a statistical reviewer. But none of these experts saw the raw data from Surgisphere, she said.

Both the Lancet and the New England Journal of Medicine subscribe to the voluntary editorial recommendations of the International Committee of Medical Journal Editors. Those guidelines state: “As part of peer review, editors are encouraged to review research protocols, plans for statistical analysis if separate from the protocol, and/or contracts associated with project-specific studies. Editors should encourage authors to make such documents publicly available at the time of or after publication, before accepting such studies for publication.”

The New England Journal of Medicine spokeswoman said following the Surgisphere paper retraction earlier in June, the journal completed an internal review of the editorial process. She said the Surgisphere study “received excellent peer reviews, and the reviewers raised relevant questions about the work”.

“We learned two things from this review that will result in changes to our process,” she said.

“We have limited experience with reviewing or publishing studies like this one, which used a large database based on electronic medical records. The reviewers and editors asked the authors questions about the data sources and data validity. The editors accepted the authors’ responses, rather than asking for help from reviewers with expertise in this type of data. In the future, our review process of big data research will include reviewers with such specific expertise.”

Related: Surgisphere: governments and WHO changed Covid-19 policy based on suspect data from tiny US company

The journal is now in the process of assessing existing guidelines for the conduct and reporting of research on big data, and is also developing internal policies for reviewing and reporting these articles.

The Lancet did not provide as much detail when asked the same questions by the Guardian about how the Surgisphere hydroxychloroquine study passed peer review, and whether the incident would trigger a review of the process. It was the Lancet that under the same editor, Richard Horton, published a fraudulent study linking the measles, mumps and rubella vaccine with autism. In 2010, 12 years after the paper was first published, the Lancet retracted the paper. The author of the paper, Dr Andrew Wakefield, was also banned from practising medicine in the UK due to his fraudulent work.

The latest retraction from the journal has left researchers and the public questioning whether the peer review process and editorial policies are rigorous enough.

A spokeswoman for the Lancet said all research articles undergo independent, external peer review, including statistical review. She said the Surgisphere paper was peer reviewed and edited according to The Lancet’s “usual editorial procedures”. “We are reviewing our requirements for data sharing and validation among authors, and data sharing following publication,” she said.

The Surgisphere website and Twitter account have both been deleted since the retractions on 5 June.