How were medical journals and WHO caught out over hydroxychloroquine?

Some scientific papers stop the world in its tracks. In the middle of a raging pandemic, a study in the world’s leading global health journal that seemed to prove President Trump wrong to laud the drug hydroxychloroquine for Covid-19 was always going to have a massive impact. It did. When the paper flagged a higher risk of death on the drug, trials were stopped all over the world, including one by the World Health Organization.

Now it’s in danger of unravelling.

The data on more than 96,000 patients from 671 hospitals worldwide is under the microscope. Errors have already been confirmed. A Guardian investigation has revealed major issues with the small US company Surgisphere that owned the database. Two major journals – the Lancet, which published the hydroxychloroquine paper, and the New England Journal of Medicine, which ran an earlier paper from the same authors with data from Surgisphere acquitting blood pressure drugs of any adverse effect in Covid-19 – have issued an Expression of Concern. That’s a formal statement that something is not right, but short of a retraction. Depending on what the audit says, a retraction may follow.

This is difficult territory for the journals, who will doubtless be criticised if it turns out the published material was seriously flawed.

But it’s easy to see why they would consider this an important paper to publish. In the middle of a pandemic, with answers to potential treatments desperately needed, a US president is inciting people to take drugs that could potentially be dangerous and are suddenly in short supply, even for people who need them for another condition – lupus. Countries are scrabbling to buy up stocks in hope of saving lives.

Meanwhile, dozens of papers on hydroxychloroquine and its older version, chloroquine, are being published online in what is called pre-print form, which means without the peer review that major journals habitually carry out.

Peer-review is seen as a safeguard against error and fraud. Journals send out draft studies to a number of experts in the particular field – perhaps three or four leading experts who will mull over the contents, methods and plausibility of the findings, make criticisms and give the thumbs-up or down to publication, which the editor does not have to accept.

The study by Prof Mandeep Mehra of Harvard and the Brigham and Women’s hospital in the US on hydroxychloroquine and the original anti-malarial chloroquine went through a form of peer-review that was more rapid than usual because of the pandemic.

Nobody appears to have spotted the error that the Lancet has since corrected concerning the numbers of patients in Australia – who turned out to be in Asia. That may be because they were overworked and rushing, as scientists now are, but it may also be because they didn’t see the database.

But that is normal. The database belongs to Surgisphere. The company’s role in the study was to provide the data from the hospitals on how patients fared on the drugs or without the drugs. That was why Sapan Desai, CEO of Surgisphere, was second author. The authors of a paper are responsible for the accuracy of the data they use. Where they have obtained it from a company, they would normally make some spot checks to be certain that what they have been given to analyse is reliable.

Prof Mehra and the other co-authors who are not part of Surgisphere, dismayed by the criticism, have urgently set up an independent review of the data – how and where it was collected as well as its accuracy – from an external audit company in Bethesda. Their report is expected by the end of the week. Mehra said he never expected his observational study to be the last word on these drugs and stressed that they used the data they could lay their hands on because the matter was urgent.

“I have routinely underscored the importance and value of randomized, clinical trials and articulated that such trials will be necessary before any conclusions can be reached,” he said in a statement.

default

“Until findings from such studies are available, given the urgency of the situation, using the available dataset was an intermediary step. I eagerly await word from the independent audits, the results of which will inform any further action.”

There are lessons here. One is that peer-review is a flawed process, as scientists will all tell you. It is a safety net, but it depends on many things, including the time reviewers have to spend on a study. And they see the draft paper, not the data it is based on. So it is hardly surprising that it was not the selected experts who spotted mistakes, but scientists from around the world who realised the numbers of patients in some of the hospitals could not be right.

But most importantly, observational studies have serious weaknesses and should never deliver the final verdict on drug treatments. This was a very big observational study, which could show trends, but there is always a danger it was not comparing like with like. The patients given hydroxychloroquine in India may not have been treated in the same way as those given it in France. The gold standard is the randomised controlled trial, set up purposefully to answer the question. The Recovery trial, with more than 11,000 patients enrolled in every acute hospital in the UK, is the biggest such trial in the world. It is testing seven treatments including hydroxychloroquine. By July, we should have a genuine answer – not one that merely muddies the waters.