The authors of the Florida analysis appear to have modeled their approach on an unpublished paper posted to a preprint server medRxiv in March, which they cited. The preprint, by scientists in the U.K., also evaluated the risk of all-cause and cardiac death following COVID-19 vaccination by doing an SCCS analysis on the last vaccine dose given.
Unlike the Florida analysis, however, it did not identify any increased risk of death after vaccination. It also performed a similar analysis after a positive COVID-19 test — which the Florida analysis did not do — and found a large increase in the frequency of both all-cause and cardiac-related deaths after infection in people who were unvaccinated.
The lead author of the preprint, Vahé Nafilyan, a principal statistician for the U.K.’s Office for National Statistics, told us in an email that the preprint was “conducted very quickly to respond to rising concerns about the safety of vaccine in young people,” and his group had rerun the analysis using the methodology Farrington recommended. The new paper is currently under review, he said, and the preprint is being released this week.
But Nafilyan also said his team had a work-around to avoid the concern Farrington had with analyzing the last vaccine doses.
“In the initial analysis, we purposely restricted the follow-up period to 12 weeks – the minimum separation between doses,” he said in an email, referring to the longer interval used in the U.K. when the vaccines were first rolled out. “This was a quick way to circumvent the issue. The Florida study does not apply such a restriction and that is the issue.”
“For the Florida study, it would be far more tricky since the minimum gap between doses in the US was 3 weeks,” he added, “making it impossible to use the same ‘trick’ as in our initial analysis.”
Farrington agreed that this difference between the studies would mean the U.K. paper “is probably OK, whereas the Florida one is not!”
Additional Concerns
Other critics have raised additional concerns and caveats about the Florida analysis. For example, cardiac deaths were not checked by reviewing medical records, but instead were defined by the presence of certain codes on death certificates, which aren’t a guarantee that a person died from a cardiac problem.
As Dr. Kristen Panthagani, an emergency medicine resident at Yale New Haven Hospital, wrote in a blog post, the codes included cardiac arrest, “which simply means ‘the heart stopped’ and can be the terminal event for many different diseases, not just cardiac issues.”
The authors say as much in the limitations section, noting the study “cannot determine the causative nature of a participant’s death” and the “underlying cause of death may not be cardiac-related.”
The limitations section, notably, also mentions that the increased risk in cardiac deaths the analysis observed in the overall population and the 60 and over group could be due to confounding by age in the older group. Because so many more deaths occurred in older people, when the 60-plus group was removed, there was no longer any statistically significant result for cardiac-related deaths after vaccination, for mRNA vaccines, or males who received mRNA vaccines.
Panthagani and others have said that given the relatively small number of deaths in the “risk” period for younger males receiving mRNA vaccines — 20 — a change of just a few because of inaccurate coding could render the finding insignificant.
Another issue is how exactly the authors dealt with COVID-19. People with COVID-19 listed as a cause or contributing factor to death were correctly excluded from the analysis, but it’s not entirely clear if the same was done for people who tested positive for COVID-19. This could mean that some of the cardiac deaths being attributed to the vaccine are actually from the disease, Panthagani noted.
In one part of the write-up, the analysis says people with positive COVID-19 tests were excluded, but in another, “COVID testing status was unknown for those who did not die of/with COVID.”
On Oct. 10, Ladapo responded to some of the criticisms on Twitter, arguing that it didn’t matter if the diagnosis codes for cardiac deaths were “imperfect,” since they were the same for all groups. He also said the state “used all of our data resources,” including test results, to exclude people with documented coronavirus infection.
But if that’s the case, then what is written in the limitations section is incorrect. We reached out to the Florida Department of Health to clarify this issue, among others, but did not hear back.
Morris, the University of Pennsylvania biostatistician, told us that the analysis might be fine “if properly executed,” but he couldn’t verify if that was the case given the lack of detail in the write-up — and he had “several unresolved questions that make me wonder.”
Along with the confusion over the COVID-19 testing and “potential bias from considering the last dose only,” Morris said that he did not understand why some of the average follow-up times in some cases were longer than the 25-week period set for the entire analysis. He said he had asked for more information from the state health department on this and other issues, but had not received a reply.
Morris and Farrington also wondered why the analysis excluded booster doses. And critically, despite a mention of “infection” in the title and in another sentence of the discussion — which could be a typo — Morris did not understand why the analysis did not also evaluate the risk of cardiac deaths following coronavirus infection. Not only is this what the U.K. study did, but this would be “a relevant factor to weight against any vaccine-related risks.”
Analysis Poor Basis for Changing Vaccine Recommendations
Indeed, several critics have pointed out that even if the analysis is taken at face value, it doesn’t include a comparison of the risks of the vaccine with its benefits, so the analysis itself fails to show that the risks outweigh the benefits. And as a single unpublished, preliminary analysis that contradicts other published papers, it’s hardly enough to justify modifying public health guidance.
“There is a large literature on the safety of mRNA vaccines, and recommendations should be based on a comprehensive overview of the available evidence, not any one single study (even less so if it is unpublished),” Farrington said. “What I find extraordinary in the Florida saga,” he added, “is that recommendations on vaccination can be made based on a single (in this case, flawed) study, when there is ample other evidence to the contrary.”
As we said, myocarditis and pericarditis have been identified as rare side effects of the mRNA vaccines. Studies have consistently shown that for the overall population, the risk of myocarditis is much greater after a coronavirus infection than after vaccination. For some groups, such as men under 40, the risk of myocarditis may be higher after vaccination than after an infection, as a large U.K. study recently found for the second dose of the Moderna vaccine. But considering all the other possible bad outcomes from COVID-19, numerous teams of experts have concluded that the benefits of the vaccine outweigh the risks, even for younger males.
“Florida’s public health recommendation is specifically for men age 18-39. Despite the fact that this age group is at lower risk of severe complications of COVID-19 than older Americans, CDC, FDA, and other federal agencies continuously have found that the evidence is clear: the benefits of vaccines clearly outweigh any risks,” Sarah Lovenheim, assistant secretary for public affairs at the department of Health & Human Services, said in a statement. “This is why FDA authorized, and CDC recommended, that all individuals in this age group get vaccinated against COVID-19.”
Lovenheim called Florida’s decision “flawed and a far cry from the science.”
According to an expert consensus paper published in March by the American College of Cardiology, “a very favorable benefit-to-risk ratio exists with the COVID-19 vaccine for all age and sex groups evaluated thus far.”
While it’s conceivable that some deaths have occurred because of the vaccines, this risk is exceedingly small. During a CDC vaccine advisory committee meeting in February, the agency presented data showing that 13 deaths involving myocarditis had been reported among people 30 years and younger who had received a first or second mRNA dose. However, none of these was thought to be vaccine-related. We asked the CDC for an update on the numbers, but didn’t receive a response.
Studies suggest that post-vaccination myocarditis is milder than typical viral myocarditis and that most people recover within three months.
In interviews with Politico and the Washington Post, Ladapo defended the analysis and his decision to stop recommending the mRNA vaccines for young males, given high levels of immunity in the population now.
Ladapo told the Post that he hoped his former mentor, Harvard economist David Cutler, would endorse the methods used in the analysis. But according to the Post, Cutler said the analysis was flawed and should not serve as the basis for state vaccine policy.
“If I was a reviewer at a journal, I would recommend rejecting it,” Cutler said.
Editor’s note: SciCheck’s COVID-19/Vaccination Project is made possible by a grant from the Robert Wood Johnson Foundation. The foundation has no control over FactCheck.org’s editorial decisions, and the views expressed in our articles do not necessarily reflect the views of the foundation. The goal of the project is to increase exposure to accurate information about COVID-19 and vaccines, while decreasing the impact of misinformation.
