Florida’s Surgeon General, Dr. Joseph A. Lapado, made waves when he announced in early March that Florida would “be the first state to officially recommend against the COVID-19 vaccines for healthy children” and directly contradict CDC recommendations.
The recommendation is based in part on data from New York, reported in an online preprint in late February. The paper described a large drop in Pfizer vaccine effectiveness (VE) for 5-17 year olds after 6 weeks. The findings made headlines, fueling alarm about supposedly rapidly-waning immunity among children. The results were ripe for cherry-picked “proof” that there’s not much reason to vaccinate kids—if the vaccine protects for only a few weeks, why bother?
Researchers have shared their work at a rapid pace throughout the pandemic, often by posting results online, where they are covered by journalists before peer reviewers have a chance to weigh in. Even in this environment, we find it astonishing just how quickly a study influenced policy—especially given the key inconsistencies we were able to identify in the work.
First, let’s look at what the authors of the preprint did to examine how quickly vaccine-induced protection decreased in children. Using immunization records and lab testing data, they identified children in New York aged 5 to 17 years old who became fully vaccinated (that is, they were at least 14 days out from a second Pfizer dose) between December 13, 2021 and January 2, 2022. They created six groups of children based on two factors: the calendar week in which they were first fully vaccinated (week of December 13th, 20th, or 27th) and age group (5-11 or 12-17 years of age). The age groupings are based on the Pfizer vaccine dosage, since a higher dosage is approved for 12-17 year olds than for 5-11 year olds.
In order to tell how well the vaccine was holding up, the researchers looked at the rates of COVID-19 each group of kids experienced throughout the month of January. The goal was to see how much lower rates were in children fully vaccinated two weeks ago, three weeks ago, and so on, compared to unvaccinated children. If the vaccine wanes, the difference in the rate of COVID cases among the vaccinated and unvaccinated children gets smaller with each successive week since vaccination.
The authors report that vaccine effectiveness for 5-11 year olds dropped from 65 percent at less than 2 weeks after full immunization to -41 percent at 6 weeks after immunization. Yes, that’s a negative number for vaccine effectiveness, and it’s biologically implausible. As vaccine epidemiologists, that result was the first flag telling us to take a closer look. The authors include a footnote that “negative [vaccine effectiveness] values observed in later timepoints likely reflect estimator instability and/or residual confounding” but these issues would affect all values, not just the negative values. In plainer terms: the authors are saying “hey, we know there could be a problem here.”
The drop in VE for 12-17 year olds is not as dramatic, but still weird: 76 percent to 46 percent over this same time period. Based on these decreases, the authors concluded that vaccine-induced immunity waned for both dosages of the Pfizer vaccine but waned faster for 5-11 year olds, triggering lots of discussion about whether Pfizer had picked the right dose for young kids.
Such a fast and dramatic drop over a six-week period is biologically suspect and motivated us to dig deeper into the numbers. Using data in the appendix of the paper, we plotted vaccine waning in 5-11 year olds, keeping the groups of children—who became fully vaccinated at different times—separate. If waning is a purely biological phenomenon, the three groups of children should have a similar level of vaccine protection at three, four and five weeks after becoming fully vaccinated—that is, the lines on the graph would overlap. Yet we see that the line for the earliest vaccinated group of 5-11 year olds – those first fully vaccinated the week of December 13th – isn’t anywhere close to the lines for the other groups. This suggests some fundamental (and unaccounted for) differences between this group and others.
Something else is likely going on with the New York data to cause this strange result. Maybe the children in the first group to be vaccinated—the ones for whom the vaccine seemed to wane the fastest—are at highest risk for catching the virus, and therefore would be expected to have a higher rate of breakthrough infections and lower VE. Or maybe, children in that group are the ones most likely to be tested and have their cases recorded in the data. If we see such big differences between kids vaccinated only a few weeks apart, imagine how different these kids are from the unvaccinated. Unvaccinated children—against which the vaccinated children were compared—may be the least likely to be tested. They could also be more likely to have been infected in previous waves.
Unfortunately, our hypotheses cannot be explored further with the data available. Unlike some other studies on vaccine effectiveness in children, the New York database does not include a lot of detail on the children studied, like testing behaviors or whether they have comorbidities. That lack of detail leaves greater potential for bias as researchers cannot adjust for the many complex reasons why some children are more likely to be vaccinated, more likely to get infected, and more likely to be tested.
Other studies published after this preprint have helped to fill in our understanding, and findings don’t always agree. A Centers for Disease Control and Prevention study published in early March found that protection against disease was pretty similar between 5-11 and 12-17 year olds, unlike the findings in the New York preprint. A study published this week in New England Journal of Medicine looked at pediatric hospitalizations, finding 68 percent protection in 5-11 year olds. (The New York study reported waning protection against hospitalization, but due to small numbers, the estimates were overwhelmed by uncertainty.) Protection against hospitalization is typically much longer lasting than protection against infection.
As frustrating as it can be, it takes time for the full picture to develop. While vaccine protection against infection has been lower against omicron, and does wane over time, the fast and dramatic drop seen in New York is likely a data artifact and not a reliable reflection of reality. We recommend proceeding cautiously with this preprint instead of jumping into major policy shifts. Unfortunately, it’s already been snapped up and misused by policymakers looking for an excuse to undercut COVID-19 vaccines.
