A controversial new review of the HPV vaccine, which is recommended for boys and girls to prevent different types of cancer, suggests the shot’s safety was not adequately tested in the clinical trials leading up to its approval.
Scientists widely agree that the vaccine has enormous potential to save lives globally, but unproven fears of side effects have hampered its uptake. An investigation I did for Slate in 2017 revealed significant flaws in the manufacturer’s attempt to vet its product’s risks. While the new research supports these findings, anyone looking for quick conclusions can stop reading now. This is a story about process, about the messy, difficult nature of scientific inquiry, and it offers no easy answers. As Dr. Lars Jørgensen, who worked on the review as part of his Ph.D. studies under Dr. Peter Gøtzsche, an outspoken critic of the pharmaceutical industry, told me, “I think it’s really hard to come up with a conclusion set in stone based on this kind of data.”
Safety concerns have been swirling around the HPV vaccine for years—in the news, on social media, and in the scientific literature—but health officials across the globe have rejected those misgivings nearly unanimously. And the vast majority of studies, many of which have considered real-world data from the decade-plus the vaccine has been on the market, have failed to support the theory that it is dangerous.
To dig deeper, Jørgensen and his colleagues launched a sweeping analysis of drug companies’ vast study reports on the vaccine, which are largely kept confidential. And while they found much to criticize, their study ultimately revealed no solid evidence of serious side effects either. So what does this mean for people considering the HPV vaccine? Jørgensen’s advice to parents is to continue to follow the recommendations from health authorities. In turn, he hopes those agencies will take stock of his research and use it to continue to bolster testing for new medicines, particularly those that will be used widely and prophylactically like the HPV vaccine (his full dissertation is available here).
The review is the most ambitious to date of the clinical trials of Merck’s Gardasil and GlaxoSmithKline’s Cervarix, two HPV vaccines that have been given to hundreds of millions of young people worldwide. It looked at studies that tested the vaccines against a nonvaccine solution or an already approved vaccine, a type of research design intended to gauge a treatment’s risks and benefits before it is licensed for marketing. While not conclusive, the findings do spotlight potential signs of rare neurological harms that outside experts say warrant a comprehensive look at the raw data, and they paint a damning picture of how the manufacturers evaluated their products’ safety. Dr. Tom Jefferson, Jørgensen’s co-supervisor at the Nordic Cochrane Centre in Copenhagen, where the research was done, said safety was reported “higgledy-piggledy” in the 22 industry trials he and his colleagues examined. He added that the trials were “all at high risk of bias” and not “a fair test” of the vaccines.
For instance, for nearly three-quarters of the trial participants, the reporting of serious harms was hampered by incomplete follow-up that lasted only a fraction of the studies’ duration—a flawed design that was directly dictated by confidential research protocols. In the Gardasil trials, some of which ran for years, study personnel were only required to report serious medical events for 14 days following each shot. Participants who developed life-threatening or disabling disease after, say, 16 days could not count on having their ailments thoroughly assessed and reported as a potential side effect (referred to in medical-speak as an “adverse event”).
The truncated safety follow-up in many trials meant that they found very few serious adverse events, whereas other trials, with longer follow-up, found lots. One four-year Cervarix trial with full follow-up reported such events for 9 percent of participants, for example, compared with less than 1 percent in the largest-ever randomized placebo-controlled trial of Gardasil, which also lasted years.
Researchers use percentages like these to find out if an adverse event could be caused by the vaccine or if it’s just “background noise.” In the latter case, such events should occur with equal frequency in the trial’s two groups of participants, or “arms”: those randomly assigned to the vaccine and the comparator, respectively. The fewer adverse events a trial is set up to record, the less likely it is that researchers will be able to discern a real difference between its two arms from noise—in other words, to detect potential side effects. By the same logic, they would also be less apt to pick up rarer side effects, or those that tend not to show up immediately.
What’s more, the manufacturers virtually always benchmarked Cervarix and Gardasil against licensed vaccines or proprietary vaccine ingredients known as aluminum adjuvants rather than against an inert saltwater placebo. This could mask potential side effects, Jørgensen and his colleagues argue, if the comparators turn out to carry risks of their own (health authorities say years of use have showed adjuvants to be safe).
The bottom line? “As the included trials were primarily designed to assess benefits and were not adequately designed to assess harms, the extent to which the HPV vaccines’ benefits outweigh their harms is unclear,” the team concludes in their article, published online on Feb. 28 in the journal Systematic Reviews.
Joshua Wallach, who studies research practices, or meta-research, at the Yale School of Public Health in New Haven, Connecticut, agrees. But he added that this is not at all unusual in drug development and cautioned about jumping to conclusions. “The takeaway is really about transparency,” Wallach said. “When reading this study, I was left with the opinion of, OK, there is uncertainty, we need additional studies with longer follow-up as well as meta-analyses that have access to individual-patient-level data.”
But Dr. Cody Meissner of Tufts University School of Medicine in Boston, who has advised the government on Gardasil and other vaccines, rejected the research flat-out. “My first impression is that the findings of the authors are so inconsistent with other publications regarding HPV and the HPV vaccine that to me this paper seems to be a pretty dramatic outlier,” he told me last March. “I think it is very unfortunate that this paper is going to be published, because it will generate concern that is not justified. This is a remarkably safe and remarkably effective vaccine.”
A commentary published along with the review strikes a similar note, arguing that the authors’ statistical methods “systematically distorted the presentation of rates” of certain adverse events. The review’s “claims about serious and rare neurological harms” are sure to be its “most influential conclusions,” writes Hilda Bastian, who has participated in earlier reviews of the vaccine and has criticized Jørgensen and his colleagues’ work before. “That is extremely worrying because I believe the authors are on very shaky ground here.”
Given the drops in HPV vaccination rates that have followed safety concerns raised in the media, she adds, “the stakes in discussing potential vaccine harms are high, both in the need to openly scrutinize the potential for harm and the need to do it responsibly. Only a very rigorous assessment could move us forward. I do not believe the Jørgensen et al. systematic review provides that.”
The journal did take a very long time to release the review after accepting it more than a year ago, when most of the reporting for this article was done. In a note on its website, it hints at why publishing a study whose conclusions are “contrary to current thought” might still be worth it. “One of the most important features of science is the debate between similarly qualified experts about the relative merits of a hypothesis or results of a study,” the journal posits.
The first versions of the HPV vaccine became available in 2006. Infection with HPV, or human papillomavirus, is estimated to be responsible for more than 33,000 cancer cases every year in the U.S., most of which are thought to be preventable with vaccination. In December 2017, researchers reported a drop in HPV-related cancers among women vaccinated as part of two clinical trials. And last April, a study showed that routine Cervarix shots had led to a “dramatic reduction” in precancerous cervical disease in young women in Scotland.
As Meissner put it, “We’ve always wanted a vaccine against cancer. We’ve now got one. Let’s use it!”
Neither manufacturer addressed the shortcomings in trial design after I emailed them the review for comment. Instead, they echoed Meissner’s point, emphasizing that regulators and other health authorities had repeatedly found the benefits of their products to outweigh the risks. Merck also pointed to a 2017 systematic review commissioned by the World Health Organization. That review, based on published studies, found that both clinical trials and post-marketing observational research were “very consistent in finding that there is no relationship between any serious adverse event and HPV vaccination.” But the authors also noted that the clinical trials were erratic in how they reported serious harms, observing that “nearly every trial included in this review claimed to be a ‘safety and efficacy’ study. However, the focus of the vast majority of studies was on efficacy and immunogenicity, with safety a secondary concern and affording a small portion of the published study report.”
Jørgensen and his colleagues wanted to go further than just analyzing the medical literature. Traditionally, independent researchers who are interested in the evidence base for a medical treatment have been left to pore over published reports. They may not be seeing the full picture, though, because studies that find a treatment to be ineffective are less likely to make it into medical journals—an example of the phenomenon known as reporting bias. Studies have also been found to selectively describe some results, typically positive ones, while omitting or spending little time on those that are less flattering, such as adverse events. To work around these problems, the team created a thorough index of all the clinical studies of HPV vaccines that they could sniff out, published or not; then they requested the corresponding clinical study reports from the European Medicines Agency under a transparency policy that grants access to the agency’s documents.
Clinical study reports are massive scientific documents that companies create based on proprietary research. They may share them with regulators as part of marketing applications for their products. Compared with the short articles published in medical journals, these reports are treasure troves of data. But they are typically kept secret and therefore rarely included in systematic reviews and meta-analyses, which are generally seen as the best tools for summarizing medical evidence. (When they are included, the results can show a very different story than the one told by published studies, as with the influenza drug Tamiflu, which Jefferson found in 2014 was not as effective as its manufacturer had let on.)
Thus began a three-year marathon of grueling back-and-forth that eventually resulted in the piecemeal release of 18 heavily redacted reports, out of 29 held by the agency. The researchers also obtained documents from an online trial register, where GlaxoSmithKline has made certain redacted study reports available. In total, they retrieved 24 of 50 eligible reports, or more than 58,000 pages, representing data on some 95,000 study participants. (This was 22,000 participants more than included in a recent Cochrane review that the team has criticized.) “It started out as good old grunt work, where you have to go through all these pages and find out where the important things are. It takes a long time—it’s something like reading 200 pages a day,” said Jørgensen.
What the team found was generally consistent with the findings that have already been published, as they detail in a companion paper in Systematic Reviews. While the vaccine didn’t cut the rate of cancers in the trials, that’s hardly surprising; most HPV infections are cleared by the body without causing any harm, the few that remain and trigger malignant growths typically take decades to do so, and most trials lasted no more than a few years. Meanwhile, the vaccine did appear to cut the number of early cell abnormalities that can progress to cancer, as well as HPV-treatment-related procedures such as cone biopsies of the cervix.
Like the WHO review, Jørgensen and his colleagues found no evidence that Gardasil or Cervarix triggered an overall increase in “serious harms.” These are typically defined as medical problems that lead to death or disability, are life-threatening, or put a study participant in the hospital. The data they studied also contained no diagnosed cases of any of the rare neurologic disorders that have been described in scientific case reports and in the media as being possibly connected to the vaccine, including chronic fatigue syndrome; complex regional pain syndrome, or CRPS; and postural orthostatic tachycardia syndrome, or POTS. (These conditions have only recently come on the radar of most physicians, so they might not have been diagnosed, even if they were present.)
The researchers did find a lot more “general harms” in trial participants who got the HPV vaccines rather than the comparator. These were things like muscle aches, fatigue, and headaches—well-known and generally minor side effects of all vaccines. Whether some participants experienced problematic clusters of these symptoms, however, was unclear because the researchers didn’t have access to individual participant records.
There were also a few dozen extra cases of serious nervous system disorders in vaccine recipients, as well as a small increase in serious harms that were judged “definitely associated” with POTS or CRPS by a physician who didn’t know whether a given participant had received the vaccine or not. But as Bastian notes in her commentary, those symptoms “are exceedingly more likely not to be associated with POTS or CRPS than they are to be a signal of a rare neurological condition.”
There is a deeper reason that giving these findings a lot of prominence is a dicey proposition. The problem hinges on statistics: Each of the results was statistically significant on its own, which means it would be unlikely to have occurred because of random noise. But the researchers did so many statistical comparisons—166 to be exact—that they would expect to see more than a handful of false alarms purely by chance. “I would think that there are going to be spurious findings among 166 analyses,” said Yale’s Wallach. “This is really looking for potential signals worth further exploring.”
Jefferson stressed that he sees the review as “interim,” and he and his colleagues are currently working to expand on it based on newly released information (they describe one ongoing analysis here). In 2018, a Canadian federal court ordered that country’s government to release all the drug company data it holds on HPV vaccines, without redactions, to Peter Doshi, one of Jefferson’s collaborators. “Regulators shouldn’t have a monopoly on judging the risks and benefits of medicines or hinder others from doing the same via confidentiality agreements,” Doshi, an associate professor at the University of Maryland School of Pharmacy in Baltimore, told the Canadian Broadcasting Corporation after the ruling.
Regulators are not worried about the new review and have not changed any recommendations in response to it. In a number of emails to me, the European Medicines Agency said it had found “no evidence” in its safety assessments that the vaccines cause “serious neurological problems.” And the U.S. Food and Drug Administration wrote in an email, “Based on the robust body of scientific information evaluated by FDA, we are assured of the safety and effectiveness of Gardasil, Gardasil 9, and Cervarix, and as we do with all vaccines, we will continue to monitor the safety of Gardasil 9.” The agency did not answer any of my questions directly, however. The Centers for Disease Control and Prevention, for its part, declined to comment, citing “a long-standing tradition of not commenting on research or papers authored by those outside of CDC.”
But Susan Ellenberg, who directed the Office of Biostatistics and Epidemiology at the FDA when the design of the Gardasil studies was discussed and now works at the University of Pennsylvania Perelman School of Medicine in Philadelphia, welcomed the new review. She emphasized that “the FDA is exquisitely sensitive to potential adverse effects,” but added that she, like Yale’s Wallach, found the researchers’ criticism of the trials’ design “legitimate.” While she worked at the FDA, Ellenberg pushed for making trials of childhood vaccines “much bigger,” and she said she would have liked “more intense follow-up” of safety. “I think in general we don’t get enough long-term information about serious adverse events from vaccine trials,” she said. “There is an assumption that vaccines are safe, and if I have to bet one way or the other, I believe that they are safe, but I would like to have more data about it.”
