In 2003, a committee established by the National Football League began releasing a series of studies that downplayed the seriousness of getting your brain bashed into your skull. On Thursday, the New York Times published a front-page story reporting that the database the league’s concussion committee relied on for those studies omitted more than 100 diagnosed head injuries—including serious ones to star players like quarterbacks Steve Young and Troy Aikman. In a related editorial, the Times details how it decoded the NFL’s concussion database, which cataloged injuries from 1996 to 2001, to find out which teams and which individual players sustained injuries.
Clearly, this is important stuff. But to the casual onlooker, it isn’t clear what’s new in the Times report. Let’s break it down. (The story also reveals alleged links between the NFL and the tobacco industry, which we won’t address here.)
The headline of the Times piece, written by reporters Alan Schwarz, Walt Bogdanich, and Jacqueline Williams, reads “In N.F.L, Deeply Flawed Concussion Research.” That premise has been well-established—we’ve known for years that the NFL’s in-house concussion studies were based on faulty data. Since 2006, ESPN’s Peter Keating and others have been writing high-profile stories questioning the methods researchers used to make far-fetched claims, like suggesting that repeated serious head injuries had few long-term consequences. Mark Fainaru-Wada and Steve Fainaru went further in their 2013 book League of Denial, speaking to Bill Barr, who worked as the team neuropsychologist for the New York Jets for nearly 10 years. In an interview with Barr, the Fainarus learned that the league’s concussion committee—which included neuropsychologist Mark Lovell and the committee’s chairman, rheumatologist Elliot Pellman—omitted far more than 100 data points.
From page 177 of League of Denial:
By the time Barr was done canvassing neuropsychologists from Lovell’s network, he calculated that at least 850 baseline tests—and perhaps thousands more—had been excluded from the NFL’s results. Barr was concerned that Pellman and Lovell had cherry-picked the data to reinforce the league’s argument that the impact of concussions was negligible and players recovered quickly.
As Fainaru-Wada and Fainaru explain, Barr asserted at the 2007 NFL Concussion Summit that the committee had left out thousands of baseline tests from its sixth report, published in the journal Neurosurgery:
As Goodell looked on, Barr repeated his allegations that Pellman and Lovell had left out thousands of baseline tests in NFL Paper Number 6, the paper that supposedly had shown that players recover quickly from concussions.
“I said that data collection is all biased,” Barr said. “And I showed slides of that. Basically I pointed out that we had been obtaining baselines on players for 10 years, and when you look at the study it only included a small amount of data. My calculations were that their published studies only included 15 percent of the available data. Let’s put it this way: There were nearly 5,000 baseline studies that had been obtained in that 10-year period. And only 655 were published in the study.”
Barr hadn’t come right out and said it, but essentially he was accusing the NFL’s researchers of fraud. The implication was that Pellman and Lovell had purposely excluded data that didn’t support their findings.
OK, back to the New York Times piece. To do good science, you need an accurate dataset—which the NFL clearly didn’t have. But if Barr had already publicly accused the NFL of ignoring thousands of baseline tests, why is it important that the league might have omitted more than 100 diagnosed concussions?
Well, not all data points are created equal. As the Jets’ team neuropsychologist, Barr assessed healthy players at the start of the season. That meant giving each player a “baseline test,” a neurological exam that tests basic brain function, concentration, memory, and motor skills. It’s the same diagnostic you use to assess whether a concussion has occurred and, if so, how severe it is: The player takes the test again shortly after sustaining a potential head injury, and the results from that second test are compared to the baseline.
Back in 2007, Barr asserted that a huge number of those original baseline exams—tests on healthy players at the beginning of the season—were missing from the NFL’s data set. The Times, by contrast, is talking specifically about concussed players. This data on players who’d suffered head injuries formed the backbone for most of the 13 peer-reviewed studies the league released between 2003 and 2006, including the one Barr referenced in League of Denial.
Based on the Times’ calculations, the committee omitted more than 10 percent of the total number of concussions sustained by NFL players between 1996 and 2001. Then, it used those artificially low numbers to calculate the NFL’s rates of head injuries. The league drew upon that incomplete data when it wrote studies concluding that players recover quickly from concussions, that players with several concussions experience similar damage as those who’ve had just one concussion, and that returning to play soon after sustaining a concussion does not predispose you to more brain damage.
In an interview, Barr told me it’s bad enough to remove data about healthy players. What the Times is reporting is even more egregious: “It just shows us that the data omission went through to all levels of the analysis.”
Even knowing what he knows about the NFL, Barr says he was still surprised that the committee would omit such a large number of diagnosed concussions, many of them well-known and in the public record. “I personally thought that [the data omission] just involved the neuropsychological baseline data,” says Barr, who now directs the neuropsychology division of the Comprehensive Epilepsy Center at New York University Langone Medical Center.
In its statement on the Times’ story, the NFL acknowledges that “the data set had limitations” and that the “studies never claimed to be based on every concussion that was reported or that occurred.” But according to the Fainarus, this is not an accurate characterization. When the NFL published its concussion studies, it implied that they were based on a comprehensive tally of concussions from 1996–2001. For instance, the seventh study reads: “Earlier articles in this series have documented … the entire cohort of all NFL players in the [mild traumatic brain injury] registry who sustained concussion during a recent 6-year period.”
Here, in the simplest possible terms, is what’s new in the Times’ Page 1 concussion story: We now know that the league’s scientific committee removed players who got concussions from its studies on how many players get concussions. In the big picture, that revelation might not change much: The league looked bad before the Times report, and it looks bad after. It’s another important incremental step, though, in revealing what the NFL knew and what it didn’t want us to know.