Can We Even Trust the Testing Data?

Listen to this episode

S1: So, Robinson, when people ask you, like, how many people have had the corona virus in the United States? I’m kind of curious. You’re really deep in the numbers. Do you just kind of laugh?

S2: Yeah, there’s not I don’t think we have any where close to an answer for that right now.

S1: Robbins and Meyer is a reporter for The Atlantic.

S2: Even with perfect data, we would probably never fully know. It’s so funny because it’s the most important question to some degree. Like it’s the one question we’d really, really love to know about how bad this outbreak is. And it’s the one we can answer.

Advertisement

S1: Usually, Robinson covers the energy and climate beat, but in the last couple of months, he’s reinvented himself as an expert on testing for the Corona virus. He’s helping run The Atlantic’s Cauvin Tracking Project. It’s made him hyper aware that asking who’s had the Corona virus? It may sound simple, but answering that question is anything but.

S2: The question we’ve always been trying to answer. To some degree with the Kobe tracking project from the beginning was not even how many people have gotten sick, but just trying to understand the scale of how many people have gotten sick. And that meant asking how many people have been tested for COPD.

Advertisement

S3: And the CDC was releasing testing numbers basically through January and February, and then on February twenty ninth, it stopped publishing them. It just said, hey, state labs are going to be doing a lot of this testing and that’s it.

Advertisement
Advertisement
Advertisement

S4: So Robinson and his colleagues assembled a team of volunteers to meticulously track what each state was reporting. Every day, dozens of them watch press conferences, download testing data from the local health departments. They’ve been doing this for months.

S2: Our first goal was we assumed the CDC had this data somewhere in the agency and that it had just stopped publishing it because the numbers were flattering. So was a little bit of like naming and shaming. It was, yeah, it was very much naming and shaming. That was our theory of change for the first two weeks. And then we kept doing it. And the CDC kept not publishing the data. Then we would announce milestones like, oh, the country has tested one hundred thousand people and then the CDC would announce them the next day or the White House would announce that the next day.

Advertisement
Advertisement

S1: So they were following you?

S2: Yeah, I think slowly started to dawn on us that the White House didn’t actually have a better internal source of data.

S1: This volunteer project is now one of the most comprehensive and reliable sources of coronavirus testing information out there. But last week, very quietly, the CDC launched a kind of Kovik tracking project of its own, a Web site keeping tabs on infection rates, testing numbers, exactly what Robinson and his colleagues had been hoping to shame the CDC into doing months ago.

S2: Basically, someone happened to see it while looking for other data, and they were like, do we know about this? We were just like, wait, this is up. This is online. When did this go online?

Advertisement
Advertisement

S3: We were like, they’re publishing data. It’s amazing. But when you dug into the data, what did you find? We found it. It’s it’s funny. I mean, so there are state by state test counts really didn’t match up with ours. Do we know why there’s this discrepancy? We don’t. We’ve asked the CDC why and they haven’t.

Advertisement
Advertisement
Advertisement

S4: They haven’t provided an answer on today’s show. We closed the country up because we just didn’t have enough tests. But now that our capacity is ramping up, there’s a different problem figuring out what all this new data means and who’s giving the full story about it. I’m Mary Harris. You’re listening to what next? Stick with us.

Advertisement

S1: For the last two months, the story’s been the same. It’s been this continual drumbeat. We need more tests. We need more tests. And that’s still true, right?

S2: It is still true. It is absolutely still true. At the same time, things are much better now than they were in mid-April and they’re much better now even than they were two weeks ago. However, we still need more tests.

S1: Can we be really clear about why the testing is so important? What it allows the government to do? Because it allows you to get control and this really important way that I think we should articulate.

S2: Yeah. So testing is the first most important step of basically the only strategy we have for dealing with a pandemic until there’s a vaccine which is keeping infectious people and contagious people away from susceptible people. And testing is so important because, like, that’s the whole strategy, right? When we lock down, even though it’s not a lockdown, but when we do the thing we’ve decided to call lockdown, what we’re effectively doing is being like there’s so many infected people out there that just everyone needs to stay in their homes. And that’s how we’re gonna keep infected people away from susceptible people, because we just don’t know, because we just don’t know. And so and there’s no way to find out. We don’t have enough tests to find out the diseases possibly so spread. We can’t find out over time. What we want to be able to do is be able to say, okay, these are the infectious people were identifying them and we can just preemptively keep them away from the susceptible population. And and then this is several population can live something closer to its normal life. And that’s how we’ll manage the disease.

Advertisement
Advertisement
Advertisement
Advertisement
Advertisement

S1: Think of it like putting a dog on a leash. Like, you know, if you are walking your dog without a leash, it’s going to go everywhere and be crazy. But if you have the dog on a leash, you begins to go crazy. Pull it back, you pull it back. And so if your tests are showing more coronavirus, you can pull back a little bit. You can say, OK, maybe some schools are going to close. Maybe some stores are going to close, maybe people are going to stay home.

S2: Well, and even at the most basic level, if your tests are showing more, kind of like the best case scenario here is that you just find all the cases of coronavirus before there’s too much mixing in this general population. And even before you need to put in these society wide measures, you can just basically place people in quarantine. Right. You can say you are infected. You need to stay in your home, but everyone else can just kind of go about their lives. So it’s not only a like a temperature check or a leash for society. It’s also at an individual level. It’s the most important tool we have.

Advertisement

S1: There are two kinds of coronavirus tests. We’ve all been hearing about. One is the test you get when you’re sick. The viral test, usually they do it with a nasal swab. Sometimes it’s a saliva sample. The tests will tell you if you are currently infected with the virus. The other test is a blood test, also called a serology test. It looks for antibodies that show whether you’ve already fought the virus off. So it’ll tell you if you had the virus in the past where Robinson found this discrepancy between the states and the CDC. It’s with the way they report that first test, the viral test. And the reason he’s been so focused on that particular metric is that it tells you a number of things.

Advertisement
Advertisement
Advertisement
Advertisement

S2: At the same time, you care about testing for active infections, for looking for active infection for two reasons. The first is that basically by the number of people who are getting tested for Kofod, who turn out not to be sick. That gives you some confidence that, like a lot of people are getting tested and you’re probably catching almost all of the sick people in your testing regime. The second is that it basically gives you your baseline understanding of just how many people are sick in your area.

S1: So you’re saying looking at this data can do a couple of things. It doesn’t just give you a sense of how many people are testing positive, like how many people have the coronavirus out there. It also gives you a sense of just the health of the overall system and its capacity. Yes, exactly. You said that the CDC hasn’t really said much about these differences in their data versus the state level data. When you talked to experts, when you talked to people who would know what might be going on here, what did they say to you about what could be happening?

Advertisement

S2: So there’s this funny aspect of this, which is the CDC basically says as many tests have been conducted nationwide as we do, we both think as of Friday that there’s ten point seven, ten point eight million tests have happened nationwide, but it apportions them across states. In a completely different way than we do. Oh, that’s weird. Yeah, exactly. And some states are higher. Some states are lower. So it’s not like the CDC is just underreporting states across the board. It’s overreporting some states and it’s underreporting other states. And that’s kind of worrying because it means that there’s no consistency. It’s harder to cite a single methodological reason why it might be happening. Here are some theories. The first is that in some states, the CDC is including the results of antibody tests. It’s not only just counting these nose swab viral tests.

Advertisement
Advertisement
Advertisement
Advertisement

S1: So not just whether you actively have the virus, but whether maybe you had the virus.

S2: Exactly. Those tests are like looking for a completely different thing than we’d want the CDC to be looking for in this data set. There’s another theory, basically, that the states are reporting test results that affect their residents. While the CDC is reporting test results that happen territorially within that state. And so if I get tested for coded in Camden, New Jersey, and my test result goes to Philadelphia to be tested, that test result is reported by the state government of New Jersey because I’m a New Jersey resident, but not the state government of Pennsylvania. However, the CDC reports it is happening in Pennsylvania. Because it was analyzed by a lab in Pennsylvania. This is another theory about how this discrepancy could be happening.

S1: It seems like they would have worked that out.

S2: It does seem like they would have worked that out.

S1: But you also reported that in the early days of this pandemic, the vice president’s coronavirus task force sort of reorganized the way all this information even reached the CDC.

S2: Exactly. So normally what happens with the CDC is that they they do almost exactly what the covert tracking project does, which is they go to states. And they collect data from states and that makes up the CDC data and in fact, the CDC has staff seconded to state governments often. So there’s CDC employees within state health departments. That is not what they’ve done. Corona virus, what they’ve done for the current virus is set up an entirely separate reporting system that hospital administrators and health care providers and doctors offices and these commercial labs and nonprofit labs, university labs are all supposed to use where they report their COVA test results directly to the CDC. And that’s new, that isn’t usually what happens in disease surveillance at the federal level.

Advertisement
Advertisement
Advertisement
Advertisement

S1: Yeah, I think you said in 13 states the data the states are giving diverges from the CDC is by 25 percent, which is a pretty big gap.

S2: Exactly. Yeah. So. So in Florida, for instance, the CDC reports there being 30 percent more tests happening in Florida than the state government of Florida reports.

S1: To put a finer point on this, the state of Florida says it’s done around 700000 corona virus tests, but the CDC pegs that figure. Way higher, like 900, 19000.

S2: And that’s be on the scope for paperwork error, basically, because at the current rate, the state governor of Florida says it is conducting tests. It will hit that nine hundred nineteen number in two weeks like that is either we are getting results for two weeks in the future from the CDC point of view or there are real major discrepancies here.

S1: I want to talk about Virginia, because I know that Virginia at the state level, they were doing one of these things that can confuse.

S2: Exactly. And they were doing it, in fact, with the intent to deceive.

S1: Huh. So tell me a little bit about that. I know they were counting all of these Corona virus tests, both blood tests that you get after the fact and the nasal swabs and sort of putting them in one pile. Why is that problematic and why would they be doing it in the first place?

S2: So the two kinds of tests tell you different things, right? The antibody tests tell you how many people across the general population are sick, have been sick, and the viral tests tell you how many people are sick right now. And because of that, they’re testing very different populations. Right. The viral tests are really only testing at risk people and people who might be sick. Zero tests. The blood tests are trying to test the whole population. They’re trying to get a good sample of everyone in Virginia because of that, the positive rate on the viral tests is much higher. For purely natural reasons than it is on the blood tests, because with the viral tests, you’re trying to test sick people or people who might be sick or people who are very at risk or working at risk environments. That’s a population much more likely to have covered in it than this kind of general population sample that you’re looking for in the blood tests, which is everyone. Virginia right now or Virginia, as of late last week, was in the last five states. By a number of testing metrics, they had tested among the fewest people per capita of their population and they had also had a fairly high positive rate. And so what the state government of Virginia started to do was it started to report viral tests and antibody test together in the same metric. They just said, oh, we’ve done x many tests. We’ve done this many tests.

Advertisement
Advertisement
Advertisement

S1: Did that move them up the charts? Like, did they look much better?

S2: Move them slightly up the per capita try to batho about 117 people tested per 100000. And in fact, the chief of staff of the governor of Virginia said in a press conference they were asked why they were doing this. Reporter for the Virginia Times Dispatch discovered this. They were doing this and she asked them about it in the press conference and he said, well, we think other states are doing this and there is no way to win. We. We get criticized if we don’t do it because we’re at the bottom of this ranking and now we’re getting criticized. If we do do it. There’s no way to win. So we’re going to do it. But the issue is that it is it it. Takes two useful data sources and combines them into one useless data source, because we know how to read viral data. We can’t. We can interpret viral data in certain ways. And we know how to read serology data. We can predict in certain ways, but we don’t really know how to read them together. They’re not really useful together.

S1: That’s so craven. That’s just like just saying like, listen, other places are juking the stats. So we’re going to this.

S2: We’re going to do it, too. And like me, if it means we can reopen sooner. That’s what we’re gonna do.

S1: I mean, in Governor Northam’s defense, he did have a press conference where his response was, listen, if you separate out the antibody tests and the nasal swab tests, the trends remain the same. Is that not a legitimate defense here?

Advertisement
Advertisement
Advertisement

S2: That’s fine, though. The trends we care about for antibody tests and viral tests are different. Right. Like viral tests, we want the positive rate to go down because ultimately we want fewer and fewer people to be sick. But antibody tests, the positive rate won’t really go down. If you’re sampling right now, it can only go up because more and more people will be exposed to the virus. So basically, Virginia’s positivity rate of their tests did not change very much when they took out the antibody test. So I think it was 14 percent. And then they took out the antibody test and then it was 15 percent. However, there weren’t that many antibody tests in the sample and they were mostly negative. And so they like they were getting a lot of they were getting a big shot in the arm of negative tests by including these antibody tests.

S1: It’s funny because listening to you, I just feel like there are these levels of chaos, like each level has its own world of chaos. You have the tests themselves and there are a lot of different tests. And then you have kind of how the data from these tests is presented to the public. And that has its whole other level of chaos because the states are doing one thing with it. And now it looks like the federal government is doing its own thing with it. It just it feels like it’s hard to know the truth when there are so many complicating factors at every level of information.

S2: I think that’s right. The further you get into this data, the more problems you see with it. But to some degree, that’s always what data is like. Right. Data always has to be made. It always has wrinkles that the people who work with the data know about, but that if you don’t work with the data are not as clear to you. And I think one of the things that’s been happening with this pandemic is we’ve all had to engage with the kind of ambiguity and uncertainty associated with working with scientific information, which is to say that we science doesn’t know anything. It knows a lot, but it’s always fighting to people to say things clearly and say things truthfully. And there’s always a big there’s always a ton of work you have to put into people to say something is true or accurate. And where we’re working right now is that like a society level in France.

Advertisement
Advertisement
Advertisement

S1: Some of this ambiguity would just be clarified with rules, rules about here’s what each state needs to be reporting and who they need to be reporting it to. Rules about when the CDC releases testing data to the public.

S2: I think the rules that we really need are rules from the CDC about exactly how each state should be publishing its data. And what they’re allowed to do and what they’re not. And right now, that’s what we don’t have. And so it means that all the states are reporting data a little differently. And also, the CDC is collecting data through a completely different method than the states. I mean, right now, the country’s testing situation is improving, but it is improving like 56 cats, all kind of arriving to eat at the same moment at the same time of the day. Everyone, every individual cat having take it. It’s different out there. There is no central authority. There is no central task force that is planning how we’re going to build out test capacity.

S1: Even though the CDC has started compiling this testing data, you’re covered. Tracking project still seems to be the testing tracker of record like the White House cites you. What do you make of that? I mean, as a journalist, I guess it’s rewarding.

S2: It’s rewarding to be on a team that is doing work that so many people find useful. Right now, it is frustrating that this work is being done right now by journalists and by. Volunteers and by experts who are donating their time, and we’d much prefer that experts paid by the government are collecting this data, are the ones who do it.

Advertisement
Advertisement
Advertisement

S1: Yeah. I mean, here is this weird thing, which is that. The CDC isn’t publishing its data on the weekends, but you guys are, yes.

S2: The CDC doesn’t update their data on the weekends during this public health emergency.

S4: It’s just strange. Yeah. Yeah.

S3: Robinson, Meyer, thank you so much for joining me. Absolutely. Thank you.

S1: Robinson Meyer is a staff writer at The Atlantic and part of their Cauvin tracking project.

S4: What Next is produced by Daniel Hewitt, Mary Wilson and Jason de Leon. We have help from Alicia McMurry and Alison Benedict. And today, our honorary producer is Rubinson Meyer himself, who sweated it out to make sure we had decent audio.

S3: Yeah, I’m I’m I’m recording again. Oh. Oh, oh. Okay.

S4: Thanks for listening. I’m Mary Harris. I’ll talk to you tomorrow.