Not All of the Polls Were Wrong

Listen to this episode

S1: In Iowa, Ann Selzer is known for her political polls, polls that surprise people. I’ve heard all of these nicknames for you, like my favorite is the Cassandre of Des Moines. But do you have a favorite?

S2: Well, I’ve recently sort of resurrected the title of Outlier Queen, that is. I’ve had any number of polls that looked like an outlier, just dismissed as an outlier and then turned out to be pretty, pretty close. So, Queen, I have a little tiara collection, so, you know, that fits me just fine.

Advertisement

S1: Do you literally have a tiara collection? I literally have a tiara collection and polls don’t just catch people by surprise. They’re also surprisingly accurate. She started getting attention back in 2008. That’s when she was one of the only people to predict Barack Obama would dominate the Iowa caucuses since election results began coming in last week. There’s been this debate about the polls, how good they are, whether we should pay attention to them anymore. So many of them simply got this election wrong, predicted Joe Biden would run up big margins and that Democrats in the House and Senate would do the same. But I wanted to talk to Anne because this year she really earned her title The Outlier Queen, you’re one of the few pollsters who got something right because your last poll in Iowa really wasn’t exact, because nothing’s exact, but it showed the direction things were heading. I wonder if you can describe when that poll was released, what it showed and what was the reaction you got to it.

Advertisement
Advertisement
Advertisement
Advertisement
Advertisement

S3: So we were in the field starting the Monday, a week ahead of the election. So Monday night, Tuesday night, Wednesday night, Thursday night. We’re, of course, looking at our data on a daily basis, seeing how things are going. And it’s with relatively few interviews per night, just a couple of hundred, and there’s a fair amount of bounciness to it.

S1: You’ve talked about how, like the polling, there’s like bounce in it in the first few days, like it wobbles around before stabilizing. Is that how you frame it?

S3: It is. And there’s a reason for that. It’s very technical, which is that as the phone bank is making their first calling attempt, they’re going to get the people who are easiest to catch. So we’re always going to look older and always typically look more female, though that one doesn’t bounce quite as much.

Advertisement

S1: So how are your results shifting as you got more and more of them?

S3: Well, we had I think this is from memory, a good first day for President Trump, a good first day for Vice President Biden. And then it was a Trump day, followed by another Trump day.

S1: Those Trump days translated to a seven point lead for the president, which is about where he ended up on Election Day in Iowa. But it isn’t just that and revealed the president’s electoral strength in her home state. She also foreshadowed another trend that played out across the country the durability of Republican candidates for Congress. She predicted Senator Joni Ernst would keep her seat and her poll also predicted a Democratic House incumbent. I think an hour would lose to her Republican challenger.

Advertisement
Advertisement
Advertisement
Advertisement
Advertisement

S2: Now that’s ousting a sitting congresswoman, albeit she’s only been in one term. But that was, I think, the thing that was more alarming than the fact that we had Trump leading, which he did, and won the state in about a similar margin in twenty sixteen.

S1: You say it was more alarming. Why do you say that?

S2: It wasn’t alarming to me. It was alarming to the people who let’s be clear, primarily Democrats who felt like that race was locked up.

S1: Can you describe the heat you took after this poll came out?

S3: I describe it as a cloud of anxiety. So I have a little conversation with myself from time to time that says, look, you’ll either be golden or a goat and just be prepared either way.

Advertisement

S1: Just just you know, we’re going to we’ll know soon enough and just try to keep breathing today on the show since and Sulzer got it right, we decided to ask for her insider take on what her colleagues got wrong. I’m Mary Harris. You’re listening to what next? Stick with us. Talking to Insulza is a strange experience, on the one hand, her polling numbers were really right this election, but on the other, she doesn’t want to gloat about it.

S3: I don’t like the idea that I would be seen as bashing my polling colleagues. There’s clearly something I feel like I did. Right. So I’m apart from the the general dismay that the public is having about polling, but I don’t like to crow either.

Advertisement

S1: So it’s a little bit of a, you know, funny position to find myself and says her secret sauce that led her to get this 20 20 election right isn’t really much of a secret or a source. It’s published at the top of every poll she releases. And it’s simple. She uses random digit dialing for landline and cell and she asks people how they’re going to vote. Afterwards, she adjusts her data to make sure it reflects the population of the place she’s polling. And that’s it. She tries not to bake in any assumptions, especially about past behavior, into her current polling data, which is where she thinks some pollsters can get tripped up.

Advertisement
Advertisement
Advertisement
Advertisement
Advertisement

S3: So one of the thoughts I’ve had about Florida in particular is that there might have been pollsters who took a look at their numbers and thought, well, this isn’t the outcome I was expecting as they delved into some of the cross tabs, perhaps with the Latino community, perhaps with the African-American community, and they may have done some additional adjusting. I don’t know that to be we we would never do that. We would not think that we know better what’s going to happen in the future except what our data will reveal to us. So I’m not there trying to figure out, well, do I expect the turnout among non college educated white men to be up or down compared to last year? That’s a judgment call. I worked with a guy with a colleague, and he kept saying to me, well, you know, you’ve got to figure out the size and shape of the electorate. And I don’t know what you’re talking about. And, you know, there’s just a complete disconnect of him thinking he he can outthink the future. And my position of saying I don’t think I can do that. I think I can be in a best position to see the future when my data shows it to me. But I’ll tell you, my approach is probably the most simplistic of any of these polling firms. And so there’s perhaps a commercial advantage to having a complicated scheme and I mean scheme just in a technique and and a way of approaching it that makes people think it’s all science and therefore it’s better and it is secret and it’s protected by patterns and trademarks and all of that. And so mine looks like a second grader dream just by comparison. And all I can say is that has worked for me.

Advertisement
Advertisement
Advertisement
Advertisement

S1: This simple technique means and simply trusts what people tell her. She classifies respondents as likely voters if they tell her they’ve already voted or they definitely will. Other pollsters, they might adjust their data based on past political behavior. That means they could miss a surge of new voters. It’s not like and never gotten anything wrong, she says. In the 2004 general election, she predicted John Kerry would win Iowa’s seven electoral votes, but he lost 2004.

S3: We had in our final poll that John Kerry would win Iowa not by a lot, but enough and know that didn’t happen. George W. Bush won Iowa. And at the time I had a new client, which was Des Moines University, and I was meeting with one of their senior vice presidents and then president of the medical school as governor, Terry Branstad. And he saw me and he came in and he spent he spent 20 minutes with me with a map of Iowa, showing me where the votes were and how this thing happened and that we couldn’t have seen it in our poll because we had to stop calling people on Friday night. And at that time, George W. Bush was doing a lot of closed rallies so the press couldn’t report on them, didn’t really know. And Governor Branstad said the Sioux City rally, he’s never seen any sort of organizing event like that. And people were sent out with very specific instructions for what to do for the next twenty four hours, 48 hours. And he credited that rally for Bush’s win in Iowa. And he said he credited it for taking down Senator Tom Daschle in South Dakota. And it it it helped me some, but it was still a gut punch, did not get that right. So it took me a while to become. To peace with it, there’s there are things that polls can’t do, we have to stop polling at some point because the point of a poll is for people to use the data. And for me, my clients, typically, they’re they’re going to write stories about it. So I think people get a misguided idea of what’s the good of a poll. If it’s not perfect, if it’s not proven to be perfect and coal is a is a device to help reporters, in my case tell stories of what’s happening. And if you didn’t have polls, how would an assignment editor know how to disperse the resources? If you don’t know who’s leading, if you don’t know if a third party candidate is getting traction and going to take away some of the vote, how do you know how to report on the race? So I don’t I don’t aim for perfection. It’s nice when I get close to it. But there are other things that a poll can help you do.

Advertisement
Advertisement
Advertisement
Advertisement

S1: Hmm. It’s interesting that that story is about a sort of last minute change to campaign style and like the sort of last minute surge of energy, because when you look at your polls this year, it looks like in Iowa there was this kind of last minute Republican surge. And I’m wondering if you can talk about how you found that in the data and what you thought of it when you saw it.

S3: This is partly it’s sort of it’s a data driven speculation, let’s put it that way. So our September poll had had Teresa Greenfield leading Joni Ernst by a few points.

S1: That’s the Senate race. Teresa Greenfield was the opponent for Joni Ernst.

Advertisement

S3: That was the Senate race. Yes. And then for the presidency, a dead heat. And what transpired after that? This is before early voting started. But if you remember, as state by state started their early voting, those initial days, you would see long lines of people standing in line for hours so that they could cast their vote early and then that kind of dwindled down. So I wondered if the Democratic playbook was to really put the effort into getting that early vote out in strength and that the peak of their arc of their final push was a little early and that the Republicans who knew that their supporters were more likely to vote on Election Day, their arc peaked a little later. And this is informed a bit by having listened to a bit of Rush Limbaugh on Monday, last week, the day before the election, he was talking about my poll.

S4: And it is a gospel. It has become a gospel poll. And the woman that runs the poll is considered the epitome of fairness and so forth, because she really does care about the sanctity of her or her newspaper’s reputation or pole reputation. All that and the news in the Iowa poll on Sunday is devastating for the Democrats, devastating for Biden.

Advertisement
Advertisement
Advertisement
Advertisement

S3: But he was also saying that if you were an alien and you came from another planet and you couldn’t understand anything, people were saying you just had the optics. Who would you think is winning? That there was more energy, there was more there was just more electricity happening on the Republican side and more sedate activity on the Democratic side, and that kind of got me thinking about, well, where was the peak of the Democratic arc? And it just because it was early and the feeling that you have your support banked in early voting, there just didn’t seem to be much of a final push except on the Republican side. The president came here a couple of times, the vice president came here. So I think that’s just interesting and it’s perhaps is unique to this year because of the pandemic and the decisions that people are making about being out in public at all.

S1: I wonder how much you think the criticism of polling has been strengthened by the fact that we had this long, drawn out process of getting to the place we are now where we have a assuming president elect with Biden and that it just took so much time. There was all this space for us to think about polling and pick it apart.

S3: I think when there are misses and polling that are widespread, it’s it’s just a matter of time that people are going to start complaining about it. I don’t fault the media for picking at it, but I do want to say that the idea that the polling industry needs a reckoning, this is a sentence that doesn’t make that much sense to me. It’s not like the polling community acts as one and that we all do things the same way and that we’re just a commodity kind of of product that you can switch in and out and and get the same result. It’s it’s there’s a lot of diversity, a lot of different approaches. And it’s a commercial enterprise.

Advertisement
Advertisement
Advertisement

S1: So where there is demand, there will be supply and is, of course, in this industry. So it makes sense that she doesn’t want to blow it up. It behooves her to think that the demand for polling will always be there is what pays her bills. But she does think there’s something that can make polls like hers seem like outliers.

S3: I will tell you that if you’re thinking about an aggregate that is an average of polling and whether that’s a weighted average or clean average, it gives the illusion that that’s going to be more accurate.

S1: Hmm. You’re talking about like forecasting like what 538 does, where they sort of combine everything and come out with something like a cumulative score. Right.

S3: And what 538 does that others might not do is that they factor in the pollsters history of being accurate. So it’s not just a straight average, it’s a handicapped average. And again, I hate to bring this back to me, but if you average my poll in the average for Iowa would look like it was going to be a close race, even with my one outlier poll, because it’s going to be, you know, that influence is going to be depressed. I suspect there would be some polls in Ohio that got it right and some polls in Florida that got it right. But when all of the polls are looked at together, that can mask something that that’s actually happening out there. So until that work has been done to clarify in Florida, to clarify in Ohio, I’ll sit on my hands here.

S1: It’s interesting. I wonder if you think that’s kind of a problem where each polling group is doing their own thing. But because there’s not consistency, you might be comparing apples to oranges in terms of what they’re doing and what they’re showing.

Advertisement
Advertisement
Advertisement

S3: I don’t think it’s a problem. I think it’s the reality. There’s a sort of part of the way people are talking about polling as though we’re a public utility. And so, you know, everyone should be doing everything the exact same way. And I don’t envision that that’s ever going to happen. It doesn’t happen in political consulting. It doesn’t happen in the way people choose to advertise. It’s everybody has their own way.

S1: But I mean, you clearly you have a method where you don’t do this, adjusting for what you think is going to happen after the fact. When you do your polling, do you think it’s a mistake that more people don’t follow that straight ahead method?

S3: Is it a mistake? A lot of times they have come up with a data driven approach that has worked for them in the past, and they can show you the regression equations which lead them to think this is what’s going to happen and how they should define a likely voter. So I’m. I’m sure that they would push back and say, hey, it was 20, 20, what made you think anything was going to work this year? I do say we publish our methodology. It’s available to anybody who thinks, well, you know, maybe we should try something different.

S1: Well, you do train your phone bankers in like manners training. And I was thinking about that when I heard some folks say when these polling misses started happening, oh, well, we’re missing people who would hang up the phone if we said, oh, this is The New York Times on the phone. And it may be it just made me think about your method and whether there are things to be gleaned from it, both in the fact that you’re a local pollster, so a known entity with voters may be trusted. And then also whether there’s something to even the way the people you work with talk to the people you call.

Advertisement
Advertisement
Advertisement

S2: Well, to be clear, I work with a phone bank that is a separate entity. They’re not my employees, but they are very good at what they do. I could not be successful without them being very, very good. But yes, I think that politeness forms can matter. This goes back to when I was on the staff of the Des Moines Register trying to teach people how to say, sir and ma’am, in a way that didn’t sound like it was the first time they never uttered those words. I think it’s really important and this is what I’m reflecting on. Increasingly, polling as an industry pretty much relies on the kindness of strangers to check their phone and see that it’s ringing to decide to answer and then to decide to continue the conversation for no compensation. And in twenty, twenty and really beyond, I just wonder how that business model can stand up. There are pollsters who recruit panels of people who say, sure, hold me, pull me for me. And I’ve never found that model to sit well with me. But I understand why they do it, why, you know, there’s a thing in social science called the instrument effect that says the mere act of measuring something can change its. So if you take your tire pressure, you’re releasing air from your tire in order to measure it. You changed the air pressure inside your tire in order to measure it. And the idea of someone joining a panel agreeing to be pulled repeatedly, do they now behave differently? Do they do it? Do they now engage in different kinds of political activity, expose themselves to different kinds of political messages? Does it shape them as they’re thinking? Oh, well, you know, the next time I’m told that, I worry about that.

Advertisement
Advertisement
Advertisement

S1: Yeah. I mean, in one of these critical essays, the author pointed out that if you’re exposed to a forecasting prediction, this is sort of a similar idea. It can change the way you vote, decrease turnout, confuse people. Do you worry about that in the work that you do, that giving people some kind of prediction can become part of the story?

S2: I’ve never seen data that would say what the impact of that might be. That is, if the person you’re inclined to support is behind, are you more inclined to to show up or are you more inclined to not? And and if you’re supporting the the leading candidate, again, what what impact does that have or would be expected to have consistently from voter to voter to voter and at a higher level, the idea that people think polls are evil because they might influence the outcome of an election, I say, well, is political advertising in our campaigns, you’ve got a lot of people spending a lot of money trying to influence the outcome of the election. So polls are a piece of it, but I don’t think they need to be taken out and shot more.

S1: What next? After the break. There’s been a lot of public hand-wringing with Democrats about this election, even though there’s this presidential win, because they wonder what it tells them about their agenda moving forward. And I wonder what your polling in Iowa would tell them, especially now that the Democrats are looking to go to Georgia and make a big push with two special elections to try to have some kind of foothold in the Senate. I mean, just because you talk to so many voters, I wonder what you thought about this debate, about, you know, whether the party’s gone too far left, whether there’s something else afoot.

Advertisement
Advertisement
Advertisement

S3: There will be some hashing of all of that out to come. And I look forward to the way the contest shapes up in Georgia and how that might differ from the way it was run before we knew the outcome here. I think our data said one thing rather clearly, which was the Republican side had a singular theme that was the most important thing for their voters, and that was the economy. They said it over and over and over and over and over. And then when you asked Democrats to say, what’s the single most important thing? Well, a segment of them said, well, it’s competent leadership. It’s really different from that. It’s really being able to restore what’s good about America. Oh, it might be that it’s covered and we’ve got to do this. But there was no one unifying theme. And when I listened to interviews of Trump supporters, you heard them say two things repeatedly. He’s built the best economy ever and we’re having a bump in the road, but he’s the one to build it back. And you heard them say that socialism would be terrifying for this country. They can’t they can’t imagine socialism. And then they might throw and defund the police and say this is just a future that is fraught with one bad thing after another, after another. And we just need to stay with President Trump. And when you talk to Democrats, it I don’t think I could begin to sum up what they were saying beyond the single unifying thing was, look at that. We don’t want any more of that. Trump without without really being making a more logical recent case. It was a more emotional we have we have to save ourselves from that.

Advertisement
Advertisement
Advertisement

S1: Yeah. I mean, to me, it’s going to be really interesting to see what happens in Georgia, because I agree with you that there has been almost an intentional lack of of laying out an agenda by the Democrats. It seemed to me like almost quietness of just like keep all the attention on Trump and you’re voting against Trump. And now that Biden is the president elect, I think you need to have an agenda that you’re selling people, because otherwise you don’t you don’t have Trump to sort of run against anymore.

S3: Yes. So so the person they have to run against is Mitch McConnell. So one of the things that we saw in our final Iowa poll data was that the most common reason to support Teresa Greenfield, there was more to two for Democrats to take control of the Senate than their enthusiasm for the candidate. The same is true for those who are supporting Joni Ernst. More of them said for Republicans to keep control of the Senate, then said that it was enthusiasm for the candidate, but it wasn’t as big. And so if that is kind of how Senate races are getting decided, I think that will be interesting in Georgia, because you’ve got to.

S1: Right? Yeah, it’s just I’m just fascinated. Ann Selzer, thank you so much for joining me. My pleasure. Ms. Selzer is the president of Selzer and Co., an Iowa based polling organization. And that’s the show What Next is produced by Mary Wilson, Daniel Hewitt, Alan Schwarz and Frannie Kelley, Jason de Leon contributed one more time to this show. Thanks, Jason. Our overlords in a good way are Allison Benedict and Alicia Montgomery. And I’m Mary Harris. You can find me on Twitter at Mary’s Desk. Lizzie O’Leary will take the reins tomorrow for our Friday show. What next? TBD. And I will catch you back here on Monday.