The Bad Research Behind the Bogus Claim That North Carolina Is No Longer a Democracy

No, North Korea isn’t more democratic than the Tar Heel State.

Gov. Pat McCrory notwithstanding, hope for a democratic North Carolina is not lost. Above, McCrory arrives at Trump Tower in New York City, Dec. 7, 2016.

Spencer Platt/Getty Images

In a recent op-ed in the Raleigh News & Observer that was picked up far and wide, including here on Slate, political science professor Andrew Reynolds wrote:

In 2005, in the midst of a career of traveling around the world to help set up elections in some of the most challenging places on earth … my Danish colleague, Jorgen Elklit, and I designed the first comprehensive method for evaluating the quality of elections around the world. … In 2012 Elklit and I worked with Pippa Norris of Harvard University, who used the system as the cornerstone of the Electoral Integrity Project. Since then the EIP has measured 213 elections in 153 countries and is widely agreed to be the most accurate method for evaluating how free and fair and democratic elections are across time and place.


So far, so good. But then comes the punchline:

In the just released EIP report, North Carolina’s overall electoral integrity score of 58/100 for the 2016 election places us alongside authoritarian states and pseudo-democracies like Cuba, Indonesia and Sierra Leone. If it were a nation state, North Carolina would rank right in the middle of the global league table—a deeply flawed, partly free democracy that is only slightly ahead of the failed democracies that constitute much of the developing world.

I searched on the web and could not find a copy of the just-released EIP report, but I did come across this page, which lists all 50 states plus Washington, D.C.

North Carolina is not even the lowest-ranked state! Alabama, Michigan, Ohio, Georgia, Rhode Island, Pennsylvania, South Carolina, Mississippi, Oklahoma, Tennessee, Wisconsin, and Arizona are lower.


Hmmm. What’s up with that?

On page 12 of “The Year in Elections, 2014,” by Pippa Norris, Ferran Martinez i Coma, and Max Gromping, you’ll find an international map, color-coded according to electoral integrity. There’s North Korea in yellow, one of the countries with “moderate” electoral integrity. Indeed, go to the chart and they list North Korea as No. 65 out of 127 countries. The poor saps in Bulgaria and Romania are ranked Nos. 90 and No. 92, respectively. Clearly what they need is a dose of Kim Jong-il.


Let’s see what this measure actually is. From the report:

The survey asks experts to evaluate elections using 49 indicators, grouped into eleven categories reflecting the whole electoral cycle. Using a comprehensive instrument, listed at the end of the report, experts assess whether each national parliamentary and presidential contest meets international standards during the pre-election period, the campaign, polling day and its aftermath. The overall PEI index is constructed by summing the 49 separate indicators for each election and for each country. …


Around forty domestic and international experts were consulted about each election, with requests to participate sent to a total of 4,970 experts, producing an overall mean response rate of 29%. The rolling survey results presented in this report are drawn from the views of 1,429 election experts.

OK, let’s see how the experts scored North Korea. It’s on Page 9 of the report. Each of these is on a 0–100 scale, with higher numbers indicating a higher level of electoral integrity.

Electoral laws: 53
Electoral procedures: 73
District boundaries: 73
Voter registration: 83
Party and candidate registration: 54
Media coverage: 78
Campaign finance: 84
Voting process: 53
Vote count: 74
Results: 80
Electoral authorities: 60


So, you got it, North Korea is above 50 in every category on the scale.

Who did they get to fill out this survey? Walter Duranty?

OK, let’s look more carefully. In this table, the response rate for North Korea is given as 6 percent. And the report said the authors consulted about 40 “domestic and international experts” for each election. Hmmm … 6 percent of 40 is 2.4, so maybe they got three respondents for North Korea, two of whom were Stalinists.

That 2014 report mentioned above gave North Korea a rating of 65.3 out of 100 and Cuba a rating of 65.6. Both these numbers are higher than at least 27 of the 50 U.S. states in 2016, according to the savants at the Electoral Integrity Project.


Political science, indeed.

How’s North Korea been doing lately? In an email to me, a reader named Nick Stevenson writes:

North Korea is in “The Year in Elections, 2014” but was quietly removed from “The Year in Elections, 2015.” It’s not a matter of the 2014 elections not being in the 2015 timeframe either—Diagram 5 of “The Year in Elections, 2015” says “PEI Index 2012–2015,” and North Korea was in Diagram 1 of “The Year in Elections 2014,” PEI Index 2012–2014. They have North Korea in gray in the later world map as “not yet covered.” On Page 73 of “The Year in Elections, 2015,” they list their criteria for inclusion in the survey (no microstates, no Taiwan, etc.) but don’t explain why PRK_09032014_L1 has suddenly gone missing.


Perhaps North Korea was too embarrassing for them?

Stevenson continues:

This is terrible research that I think has the potential to do real damage in the real world with their absurdly high scores for fake elections in places like Oman, Kuwait, Rwanda, and Cuba. Suppose Oman’s government arrests an opposition politician or cracks down on a peaceful demonstration and the EU and U.S. ambassadors protest. What if the Omani government argues that according to Harvard University’s measure, which is “widely agreed to be the most accurate method for evaluating how free and fair and democratic elections are across time and place,” Oman is in much better shape than many EU countries and U.S. states and that they should get their own houses in order before criticizing others? The EIP is just as likely to serve as a freebie to repressive governments that somehow fluke a high score as it is to spur the repeal of Wisconsin’s ID law.


What went wrong here? It all seems like an unstable combination of political ideology, academic self-promotion, credulous journalism, and plain old incompetence—like this similar thing from a few years ago with the so-called Human Development Index.


If Reynolds, et al, don’t like what the North Carolina legislature has been doing, fine. It could even be unconstitutional—I have no sense of such things. And I agree with the general point that there are degrees of electoral integrity or democracy or whatever. Vote suppression is not the same thing as a one-party state, and any number-juggling that suggests that is just silly, but, sure, put together enough restrictions and gerrymandering and ex post facto laws and so on, and that can add up.

Electoral integrity is an important issue, and it’s worth studying. In a sensible way.

Originally published in slightly different form on AndrewGelman.com. A response from one of the survey’s organizers, Pippa Norris, can be found here.