Skip to content

About that bogus claim that North Carolina is no longer a democracy . . .

Nick Stevenson directed me to a recent op-ed in the Raleigh News & Observer, where political science professor Andrew Reynolds wrote:

In 2005, in the midst of a career of traveling around the world to help set up elections in some of the most challenging places on earth . . . my Danish colleague, Jorgen Elklit, and I designed the first comprehensive method for evaluating the quality of elections around the world. . . . In 2012 Elklit and I worked with Pippa Norris of Harvard University, who used the system as the cornerstone of the Electoral Integrity Project. Since then the EIP has measured 213 elections in 153 countries and is widely agreed to be the most accurate method for evaluating how free and fair and democratic elections are across time and place. . . .

So far so good. But then comes the punchline:

In the just released EIP report, North Carolina’s overall electoral integrity score of 58/100 for the 2016 election places us alongside authoritarian states and pseudo-democracies like Cuba, Indonesia and Sierra Leone. If it were a nation state, North Carolina would rank right in the middle of the global league table – a deeply flawed, partly free democracy that is only slightly ahead of the failed democracies that constitute much of the developing world.

I searched on the web and could not find a copy of the just released EIP report but I did come across this page which lists all 50 states plus DC.

North Carolina is not even the lowest-ranked state! Alabama, Michigan, Ohio, Georgia, Rhode Island, Pennsylvania, South Carolina, Mississippi, Oklahoma, Tennessee, Wisconsin, and Arizona are lower.

Hmmm. Whassup with that?

Here’s the international map from The Year in Elections, 2014, by Pippa Norris, Ferran Martinez i Coma, and Max Grömping:

There’s North Korea in yellow, one of the countries with “moderate” electoral integrity. Indeed, go to the chart and they list North Korea as #65 out of 127 countries. The poor saps in Bulgaria and Romania are ranked #90 and 92, respectively. Clearly what they need is a dose of Kim Jong-il.

Let’s see what this measure actually is. From the report:

The survey asks experts to evaluate elections using 49 indicators, grouped into eleven categories reflecting the whole electoral cycle. Using a comprehensive instrument, listed at the end of the report, experts assess whether each national parliamentary and presidential contest meets international standards during the pre-election period, the campaign, polling day and its aftermath. The overall PEI index is constructed by summing the 49 separate indicators for each election and for each country. . . .

Around forty domestic and international experts were consulted about each election, with requests to participate sent to a total of 4,970 experts, producing an overall mean response rate of 29%. The rolling survey results presented in this report are drawn from the views of 1,429 election experts.

OK, let’s check what the experts said about North Korea; it’s on page 9 of the report:
Electoral laws 53
Electoral procedures 73
District boundaries 73
Voter registration 83
Party and candidate registration 54
Media coverage 78
Campaign finance 84
Voting process 53
Vote count 74
Results 80
Electoral authorities 60

Each of these is on a 0-100 scale with 100 being good. So, you got it, North Korea is above 50 in every category on the scale.

Who did they get to fill out this survey? Walter Duranty?

OK, let’s look more carefully. In this table, the response rate for North Korea is given as 6%. And the report said they consulted about 40 “domestic and international experts” for each election. Hmmm . . . 6% of 40 is 2.4, so maybe they got 3 respondents for North Korea, 2 of whom were Stalinists.

That 2014 report mentioned above gave North Korea a rating of 65.3 out of 100 and Cuba a rating of 65.6. Both these numbers are higher than at least 27 of the 50 U.S. states in 2016, according to the savants at the Electoral Integrity Project.

Political science, indeed.

How’s North Korea been doing lately? Stevenson writes:

North Korea is in The Year in Elections 2014 but was quietly removed from The Year in Elections 2015. It’s not a matter of the 2014 elections not being in the 2015 timeframe either – diagram 5 of The Year in Elections 2015 says ‘PEI Index 2012-2015’ and North Korea was in Diagram 1 of The Year in Elections 2014, PEI Index 2012-2014. They have North Korea in gray in the later world map as ‘Not yet covered’. On p. 73 of The Year in Elections 2015 they list their criteria for inclusion in the survey (no microstates, no Taiwan, etc) but don’t explain why PRK_09032014_L1 has suddenly gone missing.

Perhaps North Korea was too embarrassing for them?

In his email to me, Stevenson wrote:

This is terrible research that I [Stevenson] think has the potential to do real damage in the real world with their absurdly high scores for fake elections in places like Oman, Kuwait, Rwanda, and Cuba. Suppose Oman’s government arrests an opposition politician or cracks down on a peaceful demonstration and the EU and US ambassadors protest. What if the Omani government argues that according to Harvard University’s measure which is “widely agreed to be the most accurate method for evaluating how free and fair and democratic elections are across time and place”, Oman is in much better shape than many EU countries and US states and that they should get their own houses in order before criticizing others? The EIP is just as likely to serve as a freebie to repressive governments that somehow fluke a high score as it is to spur the repeal of Wisconsin’s ID law.

If Reynolds, Norris, etc., don’t like what the North Carolina legislature has been doing, fine. It could even be unconstitutional, I have no sense of such things. And I agree with the general point that there are degrees of electoral integrity or democracy or whatever. Vote suppression is not the same thing as an one-party state and any number-juggling that suggests that is just silly, but, sure, put together enough restrictions and gerrymandering and ex post facto laws and so on, and that can add up.

Electoral integrity is an important issue, and it’s worth studying. In a sensible way.

What went wrong here? It all seems like an unstable combination of political ideology, academic self-promotion, credulous journalism, and plain old incompetence. Kinda like this similar thing from a few years ago with the so-called Human Development Index.

P.S. I googled *reynolds north carolina democracy* to see how much exposure this story got, and I found links to Democracy Now, Vox, Slate, Daily Caller, Common Dreams, American Thinker,, MSNBC, Huffington Post, Think Progress, The Week, . . . basically a lot of obscure outlets. I write for Slate and Vox, so I was sorry to see them pick this one up.

But the good news is that the usual suspects such as ABC, NBC, CBS, CNN, NPR, BBC, NYT didn’t fall for it. I give these core media outlets such a hard time when they screw up, and they deserve our respect when they don’t take the bait on this sort of juicy, but bogus, story.

P.P.S. See here for more from Pippa Norris.


  1. Jonathan says:

    I received this about 15 times from friends and relatives and took to posting a simple statement about how ridiculous it was, but then it feeds into the general paranoia of today. But to add something (possibly) humorous: at Yale in days of yore, I took a Soviet history seminar in which one of the students was a Communist from India and then the Indian Communist Party was unreconstructed Stalinist (at least partly). Almost the entire content of the course thus consisted of two versions of reality. It was startling to hear him talk about how Stalin only killed a few Kulaks and how what we call show trials were real. As an extra level, the professor was a real life acquaintance – as I remember, he was related to a roommate – and he confided to me that he had no idea what to do: the kid’s work product, etc. all was exact Stalinist party doctrine so should he grade him based on the quality of that or versus actual reality? The kid, btw, was otherwise nice and extremely patient and determined that you were wrong on such a deep level that you were not even an annoyance to him. That is something people don’t recognize: they assume fanatics are anxious too but they aren’t and that implacability makes them dangerous. I have no idea what the kid got as a grade.

    As to N. Korea, when I was but a wee lad, we learned that the Soviet Constitution had far more democracy in it than ours. Pretty much true except of course the application lacked. I sometimes think of Kurt Goedel in this regard, notably his belief that our Constitution has a fatal flaw which if exploited could turn the US into a dictatorship – see here: – and that this wasn’t as crazy as one might think given the Soviet Constitution.

  2. Dane says:

    I have trouble picking up on Internet sarcasm sometimes, so forgive me if I missed it in your P.S., but the Daily Caller, HuffPo, Think Progress, and MSNBC are not obscure outlets. I would argue that the Daily Caller and Huffington Post are some of the most dangerous places for something like this to land, given their reputation as partisan echo chambers. I don’t know anyone under 30 who would regularly read a network news site.

  3. Dan Riley says:

    Vox picked up on some of the same issues, noticing that NC wasn’t the worst US state, and that some of the rankings seem spurious: “For instance, the US’s overall rating (62) is below that of Rwanda, a full-on autocracy under strongman Paul Kagame; it seems foolish to infer from that that the US is less of a democracy than Rwanda.” So I actually found their article useful for calibrating some of the others.

  4. I saw this story floating around on Facebook after the NC election results. While I find some of the political actions of the current NC legislature ridiculous, it’s equally ridiculous to say that the state is no longer a democracy. Or even worse, that it scores lower than autocratic regimes across the globe.

    What’s surprising is that they don’t use this as an opportunity to talk about some obvious peculiarities with their measures. It’s really unfortunate because I think the project is important — I mean, who wouldn’t be interested in protecting the integrity of elections (well, besides those who are rigging/benefitting from unfair institutions).

    • Andrew says:


      Yeah, it’s almost as if they’re rather get publicity than understand what they’re doing.

      • Rahul says:

        Are they being intentionally ridiculous or only accidentally so?

        • Andrew says:


          I think these researchers think of themselves as the good guys, and I’d guess they’d dismiss any criticism as being either nitpicking or politically motivated. Nick Stevenson did contact Pippa Norris awhile ago on the North Korea thing and she didn’t seem to get the point. This happens when people do quantitative research: they just take the numbers and run with them.

          One thing I sometimes say is that a researcher should have the capacity of being upset. You should want to stress-test your model and methods and see where they break down. Unfortunately not everyone thinks this way.

          • Nick says:

            Yes, she basically said lots of items on the questionnaire, lots of ‘experts’ polled -> meaningful output.

          • Pippa Norris says:

            No we take criticism very seriously and I answered a stream of 5-6 emails on New Year’s Eve from Nick Stevenson by providing a lot of information ! He then got annoyed because the university office was shut and the research team not at their desks over the break.

            Note also that in all our research we use mixed methods to triangulate combining cases and mass surveys and many other techniques, and PEI is only a fraction of the EIP project. Please read the books and articles we produce and not a second-hand newspaper report about a blog that was not even written by the team.

            • Nick says:

              Pippa, as I said in our emails I appreciated you answering these questions from a random person off the internet over the holiday period. I asked you why 2012 US Presidential election was 70.2 and the 2014 midterms were 69.3 according to ‘The Year in Elections 2014’ when you told me “the PEI Index was 63 in 2012 and 62 in 2014” for the USA. I asked if you had recalibrated the scores and you said no recalibration had been made and referred me to Max Groemping, but when you cc’ed him into an email I got an auto-reply saying he wouldn’t be back until March, so I asked you if Ferran or Alessandro might know.

      • Pippa Norris says:

        Again this refers to the reaction to the News and Observer blog, not the EIP project. Read our research, please.

        • Rahul says:

          Having lived in both India and the US for a substantial time, any index that puts the “electoral integrity” of both nations in the same bracket seems highly suspect to me!

          What’s your working definition of “integrity”?

        • Rahul says:

          And this is what I call a functionally useless definition:

          “The idea of electoral integrity is defined by the project to refer to agreed international conventions and global norms, applying universally to all countries worldwide through the election cycle, including during the pre-election period, the campaign, on polling day, and its aftermath.”

          Do you have anything better for us to chew on?

  5. Bob Rudis says:

    “Searched”. It took me abt 2m to find it a cpl weeks ago. Full data is available here: & I did a quick comparison between the US scores and global scores here:

  6. rtd says:

    “I write for Slate and Vox, so I was sorry to see them pick this one up.” Seriously? There’s zero chance that Slate or Vox (ESPECIALLY Vox) picking up on this could objectively surprise anyone who follows the news.

  7. Realist Writer says:

    “OK, let’s look more carefully. In this table, the response rate for North Korea is given as 6%. And the report said they consulted about 40 “domestic and international experts” for each election. Hmmm . . . 6% of 40 is 2.4, so maybe they got 3 respondents for North Korea, 2 of whom were Stalinists.”

    But North Korea’s elections could be arguably be seen as free and fair. Only one candidate is allowed on the ballot, having been handpicked by the government beforehand. If you happen to hate the candidate, you can cross the name of the candidate publicly, in front of election officials or in a separate box (and thus face retaliation *after* the elections). There’s no reason to suppose that the election would be rigged, because how can you even rig an election where there’s only one possible outcome?

    That’s not quite true. I suppose one could argue that North Korea could undercount or suppress signs of dissent such as defaced ballots or ballots with the name of the government-preferred candidate crossed out. And the government could use media coverage to boost turnout in these elections. My point is not to glorify the North Korean elections but point out that you can divorce elections from “democracy”. Elections may not have to serve “democratic” norms, but instead serve other purposes. In North Korea, the elections are little more than a propaganda stunt and a way to keep tabs on the general population by checking who’s voting and who’s not. “Election integrity” is missing the point of holding elections in North Korea (or indeed, in other autocracies with really high vote percentages for the incumbents).

    • Pippa Norris says:

      Let’s not cherry pick by focusing upon this single case, which was dropped from the study in 2015! We did have some problems with the experts there, which is why the case was dropped, but this is highly atypical. PEI-4.5 covers 153 countries.

      • Randy says:

        Well, hold on there partner…if you had problems with the experts there…who’s to say you don’t have similar problems elsewhere? You say “cherry-picking” but the reality is it’s such a farce to rank North Korea so highly that even a novice like me, not a so-called “expert” can readily see the problem with the outcome of your methodology. What I want to know is if whatever…errr…analysis tools you’re bringing to bear on this problem are all that accurate at all?
        If your process can’t even properly place countries like Cuba, Rowanda, North Korea…well no layman such as myself will ever take it seriously.

        I saw one of your representatives on Fox news being interviewed tonight. I was out with friends when we saw this on a TV in a bar. We are all in town for conference. I have a BS and MS in EE from Texas A&M and I was the least educated engineer at the table. We laughed long and hard as this guy from NC Chapel Hill was dodging, weaving, and back-stepping from the report. I just wish to God the interviewer from Fox had brought up North Korea or Cuba, all he seemed interested in asking about is NC not being a democracy any longer.

        • Patrick Henry, the 2nd says:

          This is a great point. If a non-expert can easily poke holes in an analysis, is it actually correct at all? I mean there is no reason that Cuba or North Korea is better than NC for electoral reasons.

  8. Tracy Lightcap says:

    I used to work for a government agency that shall remain nameless that used expert opinion like this to determine if new judges should be requested for judicial districts. But we did it differently.

    We got the judges (who else?) to rank the amount of time they spent on different kinds of actions in different types of cases. We then weighted the actions and cases and got an estimate of how much time was necessary. We took that estimate and sent it back to the judges so they could adjust their estimates. Those adjusted estimates were then used to determine how much excess work there was in a particular district and, hence, which one would get a request.

    This called Delphi estimate and actually works pretty well. (We did some actual surveys to see.) And that’s exactly the problem here. You have a load of different indicators that are susceptible to varying evaluations. Well and good. Problem = there isn’t any feedback allowed for the expert panel o correct their point estimates in light of the aggregate score. So some estimates end up looking a little squirrelly.

    There’s nothing wrong with the initial evaluations. But if experts can change their estimates if they find their aggregate scores unreasonable then things can get more reasonable. Of course, the easiest solution would be to use harder data. Good luck with that if you are rating North Korea.

    • Pippa Norris says:

      Yes this is another way of doing it, for example Global Integrity goes back and forth with their ‘experts’ and Freedom House discuss their ratings as a group. But this can also cause other problems, like group-think where the conventional wisdom outweighs contrary judgments (“of course Trump cannot win,?” etc etc). If everyone believes the world is flat, then it brings us no nearer the truth to ask Copernicus to get in line with what everyone knows. May be better to publish the St. dev and look at the differences or convergence across independent experts.

      • Tracy Lightcap says:

        We tried this – using feedback from the entire group of judges and getting them to discuss the ratings among themselves – and it led to the very problems you point out. However, getting the judges to revise their own ratings on a district by district basis worked pretty well. They got to revise their estimates in private before any were presented and, initially to our surprise, they became quite serious about it. As near as we could figure, what was happening is that each group of judges was afraid of not being taken seriously in their requests unless they were reporting as near to what they actually thought the situation was as they could. Early on, we had a couple of districts that were obviously trying to game the system and their estimates were greeted with open derision by the judges when they were presented for a vote of approval. The example – game theory strikes again – stuck and the whole process became something that our judiciary was willing to stand behind when they took the plans to the legislature.

        Ok, your project isn’t like ours, but the mechanism might work better with revisions along these lines.

      • Andrew says:


        As I wrote in a different comment somewhere, the standard deviation is only part of the story. It ignores bias, which for your survey seems to be huge. There’s bias in the ratings and bias with respondents not being representative of some population of raters.

        • Nick says:

          ‘There’s bias in the ratings and bias with respondents not being representative of some population of raters.’ I agree that there are some sources of bias in the ratings – e.g. some questions in the survey have the systematic effect of biasing the ratings (in tightly managed authoritarian pseudo-elections) away from what most academics mean by actual electoral integrity (i.e. the ability of the people to make a free and fair democratic choice). But do we really know that there’s bias from ‘respondents not being representative of some population of raters’? We don’t have evidence that there was bias caused by proponents of any particular point of view being systematically more likely to answer the survey in the case of any particular country. If we did have evidence that the responders were unrepresentative of the wider pool to whom the survey was sent, it’s more parsimonious to put it down to variance.

          Any given surprising rating could be caused by the naively additive design of the survey rating, the subjective nature of some of questions, the fact that respondents have no way of indicating how severe a particular problem is, etc.

  9. Tom Maguire says:

    Re; “P.S. I googled *reynolds north carolina democracy* to see how much exposure this story got, and I found links to Democracy Now, Vox, Slate, Daily Caller, Common Dreams, American Thinker,, MSNBC, Huffington Post, Think Progress, The Week, . . . basically a lot of obscure outlets. I write for Slate and Vox, so I was sorry to see them pick this one up.”

    FWIW The American Thinker is a right-wing site for whom an article like this is catnip only as an opportunity to denounce cloistered left-leaning academics, which Rick Moran did with gusto.

    Vox noted a few issues; the Huffington Post and Think Progress were as credulous as one might expect.

  10. Nathan says:

    There are definitely some good points here, but overall I’m not sure I totally buy the argument here.

    First, it took me less than 30 seconds to find the latest report and its corresponding data [1]. There is not necessarily anything wrong with using the 2014 data to critique Reynold’s piece (things shouldn’t have changed that much), but not being able to find it isn’t really a valid excuse.

    I do have issues with how the 2014 data is used, however. The argument here focuses on the poor quality of the data for undemocratic states like North Korea and asserts that the measure in general for all states is deeply flawed and therefore invalid. I agree on the first part. The data for these undemocratic regimes is very poor and I don’t really buy any conclusions drawn from them. Stevenson is certainly right that it allows states like Rwanda and Cuba to skate by with a much higher score than they deserve.

    Experts who study US elections, however, have a lot more access to information, so there’s reason to assume that the data about US elections is going to be of higher quality. While I’d say the confidence intervals around the results for undemocratic regimes like Cuba and Oman should be very large, those for the US states should be much smaller. So making any meaningful comparison between the US or North Korea on this measure is problematic, but that does not make the measure necessarily invalid (for the reasons listed in the blog post) for the US states. And by no means am I arguing that the measure is perfect. Empirical work comes with a whole host of issues, but I’d say a survey of experts about US state elections might still have some useful information to tell us.

    Also, the fact that North Carolina (or Alabama or Mississippi) rank so low on this measure shouldn’t be a surprise: the US South has a long history of engaging in undemocratic behavior. This is where Reynold’s framing of the situation is problematic. The recent behavior of NC legislators didn’t flip and change NC’s status — the state was already a deeply flawed, semi-free democracy.

    1.a Report:
    1.b US data on dataverse:
    1.c World data on dataverse:

    • Andrew says:


      1. The link you give as “the latest report” does not have data on U.S. states so it cannot be “the just released EIP report” mentioned by Reynolds. As I wrote in my post, I did find a link to the U.S. numbers but I did not find “the just released EIP report” mentioned by Reynolds. Since I was writing about Reynolds’s op-ed, I wanted to take a look at the report he was pointing to, but I couldn’t find it.

      2. I haven’t looked that the U.S. numbers in detail and perhaps the relative rankings of the different states is reasonable, as the states do differ in levels of voter suppression, gerrymandering, etc. But the headline was not “North Carolina is 13th-worst U.S. state in electoral integrity,” it was “North Carolina is no longer classified as a democracy.” So, yes, it seems relevant to point out that this ridiculous measure had over half the U.S. states in 2016 as being less democratic than Cuba and North Korea in 2014.

      • Nathan says:

        1. You’re right. I was wrong here. I was conflating the 2014 world report you were discussing and the 2016 results you were also discussing.

        2. I’m wrong here too. Ok so I think you’ve totally brought me over to your side. I think I was hung up on trying to ‘save’ the measure as it applied to US states. In the process, however, I missed the overall point you were making and I agree. The framing of “North Carolina is no longer a democracy” is incredibly misleading and in fact -bogus- as you say. At the same time, I don’t think this lets NC (or other states with restrictive policies) off the hook.

        • Andrew says:


          I agree about not letting N.C. or other U.S. states off the hook. As Thomas Jefferson never said, eternal vigilance is the price of liberty. Just as we still have plane crashes but we try hard to get that error rate down to zero, I think we should be similarly serious that votes are counted correctly, that everyone who is eligible to vote and wants to vote is able to do so, and that all election laws are enforced.

  11. Max Effort says:

    Any response to this part of the op-ed?

    “Third, government in North Carolina has become arbitrary and detached from popular will. When, in response to losing the governorship, one party uses its legislative dominance to take away significant executive power, it is a direct attack upon the separation of powers that defines American democracy. When a wounded legislative leadership, and a lame-duck executive, force through draconian changes with no time for robust review and debate it leaves Carolina no better than the authoritarian regimes we look down upon.”

    • Andrew says:


      I’d recommending studying this by looking at the history and seeing how often outgoing governments do this sort of rule-changing behavior. I think this is worth studying.

    • Patrick Henry, the 2nd says:

      The problem with that is the legislature is the representative of “popular will” just as much as the governor- and given the election for governor was so close, the legislature may be a better indication. Dems gained one seat in the House from an unaffiliated seat, and Repubs gained one seat in the Senate from a Dem seat.

      So the people elected a legislature that didn’t change much. And if anything is a definition of American democracy, its reducing the power of the executive. And it had nothing to do with attacking separation of powers.

      Nothing NC did was anything close to what an authoritarian regime does- because the power is still within the legislature.

      • tom c says:

        “The problem with that is the legislature is the representative of “popular will” just as much as the governor-“

        Sadly, no. In fact, federal courts told NC’s legislature that their districts are so unfair to groups of NC’s citizens that they have to redraw them and hold a special election in 2015. Republicans did win an election in 2010. Their majority was reflective of the popular will that one year. Importantly they didn’t win veto proof majorities in that year. Those only materialized after the 2010-11 redistricting process. Also interesting to point out that the veto proof majorities emerged through elections where republicans drew smaller percentages of the vote than they did in 2010. If they’d run under the old districts in 2012, not only wouldn’t they have veto proof majorities in both houses, they likely would have lost a legislative majority altogether. (And one has to wonder how the hell they managed veto proof majorities when the Governor’s race, the AG race, and just about very other statewide race was so close in 2016.) So, no, the state legislature in NC isn’t reflective of the popular will just as much as the Governor is.

        The special section laws passed this past winter also address rules well beyond the chief executive’s powers. They also address ways in which their own power can be challenged. They’ve already hopelessly stacked the deck on elections through the redistricting process. Now they’ve changed the manner in which legal challenges can be brought against them. Cases which previously would have gone before the State Supreme Court now have been channeled toward other venues. Why? Because Democrats won a State Supreme Court election, giving them a majority of seats. So their powers need to be curtailed, too. Yay democracy!

        The changes the NC legislature made weren’t designed to change who the Governor can appoint to a single office. They address just about every kind of power the Governor still had.

        And just to point out, the whole special session was supposed to be in convened strictly for the purpose of providing assistance to flooding/hurricane victims. I guess though, that’s just what democracies do.

        • Andrew says:

          Patrick, Tom:

          The discussion in your two comments above is useful. I wish Pippa Norris and Andrew Reynolds had just addressed things at this direct level rather than trying to support their points with garbage data.

          • tom c says:

            Patrick and I aren’t having a discussion. We are making two mutually exclusive assertions about the NC state legislature. To use your terms, one of us is using “garbage data.”

        • tom c says:

          Typo above (sorry): 2015 should be 2017. We are (at least as of now) going back to the polls this coming year to redo the state legislature election we just had. (Though there’s every chance newly draw districts will have almost all the problems the old ones had.)

  12. Gerry says:

    NC ranks 58 but that paragon of liberal democracy NY ranks 61. So 3 points means the difference between a good democratic state and a bad one. Right!!!

  13. aj1575 says:

    Let’s face it, the USA is moving further and further away from being the cradle of democracy, the one that ever other nation should look up to. There is a trend going on in the USA, that takes away democratic rights from people, in order to preserve the power of a party. In a real democracy it is a contest of who has the better ideas and therefore gets more votes, and not about how to game the system in order to keep the power (and then label it as a democratic process).

  14. Richard Belzer says:

    The first thing I do with all survey research is look for the response rate. If this is not reported, I discard the study. If it is reported and is less than 80%, I discard the study unless there is a rigorous nonresponse-bias analysis persuasively showing that nonresponders are no different from responders.

    Only survey research that passes these tests is worth examining further, such as for representativeness of the sample, framing and bias in survey design, and a host of important but subtle issues.

    Few survey research projects pass by initial screen, but that is not a bug in my screen; it’s a feature. In the newfound realm of “fake news,” survey research is disproportionately res presented.

  15. Physical Scientist says:

    The need to subjectively omit data that does not agree with preconceived notions is a red flag in any scientific study. Reasons can always be found to omit data that could expose flawed methodology. We must remember that social science is not science and give the conclusions the weight they deserve.

  16. Wayne says:

    Here is an explanation of why they included North Korea and Cuba, etc…

    What is the role of legal restrictions on party competition?
    The survey seeks to be comprehensive by evaluating
    all national parliamentary and presidential elections worldwide. Hence it assesses states where all political
    parties are banned (such as Bahrain and Swaziland), countries where specific types of political parties are
    disqualified from standing for election (including the Muslim Brotherhood’s Freedom and Justice party in Egypt,
    ethnic parties in Africa, and neo-Nazi parties in Germany), countries where candidates are restricted from ballot
    access (such as Iran), as well as one-party states (like Cuba and North Korea) with outright legal bans on all
    opposition parties.

    Note: The seat share is calculated by the proportion of seats in the lower house of the national parliament held
    by the largest party following the election. Source: Electoral Integrity Project. 2014.

    Some researchers might automatically exclude one-party states like North Korea from the comparison, on the
    grounds that human rights in these countries are so deeply flawed as to make the elections just a façade
    disguising autocratic rule. We feel, however, that there are several reasons for documenting levels of integrity in
    all these diverse cases.
    One is that the degree of party competition varies substantially worldwide, as illustrated in Figure 3. Legal bans,
    while a major violation of human rights, are only one mechanism to restrict opposition. It is an empirical matter
    to measure the degree of party competition, such as by monitoring the seat or vote share won by the leading
    party in parliamentary contests, or the vote share of the winning presidential candidate. The PEI is designed to
    measure all the ways that party competition can be limited, for example through lack of a level playing field in
    access to party finance or state resources, partisan manipulation of district boundaries (gerrymandering),
    excessive legal requirements for ballot access, and high de jure or de facto vote-seat electoral thresholds. In
    several micro-states, small legislatures with majoritarian electoral systems also allow a clean sweep in a landslide
    victory for one party.
    In addition, it is also important to monitor the contemporary quality of all elections worldwide to create
    benchmarks for future change, should states loosen legal restrictions on party and candidate competition in
    subsequent elections.
    Finally, several aspects of electoral governance may still function relatively cleanly and efficiently even in states
    with restricted party competition and human rights. Indeed the quality of electoral governance may be higher in
    these cases than in several fledgling democracies with weak state capacity and insufficient resources to stampout
    malpractices and irregularities such as vote-buying, ballot-stuffing, or security threats. In Cuba, for example,
    during the nomination process some genuine competition is reported among rival candidates.

    Page 15:

  17. Jon says:

    People who make claims like “North Carolina is No Longer A Democracy” and cite surveys that show the US like a “stan” or “North Korea” harm their own credibility. Certainly North Carolina is far from perfect, as is any place, but statements like that get the writer only a notch above people who say “Obama is a Muslim” or “Cuba is a great success because of its low energy usage per capita”.

  18. kimberly conroy says:

    I would not go so far to say it is bogus. To be fair, this was an Op-Ed piece and he was writing about HIS home state. Sure, he could have said that it was not ranked the lowest but he was only talking about his state in a local paper in his own state.

  19. Luís Vítor Antunes Coutinho says:

    In social science research there is (or was at least in my time) a lot of discussion about “face validity”. If something in your results is odd “on the face of it”, you better rethink your whole approach. Unfortunately, too many researchers get carried away by their number crunching and lose sight of the essential.
    It is unclear to me (I admit I haven’t looked at the original EIP) how they came to their index based on multiple measures, but there is a simple way of avoiding the pitfalls. How about multiple cut-off points? If you don’t pass on one or more essential measures, you simply score zero. I know, you might lose information but it’s better than losing your face.
    Second ─ and I am riding here a hobby-horse of mine ─ what is all this talk about significance and confidence intervals about? Does it occur to the authors that they neither have a random sample of experts nor of countries ─ quite the opposite? What then is the worth of an exercise going through statistical inference when none is possible? Scientific adacadabra?

  20. Anonymous says:

    Analytics are fine, but living here in NC as a liberal Democrat it sure feels like my vote is meaningless. And I don’t mean that because my side didn’t win, but because even when we do, the other side controls the legislature. Yes, there have been elections where Democratic candidates have received more votes statewide while remaining the minority in the legislature. And when Republicans get the majority of votes, even then the veto-proof majority of seats won come nowhere near reflecting their slim margin in votes. This year the voters of the state ousted McCrory and put a Democrat in his place, only to see the legislature gut his powers. Does that seem like a democracy? I won’t even get into the less-than-democratic voter suppression drive here. Maybe the original article was flawed, but it’s articles like Gelman’s that give an opening for new organizations to print headlines saying all is well in North Carolina after all (I linked here from Slate, who did just that).

  21. Matt says:

    The map also rates Thailand as “moderate” over the period 2012-2014. Thailand had a coup d’etat in 2014.

  22. Jonathan says:

    “If Reynolds, Norris, etc., don’t like what the North Carolina legislature has been doing, fine. It could even be unconstitutional, I have no sense of such things.”

    …kind of sidestepped the main issue there.

    • Andrew says:


      Huh? Just to remind you, the survey in question rated many of the U.S. states in 2014 as lower than North Korea and Cuba in 2016. North Carolina was only the 13th-lowest-ranked state in the U.S. So in what sense is a particular action of the North Carolina legislature “the main issue” in evaluating those ridiculous claims? Suppose the N.C. legislature was acting unconstitutionally in its recent setting. Then do you say that Michigan, Ohio, Georgia, etc. are not democracies?

      I agree that vote suppression, gerrymandering, etc., are real problems. I just don’t see this measure as contributing in any useful way to the understanding, evaluation, or solution of these problems.

Leave a Reply