Methods 101: Why the NH Polls Were Wrong

Jon Krosnick, a professor of Communication at Stanford and perhaps the top expert in survey methodology, hypothesizes that the pre-primary polls in New Hampshire might have been wrong because they failed to take into account the NH ballot design and the miserly nature of voters. In public opinion research, the tendency for miserly voters to choose the candidate listed first or closer to the top of the list of candidates is called the primacy effect:

A further potential source of error stems from New Hampshire ballot rules. In previous contests, the state rotated candidate names from precinct to precinct, but this year the names were in alphabetical order, with Clinton near the top and Obama lower down. Stanford Professor Jon Krosnick, a survey specialist and expert witness in a lawsuit about ballot order in New Hampshire, has estimated a three percentage point or greater bounce for a big name candidate appearing high on the ballot. Therefore, if pre-election polls randomized candidate names, as most do, they would have underestimated Clinton's support by at least three points.

Regardless, there were no immediate clear answers, and lots of data analysis ahead.

More like this

Did Clinton win the New Hampshire Democratic primary Tuesday because her name came before Obama's on the ballot? SciBlogger Matt "Framing Science" Nisbet has a couple of posts referring us to someone who seems convinced she did. I'm not so sure, but find the mainstream media's reluctance to at…
At ABC News.com, survey expert and Stanford professor Jon Krosnick has more on the likely primacy ballot effect that I reported on this morning: Until this year, New Hampshire rotated candidate name order from precinct to precinct, which allowed us to do that analysis. This year, the secretary of…
Remember the Democratic New Hampshire primary? According to news organizations and many pollsters, the NH primary was supposed to be the loss that put Hillary Clinton out of contention and sealed an early nomination for Obama. Yet Clinton staged a surprise victory. At the time, Washington Post…
This post was written in two parts, pre-primary and post-primary. To see the result and a discussion of what they mean, skip down to the last part of the post, where I'll discuss why Tuesday's results may mean that Sanders could win the primary. Pre-Primary As already discussed, Clinton is likely…

Another possibility: a clever, if paranoid, reporter, has noted that New Hampshire has two methods of voting, paper ballots (about 20% of the state) and Diebold electronic machines (about 80 % of the state). The reporter says this is the voting breakdown:

Machine count: Clinton, 82860; Obama, 72807
Hand count: Clinton, 18898; Obama, 20912

The report is found at http://presscue.com/node/38034

Personally, I'd like to believe the numbers are wrong, or else that there is some plausible explanation for the apparently significant discrepancy between the results from the two methods of voting. Prof. Nisbet?

By Albion Tourgee (not verified) on 09 Jan 2008 #permalink

"...that there is some plausible explanation for the apparently significant discrepancy between the results from the two methods of voting."

You are assuming that the voting methods are distributed randomly, with no correlation to preferred candidates. That would be nice, but in real life it won't happen.

By Lassi Hippeläinen (not verified) on 09 Jan 2008 #permalink

I don't believe this in this case. Only the most hard-core political wonks go out for primaries. They're not going to drive all the way to their polling place, only to be too lazy to look at the entire ballot. I'd bet that for a primary, the vast majority of voters had their candidate in mind before they got there.

Thanks Lassi for the response, but aren't you assuming the conclusion? Why would there be a correlation of voting methods (which are determined town-by-town in New Hampshire) and voter preferences? Obviously voting methods aren't distributed randomly, but why would there be a correlation between that distribution and Clinton / Obama votes? It seems all the more puzzling to me, because Edwards' vote apparently was consistent across voting machine types, according to some information I've seen.

If non-random distribution of variables precluded making statistical comparisons, then what real-world comparisons could be made? But we frequently do statistical workups of non-random distributions. For example, voting patterns are clearly not randomly distributed in any real population. We even accept usage of polls where, for example, followers of one candidate may be more inclined not to answer than followers of another. Yet, meaning still can be derived from the information.

By Albion Tourgee (not verified) on 09 Jan 2008 #permalink

Your assumption is incorrect, Jeff. There were more than half a million votes cast (~280K for Democrats, ~230K for Republicans) in a state with a total population (including children and noncitizens) of not quite 1.3 million. That would be a turnout of between 50% and 60% of all eligible voters (not just those registered), a rate that we are lucky to see in this country for an off-year election.

The distribution of machines vs. hand-count is, as Lassi said, not random. Towns using hand counting are overwhelmingly likely to be small and rural; cities and larger towns use machines. Of the state's 13 cities, Hillary carried 9 (Berlin, Claremont, Dover, Franklin, Laconia, Manchester, Nashua, Rochester, and Somersworth) to Obama's 4 (Concord, Keene, Lebanon, and Portsmouth). The general pattern is that the smaller and wealthier towns and cities favored Obama, but Hillary got a solid vote from working-class cities and towns, including the less well-off of the suburban towns.

My source for town-by-town results is AP via the Concord Monitor:
http://hosted.ap.org/dynamic/files/elections/2008/by_county/NH_Page_010…

There might have been bad programming, or deliberate hanky-panky, with the electronic machines, but short of doing a hand recount there is no way to prove it. The results were consistent with exit polls.

By Eric Lund (not verified) on 09 Jan 2008 #permalink

Isn't the disparity more likely to be the " I'm not a racist so I'm not telling the pollster I'm not voting for a black man" theory much more likely?
I recall just reading an article about the 1982 California governors race, when Tom Bradley (black) had an 8 to 22 point lead in the polls and lost.

Furthermore, 20% of the voters who had a high school degree or less, who are the most likely people to vote heuristically and be susceptible by a primacy effect...of these Clinton got 48% of the low education vote vs. 30% for Obama of this demographic

The fact is that there is an inverse relationship between education and support for Clinton...the less education, the more likely to vote Clinton, whereas more educated voters prefer Obama.

At the end of the day women and low-educated voters turned out for Hillary...everyone else preferred Obama

What I want to know is why is it that head to head Dems vs GOP, Edwards beats the Republicans hands down, but he can't win his own party's nomination. Edwards beat the GOP candidates by the biggest margins of any Dem.

"At the end of the day women and low-educated voters turned out for Hillary...everyone else preferred Obama"

Clearly not everyone, as 'women and low-educated voters' do not make up a plurality. As has been noted, Hillary also did better among older voters and registered Democrats. And note that Edwards still took a sizable chunk.

I have no horse in this race, I find all three Dem frontrunners to be tolerable yet disappointing in roughly equal amounts. But declaring that 'only women and the uneducated' voted for Hillary is wildly inaccurate and frankly insulting to the candidates and the voters.

Um, Phil, Perhaps you didn't notice, but Edwards is the only white male in the top tier. That does some interesting things in the head-to-head match-ups. You can actually begin to estimate what percent of the population is either sexist or racist from those numbers. Now if we just had Condi Rice in the race in those polls we could really sort everyone out. Too bad we don't. It'd be fascinating to see how she matched up.

Phil said,

Isn't the disparity more likely to be the " I'm not a racist so I'm not telling the pollster I'm not voting for a black man" theory much more likely?

In Canada, something similar is called the Flora MacDonald effect or the Flora Factor:
Flora MacDonald should have won the leadership of the Progressive Conservative Party in 1976, according to the number of delegates who had committed to her. But when the results were counted... it turned out that delegates abandoned her in the polling booth on the first ballot. The reason seemed to be reluctance to vote for a woman.

It's a measure of how things have changed that nobody seems to think that the race or sex of the candidates is worth discussing. But I wondered if the Iowa results weren't the Flora Factor proving what black women used to say: that they faced more descrimination as women than as blacks. Not that you could actually call Mr. Obama black: light brown, maybe.

Or maybe her better showing in N.H. is the racial version of the Flora Factor.

Could the difference between paper and polling machine results represent some ergonomic factor? Has anyone looked at results from a previous election for similar differences.

I have to recommend the Toronto voting system. It's not national but it works very well. We mark our election choices on an optical card. We cover them up and hand them to a polling agent, who runs them through an optical reader face down. If the card reads cleanly, you're done. If the reader can't make out the choices, you get another card and try again to stay within the lines. Results are tabulated and transmitted as soon as the polls close; but there is a physical ballot that can be checked in case of disputes.

@Albion Tourgee: I was assuming that machines would be used in big towns, where the number of voters is high. Voting methods would correlate with demographics. Eric Lund has the details.

By Lassi Hippeläinen (not verified) on 10 Jan 2008 #permalink