Good Math, Bad Math

What happens is that they look at polls, and they talk about the results and what they mean. But they, like almost everyone, use the margin of error as if it means something very different than what it really does.

What you hear is that, for example, Barak Obama is leading Florida by 5 points, but the margin of error is +/- 4%, so it's really not a significant lead. What the journalists seem to think it means is that the margin of error is a total measure of the accuracy of the polls - that the poll result is within the margin of error of the "true" result that the poll measures. So, by that interpretation, the poll is predicting an outcome of 52/48, and the margin of error means that the range of actual voter preferences ranges between 48/52 and 56/44.

The thing is, that's not what the margin of error means. The margin of error is a statistical measure of the probabilistic size of errors caused by unintentional sampling errors.

Polls - and much of statistics in general - are based on the idea of sampling. Given a large, relatively uniform population, you can get an amazingly accurate measure of that population by looking at a small subset of it, called a representative sample. A sample is a randomly selected group that is intended to be a microcosm of the entire population. In an ideal representative sample, the sample must have the same distribution of differences as the population as a whole.

There's a big problem there: how can you be sure that your sample is representative? The answer is, you can't! The only way to know for certain that a sample is representative is to measure the entire population, and compare the results of doing that to the sample. But once you've measured the entire population, what's the point of looking at a sample?

Fortunately, we can assess how likely it is that our sample is a good representation of the population. That's what the margin of error does - it measures the likelihood of the sample being representative of the population. It's computed by combining a bunch of factors, the primary ones most commonly being the size of the population and the size of the sample. Given those, we can assess how certain you can be of your measure being pretty close to accurate. Typically, we describe that certainty by stating how large an interval you need to define on either side of the measured statistic to be 95% certain that the "actual" value is within that interval. The size of that interval is the margin of error.

So when you hear a pollster talking about a "poll of likely voters showing that Obama is ahead by 8 points with a margin of error of +/-4%", the big thing you should do is realize what they're measuring. In that case, the population isn't "the set of people who are going to vote next tuesday" (even though that's what the journalists try to make you think); the population is "the set of people who the poll believes are likely to vote next tuesday". So the margin of error is a measure of how well their poll matches the population of people who they believe are likely to vote - which is quite a different thing from the population of people who actually do vote. In fact, it actually does slightly less, even, than that: it measures how much sampling error is contained in their poll due to unintentionally selecting a non-representative sample. That's not really saying very much in an election poll.

The population being sampled by polls is likely to be quite different from the actual population of voters for a number of reasons, and this difference produces measurement errors that almost certainly significantly outweigh the unintentional sampling errors measured by the margin of error. For example:

Intentional Sample Bias

Intentional sample bias covers a variety of techniques that pollsters use when they select people for the sample. For an extreme example, some polls (like, I think, Zogby) try to get an equal number of people who self-identify as republicans and democrats. But in most states, the number of party members in the two major parties are not equal. They are, in fact, often pretty dramatically uneven. A less dramatic but still significant one is that many polls do their polling through phone calls, and only call land-lines. Many younger people no longer have land-lines; the exclusion of cell-phone numbers therefore excludes some portion of the population from the sample. These kinds of sample bias produce a significant mismatch between the population of real voters, and the population being sampled.

Unknown Population

The biggest of polling errors leading up to an election is the fact that the real population is unknown. No one is sure who's going to vote - which means that no one is certain of what the correct population to sample is. Pollsters try to identify a sample of people who are likely to vote. But since the population is unknown, they don't know if they're including people in the sample who aren't in the actual population of voters, and they don't know if they're excluding people from their samples who are going to vote. In this election, this is likely to be a significant effect, because huge numbers of people registered to vote for the first time, but no one knows how many of those newly registered voters are likely to show up and vote. Once again, there's a problem related to the fact that the population that they're sampling isn't the same as the population that the poll is trying to measure - so that error factor is outside the margin of error.

Phrasing Bias

You can get significant differences in polls based on how the question is phrased. "Who are you going to vote for?" will likely generate different results from "Are you going to vote for Obama or McCain?", which will likely generate different results from "Do you plan to vote for McCain or Obama?", which will generate different results from "Do you plan to vote for a Democrat or a Republican in the presidential election?". This is a well-known problem, but it still has a significant effect.

Dishonest Answers

People aren't entirely trustworthy. They don't necessarily answer questions honestly. A frequently discussed version of this is called the Bradley effect. The Bradley effect is a phenomenon where people are reluctant to admit to being racist. So when a pollster asks them if they're going to vote for a black man, they'll say "yes", but when it actually comes to voting, they'll vote for the white guy. I've heard some people speculate on a reverse Bradley effect this year in some southern states, where people are reluctant to admit that they're going to vote for a black man, so they lie and say they're voting McCain. But the truth of the matter is, we don't know if the people answering the polls are answering honestly. If they're not, that skews the poll results, and once again, it's not covered by the margin of error.