The margin of error is the most widely misunderstood and misleading concept in
statistics. It’s positively frightening to people who actually understand what it means to see
how it’s commonly used in the media, in conversation, sometimes even by other scientists!
The basic idea of it is very simple. Most of the time when we’re doing statistics, we’re doing statistics based on a sample – that is, the entire population we’re interested in is difficult to study; so what we try to do is pick a representative subset called a sample. If the subset is truly representative, then the statistics you generate using information gathered from the sample will be the same as information gathered from the population as a whole.
But life is never simple. We never have perfectly representative samples; in fact, it’s
impossible to select a perfectly representative sample. So we do our best to pick good
samples, and we use probability theory to work out a predication of how confident we can be that the
statistics from our sample are representative of the entire population. That’s basically what the
margin of error represents: how well we think that the selected sample will allow us to
predict things about the entire population.
The way that we compute a margin of error consists of a couple of factors:
- The size of the sample.
- Magnitude of known problems in the sample.
- The extremity of the statistic.
Let’s look at each of those in a bit more detail to understand what they mean, and then I’ll explain how we use them to compute a margin of error.
The larger a sample is, the more likely it is to be representative. The intuition behind that is
pretty simple: the more individuals that we include in the sample, the less likely we are to
accidentally omit some group or some trend that exists within the population. Taking a presidential
election as an example: if you polled 10 randomly selected people in Manhattan, you’d probably get
mostly democrats, and a few republicans. But Manhattan actually has a fairly sizeable group of people
who vote for independents, like the green party. If you sampled 100 people, you’d probably get at
least three or four greens. With the smaller sample size, you’d wind up with statistics that overstated the number of democrats in Manhattan, because the green voters, who tend to be very liberal, would probably be “hidden” inside of the democratic stat. If you sampled 1000 people, you’d be more likely to get a really good picture of NYC: you’d get the democrats and republicans, the conservative party, the working families party, and so on – all groups that you’d miss in the
When we start to look at a statistic, we start with an expectation: a very rough sense
of what the outcome is likely to be. (Given a complete unknown, we generally start with the Bayesian
assumption that in the presence of zero knowledge, you can generally assign a 50/50 split as an
initial guess about the division of any population into exactly two categories.) When we work with a
sample, we tend to be less confident about how representative that sample is the farther the
measured statistic varies from the expected value.
Finally, sometimes we know that the mechanism we use for our sample is imperfect – that
is, we know that our sample contains an unavoidable bias. In that case, we expand the margin of error
to try to represent the reduced certainty caused by the known bias. For example, in elections, we
know that in general, there are certain groups of people who simply are less likely
to participate in exit polls. An exit poll simple cannot generate an unbiased sample, because the outcome is partially determined by who is willing to stop and take the poll. Another example is in polls involving things like sexuality, where because of social factors, people are less likely to admit to certain things. If you’re trying to measure something like “What percentage of people have had extramarital affairs?”, you know that many people are not going to tell the truth – so your result will include an expected bias.
Given those, how do we compute the margin of error? It depends a bit on how you’re measuring. The
easiest (and most common) case is a percentage based statistic, so that’s what we’ll stick
with for this article. The margin of error is computed from the standard error, which is in
turn derived from an approximation of the standard deviation. Given a population of size P;
and a measured statistic of X (where X is in decimal form – so 50% means X=0.5), the standard error E