On reading evolution polls

A couple weeks ago, Fox News released a new poll asking about evolution and creationism. It didn't strike me as especially noteworthy, though it does show a statistically significant rise in acceptance of evolution (21% think "the theory of evolution as outlined by Darwin and other scientist" is "more likely to be the explanation for the origin of human life on earth") since they last asked the same question in 1999 (when it was just 15%).

That matches the small but statistically significant rise in support for unguided evolution seen in the nearly 30 years that Gallup has been polling on evolution, where support for evolution without divine involvement has risen from 9% in 1999 to 16% in 2010. The question was asked differently, so the results can't be directly compared, but we see the same shift in opinion over the same range.
When I see a new poll on evolution, I always compare it to that Gallup trend, and to the trend from National Science Board surveys conducted since 1985 (the dotted green line in the graph above). Those two long-term datasets give us a context for most other polls, giving a baseline for comparison. Gallup's question is hardly perfect â as it conflates several issues: evolution, God, age of the earth â and it has three options, with the middle option encompassing theologies that set themselves at odds with sceince (such as old earth creationism or intelligent design creationism) and theologies that embrace scientific findings, such as theistic evolution.

Alas, we can't just plug the new Fox poll into that graph, because their question is worded differently, and because it's conducted by a different pollster. Indeed, you should never blindly compare results from two pollsters, even if they ask exactly the same question. Different pollsters use different techniques for identifying folks to call, they train their surveytakers differently, and place questions about evolution in different contexts. If you just asked a series of science questions, then a question about evolution, you'll get different results than if you just asked a series of questions about politics, or about religion. An individual pollster often makes efforts to control those effects, while different pollsters will vary in consistent ways.

Furthermore, as George Bishop shows convincingly, subtle differences in question wording have big effects on polls of attitudes on evolution. Fox uses the word "evolution," and the word "Darwin," both of which tend to shift people's responses compared to polls using "develop" as Gallup and the NSB both do. Fox explicitly contrasts "the theory of evolution as outlined by Darwin and other scientists" with "the Biblical account of creation as told in the Bible," and allowed people to say they "are both true." Gallup contrasts development by natural processes with "God created human beingsâ¦within the last 10,000 years." The NSB just asks whether human evolution is true. The NSB survey thus isn't explicitly contrasting science and religion, while Gallup and Fox are doing so, but to different extents. Fox's question is about the Bible, while Gallup's is about God. Gallup puts a timeframe on creation and evolution, while Fox doesn't.

Fox's wording explicitly contrasts evolution â a "theory" â with "the Biblical account," and "Darwin and other scientists" against "the Bible." Not surprisingly, Fox gets more people taking the creationist option, 45% in 2011 vs 39% from Gallup in 2010, and because they draw such a sharp contrast, they get fewer people choosing the middle ground option, only 27%, compared to 40% for Gallup's middle ground option ("Humans developedâ¦but God guided the process"). You get essentially the same amount of movement from the middle ground option to each other option (5% switch to the pro-evolution view, 6% go to the creationist view).

Indeed, it's a bit surprising that so many people took Fox's evolution-only option: 21% compared to 16% in Gallup. I'd guess that's partly a result of people switching from the less-appealing middle ground option to the evolution-only option, and also from Fox's heavier weighting of Democrats in their sample. Fox has about 41% Democrats in their sample, while Gallup only finds 31% in their 2010 samples. Fox has 36% Republicans compared to 29% in Gallup's survey, and Fox has 19% Independents, relative to 38% in Gallup. Fox seems to be pushing Independents harder to pick a party, but even ignoring independents, the ratio of Democrats to Republicans is higher in Fox's sample than in Gallup's. That would mean more evolution-supporters, on average.

So what does Fox's poll tell us? It can help guide us toward better framings of the issue, but since they made so many changes relative to Gallup's question, it's hard to separate out the effect of any one change in framing. It confirms the increase in acceptance for naturalistic evolution seen in Gallup, but not seen in the NSB data. If we combine both pro-evolution answers, Gallup's survey shows a much weaker trend (no change at all relative to 1999, which is a low outlier in Gallup's data), while we still see a shift towards evolution in the Fox data. There are various ways to interpret that (Gallup's vague middle ground may be obscuring a shift within the pro-evolution theists), but I don't think we have enough data to make a firm interpretation.

Part of my motive for delving into this is that Allen MacNeill, a sharp observer of the creation/evolution landscape, put forward a rather mangled interpretation of the Fox poll, a misreading that the excellent John Farrell repeated. MacNeill argues:

The questions asked in the poll are very similar to those in the periodic polls on this question conducted by the Gallup organizationâ¦

I find the Fox News poll results surprisingly encouraging. Although the fraction of the American public that agrees with the Young Earth Creationist position hasn't changed significantly for almost half a century, the fraction that agrees with the position taken by evolutionary biologists has increased very significantly since the Gallup organization first polled Americans on this question in 1982.

He also has a graph, tacking the 2011 Fox result on to the Gallup time series, as if it were directly comparable, and writing:

From 9% to 21% in only twenty-nine years (i.e. less than two generations)! If you plot the data, the increase is clearly exponential, with the inflection point at around 2006 (i.e. following the Kitzmiller-Dover decision). At the current exponential rate of increase, the "evolutionary biology" position should be the majority position within another generation.

NONONONONO! You cannot extrapolate like that, and you cannot just tack the Fox and Gallup data together. The only thing that makes this look exponential (which he also claims in the title of his post), is that he is comparing apples and oranges. The Fox and Gallup questions are not "very similar." And in order for the pro-evolution stance to become a majority position, you'd need to have some movement in the creationist view, and the Gallup polls show no such movement.

Farrell writes, correctly:

At first, I wasnât that excited by the recent Fox News poll on Creationism. It shows that 45% of Americans take the Human Origins story in the Book of Genesis literally.

This is consistent with past polls.

But then he buys into MacNeill's apples-to-oranges comparison, taking a small change and inflating it into a major shift. A small shift â 6 points in 12 years â is in line with other polling groups, but a half point per year is hardly news, and a sharper rise than Gallup's 7 points in 28 years (a quarter point per year), but hard to get excited about. Plus, the only increase is within the naturalistic evolution option, none of that shift coming from the creationist group, only from the theistic evolution group. In terms of the political debate, then, no substantive change in how many people endorse evolutionary explanations (matched by the NSB data, which show no trend over 25 years).