…hypothesis testing!

To recap, Jason Rosenhouse, who I love like a brother, put up a post using a poll from VCU and data from Gallup polling to address a hypothesis about “New Atheists,” a hypothesis he attributes broadly to the critics of “New Atheism.” He’s since clarified (in a comment pledging not to reply further here, alas) that he was thinking of a comment by Michael Ruse that New Atheists have been “a bloody disaster,” with Jason add: “it is hard to find a critic of the NA’s who has not” claimed “the NA’s are hurting the cause of good science education.”

He proposed to test this hypothesis by looking at change in public answers to the questions:

Which of these statements comes closest to your views on the origin of biological life: biological life developed over time from simple substances, but God guided this process, biological life developed over time from simple substances but God did not guide this process, God directly created biological life in its present form at one point in time?

and

Which of the following statements comes closest to your views on the origin and development of human beings? (1) Human beings have developed over millions of years from less advanced forms of life, but God guided this process. (2) Human beings have developed over millions of years from less advanced forms of life, but God had no part in this process. (3) God created human beings pretty much in their present form at one time within the last 10,000 years or so.

The latter has been asked for almost 30 years in Gallup polls, the former in two polls from Virginia Commonwealth University. Jason then brings out data from only the most recent VCU poll and from Gallup polls since 1998.

So far as I can tell, the specific hypothesis being tested is as follows: If New/Explicit/Affirmative Atheists are really hurting science education, there should be a decline in acceptance of the claim that evolution unguided by God explains the origins of humans and of life more generally.

Before getting to testing this hypothesis, it’s worth noting that this is hardly the only hypothesis one could form from the remarks Jason cites. Ruse’s “bloody disaster” remark is hardly a prediction that acceptance of evolution would decline, nor that acceptance of evolution unguided by God would decline. In my last post, I clarified my own views:

I don’t know that Chris [Mooney] has ever claimed explicitly that New Atheism will actively decrease public acceptance of evolution or increase acceptance of creationism. I certainly know I haven’t made that claim. Most significantly, I wouldn’t claim that because I think the main effect of that fracas is to simply turn people away from experts whose opinion they might otherwise listen to. This could swell the ranks of the undecided, but I doubt it will move many people out of that broad middle. My concern, and what I understand Chris (and his co-authors Matt Nisbet and Sheril Kirshenbaum, at various times) to have principally argued, is that the New Atheism is likely to turn people away from folks they’d otherwise regard as trustworthy experts, and that this could hold back ongoing outreach efforts. These data don’t really allow testing of that hypothesis. (I’ve hedged my language here because it may be that there is such an explicit prediction, and this paragraph would be moot.)

Jason responded to that remark by accusing me of moving the goalposts. But I don’t see it. First, because he never shows, or even makes an attempt to show, that this is different than anything I’ve said before. Would this effect still “hurt[] the cause of good science education” (Jason’s account of the claim)? I think it would, so based on what Jason’s offered as evidence, it looks more like he created a straw man than that I’ve moved the goalposts. But my parenthetical above still applies. If someone offers actual evidence of the prediction he’s testing having been offered by any of the folks he cites, I’ll happily withdraw the objection.

Regardless of whether anyone has formulated this exact hypothesis before, it’s not unreasonable or uninteresting, so we can still try to test it. To test this hypothesis against polling data, one poses the null hypothesis that there is no decline in the acceptance of unguided evolution since the rise of New Atheists. If we reject this null hypothesis, it doesn’t prove causation, but it does indicate that there is a meaningful association, thus waggling its eyebrowss suggestively at causation.

If we fail to reject the null hypothesis, it doesn’t really tell us anything. If we’d asked every American adult these same questions, and had a perfect census rather than a sample, it’s extraordinarily unlikely that that we’d find the numbers staying perfectly constant. If the change is small in the sample, it will be statistically indistinguishable from random noise, which might mean that it is both statistically non-significant and practically non-significant. But with only 9 data points, we could have practically significant trends but not enough statistical power to identify them. Failing to reject a null hypothesis does not mean that the null hypothesis is true, let alone that the alternative hypothesis (the really interesting one) is false!

On this basis, we can set up the 9 data points from Gallup in a generalized linear model, using a binomial model with logit link, and test whether there is a significant trend to the data on unguided evolution. And when I do that in R, I get:

              Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.655e+00  4.583e+00  -0.579    0.562
slope        5.639e-05  3.934e-04   0.143    0.886

              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -2.655e+00  1.449e-01 -18.320  < 2e-16 ***
slope        5.639e-05  1.244e-05   4.533 5.82e-06 ***

Doing the same thing with only the results since 1998 (a cutoff of Jason's choosing, and somewhat arbitrary IMHO):

              Estimate Std. Error z value Pr(>|z|)
(Intercept)  1.911e-01  8.841e+00   0.022    0.983
slope       -3.101e-05  6.968e-04  -0.044    0.965

              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -3.520e+00  4.383e-01  -8.032 9.62e-16 ***
date         1.240e-04  3.421e-05   3.624 0.000290 ***

The important part is that last column, where the p-values are well above 0.05 well below 0.05, indicating that there is no a significant statistical difference from a flat line. Again, this doesn't mean no trend at all, it just means that we can't detect any trend given the data available.

So we fail to falsify the null hypothesis. This means Jason's eyeball estimates are valid, but for reasons discussed below, doesn't mean critics of New Atheists (by Jason's account) are wrong. For one thing, the only place where we see significant change is in the "unguided evolution" option, not in either of the other options nor in the combined acceptance of evolution (combining guided and unguided evolution).

Jason, in his comments, hints at a second hypothesis, arguing that a finding of no trend or a positive trend for unguided evolution tends to show that New Atheists are doing no harm.

This is trickier to test. Failing to falsify the null hypothesis of no trend does not let us say that there's really no trend, and The statistically detectable rise in acceptance of unguided evolution would not necessarily demonstrate that New Atheists are winning. The problem here is with inferring causation based on a time series when you haven't established any controls. If something else changed in US politics at the same time New Atheists came on the scene, how do you separate those effects?

And various things have been happening in the world of creationism (ID and otherwise) at the same time that things were happening in the world of New Atheism. Interesting things have been happening in the world of theistic evolution, too. Testing the effect of New Atheists requires having some way to separate their actions from others.

An analogy: Since the collapse of the world economy in 2008, the US economy has shed jobs at a horrific pace. In early 2009, Congress and the Obama administration rushed through a stimulus package, the American Recovery and Reinvestment Act, that did a lot to stop the bleeding and get the economy back on the path to recovery. The well-respected Council of Economic Advisers estimates that ARRA saved or created 2.5 to 3.6 million jobs. But the ARRA had a smaller effect than the collapse of world financial markets, the banking sector, etc., and the economy has lost jobs since ARRA passed. Teabaggers point to that and insist that this means ARRA didn't work. But it did! As the figure above illustrates, things would have been much worse if ARRA hadn't passed. The valid comparison is not between today's employment figures and those when ARRA passed, but between today's employment figures and those which would have been but for ARRA.

To get an estimate of the impact of New Atheists, then, we need to have something more than just the polling data. We also need a model that accounts for everything else that happened. Biologists often get such a model by simply creating a control group that experiences the same general conditions as their experimental group, allowing direct evaluation of the effect of the experimentally varied factor. Clearly we cannot go back and create a United States that is identical in every way to the USA of 2004, and then allow it to continue with the only difference being that Sam Harris and Richard Dawkins and Christopher Hitchens and Daniel Dennett (to a lesser extent) never publish their books, or publish different books.

Instead, we might do something like the CEA did: construct a statistical model that accounts for all the factors bearing on acceptance of creationism or evolution, and then run that model with New Atheism as a factor, and again without. The problem is, we haven't got anything like the amount of data CEA has, and what we have would never let us parameterize such a model.

Here are a few of the things that might countervail upon the effects of the NAs: The Clergy Letter Project kicked off in 2004, roughly when Harris's The End of Faith came out. CLP kicked off Evolution Weekend, when clergy present sermons on the compatibility of science and religion, in 2006, the year Dawkins published The God Delusion. Kitzmiller and Tiktaalik both came out in 2005, and movies, TV shows, plays, magazine articles, newspaper stories, blog posts, and other media about Kitzmiller and Tiktaalik and other scientific advances in evolution followed steadily. There was the 2009 Year of Science and the Darwin bicentennial and Origin of Species sesquicentennial. Jerry Coyne and Richard Dawkins and Ken Miller wrote widely discussed books about the science of evolution, that set aside religious polemic. Before 2003, there were no books by evangelical Christians defending evolution books, but there have been at least 10 since then. Science blogs, including blogs by religious scientists and by New Atheists, gained prominence. Youtube and podcasting rose as methods of narrowcasting, gaining use by all factions.

Then you have the creationists. AiG opened its creationism museum in 2007, as Hitchens released God is Not Great. Creationists in Kansas got a majority of the state board of education in 2004, enacted creationist standards in 2005, and lost their majority in 2006, after which the standards were fixed. Ohio creationists backed down from a creationist standard and associated curriculum guide in 2006, shortly after the Kitzmiller ruling, but also after public hearings where Ken Miller, a religious scientist, defended evolution. ID promoters managed to sell a lot of books and sway a few influential folks to their side before Kitzmiller, then regrouped under the banner of "critical analysis" and "academic freedom," passing the first creationist legislation in a long time. The Texas Board of Education fought a widely-reported battle over evolution in textbooks in 2003, and in 2008 fought a battle over evolution in state science standards. Ben Stein stumped hard for Expelled, which did pretty well at the box office.

All of that will impact public opinion, either for creationism, for unguided evolution, or for factions within the guided evolution camp. Some of those phenomena could tend to dull the effect of New Atheism's attack on religion, or to counteract it, or to promote it. There's no obvious way to tease out these interacting effects. As this was all happening and before, back to the '90s, nonbelief (though not atheism) has become the fastest growing religious affiliation in the US, a factor which will both cause a trend in views on evolution and New Atheism and will also change in response to evolution and New Atheism.

As I said before, it may be that careful work with the GSS would give enough demographic controls that you could pick some of this apart. Were there different trajectories in people's views of evolution in areas with active creationist efforts? How do the many variables tracking religiosity interact with people's views on evolution? How does that match against demographic trends in polls by Pew, Gallup, and Harris, all of whom have asked the same questions for several years.

I'd like to see someone do this work, and I'd welcome citations of papers which might serve as a basis for such an analysis. But the evidence at hand simply isn't adequate for what Jason would like do with it. Noting those complications is not shifting goalposts, nor is it making excuses. It's the way I would think about any challenge in hypothesis testing. If we want to promote science as a way of knowing, I think it behooves us to model good scientific practices, and that's my agenda with this post and the post it follows from.

Update: Thanks to a commenter at Jason's blog, I've fixed a stupid error in my binomial regressions, and now get the statistically significant rise in unguided evolution that I should have gotten all along. I've indicated the changed statistical results and updates to the text above as needed. Let this be a lesson not to do statistical analysis on the fly!