â¦hypothesis testing!
To recap, Jason Rosenhouse, who I love like a brother, put up a post using a poll from VCU and data from Gallup polling to address a hypothesis about "New Atheists," a hypothesis he attributes broadly to the critics of "New Atheism." He's since clarified (in a comment pledging not to reply further here, alas) that he was thinking of a comment by Michael Ruse that New Atheists have been "a bloody disaster," with Jason add: "it is hard to find a critic of the NA's who has not" claimed "the NA's are hurting the cause of good science education."
He proposed to test this hypothesis by looking at change in public answers to the questions:
Which of these statements comes closest to your views on the origin of biological life: biological life developed over time from simple substances, but God guided this process, biological life developed over time from simple substances but God did not guide this process, God directly created biological life in its present form at one point in time?
and
Which of the following statements comes closest to your views on the origin and development of human beings? (1) Human beings have developed over millions of years from less advanced forms of life, but God guided this process. (2) Human beings have developed over millions of years from less advanced forms of life, but God had no part in this process. (3) God created human beings pretty much in their present form at one time within the last 10,000 years or so.
The latter has been asked for almost 30 years in Gallup polls, the former in two polls from Virginia Commonwealth University. Jason then brings out data from only the most recent VCU poll and from Gallup polls since 1998.
So far as I can tell, the specific hypothesis being tested is as follows: If New/Explicit/Affirmative Atheists are really hurting science education, there should be a decline in acceptance of the claim that evolution unguided by God explains the origins of humans and of life more generally.
Before getting to testing this hypothesis, it's worth noting that this is hardly the only hypothesis one could form from the remarks Jason cites. Ruse's "bloody disaster" remark is hardly a prediction that acceptance of evolution would decline, nor that acceptance of evolution unguided by God would decline. In my last post, I clarified my own views:
I don't know that Chris [Mooney] has ever claimed explicitly that New Atheism will actively decrease public acceptance of evolution or increase acceptance of creationism. I certainly know I haven't made that claim. Most significantly, I wouldn't claim that because I think the main effect of that fracas is to simply turn people away from experts whose opinion they might otherwise listen to. This could swell the ranks of the undecided, but I doubt it will move many people out of that broad middle. My concern, and what I understand Chris (and his co-authors Matt Nisbet and Sheril Kirshenbaum, at various times) to have principally argued, is that the New Atheism is likely to turn people away from folks they'd otherwise regard as trustworthy experts, and that this could hold back ongoing outreach efforts. These data don't really allow testing of that hypothesis. (I've hedged my language here because it may be that there is such an explicit prediction, and this paragraph would be moot.)
Jason responded to that remark by accusing me of moving the goalposts. But I don't see it. First, because he never shows, or even makes an attempt to show, that this is different than anything I've said before. Would this effect still "hurt[] the cause of good science education" (Jason's account of the claim)? I think it would, so based on what Jason's offered as evidence, it looks more like he created a straw man than that I've moved the goalposts. But my parenthetical above still applies. If someone offers actual evidence of the prediction he's testing having been offered by any of the folks he cites, I'll happily withdraw the objection.
Regardless of whether anyone has formulated this exact hypothesis before, it's not unreasonable or uninteresting, so we can still try to test it. To test this hypothesis against polling data, one poses the null hypothesis that there is no decline in the acceptance of unguided evolution since the rise of New Atheists. If we reject this null hypothesis, it doesn't prove causation, but it does indicate that there is a meaningful association, thus waggling its eyebrowss suggestively at causation.
If we fail to reject the null hypothesis, it doesn't really tell us anything. If we'd asked every American adult these same questions, and had a perfect census rather than a sample, it's extraordinarily unlikely that that we'd find the numbers staying perfectly constant. If the change is small in the sample, it will be statistically indistinguishable from random noise, which might mean that it is both statistically non-significant and practically non-significant. But with only 9 data points, we could have practically significant trends but not enough statistical power to identify them. Failing to reject a null hypothesis does not mean that the null hypothesis is true, let alone that the alternative hypothesis (the really interesting one) is false!
On this basis, we can set up the 9 data points from Gallup in a generalized linear model, using a binomial model with logit link, and test whether there is a significant trend to the data on unguided evolution. And when I do that in R, I get:
Estimate Std. Error z value Pr(>|z|) (Intercept) -2.655e+00 4.583e+00 -0.579 0.562 slope 5.639e-05 3.934e-04 0.143 0.886Estimate Std. Error z value Pr(>|z|) (Intercept) -2.655e+00 1.449e-01 -18.320Doing the same thing with only the results since 1998 (a cutoff of Jason's choosing, and somewhat arbitrary IMHO):
Estimate Std. Error z value Pr(>|z|) (Intercept) 1.911e-01 8.841e+00 0.022 0.983 slope -3.101e-05 6.968e-04 -0.044 0.965Estimate Std. Error z value Pr(>|z|) (Intercept) -3.520e+00 4.383e-01 -8.032 9.62e-16 *** date 1.240e-04 3.421e-05 3.624 0.000290 ***The important part is that last column, where the p-values are
well above 0.05well below 0.05, indicating that there isnoa significant statistical difference from a flat line.Again, this doesn't mean no trend at all, it just means that we can't detect any trend given the data available.So we
fail tofalsify the null hypothesis. This means Jason's eyeball estimates are valid, but for reasons discussed below, doesn't mean critics of New Atheists (by Jason's account) are wrong. For one thing, the only place where we see significant change is in the "unguided evolution" option, not in either of the other options nor in the combined acceptance of evolution (combining guided and unguided evolution).Jason, in his comments, hints at a second hypothesis, arguing that a finding of no trend or a positive trend for unguided evolution tends to show that New Atheists are doing no harm.
This is trickier to test.
Failing to falsify the null hypothesis of no trend does not let us say that there's really no trend, andThe statistically detectable rise in acceptance of unguided evolution would not necessarily demonstrate that New Atheists are winning. The problem here is with inferring causation based on a time series when you haven't established any controls. If something else changed in US politics at the same time New Atheists came on the scene, how do you separate those effects?And various things have been happening in the world of creationism (ID and otherwise) at the same time that things were happening in the world of New Atheism. Interesting things have been happening in the world of theistic evolution, too. Testing the effect of New Atheists requires having some way to separate their actions from others.
An analogy: Since the collapse of the world economy in 2008, the US economy has shed jobs at a horrific pace. In early 2009, Congress and the Obama administration rushed through a stimulus package, the American Recovery and Reinvestment Act, that did a lot to stop the bleeding and get the economy back on the path to recovery. The well-respected Council of Economic Advisers estimates that ARRA saved or created 2.5 to 3.6 million jobs. But the ARRA had a smaller effect than the collapse of world financial markets, the banking sector, etc., and the economy has lost jobs since ARRA passed. Teabaggers point to that and insist that this means ARRA didn't work. But it did! As the figure above illustrates, things would have been much worse if ARRA hadn't passed. The valid comparison is not between today's employment figures and those when ARRA passed, but between today's employment figures and those which would have been but for ARRA.
To get an estimate of the impact of New Atheists, then, we need to have something more than just the polling data. We also need a model that accounts for everything else that happened. Biologists often get such a model by simply creating a control group that experiences the same general conditions as their experimental group, allowing direct evaluation of the effect of the experimentally varied factor. Clearly we cannot go back and create a United States that is identical in every way to the USA of 2004, and then allow it to continue with the only difference being that Sam Harris and Richard Dawkins and Christopher Hitchens and Daniel Dennett (to a lesser extent) never publish their books, or publish different books.
Instead, we might do something like the CEA did: construct a statistical model that accounts for all the factors bearing on acceptance of creationism or evolution, and then run that model with New Atheism as a factor, and again without. The problem is, we haven't got anything like the amount of data CEA has, and what we have would never let us parameterize such a model.
Here are a few of the things that might countervail upon the effects of the NAs: The Clergy Letter Project kicked off in 2004, roughly when Harris's The End of Faith came out. CLP kicked off Evolution Weekend, when clergy present sermons on the compatibility of science and religion, in 2006, the year Dawkins published The God Delusion. Kitzmiller and Tiktaalik both came out in 2005, and movies, TV shows, plays, magazine articles, newspaper stories, blog posts, and other media about Kitzmiller and Tiktaalik and other scientific advances in evolution followed steadily. There was the 2009 Year of Science and the Darwin bicentennial and Origin of Species sesquicentennial. Jerry Coyne and Richard Dawkins and Ken Miller wrote widely discussed books about the science of evolution, that set aside religious polemic. Before 2003, there were no books by evangelical Christians defending evolution books, but there have been at least 10 since then. Science blogs, including blogs by religious scientists and by New Atheists, gained prominence. Youtube and podcasting rose as methods of narrowcasting, gaining use by all factions.
Then you have the creationists. AiG opened its creationism museum in 2007, as Hitchens released God is Not Great. Creationists in Kansas got a majority of the state board of education in 2004, enacted creationist standards in 2005, and lost their majority in 2006, after which the standards were fixed. Ohio creationists backed down from a creationist standard and associated curriculum guide in 2006, shortly after the Kitzmiller ruling, but also after public hearings where Ken Miller, a religious scientist, defended evolution. ID promoters managed to sell a lot of books and sway a few influential folks to their side before Kitzmiller, then regrouped under the banner of "critical analysis" and "academic freedom," passing the first creationist legislation in a long time. The Texas Board of Education fought a widely-reported battle over evolution in textbooks in 2003, and in 2008 fought a battle over evolution in state science standards. Ben Stein stumped hard for Expelled, which did pretty well at the box office.
All of that will impact public opinion, either for creationism, for unguided evolution, or for factions within the guided evolution camp. Some of those phenomena could tend to dull the effect of New Atheism's attack on religion, or to counteract it, or to promote it. There's no obvious way to tease out these interacting effects. As this was all happening and before, back to the '90s, nonbelief (though not atheism) has become the fastest growing religious affiliation in the US, a factor which will both cause a trend in views on evolution and New Atheism and will also change in response to evolution and New Atheism.
As I said before, it may be that careful work with the GSS would give enough demographic controls that you could pick some of this apart. Were there different trajectories in people's views of evolution in areas with active creationist efforts? How do the many variables tracking religiosity interact with people's views on evolution? How does that match against demographic trends in polls by Pew, Gallup, and Harris, all of whom have asked the same questions for several years.
I'd like to see someone do this work, and I'd welcome citations of papers which might serve as a basis for such an analysis. But the evidence at hand simply isn't adequate for what Jason would like do with it. Noting those complications is not shifting goalposts, nor is it making excuses. It's the way I would think about any challenge in hypothesis testing. If we want to promote science as a way of knowing, I think it behooves us to model good scientific practices, and that's my agenda with this post and the post it follows from.
Update: Thanks to a commenter at Jason's blog, I've fixed a stupid error in my binomial regressions, and now get the statistically significant rise in unguided evolution that I should have gotten all along. I've indicated the changed statistical results and updates to the text above as needed. Let this be a lesson not to do statistical analysis on the fly!
- Log in to post comments
Mr. Rosenau gave a medium-sized lecture of statistics. I thank him for that, but wonder what motivated him to do so. Rosenhouse basically just said that there is no evidence of a backlash on science education due to the outspokenness of NA's. Mr. Rosenau does not disagree with him on this matter.
Why make such a big fuzz about statistics and polls? In frank political discussions, one does not think of the polls, but speaks out one's mind sincerely. That is what the NA's are doing. What's wrong with that?
In politics there is of course another dimension, collecting popular support. Many accommodationists argue that NA's should be less outspoken for tactical political reasons. However, they have no real evidence that the actions of NA's are harming the cause of scientific outreach. What they are basically saying is: "I do not like what and how you say, so could you please tune it down?"
It seems to me that with his latest post, Mr. Rosenau tries to show that it cannot be ruled out that NA's harm scientific outreach. I suspect that the outspoken atheists will not limit the expression of their sincere conclusions due to this speculation.
What happens if you use normal regression, rather than a GLM with binomial error? My guess is that you'll find wider confidence intervals (I might look later, but I don't want to piss around with formatting data right now). BTW, it would be nice to plot the data too, so we can see what it looks like (R has all these great graphical facilities. :-)).
Over all, I think this sort of analysis is pretty crude, and I wouldn't infer anything about New Atheists either way: there are simply so many other things that could affect these polls (as you state).
For me a more interesting question than whether NAs(1) have had an overall effect is whether their arguments have an effect on individuals' views: present people with NA arguments and rhetoric, and see if that changes their views. I assume there is research on this (perhaps not on atheism, but on different ways of presenting arguments), and the Framers can probably give us the references.
(1) I'm sure there's a joke there for users of R and S-plus.
Right back atcha! But sometimes siblings fight.
You keep saying I was testing an hypothesis. I wasn't! I wasn't even putting forth an hypothesis. I did not even say there wasn't a backlash. My claim was simply that the poll numbers show no evidence of a backlash, and they don't. Yet on and on you go. There's a reason my post was so short, whereas your posts are very long.
I accused you of moving the goalposts because you did. Neither Ruse, nor Mooney and Kirshenbaum, said anything about turning people away from folks they would otherwise regard as experts thereby hampering outreach. Mooney and Kirshenbaum's argument, in their NA chapter in UA, is that NA's are forcing people to choose between science and religion, and given that choice they will choose religion. Then they go on to give a truly asinine lecture about the limits of science. Ruse says very similar things in his “bloody disaster” essay. Their rhetoric is much stronger that what is suggested in your characterization of their argument.
As I said in the comments to my post, my view is that the NA's have done obvious, unambiguous good for the cause of promoting atheism. Atheists are part of the public conversation today in ways they never were previously, and I regard that as an essential first step to any attempt to mainstream it. But many people I respect tell me they are actually doing harm to the cause of good science education. If that were true I would certainly have to rethink my attitude towards the NA's, but the burden of proof lies with the people making the claim. It is not my job to prove there isn't a backlash, it is theirs to prove that there is.
» Josh Rosenau:
⦠would not necessarily demonstrate that New Atheists are winning â¦
Looks like the windmills are winning, Josh. As I said at Jasonâs place, youâre belabouring a peripheral, actually very probably irrelevant, point. Jasonâs point, as he has pointed out to you and as is readily apparent from the headline and the closing paragraph of his post, is that those in the accommodationist camp have never shown any evidence for their claims that the Spiffy New Atheistsâ approach is destructive, counterproductive, or otherwise negative in some way or other. One possible place to look for such evidence would be the poll numbers Jason is talking about. And it is about those that he makes one simple observation: there is no consolation in them for the accommodationists. He is saying: there is, other things being equal, no prima facie evidence of negative NA effects in these numbers. I.e., weâre still waiting for you to produce something that might make your assertions debatable.
I make a fuss over statistics and polls because I think a lot of people don't understand statistics and polling, and I think these are important topics. So it's a teachable moment.
There's a place for poll analysis in politics. Not that one simply says "the majority has spoken, so let's do what they want," but because looking at the polls tells you how the populace is thinking about issues. If one wants to change public opinion, for instance on evolution, it's important to know what the public thinks, and to understand why. Polls are how we do that, and statistics are the tool we use to get the important answers from polls.
I think the "real evidence ⦠[about] the actions of NAs" will come not from big national surveys, which are blunt instruments, but from careful sociological research.
rni.boh: Thanks to a commenter at Jason's I fixed an error in my approach to regression and am making an update now.
Jason: It is only moving the goalposts if any of the people you cite had ever claimed that poll numbers should show this backlash. I've listed several other places the backlash could be seen, and sound statistical reasons why it is inappropriate to make the sorts of comparisons you are at least hesitantly attempting. It's true that the burden of proof for showing a backlash falls to NAs' critics, but the burden of proof falls on NAs to demonstrate the positive effects (you offer some useful anecdotal evidence, but something more quantifiable would be nice). Jerry Coyne, for instance, claims that NAism is necessary in part because the old "accommodationist" ways have not worked in changing public polling on evolution. I'd like to see you ask him for evidence that NAism does a better job.
Mr. Rosenau: "It's true that the burden of proof for showing a backlash falls to NAs' critics, but the burden of proof falls on NAs to demonstrate the positive effects (you offer some useful anecdotal evidence, but something more quantifiable would be nice)."
Well, one cannot deny that the NA's have increased the public discussion of atheism and it's clashes with religion. This can be measured f. ex. by real estate in traditional media.
Many social reforms are preceded by increased discussion, so intensified discussion should be considered a positive thing. Unfortunately, many accommodationists seem to prefer tactical clamming up when criticizing religion.
"Jerry Coyne, for instance, claims that NAism is necessary in part because the old "accommodationist" ways have not worked in changing public polling on evolution. I'd like to see you ask him for evidence that NAism does a better job."
Are you willing to give the NA the same number of decades the accommodationists have had to make a difference?
The accommodationists have been saying for years "oh those nasty 'New Atheists, they're forcing people away from science." Now some real data says this doesn't appear to be the case. Do accommodationists say "sorry, 'New Atheists', we might have been wrong"? Not them. Folks like Mooney & Kirshenbaum and Rosenau would rather slash their wrists than admit a possible error in their whining about "New Atheists."
Josh, your apology, while not expected, will be accepted.
Josh Rosenau wrote:
I believe that every "New Atheist" I've ever read talking about this claims that the "NA way" is better only because it is intellectually honest and not condescending, not that it will necessarily increase acceptance of science. I, who am more "New Atheisty" than most "New Atheists," actually expect a backlash as a cost of taking a stand on principle, and I'm still expecting it (I'm disappointed that it hasn't happened yet, as it means - to me - that the proper societal change is taking longer than I'd prefer).
I, for one, would like to see a link or other citation to this claim.
@Tis Himself:
Let's not hold our breaths while we wait. ;)
My thoughts on Josh's post: My, that certainly is a lot of words.
Do you consider yourself an educator? I hope so, and I appreciate the work you and the rest of the NCSE do, but I don't think you're doing a very good job of educating right now. Of course those are important topics. But there are always important topics that get transcended by a teachable moment.
There are lots of different ways to define teachable moment, but the commonality is the "moment" part: a particular moment when learners will be extra receptive to learning. And the most common usage is when discussing particular details, a larger and deeper point can be made using the details as an example or starting point.
However, you take the situation of debating whether there is a backlash against NA's, to drill down into the details of polls and statistics. Another time, dude. Readers are not extra receptive to learning about polling and statistics through this debate. People have told you that you are missing the forest for the trees. Please listen to them. That's the ultimate quality of an educator, good listening skills.
Well to me it seems like you agree.
Neither you nor Mooney, Nisbet or Kirshenbaum shown the slightest evidence that affirmative atheists in any way what so ever causes harm to science education or acceptance of science, and you freely admit that.
So why not just accept Jasons point - the polls show no backlash against affirmative atheists?
After all, its just status quo - still no evidence of any backlash.