Posted by There was a lot of talk on the ScienceBlogs back channel last week about Mike Dunford’s post on President Bush’s wrangling with Congress over funding the Iraq war. The post attracted a lot of attention, including many comments from readers who claimed Dunford didn’t “support the troops.” If they had actually read the post they would have realized that the “troops” include Mike’s wife and two brothers.
Bora Zivcovic remembered a post by Chris Clarke, which argued that very few readers were willing to read very long blog posts. In the past, he had written several three-part articles, requiring readers to click from page to page. He found that the first page got fifty times as much traffic as the second page, which got fifty times as much traffic as the third page.
So that brought up the question: if a blog post is on a single page, how many readers bother to read it all the way to the end? Mike’s post, at 1,700 words, was a comparable length to the articles Clarke was writing about, so it seemed like the ideal opportunity to conduct an experiment.
In order to do it, a little deception was required: we claimed that the Casual Friday study for last week was actually about political attitudes and legal knowledge, but in reality it was nothing more than a reading comprehension test. We “suggested” that readers consult with Mike’s post before participating in the study. Then half the participants were given a second chance to reread the study before taking the test, and half of those were told that it really was a test of how closely we read blog posts, and encouraged to read it closely. Here are the results:
There were seven questions about the content of the post, corresponding to information given at progressively later points. If blog readers behaved like readers of multi-part posts, you’d expect that they’d get less accurate as the test progressed. Indeed, it does appear that the readers we surprised by linking only to the blog post, without warning them they’d be tested on the content, fared worse as the test went on: their scores for the last three questions are significantly worse than those for readers who were given a second chance to read the post, as well as those who were aware of the real purpose of the test.
But perhaps these readers simply skipped the post entirely — they weren’t just bad at the end, they were bad all the way through. Since we asked readers whether they read the post at all, we could recalculate the scores, throwing out those from readers who skipped the post entirely. Here are those results:
The green line shows the average scores for readers who were only given a link to the post. For six out of the seven questions, their scores were statistically indistinguishable from the other participants — even those who were told they would be tested on the reading. (They scored better on question 2, but this is because the question asked how many follow-up posts Dunford had written, and this could only be answered if you had visited his blog.)
So Cognitive Daily readers, at least, don’t appear to pay less attention to the beginning of a post than the end — unlike the experience Chris Clarke had when he used to break his articles into several pages.
Thus, this study also makes an very good case for not breaking articles up into separate pages — your readers are very likely to pay attention all the way through to the end, while Clarke’s experience suggests they are very unlikely to do so if they have to click through to visit several new pages.
So why did some commenters on Dunford’s post not acknowledge his family’s military commitment? They, like 20 percent of our respondents who got the question about his family wrong, must have just picked the wrong paragraph to skim over.
P.S. Mike Dunford has yet another follow-up to his post here, where he recommends an excellent way for the rest of America to understand the type of sacrifice his family has been making throughout this conflict.
April 13, 2007
April 13, 2007
April 13, 2007
Ok, fair enough Andrew, it could be three dudes and you could be misreading 66% as 63% (it would also be good practice to include actual numerical values on the graph or as a separate table). The point is that not including sample sizes is just bad science. Sure, these casual Friday posts aren’t exactly scientifically sound anyway (no one is claiming that the poll respondents constitute a probability sample), but having sample sizes is just elementary good scientific citizenship.
April 13, 2007
April 13, 2007
April 14, 2007
April 17, 2007