A couple hours ago I posted a quick poll, in what might be construed as an unbiased fashion. I simply asked respondents for their sexual orientation, offering a wide array of choices ranging from “straight” to “mostly gay” to “gay” to “other.”
In fact, my poll was biased — not because the question itself was slanted, but because of the way respondents were recruited: I titled the post “Are you homosexual?” Potential respondents who are homosexual or who don’t have traditional sexual preferences are more likely to be interested in the question, and therefore more likely to respond. How do I know this biased the sample? Because I collected similar data last week in the Casual Fridays survey about romantic gifts. In that survey, women reported same-gender partners 5.7 percent of the time, and men reported same-gender partners 3.7 percent of the time.
I posted the poll because I had seen a similar poll on Twitter: Bruce Wagner asked “How Gay is Twitter?” and linked to his own poll. I suggested Bruce’s selection of responses would be biased, and he challenged me to prove it. Here’s the evidence:
As you can see, in both the Cognitive Daily poll and the Twitter poll, significantly more respondents claimed to be homosexual (“mostly gay” or “gay”) compared to the Casual Friday survey. It all comes down to sample bias. It’s the same reason that Alfred Kinsey reported that ten percent of Americans were homosexual when other studies consistently find about a five percent homosexuality rate. He recruited respondents for a study on sexual behavior, which biased his sample.
So why did Bruce find an even higher proportion of homosexual responses than we did here? It could be that there are more gay people on Twitter, but I suspect it has more to do with Twitter culture. People on Twitter like to stay on Twitter. His poll requires respondents to click on a link to get to the poll, so only the most motivated respondents replied. By contrast, Cognitive Daily readers didn’t have to leave their comfort zone to respond. Even so, we still had significantly more responses of “gay” or “mostly gay” than in our Casual Friday study which mentioned nothing about sexual preference in the recruitment post — thus demonstrating that selection bias can be found even in seemingly innocuous polls.