A couple hours ago I posted a quick poll, in what might be construed as an unbiased fashion. I simply asked respondents for their sexual orientation, offering a wide array of choices ranging from "straight" to "mostly gay" to "gay" to "other."
In fact, my poll was biased -- not because the question itself was slanted, but because of the way respondents were recruited: I titled the post "Are you homosexual?" Potential respondents who are homosexual or who don't have traditional sexual preferences are more likely to be interested in the question, and therefore more likely to respond. How do I know this biased the sample? Because I collected similar data last week in the Casual Fridays survey about romantic gifts. In that survey, women reported same-gender partners 5.7 percent of the time, and men reported same-gender partners 3.7 percent of the time.
I posted the poll because I had seen a similar poll on Twitter: Bruce Wagner asked "How Gay is Twitter?" and linked to his own poll. I suggested Bruce's selection of responses would be biased, and he challenged me to prove it. Here's the evidence:
As you can see, in both the Cognitive Daily poll and the Twitter poll, significantly more respondents claimed to be homosexual ("mostly gay" or "gay") compared to the Casual Friday survey. It all comes down to sample bias. It's the same reason that Alfred Kinsey reported that ten percent of Americans were homosexual when other studies consistently find about a five percent homosexuality rate. He recruited respondents for a study on sexual behavior, which biased his sample.
So why did Bruce find an even higher proportion of homosexual responses than we did here? It could be that there are more gay people on Twitter, but I suspect it has more to do with Twitter culture. People on Twitter like to stay on Twitter. His poll requires respondents to click on a link to get to the poll, so only the most motivated respondents replied. By contrast, Cognitive Daily readers didn't have to leave their comfort zone to respond. Even so, we still had significantly more responses of "gay" or "mostly gay" than in our Casual Friday study which mentioned nothing about sexual preference in the recruitment post -- thus demonstrating that selection bias can be found even in seemingly innocuous polls.
- Log in to post comments
The results of these types of polls are usually skewed by 1. the nature of the question asked and 2. the degree of anonymity allowed for responders.
Your poll (biased, as you admit), seems to prove the OPPOSITE of the theory you stated...
You seem to be assuming that the question "Are you homosexual" biased the poll but that "Romantic gifts" didn't. I responded to the gifts poll and I read the results, and I had assumed that from the way the question was phrased and the results presented that it was addressed to heterosexuals. I obviously must have missed the male-male or female-female results, but at the same time something in the way the poll was presented led me to think that way. In fact I thought it was a fun study with interesting results, and I posted about it on my blog with the comment that it would be interesting to compare the results with same sex preferences and assumptions. So I'm wondering if perhaps the bias occurs the other way as well and that a smaller number of people in same sex relationships responded to that poll than would be representative.
Most of the gay men I know are not particularly interested in the whole "romantic gifts" nonsense. I skipped past those entries without reading them.
Now that I've looked at the romantic gifts poll, it's clearly about heterosexual culture, it seems pointless for me to answer it.
Bruce:
The point of the poll was to see if we could introduce selection bias.
Last week we asked the same question in a generic, non-biased way -- by asking for respondent's gender and their partner's gender in an unrelated survey that didn't mention sexual orientation at all. Today I asked the question in a poll specifically mentioning the term "homosexual." The question is, would this phrasing attract more homesexual people, thus introducing a selection bias.
Comparing our results from earlier today with last week's survey, it's clear we did.
Even if Felix was correct and our survey disproportionately did not select gay men, we need only look at the results for gay women, and again we see that significantly more respondents to today's survey indicated they were gay than did last week. That's selection bias.
Your poll had an even more dramatic selection bias because respondents had to do more to answer your poll -- thus only the most motivated individuals responded.
One more thing: do the "bisexual" and "other" choices negate our findings? Absolutely not. If anything, they only dilute the pool of responses. Despite having additional choices, significantly more people chose "gay" as their response than those who indicated their partner was the same sex last week.
A little point (and possibly moot -- I didn't take the "romantic gifts" poll, so I don't know): was the earlier poll biased toward those who are partnered? If, for some reason, a higher proportion of straight people (or gay people) were unpartnered, would that bias the results? What about bi people who happen to be partnered with an opposite-sex partner?
> reported same-gender partners
This doesn't prove orientation. Bad poll, sorry.
>>This doesn't prove orientation. Bad poll, sorry.
But you could equally say that actually having a same-gender partner is the only meaningful measure of orientation and that just asking people "Are you gay?" is a bad poll.
In reality, neither is ideal. You ought to ask both (and lot more) if you're serious.
Since in this case the poll was just designed to prove a point (and it did so very successfully, sorry Bruce) it's all good.
Yes, I'm with those who'll say that it's a big assumption (one I'm surprised you let yourself make) to say that the earlier poll wasn't unbiased.
But the Twitter poll is obviously way off too. All these polls are skewed in one way or another.
Kinsey's research never found that 10% of men were gay; in fact he found that 4% of men were more-or-less exclusively homosexual. The 10% figure was the result of cherry-picking by activists: it represented the number of Kinsey's subjects who had homosexual, but not heterosexual, experience in any three-year period when they were 16 or older. The problem is that that group is going to include a lot of people who had incidental homosexual experience in their late teens before losing their heterosexual virginity, and most of those people would be regarded as heterosexual for most of their adulthood.
Related article:
Savin-Williams, R.C. (2006). Who's gay? Does it matter? Current Directions in Psychological Science, 15, 40-44.
Abstract
To answer the question âWho's gay?ââand its logical follow-up, âDoes it matter?ââresearchers usually define homosexuality with reference to one of three components or expressions of sexual orientation: sexual/romantic attraction or arousal, sexual behavior, and sexual identity. Yet, the three components are imperfectly correlated and inconsistently predictive of each other, resulting in dissimilar conclusions regarding the number and nature of homosexual populations. Depending on which component is assessed, the prevalence rate of homosexuality in the general population ranges from 1 to 21%. When investigators define the homosexual population based on same-sex behavior or identity, they enhance the possibility of finding a biological basis for homosexuality and a compromised mental health (suicidality).
Funny that this was your goal. Not sure if you saw my comment at the polldaddy site, but here is what I said:
Seeing this blog post in my RSS reader, I quickly skipped over it. But, then I realized a funny thing: if I was gay, I would have undoubtedly stopped and voted. I'd want to make the point, "Heck yeah! I am gay!" What does that tell you about psychology? [It certainly tells you something about the reliability of online polls, but that's nothing new. :-)]
It turns out that Bruce Wagner is just a spammer and a scammer, as his suspension from Twitter clearly illustrates:
RT @mangaaa @brucewagner had an army of fake Twitter accounts which retweeted on command, http://is.gd/jylH Clever yet unethical!
http://twitter.com/mangaaa/statuses/1212303333