Cognitive Daily

Yesterday’s post offered a simple poll question:

# How Many People Will Select The Same Option As You?

0%

1-25%

26-50%

51-75%

76-99%

100%

I didn’t think this poll would attract as much interest as it did because in one sense I was just copying a poll from Slashdot. I guess people liked the fact that I offered one twist: I attempted to shape the results by suggesting that the “obvious” answer of 1-25% wouldn’t work. If everyone was choosing their responses randomly, we might expect about 16 percent of respondents to choose this option, and therefore it would be the “correct” response.

But surely some respondents would realize that if everyone was rational, then everyone would choose 1-25%, and it would no longer be “correct.” I didn’t spell it out in my post, but it only make sense that if everyone follows this pattern of logic to its rational conclusion, then everyone should choose 100%, and everyone will be “correct.” How many of our readers took the hint? The graph below shows how our readers compare to Slashdotters:

Clearly our readers were, at the very least, paying some attention to the hint. A much higher proportion of our readers selected 100%, and a whopping 42 percent selected 75-99% or 100%, compared to just 22 percent of Slashdotters.

Interestingly, the effect was large enough that those responding 26-50% in our poll are now technically “wrong” — just 25.9 percent chose that option, compared to 31.6 percent of Slashdotters (I removed the Cowboy Neal option from my analysis).

Some commenters, both here and on the Reddit thread about yesterday’s post, have suggested that if the poll was repeated several times, the real-world results would skew closer and closer to 100 percent. Now that’s something we can verify empirically — let’s try this again:

Comments

#1 Olaf Davis
January 29, 2008

“I removed the Cowboy Neal option from my analysis” – trouble is, if the Slashdotters were voting based on their assumptions about how many people would pick Neal, it introduces another possible explanation for differences between their poll and yours.
#2 Andrew G
January 29, 2008

Maybe results can be skewed towards 100% with many repetitions, but I can’t imagine anyone voting the 100% option itself would ever actually be correct. An equilibrium of 76-99% of voters voting the second to last option and 1-25% of voters voting the second option is much more flexible. All it takes is a single dissenter that doesn’t vote 100% and everyone loses.

I think this experiment would have dramatically different results if something were at stake – money, or even just “points” awarded for success over several repetitions. I would be surprised if even a single person voted 100% if they had something valuable to them at stake.
#3 Dave Munger
January 29, 2008

But what if there was a LOT at stake — like a million dollars. The 100 percent solution requires less negotiation than the other options.

What if no one got paid unless everyone was correct? I think you might see a very strong trend toward 100 percent. But I also think you’re right — someone still wouldn’t “get it.” But I also think the 75-99 / 1-25 equilibrium is problematic. How do you know if you should pick 75-99 or 1-25? What if 50 percent choose each option?
#4 Dan Armak
January 29, 2008

if the poll was repeated several times, the real-world results would skew closer and closer to 100 percent

At this point, isn’t there a serious risk that the responders are not primarily motivated by trying to choose the correct answer? Many may be trying to prove your point, or disprove it, or otherwise affect the results.

I think another data point is needed for comparison. How about a second poll asking “which answer to the first poll will be selected by the greatest amount of people?”
#5 KenBW2
January 29, 2008

Surely it’s an impossible task. If you thought that by probability 16.6% would choose your answer then (assuming everyone thought that) you’d get 100%. But then you’ve ticked 100% and… wait, I’m confused…
#6 Dunc
January 29, 2008

Is there any rational means of estimating how many people will give an irrational answer?
#7 Gordon Worley
January 29, 2008

Andrew, I think the question is if you have rational (Bayesian) agents, or actual humans. A Bayesian voting with only other Bayesians should of course choose 100% since that’s exactly what everyone else is going to do. With actual humans, however, we have the problem of a random dissenter. If the group is sufficiently large, the probability of a dissenter increases dramatically.

Consider these cases for dissenting. If everyone but the dissenter votes 100% and the dissenter votes 1-25% (I take this option to include nonzero percentages less than 1), the dissenter is the only one to give a correct response. If everyone but the dissenter votes 76-99, and the dissenter votes 1-25, then everyone is right (but it involves risk, so the Bayesians won’t choose it when everyone voting 100% guarantees that everyone is right, since the expected utility will slip to slightly less than 1). And, we can’t forget that humans are not Bayesian agents, so we have to expect irrational behavior, even irrational behavior that isn’t trying to be rational.

Some additional poll questions in this vein, but a little different. Use the same answer set as the poll in this post for each one.

Which will be the most popular?

How many will vote for choice X?

How many will vote for a choice above yours?

How many will vote for a choice below yours?

How many will not choose the same answer as you?

(Some of these may not have stable solutions, so be warned if you play with the mathematics.)
#8 Karl
January 29, 2008

Dave & Andrew: As I pointed out in my comments to the other post, the moment this would become an explicitly collaborative task (that is, a coordination problem to solve), 100% is the only reasonable option to choose. If something was at stake this should be more pronounced still, because 100% is the ONLY option that guarantees that you are correct AND everyone will reach this conclusion. (This doesn’t even necessarily assume collaboration, but it does assume that no one wants to ruin the chances of others.)

As this isn’t a collaborative task, people consider the possibility that other people will act irrationally and base their answer on that, and this skews the proportions away from 100%. However, this might well be the only reason for not picking 100%, that is, everybody who doesn’t pick 100% do it because they assume that some other people won’t as well, while everyone still realizes that 100% is the rational choice.
#9 Pat
January 29, 2008

So the percentage of slashdotters who got the correct answer is around 53% whereas only 10% of CogDaily readers got the correct answer!

That’s a bit disappointing!

> the “obvious” answer of 1-25% wouldn’t work.

Actually it was the only answer that worked!
Everyone who voted differently got a wrong answer – at least in the CogDaily poll.
#10 Pat
January 29, 2008

Actually if you round up the 25.9% who voted 26-50 are correct too so 36% of CogDaily readers go the right answer …

That’s better.
#11 Prateek
January 29, 2008

My reasoning for voting in the 76-99% category:
I assumed that about 1-25% of people will vote irrationally. The rest of the people, who will reason rationally (sounds redundant), will see that they must all vote in the 76-99% spot to be correct.
#12 Lagan
January 29, 2008

Did I miss something? About 55% of the Slashdot readers were ultimately correct in their choice. Even if we give CogDaily readers credit for the 26%-50% responses, only about 35% of us were correct in our choice.

Rather than focus on that, you dive deeper. Let me summarize: A greater number of CogDaily readers picked answers that could only be correct in the event that their conclusion and approach is the only way to think about the problem.

I can think of several words to describe that sort of world view, though “rational” was not the first to jump to mind.
#13 David
January 29, 2008

There’s no reason to assume that this is collaborative. Perhaps this is a competitive game. In which case I will continue to select 1-25 to sabotage the hundred percenters. Maybe I feel better knowing that I’m right, regardless of whether everyone else gets there.

I think the feedback will be much more complex. Each time, one looks at the results, compares one’s results with pass successful responses, and decides how other people are likely to vote. Based on your statements in the post, you implied that 100% is the best choice. I know everyone will not pick this choice (there’s always a spoiler, an outlier, a renegade), but many people will because your influence. So I would rather be the spoiler and pick 1-25, since the reward for being right (subjective as it may be) seems better than the reward for sacrificing myself to select the “best” solution. The categorical imperative is a weak argument in a web poll.
#14 James Bach
January 29, 2008

0% was the correct and rational choice the first time. It still is. Since most of you ignored my reasoning on the first go around– as I expected– I’m confident that 0% will continue to be the most precise and accurate choice.

However, I see that the webmaster is playing a little bit dirty. The question actually was: “How many people will choose the same answer as you?” yet when reporting the results, the webmaster wrote “How Many People Will Select The Same Option As You?”

Selecting an option is not the same thing as choosing an answer. I see “answer” as a bigger idea than option. I would expect these two questions to be conflated on a non-science blog, but here? I’m shattered.

Maybe you think I’m splitting hairs. Okay, but in social research, isn’t such hairsplitting about what people think instructions mean the difference between something like a CNN “Cafferty File” poll and research worthy of being taken seriously?
#15 James Bach
January 29, 2008

If you look at the middle of each bin, which one has a result that comes the closest to matching itself?

0% is in second place, by that measure.

13% is the middle of 1-25%, so that’s 4 points off. The current reading on “0%” is 5%, so it’s a close second.

By the way, the “answer I chose” was “clerisy”, and I expected that 0% of other people would answer that, which is why 0% is the most reasonable “option to select”.

All you who think 0% is an illogical answer… you should bone up on discourse analysis and critical theory.
#16 John
January 29, 2008

Rationally, 100% is the best answer. But we also know that people are irrational, which would automatically suggest that 100% is wrong. Reason isn’t always in step with reality. The people who didn’t pick 100% aren’t necesarily irrational, just sensible.
#17 Freiddie
January 29, 2008

Nearly everyone picks 0% because they don’t WANT your theory to be right.
#18 John Carter
January 29, 2008

Trouble with a Bayesian estimate is we have several populations here. True 100% static point believers, Cynics, Realists, Contrarians, Poll Hackers…

You first need a model of the size of each of those populations to come up with a true bayesian estimate.

Of course one could construct a nice Prisoners Dilemma out of this poll. Create a bin for every percentage point…
0,0-1,1-2,….,98-99,99-100,100

Let every respondee contribute a dollar.

Share the pot amongst those who where closest to the correct answer.

What we mean by “closest to the correct answer”?

If everyone selects 100%, everyone gets their dollar back.

If one person says 0% and the rest says 100%. That person is saying “nobody will pick the same answer as me” and in this case he will be right. The rest saying, “everyone will pick the same answer as me”, will be almost, but not exactly right.

Thus the guy who picked 0% wins the entire pot.

If everyone but one says 0%, and one person says 0-1% (an option not available on current poll I see), that one person gets all the dollars.

If the distribution is say p_i for each bin i, then a person picking bin i is saying i% of the people will pick bin i. Thus he is |p_i-i/100| away from the correct answer.

Thus everyone in bin i will be that far from the correct answer, and the other bins may perhaps be equally far. In which case they will share the pot.

The interesting thing is this game seems to reward the contrarians. Whatever rational basis you choose for coming up with the answer, it probably pays you to choose an irrational one!

Does there exists a choice which will not reward a contrarian?
#19 Paul
January 29, 2008

“rationality is a much broader term than logic, as it includes “uncertain but sensible” arguments based on probability, expectation, personal experience and the like” — Wikipedia article on rationality.

100% seems to be an *irrational* choice to me. Given the large number of players there is a high probability that someone will not choose 100 out of simple randomness. There is also a decent probability that a number of players will realize this and decide that 1-25%, 26-50%, or 51-75% is safer than 100%. One’s answer is rational if it is consistent with one’s expected probability of results. 100% is only a rational choice if you believe 100% of readers will choose it (not if you believe they should choose it). Given that Slashdot readers were correct more often, I think they are more rational.
#20 Dave Munger
January 29, 2008

Whoah, someone spammed the poll. It looks like they just put 10,000 votes for the 0% choice, so you can still get a sense of the results if you just subtract out 10,000 from that entry.

The 100 percent choice was at about 44 percent before the data was compromised, which indeed suggests that when people are prompted, they are more likely to trend to the 100 percent option.
#21 Stephen Downes
January 29, 2008

I find it interesting that a much larger number of Slashdot readers picked a ‘correct’ response (more than half the Slashdot readers were ‘right’ by my count) without any guidance than Cog Daily readers did with the guidance. There should be a lesson in that. Something like: people are more likely to get the right answer if you leave them along and don’t try to guide them to it.
#22 John Carter
January 29, 2008

Actually phrasing it a prisoner’s dilemma is lovely.

Can you find strategy A such if N people publically agree to follow it, their expected reward is greater than if a conspiracy of M people defect to any other strategy?

And if you can’t, is there any strategy for the defectors that is robust against betrayal by a conspiracy within the conspiracy…
#23 Celeste
January 29, 2008

The problem I see, is that now, I know that you want everyone to choose 100%. By telling what should happen you’ve incited rebellion. however, if I know that most people will choose %100, I can be right by choosing %1-25, or even %0 if the number of votes for %100 is large enough.
#24 Caledonian
January 29, 2008

Are you trying to predict the outcome of a chaotic feedback loop?

That’s not very rational.
#25 Lord
January 29, 2008

One can argue that everyone should choose 100%, but everyone also knows everyone won’t, so if you reject 100%, they why not reject the reasoning that led all the way to 100%? Soon by other arguments you end up with something close to random choice so 1/6 or 1-25% makes the most sense. But if most ended up with that conclusion it would be wrong.
#26 nitzan
January 30, 2008

let’s say that this was really for a big sum of money that would be divided only betweens the winners.

i say it wont be hard to find the one that wont resist the temptation of voting 0% after everybody else allegedly voted 100% to try and be the only winner.

indeed – the prisoner’s dilemma is very nice here.
#27 Brutha
January 31, 2008

100% isn’t rational. If I expect that everyone else votes 100% I am both right when I vote 0% and when I vote 100% -> the chance that I vote 0% is 50% and the chance that I vote 100% is also 50%.
That means if everyone will be wrong. If 1% sees that your logic is faulty and selects 1%-25% that bunch of people is right.
->voting 100% is really bad.
On the other hand you get there could be a Nash equilibrium is you assume that: 0,4 % of the people vote 0% and are right, 10% vote 1%-25% and are right, 35% vote 36-50% and are right and 54,6% vote 51%-75%.
->voting 75%-99% is also bad.
#28 Michael Chermside
January 31, 2008

Oh my, now THAT’s an interesting switch.

Now the person presenting the puzzle is ACTIVELY campaigning for people to pick a particular answer (100%). But the 100% answer has approximately NO chance of occurring — it only requires a single dissenter (Okay, 0.5% dissenters if you round to the nearest %.) That and 0% are VERY unlikely to succeed.

Anyhow, this time I guessed a LOWER field. I can’t predict just how high up the chart people will clump. My goal is _NOT_ to maximize the number of right answers in the whole population, my goal is to maximize the odds of my _OWN_ answer being correct. So I’ll take the 1-25% category: harming the group slightly to the benefit of myself.

I suppose not everyone has the same goal in mind… perhaps some had the goal of maximizing the number of correct answers without regard to the affect on their own answer? I’m guessing that when Dave posed it, HIS goal was to maximize the (negligible) chance that 100% of the responders would get it correct.

This raises some interesting questions. Suppose that you are Dave, and you’re about to present this for the THIRD time… what advice do you give? If your goal is to maximize the percentage of correct answers, then I’d suggest presenting all of the options and specifically requesting (in the instructions) that the users select the 76-99% category. If the number of people NOT following instructions is at least 0.5% (quite likely!) and less than 25% (reasonable) then you ought to succeed.

I’m not sure what Dave’s strategy is for certain other goals. For instance, if his goal is to maximize the chance that 100% get it correct, he’d better ask for the 100% category and ALSO try to discourage participation (there’s a certain irreducible percentage of contrary individuals and if that percentage exceeds 0.5% your best chance is to have none of those in your sample due to small sample size). If his goal is, say, to spread out the votes as much as possible, then that’s a bit trickier… I’m not quite sure WHAT he should say to achieve that goal.
#29 Laura
January 31, 2008

If most people vote trying to be in a winning category, I predict that after several rounds, there will be no winners. People will flock to the 1-25% and 26-50% categories as they’ve been successful in the past, pushing the percentages above 25% and 50% respectively. A few will stick with each of the other four categories.

And this isn’t necessarily irrational behavior – all winning or all losing still makes us all the same. And there’s much less reliance on other (unpredictable) people to get to an all losing outcome.

The only way to avoid an all-losing state is if over 25% sacrifice themselves to the 0%, 51-75%, 76-99% and 100% categories (which I predict will never be winning categories), allowing the remaining respondents to win in the 1-25% and 26-50% categories.

If these polls continue, I’m guessing that 25-50% will be a winning category for the next few rounds, and that it will be the last to become a losing category, so I’ll probably vote that category for the duration.
#30 Dave J.
February 5, 2008

I expect I’m too slow to comment and interest has died down by now but I can’t resist anyhow..
If you made this experiment one of a series then it’s an unavoidable shame for each to be followed by discussion; it can’t help but taint the later results. If it could be carried out under ‘experimentally clean’ conditions (and perhaps even if not) then I’d be curious to know the result of some alterations.

a) set it out as a competition

b) offer an edit box for the entrants to type their estimate, perhaps so that it’s to the nearest percent or perhaps even more interestingly

c) with it unrounded, so the entrants can include as many decimal places as they desire.

I think a short succession of the latter option (ie all three ‘tweaks’) would be my personal choice. Sadly it could never be ‘clean’ enough to satisfy anything over idle interest, there will always be deliberate stirrers. They just have to be seen as a part of the data.