The Wisdom of a Crowded Individuals

A lot of people have read The Wisdom of Crowds by James Surowiecki. In the book, he gives an example of a group of people forced to estimate the weight of a cow. (This was actually an experiment that geneticist Franics Galton attempted.) When you do this, you find that the accuracy of the average response from the group is much greater than the accuracy of each individual estimate.

This is the so-called wisdom of crowds. The assumption with experiments like this is that each individual will make the best guess they can and this guess will be stable over time -- minus any new information acquired in the interim. However, two psychologists have discovered that this assumption isn't correct:

Until now, psychologists have assumed that when people make a guess, they make the most accurate guess that they can. Ask them to make a second and it should, by definition, be less accurate. If that were true, averaging the first and second guesses should decrease the accuracy. Yet Edward Vul at the Massachusetts Institute of Technology and Harold Pashler at the University of California, San Diego, have revealed in a study just published in Psychological Science that the average of first and second guesses is indeed better than either guess on its own.

The two researchers asked 428 people eight questions drawn from the "CIA World Factbook": for example, "What percentage of the world's airports are in the USA?" Half the participants were unexpectedly asked to make a second, different guess immediately after they completed the initial questionnaire. The other half were asked to make a second guess three weeks later.

Dr Vul and Dr Pashler found that in both circumstances the average of the two guesses was better than either guess on its own. They also noticed that the interval between the first and second guesses determined how accurate that average was. Second guesses made immediately improved accuracy by an average of 6.5%; those made after three weeks improved the accuracy by 16%.

(The paper itself does not appear to be online yet.)

This result suggests that guesses aren't stable over time. By virtue of the additional guess, the average regresses towards the mean of the real value, and you get improved accuracy. It is almost as if the single individual is showing the same effect that groups have for improved accuracy.

I think that there are two explanations for this.

First, you could say that the participants got additional information between the first and second guess, and this improved their estimate. However, this hypothesis has difficulty explaining the improvement in accuracy from the second guess taken immediately afterwards. Maybe the authors prompting the participants to make additional guesses made them think harder about the problem improving their accuracy. It is a little difficult to tell.

Another interesting idea is that the individual guesses are drawn from a distribution. Rather than having a discrete number that you think the answer is, you have a distribution of answers from which the answer you give is drawn. This accounts better for the absence of stability in your answer, but it doesn't necessarily account for the improvement in guesses over three weeks. Perhaps the acquisition of additional information over time causes you to modify and improve your distribution rather than your answer.

It may seem a bit odd, but we see something similar to this idea of a distribution of responses in the lab working with rats. Say I train a rat to go right in a forced choice between right and left. One, the rat doesn't always go right, even if I have trained them for days and days. This is because the rat has an innate desire to forage and check out novel locations, and it is willing to forgo a reward on a given trial to do so. Therefore, it would be reasonable to say that whatever the probability a highly rat has of turning right is, it isn't 100%. Two, say I change the rewarded arm to left instead of right. What you see is a slow change in behavior between the two arms. The rat might try the left, then go back to right for a couple, then left and so on. Eventually, the rat will now go left every trial, but only after many trials.

The question when you are observing this is what generates this variation in behavior. Why doesn't the rat immediately go to the other arm all the time when the rewarded arm is changed? Does the animal have a hidden variable that represents its probability of turning right on any given trial? Does the improvement in behavior represent the change in this probability?

Other authors have suggested that we should model learning as the change in a probability of giving the right answer to increase the likelihood of receiving a reward. These methods capture the intrinsic randomness that is present in lots of animal behaviors -- you can never expect them to get it perfectly right -- and applies it to models of learning.

Maybe in human responses -- as well as animals -- what we have is a distribution of responses rather than a set response. This would mean that if you want to give the right answer on a test, you should collect multiple answers from yourself -- hopefully separated by a reasonable period of time. It sounds odd, but it just might work.

Hat-tip: Mind Hacks