The Illusion of Streaks

Someone should really tell the NCAA tournament television commentators that "the hot hand" doesn't exist. I've gotten pretty tired of hearing these tired cliches about Texas going cold, or Stephen Curry catching fire yet again. Never has a cognitive illusion gotten so much play.

The illusory nature of basketball shooting streaks was first demonstrated by Amos Tversky (of kahnemanandtversky fame) and Thomas Gilovich, a psychologist at Cornell. They began the investigation by sifting through years of Philadelphia 76er statistics. They looked at every single shot taken by ever single player, and recorded whether or not that shot had been preceded by a string of hits or misses. If "the hot hand" was a real phenomenon, then players should have a higher field goal percentage after making several previous shots. The streak should elevate their game.

So what did the scientists find? There was absolutely no evidence of "the hot hand". A player's chance of making a shot was not affected by whether or not their previous shots had gone in. Each field goal attempt was its own independent event. The short runs experienced by the 76ers were no different than the short runs that naturally emerge from any random process. Taking a jumper was like flipping a coin. The streaks were a figment of our imagination.

The 76ers were shocked by the evidence. Andrew Toney, the shooting guard, was particularly hard to convince: he was sure that he was a streaky shooter, and went through distinct "hot" and "cold" periods. (Toney is still regarded as a great clutch player. Charles Barkley has called him "one of the best kept secrets in the history of the NBA.") But the statistics told a different story. During the regular season, Tooney made 46 percent of all of his shots. After hitting three shots in a row--a sure sign that he was now "in the zone"--Tooney's field goal percentage dropped to 34 percent. When Tooney thought he was "hot," he was actually freezing cold. And when he thought he was cold, he was just getting warmed up: after missing three shots in a row, Tooney made 52 percent of his shots, which was significantly higher than his normal average.

But maybe the 76ers were a statistical outlier. After all, according to a survey conducted by the scientists, 91 percent of serious NBA fans believed in "the hot hand". They just knew that players were streaky. So Tversky and Gilovich decided to analyze another basketball team: the Boston Celtics. This time, they looked at free throw attempts, and not just field goals. Once again, they found absolutely no evidence of hot hands. Larry Bird was just like Andrew Tooney: after making several free throws in a row, his free throw percentage actually declined. Bird got complacent, and started missing shots he should have made.

Why do we believe in streaky shooters? The danger of random processesâ¯things like slot machines, stock markets and basketball shotsâ¯is that they are full of intermittent rewards that feel really good. The problem arises when we try to fit these stochastic events into a neat pattern, when we try to explain why the stock market went up 50 points, or why Stephen Curry just made three shots in a row. In both cases, the answer is the same: random chance.

I always assumed that there was one major exception to this psychology of streaks: Joe Dimaggio's 56 game hitting streak in 1941. Here is the late, great Steven Jay Gould, summarizing the evidence in 1988:

There is one major exception [to the hot hand rule], and absolutely only one--one sequence so many standard deviations above the expected distribution that it should not have occurred at all. Joe DiMaggio's fifty-six-game hitting streak in 1941. The intuition of baseball aficionados has been vindicated. Purcell calculated that to make it likely (probability greater than 50 percent) that a run of even fifty games will occur once in the history of baseball up to now (and fifty-six is a lot more than fifty in this kind of league), baseball's rosters would have to include either four lifetime .400 batters or fifty-two lifetime .350 batters over careers of one thousand games. In actuality, only three men have lifetime batting averages in excess of .350, and no one is anywhere near .400 (Ty Cobb at .367, Rogers Hornsby at .358, and Shoeless Joe Jackson at .356). DiMaggio's streak is the most extraordinary thing that ever happened in American sports. He sits on the shoulders of two bearers--mythology and science. For Joe DiMaggio accomplished what no other ballplayer has done. He beat the hardest taskmaster of all, a woman who makes Nolan Ryan's fastball look like a cantaloupe in slow motion--Lady Luck.

Sorry, DiMaggio. Looks like your streak was also a by-product of random chance. A new, more comprehensive statistical simulation has demonstrated that even a 56 game hitting streak will naturally emerge from the history of baseball:

Using a comprehensive collection of baseball statistics from 1871 to 2005, we simulated the entire history of baseball 10,000 times in a computer. In essence, we programmed the computer to construct an enormous set of parallel baseball universes, all with the same players but subject to the vagaries of chance in each one.

To tease out the meaningful lessons from random effects (fluky streaks that happen by luck), we redid the whole thing 10,000 times. In each of these simulated histories, somebody holds the record for the longest hitting streak. We tabulated who that player was, when he did it, and how long his streak was.

And suddenly the unlikely becomes likely: we get a very long streak each time we run baseball history. These results are shown in Figure 1. The streaks ranged from 39 games at the shortest, to a freakish baseball universe where the record was a remarkable (and remarkably rare) 109 games.

More than half the time, or in 5,295 baseball universes, the record for the longest hitting streak exceeded 53 games. Two-thirds of the time, the best streak was between 50 and 64 games.

In other words, streaks of 56 games or longer are not at all an unusual occurrence. Forty-two percent of the simulated baseball histories have a streak of DiMaggio's length or longer. You shouldn't be too surprised that someone, at some time in the history of the game, accomplished what DiMaggio did.