The Frontal Cortex

New Hampshire and Political Punditry

Posted by Jonah Lehrer on January 9, 2008
(8)
More »

Needless to say, the political pundits were hilariously wrong about the New Hampshire primary. I won’t hypothesize about what actually happened, other than to say that I think many voters here wanted a longer primary. They didn’t want an Obama coronation in the beginning of January. This says less about Obama and Clinton and more about the over-hyped press coverage and shortened primary schedule. I voted for Obama, but I’m looking forward to a drawn out race for the Democratic nomination. This whole democracy thing is pretty entertaining.

But back to the failures of the political pundits. Nobody should be surprised. In 1984, the Berkeley psychologist Philip Tetlock began an epic research project: He picked two hundred and eighty-four people who made their living “commenting or offering advice on political and economic trends” – they were professional pundits – and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be re-elected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the experts were asked to rate the probability of several different possible outcomes. Tetlock then interrogated the experts about their thought process, so that he could better understand how they made up their mind. By the end of the study, Tetlock had quantified 82,361 different predictions.

After Tetlock tallied up the data, the predictive failures of most experts became painfully obvious. When asked to forecast the probability of a specific event happening, pundits tended to perform worse than random chance. A dart throwing chimp would have beaten the majority of well-informed experts. Tetlock also found that academic specialists – say, an expert in Middle Eastern affairs or a specialist on the New Hampshire primary – weren’t any better than the-man-on-the-street at predicting the future. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” Tetlock writes in Expert Political Judgment. “There is no reason for supposing that contributors to top journals–distinguished political scientists, area study specialists, economists, and so on–are any better than journalists or attentive readers of The New York Times in ‘reading’ emerging situations.” Furthermore, the most famous experts in Tetlock’s study tended to be the least accurate, consistently churning out overblown and overconfident forecasts. Eminence was a handicap.

So think about that the next time you watch those talking heads on CNN.

(8)
More »

Comments

  1. #1 Jillian
    January 9, 2008

    There is a great book that goes into this called The Wisdom of Crowds by James Surowiecki. He talks about how (and why) experts have such bad track records and comes up with some pretty interesting (and highly effective) ways of making good predictions. Reading it definitly changed the way I thought about and responded to the idiotic comments made by the talking heads during election years, that’s for sure!

  2. #2 Vnend
    January 9, 2008

    1984? Or 1994, when the questions listed make much more sense?

  3. #3 Daniel
    January 9, 2008

    When I was watching the funeral of Princess Dianna on TV, Tom Brokaw made the briliant observation, “Just think, the next time we witness a spectacle such as this, the queen will be dead.”

    I thought, “Yes, Tom, unless her son Prince Charles dies first, or her husband Prince Phillip dies first, or Princess Anne…” yada yada yada.

    Another time, in the recent past, when the Senate was divided 50/50 between the Republicans and Democrats, a reporter was commenting on the very advanced age of Senator Srom Thurmand, whom I believe was 100 years old. And she said, “Just think, if anything happens to Strom Thurmand, then the whole balance of power would be upset.”

    And I thought, “Yes, you’re right. If anything happens to ANY Senator, the whole balance of power would change.”

  4. #4 jonah
    January 9, 2008

    the study started in 1984, but the study went on for more than 15 years…sorry i wasn’t clear.

  5. #5 Scott Belyea
    January 9, 2008

    They went meta on the race and decided that they didn’t want to coronate Obama in the beginning of January.

    Arrrggghh!

    Sorry for the scream of pain, but your use of language is normally much more elegant. I mean – “went meta”? and (heaven forbid) “coronate”???

    But overall, looking at it from outside the US (from Canada, actually), I’m struck by what a bizarre process this is. Look at the % of the US voting population who participated in these two states, and the only rational reaction ought to be, “Who cares??”.

  6. #6 jonah
    January 9, 2008

    Scott, thanks so much for the writing tip. You’re absolutely right: that sentence was atrocious. (Can i blame it on lack of sleep? I was up late watching returns.) anyways, I’ve fixed the text.

  7. #7 tim
    January 9, 2008

    I didn’t watch or listen to any of the pundits. My brand new 46″ LCD flat panel was on mute. But I observed that statistically speaking the polls showed both Obama and Clinton in a dead heat taking into account the margin of error. And that is exactly how it turned out. Clinton only won the vote by 2 percentage points. But that doesn’t matter because in the end its not the popular vote that counts. Its the number of delegates and in NH they both received -9- delegates. Its not an all or nothing state. So all this talking and arguing over who won doesn’t matter unless you really buy the “momentum” bull that pundits love to flap about.

  8. #8 peggy
    January 9, 2008

    A friend of mine who teaches literature at a well-known university was once interviewed on television about Orwell (speaking of 1984!). In reference to a question about Orwell’s semi-autobiographical work, he said “Down and Out in Paris and London is about …. being down and out in Paris and London.”
    I laughed out loud when I heard it. In his defense, it turned out that the interview was edited and something he said in between ended up on the cutting room floor. But on the screen, this was barely perceptible except as an ever so slight pause. He just sounded like a tautologist.
    When I listen to the pundits on any number of subjects–but usually on politics or the economy–I often hear what amount to tautological statements. Talk about having mastered the art of stating the obvious. Either that or, as this study suggests, making wild predictions. Someone should invent a system for tracking the pronouncements of the main pundits of the day and giving us the score every now and again. Maybe their track record could actually appear graphically on the screen below their talking heads, the way the stats of athletes are.