The futility of predicting unlikely events

When Joanne Rowling sat in an Edinburgh coffee shop, nearly broke, her baby sleeping nearby in a stroller, penning a fantastic story about a school for wizards, could anyone have predicted that she would soon be the most successful novelist in history?

Certainly not the twelve publishers who rejected her manuscript before Bloomsbury finally offered her a trifling £1,500 advance for a work that would ultimately become the basis for a billion-dollar publishing empire.

Probability expert Nassim Nicholas Taleb believes that our inability to predict blockbuster successes like Harry Potter is mirrored by an equivalent deficiency in predicting catastrophic disasters like World War I. Yesterday he participated in a fascinating interview on NPR where he discussed his theory and its potential applications.

Taleb argues that history books make up reasons for events that are by their very nature improbable. If someone had sat in a coffee house in Vienna in 1913 and related the history-book explanation of the situation in at that time, explaining that Europe was on the brink of an unprecedented continent-wide war, Taleb claims, he would have been carted off as a lunatic.

Despite the inherent unpredictability of events like this, Taleb argues that there are ways to manage that uncertainty to our advantage, but it requires an understanding of basic psychology. "We're suckers for a narrative," he says. When a story is attached to an event, it seems more probable than it actually is, causing us to err on the conservative side for negative events, and on the risky side for positive ones.

He also mentions the propensity to risk more to avoid losses than to make gains. This can explain why publishers are so quick to reject even promising works by unproven writers. Taleb suggests that publishers should be offering more contracts to writers because that small risk can have such a vast reward.

Overall, a fascinating interview, and I highly recommend listening to the whole thing. (Beware, however -- it crashed Firefox the first time I tried to listen.)

Tags

More like this

Psychiatrists of old, never gave advice.  But here's some advice: And never trust with your money anyone making a potential bonus. From href="http://www.ft.com/cms/s/0/fa89be08-02aa-11de-b58b-000077b07658.html">Nassim Nicholas Taleb, author of Black Swan; HT: href="http://theautomaticearth.…
Or: Deconstructing Dumbledore. (Major and serious spoilers for Half-Blood Prince and Deathly Hallows follow. In fact if you haven't read them this will make very little sense.) There's a lot of villains in the Harry Potter series. They are young and old, male and female, human and otherwise,…
JK Rowling, Author of the Harry Potter book series. Harry Potter and the Deathy Hallows, the last book in the popular Harry Potter book series, is due to be released on 21 July 2007, according to JK Rowling, the series' author. Bloomsbury, her British publisher, said it would publish a children…
tags: Harry Potter, JK Rowling, charities, prequel From the prequel I am not working on -- but that was fun! JK Rowling 2008 At the request of UK bookseller, Waterstone's, JK Rowling recently wrote an 800-word outline for a Harry Potter prequel, all to benefit the British charity Dyslexia Action…

It ties in with your previous post. Living a happy prosperous life among optimistic people in Belgrade, I did not in May 1991 (and nobody around me either) expect there to be a war. We were all richer than ever before in our lives, starting businesses, buying land and looking forward to the future. We fed each other's optimism. The war broke out in June 1991.

Taleb's an interesting guy, along the lines of the previous comment, he came from what was a prosperous family in Lebanon prior to the civil war there.

It ties in with your previous post. Living a happy prosperous life among optimistic people in Belgrade, I did not in May 1991 (and nobody around me either) expect there to be a war. We were all richer than ever before in our lives, starting businesses, buying land and looking forward to the future. We fed each other's optimism. The war broke out in June 1991.

You probably didn't concern yourself with the very real social calamity that was created with the loss of NATO's mission in 1989.

Despite the inherent unpredictability of events like this, Taleb argues that there are ways to manage that uncertainty to our advantage, but it requires an understanding of basic psychology. "We're suckers for a narrative," he says. When a story is attached to an event, it seems more probable than it actually is, causing us to err on the conservative side for negative events, and on the risky side for positive ones.

We could also actively manage the uncertainty to our advantage given enough access to media and power. Example: although we spend more on defense than all the other countries put together, we still feel insecure. I'd be surprised if there isn't an industry or two behind that providing suitable narrative.

Unfortunately, the premise is crap. People could and did predict before 1914 that there was going to be a war. What do you think they were having an arms race, drawing up hideously complex mobilisation plans, and training millions of conscripts for?

Bismarck said not long before he died in 1890-odd that "the great European war will begin over some damn fool thing in the Balkans". Jan/Ivan Bloch commissioned a whole squad of international military experts to write his study that predicted...well, precisely a long, unprecedentedly awful industrial massacre that would only end when whole nations broke under the strain. Sir Henry Wilson, Jackie Fisher, and Winston Churchill repeatedly predicted that there would be a war with Germany *and the Germans would march through Belgium and down to Paris from the northwest*.

The mention of JK Rowling is interesting. The NY Times had an article yesterday about something called MediaPredict.com. This group is asking people to predict which of a group of book proposals will be the most popular, and Simon and Schuster may publish it. They provide $5000 fantasy, with which you buy shares in the proposals you think will make it. Note that they don't ask for which ones you like best, but which will be the most successful.
As a reader, I find this gives me the willies. As a scientist, I'm interested in it, both the process and the two sets of results. On the whole, though, I'm sad about it.

A very fascinating interview with Taleb on NPR as you indicated.

My sense is that Taleb's research is strongly connected to research on hindsight bias, which refers to people's exaggerated sense that an outcome (e.g., a Black Swan event such as 9/11) was inevitable. The comment that we are suckers for narratives in interpreting history is representative of a hindsight bias phenomenon.

Sanna and Schwarz (2006) have recently written that hindsight bias involves an interaction of thought content (e.g., thinking about possible alternatives to an event reduces hindsight bias) and metacognitive experiences (e.g., the extent to which an event is surprising reduces hindsight bias). In relation to the previous posting about how to make an opinion popular, the authors also note that hindsight bias is increased by constant repetition of information. As an example concerning the 9/11 attacks, they write:

"....as people strive to understand what happened, the search for explanations makes potential causes highly accessible, resulting in the conclusion that it could have been foreseen and should have been prevented--possibly leading to more blaming than actually working to further defense. Public discouses following 9/11 is certainly consistent with this conjecture. Media coverage may further change the metacognitive experiences associated with events. For example, frequent repetition of key event scenes in the news may contribute to hindsight bias by rendering the event more fluent, and this may have far-reaching implications for public opinion, calls for relevant policy, and individual coping strategies. " (Sanna and Schwarz, 2006, p. 176).

Reference

Sanna, L.J., & Schwarz, N. (2006). Metacognitive experiences and human judgement: The case of hindsight bias and its debiasing. Current Directions in Psychological Science, 15, 172-176.

By Tony Jeremiah (not verified) on 22 May 2007 #permalink

Maxine's comment about MediaPredict.com is interesting. There seems to be evidence that this type of bidding process can be a good method of predicting the outcome of an event.

Back around 2001, the CIA setup a website for trading "terrorism futures" (I kid you not, I actually visited the site) The premis was that the "price" of the terrorism future would be an indicator of the current risk of an 'event' occurring. Whatever, there was an immediate public outcry and the website was removed the same day.