The Quantum Pontiff

Can markets predict elections? Alea summarizes last night’s primary results: Ooops! From my perspective, I find the ideas of markets predicting future events fascinating, if for no other reason than my original motivation for studying physics was tied up deeply in questions about predicting the future.

I believe that, fundamentally, we cannot predict the future. Why? Not necessarily because of quantum theory (did I surprise you there?) and not because of arguments based on chaos theory (and I worked at the Santa Fe Institute ), but because of the locality of physics. In a world with local laws, local entities do not have access to information which will allow them to predict their future. Information in your past light cone is not enough to predict your future evolution. To slightly bastardize John Archibald Wheeler: space-time locality is what prevents anyone from knowing everything. For want of locality, we would be time travelers, able to predict our future with Laplacian certainty.

There are good reasons why markets can sometimes be efficient predictors. One central reason is that they aggregate information across multiple sources. But, just as in physics, information external to those who are betting in the market can become important and can ruin the predictive powers of the market. Last night, I think, information external to the market players and that influenced the election result was not available to the market. What was this information? My bet would be on people’s backreaction to misogyny like this (warning: clicking on that link may make your blood boil, or may have the opposite effect, depending on the color of your political underwear.), but in truth, I don’t know at this point what that information was.

But this is exactly what I find fascinating about last night’s results in the primary: unlike in other cases where the information was readily available far prior to the market being priced incorrectly (as in, say, a bubble market), it seems that last night was an example of the information simply not being available to the market players. The failure seems less from a bubble of speculation, and more from a fundamental property of markets as information processing entities. And it is for this reason that I don’t spend my days worshiping at the feet of the efficient market hypothesis, or thinking that markets for predicting terrorism have any chance at succeeding. While the former may have application as an approximation, I believe at a very fundamental level the efficient market hypothesis cannot possibly be true. And as for the later, since terrorism is (sadly) centered on secretly coming up with new evils to enact upon our collective security, I believe markets for predicting future terrorism are next to useless. Markets are local information processing entities with their own sense of locality (to be clear the speed of information flow in markets is not as clear cut as in physics, but it is there, nonetheless), and, just like the world we live in, this means that fundamentally we cannot rely on markets to predict the future, any more than we can use the laws of physics to predict our future. Now, if you don’t mind, I’ve got to go and check on the prices of equities I own.

Comments

#1 ch
January 9, 2008

I think that another reason why the markets have done poorly in predicting primary results is that the group of people who even bother playing in those markets (and it is play for them) bears very little resemblance to the general electorate. Before the primaries started, they let their own prejudices/preferences sway their judgment, as is indicated by the still overvalued price on Giuliani.

Now they seem to be overreacting to each event as the results come in. It will be interesting to see if this evens out as we get more and more primary results.
#2 John Sidles
January 9, 2008

Your “free-will” essay was IMHO very fine (so please consider this to be fan mail).

As for markets, my understanding is that one of the main ideals of the Enlightenment was that markets would support creative individual enterprise.

But alas … once folks realized that finding optimal market strategies is NP-hard, markets began an inexorable march toward exponential complexity … this has become a 21st century version of the “tragedy of the commons”.

It would be interesting to analyze a game in which the winning players were those who discovered the most intricate and powerful theorems …
#3 Dave Bacon
January 9, 2008

“It would be interesting to analyze a game in which the winning players were those who discovered the most intricate and powerful theorems … :)”

Mathematics Faculty Hiring, the Game (TM)?
#4 Geordie
January 9, 2008

John: In what sense do you mean that “finding optimal market strategies is NP-hard”?
#5 Dunc
January 10, 2008

Real futures markets do not even attempt predict the future – to say they do is a fundamental misunderstanding of what a futures market is all about. Markets set current values for the ability to hedge against perceived future uncertainty. They do not make predictions.

For example, if crude oil for December 2008 delivery is currently trading at 90.40, that does not mean that the market expects crude to sell for that price at that time. It means that buyers and sellers are prepared to trade at that price today in order to reduce their exposure to future price movements.

The question a futures market addresses is not “what will the future price be?”, it’s “who is prepared to pay money now to avoid having to worry about what the future price will be, and how much?”
#6 Dave Bacon
January 10, 2008

I agree Dunc. But there is certainly a belief floating around that markets are good predictors.
#7 John Sidles
January 10, 2008

Gordie sez: John,in what sense do you mean that “finding optimal market strategies is NP-hard”?

If you Google for ‘NP-Hard “market equilibrium”‘ you will find plenty of hits! The heuristic reason is simple: most interesting games are NP-hard to play optimally, markets are an interesting game to play, therefore playing markets optimally is very likely to be NP-hard.

The above is said tongue-in-cheek, but it is true too.

Dave Bacon is right that these considerations apply in academia — my original post was intended as a humorous comment upon this!
#8 Dunc
January 11, 2008

I agree Dunc. But there is certainly a belief floating around that markets are good predictors.

Oh, I know it’s there – I just don’t really understand where it’s come from, or why. It’s flat out silly to anyone with even a tiny sliver of understanding. Heck, I’ve barely got a tiny sliver of understanding of these matters, but I know what a futures market is for.

I find it really scary that people who are feted as having some sort of expertise can promulgate such obvious nonsense and get away with it.
#9 wolfgang
January 12, 2008

Dave,

> I believe that, fundamentally, we cannot predict the future. Why? … because of the locality of physics.

as you know, I completely agree with you on that. But there are two more important ingredients imho: gravitation (because it cannot be shielded) and the fact that in the past light cone we find ourselves…
#10 Kaleberg
January 13, 2008

Drat! I clicked on this link because I thought I had read “Markets Predicting Electrons”. That would be neat a approach to quantum physics.
#11 Kaleberg
January 13, 2008

Actually markets are terrible predictors. Again and again we see companies and industries far over the cliff with no apparent market reaction. Then, suddenly, someone looks down and the stock price plummets. Where is the predictive value? The event is in the past. In fact, the way to make money in the market is to react to events that no one else is reacting to and then waiting for someone to look down, or up. You make a lot less money if you react when everyone else does. Too many people forget the old trick, buy low and sell high.

Markets also have a long, multi-day settling time. If you want to make money on this, go for the “dead cat bounce”. Buy quickly after the bad news reaction and sell on the upward oscillation. There is an upside variation on this, but it involves short selling so it is much less efficient.

What markets are good at is setting prices. Prices allow people to estimate values. Values are useful for insurance, collateral and elsewhere in auditing and bookkeeping. If you don’t have a market to set prices, it is harder to figure out where to allocate your resources.
#12 Sanmay
January 17, 2008

Hmm, two key points seem to have emerged that I largely agree with:
1. Markets don’t create information out of nothing. They are (can be?) good aggregators of information that is out there, though. In elections, generally over the past few years, the prediction markets have seemed to follow public information rather than “lead” it.
2. Markets are good at setting prices. Without them we’d have no idea of what a company is worth, or of the world’s general belief that, say, Hillary Clinton will win the democratic nomination.

But neither of these means the markets got anything wrong in terms of the NH election (or any other) — sure there was a general perception that Clinton would lose, and the price the contract was trading at reflected that. This was the right price for it to be trading at, and assigning a probability of 25% that something will happen doesn’t mean that it won’t!

There are two separate questions. The first is whether prediction markets do a good job of aggregating information into probabilities. The second is whether the predictions of these markets are in general “good.” Assuming that the answer to the first question is yes, if the answer to the second question turns out to be no, then that basically means that even the collective intelligence of people isn’t great at predicting some future events. The right way to test question number 2 is of course to get a large sample of events that traded at some set of prices and see if they came out “right” the appropriate percentage of the time (many details would have to be taken care of, I’m thinking!)