It's one of the more annoying side-effects of the financial collapse: instant updates of the Dow Jones Industrial Average are suddenly everywhere, popping up in the corner of cable news shows, in between weather reports on the radio, highlighted on websites, etc. It's a bizarre form of financial melodrama, as the moods of the market seem to lurch and pivot for no good reason. Yesterday afternoon, on the same day a terrible unemployment report came out, the Dow swung upwards and closed 500 points higher. This morning, it's down 350 points, although nobody seems to know why. This chart captures the recent flux:
While it's certainly entertaining to spin post-hoc explanations of market activity, it's also utterly futile. The market, after all, is a classic example of a "random walk," since the past movement of any particular stock cannot be used to predict its future movement. This inherent randomness was first proposed by the economist Eugene Fama, in the early 1960's. Fama looked at decades of stock market data in order to prove that no amount of rational analysis or knowledge (unless it was illicit insider information) could help you figure out what would happen next. All of the esoteric tools and elaborate theories used by investors to make sense of the market were pure nonsense. Wall Street was like a slot machine.
Alas, the human mind can't resist the allure of explanations, even if they make no sense. We're so eager to find correlations and causation that, when confronted with an inherently stochastic process - like the DJIA, or a slot machine - we invent factors to fixate on. The end result is a blinkered sort of overconfidence, in which we're convinced we've solved a system that has no solution.
Look, for example, at this elegant little experiment. A rat was put in a T-shaped maze with a few morsels of food placed on either the far right or left side of the enclosure. The placement of the food is randomly determined, but the dice is rigged: over the long run, the food was placed on the left side sixty per cent of the time. How did the rat respond? It quickly realized that the left side was more rewarding. As a result, it always went to the left, which resulted in a sixty percent success rate. The rat didn't strive for perfection. It didn't search for a Unified Theory of the T-shaped maze, or try to decipher the disorder. Instead, it accepted the inherent uncertainty of the reward and learned to settle for the best possible alternative.
The experiment was then repeated with Yale undergraduates. Unlike the rat, their swollen brains stubbornly searched for the elusive pattern that determined the placement of the reward. They made predictions and then tried to learn from their prediction errors. The problem was that there was nothing to predict: the randomness was real. Because the students refused to settle for a 60 percent success rate, they ended up with a 52 percent success rate. Although most of the students were convinced they were making progress towards identifying the underlying algorithm, they were actually being outsmarted by a rat.
So don't listen to those talking heads telling you why the market rose or fell. They're just like those Yalies, convinced they've found a pattern where none exists.
- Log in to post comments
Loved that study! And couldn't this be applied to politics as well as the stock market? Everyone is thirsting to predict the future in both cases, and the randomness of daily polling and daily market activity feels like it should be tied to real events. Was it Sarah Palin that crashed the McCain campaign or Lehman Brothers? It's as if only we could get enough data and know enough about all of the innumerable potential influences, then we'd be able to generate a Unified Theory of how to win an election or beat the stock market.
The big question for me, as a neuroscientist, is whether this applies to the brain as well. If we just put enough ion channel and anatomical data into the Blue Brain project, will we really be able to build a Unified Theory of consciousness?
Or should we all just get a little more comfortable with uncertainty--like the rats?
Looking for the rat story, I found this post which argues that the yalies got different information: primarily that they would have gotten food from the other arm of the T, while the rat only information that there is no food when it chooses badly.
According to the not yet materialized dream of Friedrich Hayek of making possible the transition from the realm of the sensory order to the realm of social order, the brain is similar to a financial market.
Waves of electrical activity in the brain equals waves of tantalizing ups and downs of the market
There are many scientists in the science of the complexity or graph theory that shares this view.
I m not so sure about it, because this has political implications.
For them the brain has an spontaneous order not a central commander or mechanism of regulation, if we translate this to the workings of economy...
Thanks so much for that link, Dave! It's a really interesting take on the experiment I describe, which I first learned about in Tetlock's book. I'm headed to the library today, so I'm going to check out that book on learning and get back to you. But thanks for the update.
I would have to conclude that Yale students are like pigeons. There is a famous experiment where pigeons were given food through a device, the food arrived one morsel at a time in a series of successive fixed intervals (say once every 3min). However the exact time it arrived during that interval was random.
Each pigeon quickly developed an individual "dance" (pecking at the slot, spinning round, etc) in an effort to influence the timing of the reward. Occasionally the pigeon would do something and the reward would just happen to arrive so the pigeon would incorporate it's last movement into the dance. The dance developed into an intricate routine that lasted long enough to give a 50/50 chance of coinciding with the reward.
Like tribal rain dancers (or Yale students), once the pigeons had it in their head that the dance "worked" they would repeat it for long periods of time without reward. Simarly the pragmatic rat found a right turn "worked".
Finally when you think about it we all perform rituals we don't understand for the simple pragmatic reason that they "just work".
this post is excellent. thnx.
While it isn't possible to predict whether the market will go up or down on any day, it is possible to predict which stocks will do better or worse then it over the long run.
Peter Lynch and Warren Buffet--- among others--- have basically shown such high consistent returns that they effectively couldn't have achieved through chance. And they've done this using publicly available information.
Brilliant.
"While it isn't possible to predict whether the market will go up or down on any day, it is possible to predict which stocks will do better or worse then it over the long run."
If that were true, mutual fund managers would do it. They can't, so it's not. They guy who wrote "A Random Walk Down Wall Street" used to challenge professional stock pickers to beat a portfolio he picked using the stock pages and a dartboard. I don't think anyone's willing to take him up on it any more.
The law of large numbers means there will occasionally be a Buffet, but it's a rarity. Berkshire Hathaway has taken a beating of late anyway.
I don't agree with all aspects of the argument. Searching for patterns, algorithms, etc. is an important part of the scientific process, as is failure. It is easy to call the students foolish when we know the answer involves no pattern, only randomness. It seems the biggest fault of the students is not knowing when to accept there was no pattern, but again, it is hard to lay blame here as well. At some point man realized the world isn't flat, the sun doesn't revolve around the earth, ocean currents do have patterns that impact weather and boating, and diseases do have cures if we try enough experiments. Had we given up too soon, would we have missed an important discovery...or worse, have we already given up too soon on something important we'll not know for years?
The thing that will ultimately separate the Yale student from the rat is that they will strive for better than the rat-standard 60%, and more importantly, some will achieve it.
(Incidentally, I attended two crummy Pennsylvania state schools, so I have no general stake or opposition for laughing at Yale students.)
The stock market argument reminds me of one of my favorite movies, Pi, in which the plot is the desire for one mathematical genius to find the pattern behind the daily ups and downs. It is, of course, fiction, and the fact that he found a 216 digit code that not only controlled the market but also happened to translate in Hebrew to the true name of God is probably unlikely in real life (that didn't really spoil much, if you are interested). But there is some interesting mathematical philosophy discussed in the film, most notably in that just because we aren't sophisticated enough to see a pattern or recognize a system doesn't mean one isn't out there (sorry for the double negative).
The stock market IS based on real events; confidence is founded on many real life factors, and on a minute, individual level, there are people who recognize a good investment over a bad with better than rat-standard accuracy. The sum of all its parts is, for all intents and purposes, random, but individually the parts are very sophisticated.
Beyond the 24 hour news stations what report fluff to fill up the 23.5 hours where no real news actually occurs, I suspect there are people who believe there are patterns that can be found in the parts that make the market predictable. An exact pattern to find - highly unlikely; but I wouldn't disregard the potential for a better understanding than what we have today.
"rat-standard accuracy."
Ooh, that one's a keeper.
Alan,
That pigeon study you mentioned reminds me of a young adult's book by an amazing author named William Sleator. The book is called "The House of Stairs".
http://en.wikipedia.org/wiki/House_of_Stairs_(William_Sleator_novel)
Following up on Dave X's comment -- the interpretation given here of the rats-vs.-yalies experiment is a good story, but unfortunately it's a sort of a scientific urban legend.
The details are given in the weblog post that Dave X cited, and which I wrote -- "Rats beat Yalies: Doing better by getting less information?". In a nutshell, the behavior of both species is accurately predicted by an elegant and simple model of "expected rate learning", given the facts of their (different) experiences in the context of the experiment.
The story about "the allure of expectations" is probably a post-hoc fable; at least, Occam's Razor would suggest discarding it, given that a much simpler and much more broadly-supported model predicts the facts nicely.
I'm about halfway through "Fooled By Randomness" and I will keep this post in mind as I finish it.
Years ago, I was told that the ability to perceive a pattern was a sign of intelligence. What is it a sign of to perceive patterns that aren't there?
I agree with you that the media's "explanation" of stock market moves are random, but not the movement of the stock market itself. The media grasps at anything to explain the moves because the average person cannot accept it if the reporter said that the stock market sold off today because there were more sellers than buyers. The stock market on any given day is a reflection of human behavior and emotion. I believe that "most" humans are like rats and are motivated predictably by fear and greed. There are many successful traders that profit from this daily.
the stock market isn't a slot machine, it's just that previous performance has no bearing on future performance. Those two statements are very different. It still matters what a companies growth prospects look like to their individual valuation, and that valuation does impact things like the DOW.
It is certainly true that investor panic, speculation, or just plain noise can easily overwhelm the underlying information, but that doesn't mean that the information isn't there or influential. At that point, you really just talking signal to noise ratios.
Your point about humans needing to find a pattern, even if they are given noise is still quite on the mark however.
great post thanks
The movement of any stock in the short term is random, and day-trading is no different from a slot machine.
However, stocks aren't just pieces of paper-- they have companies behind them, and the success or failure of those companies drives the LONG TERM price. It sometimes is possible to see that one company has a better business strategy than another-- not with 100% accuracy, but better than the rat's 60%.
You can't lump in the short-term gyrations of "the market" with the long-term performance of the companies which make it up.
Turkiyeni en iyi online video sitesi