Are failures good for scientists - and everyone else?

I ran across first one, then two articles on the scientific benefits of failure in the past few weeks: this one by Jonah Lehrer for Wired (originally out in December), and this one by W. Barksdale Maynard for the April issue of the Princeton Alumni Weekly. Odd - or maybe not so odd; with the economy crashing down, oil rigs blowing up, etc., it's awfully reassuring to reflect on the good side of screwups.

Lehrer's piece begins by debunking an overly romantic and simplistic view of science, in which experiments come out the way you expect (ha):

Dunbar came away from his in vivo studies with an unsettling insight: Science is a deeply frustrating pursuit. Although the researchers were mostly using established techniques, more than 50 percent of their data was unexpected. (In some labs, the figure exceeded 75 percent.) "The scientists had these elaborate theories about what was supposed to happen," Dunbar says. "But the results kept contradicting their theories. It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense.". . .

The scientific process, after all, is supposed to be an orderly pursuit of the truth, full of elegant hypotheses and control variables. (Twentieth-century science philosopher Thomas Kuhn, for instance, defined normal science as the kind of research in which "everything but the most esoteric detail of the result is known in advance.") However, when experiments were observed up close -- and Dunbar interviewed the scientists about even the most trifling details -- this idealized version of the lab fell apart, replaced by an endless supply of disappointing surprises. There were models that didn't work and data that couldn't be replicated and simple studies riddled with anomalies. "These weren't sloppy people," Dunbar says. "They were working in some of the finest labs in the world. But experiments rarely tell us what we think they're going to tell us. That's the dirty secret of science."

Uh - that"s a secret? Any undergrad who's done time in enjoyed the unique ambiance of a research lab should realize that the experiment that goes just as planned is the anomaly, not vice versa! Fortunately, Lehrer's article goes on to talk about something a little more complex: the (nascent) psychology of how we react to frustration, and why it can stymie researchers. He argues that talking to others - especially those outside your immediate field - is one of the best ways to get past experimental results that just don't make sense. Score one for interdisciplinary discourse! And Maynard's article catalogs some of the delicious fruit of those unexpected results, through interviews with a number of Princteon faculty:

history shows that breakthroughs often spring not from carefully laid plans, but from mischance or even sheer, ridiculous accidents. A stovetop spill Âheralded vulcanized rubber; the potency of uranium was revealed when a rock was left in a drawer among photographic plates. And great research seldom follows an unswerving path. At RCA in Princeton in the 1950s, David Sarnoff exhorted his team to invent a flat television that could hang on a wall. "There were an enormous number of failures," says Princeton historian of science Michael Gordin -- and instead of TVs, the world got the Seiko digital watch in 1973.

With hindsight, if I had my heart set on a flatscreen TV, I would not be thrilled with a digital watch. But the idea that serendipity and surprise and failure combine to generate value is one of the commonly cited reasons for funding basic research: because you can't predict when or where a discovery with significant practical utility will materialize. In science, serendipity and failure are two sides of the same coin - and we've all been on the wrong side before. Fortunately, knowing there's another side is what keeps researchers going.