As I endlessly repeat, I'm an experimentalist by training an inclination, so I especially appreciate stories about experimental science. There's something particularly wonderful about the moment when an experiment clicks together, usually after weeks or months of hard, frustrating work, when things just keep breaking.
Of course, sometimes, breaking stuff can be a Good Thing.
Possibly my favorite story from the development of quantum physics involves just such an occasion, around 1924, when Clinton Davisson and Lester Germer at Bell Labs were trying to characterize a nickel surface by bouncing an electron beam off it. They expected and were seeing diffuse reflection-- when they shot a beam straight into the surface, they saw scattered electrons at many different angles away from the vertical, with the number of electrons bouncing off the surface at a particular angle dropping smoothly to zero as they increased the angle.
Then, they broke a bit of their apparatus. Specifically, they cracked something in the vacuum system containing the nickel target, electron beam, and detector. Air rushed in, and oxidized the surface of the nickel. As they wanted to study pure nickel, not an oxide layer, they dealt with this in the traditional manner: by heating their apparatus to drive off the oxide.
When they got their apparatus up and running again, they found that their scattering results had changed dramatically. Instead of smoothly decreasing as they increased the angle of the detector, the signal decreased for a while, then rose to a dramatic peak at an angle of 50-odd degrees, before dropping again. This was unexpected and puzzling, but turned out to be one of the most important experimental results in history. Davisson and Germer didn't really know what to make of it when they first saw this in 1925, so on a visit to England in 1926, Davisson was shocked to hear Max Born explain his weird results via a fundamental revision of the nature of matter.
Around the time Davisson and Germer began working together, a French Ph.D. student named Louis de Broglie turned in a thesis in which he suggested that electrons should behave like waves. He was motivated by trying to justify Niels Bohr's quantum model of the atom, and it worked brilliantly for that. Of course, nobody had any direct evidence of electrons behaving like waves, until Born spotted Davisson and Germer's results. It's not clear that Davisson and Germer were even aware of de Broglie's wave theory (Davisson's a little vague on this in his Nobel lecture). People in Europe certainly knew about it, though, and had started looking for wave behavior of electrons, but Davisson and Germer had scooped them all by accident.
The weird increase in the number of electrons they were seeing at a particular angle is a diffraction effect-- the electrons were acting like waves, and at just the right angle, electron waves that reflected off the very first plane of atoms in the nickel arrived at the detector exactly in phase with electron waves reflected off the second plane. These interfered constructively, giving the large peak in the scattered electron intensity. This is a phenomenon that had earlier been demonstrated by the Braggs with x-rays, called "Bragg diffraction" in their honor.
The Davisson-Germer result clearly and unambiguously showed the wave nature of electrons. And it only worked because of the time they broke their apparatus. Their initial results didn't show any diffraction peaks because their initial nickel surface was made up of zillions of little crystals, all oriented randomly. The diffraction peaks from all the individual crystals occurred at different angles, and smeared together to give the boring result they were seeing at first. When they heated the surface to clean it, though, they overdid things a little, and actually melted the nickel. As it cooled down, it re-crystallized into a much smoother surface, with large single crystals. These were big enough to produce a clear signal.
Davisson went on to share the 1937 Nobel Prize in Physics with George Paget Thompson, who also observed diffraction of electrons in the late 1920's, in his case by shooting beams of electrons through thin films.
There are a lot of important messages to take away from the Davisson and Germer experiment about the process of science, particularly experimental science-- the importance of sharing anomalous results, the fact that revolutionary discoveries can often happen when you're looking for something else, the importance of being the senior partner in any revolutionary experiment (Germer got left out of the Nobel because the Nobels have always been really bad about that...), etc. The most important one, though, is one to sustain experimentalists through those long days and weeks where everything just keeps breaking: sometimes, breaking your apparatus is the best thing that can possibly happen.