Chad, can I get a post about how you (or scientists in general) come up with ideas for experiments? You've covered some of the gory detail with the lab info posts, but I think it would be useful for your readers to see where the ideas come from.
The answer is obvious: Ideas come from Schenectady. Which, not coincidentally, is where I live...
More seriously, in my area of experimental physics, I think there are two main ways people come up with ideas for new experiments: straightforward extensions, and cross-pollination.
Straightforward extensions are pretty much what they sound like: Taking an established technique, and applying it to an existing problem, or refining an existing technique to do a better job of a previous measurement. Most of the stuff I've done has fallen into this category: in grad school, I studied collisions of metastable xenon, and used fairly well-understood technology (electron multiplier detectors, basic laser cooling techniques) to look at those collisions in a new regime. We came up with some novel ways to reach ultra-low temperatures for the spin-polarized collisions experiment, and I adapted a technique developed by the UConn group for the time-resolved collisions experiment, but those were straightforward applications of existing ideas.
My initial research plan (and first grant) here at Union was for a similar project, a return to looking at low-temperature collisions, just in different elements. There are a few small tweaks to the basic plan I followed in grad school, but it's essentially just a refinement of existing work.
The other big category of experimental ideas is cross-pollination, which usually comes about through talking to people in other fields, or even just reading about other fields. These are experiments that take a technology from one field, and use it to study problems in a completely different field.
I've done a little of this, as well. My current project, using single-atom trapping and detection to measure krypton contamination in other rare gases, came about through a conversation with a visiting speaker. Dan McKinsey from Yale came up here to give a colloquium talk, and while I was giving a tour, he asked if I knew about the single-atom trapping technique being used at Argonne for radioisotope dating, and we talked about whether it might be useful for measuring background contamination. We made a few estimates, decided it would work, and wrote a paper and a grant proposal, and started my lab in a completely different direction.
That's not the most radical sort of cross-pollination, but it's a small example of how this sort of thing works. More radical changes often come about in similar ways-- I toured a lab at Maryland a while back where they're looking at combining laser cooling and a SQUID. That experiment, which is a huge technical challenge, came about basically because some people from the SQUID experiment gave a colloquium talk, and mentioned in passing that its resonant frequency was fairly close to the ground-state hyperfine splitting of one of the alkalis (either 6.8GHz or 9.1 GHZ, I forget which). That led the laser coolers to wonder whether there wasn't a way to marry the two techniques.
Sometimes, these sorts of developments come about through people getting diverse training-- people whose Ph.D. work was in one field, who went on to a post-doc in a very different field. Some of the best experiments in the field have come about through this kind of cross-pollination by training-- people whose Ph.D. work was in experimental nuclear physics who move to atomic physics, or people with a background in cryogenic condensed matter physics moving over to work with cold systems using laser cooling. They bring a different skill-set, and knowledge of different problems and systems, and can open up new areas of study.
This can take some interesting forms, such as the "NIST paradigm" that we used to joke about when I was a grad student. We used to joke that there were two styles of research in the laser cooling world: the "French paradigm" exemplified by Jean Dalibard and Claude Cohen-Tannoudji and their groups, who do extremely careful calculations to determine what effects they should see, and then go into the lab and measure exactly that. The "NIST paradigm," in contrast, was exemplified by our group, where the approach was basically, "Hey, we have this other laser. I wonder what would happen if we hit the atoms with that?" A lot of times, this led to us recording completely bizarre signals, then scrambling around trying to figure out what was going on (which occasionally turned out to be a well-known problem from another field), but it was kind of a fun way to operate.
So, that's my schematic answer to the question of where experimental ideas come from. In the end, I think the cross-pollination method tends to lead to the best advances in science, and if I were still in the ultracold quantum gases business, I'd be looking for some way to make cold atoms look like electrons in graphene, or some other condensed-matter system, and probably trying to hire people with a condensed matter background.
Anyway, those are my very rough thoughts on the matter. What methods do people in your field use to come up with ideas?
That was fast - thanks Chad!
You summarized very well what I have been doing (not in Physics of course). My PhD was all about 'refinement' and 'efficiency' and now, my PostDoc is about 'cross-pollination'. I am usually very excited about cross-pollination stuff but I think it was useful to get a good grounding with refinement stuff.
Great take here. We recently blogged about this issue at Lofty Ambitions, too. It's been of great interest to us, and we recently happened upon Steven Johnson talking about his new book Where Good Ideas Come From. Yes, yes, to your straightforward extensions and cross-pollinations. We are especially fond of serendipity as well, and we define it in similar ways to Johnson. He categorizes innovation in ways that make sense: the adjacent possible, the slow hunch, error, etc.