"This isn't my life anymore, Mulder. I'm done chasing monsters in the dark." -Dana Scully, X-files
One of the most exciting things a scientist can experience is when they look at a sample of data, expecting to not see a particular exciting effect, and yet, something’s there. Immediately, you have to check yourself: what are all the other things it could be? What are the mundane possibilities that could mimic that effect? And what can I do, if anything, to rule them out? Whenever you have a new idea, your main job, first and foremost, is to try and blow the biggest holes in it you possibly can.
Last week, a publication came out claiming that, out of 2.5 million SDSS spectra that were analyzed, about 0.01% of them showed an unusual high-frequency variation. After exhausting the astrophysical possibilities for this effect, two scientists came to the conclusion that it must be aliens; that no other possibility sufficed. And yet, despite passing peer review, that’s not at all the opinion of a majority (or even a reasonable minority) of scientists.