Low hanging fruit

Even with cable, there isn't much on TV before six in the morning. On the stationary bike today I was flipping through the channels and I just had to watch the infomercial for No Evil Oil. According to the testimonials, this stuff not only keeps out the Devil, but gets rid of shingles and makes you rich. The smiling preacher with the mullet will send it to you for free (but there are plenty of donation buttons).

You may be asking yourself, "How dumb/desperate/gullible do you have to be to use the No Evil Oil?" Even my most credulous readers would agree that this stuff is snake oil.

While we probably can't design a study to measure it's Satan-shunning capabilities, we certainly could investigate its medical properties. What if a double-blind, randomized placebo-controlled trial, conducted technically well, showed that No Evil Oil was superior to placebo in reducing the severity and duration of flu symptoms? Should we start prescribing it?

There are many ways to explain seemingly unexplainable results. Here are some options for our hypothetical study:

1. No Evil Oil works. We don't know how it works, but it does.
2. The study was subject to bias: perhaps randomization was poorly done, or the operational measures were systematically flawed, or there were unequal attrition rates in the groups---there are many ways to screw up an experiment.
3. There was a confounding variable responsible for the disparity between groups
4. There was reporting bias: maybe fifteen other studies failed to find a significant effect.

There are, in fact, many possible explanations for a "positive" study other than "No Evil Oil Works"---that is but one of many possibilities. How are we to figure out which explanation best describes reality?

One way is to look at our basic premise and assess how likely it is that "NEO works" is the right answer. We can look at this a number of ways. Let's do another thought experiment. Let's say that we didn't do a randomized controlled trial but instead a sort of cohort study. We dig up the charts from a clinic and grab out all the flu patients and divide them into groups of "got better fast" and "got better slow". We then comb through the charts to see if there are significant differences between these two groups. We may find a lot of differences: some are smokers, some have asthma, some are older, and some used the ever-popular No Evil Oil. All of these differences we find to be statistically significant. How do we decide which, if any, are important clinically?

The answer to both thought experiments is that we must consider some sort of basic plausibility. We know that No Evil Oil is just some canola with red dye #40 in it and is for all intents and purposes biologically inert. The only difference between No Evil Oil and the placebo is the preacher's magic blessing. We could posit that the blessing makes all the difference, but that would require us to ignore the bulk of our understanding of physics, chemistry, and biology. Since the overwhelming evidence supports our current understanding of science, we are unlikely to abandon this in favor of one or two odd experiments.

But, you may argue, what if our science really is wrong? Should we dismiss these results just because we don't "believe" in them?

The nature of science is to be very skeptical. If we test our hypothesis over and over, using different methodologies, and repeated experiments are done by others that find the same thing, we might---just might---be on to something. Or we might still be missing an alternate explanation or confounding variable.

One thing that we should not do is change the way we treat the flu based on one or two tests of a very improbable hypothesis. The practice of medicine deals with real people, and messing with their health and their hope is immoral and wrong. We work from the best evidence available because it is the right thing to do. That is what produces predictable outcomes and allows us to make intelligent decisions that helps our patients. If we jump on every cockamamie idea, we are performing a great disservice.

Oh, and if you got the idea that when I said "No Evil Oil" I really meant "Occillococcinum", you were right.