Low hanging fruit

Even with cable, there isn't much on TV before six in the morning. On the stationary bike today I was flipping through the channels and I just had to watch the infomercial for No Evil Oil. According to the testimonials, this stuff not only keeps out the Devil, but gets rid of shingles and makes you rich. The smiling preacher with the mullet will send it to you for free (but there are plenty of donation buttons).

You may be asking yourself, "How dumb/desperate/gullible do you have to be to use the No Evil Oil?" Even my most credulous readers would agree that this stuff is snake oil.

While we probably can't design a study to measure it's Satan-shunning capabilities, we certainly could investigate its medical properties. What if a double-blind, randomized placebo-controlled trial, conducted technically well, showed that No Evil Oil was superior to placebo in reducing the severity and duration of flu symptoms? Should we start prescribing it?

There are many ways to explain seemingly unexplainable results. Here are some options for our hypothetical study:

  1. No Evil Oil works. We don't know how it works, but it does.
  2. The study was subject to bias: perhaps randomization was poorly done, or the operational measures were systematically flawed, or there were unequal attrition rates in the groups---there are many ways to screw up an experiment.
  3. There was a confounding variable responsible for the disparity between groups
  4. There was reporting bias: maybe fifteen other studies failed to find a significant effect.

There are, in fact, many possible explanations for a "positive" study other than "No Evil Oil Works"---that is but one of many possibilities. How are we to figure out which explanation best describes reality?

One way is to look at our basic premise and assess how likely it is that "NEO works" is the right answer. We can look at this a number of ways. Let's do another thought experiment. Let's say that we didn't do a randomized controlled trial but instead a sort of cohort study. We dig up the charts from a clinic and grab out all the flu patients and divide them into groups of "got better fast" and "got better slow". We then comb through the charts to see if there are significant differences between these two groups. We may find a lot of differences: some are smokers, some have asthma, some are older, and some used the ever-popular No Evil Oil. All of these differences we find to be statistically significant. How do we decide which, if any, are important clinically?

The answer to both thought experiments is that we must consider some sort of basic plausibility. We know that No Evil Oil is just some canola with red dye #40 in it and is for all intents and purposes biologically inert. The only difference between No Evil Oil and the placebo is the preacher's magic blessing. We could posit that the blessing makes all the difference, but that would require us to ignore the bulk of our understanding of physics, chemistry, and biology. Since the overwhelming evidence supports our current understanding of science, we are unlikely to abandon this in favor of one or two odd experiments.

But, you may argue, what if our science really is wrong? Should we dismiss these results just because we don't "believe" in them?

The nature of science is to be very skeptical. If we test our hypothesis over and over, using different methodologies, and repeated experiments are done by others that find the same thing, we might---just might---be on to something. Or we might still be missing an alternate explanation or confounding variable.

One thing that we should not do is change the way we treat the flu based on one or two tests of a very improbable hypothesis. The practice of medicine deals with real people, and messing with their health and their hope is immoral and wrong. We work from the best evidence available because it is the right thing to do. That is what produces predictable outcomes and allows us to make intelligent decisions that helps our patients. If we jump on every cockamamie idea, we are performing a great disservice.

Oh, and if you got the idea that when I said "No Evil Oil" I really meant "Occillococcinum", you were right.

Categories

More like this

[Previous installments: here, here, here, here, here] After a detour through the meaning of causation and the need to find a substitute for what can't, in principle, be observed (the counterfactual), we are now ready to consider what many of you might have thought would be the starting point,…
This week Canadian public health researchers published the long awaited paper on possible association between vaccination for seasonal influenza the previous flu season and risk of having a medically diagnosed infection with pandemic influenza during the first wave of infections (April to July)…
It's like this: science requires a tolerance of failure. If your shiny, happy hypothesis fails to stand up to rigorous scrutiny, you drop it and move on. If instead of a true, disposable hypothesis, you have a fixed belief that will not change based on the data, you are delusional. Boosters of…
As most people in any empirical or scientific field know, the gold standard for experimenting and establishing causality is the randomized controlled trial (RCT). In an RCT, subjects are randomly assigned to one of two conditions: an experimental group or a control group. The experimental group…

Even with cable, there isn't much on TV before six in the morning.

Which is why my HD setup is connected to a computer with a LOT of e-books. Baen WebScriptions has a long way to go to make up for all of the novels I've read while lying immobile, but it's off to a good start.

By D. C. Sessions (not verified) on 07 Oct 2009 #permalink

There's a "morning" before six? I don't think I understand.

There's a "morning" before six? I don't think I understand.

From May through September it's the only halfway-comfortable time of day to be outside.

By D. C. Sessions (not verified) on 07 Oct 2009 #permalink

Didn't you forget to re-dilute the No Evil Oil and then shake it really hard? You know that could have thrown off the entire experiment!

By Kate from Iowa (not verified) on 08 Oct 2009 #permalink

Ah, but the placebo effect is real.... perhaps people getting red oil thought better of it than people getting plain oil.

Or perhaps it's no placebo, but since it was red, they spent longer washing it off.

I'm reminded of a trial that attempted to prove or disprove the benefits of ear candling, but didn't take account of the fact that the instructions for the ear candles told people to rub some oil into their ears first -- maybe *that* was the clinically significant part. Ya never can tell.

Whaaaaat...the font on that pastor's banner looks oddly familiar.