It’s like this: science requires a tolerance of failure. If your shiny, happy hypothesis fails to stand up to rigorous scrutiny, you drop it and move on. If instead of a true, disposable hypothesis, you have a fixed belief that will not change based on the data, you are delusional. Boosters of alternative medicine prefer the term “maverick” to “lunatic” but in the two are often the same.
It is nearly impossible to get someone to abandon a belief in alternative medicine, no matter how strong the evidence against it. Study after study has failed to validate homeopathy as anything other than bullshit, yet it’s strongest supports hang on hoping, perhaps, that someone will find out that we were wrong about physics and chemistry all along (you know, regional changes in physical constants and all that). Not all alternative medicine boosters are cynical thieves. Some really do believe that they are doing science, when in fact they are deceiving themselves about the meaning of data. When this type of thinking occurs in medicine, rather than leading to a paper retraction, it leads to quackery and sometimes death. Certain health conditions, because of their special characteristics, are more susceptible to this type of quackery than others.
Fibromyalgia is a poorly-understood and controversial pain syndrome. In brief, it includes patients who have significant chronic pain which is not due to any identifiable pathology. It probably includes a heterogeneous group of problems, but our understanding is limited. There may be changes in the way the nervous system deals with pain signals, but even this is not yet clear. It’s a disorder that can be very frustrating to treat, and even more frustrating to have. It is often co-morbid with depression, and the pain can be quite resistant to treatment.
Some practitioners deal with this by rejecting the diagnosis as being vague and useless. Others use the limited evidence we have to develop a treatment plan. Yet others turn to quackery, the topic of a recent New York Times piece. The article is a brief presentation by an expert with Q&A in the comment section. He briefly describes fibromyalgia and talks about the difficulties in finding successful treatments:
One exciting area of research in the past decade has been in the realm of complementary and alternative medicine, or CAM, treatments for fibromyalgia. These range from well recognized therapies like acupuncture and massage to more novel treatments like d-ribose and qi-gong.
As this research grows, it is increasingly possible to identify CAM therapies that have some evidence of efficacy and minimal risk that can be incorporated right along with the more conventional treatment recommendations.
This would be great if there were such evidence. A recent article from Rheumatology International looked at randomized-controlled trials (RCTs) of alternative medicine for fibromyalgia. Most of the studies were of poor quality and had conflicting results. Some of the positive results are easy to misinterpret.
Let me set us all straight on one thing first. There is a difference between “cherry-picking” data in order to support a failing hypothesis, and critically evaluating data that has questionable conclusions. One is driven by faith, the other by critical thinking.
RCTs are a strong type of evidence, but that does not make them immune from criticism. In the review in Rheumatology International, the authors reported a single positive RCT for homeopathy. There are a number of ways to interpret this. One is to look at RCTs as de facto gold standard tests, independent of their quality. Another is to think.
RCTs are only as good as their design, conduct, and interpretation. There are flaws in each step that can lead one to erroneously keep ones hypothesis rather than reject it. In the case of alternative medicine treatments for fibromyalgia there are a number of ways to misinterpret data.
Dr. Harriet Hall would remind us of Tooth Fairy Science. We can measure all of the important data about the tooth fairy, including average get per tooth, average age of visitee, etc, but if we forget to question the fairy’s existence, we have failed to ask the most important question. It may be true that an RCT showed improvement in fibromyalgia patients using homeopathy, but since homeopathy is water, there is no reason to expect causality, and the results may be better explained by some other phenomenon.
This is explained mathematically by Bayes’ Theorem. If the prior probability of a positive result being due to the intervention is very low (say, because of implausibility), then any positive finding is very likely to be due to chance rather than causality.
Confounding natural variation with causation
Fibromyalgia is a syndrome whose symptoms naturally wax and wane. It can be very easy to confuse a change in disease state that occurs during a study with an actual effect. Rigorous controlling can minimize this but not prevent it. If, by chance alone, subjects in the treatment group had improvement in their disease due to its natural history, this will look statistically like a “win”. This makes the study of such disorders difficult, and opens a big door for CAM, as it is easy to convince others to follow your misattribution of cause. This is similar to concepts such as lead-time bias and regression toward the mean.
Built into this is the common cognitive error of confirmation bias: if you are a believer in the intervention, you may be prone to attribute positive results to the intervention even if there is no causation.
Damned statistics and replicability
The statistical tools we use to interpret RCTs are designed to help us tell systematic variations in the data from chance alone. There are a number of arbitrary assumptions built into this system. For example, if results are described by a normal distribution, we may define “abnormal” as the highest and lowest 2.5% of results. If a single RCT shows statistically promising results (say, >2.5 SDs from the mean), then it’s “positive”–but this still may be due to chance alone. A well-designed study can minimize the chance of this result being due to chance alone but cannot eliminate it. This is one of the reasons a single positive test for a less plausible hypothesis must be replicated before we get too excited.
The bottom line
Fibromyalgia is a complicated syndrome whose very nature makes it susceptible to the abuses of CAM practitioners. When evaluating a therapy for a complex disorder whose natural history is variable, we must very carefully parse out causation from correlation, recognize our own biases, and remember that a positive result of a randomized-controlled trial does not necessarily confirm a hypothesis. If an intervention has no plausible way or working, any positive results are likely a statistical artifact. Science isn’t a contest to see who can crank out at least one positive study. It is a way of evaluating hypotheses to see which ones most closely fit reality. Science is hard work, but the results are worth it.
Baranowsky, J., Klose, P., Musial, F., Haeuser, W., Dobos, G., & Langhorst, J. (2009). Qualitative systemic review of randomized controlled trials on complementary and alternative medicine treatments in fibromyalgia Rheumatology International DOI: 10.1007/s00296-009-0977-5