Alternative medicine: Changing the rules after the game has started

Damn Steve Novella.

Well, not really, but I always get annoyed when someone comes up with an analogy or description of a phenomenon that I should have thought of first. I don't really get annoyed at the person who came up with such ideas, but rather at myself for not thinking of something so obvious or precious first. Whether this self-criticism is a symptom of the megalomania or massive ego that I have been accused of having by some of my less--shall we say?--enamored readers or simply a personality quirk, I'll leave to the reader to decide.

Whatever the case, writing for Science-Based Medicine, Steve Novella nails it perfectly when he describes why so many advocates of so-called "complementary and alternative medicine" (CAM) are very much like small children making stuff up as they go along:

My daughter, Julia, loves to play games and has a bit of a competitive streak. She can make any activity into a game and is adept at making up rules on the spot. When she was younger, like most children, she had a tendency to add to or change the rules on the fly - usually to ensure a favorable outcome for herself. "Oh, Daddy, I forgot to mention that the ball can bounce once and that still counts."

It was an opportunity for me to gently teach her that in order for rules to work everyone has to know what they are ahead of time and you can't change them after the fact. Her smile told me that even at five she intuitively knew this already - that changing or making up new rules was not fair. What I was really teaching her was that she wasn't going to get away with it with me, and by extension that it is socially unacceptable to mess with the rules to suit oneself.

Now, here's where Steve nails it:

Today there is a political/ideological movement within medicine and health care to change the rules after the fact. The purveyors of many sectarian methods of treatment and unscientific belief systems of health and illness have not succeeded at the fair rules of science. So now they want to change those rules. They want anecdotes to not only count but to trump rigorously controlled observations (that is, when the anecdotes are in their favor). They was to reinterpret the placebo effect after the fact as if it were a real effect. They want to count only those experiments that confirm their beliefs and ignore or reject those studies that reject their beliefs.

Being educated adults they have much more sophisticated language to express their childish desire to alter the rules.

Andrew Weil wants to relabel anecdotes he favors as "uncontrolled clinical observations." This is a way of getting to choose after the fact which observations count, rather than letting the rules of science decide.

Dr. David Katz from Yale's "Integrative Medicine" Program wants to allow for "a more fluid concept of evidence." This way modalities he favors, such as homeopathy, that have failed by the generally accepted rules of science can still win with his more "fluid" rules.

When studies of "alternative" modalities are negative, proponents want to change the rules after they see the results. They claim that the "sham" acupuncture was giving a real effect too, or that the numbers in the study were too small, or that homeopathy cannot be tested with the same methods as cookie cutter drugs, or that a statistically insignificant trend in their favor should count even though the rules say they shouldn't. Of course, when the outcome is positive, then these same rules are just fine. Heads I win, tales you lose.

Indeed. CAM aficionados will point to any study, no matter how poor, as evidence that their favorite woo works. When a skeptic or scientist then counters with numerous other studies that show that it doesn't work any better than a placebo, they then want to claim that science is inadequate to study their modality. A favorite ploy is to claim that their modality is so "holistic" that it can't be adequately evaluated within the confines of well-controlled randomized clinical trials (RCTs). It may be true that RCTs may have their limitations, but there are other scientifically valid, albeit less powerful, methods of studying the efficacy of an intervention. We do it in surgery all the time when it's not possible to do a true blinding or a true randomization. The data aren't nearly as clean, but accumulation of multiple such studies can still demonstrate or cast doubt upon whether a surgical procedure does what it is claimed to do or not.

Andrew Weil is particularly exasperating in this. For example:

But for all the other stuff, we don't have time to do that, so we have to have other methods of estimating how things work. Now one of the attitudes that I run into in the research community that just drives me up the wall is people who dismiss what they call anecdotal evidence. And I have challenged some of these people in public to strike the word "anecdote" from the medical vocabulary.

I think it is a trivializing word. If you want to call this uncontrolled clinical observation, that's fine with me. The fact is that the scientific method begins with raw observation. You notice something out there that catches your attention, that doesn't fit your conceptions. You see it again. That gives you an idea that generates a hypothesis which you can then test. It is this kind of uncontrolled observation which is the raw material from which you get hypotheses to test in a formal manner. If you dismiss all that stuff, if you drop it into a mental wastebasket labeled "anecdote," you cut yourself off from the raw material of science. ...

Of course, it's a massive straw man to claim that we dismiss anecdotal evidence. What I (and most others) tend to dismiss is in actuality testimonial evidence, for reasons that I explained in great detail very early on in the history of this blog and that Steve Novella has also described. We use anecdotal evidence all the time in science-based medicine to do exactly what Andrew Weil says: To develop hypotheses to test in a more organized, rigorous fashion. No physician practicing science-based medicine would say that anecdotes are worthless, nor have I ever said that (although I have said that testimonials are pretty much worthless). Indeed, the medical literature is littered with case reports, which are in essence a formal writeup of an anecdote. The difference is that good case reports have hard data behind them: Physical findings, lab tests, X-ray results, pathology results, etc. They also have a carefully documented timeline assessed by medical professionals as events happen. Not so "testimonials," which are usually vague stories told after the fact, rarely with much in the way of any verifiable hard evidence that can be assessed. True, it is sometimes hard not to conflate anecdotes and testimonials (CAM advocates do it all the time), and indeed I may have fallen into that trap on occasion. The point, however, is that anecdotes can indeed serve as a basis for study, but that's all they are really good for. Except in extreme cases, such as rare diseases where no better evidence exists, they should not be used as the basis for treating patients because they are the weakest form of medical evidence. As for the concept of "uncontrolled" clinical observations, there's a reason why I like to say that the plural of "anecdote" is not "data," and it's because without controls a series of anecdotes is, more than anything else, prone to amplify physician bias, confounding factors that are really behind any effect observed, or both.

The bottom line is that, as Steve describes, CAM is trying to change the rules in the middle of the game. There is a reason why medicine evolved to elevate science and randomized clinical trials to the high places they occupy, and that's because humans suffer from a number of cognitive quirks that easily lead them astray when examining the world around them. Doctors like Steve Katz and Andrew Weil would like to take us back to a time when how doctors practice was determined more by dogma, authority, and tradition than by science.