Alternative medicine: Changing the rules after the game has started

Damn Steve Novella.

Well, not really, but I always get annoyed when someone comes up with an analogy or description of a phenomenon that I should have thought of first. I don't really get annoyed at the person who came up with such ideas, but rather at myself for not thinking of something so obvious or precious first. Whether this self-criticism is a symptom of the megalomania or massive ego that I have been accused of having by some of my less--shall we say?--enamored readers or simply a personality quirk, I'll leave to the reader to decide.

Whatever the case, writing for Science-Based Medicine, Steve Novella nails it perfectly when he describes why so many advocates of so-called "complementary and alternative medicine" (CAM) are very much like small children making stuff up as they go along:

My daughter, Julia, loves to play games and has a bit of a competitive streak. She can make any activity into a game and is adept at making up rules on the spot. When she was younger, like most children, she had a tendency to add to or change the rules on the fly - usually to ensure a favorable outcome for herself. "Oh, Daddy, I forgot to mention that the ball can bounce once and that still counts."

It was an opportunity for me to gently teach her that in order for rules to work everyone has to know what they are ahead of time and you can't change them after the fact. Her smile told me that even at five she intuitively knew this already - that changing or making up new rules was not fair. What I was really teaching her was that she wasn't going to get away with it with me, and by extension that it is socially unacceptable to mess with the rules to suit oneself.

Now, here's where Steve nails it:

Today there is a political/ideological movement within medicine and health care to change the rules after the fact. The purveyors of many sectarian methods of treatment and unscientific belief systems of health and illness have not succeeded at the fair rules of science. So now they want to change those rules. They want anecdotes to not only count but to trump rigorously controlled observations (that is, when the anecdotes are in their favor). They was to reinterpret the placebo effect after the fact as if it were a real effect. They want to count only those experiments that confirm their beliefs and ignore or reject those studies that reject their beliefs.

Being educated adults they have much more sophisticated language to express their childish desire to alter the rules.

Andrew Weil wants to relabel anecdotes he favors as "uncontrolled clinical observations." This is a way of getting to choose after the fact which observations count, rather than letting the rules of science decide.

Dr. David Katz from Yale's "Integrative Medicine" Program wants to allow for "a more fluid concept of evidence." This way modalities he favors, such as homeopathy, that have failed by the generally accepted rules of science can still win with his more "fluid" rules.

When studies of "alternative" modalities are negative, proponents want to change the rules after they see the results. They claim that the "sham" acupuncture was giving a real effect too, or that the numbers in the study were too small, or that homeopathy cannot be tested with the same methods as cookie cutter drugs, or that a statistically insignificant trend in their favor should count even though the rules say they shouldn't. Of course, when the outcome is positive, then these same rules are just fine. Heads I win, tales you lose.

Indeed. CAM aficionados will point to any study, no matter how poor, as evidence that their favorite woo works. When a skeptic or scientist then counters with numerous other studies that show that it doesn't work any better than a placebo, they then want to claim that science is inadequate to study their modality. A favorite ploy is to claim that their modality is so "holistic" that it can't be adequately evaluated within the confines of well-controlled randomized clinical trials (RCTs). It may be true that RCTs may have their limitations, but there are other scientifically valid, albeit less powerful, methods of studying the efficacy of an intervention. We do it in surgery all the time when it's not possible to do a true blinding or a true randomization. The data aren't nearly as clean, but accumulation of multiple such studies can still demonstrate or cast doubt upon whether a surgical procedure does what it is claimed to do or not.

Andrew Weil is particularly exasperating in this. For example:

But for all the other stuff, we don't have time to do that, so we have to have other methods of estimating how things work. Now one of the attitudes that I run into in the research community that just drives me up the wall is people who dismiss what they call anecdotal evidence. And I have challenged some of these people in public to strike the word "anecdote" from the medical vocabulary.

I think it is a trivializing word. If you want to call this uncontrolled clinical observation, that's fine with me. The fact is that the scientific method begins with raw observation. You notice something out there that catches your attention, that doesn't fit your conceptions. You see it again. That gives you an idea that generates a hypothesis which you can then test. It is this kind of uncontrolled observation which is the raw material from which you get hypotheses to test in a formal manner. If you dismiss all that stuff, if you drop it into a mental wastebasket labeled "anecdote," you cut yourself off from the raw material of science. ...

Of course, it's a massive straw man to claim that we dismiss anecdotal evidence. What I (and most others) tend to dismiss is in actuality testimonial evidence, for reasons that I explained in great detail very early on in the history of this blog and that Steve Novella has also described. We use anecdotal evidence all the time in science-based medicine to do exactly what Andrew Weil says: To develop hypotheses to test in a more organized, rigorous fashion. No physician practicing science-based medicine would say that anecdotes are worthless, nor have I ever said that (although I have said that testimonials are pretty much worthless). Indeed, the medical literature is littered with case reports, which are in essence a formal writeup of an anecdote. The difference is that good case reports have hard data behind them: Physical findings, lab tests, X-ray results, pathology results, etc. They also have a carefully documented timeline assessed by medical professionals as events happen. Not so "testimonials," which are usually vague stories told after the fact, rarely with much in the way of any verifiable hard evidence that can be assessed. True, it is sometimes hard not to conflate anecdotes and testimonials (CAM advocates do it all the time), and indeed I may have fallen into that trap on occasion. The point, however, is that anecdotes can indeed serve as a basis for study, but that's all they are really good for. Except in extreme cases, such as rare diseases where no better evidence exists, they should not be used as the basis for treating patients because they are the weakest form of medical evidence. As for the concept of "uncontrolled" clinical observations, there's a reason why I like to say that the plural of "anecdote" is not "data," and it's because without controls a series of anecdotes is, more than anything else, prone to amplify physician bias, confounding factors that are really behind any effect observed, or both.

The bottom line is that, as Steve describes, CAM is trying to change the rules in the middle of the game. There is a reason why medicine evolved to elevate science and randomized clinical trials to the high places they occupy, and that's because humans suffer from a number of cognitive quirks that easily lead them astray when examining the world around them. Doctors like Steve Katz and Andrew Weil would like to take us back to a time when how doctors practice was determined more by dogma, authority, and tradition than by science.

Categories

More like this

Lately, I've been reviewing the literature on autism pharmacotherapy, and it is loaded with "open label" case studies in which a few patients are given some drug, and based on some kind of rating scale (because there is no lab test for autism--too bad it isn't really mercury poisoning, because then you could just measure blood mercury levels) it is claimed that they got better. Of course "case study" is really just publication-speak for anecdote. There aren't many double blind placebo controlled studies, and those that have been conducted reveal that he placebo "response" in autism tends to be huge. Which means that all of those open label studies (in which everybody knows what drug the patient is getting) are virtually worthless.

The most dramatic example is secretin, a GI peptide. There was a case report in which 3 autistic kids were said to have dramatically improved after a single injection of secretin. It was reported in the media, and people went nuts. Thousands of kids were treated with this stuff, to the point that stocks of secretin ran low. Other open-label studies "confirmed" the report. Eventually, there was a double-blind placebo controlled trial that replicated the dosing of the original report. The secretin kids got better. So did the ones injected with a placebo salt solution. No difference. People just wouldn't believe it. So there were more studies. Maybe it was just kids with diarrhea (the kids in the original study had diarrhea). No difference. Maybe it was just kids who showed regression. No difference. Maybe you needed to treat longer (never mind that the original report was after a single dose). No difference.

Secretin now seems to be better studied than any other drug treatment in autism. And it never turned out to be better than placebo in any controlled study. And there is a long, long list of drugs that continue to be used in autism based on little more than such case studies, for which properly controlled studies have never been done. Meanwhile, NIH is currently carrying out a controlled trial of a chelating agent for treatment of autism. In one sense, I suppose that this is a good thing, even though I have an uneasy feeling about subjecting all of those kids to the hazards of a chelating agent. We might hope that at least it will put an end to this reprehensible nonsense of treating kids with chelating agents. Except that it won't. I have little doubt that if (when) this study comes out negative, the same people who have been pushing chelating agents will insist that they didn't do it right, and they should do it all over again with testosterone blockers or some such nonsense.

Since trrll brings up the subject of autism, here's another example: the Hornig mouse study and the recent monkey study. There is, of course, no accepted animal model of autism (and there may never be), so what the researchers did was administer alleged causative agents to their animals, observe their behavior afterwards, and declare it to be "autism-like." Now if they had decided in advance what sort of animal behaviors should be considered "autism like" they might be doing something legitimate. But waiting to see what behavior emerges and then saying "oh yeah, that one's similar to something autistics do" is circular reasoning, plain and simple. It's like how in statistics you're supposed to come up with your hypotheses to test and your significance level thresholds before you look at your results (a lot of statistics students get confused by this and think it's some quantum-mystic effect involving "observation" and "consciousness"). If you don't, you're again engaging in circular reasoning because you're using the same set of observations to both generate and verify your hypotheses, and all that can "test" is whether your observations are equal to themselves.

Which means that all of those open label studies (in which everybody knows what drug the patient is getting) are virtually worthless.

I completely agree with trill. Autism might be special in that regard. For example, there were numerous case reports on Secretin that seemed promising. There was even a non-controlled trial that suggested Secretin was beneficial. There's too much bias, subjectivity and natural progress in autism. Whenever a non-randomized trial of autism is carried out, a positive finding is almost guaranteed. Conversely, double-blind studies in autism almost always fail to show benefit.

I guess that just means one should be extra skeptical of any non-controlled non-randomized trials when it comes to autism.

This is now a mind reading zone because tincture touched the opposite pole and declared touching it oppositely by not declaring it. The score is now 29 to Q. You may proceed.

By Militant Agnostic (not verified) on 24 May 2008 #permalink

CAM coming to the Farm! Oh the prestige!, The honor! The swirly deserving trustees!
Mind you, they have a history of disgraceful actions. This is the university that through its adjunct Hoover Institution, has VD Hanson, D D'Sousa and D Rumsfeld working as "scholars."

By Onkel Bob (not verified) on 26 May 2008 #permalink

Wait wait wait, back to the acupuncture thing.

What's the scientifically-accepted rationalization for that experiment? That is, why standard treatment for back pain couldn't do any better than a supposedly-worthless treatment method?

FWIW, acupuncture didn't work for me either, but at least it was cheaper and quicker than the 10 years I approached it with standard medicine, and accomplished exactly the same thing.

Doctors have privately confided in me that as long as the pain isn't literally killing me, pain isn't considered to a be a problem worth solving. (Each day being a nightmare doesn't matter.) Apparently, it's far, far worse to be addicted to a cheap, no-side-effect prescription drug, than it is to have every thought interrupted by terrible pain.

Seeking Human Research Participants wanted for a double blind placebo controlled trial: The parachute
Proposed: 50 males and 50 females needed for a double blind placebo control trial to measure the efficacy of the parachute in preventing death.
The 100 research subjects will be flown to an altitude of 18,000 feet above sea level. Each one will randomly receive either a pack with a parachute or a pack filled with clothing (but no parachute). The research participants will be completely blinded as to the actual contents of the pack. All 100 research subjects will be asked to jump from the airplane when it reaches 18,000 feet.

Do I have any takers???!!!

Does the efficacy of water, exercise, sleep, good food, stress-reduction, etc really need to be tested in a double blind placebo control trial?

Seeking Human Research Participants for a double blind placebo controlled trial: The parachute
Proposed: 50 males and 50 females needed for a double blind placebo control trial to measure the efficacy of the parachute in preventing death.
The 100 research subjects will be flown to an altitude of 18,000 feet above sea level. Each one will randomly receive either a pack with a parachute or a pack filled with clothing (but no parachute). The research participants will be completely blinded as to the actual contents of the pack. All 100 research subjects will be asked to jump from the airplane when it reaches 18,000 feet.

Do I have any takers???!!!

Does the efficacy of water, exercise, sleep, good food, stress-reduction, etc really need to be tested in a double blind placebo control trial?