Believe it or not, there was one area of so-called "alternative" medicine that I used to be a lot less skeptical about than I am now. Homeopathy, I always realized to be a load of pseudoscientific magical thinking. Ditto reiki, therapeutic touch, and other forms of "energy healing." It didn't take an extensive review of the literature to figure that out, although I did ultimately end up doing fairly extensive literature reviews anyway. Then, the more I looked into the hodge-podge of "healing" modalities whose basis is not science but rather prescientific and often mystical thought, the less impressed I was.
Even so, there was always one modality that I gave a bit of a pass to. There was one modality that, or so I thought, might actually have something to it. There was one modality that seemed to have a bit of suggestive evidence that it might do something more than a placebo. I'm referring to acupuncture. No, I never bought all the mystical mumbo-jumbo about how sticking needles into "meridians" somehow alters or "unblocks" the flow of a mysterious "life force" known as qi that is undetectable by science. I did wonder if perhaps it worked as a counterirritant or by releasing endorphins.
Then I actually started paying attention to the scientific literature regarding acupuncture, including literature like this and this. The more I read, the more I realized something. I realized that there was far less to acupuncture than I had previously thought, and, even with my previous openness to it, I hadn't thought all that much about it anyway. What I had thought about it was that it might have a very mild beneficial effect. What I know now is that acupuncture is almost certainly no more than an elaborate placebo. What I know now is that virtually every study of acupuncture claiming to show a positive effect has serious methodological flaws and that the better-designed the study the less likely there is to be an effect. What I now know is that any study without a true "sham" acupuncture arm is worthless, and that well-designed studies show "sham" acupuncture to be no different than "real" acupuncture; i.e., no different than placebo.
And now comes yet another in a long line of studies that is consistent with just that, and, worse, it's billed as (and probably is) the "largest randomized study of acupuncture ever done." Too bad it depends on what you mean by "randomized." Too bad the press coverage misses the point:
For the current study, published in the journal Cephalalgia, German researchers followed more than 15,000 adults with chronic headaches; all had been suffering from either migraine or tension-type headaches at least twice a month for 1 year or more.Of these patients, nearly 3,200 agreed to be randomly assigned to either have acupuncture added to their regular therapy or to stay with their usual care alone. The rest of the patients began on acupuncture treatment.
All of the acupuncture patients received up to 15 sessions over 3 months, and all study patients were reassessed after 6 months.
In the end, the study found, acupuncture patients reported greater pain improvements than those who stayed with their usual care only. At the outset, they reported an average of 8.4 headache days over 3 months; that dropped to 4.7 by the study's end.
Take a minute here. If you're a regular reader of this blog, I'm betting that you can pick out the huge methodological flaw in this study from just the press report alone. Do you have it yet? Don't worry, I'll get to it very soon. However, I don't like to rely on just the news coverage of such a study. Whenever possible, I always like to go to the original study, and, as I usually do, I did just that. Here's the abstract:
We aimed to investigate the effectiveness of acupuncture in addition to routine care in patients with primary headache (> 12 months, two or more headaches/month) compared with treatment with routine care alone and whether the effects of acupuncture differ in randomized and non-randomized patients. In a randomized controlled trial plus non-randomized cohort, patients with headache were allocated to receive up to 15 acupuncture sessions over 3 months or to a control group receiving no acupuncture during the first 3 months. Patients who did not consent to randomization received acupuncture treatment immediately. All subjects were allowed usual medical care in addition to study treatment. Number of days with headache, intensity of pain and health-related quality of life (SF-36) were assessed at baseline, and after 3 and 6 months using standardized questionnaires. Of 15 056 headache patients (mean age 44.1 ± 12.8 years, 77% female), 1613 were randomized to acupuncture and 1569 to control, and 11 874 included in the non-randomized acupuncture group. At 3 months, the number of days with headache decreased from 8.4 ± 7.2 (estimated mean ±s.e.) to 4.7 ± 5.6 in the acupuncture group and from 8.1 ± 6.8 to 7.5 ± 6.3 in the control group (P < 0.001). Similarly, intensity of pain and quality of life improvements were more pronounced in the acupuncture vs. control group (P < 0.001). Treatment success was maintained through 6 months. The outcome changes in non-randomized patients were similar to those in randomized patients. Acupuncture plus routine care in patients with headache was associated with marked clinical improvements compared with routine care alone.
Got it yet? Here's the study design graphically shown:
Got it yet?
I'm sure that most of you do; so I'll just move on. This study has three design flaws so glaring that I almost don't even care what it shows because the flaws are so significant that they scuttled the study before it even started. Here they are:
- The study is not only not double-blind, it's not even blinded in any way. Both the patients and the health care practitioners know who is receiving what therapy. That alone makes its result entirely explainable by placebo effects.
- There isn't even an attempt at a sham acupuncture group. Remember my previous posts on the importance of sham acupuncture and how sham acupuncture is indistinguishable from "real" acupuncture.
- The "randomization" isn't even really a randomization. Of 15,056 patients with a complaint of headache, only 3,404 accepted randomization to control or acupuncture. Normally a clinical investigator, when faced with this situation, studies only the patients who agreed to be randomized. Not these intrepid woo-mavens! They included the remaining 11,652 patients anyway. Actually, they included 11,874 nonrandomized patients.
There are a number of other flaws, but, really, they pale in significance to the three above and are hardly worth mentioning except in passing. For example, there were significant differences between the randomized and nonrandomized groups, including higher pain intensity and a shorter duration of chronic headaches, making them prime candidates to be prone to regression to the mean. The study is also suspect because it lumps together all headaches, rather than separating out the migraine headaches, which have a different physiological mechanism behind them than run-of-the-mill headaches. Another problem was that the authors relied on questionnaires, rather than a pain diary. Because a questionnaire relies on patient memory, rather than the patient writing down an incident as it happens, it's prone to recall bias. There are also numerous other nits to pick, but none of them even come close to the three flaws listed above.
Regarding the failure to use a sham acupuncture group or to blind, it makes me wonder if all the studies coming out showing that sham acupuncture and "real" acupuncture are indistinguishable are starting to get to acupuncture advocates to the point where they really aren't even trying anymore. After all, an unblinded study is almost guaranteed to produce an effect, but an investigator has no way of knowing whether that effect is greater than placebo if there is no valid placebo group. True, the authors did all sorts of fancy statistics and handwaving to try to take the reader's mind off of this fundamental fatal flaw, but none of that changes anything.
Reading the discussion is very instructive, as it demonstrates very well the torturing of language and logic that is used by advocates of "alternative" medicine. While they admit flat out that this was an unblinded study and that , the excuse used was that this was a "pragmatic" study designed "chosen to reflect general medical practice." Yes, that's a great excuse not to do the necessary placebo/sham acupuncture control that would make the results of this study interpretable. It's also a lovely excuse to allow patients in essence to self-select for acupuncture by refusing randomization, thus making the likelihood of a placebo effect even greater--except that there's no control that allow us to know if it's just a placebo effect or not. Here's part of what they argue:
Although differences with respect to both baseline characteristics and treatment outcomes between randomized and non-randomized patients were small in absolute numbers, our findings indicate that randomization was associated with some selection effects. Therefore, the use of study designs that also include non-randomized patients appears to be desirable.
It is of note, however, that treatment benefits were similar in the randomized and nonrandomized acupuncture groups after adjusting for baseline differences. This suggests that the results of randomized trials can be representative of routine medical care situations, at least in large pragmatic studies.
No, it suggests that the placebo effect was operative in both the randomized patients and the patients who refused to be randomized. The use of study designs that include non-randomized patients is only "desirable" if you want to maximize the chances of a seemingly positive result. Bravo, though, to the study authors for having the chutzpah to try to change this study's most glaring weakness into a strength. It was a nice try, but it won't fly.
What's truly depressing is the editorial by Dr. H-C Diener of the University Hospital Essen in Germany, where he actually makes this argument:
Despite the fact that I have major design issues with the study, my view is that studies like this have to be published in high ranked journals to promote discussion on trial design in non-drug trials
Dr. Diener actually lists two of the same flaws that I did, namely the unblinded nature of the study and the lack of adequate "sham" acupuncture controls. In spite of this, he still argues that this article should be published in high ranked journals "to promote discussion"? Funny, but I always thought that high ranked journals are high ranked because they post the most scientifically sound and medically interesting articles. Think New England Journal of Medicine. I always thought the reason for such journals to publish an article is because it is scientifically sound and studies a clinically important and/or interesting question. As a reason to publish such an article, "to promote discussion" is about as far down on the list of reasons as "because the author has nice hair."
The bottom line is that this study is yet another of a long line of studies of "complementary and alternative" medicine that are entirely consistent with the placebo effect. Worse, it didn't even really try to distinguish between a treatment effect and placebo effect. Maybe that's the point. Whatever the point was, what I do know is that if I were a German citizen, I'd be mightily pissed off that so much money was wasted on this study. I'd also wonder why it was a consortium of insurance companies who funded the study. Maybe I was wrong about insurance companies funding woo. Maybe it is cheaper in the long run for them to pay for CAM than actual scientific medicine that's more than just a highly elaborate placebo.
REFERENCES:
1. Jena, S., Witt, C., Brinkhaus, B., Wegscheider, K., Willich, S. (2008). Acupuncture in patients with headache. Cephalalgia, 28(9), 969-979. DOI: 10.1111/j.1468-2982.2008.01640.x
2. (2008). Acupuncture for the treatment of headaches: more than sticking needles into humans?. Cephalalgia, 28(9), 911-913. DOI: 10.1111/j.1468-2982.2008.01650.x
There was an interesting study profiled on Scientific American Frontiers that was testing whether the placebo effect with acupuncture was higher than the "average" placebo effect for Americans because of all the mystic aura surrounding it. I haven't been able to track down whether the results were published, though. The segment is here, called "healing rituals". I was impressed by the way he figured out how to do a sham acupuncture treatment.
have there been any studies where the effects of sticking needles into people (sham or non sham acupunture) has been compared with a procedure where the practitioner pretended to stick needles into people?
colmcq, that was the study on the show - they didn't stick needles in at all, it was a retractable needle that went back up into a hollow shaft. They used tape to keep the needle on the skin, telling participants that the tape was just a variant to make sure the needle didn't move.
In case anybody hasn't seen it, there's a rather nice little Cectic comic strip which picks up on the sham acupuncture gig. Enjoy!
I'd like to see a comparative study of two sham acupuncture treatments, one performed by Chinese sham practitioners and the other by western sham practitioners, to test the hypothesis that a (relatively benign) form of racism would make a measurable difference in results.
Some years ago I received accupuncture treatment for migraines, which at that time were quite severe.
At the time I thought that it was effective. Back then I was probably a lot less sceptical of such things but the main reason for my belief was that the first time I had the needles stuck in me - partly out of desperation - I fainted whilst lying on the couch. I felt that clearly it had done something, so there must be something to it.
I moved home and job some time after and didn't continue the treatments in the new location. In recent years the severity of the attacks has greatly decreased. (Something I put down to a more congenial environment, and "just one of those things".)
In retrospect, I'm no longer sure that my migraines were helped at all all those years ago, but that initial fainting experience - easily explained without recourse to mysticism - left me open to the idea that the accupuncture should have helped.
Ahh, there y'go, poking holes in the whole acupuncture thing again. I wonder if they'll ever get the point, or if the constant needling will have to continue.
Penetrating critique, though.
Orac wrote:
"[acupuncture posits] a mysterious 'life force' known as qi that [truly] is undetectable by science [...] acupuncture is almost certainly no more than an elaborate placebo."
I agree, in the sense that a scientific claim requires immediate, discrete evidence, and vitalism has none, and overall acupuncture as a therapy is simply 'an emptiness!'
Yet, beware, all ye seeking a higher education degree expressly labeled "science," these days!
Fraud -- of the epistemic, academic type -- abounds.
E.g.:
I turn your attention to the University of Bridgeport's "professionally accredited health sciences programs" cloud cuckoo land, which includes:
an acupuncture M.S., a chiropractic D.C., and a naturopathy N.D.
(see http://www.bridgeport.edu/pages/5254.asp?item=3388 ; http://www.bridgeport.edu/admissions/InternationalApplication03-04.pdf ).
What kind of science is the science at UB's "division of health sciences?"
Well, does it shock you to know that...
it's the kind of science that states that a "life force" (see http://www.bridgeport.edu/ub/nm/Six_Prihtm.htm ; http://www.bridgeport.edu/ub/nm/Six_Prithree.htm )
is a scientific fact (see http://www.bridgeport.edu/ub/nm/Today%27s_Nat.htm ) {as well as the supernatural}?
Therein, figments / the not-in-evidence -- particularly, the scientifically-ejected -- are labeled scientific fact!
So, where has higher education's integrity gone?
I regard this, these days, as a "confidence trick" (see http://naturocrit.blogspot.com/2008/08/university-of-bridgeports-falseh… ).
-r.c.
tee hee--"sham acupuncture" "sham TT"--makes me giggle.
How about "sham margarine" or "sham fake nails"?
I of course understand the need to have "sham acupuncture" (tee hee) in a study of acupuncture, but still, the concept of sham woo is funny.
I always wonder when/if someone will take the initiative to just study and pin down the physiolocigal mechanism(s) behind the placebo effect, and develop treatments for triggering those mechanisms directly (not indirectly via elaborate deception).
I started noting the flaws in this study on a post-it as I was reading, but ran out of room on the post-it halfway through.
With no blinding (abort study), no sham (no, really- call it off), and such a large non-randomized group (screw the parachute, just jump out), how is this study any different from just asking people who have had acupuncture if it worked for them?
For all practical purposes, this study had no control group whatsoever.
Why would you ever let people choose whether to be randomized or not, even if you had a study where the subject couldn't be blinded (like surgery)? If they don't want to be randomized, they don't want to be in the study- end of discussion.
Investigator: "Do you want to get the treatment that might help, or do you want to be in the group that we just observe suffer?"
Subject: "Unmanaged pain, please."
Acupuncture- Stick a needle it in, it's done
"Worse, it didn't even really try to distinguish between a treatment effect and placebo effect. Maybe that's the point."
Could be. I've had a hard time trying to explain to my alt-med friendly friends why it matters whether a positive effect is due to placebo or not. If people feel better, then as far as they're concerned the treatment "worked." Real physical changes, long-term benefit, wide application, and progress in cure and understanding are easily set aside in favor of the quick and superficial perception that someone has been "helped."
Remember, for thousands of years human medicine has relied primarily on the placebo effect. I think history demonstrates that it is far less useful than people think.
I'll preface this with the explanation that I'm not an MD or medical researcher (or really a researcher of any kind), so please forgive me if this is a silly question:
How exactly would one do a true double-blind study of acupuncture (or acupressure, or another physical treatment)? My understanding is that in a double-blind study, neither the researcher nor the patient knows which treatment the patient is getting. But wouldn't a patient automatically know whether he or she is getting acupuncture vs. not? Seems like it might be hard to miss. Or is that what the sham trial is? I.e., the person thinks they're getting real acupuncture, but they're actually getting sham woo? (I agree with Diane, that is a funny concept.)
I believe you that this trial is fundamentally flawed. I'm just curious how one does a double-blind study on something that a patient would have to be unconscious to miss...
"is that what the sham trial is? I.e., the person thinks they're getting real acupuncture, but they're actually getting sham woo?"
That's exactly right - the sham acupuncture is designed to look and feel like "real" acupuncture to the patient, but it's not.
SimonG,
It's very common for migraines to decrease as you get older, especially for men. I used to get them fairly often (say, 4-5 times per year) in my 20s, but now that I'm a couple of decades beyond that, I get *maybe* one a year and it's much less intense.
Advantages of aging: 1
Against: 3213
There is another glaring flaw with the study. They counted days with or without headaches and the numbers of headaches per day. Headache duration was not considered. Having 5 headaches per day with acupuncture vs. 8 without acupuncture is not beneficial if each of the 5 headaches lasts 2 hours while each of the 8 headaches per day lasts 1 hour.
I'm also astonished by the size of the study. Is chronic headache in Germany the work-avoidance equivalence of chronic lower back pain in the U.S.A.?
My apologies along with Dr. Kate if this is silly, but: okay so we can "blind" the patients with different needles, but how would one "blind" the examiner? won't that person know what kind of needles he's using?
Hmm... Will people get well if I sit here at my computer and wave my hands at their web page ?
Odds are, they probably won't. And even if they did, there is no proof that my hand-waving caused them to get well.
I have yet to see a study of acupuncture that shows it works above that of placebo that didn't have major flaws in it.
I am mystified. These researchers clearly know something about clinical trial design and epistemology. They do, after all, conduct something that resembles a proper scientific trial. They see that their methodology has flaws, and try to make excuses for them. These people clearly know better. Why would anyone go to the enormous effort of running a such a study when they know the whole thing is bullshit? These people probably have skills that would allow them to pursue a legitimate medical career. Is fraudulent woo really that much more lucrative? Or are they acting from some non-monetary motive I fail to comprehend?
Hm. From about the age of 10 to about 18 or so, I suffered about 1 severe, blinding headache a week. After that they rapidly became much less common. Now, I've no idea whether they were migraines or not - but there's NO FUCKING WAY I would accept those regular headaches back in return for losing the myriad disadvantages of age I've picked up since then.
I'm surprised no one mentioned the size of the "n" as being a problem. But the other issues are for sure a glaring problem...
The Australian Medical Association's (AMA) position statement says the following:
"Research indicates that many GPs in Australia have accepted therapies such as acupuncture, chiropractic, hypnosis and meditation as potentially beneficial. Over 80% of the GPs surveyed had referred patients for a complementary therapy at least a few times a year. At the same time, many GPs express greater confidence when therapies are practised by those who are also medically trained."
Over 80% of medically trained doctors (GPs) in Australia are supporting CAM. The remainder of the position statement actually seems to take no position on CAM except to suggest more research, etc.
Can you explain why trained doctors are recommending CAM? Is it a lack of critical thinking, a lack of training or are patients driving the bus here? Or something else?
WRT to the design of "sham" studies -
1 - the retractable needle placebo design is absurd. Even if the needle never enters the skin, there is still going to be sensation and pressure. Acupressure uses the same meridians, and there's no reason to think there won't be some effect.
2 - there HAVE been reasonable double-blind studies. These used a placebo design where the practitioner was told the patient had a different problem. SO - patients either got treatment for the right problem, or treatment for the wrong one. Same number of needles, same "mystical aura", same amount of attention from the practitioner. Needles in the wrong place. Pubmed has several reports.
3 - Claiming that meridians are "imaginary" is like claiming that the equator is imaginary. All you are proving is that you have no basis of understanding.
I volunteer for the local Red Cross. I'm also a dog-owner. Which is why I agreed to staff the doggie first-aid booth at the Canine Wellness Day that our local Kennel Club is sponsoring next Sunday.
I wish I had seen the flyer for the event before I made the offer. They are devoting a couple of hours in the afternoon to Chinese herbal medicine, canine acupuncture, and the Lord only knows what other crap.
This infuriates me. A dog has no power over its health care decisions (no more than a child does) and a responsible pet owner takes a sick dog to a qualified veterinarian, for crying out loud! Keeping an animal suffering while you monkey around with needles and dried weeds is grounds for eternal damnation in my book.
I would leave my post and go over and ask nasty questions but I'll be in my little Red Cross shirt with my little Red Cross nametag. Sigh.
Anyone care to be at the Town & Country Dog Club in Bloomington on September 7th to act as the voice of reason?
A double blind test requires that the patient and the observer do not know if the needles are real or not. The person who is installing the needles knows...but does not interact with either the observer of symptoms or the patient. Normally, the observer does not enter the room until after the practitioner has left to preclude any rolled eyes, or any other sort of communication but I suppose there could be other safeguards in place if the practitioner had to hang around to twirl the needles.
You don't need an unconscious patient, just a temporary velvet mask to preclude non-verbal communication between the patient and the practioner is plenty, and is often considered overkill. (the blind in double blind!)
That third person, the practitioner, is also the custodian of the treatment regimen but should NEVER be the recorder of symptoms.
This is a stale blog post, I know, but I just felt a need to add this:
"You know the type: People who have no knowledge about a topic or, even worse, just enough knowledge to sound as if they have a clue about it to people who don't have a clue but who are at the same time easily spotted as utterly and completely clueless by people who do have a clue."
Thanks for listening.
Looks to me like the original question was the right one - why bother with the attempted RCT at all? Anything that is even nominally tailored to the individual cannot be tested by RCT. What do you do about surgery eg for back pain? How do you double blind that? So almost everyone here has been wasting their time other than there might be a placebo effect from their having a chance to rant about hippy dippy people.
By the way, as an insurance bod myself, the explanation for insurance companies paying for the trial is that they have customers who want acupuncture, so they are trying to find out if it works. Quite sensible to me, and if the trial is badly designed then it is not really the insurers' home territory. But it's quite cheap compared with the $5 billion insurers have been bullied into spending on another piece of real junk - follow the link.
science.http://www.nytimes.com/2008/09/02/business/02vytorin.html?_r=1&ref=opin…