Yet another acupuncture meta-analysis: Garbage in, garbage out

ResearchBlogging.orgEver since I started paying attention to it, acupuncture has, at least until recently, inspired ambivalence more than anything else in me. As a skeptic and science-based physician, I found it very easy to dismiss utter quackery like homeopathy or the various "energy healing" modalities, such as reiki or therapeutic touch strictly on the science alone. After all, homeopathy is based on magical thinking more than anything else, specifically the concepts of "like cures like," the concept that dilution with vigorous shaking can make a remedy stronger, and the idea that water has "memory" all are very implausible on the basis of well-established science alone. Similarly the concept that healers can manipulate living human "energy fields" (or life energy, also known as qi) for therapeutic effect is also incredibly implausible, given that no scientist has ever detected qi or shown that healers can manipulate it for therapeutic intent or any reason.

When I first became interested in "alternative medicine"--excuse me, I mean "complementary and alternative medicine" (CAM)--I viewed acupuncture somewhat differently. No, I never bought the traditional explanation that sticking thin needles into the skin somehow alters the flow of qi in order to induce a therapeutic effect. That is no more plausible than reiki or therapeutic touch. However, there are needles breaking the skin in acupuncture. It was, at least to me, not entirely implausible that that might have some sort of physiologic effect. Then I had to go and ruin that lovely kumbaya feeling towards CAM by actually going and looking at the scientific literature on acupuncture, as I've documented on this blog on multiple occasions (for example: 1, 2, 3, 4, 5). When I actually bothered to do that, I soon realized that the evidence that acupuncture is anything more than a highly elaborate placebo is shockingly thin. More like nonexistent, actually. So what do you do if you're a CAM believer but can't find studies that strongly support the efficacy of your favorite CAM modality, like acupuncture.

Why, meta-analysis, of course!

And here comes another one, this time for acupuncture to treat headaches:

WASHINGTON (Reuters) - Acupuncture works better than drugs like aspirin to reduce the severity and frequency of chronic headaches, U.S. researchers reported on Monday.

A review of studies involving nearly 4,000 patients with migraine, tension headache and other forms of chronic headache showed that that 62 percent of the acupuncture patients reported headache relief compared to 45 percent of people taking medications, the team at Duke University found.

"Acupuncture is becoming a favorable option for a variety of purposes, ranging from enhancing fertility to decreasing post-operative pain, because people experience significantly fewer side effects and it can be less expensive than other options," Dr. Tong Joo Gan, who led the study, said in a statement.

"This analysis reinforces that acupuncture also is a successful source of relief from chronic headaches."

Before I continue on to discuss the actual meta-analysis itself, let me test what you have learned from reading this blog on a regular basis (assuming, of course, that you've been a regular reader for a while; if you're a newbie ignore this). Can you tell me what is wrong with the above result and why it is not particularly impressive? No cheating and reading ahead yet. Think and see if you can identify what the problem is. Once you've done that, feel free to click to read below the fold to see if you found the same problem that I did.

That's right. These were studies comparing acupuncture with medication, and, as such, they were almost certainly not blinded. Hello, placebo effect! What this result probably says, more than anything else, is how frustratingly ineffective medications are in bringing relief to suffers of chronic headaches. What was the other finding? Let's see:

Writing in Anesthesia and Analgesia, they said 53 percent of patients given true acupuncture were helped, compared to 45 percent receiving sham therapy involving needles inserted in non-medical positions.

"One of the barriers to treatment with acupuncture is getting people to understand that while needles are used, it is not a painful experience," Gan said. "It is a method for releasing your body's own natural painkillers."

Notice how much closer the groups have become when comparing true acupuncture with sham acupuncture. Goodbye, placebo effect, at least when the groups are closer to being blinded. Of course, the contrarian (or someone who just doesn't like me) might say: See, see! There's still a difference between true acupuncture and sham acupuncture! That means there must be something to acupuncture!

Not so fast there, pardner. Let's take a look at the actual study by Sun and Gan in the latest issue of Anesthesia and Analgesia, right off the presses. As is usually the case, there's a lot less to this than meets the eye.

To do their study, authors started out by doing a typical sort of search for randomized clinical trials of acupuncture for headache. Their strategy is summarized in the figure below:

i-3b13485963c6535a3f1236b1a28dc006-searchstrategy.jpg

Studies were rated on a quality scale defined thusly:

One point was assigned to the trials in which the patient and assessor blinding were stated and two points were given to the trial in which patients could not distinguish the group allocation by credibility or guessing test and the assessor blinding was described adequately. The maximum score was 7; trials with a score of 4 or more points were considered high quality.

One major problem that becomes apparent right away is the extreme heterogeneity of the trials included:

The majority were conventional two-arm comparison trials: true acupuncture versus sham acupuncture (16 trials), acupuncture versus medication treatment (8 trials), and acupuncture versus physiotherapy (2 trials). Five trials have three arms, acupuncture, sham, and waiting list (one trial), acupuncture, sham, and usual medication (one trial), acupuncture, behavioral program, and waiting list (one trial), acupuncture, physiotherapy, and relaxation training (one trial), acupuncture, and transcutaneous electrical nerve stimulation (TENS) at acupuncture points and medication (one trial).

A set formula of acupuncture was performed in 11 trials and individual acupuncture treatment was used in 16 trials depending on the type and distribution of pain based on diagnosis of traditional Chinese medicine. Semi-standardized acupuncture was used in four trials in which basic points were applied for each session and additional points were selected individually based on the pain topography. Three trials used electrical needling acupuncture. The treatment session was, on average, 10 sessions (range, 6-16) during a mean of 8 wks (range, 4-24 wks).

Heterogeneity is, of course, always a major problem in meta-analyses of any kind looking at the effect of at therapeutic intervention, but this set is still heterogeneous even for a meta-analysis. Another problem with is the quality of the studies. The authors somewhat disingenuously state that that 14 of the studies scored 4 or more on their scale estimated quality and that five of them scored 7/7. Sounds pretty good, right? What they fail to mention, except for listing without comment in Table 1, is that eight of the studies scored only 1/7 and two of the studies scored 2/7. That means that blinding was not well addressed at all in these studies. In other words, one third of the studies were rated as only 1/7 or 2/7. That's not good, and it's unclear to me why these studies were included. Not surprisingly, these studies also tended to be the "most positive" studies favoring acupuncture, with the highest relative risk ratios.

Surprise, surprise.

Also, when one looks at the results in more detail, rather than just reading the abstract, the results become even more underwhelming. We're talking relative risks in the 1.2 range for the most part, with confidence intervals that come close to 1.0. For migraines, there was no difference in the response rate. For none of the headaches was there a significant early difference in headache intensity between control and acupuncture at the early followup time point, and for headache frequency there was no difference between any of the groups. The authors state that restricting the analysis to high quality trials did not significantly alter the results other than decreasing the measures of variation, but with such heterogeneous studies, with such variable outcome measures and design, I have a hard time believing that a relative risk of only 1.19-1.33 range for most of the groups means much of anything. The sole exception, not surprisingly, was the acupuncture versus the medication comparison, where the relative risk was reported as 1.80 in favor of acupuncture, but that's the very group of studies where there was no blinding at all; i.e., the very studies where the placebo effect was most likely to be in play. Also, in a meta-analysis of this sort, a relative risk as low as this and differences between the experimental and control groups could well be explained by publication bias, which, as you may recall, is the phenomenon where negative studies tend not to be published while positive ones are more likely to be published.

In the discussion, the authors acknowledge some, but not all, of these limitations and largely poo-poo them, concluding that acupuncture is better than sham acupuncture and also better than pharmacological treatment of chronic headaches. Indeed, their discussion smacks of a lot of rationalization and handwaving:

Complete blinding of the control group has always been a challenging issue in acupuncture clinical trials. Different strategies have been adopted to provide the best blinding possible. Sham control, often regarded as approaching the ideal strategy, provides the control subjects with the impression closest to true acupuncture but without any real analgesic effects. The most often used sham is superficial needling in which needles are inserted superficially at nonacupuncture points. Ten of 14 studies in this analysis using this method found positive results in response rate, but no significant difference between acupuncture and sham in headache frequency and intensity. However, it has been argued that needling at nonacupuncture points may produce similar physiological effects. Biochemical evidence suggests that stimulation of nonacupuncture points may also result in the release of endorphins and hence produce analgesia. As such, the treatment effects of acupuncture might be under-estimated.55,56 Alternative methods without penetration included a blunt needle against the skin or tapping a cocktail stick against a bony surface to create the pricking sensation but without producing analgesic effects.

I find it very telling that many of the studies using sham needling showed no difference in headache frequency and intensity, which is what sufferers of chronic headache really care most about. Once again, this appears to be the same as the usual case in CAM modalities: The better the study design and the better the controls, the less likely there is to find any significant treatment effect. Indeed, one wonders if the alleged treatment effect is even clinically significant (namely, that patients and their doctors notice it an consider it worth the trouble it takes to achieve it). Once again, this meta-analysis seems to be consistent with the hypothesis that acupuncture for headaches is unlikely to be more than an elaborate placebo.

Which brings me to a second acupuncture study reported this week. Unfortunately, this study didn't get as much attention as the meta-analysis. On the other hand, it isn't published in the peer-reviewed literature yet. Even so, I think it's worth mentioning because it is so deliciously appropriate. Why? Because it found that the benefits of acupuncture do not require that the needles actually penetrate the skin:

The acupuncture study of 215 patients who were undergoing radiation treatment in the abdomen or pelvic region chose by lot one of these two acupuncture types.

109 received traditional acupuncture, with needles penetrating the skin in particular points. According to ancient Chinese tradition, the needle is twisted until a certain 'needle sensation' arises. The other 106 patients received a simulated acupuncture instead, with a telescopic, blunt placebo needle that merely touches the skin.

The acupuncture was performed by physiotherapists two or three times a week throughout the five-week radiation period.

Afterwards 95 percent of the patients in both groups felt that the acupuncture treatment had helped relieve nausea, and 67 percent had experienced other positive effects such as improved sleep, brighter mood, and less pain.

The final study shows that patients that received traditional or simulated acupuncture felt considerably better than the group that had only received care following ordinary routines. The difference, 37 percent compared with 63 percent of nauseous patients, is statistically significant. On the other hand, there was no difference between the two acupuncture groups.

In other words, the needles did not have to penetrate the skin for patients to feel better and believe that acupuncture helped them with their nausea. I can't wait for this work to be published in the peer-reviewed medical literature, because I would like to see what the methodology was and the detailed findings. The good news is that, unlike the writers of the meta-analysis, the investigator in this trial understands its results:

The effects therefore seem not be due to the traditional acupuncture method, as was previously thought, but rather a result of the increased care the treatment entails. Patients could converse with the physiotherapists, they were touched, and they had extra time for rest and relaxation.

And that's why patients undergoing acupuncture think that it helped them. It's the placebo effect, combined with the human touch that is all too often missing in modern medicine. Instead of rationalizing this result to claim that acupuncture really does do something after all, which is sometimes taken to amusing extremes in studies where sham acupuncture ends up producing apparently better results than acupuncture (as happens not too infrequently these days), the explanation is clear cut. Acupuncture is an elaborate placebo, only with a small but real risk of complications, such as infection or even occasionally pneumothorax, that can occur when needles are stuck into the body. Placebos shouldn't carry risk, even risk that small.

REFERENCE:

Yanxia Sun, Tong J. Gan (2008). Acupuncture for the Management of Chronic Headache: A Systematic Review Anesthesia & Analgesia, 107 (6), 2038-2047 DOI: 10.1213/ane.0b013e318187c76a

Categories

More like this

I think my title says it all: Can we finally just say that acupuncture is nothing more than an elaborate placebo? Can we? The reason I ask this question is because yet another large meta-analysis has been released that is entirely consistent with the hypothesis that acupuncture is a placebo.…
I have to hand it to acupuncture mavens. They are persistent. Despite numerous studies failing to find any evidence that acupuncture is anything more than an elaborate placebo whose effects, such as they are, derive from nonspecifice mechanisms having nothing to do with meridians, qi, or "…
I sense a disturbance in the skeptical blogosphere. It is something that I half-expected, but, even so, it nonetheless somewhat surprised me when it arrived in the form of comments on my blog and e-mails from readers, fellow supporters of science-based medicine, and others asking me what I thought…
Never let it be said that Orac doesn't give the people what they want. Well, most of the time, anyway. What I'm referring to is a recent German study about acupuncture for low back pain that's been making its way around the media. I had actually been planning on commenting about it yesterday, but…

All of the studies comparing acupuncture directly to medicine were non-blind?! Who on earth designs such a study? It's not like it takes a genius to figure out how to make such a study properly double-blind: everyone gets a pill, everyone gets poked with needles, whether or not there's medicine in the pill or the needles poke in the magic places.

Thanks, nice analysis.

One thing I wondered: how many of the studies included in the meta-analysis came from countries (i.e. China) where there are virtually never any negative studies involving acupuncture? This point was discussed in Bausell's "Snake Oil Science", in which it was revealed that in many Western nations the percentage of published papers finding acupuncture better than placebo was 50% or less, while in China positive findings approached 100%.

By Dangerous Bacon (not verified) on 03 Dec 2008 #permalink

In discrete mathematics, the complement of something is its opposite - the complement of true is false, the complement of 1 is 0. So CAM is the complement of medicine...

I think that you hit the most important bit right at the end.

I used to suffer chronic migraines (what "fixed" it for me was 15 years of allergy shots, which reduced the frequency and severity from "blinding pain 4 days a week" to "sever pain once or twice a month").

One of the best things that I found with my headaches was that spending some quiet time in the dark, cutting off as much sensory input as possible, could provide a lot of relief - it often accomplished more than any medication that I've tried.

I think the key to these accupuncture studies and their relative success has nothing to do with needles. The accupuncture sessions are a quiet, relaxed time, in a peaceful
place, with a soothing person helping you to relax. That is enough to create a significant effect - especially since they include tension headaches in the mix.

By Mark C. Chu-Carroll (not verified) on 03 Dec 2008 #permalink

Patients could converse with the physiotherapists, they were touched, and they had extra time for rest and relaxation.
I like the therapeutic effect of the touch and interaction with the therapist idea; now if I could just come up with a way to double blind a study of the effect of medically prescribed sex, paid for by the health insurance ...

I keep seeing releasing endorphins being put forwards as an idea of how acupuncture works, but couldn't endorphines be released by a thousand other methods? Shouldn't that idea be considerd anti-acupuncture, as the effect of the needle ritual could in fact be produced by hitting the patient across the face with a live fish?

Another issue in headaches not mentioned is the relatively high placebo response rate in headache patients (30-40%). This shows up fairly consistently in the placebo controlled trials of the triptans (Imitrex and others). This makes evaluating trials with poor randomization (or no placeo) almost impossible.

Orac, it seems to me that you've been using roughly the same 'disclaimer' (for lack of a better term) at the begining of your articles on accupuncture, wherein you state that your initial not-total-aversion to the procedure. I'm honestly curious, why do you think this is necessary?

If you look for the proof that many forms of competent Energy Healing work you will find them in abundance.

If you look for reasons why they can't of shouldn't work; you will find these as well. However now enough scientific studies have found the evidence and discovered the science; if you search for these you will find this evidence. You are probable sceptical of quantum physics.

There is abundant of evidence, that current conventional medical treatments fail in the majority of cases and are adjusted by trial and error searching for the correct treatment; yet this is accepted because it is the norm.

If an Energy Healer does not pick up on all the subtle causes they are instantly judged as or dismissed as no good or a fraud.

You should ask and accept the knowledge and experience of those who have been healed by these Energy Healing modalities.

With Love
Ian Stone - Founder of HEART Energy Healing System,
Human Energy Assessment Release Treatments
Metaphysical Institute
Metaphysical Institute Blog

Thanks for the link, posting that on this site really made my day. I can't wait for Orac's response to that (he's probably on page 4 of it right now).

You should ask and accept the knowledge and experience of those who have been healed by these Energy Healing modalities.

Wouldn't that be anecdotal evidence as thus not scientific? Sounds like a woo to me. Perhaps a Friday dose of woo?

It occurred to me recently that another reason not to be a shruggee about acupuncture and chiropracticy is that it potentially drives up your health insurance premiums. Think about it. Every year, the medical insurance premiums I pay out of my paycheck go up because the insurance company does an analysis of the previous year's payouts, and adjusts our premiums to ensure they're still making a good profit off of us. Many insurance companies are now covering various forms of CAM such as acupuncture and chiropractic care; the insurance company is going to recoup the costs of providing monthly/ weekly CAM acupuncture &/or chiropractic treatments to people in my plan who avail themselves of such services, but the cost is spread out across the whole group.

Let's say CAM gets popular enough that 10% of the plan members go once a month to some type of CAM practitioner, at a cost of $100/visit. At a 1000 person firm, that's $120,000/ year the insurance company pays out for CAM that they will recoup from the group via premiums. Spread out over 1000 employees, that's $120/ year I have to pay for other people's CAM. So now it's not a question of "why do I care if someone pisses their own money away on bogus, fake medicine?", it becomes a question of "why do I care if someone pisses away some of my money away on bogus, fake medicine?".

By Karl Withakay (not verified) on 03 Dec 2008 #permalink

Ian Stone asks us to accept "the knowledge and experience of people who have been healed by these Energy modalities."

I ask in return:
How do we know the "energy healing" worked, compared to them regressing to the mean, getting well on their own or experiencing placebo effect?

And what of those who were not healed by energy healing?
How many negative outcomes were there?

The dead do not complain that their "energy healing" didn't work, do they?

Anecdotes may be useful for generating a real study, but pretending they are actual evidence just won't work.

It really burns me that here in Canada, where those who have private insurance can get reimbursements for therapeutic massage from licensed practitioners, that somehow Reiki is included as a legitimate form of massotherapy. Imagine, your insurance company paying $80 for someone to stand around & wave their hands on you while you lay o a table for an hour.
They are paying for a nap with a babyminder.

Good point, Karl!

But DLC you don't understand! It's Quantum! You can know what the treatment was or if it worked, but you can't know both at the same time!

Seriously though the moment someone cites quantum phyisics for a medical treatment you know you're dealing with someone that has completly lost all contact with reality.

So can we look forward to a proper double-blind trial of the slap-with-a-live fish therapy?

What would the control group get, a slap with a dead fish?

By Militant Agnostic (not verified) on 04 Dec 2008 #permalink

Ian,

What energy? Can it be detected and measured? Quantified? What is the energy doing? How are you affecting it?

I would love for your claims to be true, because it would open a massive field for research and scientific inquiry and we would learn a huge amount about life and its nature.

But without evidence, nobody here will buy into your claim.

What studies, published in peer reviewed journals are you referring to for evidence?

By Robster, FCD (not verified) on 04 Dec 2008 #permalink

Robster, he has no reason to answer you. He knows that the quantum conciousness is preparing an eternal negetive energy feild for people who ask such insolent questions!

For a moment I thought that Ian was a Poe. I mean, c'mon, "You are probable sceptical of quantum physics."? Then I realized that there were 3 links, and so it would have had to go through moderation. Orac, thanks for feeding this woo-monger to we piranhas of science. Om nom nom. Although I did almost give myself a nosebleed from the facepalm. :P

PS: On one of my credit card bills the envelope has an offer to get a healing magnetic bracelet for only $19.95! -_-;; I wondered how they could get away claiming that it promotes healing, but then I looked more closely. They never claim anything except what the thing is made of. The statement about its healing effects was in the passive voice. "It is said that" [insert woo therapy claim here]. "Who says that? " "Certainly not us, but y'know people..." e_e The thing is pretty, but I wouldn't buy it because of the woo mongering. Stupid credit card. Now I have to pay it and cancel it on principle. Harumph.

http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6SYP-4SDPX46-…

Just one 2008 study conducted by a joint project between Harvard Medical School, MIT + Harvard's Martinos Center for Biomedical Imaging, the director of Neurotherapeutics at Massachusetts General Hospital using fMRI and PET scans to monitor brain activity.

"Statistical analyses with paired t-tests showed that the reduction in pre- and post-treatment pain ratings was significantly greater in the acupuncture group when compared to the placebo group(P < 0.05)."

"A priori brain regions demonstrating greater [11C]diprenorphine binding decreases (associated with greater endogenous opioid release) in the verum group versus the placebo group were observed in the right medial OFC, leftmedial PFC, right thalamus, and right insula. A greater binding increase in the verum group versus the placebo group included bilateral insula, right medial PFC/ACC, left OFC, and right brainstem. Common fMRI and PET changes (increased fMRI signal and decreased [11C]diprenorphine binding associated with greater endogenous opioid release) during verum acupuncture were seen only in the right medial OFC. There were no common fMRI and PET changes during placebo acupuncture."

Check it out for yourself.

http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6SYP-4SDPX46-…

Just one 2008 study conducted by a joint project between Harvard Medical School, MIT + Harvard's Martinos Center for Biomedical Imaging, the director of Neurotherapeutics at Massachusetts General Hospital using fMRI and PET scans to monitor brain activity.

"Statistical analyses with paired t-tests showed that the reduction in pre- and post-treatment pain ratings was significantly greater in the acupuncture group when compared to the placebo group (P < 0.05)."

"A priori brain regions demonstrating greater [11C]diprenorphine binding decreases (associated with greater endogenous opioid release) in the verum group versus the placebo group were observed in the right medial OFC, leftmedial PFC, right thalamus, and right insula. A greater binding increase in the verum group versus the placebo group included bilateral insula, right medial PFC/ACC, left OFC, and right brainstem. Common fMRI and PET changes (increased fMRI signal and decreased [11C]diprenorphine binding associated with greater endogenous opioid release) during verum acupuncture were seen only in the right medial OFC. There were no common fMRI and PET changes during placebo acupuncture."

Check it out for yourself.

sorry about that.

These were studies comparing acupuncture with medication, and, as such, they were almost certainly not blinded.

Yep, that and the apparent dependence of self-report data, which are notoriously subject to demand characteristics, which have been documented for about a half century (Orne, 1962).

Sigh.

Orne, M.T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776-783.

Thanks so much Orac! I've been looking all over the place for analysis on this dubious study. It was mentioned this week in the Science Times podcast, so hopefully you'll see a spike in readers of this article who want to see your point of view.