Yet another acupuncture meta-analysis: Garbage in, garbage out

Ever since I started paying attention to it, acupuncture has, at least until recently, inspired ambivalence more than anything else in me. As a skeptic and science-based physician, I found it very easy to dismiss utter quackery like homeopathy or the various "energy healing" modalities, such as reiki or therapeutic touch strictly on the science alone. After all, homeopathy is based on magical thinking more than anything else, specifically the concepts of "like cures like," the concept that dilution with vigorous shaking can make a remedy stronger, and the idea that water has "memory" all are very implausible on the basis of well-established science alone. Similarly the concept that healers can manipulate living human "energy fields" (or life energy, also known as qi) for therapeutic effect is also incredibly implausible, given that no scientist has ever detected qi or shown that healers can manipulate it for therapeutic intent or any reason.

When I first became interested in "alternative medicine"--excuse me, I mean "complementary and alternative medicine" (CAM)--I viewed acupuncture somewhat differently. No, I never bought the traditional explanation that sticking thin needles into the skin somehow alters the flow of qi in order to induce a therapeutic effect. That is no more plausible than reiki or therapeutic touch. However, there are needles breaking the skin in acupuncture. It was, at least to me, not entirely implausible that that might have some sort of physiologic effect. Then I had to go and ruin that lovely kumbaya feeling towards CAM by actually going and looking at the scientific literature on acupuncture, as I've documented on this blog on multiple occasions (for example: 1, 2, 3, 4, 5). When I actually bothered to do that, I soon realized that the evidence that acupuncture is anything more than a highly elaborate placebo is shockingly thin. More like nonexistent, actually. So what do you do if you're a CAM believer but can't find studies that strongly support the efficacy of your favorite CAM modality, like acupuncture.

Why, meta-analysis, of course!

And here comes another one, this time for acupuncture to treat headaches:

WASHINGTON (Reuters) - Acupuncture works better than drugs like aspirin to reduce the severity and frequency of chronic headaches, U.S. researchers reported on Monday.

A review of studies involving nearly 4,000 patients with migraine, tension headache and other forms of chronic headache showed that that 62 percent of the acupuncture patients reported headache relief compared to 45 percent of people taking medications, the team at Duke University found.

"Acupuncture is becoming a favorable option for a variety of purposes, ranging from enhancing fertility to decreasing post-operative pain, because people experience significantly fewer side effects and it can be less expensive than other options," Dr. Tong Joo Gan, who led the study, said in a statement.

"This analysis reinforces that acupuncture also is a successful source of relief from chronic headaches."

Before I continue on to discuss the actual meta-analysis itself, let me test what you have learned from reading this blog on a regular basis (assuming, of course, that you've been a regular reader for a while; if you're a newbie ignore this). Can you tell me what is wrong with the above result and why it is not particularly impressive? No cheating and reading ahead yet. Think and see if you can identify what the problem is. Once you've done that, feel free to click to read below the fold to see if you found the same problem that I did.

That's right. These were studies comparing acupuncture with medication, and, as such, they were almost certainly not blinded. Hello, placebo effect! What this result probably says, more than anything else, is how frustratingly ineffective medications are in bringing relief to suffers of chronic headaches. What was the other finding? Let's see:

Writing in Anesthesia and Analgesia, they said 53 percent of patients given true acupuncture were helped, compared to 45 percent receiving sham therapy involving needles inserted in non-medical positions.

"One of the barriers to treatment with acupuncture is getting people to understand that while needles are used, it is not a painful experience," Gan said. "It is a method for releasing your body's own natural painkillers."

Notice how much closer the groups have become when comparing true acupuncture with sham acupuncture. Goodbye, placebo effect, at least when the groups are closer to being blinded. Of course, the contrarian (or someone who just doesn't like me) might say: See, see! There's still a difference between true acupuncture and sham acupuncture! That means there must be something to acupuncture!

Not so fast there, pardner. Let's take a look at the actual study by Sun and Gan in the latest issue of Anesthesia and Analgesia, right off the presses. As is usually the case, there's a lot less to this than meets the eye.

To do their study, authors started out by doing a typical sort of search for randomized clinical trials of acupuncture for headache. Their strategy is summarized in the figure below:

Studies were rated on a quality scale defined thusly:

One point was assigned to the trials in which the patient and assessor blinding were stated and two points were given to the trial in which patients could not distinguish the group allocation by credibility or guessing test and the assessor blinding was described adequately. The maximum score was 7; trials with a score of 4 or more points were considered high quality.

One major problem that becomes apparent right away is the extreme heterogeneity of the trials included:

The majority were conventional two-arm comparison trials: true acupuncture versus sham acupuncture (16 trials), acupuncture versus medication treatment (8 trials), and acupuncture versus physiotherapy (2 trials). Five trials have three arms, acupuncture, sham, and waiting list (one trial), acupuncture, sham, and usual medication (one trial), acupuncture, behavioral program, and waiting list (one trial), acupuncture, physiotherapy, and relaxation training (one trial), acupuncture, and transcutaneous electrical nerve stimulation (TENS) at acupuncture points and medication (one trial).

A set formula of acupuncture was performed in 11 trials and individual acupuncture treatment was used in 16 trials depending on the type and distribution of pain based on diagnosis of traditional Chinese medicine. Semi-standardized acupuncture was used in four trials in which basic points were applied for each session and additional points were selected individually based on the pain topography. Three trials used electrical needling acupuncture. The treatment session was, on average, 10 sessions (range, 6-16) during a mean of 8 wks (range, 4-24 wks).

Heterogeneity is, of course, always a major problem in meta-analyses of any kind looking at the effect of at therapeutic intervention, but this set is still heterogeneous even for a meta-analysis. Another problem with is the quality of the studies. The authors somewhat disingenuously state that that 14 of the studies scored 4 or more on their scale estimated quality and that five of them scored 7/7. Sounds pretty good, right? What they fail to mention, except for listing without comment in Table 1, is that eight of the studies scored only 1/7 and two of the studies scored 2/7. That means that blinding was not well addressed at all in these studies. In other words, one third of the studies were rated as only 1/7 or 2/7. That's not good, and it's unclear to me why these studies were included. Not surprisingly, these studies also tended to be the "most positive" studies favoring acupuncture, with the highest relative risk ratios.

Surprise, surprise.

Also, when one looks at the results in more detail, rather than just reading the abstract, the results become even more underwhelming. We're talking relative risks in the 1.2 range for the most part, with confidence intervals that come close to 1.0. For migraines, there was no difference in the response rate. For none of the headaches was there a significant early difference in headache intensity between control and acupuncture at the early followup time point, and for headache frequency there was no difference between any of the groups. The authors state that restricting the analysis to high quality trials did not significantly alter the results other than decreasing the measures of variation, but with such heterogeneous studies, with such variable outcome measures and design, I have a hard time believing that a relative risk of only 1.19-1.33 range for most of the groups means much of anything. The sole exception, not surprisingly, was the acupuncture versus the medication comparison, where the relative risk was reported as 1.80 in favor of acupuncture, but that's the very group of studies where there was no blinding at all; i.e., the very studies where the placebo effect was most likely to be in play. Also, in a meta-analysis of this sort, a relative risk as low as this and differences between the experimental and control groups could well be explained by publication bias, which, as you may recall, is the phenomenon where negative studies tend not to be published while positive ones are more likely to be published.

In the discussion, the authors acknowledge some, but not all, of these limitations and largely poo-poo them, concluding that acupuncture is better than sham acupuncture and also better than pharmacological treatment of chronic headaches. Indeed, their discussion smacks of a lot of rationalization and handwaving:

Complete blinding of the control group has always been a challenging issue in acupuncture clinical trials. Different strategies have been adopted to provide the best blinding possible. Sham control, often regarded as approaching the ideal strategy, provides the control subjects with the impression closest to true acupuncture but without any real analgesic effects. The most often used sham is superficial needling in which needles are inserted superficially at nonacupuncture points. Ten of 14 studies in this analysis using this method found positive results in response rate, but no significant difference between acupuncture and sham in headache frequency and intensity. However, it has been argued that needling at nonacupuncture points may produce similar physiological effects. Biochemical evidence suggests that stimulation of nonacupuncture points may also result in the release of endorphins and hence produce analgesia. As such, the treatment effects of acupuncture might be under-estimated.55,56 Alternative methods without penetration included a blunt needle against the skin or tapping a cocktail stick against a bony surface to create the pricking sensation but without producing analgesic effects.

I find it very telling that many of the studies using sham needling showed no difference in headache frequency and intensity, which is what sufferers of chronic headache really care most about. Once again, this appears to be the same as the usual case in CAM modalities: The better the study design and the better the controls, the less likely there is to find any significant treatment effect. Indeed, one wonders if the alleged treatment effect is even clinically significant (namely, that patients and their doctors notice it an consider it worth the trouble it takes to achieve it). Once again, this meta-analysis seems to be consistent with the hypothesis that acupuncture for headaches is unlikely to be more than an elaborate placebo.

Which brings me to a second acupuncture study reported this week. Unfortunately, this study didn't get as much attention as the meta-analysis. On the other hand, it isn't published in the peer-reviewed literature yet. Even so, I think it's worth mentioning because it is so deliciously appropriate. Why? Because it found that the benefits of acupuncture do not require that the needles actually penetrate the skin:

The acupuncture study of 215 patients who were undergoing radiation treatment in the abdomen or pelvic region chose by lot one of these two acupuncture types.

109 received traditional acupuncture, with needles penetrating the skin in particular points. According to ancient Chinese tradition, the needle is twisted until a certain 'needle sensation' arises. The other 106 patients received a simulated acupuncture instead, with a telescopic, blunt placebo needle that merely touches the skin.

The acupuncture was performed by physiotherapists two or three times a week throughout the five-week radiation period.

Afterwards 95 percent of the patients in both groups felt that the acupuncture treatment had helped relieve nausea, and 67 percent had experienced other positive effects such as improved sleep, brighter mood, and less pain.

The final study shows that patients that received traditional or simulated acupuncture felt considerably better than the group that had only received care following ordinary routines. The difference, 37 percent compared with 63 percent of nauseous patients, is statistically significant. On the other hand, there was no difference between the two acupuncture groups.

In other words, the needles did not have to penetrate the skin for patients to feel better and believe that acupuncture helped them with their nausea. I can't wait for this work to be published in the peer-reviewed medical literature, because I would like to see what the methodology was and the detailed findings. The good news is that, unlike the writers of the meta-analysis, the investigator in this trial understands its results:

The effects therefore seem not be due to the traditional acupuncture method, as was previously thought, but rather a result of the increased care the treatment entails. Patients could converse with the physiotherapists, they were touched, and they had extra time for rest and relaxation.

And that's why patients undergoing acupuncture think that it helped them. It's the placebo effect, combined with the human touch that is all too often missing in modern medicine. Instead of rationalizing this result to claim that acupuncture really does do something after all, which is sometimes taken to amusing extremes in studies where sham acupuncture ends up producing apparently better results than acupuncture (as happens not too infrequently these days), the explanation is clear cut. Acupuncture is an elaborate placebo, only with a small but real risk of complications, such as infection or even occasionally pneumothorax, that can occur when needles are stuck into the body. Placebos shouldn't carry risk, even risk that small.

REFERENCE:

Yanxia Sun, Tong J. Gan (2008). Acupuncture for the Management of Chronic Headache: A Systematic Review Anesthesia & Analgesia, 107 (6), 2038-2047 DOI: 10.1213/ane.0b013e318187c76a