That'll teach 'em for using an actual valid placebo control

I almost feel sorry for acupuncturists these days. Almost.

Well, not exactly. Clearly, given the infiltration of woo into academic medicine, acupuncturists are in demand even in the most allegedly "science-based" of academic medical centers. After all, acupuncture is what I like to refer to as "gateway woo," an unscientific placebo-based therapy that has somehow come to be viewed as seemingly respectable, as though there's something to it. It's not hard to see why acupuncture has achieved this status. Indeed, there was a time when I, the arch-skeptic, the guy who has built up one of the top skeptical blogs out there, the person who three years ago took over the Skeptics' Circle willingly and enthusiastically when its founder, a blogger known as Saint Nate, decided that personal reasons prevented him from continuing to maintain his blog or the Circle, was less skeptical of acupuncture than I am now. I know, I know, it's hard to believe, but it's true. Don't get me wrong, though. I never for a minute considered that the whole rigamarole about "unblocking" or "redirecting" the flow of that mystical life force known as qi had anything to do with whether or not acupuncture did or did not have efficacy treating disease or other conditions. That was clearly a holdover from the pre-scientific medicine times in which most beliefs about the causes of disease involved either the wrath of the gods or vitalism, the latter of which is, when you come right down to it, the philosophical basis upon which many "complementary and alternative" (CAM) modalities are based, especially the so-called "energy healing" modalities, such as reiki, therapeutic touch, and, of course, acupuncture.

Still, because unlike so many other "energy healing" methods, acupuncture involved an actual physical action upon the body, namely the insertion of thin needles into the skin to specified depths, it did not seem to me entirely unreasonable that there might be some sort of physiological effect that might produce a therapeutic result. And I'm apparently not unique, if the proliferation of acupuncture offerings in ostensibly "academic" medical centers is any indication. Lots of otherwise scientific and skeptical academic physicians seem willing to give acupuncture a bit of a pass because it seems to be more of a physical intervention than other forms of "energy healing." In any case, that's what I used to think until I actually did the unthinkable, until I actually started paying attention to the published scientific literature on acupuncture. That's when I started to realize that "there's no 'there' there," if you know what I mean. Badly designed studies with either no controls or utterly inadequate controls tend to be the norm in the acupuncture "literature" (and I use the term "literature," as in "scientific literature," loosely).

Moreover, acupuncture is touted as having value for conditions and procedures for which there is no plausible (or even mildly plausible) physiological mechanism by which it could be reasonably postulated to have an effect. Arthritis, allergies, headache (especially if treated with laser acupuncture), back pain, gastroesophageal reflux disease (GERD), Parkinson's disease, post-operative nausea, hot flashes in breast cancer patients caused by the anti-estrogen drugs they have to take, infertility, it doesn't matter. Seemingly acupuncture can do it all; it's the Swiss Army knife of CAM therapies. Moreover, the "explanations" given to explain "how acupuncture works" seemed increasingly less plausible to me. Most of these explanations involve counterirritation or the release of opioids, and I've had an increasingly hard time believing either of those explanations, even if these mechanisms do turn out to be at play, they could have anything other than nonspecific effects, with no mechanism to explain how acupuncture could possibly do all things attributed to it. One rule of skepticism, especially when it comes to medicine, is that you should be very skeptical of modalities that are touted to be useful for a wide variety of medical conditions that have very different pathophysiology. From a strictly scientific view, such modalities are alway highly suspect. Oddly enough, though, it seems that pretty much every "complementary and alternative medicine" (CAM) intervention can be described that way.

More amusingly, odd thngs seem to happen when when rigorous placebo controls are introduced to studies of CAM therapies, and that's that sometimes the placebo control does better than the "true" acupuncture; i.e., the evidence for acupuncture, taken in its totality, is completely compatible with placebo effect.

As Harriet Hall put it in her excellent analysis of a study purporting to show that acupuncture is useful for GERD:

This study falls into the category of what I call Tooth Fairy science. You could measure how much money the Tooth Fairy leaves under the pillow, whether she leaves more cash for the first or last tooth, whether the payoff is greater if you leave the tooth in a plastic baggie versus wrapped in Kleenex. You can get all kinds of good data that is reproducible and statistically significant. Yes, you have learned something. But you haven't learned what you think you've learned, because you haven't bothered to establish whether the Tooth Fairy really exists.

One area that acupuncturists keep needling away at is infertility. Why, I don't know. there's no reasonable physiological rationale to conclude that acupuncture would have any effect whatsoever on infertility. Nonetheless, somehow in the CAM community it's become conventional wisdom that acupuncture can somehow increase the chance of success for couples undergoing in vitro fertilization (IVF). Indeed, early this year I wrote about a meta-analysis that concluded that acupuncture did actually increase the success rate of IVF and why it did not show what its authors thought it showed. Tooth fairy science, indeed.

Now comes yet another study being publicized in the media that examines once again the question of whether sticking needles into women before they undergo IVF can increase their chances of conceiving. I could not help but be extremely amused by the title given to the ScienceDaily story about it: Placebo Acupuncture Is Associated With Higher Pregnancy Rate After IVF Than Real Acupuncture. You have to love a headline like that, and opening paragraphs like this:

A study comparing the effects of real and placebo acupuncture on pregnancy rates during assisted reproduction has found that, surprisingly, placebo acupuncture was associated with a significantly higher overall pregnancy rate than real acupuncture.

The study, published online in the journal Human Reproduction, looked at real and placebo acupuncture given on the day of embryo transfer in 370 patients in a randomised, double blind trial (where neither the patients nor the doctors knew which treatment was being given).

The researchers found that the overall pregnancy rate (defined by a positive urinary pregnancy test) for placebo acupuncture was 55.1%, versus 43.8% for the real acupuncture.

I bet that result caused some head-scratching. Or maybe even some wailing and gnashing of teeth. Or at the very least some confusion.

This reminds me of a study not too long ago that found a similar--shall we say?--disconcerting result in which acupuncture was studied for its ability to relieve arm pain from repetitive use injuries, which was funded, amusingly enough, by the National Center for Complementary and Alternative Medicine (NCCAM). Naturally, I had to go straight to the source. Fortunately, you, too, can do the same because the manuscript is available online for free for everyone, unlike most such papers, which are hidden behind a subscription firewall, not easily accessible by any but academics. The study, coming out of China, is entitled A randomized double blind comparison of real and placebo acupuncture in IVF treatment, and its lead investigator is Dr. Ernest Hung Yu Ng, Associate Professor in the Department of Obstetrics and Gynaecology at the University of Hong Kong (People's Republic of China. So, let's take a look at this study.

The hypothesis, not surprisingly, is that acupuncture could improve the efficacy of IVF and result in a higher pregnancy rate. The specific acupuncture regimen consisted of two 25-minute sessions, one before embryo transfer and one after embryo transfer was completed. Patient were informed of the study on the day of transvaginal ultrasound-guided oocyte retrieval (TUGOR), at which time a practitioner of traditional Chinese medicine (TCM) "diagnosed" thusly subjects who agreed to participate in the study:

A registered traditional Chinese medicine (TCM) practitioner (E.W.S.S.) categorized these patients' conditions by the four diagnostic methods: observation, auscultation, interrogation and palpation according to the TCM principles (Maciocia, 1998). They were classified into the related syndromes which included Kidney Yang/Yin deficiency, Liver Qi stagnation with blood stasis, Spleen Qi deficiency with Phlegm and combination of those syndromes.

I wonder if spleen qi deficiency requires a splenectomy or if we can prescribe more yang or yin for a Yang/Yin deficiency. Sorry, couldn't resist.

Patients were randomized to receive either "real" acupuncture of "sham" (a.k.a. placebo) acupuncture, and the placebo acupuncture needle was described thusly:

The Streitberger's placebo needles (Streitberger and Kleinhenz, 1998; Asiamed, Pullach, Germany; 0.30 x 30 mm) were used. The appearance of the placebo needle was the same as the needle used in the real acupuncture group. The placebo needle was not fixed into the copper handle and the tip of the needle was blunt. When it was pushed forward against the skin, the needle slid into the handle and the whole needle appeared shortened. This also gave patients a pricking penetration sensation. To place the placebo needle in position, the Park's placebo device (Park et al., 1999; Dong Bang AcuPrime, Exeter, UK) was used. The same acupoints and procedures as in the real acupuncture group were applied.

Overall, the design of the trial appears pretty sound. It was randomized and double-blinded to the extent possible with a treatment like acupuncture, and, as all good double-blind studies do, patients were asked afterward to guess whether they were in the treatment group or the placebo group in order to assess the quality of blinding. The study was powered to detect a 15% increase in the rate of pregnancy based on the clinic's usual success rate of 35%, which required 370 patients. (Note how it was assumed that acupuncture would increase the success rate of IVF.) Overall, on first glance, the two groups seemed to be comparable, although I was a bit disturbed to see that the statistics showing that they were, in fact, comparable in all the parameters measured were not included in the table summarizing the two groups. My other quibble is that I was a bit surprised that the primary outcome the investigators chose to look at was pregnancy rates as assessed by urine pregnancy tests. As anyone who's ever undergone IVF knows, not all positive pregnancy tests result in an ongoing pregnancy or a live birth, not by a longshot. When discussing the success rate of IVF, what patients are interested in is the live birth rate. Anything less is a failure. Looking at the table of the results, I think I see why the authors were more interested in the overall pregnancy rate. It is the only outcome measure the produced a statistically significant difference between the two groups.

Let's take a closer look. The overall pregnancy rate (a positive urine pregnancy test) was 43.8% for the "real" acupuncture group and 55.1% for the placebo acupuncture group (p =0.038). However, the numbers become less and less different as we go deeper into measurements of pregnancy that matter more. For example, generally after a round of IVF that results in a positive pregnancy test, the first big hurdle afterward is the 4 week ultrasound that is done to see if there is a fetal sac, and ongoing pregnancy is usually defined as a pregnancy resulting in a fetal heartbeat detectable by ultrasound by 8 weeks. The definition of live birth is fairly self-explanatory, although for purposes of this study it was defined as a live birth occurring after 24 weeks. In any case, clinical pregnancy rates for "real" acupuncture and placebo acupuncture were 38.9% and 49.2%, respectively (p=0.059); ongoing pregnancy rates, 31.9% and 40.5% (p=0.105); live birth rates, 29.7% and 38.4% (p=0.100). There was similarly no difference between implantation rates, miscarriage rates, or ectopic pregnancy rates. In fact, this study is not a study showing that acupuncture does anything to improve or harm success rates for IVF. It's a negative study. There is no statistically significant difference between the outcomes in the only measure that patients really care about, live births. Indeed, there was in addition no difference between the groups in endometrial or subendometrial vascular density, nor were there any statistically significant differences in serum cortisol levels.

That doesn't stop the authors from doing all sorts of contortions over the results in the Discussion section. For example, the authors did find a significant decrease in serum cortisol after both "real" and placebo accupuncture, as well as an improvement in uterine blood flow parameters. To me this suggests decreased stress in both groups more than anything else, entirely compatible with the placebo effect. What's more interesting is how the authors try to explain their result after first expressing their surprise in the discussion:

We found a significantly higher overall pregnancy rate following placebo acupuncture when compared with that of real acupuncture. There was a trend of higher rates of clinical pregnancy, ongoing pregnancy, live birth and implantation in the placebo acupuncture group, although the differences did not reach statistical significance. Surprisingly, significant changes in endometrial and subendometrial vascularity, serum cortisol concentration and the anxiety level were documented following both real and placebo acupuncture.

Why surprising? After all, this is exactly what would be predicted by the hypothesis that acupuncture acts primarily as a placebo; specifically, no differences in primary outcomes (ongoing pregnancy rates and live birth rates) but possibly differences in measurements related to patient anxiety, such as cortisol levels and uterine blood flow. No surprise there, unless you believe in acupuncture. The authors do go on to make a point that I actually somewhat agree with. Specifically, they lament that they didn't include a no-acupuncture group as a control. Of course, the problem with adding such a group is that it would make it impossible to blind the study subjects in the no-acupuncture control group to their experimental group assignment and thus render this study no longer double blind. That is not an insignificant problem. However, it might be interesting to do in order to estimate the magnitude of the placebo effect.

More amusingly, Dr. Ng went into some contortions to try to explain his "surprising" result in an interview. In essence, he likened placebo acupuncture to acupressure and then waved his hands thusly:

Dr Ng gave two possible explanations for the results: "Placebo acupuncture is similar to acupressure and therefore is good enough to improve the pregnancy rate. Or else, it's possible that real acupuncture may, in some way, reduce the pregnancy rate of acupuncture.

"So far there is no evidence that real acupuncture would adversely affect IVF outcomes because, in a previous meta-analysis of several acupuncture studies, the pregnancy rate was higher in the acupuncture groups than in the control groups. However, we cannot draw a firm conclusion about this from our current study as we did not compare the two groups with a third control group patients who received neither forms of acupuncture. Further studies should be conducted to compare placebo or non-invasive acupuncture and controls without acupuncture."

You can sense the desperation. At least, I can, especially looking at his rationalization for the latter explanation, namely that "real" acupuncture somehow reduces the pregnacy rate. Of course, such a result would in fact be interesting if true, because it would indicate a physiological effect due to "real" acupuncture distinguishable from placebo acupuncture. That it would be a negative one wouldn't matter. Any physiologic effect attributable to acupuncture would be of interest, whether positive or negative.

Unfortunately for woo-meisters and acupuncture boosters, the best explanation for this result is that there is no difference between the two groups (there isn't, in fact) combined with a random finding of a seemingly "statistically significant" difference in one parameter, namely the overall pregnancy rate as measured by the rate of positive urine pregnancy tests in each subject group. Combine that with a hypothesis with a very low prior probability, which significantly increases the likelihood of a false positive result, as John Ioannidis has shown, and there's no real difficulty in explaining Dr. Ng's result. The only thing unusual in this case is that the false "positive" result was a "positive" in the "wrong" direction.

Alas for poor Dr. Ng, he was tripped up by the vagaries of comparing two different (and almost certainly equal) placebos against each other. Do such studies long enough, and inevitably sometimes the "wrong" placebo will win. I bet this study will teach Dr. Ng one lesson, though.

I bet he'll never make the mistake again of including the best possible placebo control in any future study of acupuncture that he undertakes.

REFERENCES: