Here we go again.
Tuesday night and yesterday, you probably saw it, plastered all over the media, in the newspapers, on ABC, on the radio, in press releases, and around the blogosphere. Yes, it was another bit of science by press release, with news outlets practically falling all over themselves to hype the results of an acupuncture study reported earlier this week at the annual meeting of the American Society for Therapeutic Radiology and Oncology (ASTRO). Leading the pack was ABC News:
A new medical study finds that acupuncture, an ancient form of healing that has been around for thousands of years, is as good as, or better than modern medicine in helping ease the side effects of breast cancer treatment.
The findings, which were presented today at the American Society for Therapeutic Radiology and Oncology's (ASTRO) annual meeting in Boston, suggest that this ancient therapy can give cancer patients a wide range of benefits above modern medicine.
This study suggests nothing of the sort. In fact, like most acupuncture studies, it's riddled with methodological shortcomings that make its results, in essence, meaningless. I'll explain why shortly. But first, let's check out the credulous coverage:
Most women in the study said they saw the same dramatic effect from the acupuncture treatment as Azar did.
"Acupuncture is equal to drug therapy in decreasing hot flashes," said Dr. Eleanor Walker at the Henry Ford Hospital and lead author of the study. And even better, she said, it has no side effects.
Yes, this study originates in my hometown, and Dr. Walker has been all over the news serving up such bon mots such as this:
Doctors have been prescribing the antidepressant venlafaxine -- sold under the name brand Effexor -- for off-label use in alleviating hot flashes, Walter said. But oral medications are difficult for some breast cancer patients to keep down because of the nausea associated with chemotherapy, and Effexor has a long list of side effects, including constipation.
"My patients were complaining," Walter said. "They were sick of taking pills. Wasn't there something I could do?"
Don't you just love the false dichotomy? If patients are sick of taking pills or don't like the side effects of the pills, then the answer must be woo, of course! It's so obvious! It also never fails to puzzle me that patients who hate taking a couple some pills are more than willing to inconvenience themselves to go to a clinic three times a week to have needles stuck into them for a half hour or an hour. I never could figure that out. Be that as it may, into the picture comes a familiar face, whom I've discussed before, reappearing to pontificate piously:
Barrie Cassileth, chief of integrative medicine services at Memorial Sloan-Kettering, said acupuncture can benefit most patients with "an open mind."
"I think many more patients should try acupuncture," Cassileth said. "It is easy. It's pleasant. It's inexpensive. There are no risks involved."
Dr. Cassileth, as you may recall, was the principal investigator for a trial testing whether acupuncture can alleviate hot flashes in women with breast cancer taking anti-estrogen therapy that I discussed in detail in December. In fact, it was about as well-designed and scientifically rigorous a trial as could be carried out for acupuncture in practical terms. Its main shortcoming was that the practitioners were not blinded and that it used only patient reporting rather than a combination of patient reporting of hot flashes and objective measurements. Because it was a pretty well-designed study, not surprisingly, it was also a completely negative study, failing to find any significant effect for acupuncture beyond that of a placebo. You'd think That Dr. Cassileth would have been deterred or less enthusiastic, but apparently not. Here she is saying that acupuncture works if you have an "open mind." Does that mean it doesn't work if you don't have an open mind? No wonder we nasty, close-minded skeptics think acupuncture doesn't do anything more than an elaborate placebo! But I digress. In fact, she is very much still a booster of acupuncture, to the point of praising a study that is markedly inferior to hers.
Well, let's look at the design. I had a heck of a time finding the abstract, which is all there was to go on. Granted, there was enough information in some of the news reports and press releases to tell me why this study is worthless, but I always like to go to the source whenever I can. Fortunately, despite its credulous blogging about the study, Pharmalot provided the actual abstract, entitledAcupuncture for the Treatment of Vasomotor Symptoms in Breast Cancer Patients Receiving Hormone Suppression Treatment. Basically, the design was simple. Dr. Walker compared acupuncture treatments versus venlafaxine (Effexor), the latter of which is an antidepressant sometimes used as an alternative to hormone replacement therapy (HRT) to relieve menopausal symptoms in women in whom HRT isn't a good idea, such as women taking estrogen blocking drugs for breast cancer. Forty-seven women with stage 0 to III breast cancer being treated with either Tamoxifen or Arimidex were randomized to receive either acupuncture or venlafaxine for twelve weeks. Patients logged the severity and frequency of their hot flashes daily for one week prior to the study and during the study, as well as at regular intervals for a year after treatment. Other outcome measures that were assessed before, during, and after treatment included menopause-specific quality of life, general health status (SF-12), Beck Depression Inventory, and the frequency and kinds of side effects.
The results were reported as follows:
Both acupuncture and venlafaxine groups exhibited significant decreases in hot flashes and other menopausal, quality of life symptoms, as well as decreases in depressive symptoms. These changes were similar in the 2 groups, indicating that acupuncture is at least as effective as venlafaxine in reducing vasomotor and other symptoms associated with anti-estrogen hormonal treatment of breast cancer. Additionally, numerous patients treated with venlafaxine reported negative side effects including nausea, dry mouth, headache, difficulty sleeping, dizziness, double vision, increased blood pressure, constipation, fatigue, anxiety, feeling ''spaced out,'' and body jerking during the night. Patients treated with acupuncture experienced no negative side effects. They reported increased energy, clarity of thought, sexual desire, and overall sense of well-being (compared to pretreatment).
So what's the problem? Have you spotted it yet? If you're a regular reader here, I bet you have. Yes, this study is completely unblinded. Patients knew what group they were in. But is that a problem? After all, it's represented that venlafaxine is routinely used for the treatment of hot flashes; so if there was no difference between the two groups then acupuncture must have been doing as well as the drug in relieving these patients' hot flashes, right?
Not so fast, there, pardner.
First, one needs to realize that at best venlafaxine is mildly effective when it comes to menopausal symptoms. It's nowhere near as effective as estrogen, and its relatively weak relief comes at the price of some significant side effects that lead a fair number of women to stop taking the drug because they consider the side effects worse than the hot flashes (which the drug doesn't reduce all that much anyway). Indeed, a meta-analysis from 2006 published in JAMA failed to find a significant effect on menopausal symptoms due to venlafaxine, although the meta-analysis has been criticized and there have been more recent randomized trials suggesting a benefit. The consensus of the studies is that venlafaxine appears to decrease patient self-reported hot flashes but not necessarily physiologically documented hot flashes. That's not such a big deal, given that it's the patient perception of hot flashes that's so troublesome. What's more of a confounding factor in any study looking at therapy-induced menopause is that hot flashes virtually always tend to become less frequent and troublesome regardless of treatment (or lack of treatment), and there tends to be a significant placebo effect in all such studies.
What's a bigger deal is that the abstract did not report the dose of venlafaxine used. Perusing the literature, one thing that became apparent to me is that low dose venlafaxine (37.5 mg per day) appears to be less likely to be effective than high dose venlafaxine (75 mg per day). The problem, of course, is that higher doses are more likely to produce side effects. If this acupuncture study used 37.5 mg a day in the venlafaxine group, it's quite possible that there was little or no effect from the drug. Even at 75 mg, it's not clear that the benefit would be high enough to overcome the potential confounding factors inherent in the design of this study. One study suggested that the drug did little for menopausal symptoms but, because it's an antidepressant, helped women cope with them in their everyday activities, which is a plausible possibility. The bottom line is that, given the small size of the acupuncture study, it's entirely plausible--even likely--for investigators to have found no difference between the venlafaxine group and the acupuncture group, even if there really were a difference favoring venlafaxine just on statistical power considerations alone. Indeed, for such a small trial, the effect of venlaxafine is probably not strong enough or reliable enough to justify only a head-to-head trial comparing it with another modality when the object is to test whether this other modality is effective.
Let me describe for you a much better design for this study. Because the interventions involved are so different, the best way to produce a true double blinding of the study subjects and the investigators would be to randomize the patients into at least three groups:
- Placebo acupuncture plus placebo venlaxafine (the true placebo group)
- Placebo acupuncture plus venlaxafine (the venlaxafine group)
- Acupuncture plus placebo venlaxafine (the acupuncture group)
The sham acupuncture, to be most rigorous, would have to consist of the special retractable needles that blind both the patients and practitioners to the experimental group. The above design would produce real blinding, because the patients would all think they are receiving both treatments and have no idea which of the two is the placebo. If the investigators wanted to get fancy, they could add a no treatment group to gauge the true size of the placebo effect (which in many of studies of treatments for menopausal symptoms is greater than 30%) or an acupuncture plus venlaxafine group, although the latter of the two is probably not necessary unless the investigators have a real interest in testing for additive or synergistic effects of the two modalities. Other designs are possible, but the one chosen for the study as presented at ASTRO is about the worst design possible, other than a nonrandomized design. Indeed, I find it very disappointing that the Susan J. Komen Foundation actually wasted its donors' dollars on this study, given its methodological flaws. Even with its methodological flaws, I'm surprised ASTRO accepted it for a scientific talk (as opposed to a poster). I can only guess the organizers were interested in presenting "provocative" results, which meeting organizers sometimes are.
Finally, over at Science-Based Medicine, Steve Novella makes an excellent point about this study. Normally studies progress from small pilot studies, which may be unblinded or even unrandomized, which, if promising, form the basis for larger, more rigorous clinical studies. Acupuncture study methodology has evolved to the point where there is no real need to do these sorts of pilot studies anymore. There are good placebo and sham acupuncture techniques. It may be difficult to blind practitioners, but there is really no reason why an acupuncture study in which the patients are not blinded should ever be done anymore. There's no point anymore (excuse the pun). Doing small unblinded studies of this sort represents a regression in the usual arc of small pilot studies to larger, better controlled, randomized trials. It's OK to go back to do pilot studies if new observations suggest new applications for acupuncture, but, as demonstrated by the Cassileth study from last year, CAM advocates have been studying acupuncture for hot flashes for years, and acupuncture for hot flashes due to antiestrogen therapy has been studied for quite a while now. Indeed, a recent randomized trial studying this very question concluded that their "results suggest either that there is a strong placebo effect or that both traditional and sham acupuncture significantly reduce hot flash frequency." In other words, the question of whether acupuncture can relieve menopausal symptoms is not a "new" question that requires pilot studies anymore. It's been studied more than enough to be beyond that question; yet such studies persist.
As a breast cancer surgeon, I appreciate just how bad hot flashes can be for breast cancer patients. Not only do they significantly impact quality of life, but they can jeopardize treatment. Indeed, my first breast cancer patient that I treated as an attending found them so unbearable that she ultimately decided she couldn't stand it anymore and decided to stop taking her Tamoxifen. Granted, in her case, the additional benefit of the Tamoxifen was probably relatively small, but there are patients for whom it is much larger. Failure to relieve menopausal symptoms induced by therapy can in some cases increase a patient's risk of recurrence. Consequently, because it is not safe to administer hormones to women with estrogen-responsive breast cancers, it is very important to find better, non-hormonal methods of treating these symptoms. It is thus understandable that even unscientific methods would be tempting, especially because current treatments are only mildly effective and have side effects.
The problem is that practitioners have taken advantage of this hole, as they have for other syndromes and problems for which conventional medicine is not as efficacious as we would like. Worse, they have insinuated their way into academia so that academic physicians who really should know better decide that, no matter how implausible the treatment (in this case, acupuncture) is on a scientific and physiological basis, it "deserves" study simply because it's apparently popular, never mind scientific plausibility and the very high level of improbability bordering on the impossible. This leads to a cycle of pilot studies showing an apparent benefit of this woo or other, followed by larger and better designed studies that fail to confirm that benefit. At that point, rather than doing what scientists do with most "conventional" or drug therapies and abandoning them, they keep at it. Instead of seeing the "pruning" of ineffective or less effective treatments, which are then supplanted by effective or more effective treatments, we see a "regression" back to pilot studies, which, given their methodological shortcomings, then show the same apparent "benefit" again. Meanwhile the media aids and abets this cycle by credulously trumpeting any nonsense that comes from these poorly powered studies. It's also rather interesting, don't you think, that negative studies of acupuncture never find themselves being reported all over the media, as this one did. It's not just studies of acupuncture, either, but virtually any "alternative" medicine.
Once complete for a question, the cycle then starts all over again or moves on to a different set of nonscientific and utterly implausible treatments, while patients are left not knowing what to think, or, worse, thinking that woo will help them, with the media reporting and promoting the few apparently "positive" trials and ignoring the vast mass of negative evidence. And on and on and on the cycle goes.
I see a pattern emerging from these studies - Accupuncture is an effective treatment for anything that has a strong response to placebo.
Now what could this indicate? :)
Immediately after noticing that there was no blinding, I noted the lack of true controls or placebo. The (apparently) unstated assumption in this study is that the drug venlafaxine is already established as more effective than placebo in relieving the symptoms of hot flashes.
Some other notes about the abstract:
The abstract has numerous statements that indicate they're only interested in comparing the drug to acupuncture, and never even mention the word placebo.
I know that venlafaxine is known to have some significant side effects, but many of the reported side effects (nausea, dry mouth, headache, difficulty sleeping, constipation, fatigue, etc) on the drug side are commonly reported by placebo groups.
The reported positive side effects on the acupuncture side are also "known effects" of acupuncture and are likely to be reported by anyone undergoing (or believing they are undergoing) acupuncture. The lack of true controls makes all the results of this study meaningless.
It would be interesting to do an intentionally unblinded study on the power of placebo and suggestion.
Take two groups:
Give one acupuntcture (or sham acupuncture) and tell them it is normal acupuncture.
Give the other group the same, but tell them you are testing out newly discovered, more powerful Chi points that are believed to have a much more powerful effect, but also produce strong side effects (nausea, dry mouth, headache, difficulty sleeping, constipation, fatigue, etc)
and note the differences in reported side effects.
Based on personal history, I can say that it's nutty for Dr. Cassileth to have stated that acupuncture is a "pleasant" experience. It's not what I'd call painful, but not something you'd look forward to, unless the procedure was accompanied by a soothing alcoholic beverage or other pleasing distraction (apart from distractions like wondering whether the acupuncturist is properly sterilizing the needles between patients).
It frustrates me to no end hearing about poorly-designed or poorly-conducted studies. Didn't these principal investigators ever have to take a class in experimental design, or at least biostatistics? If the experiment is poorly designed, it's nothing more than a waste of precious research hours and dollars. GAHHH!!!
As someone else speaking from personal experience I take exception to the "no side effects" comment. I recall an extremely painful hematoma in my hand that rendered it useless for several days after a treatment. As Dangerous Bacon said, not something to look forward to. And sometimes having the needles placed can be very painful.
I think you would actually need a 2X2 factorial design for this one.
1. Placebo acupuncture and Placebo Pill
2. Placebo accupuncture and Real Venlaxafine
3. Real accupuncture and Placebo Pill
4. Real accupuncture and Real Venlaxafine
The major implication I take from these accupuncture stories that you cover, Orac, is that spending time with your patients and being nice to them makes them feel better. Terribly old fashioned notion for one so young to have...
Didn't these principal investigators ever have to take a class in experimental design, or at least biostatistics?
It seems as if a lot of these people take classes in how to design experiments to confirm your own prejudices.
Another one, this time comparing homeopathy with diclofenac, is discussed here.
There is a problem in trying to use the randomized controlled trial (RCT) model to evaluate the effectiveness of acupuncture. The RCT is designed for pills. Someone can give a pill and not know whether or not they are giving a placebo. A trained acupuncturist cannot provide a treatment and be unaware whether he/she is using the correct points/procedures. You cannot do a double-blind. You cannot even do a single blind because patients will know whether or not they were poked. One could use alternate points, but there is no place on the body that is energetically inert, so it is impossible to insert a needle and not have any effect. The RCT is not the only way to gather data. Anecdotal evidence does have some merit. Acupuncture works on animals, and even skeptical animals show improvement; so there is more than just placebo at work. Of course it does not work for everyone (nothing does), but it helps a lot of people. Just because we don't understand something does not make in invalid.
For those of you who report acupuncture being an uncomfortable experience, I am sorry that was your experience. There can be great variability in the styles and techniques of practitioners. Most of my patients do look forward to treatments and many describe them as being more relaxing than massage.
President - Illinois Association of Acupuncture and Oriental Medicine
It is a common excuse of alternative medicine that their treatments are special and should be exempt from the usual scientific methods. That's nonsense. The RCT was not designed to study pills; it was designed to study ANY kind of treatment. There is no reason properly designed studies can't answer questions about acupuncture. So far, the better designed the study, the less positive the results.
If no point on the body is energetically inert, and if RCTs show no difference between acupuncture points and other points (they don't), how can you justify the complicated rigamarole of acupuncture? If "electroacupuncture" gives the same results as needles, (it does) why use needles? And what about the study with a "true" and "sham" acupuncture group that showed people were more likely to respond if they THOUGHT they got real acupuncture, whether they actually did or not?
There are recent books out by a research methodologist (R. Barker Bausell's "Snake Oil Science") and by the world's first professor of complementary medicine who has been doing research on alternative and complementary medicine for the last 15 years (Edzard Ernst, "Trick or Treatment"). Both have reviewed all the available evidence for acupuncture. Ernst allows that there is "tentative evidence that acupuncture might be effective for some forms of pain relief and nausea" but "it fails to deliver any medical benefit in any other situations and its underlying concepts are meaningless."
Bausell says there is no convincing evidence that it is more effective than placebo for any condition.
There are ways to do an RCT on accupuncture. They have been used and they indicated there was no effect above placebo. This indicates the blinding was succesful.
The real reason Jason Bussell says RCT's won't work for accupuncture is because every time a well designed one is done accupunture fails to perfomr singificantly better than placebo.
The argument that because RCTs are designed for pills, they can't be done for proceudures like acupuncture is like saying because wings were designed for airplanes, they won't work on race cars. In fact RCTs have been done for surgical procedures.
The double blind was designed to study pills. We don't study surgery with a blinded surgeon, or by cutting into people and doing a sham operation that we can compare to a properly performed operation. Techniques that require specialized skill and knowledge must be studied in other ways.
Actually, with special needles, you can keep even the practitioner from knowing whether it's the real thing or not. Orac's written about them before:
The double blind study can be and has been used to study things other than medications as you qwould know if you were a regular reader of this blog. Orac has mentioned studies that did use sham operations although I doubt that you could get something like this approved today.
The argument that because something was designed for A it is not valid for B is fallacious.
I did not argue that double blind isn't valid because it is well designed for medication. A double blind is not an appropriate way to study acupuncture and many other medical treatments because it is a technique that requires practitioner skill. Those fake needles are interesting though.
"Does that mean it doesn't work if you don't have an open mind?"
Unfortunately, Effexor is not all that effective either and it does have side effects in some people. BTW, there are some antidepressants that patients on tamoxifen should not take because they are metabolized by the same enzymes as tamoxifen (or something like that -- I forget the details).
A double blind is not an appropriate way to study acupuncture and many other medical treatments because it is a technique that requires practitioner skill.
That is simply garbage, and proves you don't know what you're talking about. The fact that it involves practitioner skill does not, by any possible argument, mean that the practitioner cannot be blinded. They can be and have been. It makes it harder, certainly, but not impossible by any stretch of the imagination.
If you understood how acupuncture is practiced in a real life situation you would know that it does matter. If those fake needles can truly fool the practitioner it could create a better study, but it would take out needle manipulation which is often part of an actual treatment, but it could still have an acceptable degree of validity. A study of twenty minutes is very questionable, few practitioners use less than thirty, and many use longer treatments
A well trained acupuncturist uses individualized point prescriptions based on patient history and other symptoms and observations. Using a set point prescription based on a single symptom would greatly decrease effectiveness. If you designed a study for a migraine medication, but the only requirement for patient admission was a headache and the administrator was not allowed to figure out if the HA was migraine, tension, cluster or sinus or other cause the results would be poor.
It is also being understood that TCM was designed for Chinese herbal treatments and is possibly not the most effective way to practice acupuncture. TCM is what most western students are taught, but if that is used for a study it needs to be understood that there are other methods. Other acupuncture treatments should be studied also.
I'm all for more acupuncture studies, but they should be designed by people who will take the time to truly understand how it is practiced whether they believe it works or not. Studying acupuncture is much different than studying a pill. To ignore that is irresponsible. It seems the people here are only interested in proving that things are woo, that's not the same thing as objectively seeking the truth. The purpose of this blog is to look for woo. More power to you, but that's a strong bias.
I'd be happy to respond further with anyone who is truly interested, but I have better things to do that get in some silly internet battle so that's not going to happen.
Something fishy here.
There are different acupuncturists and methods of
competency in this world
just as there are good and bad Medical Drs. in practice
so too with licensed acupuncturists.
Those that know they benefit from the reduction
of ailments with acupuncture know it, such as my wife.
Dems dat don't ,don't.
Please do not paint every doctor of Chinese medicine with
the same brush.
I might add who is to say the 45 minute break at the acupuncturists office (which should be soothing) from life's
problems and rush,isn't enough to give certain women some relief from post surgery hot flashes?
The process of just being touched and cared about has a
very healing dynamic.
I'd be happy to respond further with anyone who is truly interested, but I have better things to do that get in some silly internet battle so that's not going to happen.
In other words, Lindsey runs away, after laying down a whole lot of B.S. about how acupuncture "can't be studied" by conventional RCTs. Of course, no mention is made of how these acupuncturists "individualize" their treatments and especially not of what objective evidence they use to do so. In that, Lindsey's whine is very much like that of homeopaths when challenged to support their woo.
I expected a childish answer so no surprises here. It's not my job to teach you about something you haven't studied. I'm not going to teach you something that takes years to learn and you should know before you claim to know how to study something. You arguments are full of holes you don't even know about so you will always be discounted. Good luck on your woo crusade.
Ah, yes, I expected a lame retort from you, if there was any retort at all, and you didn't disappoint. Unfortunately, I've made it my calling to teach people like you about something you clearly know nothing about, namely the scientific method.
However, to see if you can actually provide compelling arguments that consist of something other than "your evil scientific method can't study my woo" (which, really, is what your argument boils down to), please name just one "hole" in my arguments and prove it is actually a hole. Please use science, reason, and evidence to do it, though. Arguments from authority or by mere assertion will not convince.
Its nice to see a doctor acknowledge how terrible hot flashes are, many seem to think it's just a little warmth. I had to pull over while driving because sweat was pouring in my eyes more than once. Another time I walked into the old Strand bookstore all bundled up because it was very cold. The Strand was cooking which caused a shocking hot flash. There I was pulling off all my outer clothes and some inner clothes ripping the scarf. If I had been the least bit unstable it could have caused me to have panic attacks, it was that sudden and torturous. Yes, I have been to the Strand many times since, must say I was happy to see the air circulation improved though now its a bit too fuu fuu.
The other problem is hot flashes come and go, you can go weeks with none and then have 20 the next day. I am sure many people have thought they cured the flashes until they returned. A good test would be to try and cause them either with hormone changes or other environmental adjustments. Changes in temperature would set mine off but they would also come when ever they felt like it. I don't have that problem anymore, but I suffered for 5 years with no treatment offered.
A final thought. Unfortunately I sometimes have to take prednesone and the first week or two on it brings (mild, thank God) hot flashes and I know I'm not the only one. You adjust to pred and they leave. Maybe you could force them using another medication?
If acupuncture worked only on placebo, nobody would ever use it. Most of my patients come to acupuncture only after trying other, more conventional treatments that they expected to work more so than they expect acupuncture to work. The University of Maryland 2003 study of osteoarthritis showed acupuncture to be more effective than PT. Why would all the patients expect acupuncture to work and not have that same belief of PT? Why would cortisone, PT, chiropractic, nsaids, and microdiscectomy not work for some patients with back pain who then report benefit from acupuncture? I think they expected those other interventions to work as well. You can believe what you want, but just because science cannot explain something doesn't mean it isn't true. Most would agree that we feel better when we are in love, but try and devise a double-blind study to prove it.