I sense a disturbance in the skeptical blogosphere. It is something that I half-expected, but, even so, it nonetheless somewhat surprised me when it arrived in the form of comments on my blog and e-mails from readers, fellow supporters of science-based medicine, and others asking me what I thought. In a way, it makes me glad that I didn't blog about this back on Monday, when the study that is the focus of this disturbance was published. Had I written about it then, all I would have had as fodder was the study itself. However, waiting a couple of days has allowed me to see the reaction of the study, who's promoting it, and how interested my readership is likely to be.
If the level of comments and e-mails is any indication, there is considerable interest. Such is the method to my madness.
It turns out that this study is about acupuncture. Well, it's not exactly a study, it's a meta-analysis that aggregates a whole lot of acupuncture studies in which this most popular of woos is administered to patients with chronic pain from a variety of causes. It's also being promoted all over the place with headlines like:
- Study: Placebo or not, acupuncture helps with pain (What a horrible headline; the very definition of "helping" is to do better than placebo. This was an AP story. Where's Marilyn Marchione when you need her?
- Acupuncture May Actually Work for Pain After All
- Biological Implausibility Aside, Acupuncture Works (Uh, no. Not quite.)
- Acupuncture works, one way or another
- Does acupuncture work? New research suggests it does, for certain conditions
- Acupuncture may be helpful for chronic pain
- Not just a stab in the dark: The ancient Chinese therapy of acupuncture 'does help to relieve pain' say US researchers
- Time for acupuncture to come in from the cold?
- Acupuncture Superior to Placebo, Usual Care for Chronic Pain (Et tu, Medscape?)
- Not Quackery: Acupuncture Does Relieve Pain
In deference to that last article, I was half-tempted to call this post Quackery: Acupuncture does not relieve pain. Then there were news reports like this:
And, of course, on accompanying the above news segment there was a story like this describing a patient with chronic pain:
In January 2009 she was referred to Dr. Jun Mao, a licensed physician and acupuncturist at the University of Pennsylvania.
"The only thing I had not tried was acupuncture," Zierler said.
Now, a new review of research suggests that this ancient technique may truly hold benefits for those suffering from certain forms of chronic pain.
In a review of 29 previous well-designed studies, which together looked at almost 18,000 patients, researchers at Memorial Sloan-Kettering Cancer Center found that acupuncture does, indeed, work for treating four chronic pain conditions: back and neck pain, osteoarthritis, chronic headache and shoulder pain.
Even "placebo" acupuncture, where the practitioner only pretends to place the needle or places the needle in a random site, is effective at relieving pain, though true acupuncture works better.
And so was born the propaganda line for this particular study, namely that it's huge; that it is the most compelling evidence thus far that acupuncture "works"; that all that stuff about "sham acupuncture" being as good as "real acupuncture" isn't true. But is this stuyd strong evidence of any of this? Let's go to the tape, as I like to say.
The study itself is from a group called the Acupuncture Trialists' Collaboration. I don't know about you, but the very existence of something called the Acupuncture Trialists' Collaboration is disturbing to me. Be that as it may, the study is Vickers et al, Acupuncture for Chronic Pain: Individual Patient Data Meta-Analysis. It was just published online in the Archives of Internal Medicine.
My first inclination when reading this was to apply a dictum to it that applies to all meta-analyses, no matter what the research question is; GIGO, Garbage In, Garbage Out, just like a meta-analysis of acupuncture to treat headache that I discussed four years ago. On the other hand, on the surface, this meta-analysis looks like it's a big deal. It uses patient level data instead of aggregated data, which allows for a better meta-analysis in most cases. It tries to restrict its included studies to those with the highest methodological quality (although that doesn't completely inoculate it from the GIGO label, as you will see).
So what the authors did was to search MEDLINE, ClinicalTrials.gov, and the Cochrane Collaboration Central Register of Controlled Trials for studies testing acupuncture against chronic pain. They then winnowed the pile of studies they found using several criteria:
Randomized controlled trials were eligible for analysis if they included at least 1 group receiving acupuncture needling and 1 group receiving either sham (placebo) acupuncture or no-acupuncture control. The RCTs must have accrued patients with 1 of 4 indications—nonspecific back or neck pain, shoulder pain, chronic headache, or osteoarthritis—with the additional criterion that the current episode of pain must be of at least 4 weeks duration for musculoskeletal disorders. There was no restriction on the type of outcomemeasure, although we specified that the primary end point must be measured more than 4 weeks after the initial acupuncture treatment.
Do you see a problem yet? I do. It is not required that all studies included have a sham placebo group. That means some studies were acupuncture versus no acupuncture controls, the latter of which could include groups that got anywhere from nothing to regular care. That's just one problem that I see, because mixing studies that compare acupuncture to no treatment, to sham treatment, or to sham treatment and no treatment are comparing apples and oranges in a way. Pooling such studies is inherently problematic.
There are other problems, but let's first discuss what the study showed. First, Vickers et al reported that patients who underwent acupuncture had less pain. That's true. However, I find it very hard to be impressed by these results. Indeed, they were most...underwhelming. Basically, the study reported that "real" acupuncture resulted in pain scores that were 0.23, 0.16, and 0.15 standard deviations lower than sham controls and 0.55, 0.57, and 0.42 standard deviations lower than no-acupuncture controls for back and neck pain, osteoarthritis, and chronic headaches, respectively. What does this mean? The authors themselves try to put it into context:
To give an example of what these effect sizes mean in real terms, a baseline pain score on a 0 to 100 scale for a typical RCT might be 60. Given a standard deviation of 25, follow- up scores might be 43 in a no acupuncture group, 35 in a sham acupuncture group, and 30 in patients receiving true acupuncture. If response were defined in terms of a pain reduction of 50% or more, response rates would be approximately 30%, 42.5%, and 50%, respectively.
One notes that Vickers et al have chosen a rather dramatic example, with large numbers. For patients with chronic pain, it's uncommon to have a 50% reduction in pain scores, and the standard deviation they chose was rather large. By their own argument, even if there weren't any methodological issues with the meta-analysis and their conclusions were completely justified, Vickers et al have just unwittingly made the argument that the effect of acupuncture might be statistically significantly greater than placebo effects but that it's almost certainly not clinically significant. What Vickers et al are arguing is that a change of 5 on a 0-100 pain scale (which would be a change of 0.5 on a 0-10 pain scale), a subjective scale, is noticeable by patients. It's probably not. There is a concept referred to as "minimally clinically important difference" (MCID) defined as "the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate…a change in the patient’s management." A recent review looking at minimal detectable and clinically relevant changes in pain scores in arthritis found a range in absolute terms between 6.8 and 19.9. Tubach et al assessed only the improvement aspect of the MCID and defined the minimal clinically important improvement (MCII) as the minimum improvement in the pain score reported by 75% of osteoarthritis patients ranking their response as "good" and reported that the MCII was -15.3 for hip osteoarthritis and -19.9 for knee osteoarthritis.
Here's a hint: -5 (the difference between sham acupuncture and "real" acupuncture) is not clinically significant. The only way you can even approach clinical significance is to compare no-acupuncture controls versus acupuncture, in which case you're adding placebo effects into any other effect observed, even if that effect is real (which I highly doubt it to be). Indeed, Vickers et al labor mightily to try to convince readers that this tiny effect, if it exists, is not just statistically significant, but clinically significant. They doth protest too much, methinks. In fact, I very much like how the grand master of the scientific analysis of "complementary and alternative medicine" (CAM), Edzard Ernst, put it:
Edzard Ernst, emeritus professor of complementary medicine at the University of Exeter, said the study "impressively and clearly" showed that the effects of acupuncture were mostly due to placebo. "The differences between the results obtained with real and sham acupuncture are small and not clinically relevant. Crucially, they are probably due to residual bias in these studies. Several investigations have shown that the verbal or non-verbal communication between the patient and the therapist is more important than the actual needling. If such factors would be accounted for, the effect of acupuncture on chronic pain might disappear completely."
Which brings me to another major problem with this meta-analysis. It's one that I noticed and one that Ernst also comments on. None of the studies included that I perused were double blind, which means that there was the potential for observational bias to creep into the study. While I concede that the authors did a pretty good job of making sure that studies in which there was a possibility of what is known in the biz as unconcealed allocation; i.e., failure to protect the randomization process to guarantee that the treatment to be allocated is not known before a subject is enrolled in the study (in studies with subjective outcomes, like pain, unclear allocation concealment is associated with bias towards beneficial effects), no attempt was made that I could identify to make sure included trials were double blind. In studies of subjective outcomes, blinding is almost certainly as important or more important than allocation concealment, and double blinding is essential. As Ernst put it so well, a trial is "either both patient and therapist-blind, or not blind at all." The investigators appear to have only assessed the selected studies for whether patient blinding was adequate, looking for descriptions of questionnaires in which patients are asked to guess which group they were assigned to. Without double blinding, it's hard to call any of these trials included in this meta-analysis "high quality." And, yes, it is possible to double blind acupuncture studies, as much as acupuncture fans try to argue otherwise.
Finally, there's the issue of heterogeneity in the trials. The authors report a lot of heterogeneity for most of the analyses that were performed but gave one of the sketchiest descriptions of how they actually calculated that heterogeneity that I've ever seen in a meta-analysis anywhere. One wonders what the reviewers were thinking. For supposedly ascribing to the PRISMA methodology for high quality meta-analyses, which specifies the calculation of a statistic (I2) for describing the heterogeneity of each meta-analysis comparison that is done, the authors don't live up to its principles in at least this one respect. They don't report that statistic. That strikes me as more sloppy than anything else, given that the authors concede considerable heterogeneity in their studies, making combining them problematic.
Finally, there's the issue of publication bias. Publication bias, as most of my readers probably know, is the tendency for published studies to be more likely to be positive than studies that remain unpublished. That's because scientists don't like publishing negative studies (they seem like "failures") and journals don't like publishing them either (because editors don't consider them very interesting). That's why, it's essential that a meta-analysis include an analysis looking for publication bias. One very common way of doing this is a funnel plot. Yet there is no funnel plot included that I could find (I couldn't get access to the supplemental material because I had to have someone e-mail the study to me and forgot to ask). Instead, they talk about looking at effect sizes in small studies and large studies and then calculate that "only if there were 47 unpublished RCTs with n = 100 patients showing an advantage to sham of 0.25SD would the difference between acupuncture and sham lose significance." How they calculated this number is not described. I must say, I've never seen this sort of analysis in a meta-analysis before, which is why it stuck out like the proverbial sore thumb, as did the lack of a description of how this estimate was calculated. Modeling? Why 47 unpublished RCTs of 100 subjects and not a smaller number of larger RCTs? The whole thing looks like a number the authors pulled out of their nether regions and then plugged into their meta-analysis software in order to see if it would affect anything. In fact, I have a sneaking suspicion that they probably tried a lot of combinations in order to find the one that would make it look as though it would take a whole boatload of studies going the other way to eliminate the statistical significance of their results. Is that unfair to say so? Well, the authors have no one to blame but themselves, and if I missed the description of how that was calculated I'll take my lumps.
In the end, I am less than impressed by this study, and it doesn't surprise me at all that it was funded by NCCAM and the Samueli Institute.
In fact, I'm pretty much unimpressed at the whole study, although no doubt it will be touted by acupuncturists for years to come as "proof" that acupuncture really and truly works and isn't just placebo medicine. It doesn't, and it is. In fact, the study strongly suggests that any effect of acupuncture observed is almost certainly due to nonspecific and placebo effects and that the "positive" result is, as Ernst describes, likely due to small residual biases. Even if we concede that there might be the small effect of "true" acupuncture reported by Vickers et al, it is almost certainly a finding that is statistically significant but clinically insignificant or, as I like to put it because I like baseball analogies, a really long run for a really short slide. As they say, garbage in, garbage out. Can we finally just say that acupuncture is nothing more than an elaborate placebo?
- Log in to post comments
Thanks for these insightful comments. I was wondering if the difference between sham and 'real' acunpuncture could be explained by the absence of double-blind protocol. I did not see anywhere if there was a double-blind protocol. I know it's quite complicated to do so with acunpuncture.
The British "quality" press divides along political/ideological lines as you might expect. The Guardian - liberal/generally skeptical says:
While the Times - right wing - goes with:
Both, to be fair, quote Edward Ernst at the end, but I fear that you are right - acupuncturists will be touting this study for years to come. On the other hand, perhaps it isn't too surprising. After all, if you are "a bit hurty" having someone calmly relaxing you, touching you, and talking away the pain, might well help.
People seem to like acupuncture (which is, I believe, available on our National Health Service in the UK.
But far more shocking than a slightly dodgy meta-analysis showing possible benefits of micro-massage is that our new Health Minister, Jeremy Hunt, believes in homeopathy...
Such is the method to my madness.
"I don't see any method at all, sir."
You just haven't looked hard enough. :-)
I am baffled. I am a chronic pain patient and have ever only been asked pain questions on a ten point scale. I have never seen a 100 point scale for pain (yes, it gives you fine gradations, you can even switch it up to a decimal ten point scale - is your pain a 6.5 or a 7.2 today?). Usually keeping track of ten numbers for something as subjective as pain is difficult enough. Someone like me, who spends much time with distracting, etc., ends up better served by a functionality scale, but even that runs across a ten point range.
I'm sure that it will be explained to me it had to be converted to a 100-point scale so they could do the meta analysis better.
This makes me a little frightened, to be honest. With the insistence by some that pain medication should only be available to patients who are near death, those on the "no narcotics unless you're end-stage terminal disease" people will begin pushing acupuncture as a perfectly useful alternative for everyone else.
I am happy to say that acupuncture is nothing more than an elaborate placebo.
A meta-analysis stacked in favour of acupuncture could not really come up with a clinically significnt effect. What more evidence is required?
No funnel plot? That seems quite unusual. This has become a pretty standard method in modern meta-analyses.
Sure, if we can say that so-called "conventional" drugs used to treat muscle aches are nothing more than slow poison.
If acupuncture works, it works. And with no side-effects, I might add.
Of course, what does make a pill so tempting (and profitable) is that it's very easy to do, slow poison or not. There is no time spent lying still on a table while someone sticks needles in you. I guess most people would prefer to take that route. But then you have to take more of them, and there is still the possibility of addiction, and nasty debilitating side-effects (some of them potentially fatal), and flushing all of those chemicals into the water supply when you pee, where they tend to remain much longer than sewage because Nature takes a long time to break down all those ten-syllable ingredients us laypeople couldn't identify without medical training, and that's why our rivers, especially the Mississippi, have measurable levels of these chemicals still in the water, where, they're doing God-only-knows-what to the life in it, and of course, these chemicals find their way into people who never needed the drugs in the first-place like second-hand smoke. Oh well, I'm sure I've made my point by now, but in case you need it broken down, don't take the pills if you don't need them--try to find some other way first.
Herr Doktor @0637: Orac wasn't asking if you thought his methods were unsound.
No doubt many of the people who frequent anti-vax and alt-med sites would say that Orac's methods are unsound. A few have even popped up in Orac's comments to say so. But I think Orac's methods are entirely sound.
@Eric Lund:
http://www.youtube.com/watch?v=WdNsltQXTVU
Orac is all method. The madness is merely feigned in order to entertain.**
So saith yours truly. And I should know.
While the woo-esphere will certainly applaud and embrace Vickers et al, they are up in arms about another review:
Smith-Spangler et al ( Annals of Internal Medicine, 4 Sept 2012) reviewed studies comparing the health effects of organic and conventional foods, concluding that evidence is LACKING that organic is more nutritious although it may reduce exposure to pesticide residues and antibiotic-resistant bacteria.
As you might expect, alt med mavens aren't particluarly thrilled with Smith-Spangler et al: various articles condemning the study and Roger Cohen ( NYT), who wrote in support of the research, have already appeared at Natural News and PRN ( as well as a rant at the latter), even Age of Autism vilified the research. It will provide grist for their mills and fuel for their fire: universities like Stanford ( where the study originated) echo corporate policy and reflect governmental authoritarianism. Never trust places like that!
I predict that we'll be hearing about both for a while.
** And never forget that all method and no madness makes Jack a dull boy. That's why we avoid Jack.
Whenever I read a new study about Acupuncture I automatically think "Hmm... I wonder what orac thinks about this."
Are there other Orac-like characters on the internet with their own blogs? If Orac gets hit by a bus then i'd like to know where else I can go to get my fix.
Mrs Woo:
I could be mistaken, but I think the intervening numbers occur because of averaging. Sort of like how the average family has 2.1 children (or whatever) but obviously children are generally born (or at least recognized by census-takers) only in integer values. ;-)
However, you might be interested in a better pain scale. Do read it; it is pretty hilarious. (Note; there is some salty language.)
http://hyperboleandahalf.blogspot.com/2010/02/boyfriend-doesnt-have-ebo…
"And never forget that all method and no madness makes Jack a dull boy. That’s why we avoid Jack"
Could you point him out to me? I don't know Jack. :)
Thaks for posting this. Acupuncture continues to be the most accepted woo in my circles, even among fecking scientists who should bloody well know better.
Look how NCCAM spins the results of the Acupuncture Study:
http://nccam.nih.gov/news/2012/091012
"A recent NCCAM-funded study, employing individual patient data meta-analyses published in the Archives of Internal Medicine, provides the most rigorous evidence to date that acupuncture may be helpful for chronic pain. In addition, results from the study provide robust evidence that the effects of acupuncture on pain are attributable to two components. The larger component includes factors such as the patient’s belief that treatment will be effective, as well as placebo and other context effects. A smaller acupuncture-specific component involves such issues as the locations of specific needling points or depth of needling."
"Meta Analyses...the most rigorous evidence to date?" The GIGO principle is heavily in play here.
Look at the two components, especially the "larger component" (which, according to the NCCAM) "includes factors such as the patient’s belief that treatment will be effective, as well as placebo and other context effects."
See also what NCCAM said about prior "unsuccessful" studies of acupuncture (August 2010)...
http://nccam.nih.gov/health/acupuncture/acupuncture-for-pain.htm
Now scroll down to see how many of these studies were funded by NCCAM. Better yet, see how many "proposed" studies of acupuncture that NCCAM intended to fund.
Orac's right...I say nothing short of nuking the NCCAM will ever get them to stop funding these bogus studies.
Funny, when I see a new study about acupuncture I always say "I bet Orac will throw a tantrum about this within 36 hours."
Comparing acupuncture to non-acupuncture treatments, such as pain meds (whose long-term use has many health risks), is a legitimate use of science as process, whether you like it or not. If people who use acupuncture get more relief of back pain than those who swallow pills, I want to know about it. If the mechanism is confined to a super-duper-extra-placebo effect, why, that's very interesting, but I still want to know what the relative total benefits are, and the scientific method is capable of addressing that question. Nobody has the authority to state that Science forbids others to ask that question.
Where studies of acupuncture vs. nonstandard or "sham" acupuncture are concerned, I am happy to say that we should focus on those that are double-blinded. However, there are a few caveats. (1) The sacred Toothpick study that we are always told Proved Acupuncture Worthless was obviously not double-blind, so must never again be mentioned. (2) If the control treatment involved dermal stimulation significant enough to be felt as possibly a needle stick, studies that lack significant differences between active and control treatments cannot be used as evidence that acupuncture does not have a biological effect. (3) Double-blinded acupuncture studies in which there IS a significant difference between active and control groups cannot be automatically labeled Worthless just because the result does not please you.
That's the scientific approach. Feel free to cling to religious beliefs if you prefer.
@jane
Who's saying that science forbids people from asking the question?
Thankfully, sham needles have been developed that do allow for true double-blinding. Studies using them have, IIRC, shown that there is no difference between the real needles and the sham needles.
What these kinds of studies would show is that, if poking people actually does have some effect, then it actually doesn't matter whether the poking bit actually breaks the skin. From that, we can draw the conclusion that if breaking the skin and not breaking the skin have the same end result, then we should not ever be using methods that break the skin, since the risk is greater for no added benefit.
Again, who is doing this? What I've seen is that there are generally other methodological flaws, either in the conduct of the study or in the resulting analysis. I've never seen someone say, "Hmm. This double-blind study has a significant difference between the experimental and control groups. Bah! It must be crap!" and just leave it at that. Rather, the approach, especially as I've seen from Orac, is "Hmm...this study suggests there's an actual effect. Let's take a closer look and see what's really going on. (discover whatever flaws were glossed over by the faithful) Ah, I see what's going on now. This is why the study doesn't say what people are saying it says..."
Now, if you're done knocking down straw men, how about addressing the criticisms that Orac actually made?
If the control treatment involved dermal stimulation significant enough to be felt as possibly a needle stick, studies that lack significant differences between active and control treatments cannot be used as evidence that acupuncture does not have a biological effect.
Two things--first, why does this not represent evidence that acupuncture does not have a biological effect? Wouldn't the optimum treatment control be one where the subject was absolutely convinced actual accupuncture needles had been inserted, although no actual 'needle stick' had occurred?
I haven't seen any well designed, reproducible double-blinded accupuncture studies demonstrating a significant difference between treated and controlled groups. Which studies are you referring to here?
I'm having a lot of fun posting on Dr. Briggs blog...aptly named "Quirky Ideas From Outside The Mainstream"...
https://nccam.nih.gov/research/blog/quirkyideas#comment-205
I've gotten three comments through, with a fourth awaiting moderation...in reply to a poster who claims a friend's "successful treatment" of sinusitis by acupuncture is "Plausible". I'm stressing the "Plausibility Factor" that Dr. Briggs *claims* her staff is employing to judge whether a proposed study, warrants further investigation...and funding...by the NCCAM.
Anecdotally, I tried acupuncture for chronic sinusitis several years ago, along with a number of even less alternative plausible treatments, including homeopathy. Such are the lengths that pain and desperation can lead me to. Firstly acupuncture and homeopathy had no effect whatsoever on my symptoms, not surprisingly. Secondly, the needles are so fine that there is, in my experience, no pain whatsoever involved. Sometimes I noticed a slightly odd feeling when the needles were twiddled. I am quite sure that without visual cues I would not have been able to tell the difference between acupuncture and sham acupuncture, or even toothpicks. That's the whole point of blinding and placebos.
Apologies to regulars for the repetition, but I will never tire of pointing out that acupuncture is not ancient - the meridians and fine needles now used were introduced in the 1930s by a Chinese pediatrician called Cheng Dan’an. Before that it was almost identical to medieval blood letting.
I have not read the study, and probably will not. Can anyone who has or will read the study help me on a question that Orac implies: If you subtract out the studies in which acupuncture is compared to zero treatment, and only look at studies in which acupuncture is compared to some reliable control such as non-puncturing devices, does the statistical significance go away, or at least get reduced? In addition, does the difference in reported pain scores go down? As Orac explains, these are different questions, and both would be of interest to somebody who has pain or to the physician who is trying to treat that pain. I'm guessing that some of the comments above relate to studies that did these controls and found no evidence of anything but placebo, but I'm wondering if the paper under discussion even tried to do this analysis.
@ Krebiozen: Why don't you join me on Dr. Briggs' blog?
My last comment about sinusitis *treatment* with acupuncture and "Plausibility" is posted there. I drew the analogy to the "friend's anecdotal story of relief of sinus pain by acupuncture" with this statement...
"Your friends anecdote has no more "scientific" proof and does not meet PLAUSIBILITY factors that scientists should be using, than the anecdote from a person who claims walking around a craps gambling table three times counterclockwise relieved his gambling debt."
I have to go offline for a few hours, so see if you can get any comments posted on Dr. Briggs' blog.
@ lilady:
I managed to talk one of the fellows- and his woo-loving friend- out of accupuncture by painstakingly outlining the toothpick study: now they both swear by massage therapy for pain releif. At least that has some basis in reality.
The ex still swears by the pain extinguishing powers of very expensive hooch. Also somewhat realistic.
I try, I really try. They're business people, not scientists.
How long will it/or did it, take for the Wretched Hive to blog about this latest NCCAM study?
When would "lilady" go *slumming* at the "Wretched Hive" to find that blog?
http://www.huffingtonpost.com/2012/09/10/acupuncture-chronic-pain-arthr…
Here's the link to the NCCAM blog...well worth a "bookmark" to comment about all their recent studies...
http://nccam.nih.gov/research/blog
Somewhat related XKCD
http://xkcd.com/883/
The good Dr Crislip has noted in the past that acupuncture studies that use subjective measurements (like pain ratings) tend to show some effect, but that studies that use objective measurements (ie range of motion) show little or no effect. Maybe in the next study, eh?
Dr Crislip also notes that acupuncture quacks don't appear to believe germ theory. All the pictures of acupuncture in action seem to show the needles being handled with bare hands. I wonder if jane or ophu would like to address that issue.
ophu states that acupuncture has no side effects. Some would beg to disagree: http://sci.waikato.ac.nz/bioblog/2010/08/but-surely-if-it-does-no-harm…
ophu also states "Oh well, I’m sure I’ve made my point by now, but in case you need it broken down, don’t take the pills if you don’t need them–try to find some other way first."
You mean try to find some other way that works \, don't you?
@ Alison: How about this acupuncture needle that was "found" in the lung of the former president of South Korea...along with this gem of a quote
http://www.reuters.com/article/2011/05/04/us-korea-roh-idUSTRE7434TZ201…
"Doctors are puzzled how the needle ended up in his lung, and acupuncturists say that none of their procedures involved penetrating the lung."
oh that's dreadful, lilady! Perhaps his 'doctors' should have said, none of their procedures involved *deliberately* penetrating the lung.
Alison, I just *knew* you would enjoy that article and the memorable quote.
How about, "none of their needles are strong enough to pierce the intercostal spaces"?
Reminds me of my daughter's penny incident. *Somehow* she managed to swallow the coin and *somehow* it got lodged in her esophagus. After a quicky trip to the OR and a failed attempt to snare it, she was hospitalized overnight to allow the swelling to recede. Thankfully, the next morning an X-Ray located it in her small intestine.
The doctor didn't have to do surgery, but I had to *watch for it*. I still have that penny, (cleaned up), but all corroded from her digestive acids.
I was involuntarily acupunked once. It was to make my knee with arthritis from having part of the meniscus removed, more flexible.
It was quite painful, I was squawking as he did it, and it didn't do a thing for my knee's flexibility. I don't know why anyone would even speculate that poking in a couple needles would cause tissues that had tightened up over decades to magically loosen. Since then I have slowly, over years, stretched those tissues for extended times every day, and they have loosened.
To Orph and Jane and anyone else who thinks painkillers are Teh Evils: I have fibromyalgia, and thankyou very much, I will continue to pop painkillers despite them being addictive slow poisons, because, gosh shucky darn, THEY WORK. Within minutes. I am less convinced about acupuncture because for every person who insists it changed their life I can find 3 who say it did diddly squat, and in some cases made the pain worse.
Hm.
I can't speak for the science of acupuncture, perhaps there is none. I do know it's worked for me, for temporarily relieving cubital tunnel. The alternative would be chowing down a bunch of NSAIDs, or surgery. I figure acupuncture is less dangerous than a constant diet of ibuprofen. If it works equally well with toothpicks I don't care; for me it's had useful, temporary effects but without associated harm.
Surgery may fix the problem once and for all, maybe. If I can find a surgeon I can trust with the procedure and if it can be done without crossing my other health issues, I'll pay up and be happy about it. But surgery around a nerve... I do want to see the numbers on mishaps about that.
"the very definition of “helping” is to do better than placebo."
Um, no, the very definition of "helping" is to do better than nothing. If you're in a lot of pain, it doesn't matter whether you do real acupuncture or sham acupuncture or hypnosis or meditation: all that matters is whatever you do makes it hurt less than not doing anything.
“none of their needles are strong enough to pierce the intercostal spaces”
30 seconds at the Great Google reveals an endless list of acupuncturists calling for special care when needling points on the chest, on account of the danger of the needles piercing the intercostal spaces and causing a pneumothorax.
"It is not required that all studies included have a sham placebo group. "
Doesn't this completely invalidate the results from this study?
OT, but that reminds me of a male patient who had a large bladder stone removed, and when it was cut in half it was found it had formed around a birthday cake candle. The patient, who had had no previous bladder surgery, swore blind he had no clue how it got there, and considering male anatomy it is indeed puzzling and somewhat cringe-inducing. This isn't a myth, by the way, I personally examined the patient's records and a detailed drawing of the calculus that a pathology consultant I worked with had kept in a small medical museum he had in his office.
Ashley,
Did you not read what Orac wrote about a lack of clinical benefit (i.e. a difference that the patient can actually notice) above? Risk without noticeable benefit makes no sense at all, and there are definitely risks of punctured lungs (see the comments above) and infection from acupuncture. The link is an article about Woo, often cited here, PMID: 20299695 which is itself behind a paywall.
Horrifying, considering the needle was 6 cm long and could have ended up penetrating his aorta or heart.
The fact that his "doctors were puzzled as to how it ended up in the lung" doesn't exactly inspire faith in conventional medical practice in Korea though. They don't even have to be able to add 2 and 2 together to get 4 in order to graduate from med school? With doctors that clueless, no wonder alternative medicine is a big thing over there.
Oops, sorry. I checked that several times before posting and still missed the missing link closing tag.
Orac, I have access to the supplement. It contains their search strategy and several tables of the selected studies with respect to blinding and controls.
No funnel plots, no further statistics.
I hate playing the ad-hom card , but I can't help noticing that among the authors of the study there are homeopaths Klaus Linde and George Lewith, as well as Prof. Claudia M. Witt who is holding the chair for Complementary Medicine and the Charité Berlin, funded by the woo-friendly Carstens Foundation. She is also the author of the "patients like homeopathy, therefore it works" study here: http://www.ncbi.nlm.nih.gov/pubmed/16036164
Doesn't exactly infuse me with confidence.
This is a biased blog where woo = anything not Pharma or conventional medicine. Prejudice, i.e., a performed judgment, is de rigeur for the regular posters
@ Shannon
And whining about Pharma/conventional medicine without bringing any evidence to support their claims is "de rigueur" for the people that disagree with the regular posters.
Preformed judgement much?
How do you double blind acupuncture?
Shannon, if you want to be taken seriously you should try reading and responding to the actual article. If you did, you'd find that it contains real, legitimate criticisms of the study as written, not "prejudice, i.e., a performed judgment."
Chris (not me):
No. Not all studies can have a placebo, especially ones that require puncturing the skin (like surgery, and acupuncture needling). Though the tooth pick study was hilarious.
Actually, shannnon, woo = things which have not been shown to be effective, or which have actually been shown to be ineffective.
You do know what doctors call woo that has been shown to be effective, don't you? Conventional medicine.
@Chris
Actually, you can have a placebo acupuncture study. Sham needles have been developed that are a bit more sophisticated than toothpicks.
And sham surgery has been done, though the ethics involved are questionable at best.
Thanks Todd for the clarification. Though, sometimes they claim that the sham treatment worked, therefore acupuncture works. Which is exactly what makes the tooth pick study hilarious.
@Chris
I haven't seen the sham needles, but they're apparently pretty neat devices. In studies that use them, both the real and the sham needles have sheathes so the practitioner can't tell if they're real or fake. The fake ones retract into themselves, with a little bit of a catch so they feel like they're actually going in and a bit of, IIRC, adhesive to make sure they stay on when the practitioner removes their hand.
I can think of two ways to do suitably double blinded acupuncture:
1. Extremely clever sham needles, as Todd described.
2. Randomize the treatment plan, where the person performing the procedure doesn't know what the expected results are (so doesn't know if the acupuncture points used are "proper" for the case) and the person evaluating progress does not know if the patient received the treatment prescribed or treatment at a completely unrelated acupuncture points.
I would ask the same question as is posed for homeopathy:
Is the supposed mechanism even plausible? The acupuncture “points” are just made-up stuff as far as I can tell and the “ancient practice” trope has been addressed before. If there is any real effect on pain, HOW is this accomplished, one wonders? (Not really)
Orac:
I think you are missing the MOST IMPORTANT point about acupuncture: the absolute and total failure of the underlying biological plausibility. We can't allow the clinical observational studies to be taken seriously when they are based on the pseudoscientific belief in chi.
You and I know that sticking a needle in someone's skin at most has a localized response from mechanoreceptors and a psychological effect on the patient's brain state. The acupuncturist does not open magical energy conduits because those magical energy conduits do not exist. If a "modern" acupuncturist wants to argue that the energy is actually nerve energy, let them do the patch clamping or sodium channel assays in a lab experiment. Don't give them a pass on the underlying biology.
If someone did clinical trials on voodoo, and there were enough practicioners of Santería with research grants from NCCAM, I have no doubt you could produce a systematic review that showed positive effects.
C.
Can I shamelessly self-promote? Because I made a video on this topic that seems like a reasonable companion to this post:
http://www.youtube.com/watch?v=pp5eiHUdwb4
Nice video cOncOrdance. How about looking at the NCCAM, where Dr. Briggs and her deputy are getting hammered for this latest study...and for all the crap studies generated by the NCCAM. If you peruse the remarks, you *might* see some familiar names:
http://nccam.nih.gov/research/blog
Im quite confused...please help...
In looking at the data provided by the investigators, I am having a hard time seeing the individual effects of say sham and acupuncture, and nothing....vs showing the "fixed effect" difference between say sham and acupuncture.
So in the example provided there is a 5 point differnece in say sham and acupuncture...but where in the data can I find out this difference? Was there only a 5 point difference in the data and I am just not calculating it right or something...im terrible with stats...alls I know is that I am confused...
People are citing the 30% vs. 35% as if this was the data from the study...is this the case?
I'm just writing my own metanalysis paper now, which prompted me to comment on this part of the acupuncture paper: "only if there were 47 unpublished RCTs with n = 100 patients showing an advantage to sham of 0.25SD would the difference between acupuncture and sham lose significance.”
Meta-analyses often calculate a "fail-safe number", which is the estimate of the # of studies that would be needed to change a significant effect to a non-significant one. This fail-safe number is generally considered to be at least 5 times larger than the number of studies being considered plus 10 (Rosenthal, 1991). However, given that there were 29 studies in this meta-analysis, their failsafe number should have been something like 150, not 47.
Sorry, gave the wrong link above for Rosenthal. The correct citation is: Rosenthal, R. (1991). Meta-analysis: A review. Psychosomatic Medicine, 53, 247–271.
Hello, I am the author of the study, and someone sent me this link.
As way of background, let me say that 95% of my research is in prostate and other urologic cancers. So it is rather interesting for me to see the reaction of folks to our meta-analysis. Clearly, views about acupuncture (for or against) are often pretty religious. If our study shows anything, it is that some individuals will never change their mind irrespective of the evidence. I guess the same would be true of acupuncturists if the study had shown results in the opposite direction. Indeed, a decade or so ago, I was involved in a study showing that homeopathy didn’t work, and it wasn’t as if homeopaths quit en masse.
What is remarkable about Orac's article is the very large number of basic errors in it. To name but a few:
a) " mixing studies that compare acupuncture to no treatment, to sham treatment, or to sham treatment and no treatment are comparing apples and oranges in a way". Comparisons between acupuncture vs. sham and acupuncture vs. no acupuncture were kept separate.
b) "Why 47 unpublished RCTs of 100 subjects and not a smaller number of larger RCTs? The whole thing looks like a number the authors pulled out of their nether regions and then plugged into their meta-analysis software in order to see if it would affect anything". Why did we test for publication bias in this way? Because it was pre-specified in the protocol which was pre-published and referenced in the paper http://www.trialsjournal.com/content/11/1/90
c) "the authors … don’t report th[e] I2 statistic. That strikes me as more sloppy than anything else, given that the authors concede considerable heterogeneity in their studies, making combining them problematic". We don't report I2 because we believe it is invalid. There is either heterogeneity or there isn't so I2 should be either 1 or 0; any number in between means you don't have enough studies. With respect to “considerable heterogeneity”, we carefully investigate sources of heterogeneity. The main numbers we report for sham vs. acupuncture exclude the Vas trials and for these analyses, there was no significant heterogeneity.
d) “For patients with chronic pain, it’s uncommon to have a 50% reduction in pain scores”. Oh really? We chose 50% as having a halving of pain in the acupuncture group as the “baseline” because this is exactly what was reported in the trials (e.g. http://www.ncbi.nlm.nih.gov/pubmed/15870415).
Anyway, enough for now, I think I’ll go back to challenge 911 deniers. On the plus side, conspiracy theorists don’t poke their noses into areas of science in which they clearly know very little and then get snooty with other scientists. Sloppy? Pulling statistics out of my nether regions? Sorry you didn’t like the results, pal, and I am very willing to have a scientific debate about, say, clinical significance in placebo controlled trials. But you clearly made up your mind long ago and are not that interested in scientific debate.
Andrew Vickers
Hi Dr. Vickers,
My apologies, I haven't read the original study (or even re-read this post), but I do have some questions and comments. In your comparison of "true" versus "sham", what sort of controls did you consider? Because there are a multitude of controls that could be used. Location and depth of penetration are two, but there is also whether penetration is used at all (vis. the sham needles that retract into the head of the needle). An extremely interesting observation was the erosion of the evidence base for acupuncture once retracting needles were introduced, since they invariably showed penetration of the skin was more-or-less optional. Once introduced as a form of control in acupuncture studies, the suggestion was that skin penetration (probably the greatest risk of acupuncture in terms of injury, sterility and adverse effects) was unnecessary. Another example of this would be the famous toothpicks study.
Also, have you seen the study regarding the effects of practitioner enthusiasm? Unfortunately I don’t have the study on-hand but I’m sure Orac or another commenter could provide it if you were interested. Essentially the findings were that if practitioner enthusiasm during diagnosis and treatment were used as an independent variable, the effects far exceeded any other independent variable studied.
My point is – treating all “sham” treatments as unitary, as a single factor to be analyzed in a meta-analysis, is a questionable assertion in my non-expert opinion.
Also problematic – which system of acupuncture is used? Chinese, Japanese, Tibetan, Korean, or French (vis. ear acupuncture)? What sort of diagnosis or diagnostic system is used? Is it still “acupuncture” if you simply insert needles close to the site of pain or muscle spasm, ignoring the elaborate and quasi-mystical systems that complicate both training and diagnosis while driving up the expense, as some “medical acupuncturists” mostly in Britain do?
Also, is comparison to “no acupuncture” really valid? Dr. Ben Goldacre has an excellent article on the problem in relying on “no acupuncture” as a control, and I think that article also briefly goes into the complexities of placebos. To whit, there is no unitary “placebo effect” – the standard notation is that one pill is better than two, saline injections are better than pills, mock surgery is better than injections, the colours involved make a difference, as does the perceived exoticness and drama of the intervention. Considering acupuncture is dramatic, exotic, invasive and, if traditional, accompanied by a length consultation, it is probably naïve to think that you can compare it to simple sugar pills, or no or waiting-list control. I would think this would be exacerbated for conditions like chronic pain where we don’t have a very good solution and most patients have tried, often for years, many other treatments.
Then of course, there is the issue that acupuncture has no validated or even plausible mechanism by which it could work. Further, it seems odd that acupuncture, among the multitude of prescientific interventions, is treated seriously despite China not relying on empirical research to produce it, its history of rejection within China itself until resurrected by Mao to compensate for the lack of real (i.e. scientific) treatments and doctors, and other vagaries of its history. Virtually no medical treatments were effective before the advent of scientific medicine, making the lengthy, expensive and largely unfruitful efforts to test acupuncture…odd. As an oncologist and researcher, are you convinced this is a good use of scarce research funding when despite thousands of studies and millions of dollars we are still debating whether there is a real effect? Particularly when even positive studies with adequate controls indicate it provides at best brief and unreliable pain and nausea relief for a subset of patients?
I can’t speak to your other comments, but I do appreciate your dedication to science. In my opinion, being willing to put in the time and effort to produce reliable empirical evidence for medical treatments is just about the most laudable thing one can dedicate their life towards.
The results of acupuncture trials are mixed, and are exactly what one would expect to find for a treatment with no specific effects and a plethora of placebo and other nonspecific treatment effects.
When a treatment is truly effective, studies tend to produce more convincing results as time passes and the weight of evidence accumulates. When a treatment is extensively studied for decades and the evidence continues to be inconsistent, it becomes more and more likely that the treatment is not truly effective.
@WLU
Is this the study you meant?
http://scienceblogs.com/insolence/2008/04/08/finally-nccam-actually-fun…
Or was it this study?
http://scienceblogs.com/insolence/2012/01/09/cam-placebos-and-the-new-p…
:-)
Two posts of dozens that could be linked to, but the second is the one I remembered :)
They suggest the true value of well-designed acupuncture studies - determining exactly what the characteristics of the placebo effect are, how to modulate them, and most importantly how to use them in clinical practice. Of far more benefit than chasing down a possibly illusory effect that hasn't revealed itself despite decades of research.
We already have a good understanding of the placebo effect. It happens when you give a real treatment, which is why we test treatments against something we reasonably suspect of doing nothing. That's why we test acupuncture against sham acupuncture needles or random insertion and drugs in pill for against sugar pills. If treatment effect + placebo is greater than placebo alone to a statistically significant level, then we can say the treatment works.
Noteworthy is that placebo doesn't affect much if anything objective, just the patient's subjective experience.
Advocating the use of the placebo effect outside of research is also ethically nasty because it gives the doctor permission to lie to their patients and goes backwards, back into the era of the paternalistic doctor. It increases the power difference between doctor and patient. I'd also imagine it insidiously encourages placebo inaction for treating subjective symptomatic discomfort while there may be a non-psychological cause that needs to be addressed.
Of course, that's the business model of a con man, which is probably a major reason why we're seeing this position touted by alties and their gurus.
... You must be new to Earth.
"Clearly, views about acupuncture (for or against) are often pretty religious."
I beg your pardon. People who think scientifically have no need to get "religious" over acupuncture or anything else (except for chocolate eclairs). It can be far more uplifting and freeing to be lead by good data than to carry the increasing burden of having to insist the unreal is real.
@ LindaRosaRN:
Are you religious about chocolate eclairs?
I'm more that way about scarves, gloves and other accoutrements.
@ Linda Rosa RN:
You certainly have been *active* on the NCCAM blogs regarding this latest meta-analysis of acupuncture.
Here...
https://nccam.nih.gov/research/blog/plausibility#comments
and here...
https://nccam.nih.gov/research/blog/quirkyideas
Those damn RNs...always posting on blogs.
lilady, RN
"Religious" is a somewhat appropriate term for believers in acupuncture. The claims made for it are pretty dramatic, there's no obvious and certainlyno proven underlying mechanism, and nearly all results are equivocal, flawed, questionable and/or borderline. Not to mention the entire historical premise is flat-out wrong. Comparing the amount of money poured into acupuncture research and practice to its evidence base and efficacy, "religious" seems an appropriate term to apply to those who vigorously insist it is effective via some hitherto-unrecognized mechanism and is not merely a potent placebo.
Given the claims made versus evidence for (and against) acupuncture, the skeptics certainly seem to have the advantage.
Just because its a Placebo doesn't mean its a sham and if elaborate then congratulations on creativity. Even real medicine is influenced by the placebo effect. By packaging it well, increasing cost to raise perceived value and effect. The colour of the tablet is carefully chosen to enhance its effectiveness. Placebo teaches us more about ourselves and who we are and how we become sick and well. Medicine is concerned with the patients outcome and whichever "placebo" works is important. Doctors will often prescribe many medicines before they find one that works for a particular patient and so it is with styles of acupuncture. http://placebocures.com asks the question could we take an imaginary placebo and it still work? and there have been studies to say it does. interesting thought
If acupuncture is a placebo, how come it has been used and still used today for particular ailments and for good health benefits.
@Margaret - because people are gullible & don't know any better, perhaps?
I hadn't noticed Derek's comment. It's worth repeating that placebos do not have any objective effects. They don't make tumors shrink, get rid of infections quicker, make broken bones heal faster or improve lung function in asthmatics. They may make people feel better, but so do real medicines, which also have an objective effect. Some people seem to forget that we get a free placebo effect from real medicines as well as fake ones.
@Margaret, Oftentimes acupuncturists do more than just inserting the needles. Many incorporate relaxation therapies with soft music and lighting, relaxing scents, as well as massage into their treatments. When those treatments, along with their resulting placebo effect are considered as a whole, the patient may feel improvement. However, patients are easily and erroneously persuaded to believe that the acupuncture needle insertion is in itself the reason for any and all improvements in their condition.
As Krebiozen said, placebos "may make people feel better, but so do real medicines, which also have an objective effect."
Margaret, that a demand for a product exists doesn't argue the product offers utility, only that it's appelaing. The fact that acupuncture has been used and is still used today doesn't argue it's effective, anymore than the fact that homeopathy has been used and is still used today or energy healing has been used and is still used today argue that they are effective.
What's needed to argue accupuncture is more than a placebo is actual evidence that it's more than a placebo.
So...got any?
If it's mathematically inescapable that you're more likely to lose money in a slot machine than make more from it, why do people still play slots hoping to get rich?
My favorite retort to those supporting acupuncture is to point out it was invented in the 1930s. Before that it was indistinguishable from medieval European bloodletting (that's why acupuncturists take your pulse), and only itinerant folk-healers carried it out. As Ben Kavoussi explains:
Incidentally, I recently found out that bloodletting is still carried out in some Arab countries. It is called Hijama. The technique is very similar to cupping, as used by some TCM practitioners, and I'm sure they have a common ancestor.
Smoking *must* be good for you or everyone would have stopped by now.