It’s amazing how fast six months can pass, isn’t it? Well, almost six months, anyway, as it was five and a half months ago that I wrote about a particularly execrable example of quackademic medicine in the form of a study that actually looked at an “energy healing” modality known as “energy chelation” as a treatment for cancer chemotherapy-induced fatigue. Actually, the study design itself wasn’t so bad, leaving aside the utter ludicrousness of the concept of “energy chelation.” Rather, it was how the authors
spun interpreted their results that set my head spinning. Surprisingly, a letter to the editor was accepted for publication describing exactly why this study was a pair of fetid dingo’s kidneys. The letter itself and the authors’ response to it are quite instructive, which is why I decided to revisit this study.
Before I do, let’s just briefly provide the CliffsNotes version why I wasn’t impressed with the original study. (If you want the gory details, feel free to go back to my original post to refresh your memory.) First, none of the primary outcomes showed statistically significant differences between placebo control and treatment groups. Unfortunately, that didn’t stop the authors from mining their dataset for secondary outcomes. Not surprisingly, they found barely statistically significant differences between the control and treatment arms in a couple of these secondary endpoints plus a surrogate endpoint that they proposed as a biomarker (cortisol slope) even though it’s never been validated as a measure of chemotherapy-induced fatigue. These deficiencies in the study interpretation led me to ask this question six months ago: What do we normally call it when there is no difference between the real treatment and a sham treatment in a clinical trial testing a drug or device? That’s right. We say there’s no effect greater than that of a placebo and that the trial is negative; i.e., the tested experimental intervention doesn’t work.
Let me remind you that that is not what the authors concluded. Here’s a snippet from the conclusion of the paper:
This RCT examined whether biofield healing, compared with both active (mock healing) and waitlist control groups, positively affected fatigue as well as cortisol slope, depression, and QOL in breast cancer survivors with persistent fatigue. In addition, this study explored the role of belief in receiving healing as a potential predictor of responses. Findings indicate that both touch-based interventions reduce fatigue in fatigued breast cancer survivors, with considerable effect sizes. Previous research by our group on a separate sample of breast cancer patients indicated that the mean Multidimensional Fatigue Symptom Inventory-short form total scores was 5.99 immediately before the start of anthracycline-based chemotherapy, and rose to 19.9 immediately before the fourth cycle.38 Our fatigued survivors in the mock healing group (mean postintervention score Â¼ 10.9) dropped to fatigue scores lower than those found for breast cancer patients toward the end of chemotherapy, and the biofield healing group (mean postintervention score Â¼ 4.2) fell to fatigue scores that were below prechemotherapy scores, as well as below previously published means noted for breast cancer patients overall.28 This drop in fatigue appears to have clinical as well as statistical significance.
In other words, according to the authors, because both the “real” biofield healing and the “mock” biofield healing resulted in a decrease in fatigue scores and a barely statistically significant difference in a questionable surrogate marker, biofield healing “works.” This is the same sort of dubious rationale frequently used to claim that acupuncture “works” when researchers find that sham acupuncture results in the same apparent measured effects as “real” acupuncture. Again, the correct interpretation of this study is that it’s a negative study, and “energy chelation’ does not work. It is placebo. Stick a fork in it; it’s done.
Surprisingly, the editors of the journal that published the original study, Cancer, accepted a, that’s essentially the point of a letter to the editor that says, in essence, exactly the same thing. One of the authors of the letter is known to many of my readers, but that’s all I’ll say about that matter. What’s far more important is the message:
The registered primary outcomes of the trial were self-reported fatigue, depressive symptoms, and quality of life. No significant differences were obtained between TT and the mock treatment, whereas both conditions were superior to waitlist control. Essentially, the authors examined 3 primary outcomes, with secondary analyses of 5 subscales of the fatigue measure, and a secondary outcome, cortisol, with all pairwise differences explored between the TT, mock treatment, and waitlist control conditions. With any control for multiple comparisons, the modest difference between TT and mock treatment in cortisol is no longer significant.
There is no known therapeutic benefit to changed cortisol slopes. To justify cortisol as a secondary outcome, the authors selectively cite findings that flatter cortisol slopes are modestly related to metastatic disease and predict mortality in breast cancer patients. These limited correlational data alone do nothing to establish that cortisol is a suitable surrogate endpoint.
The letter even explicitly pointed out that this trial was negative, something I believe to be even more true when one takes into account prior probability and the lack of correcting for multiple comparisons:
We believe that publication of this TT trial encourages more pseudoscientific studies of energy fields or auras and gives the wrong message to clinicians and patients.
Not surprisingly, the authors of the original study were not too happy about the letter. Even less surprisingly, in their response they retreated to common tropes used by apologists for reiki and “energy healing.” In fact, their response is a veritable template for defending tooth fairy science. First, the authors tried to disabuse their critics of their “misconceptions” about the study by pointing out how very wrong they were to lump “energy chelation” in with healing touch (HT), therapeutic touch (TT), and reiki as an “energy healing” modality:
We wish to clarify some misconceptions put forth by Coyne, Johansen, and Gorski regarding our reported randomized controlled trial.1 First, the intervention used was not therapeutic touch but a specific hands-on technique commonly used in many types of biofield therapies for ameliorating fatigue.
Which matters not at all. It was the authors, after all, who said that energy chelation is a “biofield therapy,” which is another name for “energy healing.” In any case, their complaint reminds me of arguing that reiki is different from TT because reiki masters use different hand motions to channel the “healing energy” or that in reiki the energy comes from the “universal source” while energy chelation removes “energy blocks” in the patient himself. Until you can convincingly demonstrate that this “universal source” exists and can be manipulated by reiki masters and/or that there are “energy blockages” that “energy chelation” practitioners can remove, all you’re doing is comparing two different forms of magic. Alternatively, you can demonstrate with overwhelming indisputable evidence so powerful as to make us question previously understood laws of physics indicating that these techniques cause objective responses, but unfortunately for the authors of this paper this study does nothing of the sort. What this study found were effects due to energy chelation that were no greater than placebo on primary endpoints, all of which are subjective responses, and an unvalidated surrogate endpoint that demonstrates a barely statistically significant effect (p=0.04), after no correction for multiple comparisons. In other words, the authors’ response completely misses the point.
Here’s their next objection:
Second, there is an evidence base for biofield therapies.2,3
The authors cite this Cochrane Review and one of their own reviews. The problem with this argument, of course, is at the very core of the difference between “science-based medicine” (EBM) and “evidence-based medicine” (SBM). In EBM is no consideration of prior plausibility based on basic science in the studies examined, which were all over the place as far as quality goes. Add to what the Cochrane Review characterizes as poor quality, equivocal data the fact that the clinical trials involved testing a class of healing modalities whose explanation if effective would require, as homeopathy would, that huge swaths of well-established physics, chemistry, and biology (particularly neurobiology) to be overthrown, and the reasonable conclusion is that “biofield therapies” do not work. In particular, this is a case where “statistically significant” doesn’t mean “clinically significant,” not by a long shot, given that the decrease in pain reported in the Cochrane review was less than 1 unit on a typical pain scale that goes from 1 to 10. One unit has commonly been viewed as the smallest decrease in pain that a patient can perceive.
Objection number three follows:
Third, the study was designed to examine nonspecific and placebo elements that may drive responses: This is why we used the mock healing group as a comparison along with the waitlist control group. We also examined patient expectancy, belief, and patient ratings of practitioner attributes all elements of placebo as potential predictors.
So what? It was the authors who concluded against the evidence in their very own study that, in essence, their “energy chelation therapy” works to relieve symptoms of chemotherapy-induced fatigue even though the “real” energy chelation and the “mock” energy chelation were indistinguishable. The correct conclusion should have been that energy chelation performed no better than placebo and therefore did not work. Again, if energy chelation were a drug therapy, would the authors conclude from a result in which the drug does no better than placebo for the primary outcome measures that the drug worked? Why the double standard?
Up next is this objection:
Fourth, despite Coyne et al.’s seemingly contradictory statements (stating that the study is underpowered while also suggesting cortisol slope results should have been Bonferroni corrected), the power analysis and statistics are correct and clearly described.
“Contradictory”? I fail to see what’s “contradictory” in pointing out that the study was underpowered and that the cortisol slope results should have been Bonferroni-corrected; i.e., corrected for multiple comparisons. They are separate criticisms. Even if the study were adequately powered, it would still be flawed because of the lack of correction for multiple comparisons. At least if the authors had properly corrected for multiple comparisons then they could have blamed their negative result on the inadequate statistical power of the study!
Finally, the authors write something that both amused and depressed me at the same time:
The larger issue is what constitutes “pseudoscience” and what information is worthy of dissemination to the public. Should the data from our well conducted, rigorous, randomized controlled trial be dismissed because the mechanisms are unknown or because some scientists do not believe in the specific therapy? We make no claims surrounding mechanisms. We do note that this intervention has significant promise for reducing fatigue, which is the most common complaint among cancer patients, and the therapy produces no harm. Therefore, it merits further investigation. Premature rejection of findings from rigorous randomized controlled trials are as big a threat to science as the continuation of falsehoods based on belief. Thus, as clinicians and scientists, our highest duty to patients should be to investigate promising solutions with high benefit/risk ratios, not to act as gatekeepers of information based on personal opinion.
This paragraph is so wrong that it’s not even wrong. Note the wounded cry about “dismissing” results based on dogma rather than science. Note the straw man argument that supporters of SBM reject the results of this study because they “do not believe in the specific therapy” or because the “mechanisms are unknown.” The first trope is a massive misstatement of SBM objections to studies such as this. A more accurate characterization would be to say that the results of this trial do not mean what the authors think they mean. The authors conclude that the trial indicates that energy chelation shows “significant promise.” (They even repeated that assertion in their response!) An SBM-derived analysis would have concluded that the study’s own results indicate that energy chelation functions no better than placebo and therefore does not work. It would have been nice if the authors had addressed the actual criticism than such an easily revealed straw man version of it.
The second trope is commonly used in defense of pseudoscience because it is difficult for many to understand that there’s a huge difference between a mechanism that is “unknown” and a mechanism that is physically impossible based on current scientific understanding. An example of the former is a drug whose mechanism of action is as yet unknown but whose effects are easily documented. We know that the drug must function through some sort of biochemical interaction with a receptor, enzyme, or other macromolecule within the cell that we can discover and that we do not need to invoke mechanisms that break the laws of physics and chemistry to explain the drug’s effects. Again, to illustrate the difference between such a drug and impossible mechanisms, I like to use the example of homeopathy, which, if it worked would necessitate the overthrow of huge amounts of exceedingly well-established science in multiple disciplines, including physics, chemistry, and biology. Let’s just put it this way. Energy chelation is the same. For it to “work,” the same sorts of vast quantities of well-established science would need to be overthrown.
As a skeptic, I have to admit that it’s certainly possible, albeit infinitessimally so, that so much of what we understand about science is not just wrong and/or incomplete, but so incredibly wrong and/or incomplete that there might be an as yet undiscovered physical mechanism by which energy chelation and “biofield therapies” could work. However, if you’re going to convince me that something like energy chelation can truly work as a treatment modality, you’d better have evidence far more compelling than a small, equivocal clinical trial like this in which the effect on primary outcomes was no greater than placebo and whose analysis of secondary outcomes didn’t even bother to correct for multiple comparisons. Bringing a study like this “energy chelation” study to argue that “biofield therapies” work (or even that they might work) is akin not just to bringing a knife to a gun fight. It’s more akin to bringing fists to an M2 Browning machine gun and grenade fight. Or maybe bringing cavalry to a tank battle.
And that’s not just our “personal opinion,” either. Of course, it’s far easier to dismiss criticisms that one can somehow label “personal opinion” than it is to address the actual criticisms. That’s probably why the authors try to characterize the objections to their study as nothing more than a disagreement of opinion, before floating off into the ether of a self-righteous and condescending lecture the “highest duty” as clinicians and scientists. I retort that, as clinicians and scientists, it is our highest duty not to engage in magical thinking that subverts science-based medicine. It’s our highest duty not to waste precious resources investigating therapies so utterly implausible that their efficacy requires that the laws of physics be overturned in favor of magic. It’s our highest duty to base our treatments in science, not in prescientific vitalism and religion, which is all that most “biofield” therapies are: faith healing, the laying on of hands. I would also retort that our tax dollars should not be funding magic like this.
Yes, you guessed it; the original energy chelation study was funded in part by NCCAM.
- Jain S, D Pavlik, J Distefan, RR Bruyere, J Acer, R Garcia, I Coulter, J Ives, SC Roesch, W Jonas, and PJ Mills (2012). Complementary medicine for fatigue and cortisol variability in breast cancer survivors: A randomized controlled trial. Cancer 118: 777-787. DOI: 10.1002/cncr.26345
- Coyne JC, C Johansen, and DH Gorski (2012). Letter re: Complementary medicine for fatigue and cortisol variability in breast cancer survivors: A randomized controlled trial. Cancer, E-pub ahead of print. DOI: 10.1002/cncr.27415.
- Jain S, D Pavlik, J Distefan, RR Bruyere, J Acer, R Garcia, I Coulter, J Ives, SC Roesch, W Jonas, and PJ Mills (2012). Response re: Complementary medicine for fatigue and cortisol variability in breast cancer survivors: A randomized controlled trial. Cancer. E-pub ahead of print. DOI: 10.1002/cncr.27421.
- So PS, Y Jiang, and Y Qin (2008). Touch therapies for pain relief in adults. Cochrane Database Syst Rev CD006535. DOI: 10.1002/14651858.CD006535.pub2