If randomized clinical trials don't show that your woo works, try anthropology!

It's grant crunch time, as the submission deadline for revised R01s is July 5. However, in a classic example of how electronic filing has actually made things more difficult, the grant has to be done and at the university grant office a week before the deadline if it is to be uploaded in time. So, my beloved Orac-philes, I'm afraid it's reruns today, but, benevolent blogger that I am, I'll post two, one older, one more recent, but both about the same topic. This one's from 2007, which means that if you haven't been reading at least four years it's new to you.

A common refrain among practitioners and advocates of alternative medicine is that the reason randomized clinical trials frequently fail to find any objective evidence of clinical efficacy for their favorite woo is because, in essence, science is not the right tool to evaluate whether it works. In essence, they either appeal to other ways of knowing, invoke postmodernist nonsense claiming that science is just one way of knowing that is not any better than any other ways, or both. The most outrageously absurd example of postmodernist silliness in this regard that I've ever seen was the infamous "microfascist" paper, in which the authors called science "microfascism." Less gleefully silly and over-the-top but peddling essentially the same message was an article claiming that double-blind randomized controlled trials of homeopathic medications are not ideally possible, and, it seems, every so often I come across alternative medicine silliness of this sort.

Today is one of those times.

This time around, it's an article by one Christine A. Barry, from the School of Social Sciences and Law at Brunel University in the UK, who makes an argument I've never heard before, specifically that the best way to study alternative medicine interventions is not those nasty, cold, data-loving randomized clinical trials. Oh, no. Not humanistic enough. No, according to Dr. Barry, the appropriate methodology to study woo like homeopathy is anthropology, according to the abstract:

C. A. Barry, The role of evidence in alternative medicine: Contrasting biomedical and anthropological approaches. School of Social Sciences and Law, Brunel University, Gaskell Room 163, Uxbridge, Middlesex UB8 3PH, UK.

The growth of alternative medicine and its insurgence into the realms of the biomedical system raises a number of questions about the nature of evidence. Calls for 'gold standard' randomised controlled trial evidence, by both biomedical and political establishments, to legitimise the integration of alternative medicine into healthcare systems, can be interpreted as deeply political. In this paper, the supposed objectivity of scientific, biomedical forms of evidence is questioned through an illumination of the multiple rhetorics embedded in the evidence-based medicine phenomenon, both within biomedicine itself and in calls for its use to evaluate alternative therapeutic systems. Anthropological notions of evidence are constructed very differently from those of biomedical science, and offer a closer resonance with the philosophy of alternative medicine. Examples are given of the kinds of evidence produced by anthropologists researching alternative medicine. Ethnographic evidence of 'what works' in alternative medicine includes concepts such as transcendent, transformational experiences; changing lived-body experience; and the gaining of meaning. It is proposed that the promotion of differently constructed modes of evidence can be used to legitimise alternative medicine by widening the definition of what works in therapy, and offering a critique of what people feel is lacking from much of orthodox medical care.

In other words, if that pesky scientific method won't show that alternative medicine has efficacy against disease or symptoms, then the answer is easy: Use a different method and "widen the definition of what works in therapy"! Ugh! I wonder who paid for this study.

Dr. Barry's paper was quite long, 12 pages of two-column text, with only two small tables and no figures; so summarizing it is difficult. However, her thesis appears to include several main arguments (see below). But, first, for you to see where she is coming from, I can't resist pulling this excerpt from near the beginning of the paper. You know it's going to be a load of postmodernist crap from the very beginning:

I wish to problematise the call from within biomedicine for more evidence of alternative medicine's effectiveness via the medium of the randomised clinical trial (RCT). This call originates in part from the motive of ensuring that alternative medicine 'works' before providing it in a publicly funded service. I will suggest, however, that this call is also, in part, political and relates to the agenda of controlling the threat posed by alternative medicine to the long-standing hegemony of biomedicine in the West. I also want to highlight possible omissions and biases inherent in the RCT method, not always visible to its supporters. RCTs usually omit the measurement of important elements of 'what works' in alternative medicine, which often acts in a different way to biomedical drugs. By presenting ethnographic evidence, I wish to show how evidence, when seen from the perspectives of the users and practitioners of alternative medicine, hinges on a very different notion of therapeutic efficacy.

My interest in this debate arises from my experience as a patient of both biomedicine and alternative medicine. I can contribute from an outsider perspective, neither being a biomedical nor an alternative practitioner, but also from an insider perspective of having experienced both as a user.

So she's a fan of the woo. That's OK; it doesn't disqualify her from discussing the matter. However, her false dichotomy between "biomedical medicine" and "alternative medicine" (a dichotomy that I have said on numerous occasions is meaningless and shouldn't exist) really grated on me, and, given her claims that alternative medicine "acts in a different way" than "biomedical medicine," it was clear right away that she was going to lay down some classic altie arguments over why their woo shouldn't be held to the same standards of evidence to which "biomedical" drugs are held. She starts by denigrating randomized clinical trial (RCT) evidence as a class (while disingenuously stating, "I do not wish to discredit the notion of the RCT," as if her bad arguments were capable of doing any such thing, her accusations of "hubris" against advocates of EBM notwithstanding). The biggest postmodernist howler in this introductory section is this:

There is no such thing as The Evidence, just competing bodies of evidence.

Microfascist that I am, I felt my hand moving towards my microtruncheon and my micro-jackbooted foot twitching, ready to do some stomping of this postmodernist drivel--that is, after I successfully (barely!) resisted the urge to spew my drink all over my keyboard and monitor. And, boy, did I have reason! It turns out that Barry's indictment of RCTs boils down to a whine full of straw men (Barry's complaints in bold, my responses in italics):

1. RCT methodology measures what is easily measurable ignoring subtle and complex effects. Not exactly. RCT methodology measures objectively measurable outcomes, so that we can actually figure out if the treatment being studied has any objective evidence that it does anything therapeutic. As for "subtle and complex effects," that's just woo-speak for complaining about EBM's emphasis on measuring actual efficacy. In actuality, most objectively measurable outcomes are in fact complex. If the effects are subtle, to me that's just another word for very weak and probably of minor consequence.
2. RCTs pay little attention to the context or provider of treatments. Barry says this as if it were a bad thing. In actuality, this is not entirely true. For one thing, randomized trials are conducted mainly in academic medical centers; physicians are acutely aware that translation out into the community may not always be easy or straightforward.
3. The placebo concept has powerful healing properties not fully acknowledged in RCT methodology. It's hard not to point out right here that Barry appears to be admitting that any therapeutic effects "observed" due to alternative medicine are in actuality placebo effects. My retort, of course, would be: Why should insurers or taxpayers pay for placebos? In addition, there really is no good evidence that the placebo effect has any real therapeutic effect, except for symptomatology. No objective anti-tumor response has ever been attributed to the placebo effect, for example. No lifethreatening disease has been cured by the placebo effect.
4. Funding for trials is patchy and biased towards products with pharmaceutical industry interests. Ah, yes, the old "Help! Help! Big pharma is repressing real cures!" trope. In any case, with the existence and rapid growth of the National Center for Complementary and Alternative Medicine here in the U.S. and Prince Charles' support of woo in the U.K., more and more money is being made available.
5. EBM can be seen as a political move to protect the medical professions'exclusive expertise in healthcare. No, EBM is a move to improve the way medicine is practiced by culling treatments that have no evidence of efficacy and keeping those that do. This doesn't just apply to alternative medicine, by the way. Quite a few conventional medical treatments are being revealed to be less effective or even ineffective, and properly used EBM will eliminate them, along with the woo. Pharmaceutical companies are none too pleased when that happens to one of their products.
6. EBM can be seen as benefiting hospital managers more than clinicians and arising from the growth of managerialism and audit culture. TRANSLATION: "Audit culture" = "accountability" and a desire to improve measurable outcomes, both of which are to be avoided at all costs in alternative medicine.
7. Compliance with guidelines derived from evidence is low. So what? All this means is that "biomedical" medicine needs to do a better job in getting practitioners to follow EBM, not that we should use our lamentably low compliance with guidelines as an excuse to give up and let woo rule.
8. Teaching of EBM to clinicians can be conceptualised as ritualised practice that does not impact on clinical practice. Once again, if EBM doesn't have an adequate impact on clinical practice, the answer is to work harder to make it more relevant to practitioners, not to let woo in on equal terms with scientific medicine.
9. Evidence may be in conflict with patients' wishes in patient-centred medicine. When this happens, patients are free to seek out woo if they so desire. Ethically, however, because there is no good evidence to support the use of the vast majority of alternative medical modalities, we are under no obligation to use them and indeed an argument can be made that it would be unethical to recommend a course of treatment that is not supported by evidence. In fact, ethically we are obligated to offer only treatments for which we have evidence of efficacy. In addition, this obsessions alties have with the supposed evils of RCTs neglects that RCTs are not the only form of evidence. Indeed, EBM takes into account a variety of evidence, from nonrandomized trials to the gold standard double-blind RCT. It's a strawman to imply that double-blind RCTs are all that is acceptable.
10. RCTs come low down in the hierarchy of patients' decision making factors. Possibly, but this is a non sequitur. If the range of options offered to patients all have RCTs giving different probabilities of efficacy, as long as the patient is informed and understands, he or she can choose whatever he wishes. And, of course, it is perfectly appropriate to take into account things like treatment difficulty and side effects. But it's impossible for a patient to make an informed choice without hard data.

In contrast, to Barry, alternative medicine is so much better, as long as you redefine what "efficacy" means. Check out this additional howler, this time about homeopathy:

The nature of Homeopathy precludes the straightforward administration of clinical trials to measure it. Based on the principle of treating like with like, homeopathic remedies are developed from substances in the natural world. A picture of the symptoms of these substances is catalogued by 'proving' the effects on healthy volunteers. A muchdiluted form of the remedy is then administered to patients suffering with a picture of symptoms that is closest to that particular remedy. Each remedy picture includes multiple physical symptoms in multiple body locations, diverse psychological and emotional states, and aspects of behaviours that are not part of biomedical diagnoses. For example symptoms that improve on violent motion, particularly dancing (irrelevant to a biomedical diagnosis) is one of the keynote aspects of the symptom picture for Sepia (Vermeulen, 2000). Different individuals with the same biomedical diagnosis will be prescribed different remedies, as their symptom and personality picture will likely be different in each case. So two important aspects of homeopathy, individual prescribing and attention to non-biomedically recognised 'symptoms', problematise the use of RCT methodology.

RCTs carried out on alternative therapies necessarily entail reducing the complexity of the intervention to fit the reductionist nature of the RCT method. As a result, the therapeutic intervention as tested in RCTs is in most cases quite different to the interventions used by alternative practitioners in everyday clinic situations.

Moroever, apparently alternative medicine "works" if the patient believes it works:

Users of homeopathy did not see a need for scientific testing and were happy with their own judgement of whether the treatment was working for them. Any existence of RCT proof that the remedy was efficacious was never sought. The science of biomedicine was perceived as oldfashioned and rejected in favour of the quantum and chaos theories of modern physics. Several referred to Capra's (1976) book on parallels between eastern mysticism and quantum physics as a rationale for favouring a more modern notion of scientific enquiry about healing. Interestingly, Verhoef (2004) showed in qualitative research with CAM patients, that RCTs came bottom of their hierarchy of evidence. Anecdotal evidence, particularly from friends and family, rated highest.

Of course, anectodal evidence is almost completely useless for determining efficacy, given the multiple biases, including confirmation bias, regression to the mean, and many others. What Barry considers "complexity," I consider not having any idea what you're doing, hiding behind "individualization" of medical treatments to each "unique" patient as a means of masking the lack of any evidence to support the efficacy of homeopathy or any scientific plausibility to suggest how or why it could possibly "work." Moreover, I always know that I'm dealing with Grade A primo woo whenever I see any sort of appeal to quantum and chaos theories as a reason for why alternative medicine works, especially with the dismissal of "biomedicine" as quaint or old-fashioned. In reality, it's alternative medicine that's old-fashioned, given that the vast majority of it was envisaged long before we understood how the body actually worked or the scientific mechanisms of disease.

Barry goes on to argue that the very act of testing alternative medicine in RCTs irrevocably alters the nature of the treatment being study by "medicalizing it" and making it more like a "biomedical intervention and that the blinding procedure interferes with treatment. She seems to think this is insight. I call it bullshit.

So what is the answer? According to Barry, we need to study the efficacy of alternative medicine using anthropology:

Just as the scientific laboratory method and the nature of population statistics have shaped the nature of RCT evidence, so too anthropological method influences what constitutes evidence. Ethnographic research is conducted in everyday reallife settings and so can pay attention to the all-important contextual features of interaction.6 Reality is seen as ever-changing through a series of processes, formed by interactions and relationships between people, and always affected by the context in which social actions take place. The method utilises an observer situated in the context, not researching from afar. The focus of research is neither wholly predetermined nor tightly structured. This allows for research to uncover issues of importance to participants that may have been ignored in the literature. Shifting the focus to the perspective of the actors involved, and seeking the native point of view, not mirroring the prior concerns of the academic community, can produce powerful new interpretations.

She concludes:

Each of these pieces of ethnographic research contributes to a different notion of evidence in alternative medicine. Each produces a different answer to the question 'Does it work?' The work of anthropologists has come much closer to investigating the power of alternative medicine as it is viewed by those who use it. What 'works' for alternative medicine users and their therapists do not just include relief from physical symptoms. It also includes changes in beliefs about health, healing and disease; the gaining of meaning of illness experience in the context of the life story; bodily experiences and changed view of body-self; transcendent, transformational and spiritual healing experiences; changed identity; and a powerful dialectic relationship with the therapist. None of these aspects of therapeutic effectiveness is measured within existing clinical trial research. Whilst there are calls to include quality of life measures in RCT research more generally, such measures still do not incorporate any of the above.

In other words, inappropriately adapting a discipline (anthropology) to study something that it was not intended to study (whether a medical intervention has a therapeutic effect) will produce the result that alternative medical practitioners want: a "transformation" of alternative therapies found ineffective using scientific medical methods to one that is now magically "effective" because its users and the sociological group they are in believe that it works.

But that's not the silliest part of the article. This is:

Such alternative evidence may prove useful in what David St. George, an NHS consultant, has called the potential for holistic transformation of the NHS through a synthesis of science and spirituality into a new paradigm. As he puts it 'Perhaps alternative and complementary therapists can help the NHS to break out of its own prison'...Anthropological and other qualitative forms of evidence may prove a political tool to assist in this enterprise of transformation.

Ethnographic research in alternative medicine is coming to be used politically as a challenge to the hegemony of a scientific biomedical construction of evidence. The introduction of ethnographic forms of evidence that represent the grounded experience of users and therapists of alternative medicine communities act as a critique of biomedical notions of evidence. Thus anthropological evidence can be used to open a debate about what one should be measuring as evidence of alternative medicine efficacy, and whether one should be measuring it at all.

My guess is that alties would say we shouldn't be measuring pesky things like objective outcomes at all, because that would reveal their favored form of woo to be ineffective, but that's just the microfascist in me talking. Of course, the very premise of this conclusion is dubious at best, because I do not accept that a "holistic transformation" incorporating spirituality into a "new paradigm" is necessarily a good thing. In reality, it's just an excuse to mingle woo with scientific medicine on supposedly equal terms. At least Barry is upfront about arguing that the purpose of advocating this shift to anthropological evidence is designed to support a political challenge to the "hegemony" of science, rather than an actual scientific argument. At least that's something, and it allows those whose minds aren't full of postmodern nonsense to recognize a crock when they see it.