The return of the revenge of a "more fluid concept of evidence"

Dr. David Katz is undoubtedly a heavy hitter in the brave new world of "integrative medicine," a specialty that seeks to "integrate" pseudoscience with science, nonsense, with sense, and quackery with real medicine. In fairness, that's not the way physicians like Dr. Katz see it. Rather, they see it as "integrating" the "best of both worlds" to the benefit of patients. However, as we've documented extensively here, on our personal blogs, and even in the biomedical literature (plug, plug), what "integrative" medicine means in practice is indeed what I characterized, the infiltration of woo into medicine. This infiltration seems to have started mainly in academia—hence the term "quackademic medicine" and "quackademia"—with the steady infiltration of nonsense into medical schools and academic medical centers, but has since metastasized to the world of community hospitals. This "integration" (or, as I like to refer to it, "infiltration") has become so pronounced that a few years ago The Atlantic published an article entitled "The Triumph of New Age Medicine", and just last December the Journal of the National Cancer Institute (JNCI) published a monograph full of articles touting "integrative oncology," including guidelines recommended by the Society of Integrative Oncology (SIO) for the "integrative" treatment of breast cancer symptoms.

I mention Dr. Katz for two reasons. First, he's taken another broadside at us at Science-Based Medicine in blog entry at The Huffington Post—where else?—entitled "Holism, Holes and Poles" that I've been meaning to address for a while. But before I address Dr. Katz's most recent complaint against science-based medicine (SBM), it's necessary to step back and look at some history.

A more "fluid" concept of evidence

Longtime readers might recall that Dr. Katz is on faculty at Steve Novella's institution, Yale. More importantly he is (in)famous for making a statement that utterly blew me away seven years ago when I first encountered it, courtesy of David Colquhoun, portrayed in this video, specifically beginning at around 6:05:

Unfortunately, the original video I cited in my original post was taken down due to a copyright claim from Yale, but this one remains. The problem is that it is missing bits that were discussed when the talk was widely discussed in the medical blogosphere. However, the money quotes are still there, such as:

I think we have to move beyond the results of RCTs in order to address patient needs, and to do that I’ve arrived at the use of a more fluid concept of evidence than many of us have imbibed from our medical educations…[Referring to an anecdote of a patient with chronic pain Dr. Katz went on.]...Now, we don’t want you on narcotics any more than you want to be on narcotics. We initiated a course of acupuncture and over the next two to three months weaned him off narcotics. He was pain-free on acupuncture and subsequently transitioned into homeopathy. Now, I don’t care to get into a discussion of how or even whether homeopathy even works, but this guy had tried everything.

[…]

And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.

Later in the talk, he mentions an NIH consensus statement about the use of multivitamins that concluded that there was insufficient evidence to determine if they work and says:

And so I would argue that here, too, is an invitation to think more fluidly about evidence. Absence of evidence is not evidence of absence.

Whether or not absence of evidence is evidence of absence depends, of course, on how hard scientists have actually looked for evidence. Of course, the proper retort to such seemingly-sensible nonsense is to suggest that maybe—just maybe—you shouldn't be making such confident recommendations until there is actually some convincing supportive evidence. Moreover, in the context of discussing a bunch of negative clinical trials that Dr. Katz apparently doesn't believe, it is fairly clear that he's implying that negative clinical trials are an "absence of evidence." They are not. They are evidence. While from a frequentist perspective, it is true that a negative clinical trial does not necessarily mean that a treatment does not work (it just means that the results observed in the treatment and control groups were statistically indistinguishable, to put it in simple form), when negative clinical trials, particularly if they are well-designed, pile up, that absence of positive evidence that a treatment works becomes fairly powerful evidence that it doesn't work. In any case, this "more fluid concept of evidence" clearly involves patient anecdotes, which are, for the many reasons explained here over the years, often unreliable and even misleading. That is why they are relegated to a much lower rung on the ladder of evidence than clinical trials.

As you might imagine, quite a few skeptical bloggers have discussed Dr. Katz's idea of "a more fluid concept of evidence," including Steve Novella, Kimball Atwood, and myself. More recently, Dr. Katz was very unhappy over how Steve Novella and I had criticized the Cleveland Clinic for opening a traditional Chinese medicine clinic run by a naturopath. I have characterized David Katz's attitude as a false dilemma: Abandon patients or abandon science. And so it is. Unfortunately, it is a false dilemma that lies at the very heart of the very concept of "integrative medicine," as you will see.

Dr. Katz lashes out again

Dr. Katz is once again very unhappy with us over here at SBM. One can't help but notice right away how he refers to us (without naming us, although he does at least link to us) as the "self-proclaimed guardians of the true definition of evidence-based medicine" (EBM). Ouch. He sure makes us sound as though we are uptight, doesn't he? Except that we don't really do that. Rather, we advocate changing the definition of EBM to be more like SBM because of the obvious defects in the definition of EBM that allow pseudoscience to infiltrate medicine. It's not as though we haven't discussed this more times than I can remember.

I get it, though. No one likes being criticized or having his favorite ox gored. I can even sort of understand why Dr. Katz seems to think that we are a bunch of peons unworthy to criticize him, as once again he has to lay down his bona fides in a rather obvious appeal to authority:

I opened an Integrative Medicine Center some 15 years ago, and have directed it, and seen patients in that context, since. I did so not because of any long suppressed desire to practice medicine under that banner, and certainly not because of any inclination to don Birkenstocks.

In fact, I was -- as I remain- a card carrying member of the evidence-based medicine club. I was already running then, as I still am, a federally funded clinical research lab. I was already then, as I am now, routinely publishing studies in the peer-reviewed literature. I was teaching then, as I did for roughly ten years, biostatistics and clinical epidemiology to Yale medical students. And while back then I had co-authored a textbook on epidemiology and biostatistics, I have since co-authored four editions of that textbook, and a textbook on evidence-based medicine as well.

In other words: How dare you miserable, worthless, cretins criticize me? Me! Yes, Dr. Katz has been using variants of this same whine since he dismissed David Colquhoun, Steve Novella, Dr. R. W. Donnell, and myself as not having "earned the right to express an opinion" on these issues and of "simply having an opinion, an Internet connection, and some time to kill." He even referred to us as "health hazards of the blogosphere" and quacks.

Although not to the extent that Dr. Katz has, I, too, have been federally funded and I, too, routinely publish in the peer-reviewed literature. Unfortunately, I probably stuck too closely to basic science too long before transitioning it into the realm of clinical research, given that a surgeon will always have a great deal of difficulty directly competing with basic scientists at basic science, hence my difficulties retaining external funding more recently. In retrospect, I'm amazed I did as well as I did for as long as I did competing with basic scientists and am hopeful that my recent forays into much more directly clinically relevant areas are not too late, given my priority score on a recent grant that just missed the draconian cutoff for funding in today's—shall we say?—highly constrained funding environment.

Of course, Dr. Katz's impressive qualifications, which I concede are more impressive than mine, do not matter much when he routinely spouts nonsense such as his justification of the use of homeopathy, naturopathy, and traditional Chinese medicine, as he has unfortunately done so often in the past. Only the quality of his arguments for the "integration" of various alternative medicine practices into conventional medicine matter, and these arguments have been uniformly of very poor quality. This time around is no different.

I'll start with his third complaint, because it is a complaint so frequently used by advocates of quackery (whom, whether Dr. Katz realizes it or not, he is emulating by repeating that complaint), which is all about the money:

And third, there is the thorny little issue of money.

The horses that pull toward "evidence" pull a cart full of money. The average cost of bringing a new FDA-approved drug to market now approximates a billion dollars. Spending millions, tens of millions, or even hundreds of millions on clinical trials makes sense if a return in the billions is the offing. It's pretty much a non-starter otherwise, and a problem even a budget the size of the NIH's cannot solve. In fact, the price tag for one, new FDA-approved drug is nearly ten times the annual operating budget of the National Center for Complementary and Integrative Health.

Hmmm. Where have we heard this one before? I wonder…

Oh, yes. It becomes clear when Dr. Katz proceeds to use his favorite tale to illustrate how money supposedly rules everything in medicine. Basically, it's a tale of co-enzyme Q10 (CoQ10), a tale he's told many times before going back to his original "more fluid concept of evidence" talk in 2008, with periodic repetitions, up to and including Katz's current post. CoQ10 is also known as ubiquinol and is found widely in plants. It acts at the level of the electron transport chain to facilitate the production of chemical energy through oxidative metabolism in our mitochondria. According to Dr. Katz, we tend to get most of what we need from our diet because our bodies don't make CoQ10 very efficiently, although a recent review calls into question Dr. Katz's claim that we get most of our CoAQ10 from our diet rather than our body's own synthesis of it.

In any case, it was thought that CoQ10 would be a good adjunct in heart failure, in order to improve the function of the heart. In 2000, there was a small randomized double-blind placebo-controlled clinical trial that concluded that CoQ10 "does not affect ejection fraction, peak oxygen consumption, or exercise duration in patients with congestive heart failure receiving standard medical therapy," for which investigators concluded "we detected no objective benefit from coenzyme Q10 administration in patients with heart failure" and that their "study shows no benefit to adding coenzyme Q10 to the standard treatment of heart failure." Dr. Katz was very unhappy about this and characterized this study as being used to "drive the final nail in the CoQ10 hypothesis," while mentioning that within a year a large RCT of a prescription drug carvedilol (trade name: Coreg) was published showing that it was safe and effective for the treatment of heart failure. To Dr. Katz, this is the evidence of big pharma primacy and the premature rejection of a promising treatment.

He then goes on:

It took more than a decade after that to undo the damage, but eventually it happened: trial data came in to show that co-Q10 could reduce heart failure mortality by about 50 percent. In other words, ten years after being declared defunct, the co-Q10 "hypothesis" was not only vindicated, but seemed to represent the greatest advance in the treatment of heart failure in years.

First of all, I can't help but note that if the CoQ10 hypothesis was truly dead, then why was a relatively large randomized, double-blind study begun a mere three years later? Something doesn't jibe here. If the 2000 study had truly "closed the door," then no IRB would have approved a bigger study. Clearly, the forces of big pharma were not as powerful as Dr. Katz insinuates. I also find it odd that he didn't actually link to the primary study by Mortensen et al, which was published in JACC Heart Failure in December. In any case, one thing that I noticed about this study right away is that there were no differences in short term endpoints; so in that, at least, this study by Mortensen agrees with the 2000 study. It was only in long-term outcomes that it showed a significant difference, with a major decrease in mortality.

The other thing that I noticed is that, judging from the introduction and discussion by Mortensen et al, Dr. Katz appears to be giving a highly biased account of what happened. There were clearly a lot more uncertainty and conflicting data than his accounts would leave you to believe, and, of course, it was completely appropriate for the original study authors to conclude that, based on their results, it was not appropriate to recommend CoQ10. That's a provisional conclusion justified by the study at the time. That's the way clinical science progresses after preclinical validation. Pilot studies are done. If they are promising, then small RCTs are done. If those are promising (and sometimes if they are mixed, as judging from the introduction and discussion of this paper), then larger RCTs are done. There's nothing outrageous here, and Mortensen et al also acknowledge that their study might have enrolled a larger proportion of patients with milder symptoms than other trials.

In an accompanying editorial, Justin Ezekowitz notes that there were accrual problems, which is why the study took ten years, noting that it's "unclear whether this situation resulted from patients’ acceptance of the experimental therapy, the ability of the sites and investigators to conduct a trial, the intensity of follow-up, or other features that are not further elucidated." He also notes that it is premature to recommend CoQ10 as part of standard guidelines for treating heart failure, noting that these "striking results should be interpreted with caution given the small population and event numbers and the large treatment effect; many of these results were not replicated when a subsequent adequately powered trial was done (11)."

Note that the reference refers to a study by John Ioannidis that found that smaller studies generally produce large treatment effects that decline when larger trials are done.

To complete my deconstruction of Dr. Katz's complaint about no money for studying supplements like CoQ10, I like to echo Dr. Ezekowitz's remonstration with Mortensen et al:

Enthusiastically, the investigators have said that CoQ10 “should be added as standard therapy” based on these results, and additionally, it is a “natural and safe substance” (12). Perhaps this second statement is driven by the fewer side effects in the CoQ10 group than in the placebo group (13% vs. 19%). Further study in adequately powered clinical trials is required because it is premature to suggest that this finding reaches the necessary efficacy or safety bar required to consider prescribing this drug. Additionally, although CoQ10 is widely available and taken as a supplement, safety cannot be assumed for any drug in this context, and so much broader and more detailed experience is required to declare this substance acceptably safe. Finally, CoQ10 is natural, naturally made in a factory by pharmaceutical-grade processes of medicinal chemistry and yeast or bacterial fermentation.

Indeed. What was that about Dr. Katz denying that he believes that "natural is better," again? After all, if CoQ10 does "work," improving outcomes in heart failure—and it very possibly does, given the results of Mortensen's trial—it's working as a drug, and the actual supplement used is manufactured in a manner very much like the methods used to manufacture pharmaceuticals. In particular, I like how Dr. Eskowitz fires a shot across the bow of advocates like Dr. Katz and the supplement manufacturers:

Nutraceutical manufacturers must step up and, like pharmaceutical and device companies, fund a trial for which an indication is sought or advertised, not solely through governmental funding agencies. If 1% of the estimated $1 billion annual market for CoQ10 was reinvested into an adequately powered, appropriately designed, academically led trial to demonstrate the efficacy and safety of CoQ10, then perhaps we would have a scientific advance worth being excited about.

Yes, the lack of patentability is a problem, but it's not an insurmountable one. The problem is that supplement manufacturers want it both ways, and, thanks to the DSHEA of 1994, a law that basically lets supplement manufacturers get away with all sorts of vague health claims without evidence, they can have it both ways. Why would they want to spend even 1% of their profits and subject themselves to FDA pharmaceutical regulation if they don't have to? There's more greed over nutraceuticals than just big pharma and the lack of patentability. Funny how Dr. Katz never seems to acknowledge that.

Integrative medicine "individualization"

Now let's go back to Dr. Katz's "more fluid concept of evidence," which, given his arguments, sure sounds like an excuse to change the rules of evidence in the middle of the game, as Steve Novella has charged. In the case above of CoQ10, he argues that if only there had been a more "fluid" concept of evidence applied to the original 2000 trial, then maybe 13 years might not have been "wasted" and we might have been able to incorporate this miraculous supplement into treatment regimens of people with heart disease. I view it more as a cherry picked example, of Katz being lucky in probably having guessed right. How much more often do clinical investigators guess wrong after an initially-promising clinical trial? That's what the whole concept of the "decline effect" is about, where initially-impressive effect sizes tend to decline with time as further studies are done. As I like to say, the "decline effect is science correcting itself over time.

Be that as it may, Dr. Katz, not satisfied with complaining about funding and nefarious pharmaceutical companies, launches into what I like to call the "personalization" gambit:

First, the evidence from any given clinical trial, however methodologically robust, may or may not pertain to any given patient. The application of evidence from trials to the care of individuals not in those trials is itself a product of the art of medicine, not the science. To my knowledge, there has never been a randomized trial examining different ways clinicians might decide if trial results pertain to a specific patient. That may result from the developing field of pharmacogenomics, but we have a ways to go. In other words, for now, the application of the evidence in evidence-based medicine to the actual care of patients is not, itself, evidence-based. That's a fact.

No, actually, it is not a "fact." It is Dr. Katz's opinion. Also notice how Dr. Katz subtly attacks a straw man version of EBM by emphasizing only RCTs, implying that nothing is known because there hasn't been an RCT of applying evidence-based practice (EBP) guidelines. For one thing, it's just not true that nothing is known about the application of EBM. True, not enough is known, but a lot is known. Moreover, there have been studies of EBP guidelines, both prospective and also both simulated and in the real world. Has Dr. Katz ever heard of a surgical checklist? These are checklists that must be done before surgery to ensure that various EBP items on the list are "checked off," thus ensuring adherence to EBP guidelines. No doubt Dr. Katz would claim that these are not applicable because they are used in a much more controlled, regimented world and don't show how physicians apply guidelines to individual patients, but surgical patients are individual patients and checklists allow for not performing an intervention if it is deemed inappropriate for a given patient.

I also can't help but note that in the case of CoQ10, Dr. Katz seems rather eager, based on one RCT, to adopt a "one size fits all" guideline recommending the addition of CoQ10 supplementation to the treatment of heart failure patients.

Be that as it may, Dr. Katz is also attacking a straw man. No one—and I mean no one—arguing for SBM and against the sort of integrative relativism espoused by practitioners like Dr. Katz claims that randomized clinical trial (RCT) results must always apply to a given patient with a disease presenting for evaluation. That's why it's called science-based medicine. Heck, that's why EBM was originally referred to as evidence-based medicine. That's because, ideally, medicine is based in science and evidence, but science and evidence are not the be-all and end-all.

What the patient values, for example, matters. For example, I routinely tell women diagnosed with breast cancer that mastectomy and lumpectomy followed by radiation therapy both produce equivalent survival rates. In women for whom it is appropriate, I generally recommend breast-conserving surgery (lumpectomy) first. However, even though the survival rates are the same for the two procedures, there is a somewhat higher chance of a local recurrence (recurrence in the breast) with breast-conserving surgery. The reason that that local recurrence doesn't portend a higher risk of death is because such local recurrences can be "salvaged" with mastectomy. Some women have a very hard time dealing with even the relatively low risk of local recurrence inherent in breast conserving surgery even if the survival is the same. There's also the issue that radiation therapy, as given, can be rather inconvenient. Most regimens involve daily treatments for four to six weeks (it used to be always six weeks), which can be a pain if the woman doesn't live close to a center with a radiation oncology facility.

So is it "wrong" to do a mastectomy to treat a small breast cancer that could be treated with breast-conserving surgery? Of course not. I admit that I will at least try to persuade a woman whom I deem a good candidate for breast conserving surgery out of a mastectomy because it's a much bigger operation, but if she insists on mastectomy I will do it. The point here is that informed consent is a critical component of EBM and SBM, but that informed consent must be truly informed. In other words, the information given to the patient must be accurate and based on the best science available. That's where integrative medicine fails, both as EBM/SBM and ethically. I've frequently referred to the concept of "misinformed consent." What integrative medicine does is to introduce treatments not based in science as though they were.

Dr. Katz then invokes a common scenario in medicine:

Second, the needs of patients all too often go on when evidence runs thin, or out entirely. This is why I went into "integrative" medicine in the first place -- and why I am now involved in the online delivery of holistic care to a larger population.

By working with colleagues trained differently than I, naturopathic physicians in particular, we had more treatment options together than we did alone. When patients had been everywhere, tried everything subtended by RCTs and still weren't better, they came to us. And we, generally, could come up with something reasonable to try. Admittedly, it wasn't yet in the textbooks -- but that was the point. These were the folks that had already run off the pages of textbooks, and still needed help.

This, of course, is another straw man view of EBM and SBM. Indeed, one of our criticisms of EBM is similar, but different, than Dr. Katz's. I like to refer to it as methodolatry, defined as the profane worship of the randomized clinical trial as the only valid method of investigation. So there we agree! The RCT should not be the be-all and end-all of investigation! After all, there will never be large, double-blind RCTs to answer every medical question that needs answering. Sometimes it's unethical to do an RCT. Sometimes it's too impractical, either because of expense, time, or both. Sometimes the condition is just so uncommon that there will never be enough patients to do an RCT. For instance, at the recent Society of Surgical Oncology meeting in Houston, it was discussed how doing an RCT comparing outcomes in patients who undergo standard pancreatectomy to those who undergo minimally invasive pancreatectomy (laparoscopic or robot-assisted) for pancreatic cancer with overall survival as the primary outcome would require over 6,000 patients and several years. In such cases, we are forced to use the best evidence available, because sometimes the necessary trials will never be done.

Here's the problem, and here's where Dr. Katz and I diverge. We at SBM advocate that patients are best served when they are not subjected to unscientific and pseudoscientific medicine and when the treatments recommended to them are firmly rooted in science. Just because there is a lack of evidence regarding the best management for a given patient does not mean you can scientifically or ethically justify recommending pseudoscientific treatments like homeopathy as though they have evidence to support them! Irish comedian Dara O'Briain has a brilliant retort to this kind of argument, which is basically of the same general form of (medical) "science doesn't know everything":

Jesus, homeopaths get on my nerves with "science doesn't know everything." Well science knows it doesn't know everything. Otherwise, it'd stop…But just because science doesn't know everything doesn't mean you can fill in the gaps with whatever fairy tale most appeals to you.

That's exactly what integrative medicine does by advocating the use of modalities such as reiki, homeopathy, naturopathy, and much of traditional Chinese medicine. And that's exactly what Dr. Katz is doing when he advocates a "more fluid concept of evidence." Don't believe me? Behold these words from Dr. Katz:

Nowhere in the argument for a more fluid application of evidence to the care of people is there an argument to reject evidence when it is clear. The evidence in support of immunization is clear. A connection between the two argument is specious; a non sequitur.

Note that apparently Dr. Katz was criticized that his a more "fluid" concept of evidence has been used by the antivaccine movement. Now contrast the above passage with these words from Dr. Katz:

And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.

If there's a form of quackery other than homeopathy for which the evidence is more clear that it does not work better than a placebo, that, physically according to what we know from chemistry and physics, it cannot work, I am unaware of it, with the possible exception of reiki. In 2008, Dr. Katz was clearly willing to reject evidence that homeopathy is pseudoscience. What other clear science has he been willing to reject when he sees fit? How "fluid" is his concept of evidence? In the final section, we shall see.

Dr. Katz's lovefest with naturopaths

Dr. Katz works a lot with naturopaths. We've known this for nearly eight years. Apparently, his definition of evidence is "fluid" enough to embrace a specialty with as tenuous a relationship with science as naturopathy whole-heartedly, to the point where he's recently given testimony to the Public Health Committee in the Connecticut State House of Representatives in support of a bill that could expand the scope of practice of naturopaths and—or so naturopaths hope—give them prescribing privileges. In his written testimony, he brags about how he has "worked extensively with naturopathic physicians in every context relevant to the advancement of medicine" and gushes shamelessly about how he has been "richly rewarded by all of these collaborative activities." He even refers to naturopaths as "ideally suited to meet the primary care needs of Connecticut residents."

What is it about naturopaths Katz so admires that he's willing to go to bat for them in front of the legislature in his own state and speak of them in such glowing terms? Is it their use of quack treatments, such as unsupported thyroid tests and treatments for fake diseases like "systemic candidiasis" and Chronic Lyme disease? Given Katz's admirable defense of vaccines in wake of the Disneyland measles outbreak, is it their near-universal antivaccine views, even among naturopaths who think of themselves as "not antivaccine"? Is it their advocacy of major quackery, including chelation therapy, a potentially deadly pseudoscientific treatment, to treat autism? Is it their advocacy of intravenous peroxide to treat a wide variety of conditions? Is it their advocating "detoxification," herbs, and acupuncture to treat infertility? Is it how naturopaths embrace the four humors? Is this what he means by embracing a "more fluid concept of evidence"? Apparently so. After all, the humors were believed to be fluids.

Or perhaps Dr. Katz really doesn't know what naturopathy is actually about. If that's the case, I'll take the opportunity to educate him and suggest that he read Kimball Atwood's classic critical appraisal of the vitalistic pseudoscientific mystical practice that is naturopathy, after which he might be interested in what naturopaths say amongst themselves when they think no one is listening. Then he should read Britt Hermes' confessions of a former naturopath and her description of naturopathic education. Let's just say, the facts about what naturopathy is and how naturopaths practice contrast starkly with the naïve view Katz presents in his testimony. A lot.

Seven years after he introduced his more "fluid" concept of evidence, Dr. Katz has failed to give a satisfactory or convincing answer to the question of just what he meant. The best he can come up with, after whining about how nasty we "self-appointed guardians of the definition of EBM" with an Internet connection and too much time on our hands are, seems to be either to substitute anecdotal evidence for RCT evidence whenever he thinks it appropriate or to seek a "middle way" between SBM and quackery. If that means embracing homeopathy, traditional Chinese medicine, and naturopathy, apparently that's fine with him. Unfortunately, Dr. Katz is not alone. He is on the vanguard of a whole movement advocating a more "fluid" application of evidence who, with good intentions, have brought rank pseudoscience into bastions of academic medicine.

Additional reading:

1. “Integrative” medicine at Yale: A more “fluid” concept of evidence?
2. Changing the Rules of Evidence
3. The false dilemma of David Katz: Abandon patients or abandon science
4. Science, Reason, Ethics, and Modern Medicine, Part 2: the Tortured Logic of David Katz