Just a day into the New Year I was feeling feisty and issued a challenge to readers and the Evidence Based Medicine (EBM) blogosphere in general. I asked for a critique of a fictitious uncontrolled, non-randomized non-blinded small scale clinical study. It was truly a fictitious study. I made it up. But I had a template in mind and I intended to go somewhere with the example and I still do. But it will take longer to get there than I anticipated because it has raised a lot of things worth thinking and talking about in the meantime. I was going to wait a week to give people a chance to read and see the original, but you responded so quickly there is already plenty to talk about.
Before I start with this installment, let me be clearer why I am doing this. At the outset (the first few key taps) the impetus was irritation over the way certain kinds of studies were automatically anointed with credibility if they had the magic words “placebo controlled” or “randomized” sprinkled over them; irritation over a failure to appreciate the value of other kinds of credible scientific evidence that weren’t seasoned with the magic words; and most importantly, I was faced with a choice whether to write a post I could do off the top of my head or work on my grant proposal. It was, as they say, a no-brainer. And no-brains is not a way to write a good grant proposal (although I’ve reviewed many that show abundant evidence they were written that way). But no-brains for a blog post can still work. Evidence: we’ve been up and running and quite successful for more than 5 years.
However there was — and is — an even more important motive. Over a long academic career I’ve sat on many a committee where a final written product was expected and I’ve observed that people fall into two main camps. Those that can’t write a word until they have a detailed outline; and those that couldn’t make a detailed outline until they saw what they had written. For better or worse I am in the latter category, like the writer of fiction who doesn’t know what is going to happen to her characters until she reads it. So this exercise is also a challenge to myself — I hope with your help — to think through this problem afresh by writing about it here. I hope when we are all done it won’t be just me who has a better idea of what this is all about, even if we disagree.
I want to start off the deconstruction of my challenge example with some meta-comments, i.e., comments about the comments (if you want to see the details of the challenge — there aren’t that many, just a couple of paragraphs, but too many to repeat each time — go to the original or David Rind’s excellent summary, here.) Rind has written an extremely interesting post of his own on our challenge at his blog, Evidence in Medicine, which I recommend highly. The first point he makes there is this:
I glanced at the comments on the blog, and interestingly a lot of people are spending time trying to apply a name to what exactly this study is (i.e. “case series”, “observational”, and the like). I often don’t find naming things all that helpful unless everyone has a clear understanding of the naming convention, but this would most commonly either be called an observational study (if we view that the therapy was going to be administered independent of the interest in collecting data about what happened), or an uncontrolled clinical trial (if we view that the primary goal was to find out if the AED works for refractory hypertension).
I think this is an extremely astute observation and it is part of my motive for the example. The desire to name study designs is common and serves a useful purpose for those of us who work with them a lot. It’s an organizing device that mentally allows us to go directly to certain issues or modes of analysis without having to go through a long inventory of concerns. If you speak the language, names and phrases carry along with them context and experience. But if you aren’t quite so fluent, it can be dangerous, and I deliberately chose an extreme — an unblinded, uncontrolled non-randomized study of a small convenience sample — that had none of the magic words and all of the things students and non-specialists are told to watch out for and avoid because they will invalidate the results. I had another end in view, too, but I will keep that to myself for the moment. I agree with Rind that in this context it isn’t very helpful. More importantly, it obscures what I want to talk about, the logic of the inquiry and the grounds that might or might not warrant paying attention to it. I want to get to them, with you. But since this post is already long, I will confine it to those meta-comments I mentioned earlier, meta-comments that are in addition to or extend Rind’s.
1. I confess I deliberately seduced you into considering this a scientific study by saying it was subsequently published in a peer reviewed journal. But I gave you no reason to believe the investigator was a scientist or that he or she was doing science. I am a physician and a scientist, but most physicians aren’t scientists and most scientists aren’t physicians. In this particular case what I described is what most of us hope our own doctors are doing: learning from experience and using it — appropriately — to help patients. Most good doctors do this. They read something in the scientific literature (while not scientists, they, like most of you, are consumers of the results of science), make some inferences or deductions and incorporate all this, provisionally, into their practice. Now EBM encourages doctors to rely on sound science, it is true, but doctors are above all else, problem solvers. If they have a patient whose blood pressure won’t respond to the usual therapies, they will cast about for something that works. If this weren’t so, medicine could be practiced by rote, by a checklist. The only difference here is that this doctor thought it through carefully ahead of time and kept special records so that he or she could harvest the results of experience better. At some point they also thought of sharing it with others.
So here’s another question for you: Does status as “a study” make any difference? Does the fact that the doctor was not a scientist or necessarily doing “science” in the way many of us understand it affect in any way how reliable the information is or whether she should be using her experience in the treatment of her patients?
2. Another meta-comment. It is interesting how many of you started your comments by saying, “I’m not a doctor . . .” or “I’m not a scientist . . . ,” or even, “I’m not an epidemiologist,” the implication being that your comments were not as well grounded as they would be if you were. Depending upon what you were opining about that may or may not be true. In this particular instance I think it’s not so relevant. The great mathematical physicist John von Neumann once said. “You never understand mathematics, you just get used to it.” While many of us would give our eye teeth to be as used to mathematics as von Neumann was, there is a virtue in trying to understand why we do certain things. Many of us in “the business” have long stopped trying to see things from a naive viewpoint. We are used to it and we substitute that familiarity for understanding. If we ever dare to look under the hood — I mean really look under the hood — we see problems that are so deep and daunting we put them in the category of settle issues (settled by others smarter than we are) or time-wasting meta-physical exercises of no practical use. The uninitiated, however, have not yet been blinded by the intensity of the difficulties and will ask naive but quite interesting questions. It is our intention, and we think it a good one, to discuss some of those things (next stop will be causality, but in the context of this example and Rinds’s post). Because those questions go to the heart of the challenge and make it much more interesting than a class exercise in critiquing a study.
That’s our new good intention, anyway. Of course we all know where the road paved with good intentions leads. We’ll take the next steps shortly (maybe a couple of days). Meanwhile we are eager to hear from you on the subject, should you be so moved.