I’ve noticed that whenever I have the temerity to suggest (e.g., here and here) that maybe the word of the Cochrane Collaboration isn’t quite the “last word” on the subject and indeed might be seriously flowed, I hear from commenters and see on other sites quelle horreur reactions and implications this blogger doesn’t believe in the scientific method. Why? Because “everyone knows” that a randomized controlled trial (RCT) automatically beats out any other kind of medical evidence and any Cochrane review that systematically summarizes extant RCTs on a subject like flu vaccines is therefore a highly reliable source of evidence. In the spirit of full disclosure, I am an academic epidemiologist who has made a living out of doing observational studies, i.e., studies where I don’t get to assign the independent variable, much less randomize. I can only “observe” the outcomes (the dependent variable) and infer causal effects from my observations. That might explain why I am over sensitive to the prevalent worship of meta-analyses of RCTs as practiced by the Cochrane collaborators. Or maybe I am just blinded (or even double-blinded) by what I do and my ignorance of scientific method.
So I’d like to try something with my readership. It’s not an “experiment” in the formal sense, just tossing something out there as “shark bait” to see what happens. I’m going to describe a fictitious observational study and I’d like you, the readers, to tell me why it isn’t something that should form a reliable basis for taking some action (e.g., treating someone). In a later post I’ll respond or at least tell you the point of the example (which, while fictitious, represents actual practice in some quarters of clinical medicine).
Here’s the set-up:
On the basis of some good data from animals and pharmacologic studies I believe an existing anti-epilepsy drug will be effective for patients with refractory hypertension, i.e., people with high blood pressure who are not successfully treated with existing therapies. This is an FDA approved and drug that is considered acceptably safe for epilepsy, but blood pressure treatment is an “off label” use (I believe this would be both legal and ethical, but since we are discussing scientific method it is not germane at this point). There is no interest in this from the drug companies because this is already a generic drug, so there isn’t enough money to be made on it and there are no funds for an expensive RCT involving thousands of subjects. So I decide I will try it on patients in my cardiology practice with refractory hypertension.
I have carefully researched the alleged biochemical mechanism ahead of time, cross-checked it against lots of clinical data, and decided, on the basis of the science, exactly which patients with refractory hypertension should be the ones for whom this therapy will work. I write the criteria for treatment down and give them to one of my medical residents so she can go through my practice’s current patient histories and monitor new patients as they enroll in the practice, selecting ones with refractory hypertension and of those, deciding which ones will get the drug on the basis of my written criteria. Over the space of two years she finds 52 candidates and decides 29 fit the criteria. I don’t pay her but promise she can have her name on any paper we publish. While not every patient is a perfect fit, all meet a rough minimum standard.
Each of the 29 selected patients is given the drug at their next office visit and told to take one pill twice a day until the next visit, usually about a month. No one gets a placebo and of course that means we know which ones got the drug. They all did. The patients also know they got the drug. So there is no blinding. On the next office visit we measure their blood pressure again. Both the before and after blood pressure measurements are made with a digital reading cuff that requires no intervention from the clinical assistant besides putting the cuff on the arm and recording the results in the chart.
The difference in blood pressures is our quantitative measure of the effect of the drug. If the drug had no effect there shouldn’t be any difference. We decided ahead of time to test the difference for “statistical significance” with a linear model and the F-statistic (null hypothesis is the difference in blood pressures was zero). Using these methods there is an apparent positive effect of the drug of about 10 mm mercury systolic pressure and 15 mm mercury diastolic pressure which, because the F-statistic is too big, we do not believe is likely a chance effect. Our study concludes this anti-epilepsy drug can be used for cases of refractory hypertension. We submit the results for publication to peer reviewed journal, it is accepted and becomes part of the medical literature.
This is obviously not a randomized controlled trial because it is neither randomized (everyone got the drug) nor controlled (everyone got the drug). But it does have some strengths and it is these strengths that got it past peer review. First, treatment is well-defined and started at a precisely determined time. This is also true of properly designed RCTs. There is a well defined comparison group: the same person before the drug use. Thus the comparison group is well matched to the treated group (same sex, race, essentially the same age, educational level, income, etc.). This makes the balance of important covariates better than most RCTs. What about measurement of outcomes? Well, that’s a bit more of a problem, perhaps, because we don’t know if the effects on blood pressure we see have any relationship to clinical outcome. We have chosen a clinical measure we think we can make reliably, blood pressure measurement with a digital cuff. It is a commonly used one, so we’ll just go with it. We don’t have the ability to follow-up all our patients for years to see if their mortality is improved. We do have exclusion criteria (or if you prefer, inclusion criteria) decided upon in advance on the basis of the scientific literature. We have trusted our medical resident to apply it properly. Since the trial lasts only a few weeks and the subjects are all long term patients in our practice, we don’t have the problem that many RCTs have of loss to follow up or switching treatment groups but we might have a non-compliance problem. But so do RCTs. Our patients are probably not a representative sample of the population of refractory hypertensives. They are a convenience sample. But they are humans and have refractory hypertension and we don’t know of any factor that specifically makes our patients biologically unlike other people that might have refractory hypertension with respect to response to this anti-epilepsy drug. We analyzed the data with an accepted and appropriate measure, decided upon ahead of time.
So here’s your challenge. This is not an RCT and likely wouldn’t be included in a Cochrane review of the use of this anti-epilepsy drug for refractory hypertension (let’s assume this is true; it is highly plausible). That suggests it is either not a reliable source of information or else that Cochrane reviews are excluding reliable sources of information.
Question: In your judgment, is this a sufficiently reliable study that a reasonable practitioner, committed to the use of scientific evidence in her practice, would consider?
I’ll give it a few days or a week to get a response from the hivemind. Of course any EBM site is free to repost and/or circulate. I’m curious what the reaction will be.