I've been meaning to go through the recent meta-analysis of Avandia published by the New England Journal of Medicine that purported to show major increase in the risk for cardiac events (myocardial infarctions and cardiac death) in patients who use Avandia, but somehow never got around to it. I'm not sure I need to now, given how, via Kevin, MD, I've found this rant byThe Angry Pharmacist, who has looked over the meta-analysis and found that there is considerably less there than meets the eye and that the value of the study is considerably different than what has been reported in the press. Key observations cherry-picked from The Angry Pharmacist's rant, which you really should read in its entirety, include:
- Look who funded this "Study" (from http://content.nejm.org/cgi/content/full/NEJMoa072761) Dr. Nissen reports receiving research support to perform clinical trials through the Cleveland Clinic Cardiovascular Coordinating Center from Pfizer, AstraZeneca, Daiichi Sankyo, Roche, Takeda, Sanofi-Aventis, and Eli Lilly. Dr. Nissen consults for many pharmaceutical companies but requires them to donate all honoraria or consulting fees directly to charity so that he receives neither income nor a tax deduction. No other potential conflict of interest relevant to this article was reported. Yeah, does anyone else see something wrong with believing a study that was funded by the COMPETITORS of GSK? I'm sure Dr. Nissen doesnt get ANYTHING from these companies. No sir. He probably doesnt get to use their houses in the Bahamas on their corporate jets and eat with the offical Pfizer credit card. Charity my ass.
- Amazing how they didnt include trials that showed no deaths: Six of the 48 trials did not report any myocardial infarctions or deaths from cardiovascular causes and therefore were not included in the analysis because the effect measure could not be calculated. You think maybe that skewed the data a bit? Lets not include the studies that showed no risk, just the ones that did. Way to go douches.
A couple of points: First, alternative medicine advocates love to point out "conflicts of interest" and are quick to dismiss a study funded by big pharma that produces results that they don't like just because of the funding source. We see this all the time when antivaccination loons immediately dismiss any studies failing to find a link between mercury in the thimerosal preservative in vaccines and autism if any funding came from a pharmaceutical company that makes vaccines. Of course, I don't mean to say that such potential conflicts of interest shouldn't be considered when interpreting the results of these studies, but alt-med advocates often seem to focus on such conflicts above all else, without actually examining whether the science in the paper is sound. Given The Angry Pharmacist's observation (which, I confess, I probably wouldn't have noticed if I had done my own deconstruction of the paper), I wonder if all the alt-med advocates who are crying "I told you so!" or castigating big pharma will even care or consider that there may well be a significant conflict of interest in this study. What's good for the goose is good for the gander, you know.
My guess is that no one will mention this.
The far more telling point seems to be The Angry Pharmacist's observation about the selection of studies to be included in the meta-analysis. Meta-analyses are highly dependent, as you might expect, on which studies are included in the analysis. I'm not a statistician, but the reasoning given by the authors for excluding these studies seems specious to me based on fairly basic principles of the design of meta-analyses and basic biostatistics. In reality, you can estimate a risk from these studies. It's zero. Of course, we know it's not really zero. What this means for these studies is that they were too small to see any deaths, which means that you can estimate the risk of death within the statistical power of the study by saying that there is an 95% probability that it is no greater than X. Unless these studies were poorly designed or had other deficiencies that made excluding them valid, just because no deaths were observed is probably not in and of itself a valid reason to exclude them. The very danger with meta-analyses is, as expressed over at Musings of a Distractable Mind:
We had an endocrinologist in our office a few days ago (not representing GSK) and we discussed this issue, and his comment was that Dr. Nissen is "the Michael Moore of the medical industry." Strong words. Mr. Moore is a crusader against the big and rich for the protection of the little guy (in his opinion). The problem is (in my opinion) that Mr. Moore does not always come to conclusions based on evidence, but starts with a conclusion and finds evidence to support this. This is precisely the danger of a meta-analysis of the sort that was done in this case.
In all fairness, let me cite a bit of the discussion of the meta-analysis, in order to point out that the authors did report many of the limitations of their study:
Our study has important limitations. We pooled the results of a group of trials that were not originally intended to explore cardiovascular outcomes. Most trials did not centrally adjudicate cardiovascular outcomes, and the definitions of myocardial infarction were not available. Many of these trials were small and short-term, resulting in few adverse cardiovascular events or deaths. Accordingly, the confidence intervals for the odds ratios for myocardial infarction and death from cardiovascular causes are wide, resulting in considerable uncertainty about the magnitude of the observed hazard. Furthermore, we did not have access to original source data for any of these trials. Thus, we based the analysis on available data from publicly disclosed summaries of events. The lack of availability of source data did not allow the use of more statistically powerful time-to-event analysis. A meta-analysis is always considered less convincing than a large prospective trial designed to assess the outcome of interest. Although such a dedicated trial has not been completed for rosiglitazone, the ongoing Rosiglitazone Evaluated for Cardiac Outcomes and Regulation of Glycaemia in Diabetes (RECORD) trial may provide useful insights.
In other words, there were few events, and they didn't have access to the primary data, leading to very wide error bars (i.e., a high level of uncertainty) around the results. This may explain why the authors left out the six studies that showed no cardiovascular events, although the lack of those studies may still have skewed the results. (At the very least, they should have attempted analysis with and without those studies.) These are some reasons that I'm not sure I can dismiss this meta-analysis as vociferously or with as much contempt as the two bloggers I cited did (although perhaps I will read it in detail over the weekend and decide if it's worthy of a little special helping Respectful Insolenceâ¢). Both of them, however, raise a number of excellent points that suggest that there may be less to this study than meets the eye than has been reported in the press and that, most disgustingly of all in my opinion, has inspired many an opportunistic malpractice attorney to put up advertisements shilling for business, just as they did for Vioxx.
The point of my post was not to blast this study, it was to point out the bind such a study puts patients and physicians in. The problem is that the press presents these things as if they are earth shattering facts that cannot be disputed because they are science. We all know that the definition of a scientific fact is one that can be challenged with contrary evidence. It is fine for Dr. Nissen to raise the question (although I wonder at his motives), but unfortunately I am the one having to field 100 phone calls on what to do about people's medicine. This is really frustrating when it seems to be more of a war against the FDA as the source. Meta-analyses have their place, but probably should not get front-billing in the NEJM (IMHO).
Rob
Rob, I thought your blog regarding this was one of the fairer and more balanced ones. In my opinion, I think that GlaxoSmithKline has been lax about long term follow up with this drug. It is something that is supposed to be taken for many years, by a great many people, and the only "end point" that has been demonstrated is a reduction in blood glucose level, and other (potential) markers for disease, not endpoints for disease (ie mortality or morbidity).
The mechanism by which rosiglitazone does this is not fully understood. It is well known to cause edema and worsen congestive heart failure.
People with heart disease were mostly excluded from the trials. Are prescribing doctors excluding those patients now?
I agree that what ever the "problem" is, it is not likely to be corrected by hoards of lawyers trolling for class action plaintifs.
And look at the consequence of a questionable meta analysis:
http://www.nytimes.com/2007/05/26/business/26drug.html?_r=1&ref=health&…
Now the trial that is actually looking into cardiovascular problems with Avandia might not be completed because of patients dropping in mass because of the press getting a hold of the Nissen article.
It's not the risk, but the change in risk that's important. This is what the paper says:
I'm reading this as saying that there was no effect in either control or treatment groups. Now, if n and m are the number of patients in the two groups, the odds ratio is (0/n)/(0/m) = (n/m)*(0/0).
The bottom line: it's difficult to estimate in any change in risk of death, when you don't see any deaths.
Bob
Excuse a question from the simple-minded regarding: (0/n)/(0/m) = (n/m)*(0/0).
This is because the "meta-analysis" doesn't go back to individual responses in all the included studies, but uses the results of the study where "0" was the number of deaths so for those it all zeros out?
The "n" and "m" values are used to weight the results of the studies that are included in the "meta" analysis?
AnnR - Yes, they say in the paper that they don't have access to the data, so they just count the numbers of deaths.
The n and m are the numbers in the trial. The basic statistic they're using is the odds ratio. The odds is the
Odds=(number of successes)/(number of failures)
And the odds ratio is the ratio of odds (!), i.e.
(Treatment Odds)/(Control Odds)
An odds ratio of 1 means no effect (i.e. the odds are the same for both control and treatment), a higher OR means that the treatment has a higher rate of events (e.g. deaths).
Bob
Fighting diabetes through its symptoms (hyperglycemia) is as old as the discovery of insulin. We monitor and than try to reduce blood glucose levels as if this is the silver bullet for the treatment of diabetes. The whole idea that hyperglycemia per se is the culprit of diabetes should be reexamined. This is not different with other old dogmas that scientists tend to stick with even when there is enough data out there to refute them.Thus, lactic acidosis as a major detrimental factor in brain damage due to cerebral ischemia is still being cited in recent textbooks and clinical papers, although it has been shown for years that lactic acid may very well be the one substance that allow the ischemic brain to recover from an ischemic attack upon reperfusion. Where diabetes is concerned, the major damage is neuropathies and there is no evidence that high glucose levels per se have anything to do with neuropathies, not even glycation of proteins that, at one point, was hypothesized to be the mechanism by which glucose exerts its "harmful effects. Diabetics, both type I and II, all suffer from high levels of stress hormones (cortisol, etc) that contribute directly and aggressively to neural degeneration. Insulin and cortisol are antagonists and it is a known fact that stress hormones can cause insulin resistance and hyperglycemia. Hyperglycemia is on its own a stress factor that is known to increase cortisol levels in the blood. One can easily see how these two factors alone can get into a type of vicious cycle, which is apparent in many of the obese people today. Unfortunately, the drug/pharma industry is making billions from glucometers and their strips and the idea that monitoring and controlling blood glucose is the way to fight diabetes, such that any drive to monitor and control any other diabetes-related factor is ignored. My point is, even in the cases where glucose levels in diabetics is meticulously controlled, they all suffer from neuropathies, sooner or later. This by itself should be enough to question the current therapy.
In terms of dealing with no occurrences when computing risk ratios, see Agresti's 'Categorical Data Analysis', the death penalty by race of victim/killer example.
Also, check Andrew Gelman's 'Bayesian Data Analysis' (1996 version; I haven't read the new one), particularly the example of tumors in rats (pooling 70+ experiments).
In addition, an extreme method (which should be done, to compare the results of other methods against) would be to pool the studies, treating them as one set, with sum(deaths) vs sum(patients or patient-years).
I'm a statistician who's never worked in clinical trials or meta-analyses, and I could come up with some methods which should be done. Particularly when the major purpose of meta-analysis is to pool 'small' sample size studies, for better power. They've got to run into studies where there were no occurrences.
It's clear that discarding 0-death studies biased the results; I'd put the burden of proof on the authors to justify that decision. And it's a freakin' high burden, because there *are* methods to deal with this.
"Where diabetes is concerned, the major damage is neuropathies..."
Really? My doctor has always maintained that the most serious compication of type 2 is carduiovascular. Neuropathy is monitored as well as kidney function and retinopathy but CV is the biggest risk of death.
It's a toss-up, as far as I'm concerned. The neuropathies can be truly horrible as far as quality of life is concerned. It's also a combination of the neuropathy and vascular disease, for instance, that leads to lower limb loss. Diabetics can't feel their feet and don't notice small cuts or injuries; then, because of vascular disease, these cuts become ulcers and become infected, leading to progressive infection and ultimately amputation.
However, because the microvascular disease characteristic of diabetes leads to diabetic complications retinopathy and blindness, heart disease, renal failure, and limb loss, I'd say that I have to agree that the vascular complications of diabetes are probably worse.
Barry - the 0-death studies will have a small impact, because (being Bayesian) the posterior distribution of the rates is small for both classes, so all it says is that the odds ratio can't be huge, e.g. with a Poisson rate of 0.1 in the control group, the OR is unlikely to be 100. Or if the control group rate is 0.01, the OR is unlikely to be 1000.
The only way I can see of getting any information from those studies is to fit a hierarchical model (like Andrew's rats example), but I doubt they'll contribute much to the estimate of the OR. In addition, you'll need to use quite a lot of information to estimate the intensities in each group (e.g. number of patients, length of follow-up, and any other important covariates for all of the studies): having zero events only tells you that they are small, but not how small. I think in the end you spend all your time building a model to squeeze almost no information out of the data.
I too am a statistician who's never worked in clinical trials, but I have done a meta-analyses. One :-)
Bob
Hey, Rivlin: I'm treating patients for diabetes. I do chase blood sugars and A1Cs. I also chase retinopathy, nephropathy (using a variety of markers), dylipidemias, neuropathy, and vasculopathy. Get with the program. It was a long time ago that the only marker for diabetic control was blood sugar, even in little backwards community suburban solo docs like me.
Bob, I wouldn't be surprised. The trick is that that's something which needs to be done, to establish that it makes no practical difference.
I disagree regarding neurological symptoms being the worst part. The worst complications from a mortality standpoint are vascular (be they peripheral, cardiac, or cerebral). The DCCT trial for Type 1 diabetics and the UKPDS trial for type 2 diabetics have shown that there is a correlation between glycemic control (specifically A1c) and complications. We always need to be aware that A1c may just be a proxy endpoint, but I know of no trial that shows a substantial A1c reduction that is not accompanied by a reduction in the major endpoints for diabetics.
Retinopathy, nephropathy, CVpathy, are all outcomes of neural degeneration (neuropathy) and that is the main point I wanted to emphasize. When it comes to diabetes, we still measure blood glucose levels as the only idicator of diabetes and the single symptom of the disease to be treated. AC1 is eventually the outcome of chronic hyperglycemia and have not shown to be the cause of any of the detrimental outcomes of diabetes. I would venture as far as to suggest that high blood glucose level is probably not the best indicator of diabetes.
I agree with Rivlin. There may be good correlation of blood glucose with adverse effects when blood glucose is controlled by insulin, but that doesn't prove that it is high glucose that causes them. His point about neuropathies wasn't that they are the worst symptom, rather that they are a symptom that is not caused by elevated glucose.
Stress exacerbates every symptom of the metabolic syndrome. Every symptom of diabetes, every symptom of cardiomyopathy, every symptom of renal failure. By what mechanism does "stress" do this? Good question. I would bet, not by raising blood glucose levels.
Nitric oxide is involved in every pathway that "stress" makes worse. Hyperglycemia lowers NO by causing oxidative stress. Low NO is observed in all of these disorders (measured by vascular reactivity). Low NO directly causes vascular abnormalities by reducing vascular dilatation.
If you look at retinal vasculature in diabetes, in the metabolic syndrome, in hypertension, congestive heart failure, in subclinical stroke. In the JAMA article, retinal microvascular abnormalities were seen even in patients without elevated blood glucose. You see a retinal microvascular abnormalities; retinal arteriolar narrowing and arteriovenous nicking. Arteriovenous nicking just screams (to me), low nitric oxide.
Presumably, for arteriovenous nicking to occur, the vessels must either be a source of a diffusible compound that makes them smaller (and so the concentration is higher where they cross), or a sink of a diffusible component that makes them larger. Hemoglobin is the sink for NO, and NO does make vessels larger via vasodilatation and via angiogenesis. Low NO does make vessels smaller via apoptosis and vasospasm.
Tortuous vessels are observed in retinopathy as well in some neuropathies. The "tortuous" vessels presumably are mediated through a flow-dependant mechanism, that is, the flow influences the shape, which influences the flow. In vessels, it is the red blood cells which are the sink for NO, RBCs are denser than plasma and so segregate due to centrifugal effects when the flow is curvilinear. The region with the lower RBC density would have higher NO, and so would experience more vasodilatation and more angiogenesis, it would expand.
Tortuous vessels are observed in leukoaraiosis, and these tortuous vessels are surrounded by voids in the white matter that are empty of cells, cell debris and scar tissue, but which contain markers for apoptosis. Presumably these vessels are either sources of pro-apoptotic factors, or sinks of anti-apoptotic factors. Low NO does cause apoptosis.
Since some retinopathy is not associated with elevated glucose, presumably it is not caused by elevated glucose. Is that retinopathy "different" than the retinopathy associated with elevated glucose (even though they "look" the same?
Cortisol (corticosterone in rodents) interferes directly with energy metabolism through interaction with its receptors and inhibition of glucose uptake via the glucose transporter. The tissue most sensitive to energy deprivation is neural tissue (some neurons are more sensitive than others, i.e., CA1 neurons in the hippocampus are among the most sensitive and when we consider that the hippocampus is very rich in glucocorticosteroid receptors, we can understand why memory and cognition are two major dysfunctions of prolonged diabetes). High levels of stress hormones directly exacerbate insulin resistance. Hyperglycemia is a stressor on its own and one of the reasons for increased cortisol levels in diabetes. The main beneficial effect of insulin is not necessarily the reduction of blood glucose levels, but rather antagonism of cortisol action via the glucocorticoid receptor. However, insulin is not a good antagonist of this receptor. There are other antagonists such as RU486 or progesterone that may have even a greater benefit than insulin in treating diabetic neuropathies (including nephropathy and retinopathy).
An issue that doesn't seem to have been addressed as yet is the impossibility of designing a clinical trial that will detect really rare occurrences (it would have to be too large and wouldn't be feasible). Hence rare occurences, which will occur "frequently" if millions or tens of millions are exposed as is the case for a popular drug for a high prevalence chronicc condition like diabetes, arthritis or CVD, can only be detected by good post market surveillance, which we do not have.
You'll get no argument from me on this one. It's massively impractical, if not outright impossible, to design clincal trials large enough to identify rare adverse events, which is what the public doesn't seem to understand. Otherwise stories like Vioxx and Avandia wouldn't get such traction. If we had a better post-market surveillance (sometimes called a "phase IV" clinical trial), we'd probably find most of these sorts of problems much earlier. That also has to be coupled with a vigorous and aggressive FDA that has the power to pull drugs rapidly that show major problems after approval and wide use.
Barry - You're right that we need to know how much information there would be. So, I ran some quick simulations. They obviously not definitive, but...
I assumed a simple situation with a probability of seeing an event, and zero events in n1 and n2 trials for control and treatment (and then set n1=n2, for simplicity). I think the Bayesian approach works best here (at least it's straightforward), so I assumed a uniform prior on the probabilities, and then simulated the posteriors (Be(1, 1+n) distributions), and calculated the odd ratio.
Even for large sample sizes (n1=n2>1000), the 75th percentile for the OR is around 3 (the 97.5% quantile is about 39!).
This is the obfuscated R code I used:
OddsR=function(n1, n2, nreps=1000000) (1/rbeta(nreps, 1, 1+n2)-1)/(1/rbeta(nreps, 1, 1+n1)-1)
Ns=seq(10,500, by=10)
Quant=sapply(Ns, function(n) { OR=OddsR(n,n); quantile(ifelse(OR>1, OR, 1/OR), 0.5)})
plot(Ns, Quant)
Bob
Thanks, Bob. I'm not quite clear on what your intention was; if it was to simulate the information from a set of 0-occurrence trials, I agree that the CI's for most things would be very large. That's why some pooling should have been done. IIRC, one of the risk ratio CI's mentioned on another blog had a lower bound of 1.03. Those six studies could have easily have made the difference between significant and non-.
"Retinopathy, nephropathy, CVpathy, are all outcomes of neural degeneration"
Retinopathy and CVpathy are vascular in nature, not neurological. It is the vessels to the brain and retinae that are affected. Nephropathy is due to proliferation of cells within the glomerular membrane. How is this due to neural degeneration? Is there something I am missing?
Regarding the effect of sugar, we do know two things: people with higher sugars have worse outcomes, and lowering those sugars (as stated in the two large prospective trials I quoted) lowers those risks. While you may quibble about the actual cause of these outcomes (I am wide open to the fact that sugar may be a proxy), practically speaking it is only an academic argument that does not really make much difference in this article. Nor does it really change the way I practice diabetes at this point.
That is precisely why this paper by Nissen is problematic. There needs to be (and is in progress right now) a prospective trial looking at cardiovascular outcomes with this drug. Using the studies that do not really address this point is clearly a weaker argument. He may be right, but we won't know until 2008.
Let me point out too that the Women's Health Initiative and the argument swirling around hormone therapy 10 years ago is a good case to consider. The majority of retrospective and meta-analytical trials pointed to the fact the post-menopausal hormone therapy was protective to the heart. It was not until the WHI was done and had to be stopped before it was due to finished because of adverse cardiovascular effects from hormones, did the truth come out. This has to be considered whenever looking at anything but prospective data.