How Alties read science (p values matter)

I couldn't make it a week without talking again about Newstarget, home of Altie-med uber-crank Mike Adams. This time, they caught my eye with a surprising read of this large study of the antioxidant vitamins C and E. Jack Challem, writing for Newstarget, tells us the good, but hidden, news from this study.

When Cook and her colleagues analyzed data from people who consistently took their supplements, they found these specific benefits:

Vitamin E led to a 22 percent reduction in the risk of heart attack.

Vitamin E led to a 27 percent less risk of stroke.

Vitamin E led to a 9 percent lower risk of death from cardiovascular disease.

Vitamin E led to a 23 percent lower combined risk of heart attack, stroke, and cardiovascular-related death.

Vitamin E and vitamin C together lowered the risk of stroke by 31 percent.

Now this is surprising given the latest meta-analysis which actually showed little benefit from supplementation (and possibly some harm). But after all, a big trial like this will often be better than retrospective analysis of several small studies. However, I was shocked, shocked, to find the authors at Newstarget may have exaggerated the findings.

First, here's how the authors summarized their findings.

Methods The Women's Antioxidant Cardiovascular Study tested the effects of ascorbic acid (500 mg/d), vitamin E (600 IU every other day), and beta carotene (50 mg every other day) on the combined outcome of myocardial infarction, stroke, coronary revascularization, or CVD death among 8171 female health professionals at increased risk in a 2 x 2 x 2 factorial design. Participants were 40 years or older with a history of CVD or 3 or more CVD risk factors and were followed up for a mean duration of 9.4 years, from 1995-1996 to 2005.

Results A total of 1450 women experienced 1 or more CVD outcomes. There was no overall effect of ascorbic acid (relative risk [RR], 1.02; 95% CI, 0.92-1.13 [P = .71]), vitamin E (RR, 0.94; 95% CI, 0.85-1.04 [P = .23]), or beta carotene (RR, 1.02; 95% CI, 0.92-1.13 [P = .71]) on the primary combined end point or on the individual secondary outcomes of myocardial infarction, stroke, coronary revascularization, or CVD death. A marginally significant reduction in the primary outcome with active vitamin E was observed among the prespecified subgroup of women with prior CVD (RR, 0.89; 95% CI, 0.79-1.00 [P = .04]; P value for interaction, .07). There were no significant interactions between agents for the primary end point, but those randomized to both active ascorbic acid and vitamin E experienced fewer strokes (P value for interaction, .03).

Conclusion There were no overall effects of ascorbic acid, vitamin E, or beta carotene on cardiovascular events among women at high risk for CVD.

No overall effects? We already have a conflict between the study authors and Newstarget. Oh I wonder how this will ever be resolved.

For one, the Newstarget folks point out that when one eliminates those who were less compliant with the study regimen is where these effects were observed. In the paper one sees, true enough, that some effects did emerge specifically in those who said they took the vitamins more faithfully.

Censoring participants on noncompliance led to a significant 13% reduction in the primary end point (RR, 0.87; 95% CI, 0.76-0.99 [P = .04]). Reductions in secondary study end points were also stronger, with a 22% reduction in MI (RR, 0.78; 95% CI, 0.58-1.06 [P = .11]), a 27% reduction in stroke (RR, 0.73; 95% CI, 0.54-0.98 [P = .04]), and a 9% reduction in CVD mortality (RR, 0.91; 95% CI, 0.66-1.25 [P = .55]). There was a 23% reduction in the combination of MI, stroke, or CVD death (RR, 0.77; 95% CI, 0.64-0.92 [P = .005]). Among those with prior CVD, the active vitamin E group experienced fewer major CVD events (RR, 0.89; 95% CI, 0.79-1.00 [P = .04]; P value for interaction, .07)

In other words they saw a nonsignificant 22% reduction in MI [p=0.11], a significant 27% reduction in stroke[p=0.04], and a completely nonsignificant 9% reduction in overall CVD death [p=0.55!].

Now, the 23% combined reduction in MI, stroke or CVD death does appear to pass muster with a p=0.005, making it unlikely this was due to chance. The combined C and E effect also was significant p =0.04. So only about half of what Newstarget said was without any basis in reality.

But in reading further I noticed another example of this selective reasoning (fitting with the bias that natural is always harmless). Challem says:

Beta-carotene did not have any clear cardiovascular benefits, and the researchers also pointed out there were no signs of harm from any of the supplements. Other studies have pointed out that beta-carotene may have benefits in preventing cancer among people who do not eat many fruits and vegetables.

What did the study actually say about Beta-carotene?

There was a nonsignificant 14% increase in CVD mortality in the active group, with a significant decline over time in the effect on CVD deaths (P = .04) but no difference in total mortality. When participants were censored on noncompliance, the effect on the primary end point remained nonsignificant (RR for major vascular disease, 1.09; 95% CI, 0.96-1.24 [P = .18]), but an increase in CVD mortality appeared to emerge (RR, 1.48; 95% CI, 1.08-2.02 [P = .02]).

Wow! So beta-carotene increases your risk of cardiovascular mortality by 48%. Stop eating carrots! They're deadly! Or not. After all, this is a small effect, the significance is there, but based on the power of the study, it's unlikely to be an effect that persists with further study - like many of the other findings.

It's interesting how effects with nonsignificant p values are so noteworthy when they support the idea that vitamins are healthy, but effects among a similar group of patients that appear to significantly increase mortality are interpreted as "no signs of harm". The authors were very conservative with their statements of what their study found - and believed by the altie-people when it conformed to their worldview since they don't want a healthy natural vitamin like beta-carotene to increase your risk of cardiovascular death by almost 50%. Overall when you look at the summary of the findings you see why the researchers were conservative with their statements. Here is a figure summarizing most of their findings:

Yeah, maybe a small probability of some effects here, but it's a stretch. These vitamins (note even combined with multivitamin use) have little significant effect on cardiovascular outcomes.

Now, Mike Adams of course takes this story and goes off into crazy land with the usual conspiratorial nonsense of how this is supression of science etc.

Following yesterday's announcement of a new study showing the phenomenal benefits of antioxidants for preventing heart disease in women, the mainstream media rallied behind a blatantly false distortion of the study designed to convince the public that vitamins E and C are somehow useless. The popular press, which maintains an incestuous relationship with the pharmaceutical industry, once again demonstrates it is little more than a mouthpiece for the pro-pharma propaganda machine. There is no scrutiny of the study's findings, no critical thinking and absolutely no independent journalism being conducted by the mainstream media on this particular topic.
...
This particular distortion comes from the drug-touting Brigham and Women's Hospital (BWH), an affiliate of Harvard Medical School. BWH goes on to state, "researchers... have found that there is no evidence of benefit or risk from vitamins C, E or beta-carotene on cardiovascular events for women at a high risk for cardiovascular disease (CVD)."
...
But when profits are at stake, and there's an industry-wide propaganda campaign to push onto the public, common sense gets thrown out the window. Scientific scrutiny turns to wishy washy statistical foolery disguised as authoritative proclamations about nutrition. This, in turn, quickly devolves into nutritional nonsense. The message from industry is very clear: Don't take vitamins! And if they have to lie with statistics by making a positive study look negative, they're more than willing to step up to the plate and state the indefensible, almost as if they lived in some alternate universe where the laws of logic have all been reversed.

Ha! Adams is always good for a laugh. Talk about the pot calling the kettle black, after lying about the results of the study, and ignoring statistically significant effects that don't conform to his worldview, he then accuses the scientists, who merely made an appropriately conservative statement about their data, of lying!