Effect Measure

I keep getting asked about the Atlantic Magazine article, Does the Vaccine Matter? by Shannon Brownlee and Jeanne Lenzer, two reporters whose particular bias is that we as a nation are “over treated.” As a generalization that’s probably true, and finding examples isn’t hard. Unfortunately by taking as their main example flu vaccine during a pandemic, they have not only picked the wrong example but created more confusion at a time when there’s already too much.

Here’s the gist:

Vaccination is central to the government?s plan for preventing deaths from swine flu. The CDC has recommended that some 159 million adults and children receive either a swine flu shot or a dose of MedImmune?s nasal vaccine this year. Shots are offered in doctors? offices, hospitals, airports, pharmacies, schools, polling places, shopping malls, and big-box stores like Wal-Mart. In August, New York state required all health-care workers to get both seasonal and swine flu shots.

[snip]

But what if everything we think we know about fighting influenza is wrong? What if flu vaccines do not protect people from dying?particularly the elderly, who account for 90 percent of deaths from seasonal flu?

Brownlee and Lenzer rely upon (and romanticize as a martyr and truth-teller) Dr. Thomas Jefferson, someone who is fast establishing himself as an “Evidence Based Medicine” (EBM) crank who who courts notoriety by being a contrarian. The kind of EBM practiced by the likes of Jefferson and some other randomized trials zealots is far from the judicious weighing of the evidence envisaged by its early proponents. For example, David Sackett, defined EBM as:

“the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.” (BMJ, 13 Jan 96, Sackett, David L.).

There is nothing judicious about Jefferson, whose problem was described by one of my colleagues as “methodolatry,” the profane worship of the randomized clinical trial as the only valid method of investigation. In this case his evidence base isn’t even relevant, because we aren’t dealing with seasonal flu but pandemic flu. For the record, what he is saying about the uncertainties about flu vaccine efficacy in the over 65 age group isn’t new. In fact we’ve discussed it here, several times (here, here), going back a couple of years. But it is also clear that the vaccine offers protection in the age groups that matter for this pandemic, the people under age 50.

Nor is Jefferson, as claimed in the article, someone who “knows the flu-vaccine literature better than anyone else on the planet.” That’s an absurd claim. The literature is vast and he knows only a tiny part of it. But insofar as there is an acknowledged expert on vaccine efficacy, it would be biostatistician Elizabeth Halloran, who reviewed the clinical trial and experimental challenge literature recently in the American Journal of Epidemiology. We wrote a longish post on the subject here. There is general agreement, even among so-called skeptics, that the vaccine works in the under 60 age group, precisely the group at issue with the swine flu vaccine.

Jefferson’s points about vaccines in the elderly are not new, as I’ve said. But even though he is extreme in his views, looking at Jefferson’s own recent papers in the literature show conclusions much weaker than the picture painted in this article:

In long-term care facilities, where vaccination is most effective against complications, the aims of the vaccination campaign are fulfilled, at least in part. However, according to reliable evidence the usefulness of vaccines in the community is modest. The apparent high effectiveness of the vaccines in preventing death from all causes may reflect a baseline imbalance in health status and other systematic differences in the two groups of participants. (Rivetti D, Jefferson T, Thomas R, Rudin M, Rivetti A, Di Pietrantonj C, Demicheli V, Vaccines for preventing influenza in the elderly, Cochrane Database Syst Rev. 2006 Jul 19;3:CD004876)

Influenza vaccines are effective in reducing cases of influenza, especially when the content predicts accurately circulating types and circulation is high. However, they are less effective in reducing cases of influenza-like illness and have a modest impact on working days lost. There is insufficient evidence to assess their impact on complications. Whole-virion monovalent vaccines may perform best in a pandemic. (Jefferson TO, Rivetti D, Di Pietrantonj C, Rivetti A, Demicheli V., Vaccines for preventing influenza in healthy adults, Cochrane Database Syst Rev. 2004;(3):CD001269.)

We concluded that there is no credible evidence that vaccination of healthy people under the age of 60, who are HCWs [health care workers] caring for the elderly, affects influenza complications in those cared for. However, as vaccinating the elderly in institutions reduces the complications of influenza and vaccinating healthy persons under 60 reduces cases of influenza, those with the responsibility of caring for the elderly in institutions may want to increase vaccine coverage and assess its effects in well-designed studies. (Thomas RE, Jefferson T, Demicheli V, Rivetti D, Influenza vaccination for healthcare workers who work with the elderly, Cochrane Database Syst Rev. 2006 Jul 19;3:CD005187)

When there is evidence for efficacy in a trial but insufficient numbers for meeting Jefferson’s arbitrary standard of precision of the estimate, Jefferson typically interprets the results as showing insufficient evidence. For any particular study that may be fair enough, but EBM means weighing the evidence, not just tabulating. When he talks to reporters he abandons rigor and says stupid things. Celebrity does that to some people, especially scientists unused to being feted as truth-telling mavericks. And like many randomized trial zealots, Jefferson frequently implies lack of statistical significance is affirmation of the null hypothesis, a serious interpretive error that slides by most journalists. And I know from personal experience journalists are always trying to get you to say something in the least nuanced way. Too many scientists succumb.

The maddening thing about this article is that there are valid points to be made here and we have tried to make them until we were blue in the face (random examples here, here, here, here, here, here). We have always felt the way to prepare for and battle a pandemic is to rely on a strengthened and robust public health and social services infrastructure, not to plan for the best (that there will be an effective and timely vaccine available to everyone) while hoping the worst won’t happen. Vaccines and antivirals are a poor second best as a strategy. But that’s what we have now, and while not the optimum, they do work. On the other hand, the nostrums also touted in this article as a substitute for vaccines and antivirals, like washing your hands, have almost no scientific support in the literature for influenza. They are still good things to do, although if Dr. Jefferson decides to review the literature, I wouldn’t count on him finding any support for them. So what?

The bottom line is this. There is excellent and credible evidence in the scientific literature that vaccination against influenza reduces infections in people under 60, evidence that even Dr. Jefferson accepts. For those over 60, there are legitimate questions that were raised by others about the extent of the benefit of seasonal flu vaccine, but they were raised before Jefferson got into the act. The argument put forward in this piece is a straw man argument as far as pandemic influenza is concerned (and in which context it was placed).

I understand the rhetorical value of having a martyr-hero when pitching a story, but this was a particularly irresponsible time to pull this stunt.

Comments

  1. #1 Lina Inverse
    October 24, 2009

    Question: What makes you think that in practice Evidence Based Medicine (EBM) will be done in the way you’d prefer vs. this concrete example of Dr. Jefferson, who clearly has something of an agenda.

    Even if in his case the agenda might be “?hat’s sufficient evidence?”, which if thoroughly applied would declare many treatments to be not effective and ultimately—if you follow the logic of EBM—not approved or allowed?

    My specific point is that we have here a concrete example of the misuse of EBM that is having real world effects, perhaps to the point that one or more people will die from novel H1N1 because this article convinced them the benefit of getting inoculated is so low it doesn’t outweigh the perceived risks.

    My general point is that before we change our system to use something like EBM we should be pretty sure that in practice it won’t make things worse.

  2. #2 Orac
    October 24, 2009

    Tom Jefferson clearly has a bug up his butt about flu vaccines. Indeed, he has such an agenda that he was invited to the National Vaccine Information Center’s vaccine conference in early October. The NVIC is the oldest and biggest antivaccine organization there is. Either he didn’t know that, in which case he’s clueless, or he didn’t care. In any case, it was clear that he was invited there because of his stance on flu vaccination, and he was even going to be awarded the NVIC “Courage in Science Award”:

    http://www.sciencebasedmedicine.org/?p=1723

    To his credit, Jefferson backed out when he found out that he would be sharing the stage with Andrew Wakefield, who was to be given the NVIC “Humanitarian Award.” He was appropriately horrified. How do I know this? Because I know (at least online) the person who e-mailed Jefferson and asked him why he was lending his name to anti-vaccine crankery, and that person, with permission, forwarded Jefferson’s e-mail exchange with Barbara Loe Fisher to me.

    In any case, I don’t take Tom Jefferson very seriously anymore. “Methodolatry” is an awesome term to describe his approach. Actually, it’s a great term to describe some of the Cochrane scientists responsible for analyzing the efficacy of mammography screening, as well; their conclusions and methods rather remind me of Jefferson’s.

  3. #3 M. Randolph Kruger
    October 24, 2009

    There is ONE good bit of news on the flu front and it comes from the Australian today and it tells us of the removal of the Wikked Witch of the West in Indonesia.

    http://www.youtube.com/watch?v=A5iseJJ5ogA&feature=PlayList&p=74849EFE91ACBD37&playnext=1&playnext_from=PL&index=19

    Yes folks, Auntie Siti Supari is no longer in charge of health in Indonesia. Thanks to Dr. Henry Huang of WUSTL for passing the info on this morning.

    To quote directly from the Australian:

    “And in a clear indication of the President putting policy over politics in at least one field, the much-criticised health minister, Siti Fadilah Supari, has been dumped for a health professional she herself had targeted.

    ” General practitioner and US-educated public health specialist Endang Rahayu Sedyaningsih will be in charge of health, a sharp elevation after she was demoted by Dr Supari two years ago for allegedly giving bird flu virus samples to US researchers.

    Dr Sedyaningsih, having weathered exile in the Health Department, said after the appointment that she now expected her job to be one of “consolidating reform” in the sector. ”

    Dr. Sed is widely regarded in and around the world because of her “health first, politics second” approach to infectious diseases.

    You may wish her well at any one of the emails below

    puskom.publik@yahoo.co.id info@puskom.depkes.go.id kontak@puskom.depkes.go.id

    Going to be some pretty small shoes to fill. The ongoing rant about NAMRU-2 is still there, but indeed the Wikked Witch is outta there. There will still be a push about vaccines and the like and how we should build them a vaccine factory at a cost of 32 billion dollars so they can compete with the West… But now we dont have to listen to the knucklehead any more.

    I can only hope that Supari hasnt been hiding the big potato behind the wall of secrecy on H5N1.

  4. #4 Paula
    October 24, 2009

    There’s lots wrong with basing any part of health care or healthcare policy on bowdlerized EBM, as Lina Inverse here notes. On the swine flu vaccine, though, this seems nearly optimistic compared with the figures reported yesterday in ScienceInsider, http://blogs.sciencemag.org/scienceinsider/2009/10/pandemic-vaccin-2.html, “Pandemic Vaccine ‘Will Arrive Too Late for Many,’ CDC Concedes,” by Jon Cohen and Martin Enserink; the article concludes: “Although Frieden explicitly refused to make predictions about future supply of the pandemic vaccine, [Nicole Lurie, assistant secretary for preparedness and response at HH] said that HHS will tell state health officers later today that the government expects to have another 10 million doses available next week. In subsequent weeks, projections are now that manufacturers will supply 8 million to 10 million doses per week. But Lurie concluded . . . ‘It’s hard to predict week to week.’” The arithmetic on that is pretty scary. And with today’s “National Emergency” announcement there is also a combined figure of 3,400 deaths either from flu or indirectly probably from flu since April, “with 20,000 hospitalizations,” a death-to-hospitalization ratio strikingly high. One would think these figures and the emergency proclamation might lead to a quick push for adjuvant, or is politics going to prevent that?

  5. #5 revere
    October 24, 2009

    Randy: I think she’s been gone for a bit and creating problems for her successor whom she accused of “selling the virus to foreigners” although she then took back the claim. Unfortunately she’s the Indonesian equivalent of the tea baggers, capitalizing on distrust of the government in general and foreign govt’s in particular, especially the US. You may answer that the tea baggers have their reasons and my response would be, so does she.

    Paula: They have the adjuvant in reserve and I suspect they won’t hesitate to use it i a dire emergency, perhaps with adequate labeling or warning so people can choose if they want to use vaccines with adjuvant or not. If that happens, that will be the point where those who fear the adjuvant more than the flu will be having an effect on others and lives will be lost.

  6. #6 Lina Inverse
    October 24, 2009

    Paula: it’s a bit late in the game to make that sort of switch, to do it with any degree of safety would take a long time, probably longer than just finishing as currently planed (I’ve seen this sort of thing in my programming career).
    And since we’ve never tried an adjuvanted flu vaccine in this country before (I gather), the anti-vaxers would scream to high heavens and I’d bet the end result would be even fewer people vaccinated.

    This isn’t *that* great a crisis, nor based on the experience of the Southern Hemisphere will it become one. If we were serious about this sort of thing we would have funded companies like Protein Sciences long ago to get their vaccines tested for safety and efficacy and would have therefore produced enough vaccine for everyone in the country who wants it in time.

    Protein Sciences is not the only company using modern technology to make flu antigen of one sort or another, but they’ve shown they can make a safe vaccine in 2 or so weeks if needed (they did an H5N1 one an emergency basis in 1999 or so as I recall). If we’re ever going to be ready for a truly deadly flu we need to get away from our dependence on the decades? old egg method, which at its best yield produces about 3 unadjuvanted doses of antigen per egg (it only reproduces in the cells of the membrane).

  7. #7 M. Randolph Kruger
    October 24, 2009

    65 trillion in the hole Revere overall… its game over anyway and ten trillion in immediate debt next year…without UHC. The tea baggers understand its a give and take and this time around the take is going to finish us off. UHC is just one aspect of it.

    The first indication of this will be that the troops take no definitive action and they just pack up and leave. Why? No money to pay for it and print, print, print wont buy your fuel. Get ready for 300 a barrel oil by summer.

    Valueless money is whats going to do us in. Cant keep the military in the field with the numbers that are out there and its a fake, tax created economy.

    http://en.wikipedia.org/wiki/File:USDebt.png

    The Chinese sold treasuries this week rather than buy and the result was four days of ugly on the markets mixed with shudders to the upside. No long term buying. Tea baggers see it for what it is and that is that the second shoe is about to drop and they’ll be eating tea rather than sending it to the W. House.

    Hows that 130 million cut in the state health care budget weighing in….? Sounds like rationing to me. Better get ready for paying more, but getting less.

    Everyone is covered, everyone is broke, and everyone is using the system. But no one is paying in because the unemployment rate is where it is.

    Your 100 million jumped to 130 since March and thats with forced compliance. The State response? Cut services, but dont raise taxes. Well now that sounds a lot like the UK and….well the rationing that we said would happen has kicked in. It only took you guys two years to get to TennCare status. We went 500 mill in the hole and it took us about 7. You are on a roll.

    Change we can believe in. Can I have my Lipton please?
    —-
    BTW I agree with your comments on Jefferson. I make it a daily thing to read and there are probably about 1 million to go on the papers on Swiney and Avian. I draw a lot of the same conclusions you do on this and that is that the newsies are full of crap. I also think the Administration is full of crap too. I have YET to see any one like Janet Napolitano or Sebilius rolling up their sleeves on TV. Now that doesnt engender any support for it one way or another, but lets get their kids in there and show them taking it. Sebilius and Napolitano both need to wheel their country butts out and head to a flu shot-a-thon and roll those sleeves up. I dont want some bozo telling me that I MUST take a flu shot when they havent done it themselves.

    To me its about your (medical types) criteria and protocols… You say it provides protection? Really… Can you really say that its a preventative? Maybe only for death though and only if you have underlying conditions even though there have been “healthy” individuals that got it… But even thats questionable. I cant find any group/study/individual that can state for the record and in this country that means on the stand, that the vax is truly a fix for flu. If thats the case then why in Hell does the seasonal flu shot not work? I always hear its mismatched when it doesnt work and they say it was a good match when the numbers were down. But maybe it could be just a natural process as well? You had flu last season so you didnt get it this season because you were naturally vax’d.

    Then there is that Canadian stuff… Disturbing. You take the seasonal in the last two years and that makes you susceptible to Swiney. So now you take the Swiney vax and that makes you (possibly) susceptible to what?

    Get my drift? If there is going to be a flu vax then in my eyes it needs to be a universal one. It doesnt exist and it likely never will. I am cringing from what may come down the pike at us after this little flight down flu lane generates.

  8. #8 Mencius Moldbug
    October 24, 2009

    Revere, how wonderful to see you come out, for real judgment, real evidence and real medicine, against the EBM fanatics! Conquest’s Law – everyone is reactionary on the subjects they understand.

    So you know what it looks like when an anti-scientist and crank, at best a second-rater, whose true talent is bureaucratic fraud, not only attains the position and professional reputation of a scientist, but inserts his proboscis into the ear of the press. Whole bogus fields can be constructed around crank anti-science, as you see. And not the Republican kind, either!

    Now, for “Thomas Jefferson,” all you need to do is read “James Hansen” or “Michael Mann.” Talk about methodolatry…

  9. #9 Paula
    October 24, 2009

    Revere, thanks for this. Wonder what would happen if some of the adjuvanted vaccine were in fact made available. I’m hearing far more people discussing whether to try for vaccine (w or w/o adjuvant)by going outside the U.S. (including over-65s who do not expect vaccine to become available to them)than hearing people scared of vaccine.
    Lina Inverse, thanks–good points. Wonder if funding will become available for testing new-tech flu antigens, after this.

  10. #10 Monado
    October 25, 2009

    A couple of things have come out of the research in Canada. One, children six and older are developing strong immunity to the flu with just one “shot” (nose sniff) instead of the two that we expected it to take. The suspected reason is that by six they’ve been exposed to some flu viruses circulating in the community.

    Two, far from being useless, the 1976 innoculations against swine flu have given those who got them about 30% immunity against this strain. Unfortunately I’ve closed the web pages so I can’t link to them here.

    Oh, yeah, 67% of the people who’ve died of it in Canada have been women in their 20s, 30s, and 40s. Most of the rest have been children. 3% have been men.

  11. #11 Otto
    October 25, 2009

    “Why? No money to pay for it and print, print, print wont buy your fuel. Get ready for 300 a barrel oil by summer.”

    It’s a shame that there isn’t a secondary market in vague remonstrations.

  12. #12 Tsu Dho Nimh
    October 25, 2009

    There is excellent and credible evidence in the scientific literature that vaccination against influenza reduces infections in people under 60, evidence that even Dr. Jefferson accepts. For those over 60, there are legitimate questions

    If those under 60 are heavily vaccinated, it is less important that those over 60 have robust immune reactions because we are less likely to get the danged virus!

  13. #13 Kristjan Wager
    October 25, 2009

    Now, for “Thomas Jefferson,” all you need to do is read “James Hansen” or “Michael Mann.”

    Except of course that Hansen and Mann are supported by both the literature and the science.

  14. #14 Lina Inverse
    October 25, 2009

    Paula: You’re welcome. Protein Sciences (again, not the only company in this area, it’s just the one I’m following) *did* finally get allocated $34 million in a program that might give them up to $147 million over five years. As I remember, even some government people said that should have happened 3 or so years ago.

    Their vaccine is being tested in Australia (sense of urgency there) and more recently they got the OK to start testing it here. But it’ll still be some time before it hits the market, the FDA is in theory very careful with vaccines since they are given to ostensibly healthy people, so the tradeoffs with side effects are much harsher.

    And of course we have a proven safe and at least somewhat effective existing vaccine technology so it’s got to be somewhere in the area as good at that. But in terms of production it’s very promising, they think one industry bioreactor can produce 100,000 doses of antigen per week and then it only takes a few weeks (5?) to purify and package. Somewhere else I recently read someone comparing the time frame of egg technology to growing corn.

  15. #15 MoM
    October 25, 2009

    Just wondering if you intentionally left the “t” out of the second word of the title…

    Nice piece.

  16. #16 Phoenix Woman
    October 25, 2009

    Kristjian @ 13: Exactly.

  17. #17 Mencius Moldbug
    October 25, 2009

    Kristjan,

    What you mean is that Jefferson, unlike Mann and Hansen, have enemies (like Revere) inside the castle. True, and a very good thing it is, too. Still, I suspect that youth and energy is all on Jefferson’s side.

    Or do you think peer review is some kind of magical ordination, like becoming Pope? Any institution is only as good as the people in it. Astrologers, too, could establish a peer-reviewed literature. They would still be astrologers, and no doubt they would defend their funding empires quite energetically.

    As for the science, have you looked into the matter for yourself? I don’t think so. I don’t think Revere has either. In fact, I am sure he has not, because he strikes me as an honest person or set thereof. And he certainly has the statistical chops to evaluate the dispute for himself.

    But he does not do this. Nor does he recuse himself on account of ignorance. He is confident that he knows the answer, because he is confident that in any battle that features Republican corporate lobbyists against good progressive professors, the former must be wrong and the latter must be right. While this is perhaps a good rule-of-thumb way of arriving at the truth, it is not a very scientific one.

  18. #18 Shannon Brownlee and Jeanne Lenzer
    October 25, 2009

    Faith-based science not methodolatry is the problem

    Blogger Revere describes our article, published in The Atlantic, “Does the Vaccine Matter?” as an instance of “methodolatry.” Revere and his followers, are so enamored with their new-found term that they fail to examine, using facts rather than insinuation and emotion, whether this article was worthy of that term – a term that is useful and could be used to describe a very real problem.

    What is most remarkable about Revere’s critique is that it is largely unburdened by data or even basic facts. So, we’ll recap some of the key points we made in our article since almost none are mentioned by Revere:
    1. Several researchers pointed out what most doctors didn’t know; that the claim that flu vaccines reduce deaths by 50-90% was not based on a reduction in flu deaths, but on a reduction in all-cause mortality. Most doctors think that the studies show a reduction in flu deaths.
    2. The claim that flu vaccine reduces all-cause mortality by 50 % is rather extraordinary, given that the CDC itself only finds about 10% of all deaths among the elderly to be due to flu and the deaths exacerbated or caused by flu.
    3. Several researchers including some highly respected international experts hypothesized that the apparent reduction in all-cause mortality might be due to a healthy user effect (or conversely, a frailty bias). When they tested their hypothesis, they found a 60% baseline imbalance suggesting that virtually all of the benefit seen could potentially be due to this effect.
    4. Despite a dramatic rise in vaccination rates from 15% to 65% among the elderly over the past decade – flu deaths have not decreased.
    5. During two years of complete vaccine mismatch – the death toll did not rise.
    6. During a year when vaccine production fell 40% – the death toll did not rise
    7. Claims of “benefit” from vaccine are often based on the ability to generate antibodies – yet the very people who generate antibodies, young healthy people, almost never develop complications from flu – raising the conundrum: Does it work in those for whom it’s necessary and is it necessary for those in whom it works?

    These and other points have been raised and carefully studied by a number of experts we quoted, and a few of them believe that RCTs are necessary to tease out the answers raised by these questions.

    It is surprising that almost none of those points were addressed by Revere. Instead, like the proverbial man with a hammer, he and other critics happily swung away at what they just know is a nail – without providing any data, any evidence at all that what they’re hammering away at is, indeed, a nail. Instead, they used emotion, false-associations, and belief – or what we call faith-based science, not real science, to take their swings.

    We will address just a few of these faith-based, arguments that Revere and others substituted for genuine scientific debate:

    Revere: Brownlee and Lenzer rely upon (and romanticize as a martyr and truth-teller) Thomas Jefferson, someone who is fast establishing himself as an “Evidence Based Medicine (EBM) crank who courts notoriety by being a contrarian” in Revere’s words.
    B&L Respond: Revere declines to give a data- or fact-based challenge to any of Jefferson’s findings; instead he calls Jefferson a “martyr,” a “crank” and a “contrarian.” OK, now that Revere smeared him – did he offer a challenge to a single one of Jefferson’s findings? Did Revere provide any evidence that Jefferson was wrong in his assessments of the data or the shoddiness of the studies? The answer is no.
    Finally, Revere says we “rely on and romanticize” Jefferson. This is simply not the case. We presented the work of several other researchers – mainstream and highly respected researchers including Lone Simonsen, Lisa Jackson and Sumit Majumdar –each of whom raises similar questions about the validity of the assumptions made about flu vaccine. It is interesting that Revere never once mentions any of these researchers and their findings, instead he attacks the person he appears to see as the weak link, Jefferson. The degree to which we did raise some points about the way Jefferson is treated was not to “romanticize” him but to point out that emotion rather than clear-headed, sober, discussion of data and facts seem to be the way that purportedly dispassionate scientists are responding to the challenge to widely held assumptions. That is troubling.

    Revere writes: “There is nothing judicious about Jefferson” and “he is extreme in his views, looking at Jefferson’s own recent papers in the literature show conclusions much weaker than the picture painted in this article:”
    B&L respond: It is interesting that when Jefferson writes in a manner that is neither injudicious nor extreme, that Revere, instead of acknowledging his work as nuanced or careful, then calls his writings “much weaker” – much weaker than what? The most extreme thing Jefferson suggested in interviews with us was that scientists don’t have all the answers to the flu vaccine story and that some RCTs are in order. That hardly seems injudicious or extreme.

    Revere writes: “Jefferson frequently implies lack of statistical significance is affirmation of the null hypothesis, a serious interpretive error that slides by most journalists.”
    B&L respond: Here Revere smears both Jefferson and us by a sly insinuation. Neither we nor Jefferson state or imply that the lack of statistical significance implies a null hypothesis. Indeed, despite the studies suggesting that vaccine may not protect against serious complications and death – Jefferson calls for RCTs. Why? Because he is not ready to assume that the lack of statistical significance that he found in his comprehensive reviews, nor the suggestive findings of researchers like Simonsen, Jackson, Majumdar and others, are conclusive. So, he makes his radical call. He’d like to see some placebo-controlled RCTs. As for that sliding by “most journalists” – we won’t contest that – but in our carefully written article, we never conflate a lack of statistical significance with the null hypothesis. Indeed, we suggest that if benefit is found, it might compel some genuinely perplexed health care workers, half of whom now decline flu vaccine, to get vaccinated.

    Revere writes: “I understand the rhetorical value of having a martyr-hero when pitching a story, but this was a particularly irresponsible time to pull this stunt.”
    B&L respond: Once again, Revere reverts to emotional suasion rather than fact to argue his case. We raised point by point concerns that researchers have had about the flu vaccine narrative. We demonstrated that those who raise questions about vaccine efficacy are sometimes subjected to faith-based, emotional attacks rather than sober discussion. The last part of Revere’s sentence is particularly troubling. He writes, “…this was a particularly irresponsible time to pull this stunt.” The implication is that during this time of the 2009 H1N1 flu our article will be responsible for the loss of lives if people decline the flu vaccine. We’ve heard that argument before. When we challenged widely held beliefs about various interventions, including cancer screening campaigns, we were told that we were causing the deaths of untold numbers of people because of our articles. Years later, we are now joined by many top experts and professional organizations, such as the American Cancer Society, who now agree that the early voices were right. We could just as easily charge those whose work we criticize are causing the deaths of untold numbers of patients for publishing their work. But such emotional appeals have no place in science. We suggest that for those concerned about this issue, there is no better book to read than Arrowsmith.

    Another blogger, Dr Gorski, who lauds Revere’s review succumbs to some of the same poorly made arguments as Revere and adds another prejudicial association and non-fact: that Tom Jefferson is associated with the anti-vaccine crowd because an anti-vaccine group invited him to speak. Not one of the experts we interviewed, including Jefferson, falls into that camp. In fact, Jefferson was appalled when he found out an anti-vaccine group had used his work in a way that greatly disturbed him. He wrote a strong rebuke to the group. But that hasn’t stopped critics from smearing him any way they can and without any basis in fact.

    Finally, at one point Revere asserts, “On the other hand, the nostrums also touted in this article as a substitute for vaccines and antivirals, like washing your hands, have almost no scientific support in the literature for influenza.” This is, to some degree, an educated assertion: studies suggesting efficacy for physical interventions (e.g. hand washing) do not confine themselves to influenza prevention but rather address respiratory virus transmission in general. While this research, unlike the vaccine research, includes randomized trials and is almost certain to be relevant to influenza, his point is salient—before accepting the utility of hand washing he would like to see high quality research directly addressing the question.

    Ironically, Revere’s argument about hand washing is precisely the same as Jefferson’s (and our) argument about vaccine: mortality reduction needs further direct testing in high quality research. The irony appears to be lost on Revere, however, who then goes on to say, “They [physical interventions] are still good things to do, although if Dr. Jefferson decides to review the literature, I wouldn’t count on him finding any support for them.” Actually, Jefferson wrote the Cochrane review on hand washing and other physical interventions, and indeed, he found support.

    A layperson wrote the following response to Dr Gorski’s praise for Revere, which we think is relevant here. She wrote:
    Can someone…address the points of the article rather than just saying [RCT] for flu vaccine is unethical? I understand that point very well.
    But, can someone discuss the challenges the authors made to the 50% mortality reduction statistic? How about the 2004 low production year where mortality didn’t fall? And, I’m most interested in their assertion that the ethical moratorium on [RCT] for aggressive breast cancer treatment increased mortality risk. What about situations like that?
    I’m not an anti-vaccine person (I got my flu shot last week), I’m just a curious lay person.

    Does it take a layperson to get scientists to act like scientists?

  19. #19 Scott Wiersdorf
    October 25, 2009

    Count me as one of the unwashed laypeople just trying to understand if a flu shot will help me avoid being really sick this winter. I’m not anti-vaccine (I and all my family are all up to date on the standard US vaccinations), but it seems to me that getting vaccinated for flu is like patching only 3 of 20 holes in my roof. It’ll help for the rain falling on that spot, but what of the rest of the house?

    I understand that the vaccine’s efficacy is often good (86% is good, right?) when battling the same strain(s) the vaccine was designed for, but with dozens (perhaps scores?) of active viruses out there this year (flu and other), odds are the vaccine will probably do nothing for my overall health this winter. Am I reasoning that correctly?

  20. #20 WKluwer
    October 25, 2009

    To this layperson, the arguments in the Atlantic article were very familiar. There have been many, many news articles about the well-known flu vaccine “skeptics” such as Jefferson and Simonsen and their arguments that flu vaccination probably does not reduce all-cause mortality to the extent claimed. What is new here?

    This article leaves me with the same bottom-line question that I have had about the previous articles: so what? Is not a vaccine that reduces disease, sick days, hospital use and so on an individual and societal good? Death is not the only endpoint.

    Some of the references about flu vaccine efficacy here are quite convincing http://www.cdc.gov/mmwr/preview/mmwrhtml/rr5707a1.htm. They suggest that flu vaccine has many benefits other than all-cause mortality.

    The cover of The Atlantic teases you with this:
    SWINE FLU: DOES THE VACCINE REALLY WORK?

    Lenzer and Brownlee never tell us what is meant by “work.”

    I wonder if “keeping you out of the ICU” qualifies as “working”?

  21. #21 revere
    October 25, 2009

    Brownless/Lenzer: It is a sprited defense of a straw man. You interjected the issue of efficacy of flu vaccine among the elderly, where the RCT data are sparse, into the question of pandemic preparedness, where the vaccine is primarily aimed at the young age group, for which RCTs for influenza vaccine do exist (see the link to one of our posts discussing the Halloran review). I didn’t address the points you raised for two reasons. I have addressed them in posts here (links provided in the post) at least twice in the last several years; but more importantly, they are not relevant to the issue at hand, use of vaccines for the swine flu pandemic. I have also posted here, several times, on the method used for estimating excess mortality in flu (here’s the link). Both P&I and All Cause is used, and is a choice. Both Simonsen and Thompson have done so, as I recall, and it is Simonsen’s work that called attention to the lack of mortality change despite increase in vaccination among the elderly over an extended time period. Note that such a comparison is much more fraught than comparing the observational and RCT studies, because it is an ecological design comparison: you are trying to infer the interior of the tables (the joint probabilities) from only a knowledge of the marginal probability estimates. We know that both confounding of all kinds (especially temporal and geographic) and effect modification can drastically affect that comparison (even reversing its direction) and clearly over this time period both are operating.

    I only mention these technical issues to say that I have considered them and written about them here, numerous times. I didn’t mention them in this post because they are irrelevant, pertaining as they do to the questions of efficacy among the elderly which is not at issue in the use of a well matched vaccine for this pandemic. It is, in effect, a sleight of hand.

    Your piece is chock full of rhetorical rather than factual devices, for which I don’t blame you. You are not writing for a scientific audience but for a general audience magazine. But your piece, which again I want to emphasize is not relevant to the target you aimed at, takes on a special importance because of its timing and context. It is bouncing around the internet as debunking the current vaccine policy, but it does nothing of the sort. It is disingenuous to imply your intention was to educate doctors who aren’t aware of the basis for excess mortality estimates in influenza. The intention was clear: to debunk current vaccine policy for the pandemic. In the current context, such an intention is irresponsible journalism in our view, if it is not solidly based and yours is not because it addresses the wrong question but nowhere reveals this to the reader.

    I appreciate that no one likes to be criticized the way we criticized you. But neither do the many people who have to deal with this incredibly complex and difficult public health problem — for which the answers are not easy and not obvious — like also having to deal with the added nuisance of articles like yours which seed confusion and fear where there is already too much confusion and fear. For the public health community, it isn’t just another magazine article.

    Scott: What the swine flu vaccine will do for your health is provide some protection against one specific virus, the current pandemic swine flu. It will not protect you from seasonal influenza or the many other respiratory viruses that cause influenza-like illnesses.

  22. #22 Brian
    October 26, 2009

    The high-handed rhetoric of the reply by Brownlee and Lenzer is telling, and Revere’s substantive sur-reply devastating. These two journalists owe the readers of The Atlantic an apology.

  23. #23 tl
    October 26, 2009

    I find it amusing that although they mention Gorski’s arguments in their reply to this blog, they fail to address his statements in comments to his SBM blog.

  24. #24 Orac
    October 26, 2009

    Another blogger, Dr Gorski, who lauds Revere’s review succumbs to some of the same poorly made arguments as Revere and adds another prejudicial association and non-fact: that Tom Jefferson is associated with the anti-vaccine crowd because an anti-vaccine group invited him to speak.

    Straw man argument. The point was that Jefferson is clueless about whom he associates with and is, apparently unwittingly, giving aid and comfort to the anti-vaccine fringe.

    In any case, revere’s post was the best post about your article I have yet seen. Your article is a panoply of half-truths, questionable spin on the science, and misunderstanding, all spun around a hackneyed “brave martyr Cassandra” story structure with a doloop of the equally hackneyed “What if everything you think you know about X is wrong?” tagline. revere has written about flu vaccines for this blog for years now, and his posts have been among the most educational. He’s taught me a lot. Not only does he recognize the shortcomings in the data supporting flu vaccines, but he has written about them extensively right here. You would have done well to peruse his recent posts on H1N1 and flu vaccination before writing your article.

  25. #25 Jason Tetro
    October 26, 2009

    Here’s an wasy way to discern the truth from spin: Can anyone comment on how the flu vaccine (past and/or present) may contribute to original antigenic sin?

  26. #26 Catte Nappe
    October 26, 2009

    #18 – The article’s authors use an awful lot of defensive words, and still fail to address the point.

    How is it pertinent that there are studies showing minimal efficacy in vaccinating elderly people (which the Revere’s admit to); when this new virus is sickening (and killing) an inordinate number of children, teens and young adults?

  27. #27 Annodeus
    October 26, 2009

    Knowing a lot of journalists, I can’t help but wonder if an Atlantic editor in search of a stronger news angle led the authors astray, or if they themselves pitched the article that way. Either way, if they’re as smart as they seem, they knew the final result was a straw man setup.

  28. #28 Orac
    October 26, 2009

    Given Brownlee’s book (Overtreated), I think Brownlee pitched the story and decided on its spin, and, no, I don’t think Brownlee and Lenzer realize what a huge straw man their story is abased on.

  29. #29 david egilman MD MPH
    October 27, 2009

    There is good data that indicates that rhinovirus is spread by hand contact and that this is more important than droplet spread.

    There is no “need” to shake hands. Not shaking hands is cheap but not culturally favored. Culture is not science based.

    I have designed a button that can mitigate the adverse social consequences of no shaking hands. This and the literature n spread can be found at http://www.smilesnotgerms.org/.

    Profits if any wil be donated to African medical schools.

  30. #30 SJT
    October 27, 2009

    As an apparently unquestioning disciple of the social medicine theory of “medicalization,” Ms. Brownlee sometimes fails to meet the basic tenets of journalism which require not only balance and fairness, but accurate, reliable facts in meaningful context. It may be possible that confirmation bias clouds Ms. Brownlee’s news judgment at times. This would appear to have been one of them.

  31. #31 John Thacker
    October 30, 2009

    “Vaccines and antivirals are a poor second best as a strategy. But that’s what we have now, and while not the optimum, they do work”

    That’s not all we have. We also have encouraging people to wash their hands, wear surgical masks if they’re sick like they do in East Asia, stay home if they’re sick instead of going into work or school, and other behavior, perhaps going so far as to eliminate hand-shaking as suggested above. All of those work even better. Perhaps the money spent on the vaccine could be better spent encouraging people to do that; it’s hard to say.

    Apart from the vaccine, I do have severe worries that people clogging hospitals, emergency rooms, and doctors’ offices with mild cold or flu-like symptoms are both going to spread those diseases to people already sick and in the hospital or emergency room, and take up time and energy that could be used with other patients. We know that when hospitals and emergency rooms are crowded, people die. So too much of a H1N1 panic could indeed kill more people than it saves. I know from personal experience that hospitals, emergency rooms, and family doctors’ offices are absolutely swamped right now with people with minor symptoms or regular colds who are deathly afraid of H1N1.

    OTOH, we know that people don’t take seasonal flu and hygiene seriously enough, so the warnings also save people. I would bet that the warnings save more than they kill, but both effects surely exist.

    And I know from personal experience journalists are always trying to get you to say something in the least nuanced way. Too many scientists succumb.

    Sure, but it cuts both ways. We don’t want to tell people that the evidence for the vaccine is weaker and somewhat mixed but you should get it anyway because it’s better than nothing. Scientists are understandably worried about people not getting it if the scientists admit that it’s not as strong as the advice makes it out to be. When trying to persuade the public to get the flu vaccine, they understandably choose to be less nuanced. But you should admit that this is also a case of being less nuanced; perhaps positive Straussian-type bias, but there you go.

  32. #32 John Thacker
    October 30, 2009

    If the vaccine is effective but not as effective as we think, then it may not be the best use of public health resources. In that case, we could be wasting time, money, and energy on it. In that case, we would be killing people on net. More people would die because of misallocation of public health resources.

  33. #33 John Thacker
    October 30, 2009

    Only a zealot who didn’t understand statistics would think that the magnitude of the effect, as separate from statistical significance, was unimportant. Even if we’re absolutely sure that it has some benefit, if the benefit is small, then it may be a waste of resources.

    Perhaps, and this is completely premature of course, we should be spending our money investigating to see if statins really do reduce influenza mortality, or if the various claims about antioxidants hold up.

    You can’t say that “this helps, so we must do it.” It’s not a choice of massive mobilization to manufacture and administer a vaccine or nothing. It’s a choice between one and doing something else with our time, money, and energy.

  34. #34 John Thacker
    October 30, 2009

    I’m heartened by the New Zealand and Australia examples so far; thankfully, while H1N1 flu season started several months earlier than seasonal flu, it also seems to have ended several months earlier rather than, as I feared, only getting worse and peaking at the same time as seasonal flu but with hugely greater numbers. The mortality numbers don’t look much worse than a regular seasonal flu. (Mortality numbers are always going to be more reliable than hospitalizations or especially visits to a doctor; people definitely are going to a doctor this year for symptoms that they’d stay home with in other years. OTOH, that aggressive treatment may actually save lives and make H1N1 look less virulent in the numbers than it actually is. In other words, it could be more virulent but people take the precautions that they ought to but don’t against seasonal flu.)

    At the same time, I’m very worried that too much hyping of H1N1 will give credence to the anti-MMR, anti-DPT, anti-other vaccine kooks out there if Australian and New Zealand numbers hold up. The “MMR causes autism” kooks are already pointing to Australia and saying, “See, they didn’t get the right vaccine and had a small, near seasonal number of deaths, therefore all vaccines are a scam!” Do we all really believe that the way to bury the kooks is to swallow our own ambiguity and nuance when it comes to warning against H1N1?

  35. #35 jay
    October 30, 2009

    but with dozens (perhaps scores?) of active viruses out there this year (flu and other), odds are the vaccine will probably do nothing for my overall health this winter. Am I reasoning that correctly?

    I tend to feel similarly. I’m 60 years old, I don’t remember ever getting the ‘seasonal’ flu, i.e. the specific one that the vaccine addressed. I had a few flu shots in the past, but haven’t bothered for years. Of course I’ve been sick from time to time, but none of those cases would have been helped by the vaccine.

  36. #36 jay
    October 30, 2009

    BTW the son of a co-worker was recently diagnosed with ‘swine’ flu (despite the official push to modify the language to the much more awkward H1N1, people still stick to the easily remembered nomenclature, especially since there are H1N1 variants that are not swine flu)

    Anyhow, after a regimen with Tamiflu, in less than a week his fever’s gone and he’s getting ready to go back to school

  37. #37 flu vacskeptic
    November 1, 2009

    Doesnt anyone use citations in their “scientific” articles anymore?
    i see both sides of the vaccine argument making outrageous claims like “most literature shows” or “vaccines prevent deaths”
    with no citations whatsoever. I believe in evidence based medicine not hyperbole science without citations

  38. #38 revere
    November 1, 2009

    flu vac skeptic: Follow the links we have written on the subject (vaccine category, left sidebar). Many of them have the cites in them, but this is a blog, not a journal (I know the difference because i am also a journal editor). That said, we do pay attention to evidence. Here’s a post we did on vaccine efficacy (in part) that has a link to the literature and discussion of it:

    http://scienceblogs.com/effectmeasure/2009/07/transmission_pathogenicity_vir_1.php

  39. #39 Kelly Duke
    November 2, 2009

    Just one question:

    In what other “science” would sufficient controlled experimentation to prove the advanced hypothesis (that the overall health benefits of flu vaccines outweigh their health risks) be regarded as radical and unethical?

  40. #40 revere
    November 2, 2009

    Kelly: There are strict controls on experimenting with human subjects, so any science that used human beings as research subjects would come under that heading. Moreover, it is just plain infeasible to do this during a pandemic and anybody who thinks about this will realize why.

  41. #41 Kelly Duke
    November 3, 2009

    I am not suggesting that we do these trials during a presumed pandemic. What I am suggesting is that we do these trials at some point. Otherwise we simply continue, in perpetuity, to presuppose what we seek to prove conclusively.

    Do you agree or disagree?

  42. #42 revere
    November 3, 2009

    Kelly: It may be past time when it is possible to carry out such a trial, since it is accepted practice. We’ll have to see what observational studies reveal and how strong that evidence is. I would caution that RCTs are not the last word (they often conflict) and are themselves not really purely experimental studies. Only the treatment allocation part is experimental. From then on it is the usual problem of doing uncontrolled human observation since we don’t have complete control over the subjects (if that weren’t true we’d only have to do a single RCT for each treatment we ever come up with). It’s also incorrect to assume that non RCTs don’t provide solid and reliable evidence. If you’ve ever tried to do an RCT you’d realize they aren’t as solid as some people think. They are difficult and leave all sorts of loose ends to resolve. So I guess it isn’t a “yes” or “no” answer. It will depend on what we find out after the flu season is over.

  43. #43 Kelly Duke
    November 3, 2009

    What has become “accepted practice” is to fasttrack new flu vaccines with no clinical trials of any sort, no toxicology studies of any sort and no animal reproduction studies of any sort based on our assumptions that benefits always outweigh risks for all flu vaccines developed using existing seasonal flu vaccine manufacturing processes that were subjected to clinical trials that showed reduced efficacy in the most at risk groups (young children & seniors) but were also never subjected to any toxicology studies, animal reproduction studies or RCTs of any sort.

    Here is how every insert for every CDC approved injectable H1N1 vaccine reads:

    Pregnancy Category C: Animal reproduction studies have not been conducted with (the new swine flu vaccine) or (the seasonal flu vaccine it is based on). It is also not known whether these vaccines can cause fetal harm when administered to a pregnant woman or can affect reproduction capacity. (The new swine flu vaccine) should be given to a pregnant woman only if clearly needed.

    Yet for some reason, all public health officials are promoting this (generally thimerosal laden) vaccine to pregnant women first and foremost. Based on what? This mass vaccination program for H1N1 variant virus will be the largest vaccination experiment on pregnant women in US medical history.

  44. #44 revere
    November 3, 2009

    Kelly: the current vaccines have all had clinical trials. They are classified (correctly as far as I can see) as a strain change, as happens every other year or so. You’ve made your point. You are opposed to the vaccine. Let’s move on.

  45. #45 dolphin
    November 4, 2009

    I’m not sure I get this right, but is it possible that one of the core arguments in your post goes _against_ randomized trials…?

  46. #46 Kelly Duke
    November 8, 2009

    The CDC’s own clinical trials are all using the thimerosal-free versions of the injectable vaccines while 98% of the public gets the thimerosal-spiked versions.

    Do you also approve of that “science”, revere?

  47. #47 revere
    November 8, 2009

    Kelly: And your cite for this?

  48. #48 ct
    November 17, 2009

    No mention of how Vitamin D levels help immunity here. Only that the two authors basically are too biased to have anything worthwhile to say. You need to see the site http://www.putchildrenfirst.org/ whereas the CDC’s not-so-honest behavior is discussed and real examples are put forth. Why give immunity to vaccination makers if the vaccination is so good?

  49. #49 Gordon
    September 21, 2011

    I think the most important thing is that people need to stay calm and not be so fearful. When we are scared we make emotional decision’s and they are not logical. Everyone so afraid when emotional stress itself make’s our immune system more prone to illness.