Effect Measure

A new article in the British scientific journal, The Lancet, suggests that seasonal influenza vaccines may not be effective in preventing community acquired pneumonia in people 65 years old and older. This is the group specially targeted by CDC for vaccination each year and, not coincidentally, an age group that includes me. So I have both a scientific and personal interest in the subject. This isn’t new news. We’ve previously discussed the evidence that shows seasonal vaccines are less effective in the elderly a number of times (see here and here) over the last few years, but the proposition seems to contradict not only CDC policy but a number of studies that seem to show that flu vaccines are highly effective in the elderly. Let’s take a look at both the study and the issues.

First, most of the data are from what epidemiologists call observational studies, that is, they are not experiments where one group of elderly gets vaccinated and another similar group doesn’t and we wait to see if the vaccinated group does better. There is some but very little data of this type available from randomized clinical trials (RCTs) and we have discussed this before. Doing in RCT of this given the accepted benefits would have some serious ethical difficulties. But both RCT and surveillance data have raised plausible doubt that seasonal flu vaccination in older adults may not be as effective as claimed. The Lancet study is also observational in design (i.e., allocation of who was vaccinated was not under the control of the investigators) but it seeks to overcome some of the possible biases of previous observational studies. In particular it tries to account for possible differences in the underlying chances of getting pneumonia between those who got vaccinated and those who didn’t that are unrelated to the vaccine itself; and it counted pneumonias that didn’t require hospitalization as well as those that did.

The basic “study design” is called a nested case-control study. It starts with a defined population of people who have not yet gotten pneumonia, but some of whom would be expected to get it during the flu season. The population was members of a pre-paid health plan in western Washington state who had competent immune systems (this was guessed at by looking at pertinent information in their medical records). Medical information on this group, numbering more than 45,000 subscribers, was looked at over three seasons in which the vaccine was well matched to the virus (2000 – 2002), including pre-flu months and months when influenza was prevalent in the community. Some of these people received flu vaccine and others didn’t, depending upon their own interests and the recommendations of the health plan providers. 1173 episodes of medically verified case of pneumonia occurred in this group during the study periods (the same person could get pneumonia in more than one year so these are not necessarily all different people). Each person in one of these episodes was matched with two other people from the plan of the same age and sex. The pneumonias are called the cases. The matched people without pneumonia are called the controls. The resulting study group of 1173 cases and 2346 controls is said to be nested within the larger cohort of 45,000 subscribers. That’s why this kind of design is called a nested case-control design. The underlying idea is to see if the people without pneumonia were more likely to have gotten flu vaccine than those who came down with pneumonia.

Here’s a snapshot of the pneumonia cases and how they compared with the controls:

752 (64%) cases were outpatients, 447 (38%) were younger than 75 years of age, 531 (45%) were aged between 75 and 84 years, and 195 (17%) were at least 85 years old; 597 (51%) cases were men. Compared with controls, cases were more likely to have chronic diseases recorded in their medical records and, when recorded, these conditions tended to be more severe than in the controls. Cases were also more likely to have functional status impairments and have prescriptions for lung medications and antipsychotics than were controls. (Jackson et al., The Lancet 2008; 372:398-405 [subscription firewall]).

So the people who eventually became cases were sicker than the controls at the outset, which isn’t a surprise. So the researchers needed a way to compare the effect of vaccine on these two groups who weren’t similar to begin with. They did this with an accepted method of statistical adjustment, but the trick is to figure out what to adjust for, that is, which of the differences present at the outset needed to be taken into account. They took advantage of the fact that many people get vaccinated significantly before the flu season is underway. By looking at the odds of acquiring pneumonia in vaccinated and unvaccinated people before the flu season started they could see the effect of the irrelevant differences in the two groups. During the pre-influenza period the vaccinated group had 40% of the odds of getting pneumonia than the unvaccinated group, but this difference wasn’t due to flu vaccine because it was in the period before flu was prevalent in the community. By figuring out which characteristics of people were making the difference, the researchers could use that information to adjust the case – control comparison during the flu season, essentially “subtracting off” the differences that weren’t from the vaccine, leaving the vaccine differences (this isn’t how it was done statistically but conveys the basic idea). The adjusted factors (besides age and sex) were asthma or visit for asthma exacerbation; smoking history (ever vs never, and an indicator for whether pack-years was ever recorded in the chart); antibiotics prescribed for a lower respiratory condition; abnormal pulmonary function test; use of home oxygen; previous pneumonia episode during the baseline period; use of inhaled corticosteroids, oral corticosteroids, bronchodilators, statins, or antipsychotic drugs; and any visit to an optometrist or ophthalmologist (indicating the person was mobile and well enough to use elective services). When these factors were taken into account statistically for the pre-influenza period, the differences in the odds of getting pneumonia in those vaccinated and not vaccinated disappeared. The question then was what happens when they make the adjusted comparison when influenza is circulating in the community, the time when the vaccine would be expected to make a difference.

So what are the results? Before adjusting the odds of acquiring pneumonia during flu season was actually 4% greater if you got the vaccine, but the adjustment reversed this resulting in an 8% reduction. This good result is tempered by the fact that statistical fluctuations could also be an explanation for the difference (the 95% confidence interval when from 12% reduction to a 10% increase in odds from vaccination). If attention was restricted to the peak of the influenza season (the window where half the cases occurred), there was an increase of 4% (confidence interval of 30% reduction to 55% increase — in other words, the 4% estimated increase is pretty uncertain; the data are also compatible with a large reduction or an even larger increased odds of acquiring pneumonia after being vaccinated).The odds ratios bounce around a fair amount for the different seasons and time periods, adding to the uncertainty.

These data by no means rule out a small or even moderate effect of flu vaccination in the elderly, but they don’t provide convincing evidence a meaningful effect exists. If this is true (and we will need to look at this in other ways in other populations), what’s the explanation? The authors suggest two. One is that the vaccine isn’t very effective in this age group, perhaps because other factors are more important or the immune system has become weakened. The other is that the pneumonias being counted here are mainly other viral or bacterial agents unconnected with influenza. An accompanying Commentary in the same issue of the journal by Gelongia and Shay try to estimate the possible effectiveness of the vaccine in this older age group assuming different proportions of pneumonia that are truly influenza:

Although the proportion of pneumonia caused by influenza is not known in Jackson and colleagues’ study, we can estimate the effectiveness of influenza vaccine to prevent influenza-related pneumonia. We used the 8% effectiveness they found that for prevention [of] all-cause pneumonia and several assumptions. Our analysis suggests that vaccine effectiveness against influenza-related pneumonia might be about 70% if no more than 10% of pneumonia episodes were influenza related. On the other hand, vaccine effectiveness against influenza-related pneumonia could be less than 35% if influenza caused 30% or more of all pneumonia episodes during the months included in their analysis.(Belongia and Shay, The Lancet 2008; 372:352-354, table omitted)

So the newswire stories that flu shots “fail to stem pneumonia if older patients” only tells part of the story (that particular Bloomberg article also makes a complete hash of the study design). This well-done work again raises significant questions about the value of flu shots in the over 65 age group, the group specially targeted by CDC. But it doesn’t settle them.

Meanwhile I’ll continue to get my yearly flu shot.

Comments

  1. #1 anon
    August 2, 2008

    if you could get a prepandemic H5N1 vaccine instead,
    what’s would be your choice ?

  2. #2 revere
    August 2, 2008

    anon: No way to answer that question. At the moment I would take a seasonal vaccine. It isn’t obvious to me that if a seasonal vaccine doesn’t work in the elderly a pre-pandemic one would work any better so it has little to do with the subject of the post.

  3. #3 M. Randolph Kruger
    August 2, 2008

    Anon-H5N1 vax so far is either so hot it could kill you or so cold you have to have a liter bottle of the stuff or two across two months to get a suggested resistance from the bug. Even the de and re-engineered vax that Webster put together was so hot that he said you would likely become very, very sick from it and survive. But they never went to human testing on it because it didnt work out so well in primates. Its a backup plan at best.

    As for any pandemic vax they might come up with… I wont be taking it. They can give mine to a kid if they think its safe even though I am on the qualified list behind HCW’s.

    I guess the best question that I have is for Revere…. At what point do they call you “elderly” by age? 65? 70? It would appear here that it MAKES you sick if you are older….. Hard to prove. Is the assertion that in combination with other conditions that if you take it you might end up sicker…annually?

  4. #4 revere
    August 2, 2008

    Randy; There already exists several prepandemic vaccines for H5N1. They aren’t matched to a pandemic strain so we’d have to hope for some cross-reactivity. They have been safety tested so they won’t kill you. Whether they will save you we don’t know.

    The age of subjects in this study were those over age 65 (oldest was 94). I don’t see any good evidence the vaccine was harmful. The question is whether it was helpful.

  5. #5 anon
    August 2, 2008

    it’s in US-stockpiles. Some of it is FDA-approved.
    Some of it is expiring now and they consider to administer
    it now instead of throwing it away. There is a recent
    paper which claims that booster-shots would be
    effective for years after first intake.
    It had been suggested to add one H5N1-component
    to the seasonal vaccine.
    Could it work “better” than seasonal vax for the elderly ?
    Well, the booster shot would be monovalent
    and targeted to a specific strain.

  6. #6 anon
    August 2, 2008

    intradermal vaccine seems to be better for the elderly:

    http://www.journals.uchicago.edu/doi/abs/10.1086/590434

  7. #7 revere
    August 2, 2008

    anon: Interesting. We discussed the underlying idea a couple of years ago here and here.

  8. #8 hjmler
    August 2, 2008

    i didn’t expect that a flu vaccine would give protection x pneumonia except incidentally – but in Minnesota nearly everyone offering flu shots also offers pneumonia shots as well… what i’m wondering about it how effective the pneumonia shots are …

  9. #9 revere
    August 2, 2008

    hjmler: Since the flu virus causes (viral) pneumonia and also sets up the lung for secondary bacterial pneumonia preventing it is one of the main functions of the vaccine. The pneumovax is said to be quite effective in preventing pneumococcal pneumonia but this study wasn’t designed to test that, although it took pneumovax into account.

  10. #10 pft
    August 3, 2008

    Why are we so trusting of FDA approved vaccines, especially with Bush in office.

    My father is almost 80 and jokes he wears gloves and a mask when he opens his social security check, taking care to do so outside. My mother had not had the flu for almost 3 decades until she got her 1st flu shot since it was being given out at the corner CVS (I realize this is not significant).

  11. #11 Rob
    August 4, 2008

    Revere:

    Nice post–really clear.

    Just to amplify one point: Claims about flu shot effectiveness have long been inconsistent. RCTs show the shots
    prevents symptomatic flu in healthy adults pretty effectively. Some studies indicate that flushot effectiveness by that measure–preventing symptomatic flu–may be lower in older people.

    Then there’s been a crazy claim from observational studies that flu shots can prevent HALF of all deaths among seniors that occur during flu season.

    That crazy claim is finally crumbling. It never made any sense, despite being repeated over and over. Flu leads to an average of ~5% of senior deaths during a typical flu season, so how could it prevent 50% of all deaths during flu season?

    What I don’t get is how that notion ever become part of the conversation–it’s prima facie silly.

  12. #12 Rich
    August 4, 2008

    I wonder if the value of immunization for the elderly is greater in institutional settings like nursing homes or senior centers that attract a regular clientele. Small effects with ecological concentration may help promote “herd immunity” (at least somewhat) and help contain the entry of virus into those populations.