A new article in the British scientific journal, The Lancet, suggests that seasonal influenza vaccines may not be effective in preventing community acquired pneumonia in people 65 years old and older. This is the group specially targeted by CDC for vaccination each year and, not coincidentally, an age group that includes me. So I have both a scientific and personal interest in the subject. This isn’t new news. We’ve previously discussed the evidence that shows seasonal vaccines are less effective in the elderly a number of times (see here and here) over the last few years, but the proposition seems to contradict not only CDC policy but a number of studies that seem to show that flu vaccines are highly effective in the elderly. Let’s take a look at both the study and the issues.

First, most of the data are from what epidemiologists call observational studies, that is, they are not experiments where one group of elderly gets vaccinated and another similar group doesn’t and we wait to see if the vaccinated group does better. There is some but very little data of this type available from randomized clinical trials (RCTs) and we have discussed this before. Doing in RCT of this given the accepted benefits would have some serious ethical difficulties. But both RCT and surveillance data have raised plausible doubt that seasonal flu vaccination in older adults may not be as effective as claimed. The Lancet study is also observational in design (i.e., allocation of who was vaccinated was not under the control of the investigators) but it seeks to overcome some of the possible biases of previous observational studies. In particular it tries to account for possible differences in the underlying chances of getting pneumonia between those who got vaccinated and those who didn’t that are unrelated to the vaccine itself; and it counted pneumonias that didn’t require hospitalization as well as those that did.

The basic “study design” is called a nested case-control study. It starts with a defined population of people who have not yet gotten pneumonia, but some of whom would be expected to get it during the flu season. The population was members of a pre-paid health plan in western Washington state who had competent immune systems (this was guessed at by looking at pertinent information in their medical records). Medical information on this group, numbering more than 45,000 subscribers, was looked at over three seasons in which the vaccine was well matched to the virus (2000 – 2002), including pre-flu months and months when influenza was prevalent in the community. Some of these people received flu vaccine and others didn’t, depending upon their own interests and the recommendations of the health plan providers. 1173 episodes of medically verified case of pneumonia occurred in this group during the study periods (the same person could get pneumonia in more than one year so these are not necessarily all different people). Each person in one of these episodes was matched with two other people from the plan of the same age and sex. The pneumonias are called the cases. The matched people without pneumonia are called the controls. The resulting study group of 1173 cases and 2346 controls is said to be nested within the larger cohort of 45,000 subscribers. That’s why this kind of design is called a nested case-control design. The underlying idea is to see if the people without pneumonia were more likely to have gotten flu vaccine than those who came down with pneumonia.

Here’s a snapshot of the pneumonia cases and how they compared with the controls:

752 (64%) cases were outpatients, 447 (38%) were younger than 75 years of age, 531 (45%) were aged between 75 and 84 years, and 195 (17%) were at least 85 years old; 597 (51%) cases were men. Compared with controls, cases were more likely to have chronic diseases recorded in their medical records and, when recorded, these conditions tended to be more severe than in the controls. Cases were also more likely to have functional status impairments and have prescriptions for lung medications and antipsychotics than were controls. (Jackson et al., The Lancet 2008; 372:398-405 [subscription firewall]).

So the people who eventually became cases were sicker than the controls at the outset, which isn’t a surprise. So the researchers needed a way to compare the effect of vaccine on these two groups who weren’t similar to begin with. They did this with an accepted method of statistical adjustment, but the trick is to figure out what to adjust for, that is, which of the differences present at the outset needed to be taken into account. They took advantage of the fact that many people get vaccinated significantly before the flu season is underway. By looking at the odds of acquiring pneumonia in vaccinated and unvaccinated people before the flu season started they could see the effect of the irrelevant differences in the two groups. During the pre-influenza period the vaccinated group had 40% of the odds of getting pneumonia than the unvaccinated group, but this difference wasn’t due to flu vaccine because it was in the period before flu was prevalent in the community. By figuring out which characteristics of people were making the difference, the researchers could use that information to adjust the case – control comparison during the flu season, essentially “subtracting off” the differences that weren’t from the vaccine, leaving the vaccine differences (this isn’t how it was done statistically but conveys the basic idea). The adjusted factors (besides age and sex) were asthma or visit for asthma exacerbation; smoking history (ever vs never, and an indicator for whether pack-years was ever recorded in the chart); antibiotics prescribed for a lower respiratory condition; abnormal pulmonary function test; use of home oxygen; previous pneumonia episode during the baseline period; use of inhaled corticosteroids, oral corticosteroids, bronchodilators, statins, or antipsychotic drugs; and any visit to an optometrist or ophthalmologist (indicating the person was mobile and well enough to use elective services). When these factors were taken into account statistically for the pre-influenza period, the differences in the odds of getting pneumonia in those vaccinated and not vaccinated disappeared. The question then was what happens when they make the adjusted comparison when influenza is circulating in the community, the time when the vaccine would be expected to make a difference.

So what are the results? Before adjusting the odds of acquiring pneumonia during flu season was actually 4% greater if you got the vaccine, but the adjustment reversed this resulting in an 8% reduction. This good result is tempered by the fact that statistical fluctuations could also be an explanation for the difference (the 95% confidence interval when from 12% reduction to a 10% increase in odds from vaccination). If attention was restricted to the peak of the influenza season (the window where half the cases occurred), there was an increase of 4% (confidence interval of 30% reduction to 55% increase — in other words, the 4% estimated increase is pretty uncertain; the data are also compatible with a large reduction or an even larger increased odds of acquiring pneumonia after being vaccinated).The odds ratios bounce around a fair amount for the different seasons and time periods, adding to the uncertainty.

These data by no means rule out a small or even moderate effect of flu vaccination in the elderly, but they don’t provide convincing evidence a meaningful effect exists. If this is true (and we will need to look at this in other ways in other populations), what’s the explanation? The authors suggest two. One is that the vaccine isn’t very effective in this age group, perhaps because other factors are more important or the immune system has become weakened. The other is that the pneumonias being counted here are mainly other viral or bacterial agents unconnected with influenza. An accompanying Commentary in the same issue of the journal by Gelongia and Shay try to estimate the possible effectiveness of the vaccine in this older age group assuming different proportions of pneumonia that are truly influenza:

Although the proportion of pneumonia caused by influenza is not known in Jackson and colleagues’ study, we can estimate the effectiveness of influenza vaccine to prevent influenza-related pneumonia. We used the 8% effectiveness they found that for prevention [of] all-cause pneumonia and several assumptions. Our analysis suggests that vaccine effectiveness against influenza-related pneumonia might be about 70% if no more than 10% of pneumonia episodes were influenza related. On the other hand, vaccine effectiveness against influenza-related pneumonia could be less than 35% if influenza caused 30% or more of all pneumonia episodes during the months included in their analysis.(Belongia and Shay, The Lancet 2008; 372:352-354, table omitted)

So the newswire stories that flu shots “fail to stem pneumonia if older patients” only tells part of the story (that particular Bloomberg article also makes a complete hash of the study design). This well-done work again raises significant questions about the value of flu shots in the over 65 age group, the group specially targeted by CDC. But it doesn’t settle them.

Meanwhile I’ll continue to get my yearly flu shot.