Effect Measure

This week Canadian public health researchers published the long awaited paper on possible association between vaccination for seasonal influenza the previous flu season and risk of having a medically diagnosed infection with pandemic influenza during the first wave of infections (April to July) just as that season was ending. When preliminary results were first announced there was only vaccine against seasonal flu, which was still being given, and the results were contrary to what we thought we knew about flu biology and the immune system. Inevitably it became caught up in the wider anti-vaccine controversy while there were no published data for others to make a reasoned judgment about the study itself. Maryn McKenna for CIDRAP) has done some excellent reporting on the paper already and the paper is accompanied in the journal PLoS Medicine by a Perspective from Cecile Viboud and Lone Simonsen that is a model of concision and judicious evaluation. You can read both the Perspective and the paper itself (which is long and detailed but not very difficult) since PLoS Medicine is in an Open Access journal and there is no cost. So you don’t need me to go through the paper in detail. But I do want to make a longish comment on it about things I think will help put it in better perspective, the excellent piece by Viboud and Simonsen notwithstanding. The bottom line point I will make is that the paper isn’t news, at least not science news. That’s not because it isn’t original work or doesn’t tell us something we didn’t know or isn’t good. It is a very good paper, tells us new things and is certainly original work. It isn’t science news because in a sense there is no such thing. I’ll have to explain what I mean by this but first I want to cover one other issue that has come up in the reporting on this, the fact that as an “observational” study it has some inherent weaknesses. This is true, but easily misunderstood, so the first thing to consider is the logic of this study (the authors like to consider it 4 different studies but I prefer to think of it as a single study question with 4 sub-studies; you can decide if you think this is a difference or not).

So what is an observational study? It’s a study of the effects of a treatment or exposure (one could consider a vaccine as either) under conditions where an experiment involving randomized allocation isn’t done, isn’t feasible or isn’t possible. A political poll is not an observational study or a randomized one (the random part of polling is in sample selection, not treatment allocation). Epidemiology is just the study of health conditions in populations, so not all epidemiological studies are about effects, either. Some are just descriptive (for example, who is getting what diseases and when) or for administrative purposes. Thus only some epidemiological studies are observational. This one is, because it asks about the effects of a treatment. The treatment is getting a seasonal flu vaccine in 2008-2009 in one of four provinces of Canada; and the effect is the risk being diagnosed as a case of medically attended pandemic flu. The study was done because of an impression in a school outbreak very early in the pandemic that cases had been vaccinated more often than expected. This occurred at a time when there was no vaccine against the new swine flu strain, it was ramping up in reported cases, and no one knew where it was going. Was seasonal flu going to be crowded out by the new virus or come back? It made a difference about what to do about seasonal flu vaccine, since all studies by this group (and this paper confirmed it) showed seasonal flu vaccine quite effective. A randomized trial was infeasible at that point but there was a “natural experiment” going on and they took advantage of it with their observational study.

Some people not familiar with the real world of public health (which takes place in public and not in an ivory tower) might still think a randomized trial was necessary, so let’s discuss what the difference between an observational study and a randomized clinical trial (RCT) is in this instance. In an RCT people would be randomly assigned to receive seasonal flu vaccine or not and monitored to see if they got pandemic flu later. It would have to be prospective and involve a lot of effort and there was very little evidence at that point to think it would be useful. But there were alternatives and that’s what the paper presents. Of the 4 substudies, 3 are case-control studies where people with and without pandemic flu (by lab test) were compared on the proportion who were previously vaccinated for seasonal flu. If seasonal flu vaccine cross protects you might expect the amount of pandemic flu to be less in those previously vaccinated. The expectation was that it would be the same, i.e., no cross-protection, but the surprise was that the risk was actually higher in those who were previously vaccinated for seasonal flu.

If this had been an RCT, the two compared groups (those with medically attended pandemic flu and those without; you need to read the paper to see how these determinations were made) would be balanced with respect to other factors that might be related to getting pandemic flu. This includes all the known factors like age and comorbidity and also the factors that might be related to flu we don’t know about or didn’t have information about. Randomizing distributes all the factors (“covariates”) that were there independently of the vaccine/no vaccine allocation, so it is unlikely (although possible) that the two groups have different make-ups in some systematic way that affects the outcome (e.g., that one group was much older than the other). Randomizing makes it unnecessary to even know what all the factors are. While the two groups might still be different by chance (and often are), randomization legitimates the statistical tests we do and allows us to make statements about the chance of error in our judgments about whether the vaccine changed risk of getting pandemic flu or not.

Observational studies (by definition) don’t allow that random allocation. The two groups are now given to us by circumstance and may well be very different. If we know all the differences that are important that’s not a problem. We can adjust for them in a variety of ways (stratifying, using statistical models, matching of various kinds). It’s the differences we don’t know about or have no information about that are the problem. It could well be true that there are characteristics that people have, characteristics we don’t have any information about, that affect both the chance they will get vaccinated and the chance they will get pandemic flu and that the two groups differ in these characteristics. For example, suppose that care-seeking behavior were different in the two groups. Then people who were more likely to go to a doctor or clinic to get vaccinated might be more likely to go when they have an influenza like illness (ILI) and hence more likely to have a “medically attended” case of pandemic influenza. This is something the authors were attuned to and took pains to check indirectly to see if this was a plausible explanation, but there are many other possibilities, including ones no one has yet thought of. In an RCT you don’t have to worry about them. In an observational study you do. It is called residual confounding or hidden bias (hidden because you don’t have any information that would allow you to control for the effects of the factor).

There are a number of ways to control for confounders about which you have information and that was done in these substudies. There are 3 very similar substudies that are essentially replications. They use different populations and somewhat different protocols but all have the same problem that they don’t — they can’t — control for residual confounding. They are observational. The authors suggest that having three such substudies makes the problem of residual confounding less likely, but replication doesn’t do that, since the same problem is seen in each. In my opinion these studies don’t meaningfully eliminate uncontrolled confounders found in one and not in another. I qualified this with the word “meaningfully” because there is room to object that the studies aren’t identical but I don’t think they are different enough to be significant (not in the statistical but in the semantic sense). The 4th substudy is the weakest because it has small numbers but it does solidly eliminate the care seeking and some other hidden biases and for young adults also shows that seasonal flu vaccination makes later diagnosis of pandemic flu more likely.

Residual confounding — a hidden difference in who was and wasn’t vaccinated that affects pandemic flu risk — is a type of bias, but it isn’t the only type. Epidemiologists are also adept at uncovering, inventing and ferreting out all kinds of systematic error (meaning, not random error) that are not related to confounding. For example, supposing the test for pandemic flu consistently gave false positive readings. That would be a type of systematic error that isn’t confounding and could as easily plague a RCT. RCTs allow efficient handling of one kind of error, random error, but no others. Any RCT can have many different and disqualifying kinds of systematic error (“bias”). So the difference between the observational studies and the RCTs pertains only to a specific kind of bias, residual confounding. Both kinds of studies are subject to other kinds of error. Indeed there are good RCTs and very bad RCTs. RCTs, like observational studies, are often discordant for this reason. An RCT is no guarantee of validity any more than an observational study is a bar to it.

So that brings me to the second point, why this isn’t science news. Only a very rare piece of news about science is determinative for a particular question. The philosopher of science Susan Haack likens doing science to doing a cross-word puzzle. You have clues, you try to answer them with words that must fit into a certain number of spaces but also have to be cross consistent with other words. When they aren’t, sometimes it’s the word you just filled in, sometimes it’s one or more of the cross words. It takes time to do science’s crossword puzzle. To use one of Alice Stewart’s favorite proverbs, Truth is the Daughter of Time. When science is new it isn’t really “news.” It has to take its place against a background of other results, other disciplines (here immunology and virology), studies in other populations. Maybe this result reflects a truth that is correct for Canada but not for Mexico.

What’s my bottom line on whether seasonal flu vaccine ups the risk of pandemic flu? It well might. But it might not. Or maybe it does some times and not other times. We don’t know yet. This carefully done work opens up an important set of questions we now have to pursue. It’\s immediate public health import is probably small, because the pandemic strain will be a component of the next seasonal vaccine strain. This study confirms that the flu vaccine is quite effective in preventing the specific flu virus infections it is directed against or those quite similar. The surprise was the possibility it might go the other direction in some strains that are not as close but perhaps close enough to cause mischief. But those are words in the puzzle that still need to be filled in.

Meanwhile, I think everyone would say this study provides strong support for getting vaccinated against seasonal flu for next season [clarification: because I am assuming it will have pandemic H1N1 as a component as recommended by WHO]. Just to be clear.

Comments

  1. #1 Eric
    April 9, 2010

    My wife’s immediate thought was why would they combine the two vaccines next year when they don’t know what, if anything, is going wrong this year as observed in this study. She’s no scientist and neither am I, but I usually have an answer for her and I’m at a bit of a loss here. Looks like another year of worrying about flu vaccines around my household much as I try to keep things grounded in reality.

  2. #2 revere
    April 9, 2010

    Eric: They are including it as the H1 component because it is expected it will be the dominant strain. Hence the immune response will be matched (unlike cross reaction with seasonal H1). The other two components, as I understand it, will be H3 and flu B, so the situation from this anomalous year isn’t being recreated. Since we know vaccination is effective, this seems the best choice as opposed to any other conceivable option.

  3. #3 MoM
    April 9, 2010

    Susan Haack likens doping science to doing a cross-word puzzle.

    I’m pretty sure you meant doing science, but sometimes I’m just doping out the answers, so maybe not.

    In any event, thanks for this. Now I’ve got a few more things I need to read.

  4. #4 revere
    April 9, 2010

    MoM: LOL. At this stage of the grant process I might have meant almost anything. But the grant is at Kinkos being copied this weekend — all 796 pages. Has to be in Fed Ex to NIH latest on Wed. I’m still fretting about it and hovering over everyone but now I’m starting the unwinding stage. Last time it took me a couple of weeks. Probably a bit longer this time. And I have dash off a pre-proposal for a week from Monday and a seminar the following Friday. Then maybe a breather. Maybe.

  5. #5 Marc
    April 9, 2010

    “and for young adults also shows that seasonal flu vaccination makes later diagnosis of pandemic flu more likely.”

    Oh shit. Me and Alex got both seasonal and H1N1, we live in Canada and we are in that age group. That eliminates a lot of the confounding factors. I really hope that the study is flawed somehow. Revere, are there papers that show the opposite? That there is no link between seasonal flu vaccination and getting H1N1? This is pretty scary.

  6. #6 revere
    April 9, 2010

    Marc: No uncontrolled confounders are removed by your status. Age was controlled as were sex and other factors. You proper course of action is to get vaccinated with the swine flu vaccine which is targeted to the virus. Easy. And yes, there are a number of papers showing the opposite but it isn’t an issue if you are vaccinated with the swine flu vaccine.

  7. #7 Marc
    April 9, 2010

    @Revere: You didn’t understand. I got vaxxed for both. So did Alex. But you know there’s still a high enough chance that with the H1N1 vax I still get H1N1 (not 100% effective). My worry was that getting vaxxed with the seasonal vax increased my chances of getting H1N1 if the H1N1 vax didn’t work.

  8. #8 revere
    April 9, 2010

    Marc: The study under discussion was not about people who got both vaccines. We have no information about this at all except that we know the vaccine(s) raise antibodies so you should be fine.

  9. #9 kimw
    April 10, 2010

    Revere, bear with me as a former English major. Your post states there have been no randomized trials (how could there be?), but observational stats, hind sight, show that those vaccinated against seasonal flu were more likely to acquire an infection with H1N1. Is that a correct summary of your post?

    Humph. I recall early last fall when vaccinators came to our workplace. They offered only the seasonal flu vaccine and the pneumonia vaccine. I’d heard from you that H1N1 was the only flu around, and since I have a history of arm pain that sucks and feeling like crap just for one day after flu shots, along with malaise and some vomiting, just for a day, I said I that in my mid-fifties I only wanted the pneumonia vaccine. Had to argue for a while just to get my pneumonia shot and eschew the seasonal flu shot. Turned out the pneumonia shot put my arm out of commission and made me ill for one day only as well. But from from this observational study it sounds as though I accidentally made the correct choice at the time.

    Please correct my ignorance if necessary.

  10. #10 Paula
    April 10, 2010

    Reading kimw’s post (9) reminds me that when I had the seasonal flu this September (like all old folks, I was told Hey you’ve less risk from swine flu but more from seasonal, so get this one and no you have to wait to last for the new vaccine–at the very time we were, apparently, upping our swine flu risk from the seasonal vax), the vaccination itself hurt my arm like crazy–a rarity for me with flu shots; the swine flu vaccination, however, not only didn’t hurt but I literally didn’t feel it (hope she actually jabbed me!) I thought this difference was that the nurse in September, working through Walgreen’s, perhaps had little practice, and the swine flu vax was done by a very experienced nurse in the local p.h. office—but perhaps there was some difference due to the vaccines themselves–any known that affected immediate response? I’ve heard of none.

  11. #11 revere
    April 10, 2010

    kimw: Not exactly what the post says. The post is about a single paper. Several other papers have shown no effect on pandemic flu or a protective effect from geting seasonal flu vaccine. At our age, the pneumovax was a good move. Seasonal flu vaccine made no difference in our age group for pandemic vax (the effect, if real, was only in younger age groups) and it was not an effect modifier, i.e., it did not make medically attended pandemic flu more severe. Hospitalized patients for pandemic flu showed no difference in vaccination status. So you may have accidentally done the right thing. Or maybe not

    Paula: adverse effects (this includes sore arm) is reported to be no different with the swine flu vax than seasonal. Why some people get sore arms and others don’t you do in one case and not another we don’t know.

  12. #12 Epinephrine
    April 10, 2010

    Re #9 (kimw):

    The vaccine being examined was the 2008-2009 seasonal flu vaccine, not the 2009-2010 vaccine that you were offered.

    I am certainly not convinced that there was any risk. When two things are correlated it can simply be that the two are in fact related to a third variable (e.g. health care seeking behaviours).

  13. #13 revere
    April 10, 2010

    Epinephrine: That is the whole point of the post (residual confounding cannot be excluded). However a great deal of care was taken in this paper to evaluate care seeking as a confounder, and while it couldn’t be eliminated the evidence suggests it isn’t the most likely explanation. You shouldn’t be convinced on the basis of this paper, one way or another. That is the second point of the post (why it isn’t science “news”).

  14. #14 Epinephrine
    April 10, 2010

    Revere: I realise that is the point of your post; I was addressing kimw’s comment that s/he made the “correct choice,” in that even if we were discussing the appropriate vaccine the evidence can’t support the conclusion, and was trying to summarise in a concise manner.

  15. #15 revere
    April 10, 2010

    Epineprhine: OK. Sorry. It’s a tough subject and I don’t always explain it well, so I never know when what I think I’ve written gets lost.

  16. #16 Epinephrine
    April 10, 2010

    Actually, I think you explain things very well! I work in public health and find this a really great place/resource :).

  17. #17 bar
    April 11, 2010

    Maybe I am misreading. Perhaps the concern is that the micro infections (aka the flu vaccinations) that provoke immunity do thereby in some way attenuate the immune system’s response to similar infections,

    Have any of the sub studies eliminated this possibility. By for example relating the likelihood of pandemic infection in a patient to some sort of weighted count of past flu vaccinations on the patient?

  18. #18 revere
    April 11, 2010

    bar: One question is whether the effect is real or not. It has to take its place in the body of evidence and it will take time to sort it out. A second, somewhat related to that and to your question, is whether there is a plausible biological mechanism and what it is. Take a look at the paper because the Discussion summarizes some of the possibilities in this regard. Sifting through this will also take time and will take place in the lab, not the field.

    With respect to your use of the word “concern,” the concern is understanding what is going on biologically and epidemiologically, more than a current public health concern because the situation is now different: there is a pandemic flu vaccine that is so far well matched and will continue to be even if there is some drift. So we aren’t in the same situation as the “first wave” cases. There will be subsequent analysis of second wave and after cases where we will have some info on people who got both vaccines and that is obviously of great interest, although my prediction is that the picture will remain cloudy for quite a while until we generate more data of different kinds and try to fit it all together.

    Added: regarding your question of previous years vaccinations, there was a look at the previous year before the last flu season and there seemed to be no influence.