This week Canadian public health researchers published the long awaited paper on possible association between vaccination for seasonal influenza the previous flu season and risk of having a medically diagnosed infection with pandemic influenza during the first wave of infections (April to July) just as that season was ending. When preliminary results were first announced there was only vaccine against seasonal flu, which was still being given, and the results were contrary to what we thought we knew about flu biology and the immune system. Inevitably it became caught up in the wider anti-vaccine controversy while there were no published data for others to make a reasoned judgment about the study itself. Maryn McKenna for CIDRAP) has done some excellent reporting on the paper already and the paper is accompanied in the journal PLoS Medicine by a Perspective from Cecile Viboud and Lone Simonsen that is a model of concision and judicious evaluation. You can read both the Perspective and the paper itself (which is long and detailed but not very difficult) since PLoS Medicine is in an Open Access journal and there is no cost. So you don’t need me to go through the paper in detail. But I do want to make a longish comment on it about things I think will help put it in better perspective, the excellent piece by Viboud and Simonsen notwithstanding. The bottom line point I will make is that the paper isn’t news, at least not science news. That’s not because it isn’t original work or doesn’t tell us something we didn’t know or isn’t good. It is a very good paper, tells us new things and is certainly original work. It isn’t science news because in a sense there is no such thing. I’ll have to explain what I mean by this but first I want to cover one other issue that has come up in the reporting on this, the fact that as an “observational” study it has some inherent weaknesses. This is true, but easily misunderstood, so the first thing to consider is the logic of this study (the authors like to consider it 4 different studies but I prefer to think of it as a single study question with 4 sub-studies; you can decide if you think this is a difference or not).

So what is an observational study? It’s a study of the effects of a treatment or exposure (one could consider a vaccine as either) under conditions where an experiment involving randomized allocation isn’t done, isn’t feasible or isn’t possible. A political poll is not an observational study or a randomized one (the random part of polling is in sample selection, not treatment allocation). Epidemiology is just the study of health conditions in populations, so not all epidemiological studies are about effects, either. Some are just descriptive (for example, who is getting what diseases and when) or for administrative purposes. Thus only some epidemiological studies are observational. This one is, because it asks about the effects of a treatment. The treatment is getting a seasonal flu vaccine in 2008-2009 in one of four provinces of Canada; and the effect is the risk being diagnosed as a case of medically attended pandemic flu. The study was done because of an impression in a school outbreak very early in the pandemic that cases had been vaccinated more often than expected. This occurred at a time when there was no vaccine against the new swine flu strain, it was ramping up in reported cases, and no one knew where it was going. Was seasonal flu going to be crowded out by the new virus or come back? It made a difference about what to do about seasonal flu vaccine, since all studies by this group (and this paper confirmed it) showed seasonal flu vaccine quite effective. A randomized trial was infeasible at that point but there was a “natural experiment” going on and they took advantage of it with their observational study.

Some people not familiar with the real world of public health (which takes place in public and not in an ivory tower) might still think a randomized trial was necessary, so let’s discuss what the difference between an observational study and a randomized clinical trial (RCT) is in this instance. In an RCT people would be randomly assigned to receive seasonal flu vaccine or not and monitored to see if they got pandemic flu later. It would have to be prospective and involve a lot of effort and there was very little evidence at that point to think it would be useful. But there were alternatives and that’s what the paper presents. Of the 4 substudies, 3 are case-control studies where people with and without pandemic flu (by lab test) were compared on the proportion who were previously vaccinated for seasonal flu. If seasonal flu vaccine cross protects you might expect the amount of pandemic flu to be less in those previously vaccinated. The expectation was that it would be the same, i.e., no cross-protection, but the surprise was that the risk was actually higher in those who were previously vaccinated for seasonal flu.

If this had been an RCT, the two compared groups (those with medically attended pandemic flu and those without; you need to read the paper to see how these determinations were made) would be balanced with respect to other factors that might be related to getting pandemic flu. This includes all the known factors like age and comorbidity and also the factors that might be related to flu we don’t know about or didn’t have information about. Randomizing distributes all the factors (“covariates”) that were there independently of the vaccine/no vaccine allocation, so it is unlikely (although possible) that the two groups have different make-ups in some systematic way that affects the outcome (e.g., that one group was much older than the other). Randomizing makes it unnecessary to even know what all the factors are. While the two groups might still be different by chance (and often are), randomization legitimates the statistical tests we do and allows us to make statements about the chance of error in our judgments about whether the vaccine changed risk of getting pandemic flu or not.

Observational studies (by definition) don’t allow that random allocation. The two groups are now given to us by circumstance and may well be very different. If we know all the differences that are important that’s not a problem. We can adjust for them in a variety of ways (stratifying, using statistical models, matching of various kinds). It’s the differences we don’t know about or have no information about that are the problem. It could well be true that there are characteristics that people have, characteristics we don’t have any information about, that affect both the chance they will get vaccinated and the chance they will get pandemic flu and that the two groups differ in these characteristics. For example, suppose that care-seeking behavior were different in the two groups. Then people who were more likely to go to a doctor or clinic to get vaccinated might be more likely to go when they have an influenza like illness (ILI) and hence more likely to have a “medically attended” case of pandemic influenza. This is something the authors were attuned to and took pains to check indirectly to see if this was a plausible explanation, but there are many other possibilities, including ones no one has yet thought of. In an RCT you don’t have to worry about them. In an observational study you do. It is called residual confounding or hidden bias (hidden because you don’t have any information that would allow you to control for the effects of the factor).

There are a number of ways to control for confounders about which you have information and that was done in these substudies. There are 3 very similar substudies that are essentially replications. They use different populations and somewhat different protocols but all have the same problem that they don’t — they can’t — control for residual confounding. They are observational. The authors suggest that having three such substudies makes the problem of residual confounding less likely, but replication doesn’t do that, since the same problem is seen in each. In my opinion these studies don’t meaningfully eliminate uncontrolled confounders found in one and not in another. I qualified this with the word “meaningfully” because there is room to object that the studies aren’t identical but I don’t think they are different enough to be significant (not in the statistical but in the semantic sense). The 4th substudy is the weakest because it has small numbers but it does solidly eliminate the care seeking and some other hidden biases and for young adults also shows that seasonal flu vaccination makes later diagnosis of pandemic flu more likely.

Residual confounding — a hidden difference in who was and wasn’t vaccinated that affects pandemic flu risk — is a type of bias, but it isn’t the only type. Epidemiologists are also adept at uncovering, inventing and ferreting out all kinds of systematic error (meaning, not random error) that are not related to confounding. For example, supposing the test for pandemic flu consistently gave false positive readings. That would be a type of systematic error that isn’t confounding and could as easily plague a RCT. RCTs allow efficient handling of one kind of error, random error, but no others. Any RCT can have many different and disqualifying kinds of systematic error (“bias”). So the difference between the observational studies and the RCTs pertains only to a specific kind of bias, residual confounding. Both kinds of studies are subject to other kinds of error. Indeed there are good RCTs and very bad RCTs. RCTs, like observational studies, are often discordant for this reason. An RCT is no guarantee of validity any more than an observational study is a bar to it.

So that brings me to the second point, why this isn’t science news. Only a very rare piece of news about science is determinative for a particular question. The philosopher of science Susan Haack likens doing science to doing a cross-word puzzle. You have clues, you try to answer them with words that must fit into a certain number of spaces but also have to be cross consistent with other words. When they aren’t, sometimes it’s the word you just filled in, sometimes it’s one or more of the cross words. It takes time to do science’s crossword puzzle. To use one of Alice Stewart’s favorite proverbs, Truth is the Daughter of Time. When science is new it isn’t really “news.” It has to take its place against a background of other results, other disciplines (here immunology and virology), studies in other populations. Maybe this result reflects a truth that is correct for Canada but not for Mexico.

What’s my bottom line on whether seasonal flu vaccine ups the risk of pandemic flu? It well might. But it might not. Or maybe it does some times and not other times. We don’t know yet. This carefully done work opens up an important set of questions we now have to pursue. It’\s immediate public health import is probably small, because the pandemic strain will be a component of the next seasonal vaccine strain. This study confirms that the flu vaccine is quite effective in preventing the specific flu virus infections it is directed against or those quite similar. The surprise was the possibility it might go the other direction in some strains that are not as close but perhaps close enough to cause mischief. But those are words in the puzzle that still need to be filled in.

Meanwhile, I think everyone would say this study provides strong support for getting vaccinated against seasonal flu for next season [clarification: because I am assuming it will have pandemic H1N1 as a component as recommended by WHO]. Just to be clear.