Modeling flu responsibly

Last week another mathematical modeling paper made the newswires. If you wonder how this happens, the answer is that universities and companies have PR departments that put out press releases. Services like ScienceDaily aggregate and package these press releases for journalists and others (like us). Since a mathematical paper in a specialized journal (in this case it is PLoS Computational Biology) is not likely to be read by a reporter, especially a reporter on deadline, it isn't surprising the news stories follow the press release rather than the paper. In this case, I am sorry to say, the paper authors have read their own press release and incorporated it in the paper (please ignore the self-referential paradox here). It is an example of some good and competent modeling going beyond -- far beyond -- what the model will bear:

In a report to be published in the peer-reviewed journal PLoS Computational Biology and currently available online, Sally Blower, a professor at the Semel Institute for Neuroscience and Human Behavior at UCLA, and Romulus Breban and Raffaele Vardavas, postdoctoral fellows in Blower's research group, used novel mathematical modeling techniques to predict that current health policy -- based on voluntary vaccinations -- is not adequate to control severe flu epidemics and pandemics unless vaccination programs offer incentives to individuals.

[snip]

While other models have determined what proportion of the population would need to be vaccinated in order to prevent a pandemic, none of these models have shown whether this critical coverage can actually be reached. What has been missing, according to Blower, a mathematical and evolutionary biologist, is the human factor. The human factor involves two biological characteristics, "memory and how adaptable people can be," Blower said. "These characteristics drive human behavior." (ScienceDaily, adapted from University of California press release)

Sally Blower is an experienced mathematical modeler of infectious disease. Like many of us, she attracts post-doctoral fellows who want to extend their expertise and often employ new methods. I want my graduate students and post docs to succeed (that's true of any good mentor) and there is always a temptation invest their work with more practical significance than it deserves. After all, the goal is not to have them be post docs for the rest of their lives but to land a good faculty position in a research university. So this happens. But the flu world is so volatile and the sense of urgency for results with practical significance so strong that you have to be careful about how you portray your work. These authors, in my estimation, were not careful about that. Nowhere near.

Like all models, this one abstracts a complex reality. Its innovative feature is to add a model of human behavior to a very stripped down model of disease dynamics (a simple compartmental SIR model without vital dynamics, for those of you who read our series on antiviral modeling). The "novel" model is one derived from simple game theory, where individuals make decisions based on their experience and how much they are affected by that experience. This is what Blower means when she refers to "memory and how adaptable people can be." It is a big leap from there to claiming that "these characteristics drive human behavior." It is even a bigger leap when you see how the characteristics are modeled, but we'll leave that aside as there is so much else to talk about.

In particular, the authors assume that individuals act in their own self-interest and do not communicate their vaccination decisions to each other and that the vaccine is risk free, is 100% effective and lasts exactly one influenza season. There are a couple of more assumptions thrown in, just for good measure, but these are the main ones. Under these conditions, collective behavior (which isn't a collective behavior but just the sum of atomized behavior) is insufficient to bring the level of herd immunity consistently above the critical value to prevent an epidemic. There are a lot of other "minor" assumptions here (no seasonal drivers, vaccination occurs at the outset of each season, etc.) but enough has been said, I hope, to suggest that the statement that this paper "shows" that voluntary immunization "cannot" prevent severe epidemics is vastly overstated. This is a cartoon of voluntary behavior that ignores everything but whether a person was infected in the last season and whether they were vaccinated then or not, among other things. Yet they conclude:

We found that influenza epidemics could not be prevented in most seasons if vaccination was voluntary and no incentives were offered (Figure 1A). This result was a consequence of individuals making vaccination decisions each year on the basis of their past experiences.

Of course they "found" no such thing, unless finding something means that a run of their model exhibits such behavior. Having thus disposed of what they imply is voluntary behavior in general (rather than the very constrained and specific model they use to designate this behavior), they go on to talk about modifying voluntary behavior with "incentives." They model two incentive programs, one where buying a vaccination in one year entitles free vaccines for either two more years or 14 more years; and one where a single purchase is sufficient for a whole family but only for a single year. These are rather strange incentive programs tied tightly to a model of health services delivery. It isn't at all clear what relationship they have to other possible incentive programs (like mandatory vaccination of school children). These two highly stylized examples shed little light on the voluntary versus incentive question and in fact use these words in an unusual way. The usual contrast is between voluntary and mandatory, not voluntary with incentives and voluntary without incentives. You could consider a mandatory program to be voluntary with sanctions for non-compliance, but then that would have been a more natural thing to model.

There are a bunch of results, including some portrayed as "surprising" (sometimes a way of denying the fact that the model results were obvious ahead of time). I admit I only gave this paper a quick read (translation: 3 hours), but after lifting the hood and seeing the engine I didn't have much incentive to spend a lot of time dissecting it. These are competent modelers and I didn't expect to find egregious errors and for specialists it is interesting as the introduction of a new technique. But for those involved with flu policy, it is not ready for prime time. In fact, it isn't even ready for cable community access TV aired at 3 a.m. It's a small step in a difficult road along the way of figuring out what to do. We don't even know if it is headed in the right direction. You can't blame scientists for the distortion or sensationalization of their results by non-cientists. But you can write your papers and check the press releases in ways to minimize this. Of course if that had been done here, no one in the general public would have paid much attention to it.

Modelers practice an arcane science that is often difficult to explain. They believe in its value. But in the current setting, the temptation to inflate the significance of a paper in influenza modeling should be strenuously avoided. Using a press release to hype a publication is usually harmless. But in an area of science that purports to inform flu policy it is potentially dangerous, or at the very least adds noise. One of the things we liked so much about the Lipsitch et al. paper on modeling antivirals was that it didn't do this. It was quite precise in what its results might meant and how generalizable they might be. It was also substantively informative about a vital question in managing pandemic influenza. This paper, despite the fact it got more press than the Lipsitch et al. paper, falls far short in all these respects.

I hope good modelers continue to work on this problem. It will pay off. I also hope they do it responsibly.