Lies, Damn Lies, and Health Statistics

Last week, the Washington Post took Rudy Giuliani to task for an ad where he claims that his chances of surviving prostate cancer -- which he had about 6 years ago -- were much higher in the US than in the UK. The ad is meant to indict those who wish to modify the health care system.

He says:

"I had prostate cancer, five, six years ago. My chances of surviving prostate cancer and thank God I was cured of it, in the United States, 82 percent. My chances of surviving prostate cancer in England, only 44 percent under socialized medicine."

Here is the ad itself:

Since the ad, a flurry of claims and counterclaims as to the rightness and the not-rightness of Giuliani's data has ensued. Paul Krugman was not pleased. He pointed out that mortality from prostate cancer in the US and UK are roughly equivalent. Not to be outdone in the unequivocal department, Kevin Drum describes the comment as "Giuliani is full of s--t," and Ezra Klein adds, "It's -- no pun intended -- crap." (Hat-tip: RealClearPolitics)

On the other side of the debate, Clive Cook disagreed pointing out survivability from cancer differs between the US and UK for a variety of cancers besides prostate, and Cato argues that these differences in survival for other cancers nullify the critique of that the differences in prostate cancer survival are attributable to survivor time bias and over diagnosis. David Gratzer -- from whom Giuliani got his data -- issues his own rebuttal.

Who is right in all of this?

Well, to be precise Giuliani is both right and wrong, and most of his commenters are saying correct things but are approaching the problem differently.

Giuliani statistics are incorrect. The numbers used to calculate the 82% and 44% comparison are derived from a Commonwealth Fund report using OECD and WHO data. However, the Commonwealth Fund report states nothing about survival data, only incidence and mortality data. You cannot calculate survival rates knowing only incidence and mortality because to calculate survival you need to follow people from diagnosis across five years. (More on that in this FactCheck.org article.) So the data that Giuliani was using that he got from Gratzer is in fact crap.

However, it is correct that prostate cancer 5-year survival rate -- the percentage of people who survive five years after first diagnosis -- is higher in the US. The best and most recent statistics are from Verdecchia et al. in the Lancet which looked at the EUROCARE-4 study.

Verdecchia et al. actually did follow patients from diagnosis to five years for a variety of cancers in EU countries and compared those survival rates to rates from the US during the same period. The data is below (click to enlarge, the data is Table 5 in the cited article):

The data for prostate cancer is in the second chart. If you look you will find that the five-year survival for Wales and Scotland is about 71-72% with the rest of the UK unlisted. The US figures are at the bottom at 99.3%. There is a disparity in survival between the US and at least some parts of the UK for prostate cancer. In fact, the US has better survival than Europe as a whole for nearly every cancer listed. These data are at the bottom of each of the charts.

The total survival rates for all malignancies checked is in the last chart. It shows that in Europe the survival rate for all cancers is 47.3% for men and 55.8% for women. The survival rate for all cancers in the US is 66.3% for men and 62.9% for women. These data would appear at least to support the notion of superior survival in the US as a consequence of superior care.

Hold on just a second, though. Setting aside the issue of the heterogeneous nature of European healthcare, what are the effects of diagnosis rates on these data? Diagnosis matters because if you are diagnosing the cancer very earlier but that diagnosis has little influence on mortality from the disease this will exaggerate survival. If early diagnosis does not improve survival than the improvement in survival time from early diagnosis is a fiction.

This is almost certainly the case for prostate cancer. We attempt to diagnose prostate cancer very aggressively in the US. They do not do so in Europe. However, this does not necessarily translate into lower mortality. In many cases, men with prostate cancer are quite elderly. They often die of something besides prostate cancer without any medical intervention to stop the cancer. This means that the increased survival in the US may be magnified by increased diagnosis.

You can see this effect by comparing mortality for prostate cancer in the US and the UK; they are roughly equivalent. (This data is shown in the Washington Post article.) What about all cancers? Well, a comparison from the WHO between the US and the UK in 2002 shows that the overall mortality for the US is 134 per 100,000 and for the UK is 143 per 100,000. (See page 10 of the .pdf.) It is better in the US but not by a huge margin.

What to make of all this conflicting data? I would make two comments:

1) Say it with me now: high survival rates are not always equivalent to low mortality rates. Part of the issue is trying to compare mortality to survival rates, and they provide different images of the quality of care. Mortality suggests that for prostate cancer care may be equivalent but overall the US is doing better. However, mortality throws in a lot of other factors that can vary between countries such as diet and exercise. Can we attribute differences in diet and exercise to the quality of our health care system? I don't think so. (A similar issue is seen in comparing life-expectancy between the US and Canada. Yes, it is lower in the US, but this is largely because of higher homicide and accidental deaths -- neither of which are the problem of the healthcare system.)

On the flip-side, survival rates can also be misleading. Indeed, they are much more related to the quality of healthcare, but they are also prone to bias due to over diagnosis. Giuliani picked a poor example when he picked prostate cancer because the improved survival for that is probably because of early diagnosis. On the other hand, the US has substantially better survival for a variety of cancers where early diagnosis makes a big difference. These include breast and colorectal cancer. The US health system deserves to be lauded for their good survival with respect to these cancers.

2) When debating there is always a possibility of sanctifying your favorite number and filling in the rest of the numbers to satisfy your political preferences. This is clearly not a new phenomena in politics or human life, but I will quote what Aristotle said of the Pythagoreans -- the quintessential worshipers of numbers -- to prove my point:

Contemporaneously with these philosophers and before them, the so-called Pythagoreans, who were the first to take up mathematics, not only advanced this study, but also having been brought up in it they thought its principles were the principles of all things. Since of these principles numbers are by nature the first, and in numbers they seemed to see many resemblances to the things that exist and come into being-more than in fire and earth and water (such and such a modification of numbers being justice, another being soul and reason, another being opportunity-and similarly almost all other things being numerically expressible); since, again, they saw that the modifications and the ratios of the musical scales were expressible in numbers;-since, then, all other things seemed in their whole nature to be modelled on numbers, and numbers seemed to be the first things in the whole of nature, they supposed the elements of numbers to be the elements of all things, and the whole heaven to be a musical scale and a number. And all the properties of numbers and scales which they could show to agree with the attributes and parts and the whole arrangement of the heavens, they collected and fitted into their scheme; and if there was a gap anywhere, they readily made additions so as to make their whole theory coherent. E.g. as the number 10 is thought to be perfect and to comprise the whole nature of numbers, they say that the bodies which move through the heavens are ten, but as the visible bodies are only nine, to meet this they invent a tenth--the 'counter-earth'. (Emphasis mine, from Part 5 of Aristotle's Metaphysics)

We need to make sure that the we are not making a counter-Earth by selecting our favorite statistics and filling in the data to fit the rest.

I think that the US health care system on balance treats patients better than most healthcare systems of which I am aware. I am not suggesting that it is without problems, but on balance I think that our problem is not in patient care -- it is in paying for patient care. However, comparing two health care systems is a nearly insurmountable task. Reality is messy. Giuliani got caught with his trowsers down by being cavalier with statistics, and we all need to remember how complicated this problem can be.

Incidentally, Greg Mankiw has an excellent article in the NYTimes where he debunks a couple other health care statistical myths.