Given the usual response to terrorist threats on airplanes, we expect the latest move to protect us will be to require us to travel nude. OK. Probably not. Republicans are too skittish about public nakedness. They prefer it in the privacy of their mistresses’ beds. What we will see, instead, is yet another attempt at a technical fix, spearheaded by high priced security and aviation “consultants.” I saw one of them, Mary Schiavo (former inspector general of the Department of Transportation) the other night on the PBS Newshour. She was hawking expensive explosive sniffers for airport check-in, as well as the scanners that undress you without undressing you. That apparently works just fine for prurient Republicans.
I’m not an expert on airport security (although I am an expert victim of airport security theater), but I do know something about statistics and probability and can recognize a classic fallacy when I hear one. And I heard one, not only from Ms. Schiavo (an attorney) but also her “counter-point,” David Schanzer, director of the Triangle Center on Terrorism and Homeland Security at Duke (one Newshour correspondent, who shall remain nameless, once wryly described to me the format of the show in this way: “And now for another view of the Holocaust . . . “). Here are the relevant parts of the segment:
GWEN IFILL: David Schanzer, is there — is there a technology in place that could have avoided this kind of failure?
DAVID SCHANZER, director, Triangle Center on Terrorism and Homeland Security: Well, the full body scan machines can do a better job, and they can improve the likelihood of finding something like that.
But there’s no 100 percent screening device that’s going to be able to pick up everything. And I think you have to ask yourselves before you deploy a multibillion-dollar technology whether or not you get more bang for the buck out of things like intelligence enhancement, watch-listing, more international cooperation.
GWEN IFILL: Let’s back up a minute. You said there’s no 100 percent guarantee. And the president, in fact, said that himself today. But wouldn’t people be satisfied if — knowing that, since 9/11, we were at 75 percent, 80 percent?
DAVID SCHANZER: Well, it’s all a bit of — it’s all probabilistic.
The question is, though, that we have an adaptive adversary, that whatever technology we deploy, they’re going to take steps to try to circumvent it. And, so, the problem is, if you invest huge amounts of money in these technologies, they might become obsolete when the next type of threat comes up six months or a year from now.
GWEN IFILL: What do you think about that, Mary Schiavo?
MARY SCHIAVO: Well, that’s — that’s really not the way I would approach it at all.
Obviously, we have to invest in the technology, because it’s the technology that can spot so many of these threats. Not 100 percent? Well, it could be very close to 100 percent, because there are four different machines with four different technologies that can spot explosives and explosive materials and components of bombs.
And, here, we can’t say we rely on profiling and intelligence, because that’s what we were relying on, on September 11, 2001. We don’t always fit the profile. There have been young beautiful North Korean women to someone over Indiana in 1933 blowing up planes. We cannot rely on profiling and intelligence, because we have proven that, over the last 70 years, it has failed. Hardware is our last line of defense, and it can be pretty close to 100 percent.
[snip]
GWEN IFILL: David Schanzer, it seems like there a lot of costs that we’re talking about here, the costs of actually the physical equipment, of getting the money, the costs of what you give up once you agree to this sort of — of what some people consider to be an invasive technology.
What would you say the costs are?
DAVID SCHANZER: Well, you have named some of them. The fact of the matter is, we live in a world of limited resources. So, we have to make trade-offs and choices about which set of policies and which sets of technologies we want to deploy.
Your other guest mentioned four different types of machines. Well, I don’t think we’re going to be able to deploy four different new types of machines, not only in the United States, but we would need to deploy these things globally to truly protect us.
I’m not saying we shouldn’t have screening devices. We — absolutely, we should. All I’m saying is that you have to consider the full package and figure out what set of policies is going to do the best at reducing the risks that we all face.
GWEN IFILL: Mary Schiavo, we saw you saying, yes, we are. You know, those four machines can be deployed.
But let me ask you this. At what cost in terms of civil liberties?
MARY SCHIAVO: Well, the cost in civil liberties, the great thing about machines is, they treat everyone the same. The machines don’t violate our civil liberties.
What violates civil liberties, when you say, well, we’re going to pick out this person and look at them, or this person and look at her, et cetera. The machines treat us all the same.
The only civil liberties issue so far that makes any sense is where it reveals the shape of the human being. But even that, the machines have gotten better. And the private parts can be shaded or — or not shown. And those machines have improved as well.
[snip]
DAVID SCHANZER: Well, just to make one point, the privacy protections that your guest mentioned might have made it more difficult to detect this particular device.
I just think that, when you have limited resources, you have no choice but to make these risk trade-offs. Maybe some types of devices would be useful, but, again, this device, the body scanner, is used in secondary screening. It isn’t used for everybody.
So, at some point, you are picking who is going to go through that device and who isn’t, unless you’re, again, willing to deploy it and have everybody in the world who’s traveling at that point be screened. So, that is a very expensive endeavor. And, again, the adversary is just going to simply adapt and try to find a different way to attack us. (Transcript, PBS Newshour segment on trade-offs between security and civil liberties)
Schiavo has made the basic error. Schanz is essentially correct that it’s a cost issue, but he ratifies Schiavo’s mistake by picking on the least problematic part of her analysis, how good the technology is in picking up explosives and ignores the fallacy that would strongly support his point. Since he didn’t do it, we’ll do it for him.
The basic error is this. Schiavo is citing the probability that someone carrying explosives will be detected by these expensive machines. She claims it is essentially 100% and Schanz’s retort is that this is unlikely. Actually it is not unlikely. I can easily design a machine that will identify every single person carrying explosives onto an airplane. It is a black box with a red light on top of it. When someone with explosives walks past it, the red light is on. Now all I have to do is wire it so that it is always on. That way it picks up 100% of all explosives carrying passengers.
This is a classical screening problem in epidemiology and we know how to solve it, although most doctors have faulty intuitions because they make the same mistake as Schiavo. We’ve discussed the medical version here already, so let’s stick to airport screening. The probability of detection isn’t the relevant probability. The question we want to answer is not the probability of picking up a terrorist carrying explosives (the question Schiavo posed). It is the probability that a “detected” passenger will be carrying explosives. To address that we also need to know the probability that the detector will go off if someone is not carrying explosives (a false positive) and the probability that any passenger is carrying explosives.
First the false positives. The problem with my 100% sensitive machine is that it also says a lot of people who don’t have explosives do have them. You don’t want those cases, but if you want 100% or close to 100% detection ability, as Schiavo claims, you know you are going to have to accept some. The simple RFID anti-theft devices in stores have them all the time and that’s a relatively easy problem compared to explosives detection (NB Sciavo cites a combination of four different kinds of machine required because explosives vary and all it would take is a new kind of explosive or some other non-explosive that had a similar signature to one to screw things up). Those “minor” quibbles aside, let’s assume that by investing a gazillion dollars we could deploy some sophisticated technology at every airport within our borders and coming to and from the US that was so accurate it only had a false positive once in 100,000 passengers, i.e., it was 100% sensitive and 99.999% specific. I doubt we can make a machine that accurate, but let’s just suppose we could.
How many false positives would that produce? According to the Department of Transportation, during the last year there were about 710 million enplanements (US carriers, October 2008 – September 2009; excludes all-cargo services, includes domestic and international). That would produce 7100 false alarms, about 20 a day. How many passengers carrying explosives would the technology pick up? Well, we’ve had exactly 2 since 2001 (Richard Reid the shoe bomber and the current underpants bomber), or .25/710,000,000 enplanements (it’s actually less because enplanements have decreased substantially since 2001). So the probability of an alarm being correct is about 1 in 30,000 or .000033. For that yield there is the cost of research and development of the technology, acquiring and installing it, operating and maintaining it and the extra time of all the passengers. There will also be an effect on air travel generally, stressing an already economically desperate industry. To the extent that increases miles traveled by road, we have to add that cost and the cost in lives of motor vehicle accidents into the mix.
Of course there will be those who say it’s worth it, whatever the cost in dollars (direct cost, only, estimate of $100 billion; why “no cost too high” should be true for air travel and not health care reform baffles me, but human psychology isn’t always rational). But the “worth it” argument is only valid if it worked. As others have said, including Schanz on the PBS Newshour segment, this is essentially a reactive strategy. There’s almost always a way — often an easy way — around technological fixes like this. They usually involve human engineering exploits, not technological ones. Yet we are the proverbial generals always fighting the last war. Nor is it irrelevant to the cost accounting that there have been two examples in 8 years of passengers carrying explosives aboard airplanes but zero examples of successful detonation. Even when you get the stuff onboard, there seems to be a substantial gap between paranoid fantasy and actual practice. It’s just not that easy.
So as we head into the new year, I am expecting air travel to be even more unpleasant and as a consequence I will be even more eager to avoid it. That means I will be encouraging more videoconferencing and conference calls and less face to face meetings. There’s always something lost when you don’t meet in person. But what is gained is not having to deal with the utterly stupid, pointless and unnecessary response that is sure to come.
Happy New Year everyone!