The Transportation Safety Administration (TSA) is one of the most maligned and hated federal agencies. And for good reason. They are worse than useless, performing uncounted acts of “security theater” (in Bruce Schneier’s apt coinage), like confiscating water bottles and making you take off your shoes. The TSA also has an active use of “behavior detection officers” whose job it is recognize the telltail signs of a terrorist intention — like looking anxious before getting on an airplane. The TSA claims the program, which is now in operation at 150 major airports and employs almost 2500 behavior officers (soon to increase to 3500), is “incredibly effective” at catching airport criminals. Unfortunately the TSA’s own figures say otherwise:
A TSA program launched in early 2006 that looks for terrorists using a controversial surveillance method has led to more than 160,000 people in airports receiving scrutiny, such as a pat-down search or a brief interview. That has resulted in 1,266 arrests, often on charges of carrying drugs or fake IDs, the TSA said.
[snip]
“That’s an awful lot of people being pulled aside and inconvenienced,” said Carnegie Mellon scientist Stephen Fienberg, who studied the TSA program and other counterterrorism efforts. “I think it’s a sham. We have no evidence it works.” (Thomas Frank, USAToday)
On the one hand, a 1% success rate doesn’t sound so good, but when converted to 1266 arrests it sounds significant. On the other hand, TSA doesn’t claim it has caught a single terrorist this way. We don’t know how many of the arrests led to convictions, but they are almost all for false IDs, drugs or other non-terrorist alleged violations. We don’t know how many arrests would be made if we just randomly searched people in the general community, something forbidden by the Constitution. Thus in terms of finding terrorists, the positive predictive value of this technique would seem to be zero.
This is not a surprise. We noted recently that the National Academy of Sciences recently had a committee examine this issue and they came to the conclusion that these techniques don’t work. The reason is not that they are too insensitive, as one of the designers of the program, psychologist Paul Ekman, claimed (“The shortcoming is, we don’t know how many people are showing suspicious behaviors and aren’t being noticed,” Ekman said). The problem is that the proportion of terrorists trying to get on airplanes is vanishingly small. Suppose that behavioral detection were so sensitive that you picked every single terrorist trying to get on an airplane. There are about 750 million passenger trips this year, so let’s say one in ten million involves a terrorist, or 75 terrorist trips. Let’s also say that the behavioral test IDs them all (highly unlikely) but also makes a mistake in about one in a ten thousandth of a percent of passenger trips. That’s 750 false positives. That means for behavioral detector whose skills are 100% sensitive and 99.9999% specific (meaning he tags the wrong person only once per million times) the positive predictive value would still be only 10%. So it isn’t at all surprising that the PPV of the current system is functionally zero and probably exactly zero. It’s a fool’s errand.
The people who design and institute these systems haven’t thought them through. Actually, that’s much too charitable. They are incompetent nincompoops who have no idea what they are doing.