Effect Measure

Data mining for fool’s gold by fools

Posted by revere on October 11, 2008
(6)
More »

The US government response to the bogus war on terrorism is a mixture of the stupid, the super stupid and the evil. Taking off our shoes at the airport isn’t evil but it is probably super stupid. So is the 3 oz. liquid ban. On reflection, maybe there are only two categories: super stupid and evil. In the evil category are the pseudoscientific “datamining” and “behavior detection” scams. Data mining is my favorite, since I use one of its techniques in my research work (Association Rule Mining) and I’m familiar with a number of other techniques. Familiar enough to not be surprised that when the National Research Council (a component of the National Academies of Science) looked at it of dubious scientific value:

The most extensive government report to date on whether terrorists can be identified through data mining has yielded an important conclusion: It doesn’t really work.

A National Research Council report, years in the making and scheduled to be released Tuesday, concludes that automated identification of terrorists through data mining or any other mechanism “is neither feasible as an objective nor desirable as a goal of technology development efforts.”

Inevitable false positives will result in “ordinary, law-abiding citizens and businesses” being incorrectly flagged as suspects.

[snip]

They admit that far more Americans live their lives online, using everything from VoIP phones to Facebook to RFID tags in automobiles, than a decade ago, and the databases created by those activities are tempting targets for federal agencies. And they draw a distinction between subject-based data mining (starting with one individual and looking for connections) compared with pattern-based data mining (looking for anomalous activities that could show illegal activities).

But the authors conclude the type of data mining that government bureaucrats would like to do–perhaps inspired by watching too many episodes of the Fox series 24–can’t work. “If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so.” (CNET)

The NRC Committee, which included Charles Vest, former President of MIT and William Perry, a professor at Stanford University, former Secretary of Defense, were not any kinder to “behavior detection”:

Behavior detection, used by the Transportation Security Administration and some police departments to isolate possible criminals from crowds, likewise falls short of meeting scientific standards, the group said.

“There is not a consensus within the relevant scientific community” that behavior detection is “ready for use … given the present state of the science,” the group said.

The group cautioned that “inappropriate … responses to the terrorist threat … can do more damage to the fabric of society than terrorists would be likely to do.”

[snip]

The council said terrorism is a threat to American society, but “inappropriate or disproportionate responses to the terrorist threat also pose serious dangers to society.”

“History demonstrates that measures taken in the name of improving national security, especially in response to new threats or crises, have often proven to be both ineffective and offensive to the nation’s values and traditions of liberty and justice,” the report says.(CNN)

Data mining to detect terrorists is the phrenology of computer science. It sounds and looks scientific. But it isn’t. The same with behavior detection. Plausible sounding but with no scientific basis for detecting deception by terrorists. If you’ve ever used these tools, you’d know how limited they are.

Unless you were in the Bush administration. Then you wouldn’t care.

(6)
More »

Comments

  1. #1 Dylan
    October 11, 2008

    In the early 1990′s, the Federal Government spent twenty million dollars on the “Stargate Project,” largely in an effort to “scientifically” study the feasibility of “remote viewing;” the project was dumped in 1995. These assholes never learn.

  2. #2 phisrow
    October 11, 2008

    Not to worry, we’ll just fire up the law and order noise machine…

    “Did you know that so called “false positives” and their extreme liberal allies are standing in the way of the sophisticated intelligence tools needed to save your children from the terrorists?”

    “In our post-9/11 world, we cannot afford to tolerate the deviant behavior of the “false” positives who conceal the terrorists in our midst.”

    A couple rounds of that, establish the notion that being a false positive = obstructing justice, and no more false positives, just another reason to add people to one of the nebulous lists of enemies.

  3. #3 Charles
    October 11, 2008

    The fun part is that driftnet methods like “data mining”, and, of course, promiscuous pervasive monitoring, simply guarantee paralysis when the notional threat turns real.

    You don’t have enough analysts to keep up with the data, because there is simply too much there to analyze. The result is that your actual enforcement gets bogged down in pursuit of political and personal vendettas.

    Meanwhile, the attackers carve you up at leisure.

    Adherence to old, outdated concepts like “probable cause” would strongly mitigate this risk.

    But nobody in authority seems to care. What the hell, let’s just hurl ourselves off history’s cliff faster.

    Glad I’m in my late 50s. With any luck, I may be safely dead before I get to see how this sort of mass insanity will work out, firsthand. What I can extrapolate is more than bad enough.

  4. #4 Pineyman
    October 12, 2008

    Dylan –

    Earlier this decade the Air Force spent $20M to determine whether teleportation was possible. The money went to either Mitre or Booz Allen. If you actually read it (I did, ~3 years ago), it sounds like it was written by Bugs Bunny with a side of Roger Ramjet. If I were either company I would’ve been ashamed to publish it. But hey, $20M is $20M. If you’re curious, I believe you can google it. Just use Teleportation study & US Gov.

    Phisrow – and if you think it can’t happen to you, it can. I was tagged in the mid 90′s when I made the mistake of flying one airline one way and a second one on the return leg of a business trip. I was pulled out for questioning – and I work for the goddamn government as a contractor. Other strikes against me? I was flying alone, I had “long hair, beard and an earring” and my complexion was too dark (I am of eastern European descent, but I have been taken for arabic). I am grateful – if it can be said – that this occurred before 9/11.

  5. #5 Blake Stacey
    October 13, 2008

    There’s a wonderful textbook lesson in the False Positive Paradox and Bayes’ Theorem lurking in here.

  6. #6 g347
    October 14, 2008

    Dylan, your lack of objectivity is showing. The remote viewing studies produced a rescued fighter pilot who had been held hostage, intel on a new type of Soviet submarine, and the arrests of suspects in a kidnapping case. But you can believe whatever you want. Even if it’s wrong. And no, I’m not going to let myself get dragged into a thrash on this subject.

    As for data mining and the risks to national security: the major risk, aside from the risk of tyranny, is that of swamping the relevant agencies so they can’t do their real job.

    Consider NSA’s translation backlog that resulted in missing the key intercept about 9/11. Consider it in light of the fact that Bush apparently tasked NSA with broad-spectrum domestic collection in the months preceding the attack, even as he (Bush) was downplaying terrorism entirely. Yes, I’m asserting that Bush putting NSA on a wild goose chase swamped the agency’s translator resources and led to the delay in processing the Al Qaeda intercept.

    Next, consider this instance. FBI asked NSA for raw intercepts (something that is not ordinarily done: NSA releases digests and first-level abstracts, not the raw stuff). NSA opened the floodgates. FBI said “stop! stop!, we can’t deal with all of this stuff!” Useful lesson. Military resources are not appropriate tools for domestic law enforcement.

    Fortunately we’re about to have a smart guy in office. All the good folks who worked for the various agencies that Bush screwed up with his King Midas of Poo touch, will be rejoicing bigtime.