Data Mining: What Is It?

Able Danger, which is the DoD program that appears to have culled the names of four of the 9/11 terrorists as early as 2000 from the massive amount of data publicly available was a data mining program.

Data mining is a process by which a computer sifts through massive amounts of data looking for patterns and connections.

The NY Sun article discusses the use of data mining in more detail.

While the "Able Danger" project was little discussed until recently, a broader Pentagon data-mining effort, known originally by the Orwellian name, "Total Information Awareness," was shuttered in 2003 after an outcry from privacy advocates. Some who were critics of that program say the recent developments suggest that the data-intensive technologies now deserve a second look.

"We did dismiss it too quickly," said Sonia Arrison, the director of technology studies at a San Francisco think tank, the Pacific Research Institute. "I was really against TIA when it first came out," she said.

Ms. Arrison said it makes little sense to demand that the government abandon a technology that is being used more and more widely by retailers and others in the private sector. She said the government should move forward with the program but eschew the secrecy that usually surrounds such efforts. "Let's embrace a TIA-type system, but let's have everyone understand how it works," the analyst said. "The technology is really just a tool. It can be used for good or evil. ... You can't put it back in the bottle."

This suggests that the data mining program within the DoD was shut down sometime in 2003. It is not entirely clear that Able Danger was shut down as well, though I have seen it reported elsewhere that the program was shut down some time after 9/11.

Why it was shut down given the apparent success in culling the names of four of the 9/11 hijackers is strange, but not unexpected given the forceful resistance by civil liberties types at the ACLU and other privacy groups who thought that this was a breach of privacy.