The debate has been long on rhetoric and short on cold, hard analysis. But it may be about to improve.
Ever since 11 September 2001, governments, national security & law enforcement around the world have been arguing vigorously for hugely increased collection of information about citizens from disconnected sources and applying data mining to it. Enormous resources have gone into these initiatives and laws protecting citizen rights have been compromised to allow them. They have been equally vehemently opposed by civil liberties and privacy advocate interests.
The debate has been long on rhetoric and short on cold, hard analysis.
At last, the debate is beginning to change. It took a significant step forward on 7 October with the release of a report by the US National Academy of Sciences titled Protecting individual privacy in the struggle against terrorists. The report was funded in part by the US Department of Homeland Security and the US National Science Foundation. These are reputable researchers funded from reputable sources.
The report rather bluntly states that explosive increases in data mining have been a waste of time and resources. Hopefully it marks the beginning of the end of a rather frightening era of this form of data surveillance by government.
The Center for Information Policy Leadership (CIPL) summarises the report into 4 major points:
1. There is little evidence that "data mining" works as a tool to detect or prevent terrorism, and lots of reason to think that it is unlikely to ever work for those purposes. There are many practical problems-bad and unstructured data loom especially large-but the major obstacle is that data mining depends on having lots of identified patterns to work with in order to make predictions. Commercial data mining, for example, for fraud prevention and marketing, depends on observing millions or tens of millions of transactions to be able to recognize statistically relevant linkages or patterns. Fortunately, we have very few patterns of terrorist behavior to work with, and terrorists (unlike most consumers) are working hard to mask their transactions, so the "promise" of data mining as a counterterrorism tool seems unlikely to be realized.
2. There is similarly little evidence that behavioral or biometric monitoring works to detect or prevent terrorist acts. In fact, the committee found so little evidence of any success with behavioral or biometric monitoring that it could not reach consensus on any role that such techniques might play in counterterrorism.