Going Open Source

The Sunday Times Magazine has a lengthy piece about open source intel gathering and the problems that the intel community has in dealing with sharing intel between agencies, and the institutional hurdles.

There are many interesting points raised in the course of the article:

A profusion of spy blogs and wikis would have another, perhaps even more beneficial impact. It would drastically improve the search engines of Intelink. In a paper that won an honorable mention in the Galileo Awards, Matthew Burton — the young former D.I.A. analyst — made this case. He pointed out that the best Internet search engines, including Google, all use “link analysis” to measure the authority of documents. When you type the search “Afghanistan” into Google, it finds every page that includes that word. Then it ranks the pages in part by how many links point to the page — based on the idea that if many bloggers and sites have linked to a page, it must be more useful than others. To do its job well, Google relies on the links that millions of individuals post online every day.

This, Burton pointed out, is precisely the problem with Intelink. It has no links, no social information to help sort out which intel is significant and which isn’t. When an analyst’s report is posted online, it does not include links to other reports, even ones it cites. There’s no easy way for agents to link to a report or post a comment about it. Searching Intelink thus resembles searching the Internet before blogs and Google came along — a lot of disconnected information, hard to sort through. If spies were encouraged to blog on Intelink, Burton reasoned, their profuse linking could mend that situation. “

So, not only is there a problem with searching the volumes of material that are produced, but no good way to disseminate the material in a timely fashion. That's where wikis come into play - the intel version is known as Intellipedia.

Yet Intellipedia also courts the many dangers of wikis — including the possibility of error. What’s to stop analysts from posting assertions that turn out to be false? Fingar admits this will undoubtedly happen. But if there are enough people looking at an entry, he says, there will always be someone to catch any grave mistakes. Rasmussen notes that though there is often strong disagreement and debate on Intellipedia, it has not yet succumbed to the sort of vandalism that often plagues Wikipedia pages, including the posting of outright lies. This is partly because, unlike with Wikipedia, Intellipedia contributors are not anonymous. Whatever an analyst writes on Intellipedia can be traced to him. “If you demonstrate you’ve got something to contribute, hey, the expectation is you’re a valued member,” Fingar said. “You demonstrate you’re an idiot, that becomes known, too.”

Gee, what's stopped the CIA and other intel agencies from getting it wrong about the Soviet Union, North Korea, Iran, Iraq, Saudi Arabia, and hundreds of other issues large and small? It wasn't the technology used by the agencies, but rather the compilation of materials and analysis. Maybe this new technology and approach might be something new and worthwhile as more eyeballs can see the information from more sources and arrive at a more accurate snapshot.