AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Brian proffitt readwrite11/21/2023 ![]() If Facebook’s security is good enough to convince the NSA to hire its chief security officer, it’s not unreasonable to assume the social network has something to teach the spy agency about gleaning information about personal relationships with Hadoop, too. This process of making inferences about interests and behavior is likely at least as sophisticated as what the NSA does. Yahoo! may have taken that work and run with it to actually release Hadoop as an open-source project, but Google’s research then and now regularly pushes the industry forward.įacebook, which once claimed in 2011 to have the world’s largest Hadoop cluster at 30 petabytes, uses Hadoop to store data and Hive to analyze billions of pieces of content daily on Facebook, looking for ways to present users with the most relevant content. Google, after all, inspired Hadoop with a research paper years ago that gave the world a peek into MapReduce. With or without Federal investment, Hadoop development is extraordinarily well-funded by private industry in ways that would warm a spy’s heart. ![]() ( Disclosure: In-Q-Tel is also an investor in my company, 10gen.) Private Industry Leads The Wayīut the U.S. In-Q-Tel, the Central Intelligence Agency’s (CIA) investment arm, is an investor in Hadoop vendor Cloudera. government hasn’t been content to sit back and wait for the open-source community to build out Hadoop. So while it may change the Hadoop code to make it more applicable to the NSA’s needs, doing so establishes a fork that takes it beyond the mainline community code, making it harder for these government agencies to leverage the apparently superior efforts of the open-source community. NSA Concedes: Open Source Is Betterīut as The Wall Street Journal points out, the NSA actually turned to Hadoop precisely because it couldn’t out-innovate the open-source community. As ReadWrite‘s Brian Proffitt points out, “the agency quite likely has code in its servers that makes Hadoop, Accumulo and a lot of open source NoSQL technology do tricks that commercial users can only dream of.” Not that the NSA necessarily is using stock Hadoop. (See also Just How Closely Can The NSA Really Watch You?) When an interested 10-year old programmer has access to the same heavy-duty Big Data technology as the budget-rich NSA, we have arrived. This is a big deal, because it suggests that technology has finally become democratized. More probably it’s Hadoop, which is open source and 100% free. ![]() Ironically, however, the best technology available to fight terror likely isn’t a line item on the National Security Agency’s (NSA) $8 to $10 billion budget. has spent hundreds of billions of dollars fighting terrorists.
0 Comments
Read More
Leave a Reply. |