They closely followed the Google research paper describing BigTable — the sweeping database that underpinned many of the Google’s online services, running across tens of thousands of computer servers — but they also went a little further. In rebuilding this massive database, they beefed up the security. After all, this was the NSA.
Like Google, the agency needed a way of storing and retrieving massive amounts of data across an army of servers, but it also needed extra tools for protecting all that data from prying eyes. They added “cell level” software controls that could separate various classifications of data, ensuring that each user could only access the information they were authorized to access. It was a key part of the NSA’s effort to improve the security of its own networks.
But the NSA also saw the database as something that could improve security across the federal government — and beyond. Last September, the agency open sourced its Google mimic, releasing the code as the Accumulo project. It’s a common open source story — except that the Senate Armed Services Committee wants to put the brakes on the project.