Largest biometric database

India’s Unique Identification project (also known as the Aadhar program) has been accompanied by controversial voices from privacy and security and other aspects. It completed the collection of demographic and biometric data earlier this week. The total of more than 500 million Indian residents - is currently the world's largest biometric identification project of the same size.

The latest developments in the Aadhar project have also raised concerns about its approach to capturing, storing, and managing data, especially the role of MongoDB, a US startup.

MongoDB is a NoSQL database startup that last year raised funds from the CIA-funded In-Q-Tel organization. In-Q-Tel is an independent, non-profit capital agency backed by the CIA and some other US intelligence agencies.

Over the past few days, several Indian media reports quoted the opinions of political parties and activists in the country, and suspected that Aadhar’s private data was misappropriated and pointed to the co-founder of Infosys, the project’s head, Nandan Nilekani.

There are also some articles that include MongoDB as a target of criticism.

Governments around the world are increasingly alert to the National Security Agency (NSA) eavesdropping. Anything that has nothing to do with the U.S. government intelligence agency will make a lot of noise. Not only that, because India’s general election will take place soon, the voice of the country’s political opinions will reach an unprecedented level.

The timing of such accusations cannot come any worse than this, at least for this ambitious identification project. Aadhar is waiting for the passage of a congressional bill that will be fully constitutionally recognized this year.

The author visited the Aadhar project office in Bangalore. Truthfully, according to the staff who introduced the information to me, although it was alleged that the large-scale contract contained content to share data with MongoDB, Aadhar actually used MongoDB. Open source code does not touch sensitive data. The meeting also had the opportunity to understand how the largest biometric database on Earth currently operates and how to deal with security and privacy concerns.

Not only that, the Unique Identification Authority of India refuted allegations of sharing information on Indian national data with any US agency.

What does Aadhar mean for India?

The first thing to understand is to talk about Aadhar's context. What does this project mean for a country like India? The country’s more than 500 million people do not have any formal identification (ID) or similar proofs, which has led to many other problems, such as the inability to receive government subsidies, register bank accounts, apply for loans, and obtain driving licenses. The Aadhar database project is currently recording at a rate of 1 million Indian nationals each day. It is expected that approximately 1.2 billion people will be registered by the end of next year, which will become the largest biometric database on the planet.

The greatest advantage of getting a 12-digit length Aadhar code is that the government can link bank accounts with the poor, make direct bank transfers with direct cash equity and other subsidies. At present, there are already nearly 40 million bank accounts in India matching the Aadhar data.

According to a report by CLSA, a market research organization, more than 40% of the 250 billion U.S. dollar subsidies and other national treatment of the Indian government are targeted at the country’s poor population, but they will be wasted in government corruption in the coming years. The Aadhar plan can remove the middle of the process and transfer it directly to people who need government subsidies to curb corruption in this way.

But think tanks and activists, including the Centre for Internet & Society in Bangalore, are always skeptical about privacy issues and even question the effectiveness of the entire project.

Deeper into the world's largest biometric database

The author tried to meet with Aadhar project officials on various sides to understand the security issues, the current progress, and their reaction to criticizing the use of MongoDB technology.

On Friday, Aadhar finally agreed to meet with me at the headquarters in the southern suburbs of Bangalore. Intel and Cisco's headquarters in India are also located in the area. From a physical point of view, the Aadhar Technology Center, which stores all Indian National Data (currently a total of 5 petabytes), is not at all like a government building - it is easy to assume that it is one of the nearby Intel or Cisco office buildings.

Inside, I came to a room with a dozen TV screens in the center. Several young engineers in their twenties were sitting in front of each other excitedly and tapping on their respective computer keyboards to check the storage of data packages. Information, the entire scene is very much like an advanced control center. The TV screens they watched showed records of these data packages (about 5MB each), starting from the entry of 30,000 entry centers across the country and undergoing at least three information verification processes. The verification process package performs a repetitive inspection of each file to ensure that the same person does not generate Aadhar numbers twice.

In other words, every time a new data file is created, a "deduplication" test should be run against all existing files. At present, this figure has exceeded 500 million.

Former Intel engineer Srikanth Nadhamuni helped design Aadhar's technology platform in September 2010, which is currently running at the Khosla lab in Bangalore. He told me that these packets are processed through 2048-bit encrypted storage and trigger a self-destruction function when there is an unauthorized call attempt.

About the criticism of MongoDB

So why did Aadhar start working with MongoDB? Will this kind of cooperation continue?

Sudhir Narayana, assistant director general of the Aadhar Technology Center, said that MongoDB is just one of several products initially selected for data retrieval. Others include MySQL, Hadoop, and HBase. Unlike MySQL, which can only store population data, MongoDB can also store images.

But later Aadhar gradually shifted most of the database work to the MySQL platform because they realized that MongoDB could not handle large-scale data, that is, millions of data packages.

Currently they are already using "databasesing" technology: storing data packages on different machines ensures that the system does not crash when the amount of data increases.

This approach helped Aadhar reduce its dependency on MongoDB and instead use MySQL to store most of its data.

Ashoharlwai, deputy director of the Aadhar Technical Center, told me that MongoDB cannot call any biometric data.

"We believe that the use of open source technology can avoid over-reliance on a certain supplier, but this does not mean that we compromise in any way in terms of security," Ashokdallwai said.

A MongoDB spokesperson interviewed us and suggested that we read the In-Q-Tel investment statement on the company's website.

What's more, the UIDAI India started using MongoDB's open source software technology long before the startup got investment from In-Q-Tel. According to data from Crunchbase, MongoDB only raised a total of US$7.7 million from Red Hat, Intel Capital and In-Q-Tel in 2012.

What is the outlook for Aadhar?

Regardless of all the controversies, Aadhar will complete the goal of entering more than 1.2 billion Indian national data in 2014, and the total number of databases will reach 15 petabytes. The project is currently progressing at a rate of 1 million people per day. From next year onwards, it will achieve a speed of about 2 million people a day, and the remaining 700 million people will be included in this database system.