Facial Recognition: Face the Future

EARLY LAST MONTH, a few days before Apple announced the iPhone X with face recognition as its main unlocking tool, the journal Current Science published a paper by researchers of a Spanish University. They had come out with a new algorithm, the software used to match faces in applications. While this is a frequent occurrence in the field, what was somewhat unusual was the faces they had used to conduct the study. These included images of Anupam Kher, late Kannada superstar Rajkumar, Telugu actor Nagarjuna, actress Ramya, the late Malayalam character artist Cochin Haneefa, Anil Kapoor, and others. They were from a selection of over 34,000 faces of 100 movie actors, called the Indian Movie Face Database, which is widely used by researchers in universities here and occasionally abroad.

The database was built a few years ago at the Centre for Visual Information Technology (CVIT) in the International Institute of Information Technology, Hyderabad, as part of a research workshop. Each of the 34,512 faces is annotated by its expression (anger, happiness, sadness, surprise, fear, disgust); angle of the pose; the quality of illumination; occlusion or features that mar the image (like beards or ornaments); the actor’s age; how much make-up the face has; and gender. Vijay Kumar, a fifth-year PhD student at CVIT who was part of the database project, says that the painstaking labelling is necessary. “We started this face recognition database because we wanted to work on the Indian scenario,” he says, “Most of the research is in universities abroad and they have created their own databases from movies, sitcoms and so on. We identified 100 actors, their movies, and then searched for videos in places like YouTube (to get screen grabs of faces). We made sure there is diversity in their appearance. For instance, we did not want only Bollywood actors. We collected from six different languages and we also ensured age variation, an important factor when dealing with face recognition. For example, we collected Amitabh Bachchan’s images from his initial as well as latest movies. We had a team of 10 members; all of us manually identified the faces and gave labels to them. It is used by a lot of researchers in India now.”

For a face recognition study, a database like this is divided into two—a gallery set and a testing set. The former is used while developing the algorithm and then it is tested against the latter and factors like accuracy are measured. For a real-world application, imagine an airport where security agencies have images of potential terrorists and they are then matched through such algorithms against what CCTV cameras capture of all passengers present.

Kumar says that facial recognition technology has evolved a lot. In the earlier days, the biometric community used to focus on important facial regions like the eyes or nose to match patterns. Then it became more complex, like seeing how many edges there are at an angle of 30 or 40 degrees in a given patch, and then comparing data. In the last few years, the focus has moved on to something called ‘deep learning’ in which instead of humans, algorithms observe and decide the most important features and informative patterns to match.

The iPhone X even finds the depth at each point of a face and has infrared images so that it can be unlocked the same way at night. Kumar believes that Apple’s use of face recognition is going to be game-changer even if it’s done in a limited setting where the phone’s only job is to match the owner’s face. “But it will be really challenging to do the same kind of thing in airport or surveillance scenarios,” he says.

He believes that face recognition is just taking off. “In China, they have been using it at ATMs, in various retail stores to identify repeat customers and give specialised offers. They also use it at traffic signals to identify jaywalking offenders,” he says. Recently, there was news that facial recognition devices had been installed to prevent the theft of toilet paper rolls in a religious complex in Beijing. A Washington Post report said: ‘Six dispensers, designed by the Shoulian Zhineng company, were recently placed at the entrance to the restrooms. Those seeking relief must first stare into a computer attached to the machine for three seconds. It records their image before spitting out a two-foot long sheet of tissue paper.’

IN 2012, ADARSH Natarajan, an IIM Bangalore alumnus, was planning a start-up of technology products, and while interacting with his alma mater’s Centre of Public Policy was exposed to a programme run by it for the Karnataka government to make Public-Private Partnership models viable in the state. “As part of this project, I was posed with a challenge to ensure that subsidy leakage happening in the mid-may meal programme would be addressed. After due diligence on the field, I realised that the problem could be addressed only through a non-touch and auditable mechanism to capture the attendance of students in classrooms,” he says. This was the genesis of his Bengaluru-based company, Aindra, which built a product around facial recognition. It was found more appropriate than fingerprint scanning, the alternative, because the latter didn’t work very well in equatorial regions where sweat makes these prints difficult to capture; also, the fingertip ridges and troughs of students working in the fields would get worn out, with the result that scanners could not read them.

A pilot run was done and the product’s effectiveness demonstrated in a government school in Mavinahalli Panchayat in Tumkur, a district adjoining Bengaluru. But the company soon moved away from the government and positioned its product as a tool for other sectors. “We realised that the product was a horizontal one and sector agnostic. Any sector that had economic implications of not being able to track and monitor their employees/people were the ones that we would target as our segments. And that was how we had customers across sectors like microfinance institutions, facilities management, vocational training, et cetera,” he says.

Natarajan says it is hard to put a number on the exact size of the market, but estimates it to be in the range of $200 to $300 million and growing, given that India might be a laggard in adopting new technologies but would sooner or later catch up. Also the emergence of Deep Learning has enabled better accuracy rates which, he says, helps the adoption of this technology by mainstream applications. “I believe we are at an inflection point when it comes to face recognition, a tipping point with regard to the acceptance of this technology,” he says.

Another Indian startup that uses face recognition as a business model is Chennai-based Haliscape. Its founder, Vijay Gnanadesikan, says, “Five years ago, we got into artificial intelligence. We have been creating solutions for fashion, retail, etcetera, using augmented reality. And some of it used to include finding age and gender from a picture. We can actually determine it with a decent accuracy.”

Last year, Haliscape developed its own face recognition product called Facetagr. Gnanadesikan, too, believes the opportunities are immense. “Every place I go to, somebody comes and asks me for a different thing that they want to solve,” he says. A lot of demand for face recognition is from the retail sector. It is a way to keep track of customer profiles. For example, a mall would be interested in the gender, age group and other demographic details of the people coming in. This could help with things like customised advertisements. “Facial recognition is non-intrusive [as a method] to determine who is coming in, [unlike] other biometric systems like fingerprint or iris scanners. You don’t have to actually have people to stand there and give their fingerprints. You can just put a camera and it will know who came in,” he says.

At the beginning of this year, Gnanadesikan was driving to his office when he saw a child begging on the road and something clicked in his mind. He went to his business partner and said that they had this technology and there was this inherent Indian problem of missing children who end up on the streets. “We wanted to create a solution for that and make it user friendly. We decided to build a mobile app,” he says.

They tried to get the government to assist without success. So they started collecting images of both missing children from sites like Trackmissingchild.gov.in and created a huge database. Their app matches faces against this data. They work in coordination with government agencies and get three-four emails daily of missing children. They are also assisting an NGO in Nepal called Maiti, which works against human trafficking. Maiti’s workers get into buses at the border and use Facetagr to check whether there are girls who have been reported missing by their parents.

Vijay Kumar of CVIT is now also working on identifying people in scenarios where the face itself is not clearly visible, using other cues like the person’s hair, clothing, build and posture. “We also created a database from soccer players,” he says. “Soccer broadcast videos usually take shots from a long distance, so the resolution of faces is very small. So [we asked], how do we recognise [people] in such cases? Can we create other body cues?”

Another study on a similar theme brought out the kind of questions that face recognition will pose as it gets more and more prevalent in society. Called ‘Disguised Face Identification (DFI) with Facial KeyPoints using Spatial Fusion Convolutional Network’, the paper showed how it would be possible to recognise people even if their faces are partially covered. Its lead researcher was Amarjot Singh, a final year PhD student in Signal Processing and Communications Laboratory of Cambridge University. The co-authors were two students of National Institute of Technology, Warangal, and a professor of the Indian Institute of Science, Bengaluru.

Singh, who earlier did his BTech from NIT Warangal, says, “The problem of DFI is an extremely challenging problem that is of great interest to law enforcement [agencies], as they can use this technology to identify criminals. This is the primary reason we attempted to solve this problem. In addition, there has been no work in the past towards solving this problem, which further motivated us to have a go at it.”

For the study, they analysed numerous images and videos of crimes as well as protests all over the world to see which parts of the face are usually covered by individuals to disguise themselves. “In the majority of videos, the individuals were either wearing glasses that covered the eyes or wearing a scarf that covered the mouth. Taking this into consideration, the points on the face parts (eyes and lips) were selected which were most likely to be missing when the disguise is put on. Hence, the artificial intelligence from labelled data will learn to predict the key points for the face parts,” he says.

But such an ability for technology could also be misused by authoritarian governments. If protestors in large groups cannot gain anonymity by covering their faces, then states can identify and target them selectively later. Soon after the paper appeared in the public domain, Zeynep Tufekci, a well-known American academic and writer on technology’s impact on politics and society, issued a series of tweets with a link to the paper: ‘ohai a machine learning system that can identify ~69% of protesters who are wearing caps AND scarfs to cover their face…The authors claim the system works about half the time even when people wear glasses… And this is just the beginning; first paper… Historical crossroads. Yes, we can & should nitpick this and all papers but the trend is clear. Ever-increasing new capability that will serve authoritarians well.’

Tufekci’s tweets sparked a debate over the paper. Singh agrees that concerns about DFI being used to violate the privacy of individuals are legitimate. “However, these models can be used to take many criminals off the streets as well. As to making sure that this doesn’t fall into the wrong hands, we will just have to be aware that this technology is available only to the organisations or governments that intend to use it for the good, although this is very difficult to establish,” he says. “To be honest,” he adds, “I am not very sure what the answer to this question is, but these questions will become very interesting once these systems are deployed practically.”

facial featuresidentityinnovationiPhonerecognitionstartuptechnology