Masks That Fool Facial Recognition Technology

Surgical masks have transitioned from playing the role of protective equipment to being a societal norm to becoming a fashion statement within a couple of months due to COVID-19.

As the pandemic engulfed the world, there has been an increased deployment of AI-based solutions to keep the spread in check. Computer vision is being used to identify patients, track their movements and for other contactless precautionary measures.

When it comes to identifying people using facial recognition technology, the algorithms take cues from the landmarks around eyes, nose and mouth. Now since the bottom half of the facial features are hidden by the masks, facial recognition technology is facing new challenges.

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

To make things even more challenging for the algorithms, people are using masks that can fool these systems.

Bruce MacDonald, a software developer from Canada, has designed a mask that could act as adversarial attacks on the facial recognition models.

Fashion As An Adversarial Attack

via Bruce MacDonald

For this experiment, Bruce used a HOG or Histogram of Gradients, which is a facial detection method that divides a processed input image into cells with gradient orientations before feeding the result into a support vector machine.

For finding an input that corresponded to the most faces detected by dlib, Bruce used simulated annealing with random optimisation. Dlib is a modern C++ toolkit containing machine learning algorithms and tools.

Whereas, simulated annealing is a technique that is used to find the best solution for either a global minimum or maximum, without having to check every single possible solution that exists. 

For optimisation, mlrose, a Python package was used. This package is typically used for applying common randomised optimisation and search algorithms to a range of different optimisation problems, over both discrete- and continuous-valued parameter spaces.

Here is a snippet of code:

fitness = mlrose.CustomFitness(detected_max)

problem = mlrose.DiscreteOpt(length=24, fitness_fn=fitness,maximize=True, max_val=scale_factor)

<<First the fitness and the initial state are set and then ‘mlrose’ was configured for simulated annealing. Then the inputs are assigned, and this step is run repeatedly until an optimal result is found as shown in the picture above.>>

schedule = mlrose.ExpDecay()

best_state, max_faces = mlrose.simulated_annealing(problem, schedule=schedule, max_attempts=10, max_iters=1000,init_state=initial_state, random_state=1)

get_img_from_state(best_state)

len(detect_faces(cv2.imread(OUTPUT))))

Check the full code here.

Future Direction

The earliest large scale use of masks that fooled surveillance systems was seen during the Hong Kong protests last year. The use of these fashionable masks exposes a couple of things. First, that the public has realised the extent of surveillance, and secondly, the computer vision models have a long way to go. Imagine a scenario where a self-driving car comes across a signal or an advertisement that has these confusing imagery. The consequences can be fatal.

There have been works already to identify people even with masks. A Chinese company, Hawang, claims that it can identify everyone in a crowd with great accuracy, and it has also been deployed by the Chinese Ministry of Public Security to screen people and cross-reference images with its own database to identify and track people.

As cities became centres of mass surveillance, the yearning for anonymity is on the rise. So far, the argument for surveillance has been to make the streets safer. People have been sceptical about this, and many regulations have been passed against the inappropriate use of facial recognition, and now, the pandemic has given a touch of benevolence and made these systems mainstream.

More Great AIM Stories

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM