Masks That Fool Facial Recognition Technology

Surgical masks have transitioned from playing the role of protective equipment to being a societal norm to becoming a fashion statement within a couple of months due to COVID-19.

As the pandemic engulfed the world, there has been an increased deployment of AI-based solutions to keep the spread in check. Computer vision is being used to identify patients, track their movements and for other contactless precautionary measures.

When it comes to identifying people using facial recognition technology, the algorithms take cues from the landmarks around eyes, nose and mouth. Now since the bottom half of the facial features are hidden by the masks, facial recognition technology is facing new challenges.

To make things even more challenging for the algorithms, people are using masks that can fool these systems.

Bruce MacDonald, a software developer from Canada, has designed a mask that could act as adversarial attacks on the facial recognition models.

Fashion As An Adversarial Attack

via Bruce MacDonald

For this experiment, Bruce used a HOG or Histogram of Gradients, which is a facial detection method that divides a processed input image into cells with gradient orientations before feeding the result into a support vector machine.

For finding an input that corresponded to the most faces detected by dlib, Bruce used simulated annealing with random optimisation. Dlib is a modern C++ toolkit containing machine learning algorithms and tools.

Whereas, simulated annealing is a technique that is used to find the best solution for either a global minimum or maximum, without having to check every single possible solution that exists. 

For optimisation, mlrose, a Python package was used. This package is typically used for applying common randomised optimisation and search algorithms to a range of different optimisation problems, over both discrete- and continuous-valued parameter spaces.

Here is a snippet of code:

fitness = mlrose.CustomFitness(detected_max)

problem = mlrose.DiscreteOpt(length=24, fitness_fn=fitness,maximize=True, max_val=scale_factor)

<<First the fitness and the initial state are set and then ‘mlrose’ was configured for simulated annealing. Then the inputs are assigned, and this step is run repeatedly until an optimal result is found as shown in the picture above.>>

schedule = mlrose.ExpDecay()

best_state, max_faces = mlrose.simulated_annealing(problem, schedule=schedule, max_attempts=10, max_iters=1000,init_state=initial_state, random_state=1)



Check the full code here.

Future Direction

The earliest large scale use of masks that fooled surveillance systems was seen during the Hong Kong protests last year. The use of these fashionable masks exposes a couple of things. First, that the public has realised the extent of surveillance, and secondly, the computer vision models have a long way to go. Imagine a scenario where a self-driving car comes across a signal or an advertisement that has these confusing imagery. The consequences can be fatal.

There have been works already to identify people even with masks. A Chinese company, Hawang, claims that it can identify everyone in a crowd with great accuracy, and it has also been deployed by the Chinese Ministry of Public Security to screen people and cross-reference images with its own database to identify and track people.

As cities became centres of mass surveillance, the yearning for anonymity is on the rise. So far, the argument for surveillance has been to make the streets safer. People have been sceptical about this, and many regulations have been passed against the inappropriate use of facial recognition, and now, the pandemic has given a touch of benevolence and made these systems mainstream.

Download our Mobile App

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Subscribe to our newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day.
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Our Upcoming Events

15th June | Bangalore

Future Ready | Lead the AI Era Summit

15th June | Online

Building LLM powered applications using LangChain

17th June | Online

Mastering LangChain: A Hands-on Workshop for Building Generative AI Applications

20th June | Bangalore

Women in Data Science (WiDS) by Intuit India

Jun 23, 2023 | Bangalore

MachineCon 2023 India

26th June | Online

Accelerating inference for every workload with TensorRT

MachineCon 2023 USA

Jul 21, 2023 | New York

Cypher 2023

Oct 11-13, 2023 | Bangalore

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox