Masks That Fool Facial Recognition Technology

Surgical masks have transitioned from playing the role of protective equipment to being a societal norm to becoming a fashion statement within a couple of months due to COVID-19.

As the pandemic engulfed the world, there has been an increased deployment of AI-based solutions to keep the spread in check. Computer vision is being used to identify patients, track their movements and for other contactless precautionary measures.


Sign up for your weekly dose of what's up in emerging technology.

When it comes to identifying people using facial recognition technology, the algorithms take cues from the landmarks around eyes, nose and mouth. Now since the bottom half of the facial features are hidden by the masks, facial recognition technology is facing new challenges.

To make things even more challenging for the algorithms, people are using masks that can fool these systems.

Bruce MacDonald, a software developer from Canada, has designed a mask that could act as adversarial attacks on the facial recognition models.

Fashion As An Adversarial Attack

via Bruce MacDonald

For this experiment, Bruce used a HOG or Histogram of Gradients, which is a facial detection method that divides a processed input image into cells with gradient orientations before feeding the result into a support vector machine.

For finding an input that corresponded to the most faces detected by dlib, Bruce used simulated annealing with random optimisation. Dlib is a modern C++ toolkit containing machine learning algorithms and tools.

Whereas, simulated annealing is a technique that is used to find the best solution for either a global minimum or maximum, without having to check every single possible solution that exists. 

For optimisation, mlrose, a Python package was used. This package is typically used for applying common randomised optimisation and search algorithms to a range of different optimisation problems, over both discrete- and continuous-valued parameter spaces.

Here is a snippet of code:

fitness = mlrose.CustomFitness(detected_max)

problem = mlrose.DiscreteOpt(length=24, fitness_fn=fitness,maximize=True, max_val=scale_factor)

<<First the fitness and the initial state are set and then ‘mlrose’ was configured for simulated annealing. Then the inputs are assigned, and this step is run repeatedly until an optimal result is found as shown in the picture above.>>

schedule = mlrose.ExpDecay()

best_state, max_faces = mlrose.simulated_annealing(problem, schedule=schedule, max_attempts=10, max_iters=1000,init_state=initial_state, random_state=1)



Check the full code here.

Future Direction

The earliest large scale use of masks that fooled surveillance systems was seen during the Hong Kong protests last year. The use of these fashionable masks exposes a couple of things. First, that the public has realised the extent of surveillance, and secondly, the computer vision models have a long way to go. Imagine a scenario where a self-driving car comes across a signal or an advertisement that has these confusing imagery. The consequences can be fatal.

There have been works already to identify people even with masks. A Chinese company, Hawang, claims that it can identify everyone in a crowd with great accuracy, and it has also been deployed by the Chinese Ministry of Public Security to screen people and cross-reference images with its own database to identify and track people.

As cities became centres of mass surveillance, the yearning for anonymity is on the rise. So far, the argument for surveillance has been to make the streets safer. People have been sceptical about this, and many regulations have been passed against the inappropriate use of facial recognition, and now, the pandemic has given a touch of benevolence and made these systems mainstream.

More Great AIM Stories

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Our Upcoming Events

Masterclass, Virtual
How to achieve real-time AI inference on your CPU
7th Jul

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, Virtual
Deep Learning DevCon 2022
29th Oct

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM

What is Direct to Mobile technology?

The Department of Technology is conducting a feasibility study of a spectrum band for offering broadcast services directly to users’ smartphones.