Masks That Fool Facial Recognition Technology

Surgical masks have transitioned from playing the role of protective equipment to being a societal norm to becoming a fashion statement within a couple of months due to COVID-19.

As the pandemic engulfed the world, there has been an increased deployment of AI-based solutions to keep the spread in check. Computer vision is being used to identify patients, track their movements and for other contactless precautionary measures.

When it comes to identifying people using facial recognition technology, the algorithms take cues from the landmarks around eyes, nose and mouth. Now since the bottom half of the facial features are hidden by the masks, facial recognition technology is facing new challenges.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

To make things even more challenging for the algorithms, people are using masks that can fool these systems.

Bruce MacDonald, a software developer from Canada, has designed a mask that could act as adversarial attacks on the facial recognition models.

Fashion As An Adversarial Attack

via Bruce MacDonald

For this experiment, Bruce used a HOG or Histogram of Gradients, which is a facial detection method that divides a processed input image into cells with gradient orientations before feeding the result into a support vector machine.

For finding an input that corresponded to the most faces detected by dlib, Bruce used simulated annealing with random optimisation. Dlib is a modern C++ toolkit containing machine learning algorithms and tools.

Whereas, simulated annealing is a technique that is used to find the best solution for either a global minimum or maximum, without having to check every single possible solution that exists. 

For optimisation, mlrose, a Python package was used. This package is typically used for applying common randomised optimisation and search algorithms to a range of different optimisation problems, over both discrete- and continuous-valued parameter spaces.

Here is a snippet of code:

fitness = mlrose.CustomFitness(detected_max)

problem = mlrose.DiscreteOpt(length=24, fitness_fn=fitness,maximize=True, max_val=scale_factor)

<<First the fitness and the initial state are set and then ‘mlrose’ was configured for simulated annealing. Then the inputs are assigned, and this step is run repeatedly until an optimal result is found as shown in the picture above.>>

schedule = mlrose.ExpDecay()

best_state, max_faces = mlrose.simulated_annealing(problem, schedule=schedule, max_attempts=10, max_iters=1000,init_state=initial_state, random_state=1)

get_img_from_state(best_state)

len(detect_faces(cv2.imread(OUTPUT))))

Check the full code here.

Future Direction

The earliest large scale use of masks that fooled surveillance systems was seen during the Hong Kong protests last year. The use of these fashionable masks exposes a couple of things. First, that the public has realised the extent of surveillance, and secondly, the computer vision models have a long way to go. Imagine a scenario where a self-driving car comes across a signal or an advertisement that has these confusing imagery. The consequences can be fatal.

There have been works already to identify people even with masks. A Chinese company, Hawang, claims that it can identify everyone in a crowd with great accuracy, and it has also been deployed by the Chinese Ministry of Public Security to screen people and cross-reference images with its own database to identify and track people.

As cities became centres of mass surveillance, the yearning for anonymity is on the rise. So far, the argument for surveillance has been to make the streets safer. People have been sceptical about this, and many regulations have been passed against the inappropriate use of facial recognition, and now, the pandemic has given a touch of benevolence and made these systems mainstream.

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox
MOST POPULAR

Subscribe to Our Newsletter

The Belamy, our weekly Newsletter is a rage. Just enter your email below.