A Fun Project On Building A Face-Swapping Application With OpenCV

Most of you would have used or seen many filters that let you swap your face with your friends, celebrities or even animals. These filters are very popular in social media platforms such as Instagram, Snapchat and face app. Have you ever wondered how this is done? With Python and OpenCV it is actually very simple to build this application. The concept behind this is to detect certain points on the face and then replace it with the swapping image. Though simple, it does involve a lot of factors like lighting, facial structure and camera angle. To avoid these complications, we will implement this on two images of celebrities. 

In this article, we will implement a face-swapping technique for two images of celebrities using OpenCV and python.

Steps used for this project:

1. Taking two images – one as the source and another as a destination.
2. Using the dlib landmark detector on both these images. 
3. Joining the dots in the landmark detector to form triangles. 
4. Extracting these triangles
5. Placing the source image on the destination
6. Smoothening the face

Image selection

You can select any two images of your choice. It could be your image and your friend, it could be two celebrity images as well. I have selected two images of Shah Rukh Khan  and Katrina Kaif. Both the images are front-facing and are well lit.

Source:

Destination:

Now let us read these images using OpenCV code.

import cv2
import numpy as np
import dlib
import time
source_image = cv2.imread("srk.jpg")
source_image_gray = cv2.cvtColor(source_image, cv2.COLOR_BGR2GRAY)
dest_image = cv2.imread("katrina-kaif.jpg")
dest_image_gray = cv2.cvtColor(dest_image, cv2.COLOR_BGR2GRAY)
mask = np.zeros_like(source_image_gray)

Using the dlib landmark detector on the images

Dlib is a python library that provides us with landmark detectors to detect important facial landmarks. These 68 points are important to identify the different features in both faces. To download this use this link.

Once we have the 68 points shape predictor downloaded, let us apply them to the first face. 

Then we will use the convexhull to detect the faces after using the landmark detector. 

land_detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")
source_face = land_detector(source_image_gray)
for face in source_face:
    landmarks = predictor(source_image_gray, face)
    points = []
    for n in range(0, 68):
        x = landmarks.part(n).x
        y = landmarks.part(n).y
        points.append((x, y))
       face_point = np.array(points, np.int32)
       convexhull = cv2.convexHull(face_point)

These will create 68 points on the face as shown below.

For destination face:

dest_face = land_detector(dest_image_gray)
for face in dest_face:
    landmarks = predictor(dest_image_gray, face)
    points2 = []
    for n in range(0, 68):
        x = landmarks.part(n).x
        y = landmarks.part(n).y
        points2.append((x, y))

Joining the dots in the landmark detector to form triangles for the source image.

To cut a portion of the face and fit it to the other we need to analyse the size and perspective of both the images. To do this, we will split the entire face into smaller triangles by joining the landmarks so that the originality of the image is not lost and it becomes easier to swap the triangles with the destination image. This entire process is called Delaunay triangulation. 

Note: Since we do this with respect to the facial landmark of the source the following code has to be put inside the for loop of source image. 

rectangle = cv2.boundingRect(convexhull)
divide_2d = cv2.Subdiv2D(rectangle)
divide_2d.insert(landmarks_points)
split_triangle = divide_2d.getTriangleList()
split_triangle = np.array(split_triangle, dtype=np.int32)
cv2.fillConvexPoly(mask, convexhull, 255)
face_image_1 = cv2.bitwise_and(source_image, source_image, mask=mask)
face_points2 = np.array(points2, np.int32)
convexhull2 = cv2.convexHull(face_points2)
def extract_index_nparray(nparray):
    index = None
    for num in nparray[0]:
        index = num
        break
    return index
    join_indexes = []
    for edge in triangles:
        first = (edge[0], edge[1])
        second = (edge[2], edge[3])
        third = (edge[4], edge[5])
index_edge1 = np.where((points == first).all(axis=1))
        index_edge1 = extract_index_nparray(index_edge1)
        index_edge2 = np.where((points == pt2).all(axis=1))
        index_edge2 = extract_index_nparray(index_edge2)
        index_edge3 = np.where((points == pt3).all(axis=1))
        index_edge3 = extract_index_nparray(index_edge3)
        if index_edge1 is not None and index_edge2 is not None and index_edge3 is not None:
            triangle = [index_edge1, index_edge2, index_edge3]
            join_indexes.append(triangle)

 Now that we have the triangles for the source image we need to make sure the same space can be extracted from the destination so that the overlapping can be done smoothly. 

To do this, we follow a slightly different approach to the destination image. 

The destination image needs to have the same patterns of triangles as the source. 

To do this we will create masks for the images as follows. 

First, let us create the source image. We are trying to match the patterns of first and second images here. 

source_mask = np.zeros_like(source_image_gray)
new_face = np.zeros_like(dest_image)
for index in indexes_triangles:
    tri_one = points[index[0]]
    tri_two = points[index[1]]
    tri_three = points[index[2]]
    triangle1 = np.array([tri_one, tri_two, tri_three], np.int32)
    first_rect = cv2.boundingRect(triangle1)
    (x, y, w, h) = first_rect
    cropped_triangle = source_image[y: y + h, x: x + w]
    cropped_tr1_mask = np.zeros((h, w), np.uint8)
    pts = np.array([[tri_one[0] - x, tri_one[1] - y],
                       [tri_two[0] - x, tri_two[1] - y],
                       [tri_three[0] - x, tri_three[1] - y]], np.int32)
    cv2.fillConvexPoly(cropped_tr1_mask, pts, 255)
    cv2.line(source_mask, tri_one, tri_two, 255)
    cv2.line(source_mask, tri_two, tri_three, 255)
    cv2.line(source_mask, tri_one, tri_three, 255)

As you can see we have created a mask of the image.

Once we have the mask for the source image and the triangle points we can do the same for the destination as well. But for the destination, we need to crop out the region corresponding to the source mask. We can do this as shown below. 

tri2_one = points2[index[0]]
    tri2_two = points2[index[1]]
    tri2_three = points2[index[2]]
    triangle2 = np.array([tri2_one, tri2_two, tri2_three], np.int32)
    second_rect = cv2.boundingRect(triangle2)
    (x, y, w, h) = second_rect
    cropped = np.zeros((h, w), np.uint8)
    points2 = np.array([[tri2_one[0] - x, tri2_one[1] - y],
                        [tri2_two[0] - x, tri2_two[1] - y],
                        [tri2_three[0] - x, tri2_three[1] - y]], np.int32)
    cv2.fillConvexPoly(cropped, points2, 255)

This is the destination image triangles. 

Extracting these triangles 

Once we have the triangles in source and destination the next step is to extract them from the source image. But we also need to take the coordinates of the destination triangles so that the sizes of the two can match. This technique is also called warping. 

    points = np.float32(points)
    points2 = np.float32(points2)
    transform = cv2.getAffineTransform(points, points2)
    warping = cv2.warpAffine(cropped_triangle, transform, (w, h))
    warping = cv2.bitwise_and(warping, warping, mask=cropped)

Placing the source image on the destination

Now, we can reconstruct the destination image and start placing the source image on the destination one. First we will make some alterations in the destination face. Then, we will make sure the lines created do not appear in the final output.

ht, wt, filters = dest_image.shape
dest_face = np.zeros((ht, wt, filters), np.uint8)
 facial_area = dest_face[y: y + h, x: x + w]
    facial_area_gray = cv2.cvtColor(facial_area, cv2.COLOR_BGR2GRAY)
    _,triangle_mask = cv2.threshold(facial_area_gray, 1, 255, cv2.THRESH_BINARY_INV)
    warping = cv2.bitwise_and(warping, warping, mask=triangle_mask)
facial_area = cv2.add(facial_area, warping)
    dest_face[y: y + h, x: x + w] = facial_area

Finally, we will place the source image on the destination

final_mask = np.zeros_like(dest_image_gray)
head_mask = cv2.fillConvexPoly(final_mask, convexhull2, 255)
final_mask = cv2.bitwise_not(head_mask)
combine = cv2.bitwise_and(dest_image, dest_image, mask=final_mask)
output = cv2.add(combine, dest_face)

Thought the images have been swapped, there is no match in color or smoothness in the swapping. To eliminate this we need to do another process.

Smoothening the face

The final step is to change the colours and to make the swapping look better. To do this OpenCV provides a library called seamless cloning. 

(x, y, w, h) = cv2.boundingRect(convexhull2)
seamless= (int((x + x + w) / 2), int((y + y + h) / 2))
seamlessclone = cv2.seamlessClone(output, dest_image, head_mask, seamless, cv2.NORMAL_CLONE)
cv2.imshow("seamlessclone", seamlessclone)
cv2.waitKey(0)
cv2.destroyAllWindows()

The final output is like this 

You can see that this has some gradient and better fit when compared to the previous output. 

Conclusion

In this article, we saw how to build a face-swapping application with OpenCV and successfully built it on two photos. It is quite simple and interesting to build this and a lot of fun to build as well. You can also check out this for building your own face filters.