Blind face restoration is a process of recovering higher quality faces from lower-quality counterparts. These lower-quality faces may result from degradations such as low-resolution, noise, blur, and compression artefacts. The blind face restoration process generally relies on facial priors like a facial geometry prior or a reference prior for faithful details restoration. However, the applicability of this process is limited in real-life scenarios — low-quality inputs cannot offer accurate geometric prior, and high-quality references are inaccessible.

Generative adversarial networks (GANs) have emerged as a great technique for face restoration. There have been several pieces of research in the area, the latest of which is from scientists at Tencent who have developed Generative Facial Prior-GAN (GFP-GAN). It leverages diverse priors encapsulated in pretrained face GAN for restoration. 

GFP-GAN

In the study “Towards Real-World Blind Face Restoration with Generative Facial Prior”, the authors leveraged GFP for real-world blind face restoration. The facial priors are implicitly encapsulated in pretrained face GAN models like StyleGAN. These face GANs can generate faithful faces with high degrees of variability, providing rich and diverse geometry, facial textures, and colours. This makes it possible to restore facial details and enhance colours jointly.

The challenge here pertains to the incorporation of these generative priors into the restoration process. In the past, studies typically used GAN inversion. In this process, the degraded image is first inverted back to a latent code of pretrained GAN, after which expensive image specific optimisations are performed to reconstruct the images. While the output is visually realistic, they have low fidelity.

GFP-GAN addresses these challenges by offering delicate designs that help balance realness and fidelity in a single forward pass. GFP-GAN includes a degradation removal module and a pretrained face GAN as facial prior. These two modules are connected via direct latent code mapping and many Channel-Split Spatial Feature Transform (CS-SFT) layers in a coarse-to-fine manner. 

Credit: GFP-GAN

The CS-SFT layers perform spatial modulation on features. The left features are directly passed through for information preservation which incorporates generative prior while retaining high fidelity. The team has also introduced facial component loss using local discriminators to enhance perpetual facial details while improving fidelity through identity preserving loss.

Read the full paper here.

Similar Studies

Blind face restoration has emerged as a significant area of research. There have been several attempts to determine the most effective method to do so. 

In the recent CVPR 2021 event, a group of researchers introduced the progressive semantic-aware style transformation framework-GAN (PSFR-GAN) method. This technique restores low-quality face images through semantic aware style transformation, where a pyramid of multi-scale inputs is created first before progressively modulating different scale features from coarse-to-fine fashion.

Another method called the DeblurGAN uses an end-to-end GAN for single image motion deblurring. This method can boost state-of-the-art deblurring efficiency, flexibility, and quality and is based on a relativistic conditional GAN with a double-scale discriminator. The authors claimed that this was the first time Feature Pyramid Network was built into deblurring as a core building block in Deblur-GAN generator (version-2).

A method called mGANprior incorporates well-trained GANs as an effective prior to a variety of image processing tasks. This method employs multiple latent codes to generate multiple feature maps at the intermediate layer of the generator before composing them with adaptive channel importance to recover the input image.

Challenges with Blind Image Restoration

Despite the progress made in improving blind image restoration, inherent challenges are common across these different methods. The GFP-GAN method isn’t immune to them either. As per the authors, while GFP-GAN performs well on most faces across various population groups, a problem arises when the input images are grey-scale. This might cause the output to have a colour bias since the input does not contain sufficient colour information.

Further, if the degradation of real images is severe, the restored facial details by GFP-GAN are twisted with artefacts. The method is also prone to producing unnatural results for very large poses since synthetic degradation and training data distribution are different from those in the real world. Authors argue that this limitation can be overcome by using distributions from real data instead of just depending on synthetic data.

Face restoration is a tricky area, especially in terms of the bias factor. One of the major blunders that happened recently was when a StyleGAN-based algorithm called PULSE upscaled a low-quality picture of Barack Obama to output a high-resolution image — but of a white man. This bias problem is extremely common in machine learning and renders facial recognition algorithms almost useless on non-white and female faces.