How to Paint a random natural image using Paint Transformer?

Nowadays, AI is everywhere and boosting productivity and performance of all the industries where it is being applied. Similarly in art industries, transforming a realistic image in an artistic way is now in trend and many social platforms have this as an integral part. In context to it, in this article, we are going to discuss how neural networks are being used in the art industry and we will take a look at NN-based architecture called Paint Transformer which results in human-crafted painting images of given natural images. Following are the main points listed that are to be discussed in this article.

Neural networks in art industry
How does Paint Transformer work?
Implementing Paint Transformer

First, we will discuss how neural networks are used in art.

Neural networks in art industry

Painting has been a fantastic way for humans to preserve what they experience or even how they picture the world since ancient times. Painting has long been recognized as requiring professional knowledge/skills and is difficult for a layman. Computer-aided art production fills a big part of this void, allowing many of us to produce our own artistic works.

Neural networks are used by artists to augment or enhance their artistic efforts, as well as to generate totally new works of art. Adobe's Wetbrush technology is eroding the border between digital art and painting, despite the fact that digital art has long been recognized as a unique medium.

Creating an effect that appears like paint in a digital medium has previously been challenging, but Wetbrush employs algorithms and a physics simulation engine to make digital brushstrokes that have the feel and colour of oil paint. Wetbrush images can also be printed in three dimensions, simulating the effects of natural lighting in an oil painting.

Natural images can be transformed into artistic images via image models such as style transfer or image-to-image translation. Image creation is typically formulated as an optimization process in pixel space or a feed-forward pixel-wise image mapping with neural networks in these previous methods.

Humans, on the other hand, create paintings through a stroke-by-stroke procedure, using brushes ranging from coarse to fine, in contrast to the pixel-wise operations of neural networks. Making machines imitate such a stroke-by-stroke process has a lot of potential for producing more authentic and human-like paintings.

How does paint Transformer Work?

In order to simulate the natural behaviour of human paintings, Songhua Liu et al have proposed a neural network-based architecture that obtains a painted image for a given realistic image. In this section, we are going to discuss the architecture and in the next section, we’ll see how we can implement it.

This model predicts a set of strokes and then renders them on the initial canvas to minimize the difference between the rendered image and the target natural image is given an initial canvas and a target natural image. At K coarse-to-fine scales, the procedure is repeated. The output of the previous scale is the initial canvas at each scale, and this is referred to as a feed-forward stroke set prediction problem.

a novel self-training pipeline that makes use of synthesized stroke images To be more specific, it first generates a background canvas image with some randomly sampled strokes; then, it generates a target image by randomly sampling a foreground stroke set and rendering it on a canvas image.

Thus, the stroke predictor's training goal is to predict the foreground stroke set and minimize the differences between the synthesized canvas image and the target image, with optimization occurring at both the stroke and pixel levels.

Once trained, this self-trained Paint Transformer exhibits excellent generalization and can work with arbitrary natural images. The sample result shown below demonstrates how this feed-forward method can produce higher-quality paintings at a lower cost than existing methods.

Implementing Paint Transformer

In this section we take a closer look at the implementation, the researcher’s team has made it in a way that we implement the images like above in just 2-3 steps. Before implementation let’s again briefly summarize the procedure.

It is a step-by-step method for forecasting strokes. It predicts multiple strokes in parallel at each step in a feed-forward fashion to minimize the difference between the current canvas and the original target image.

Paint Transformer is made up of two modules to accomplish this: Stroke Predictor and Stroke Renderer. Given a target image and an intermediate canvas image, Stroke Predictor generates a set of parameters to determine the current stroke set. The Stroke Renderer then generates the stroke image for each stroke and plots it onto the canvas, resulting in the image at the output path.

Now, let’s move towards implementation. First, we’ll clone the Paint Transformers repository and set the working directory accordingly.

!git clone https://github.com/lucabeetz/PaintTransformer.git
%cd PaintTransformer/inference

That’s it, next we can do a painting for any random natural image using the run_inferance module from the inference package.

from inference import run_inference
 
input_img = '/content/jonathan-riley-VW8MUbHyxCU-unsplash.jpg'
 
run_inference(input_path=input_img,
              model_path='model.pth',
              output_dir='/content', 
              need_animation=True,  
              resize_h=None,        
              resize_w=None,
              serial=True)

Inside the run_inference first, we pass the path for the target image and second is a path for output results, and serial is set to be True if we want animated results the rest of attributes are self-explanatory. Below is our target image.

Post inference we can create the animated results of those strokes which will show how has painting formed and that can be done as,

import glob
from PIL import Image
 
# Set to dir with output images
in_dir = '/content/marine drive/*.jpg'
out_path = '/content/marine drive.gif'
 
img, *imgs = [Image.open(f) for f in sorted(glob.glob(in_dir))]
img.save(fp=out_path, format='GIF', append_images=imgs,
          save_all=True, duration=100, loop=0)

And below are the results,

The animated result,

Final words

Through this article, we have discussed how nowadays the art industry is influenced by cutting-edge technologies like AI. We briefly discussed neural paintings and different architects that can be used in art. Mainly we focused on the Paint Transformer which is a neural painting architecture and practically we saw its result.