Machine Learning algorithms for image processing have evolved at a tremendous pace. They can now help in the reconstruction of objects in ambiguous images, colouring old videos, detecting the depth in moving videos and much more.
One recurring theme in all these machine vision methods is teaching the model to identify patterns in images. The success of these models has a wide range of applications. Training an algorithm to differentiate between apples and oranges can eventually be used in something as grave as a cancer diagnosis or to unlock the hidden links of renaissance art.
In one such exciting endeavour of bridging fairly distant worlds of art and computational sciences, Tomas Jenicek and Ondrej of Czech Technical University in Prague propose an approach that links artworks through a pose similarity of human figures depicted in images.
Their work focuses on the discovery of a composition transfer among artworks containing humans using an image retrieval system.
Connecting The Dots
A physical link occurs between different machine copies of the same artwork. This appears in museum and gallery digitised collections where there are photographs of the same painting with varying detail. Using visual recognition techniques on artworks is itself challenging.
The proposed pose-based retrieval system of planar visual art consists of two steps – pose matching and geometric verification.
Poses are detected with a state-of-the-art pose detector – OpenPose.
OpenPose uses a multi-stage CNN architecture followed by a part association step to output 2D pose estimations for humans in the image. The output of the CNN is iteratively refined in each stage.
For the first half of the architecture, the output is a part affinity field which is a 2D vector field encoding the degree of association between parts of the image. The output of the second half is a set of confidence maps, one for each body part.
The dataset used for the experiment consists of a set of paintings with repeating motifs and for some paintings, also associated drawings and engravings. It was assembled by people with an art history background as a set of artworks, mainly from renaissance and baroque, that is known to contain composition transfer.
In order to evaluate the proposed approach and allow further comparison, we created a manually annotated dataset of 635 images.
The above picture depicts a few examples of discovered links between artworks in the WGA database that are impossible to retrieve using standard image retrieval methods because of a substantial change in their visual appearance. The first image, corresponding to the query, is from the composition transfer dataset, the remaining images are from WGA in the order they were retrieved by this method.
Making A Case For AI In Art
From cave paintings to Da Vinci, from Van Gogh to Banksy, how did we end up where we are now?
The aesthetically pleasing realistic faces were overshadowed by Picasso’s Cubism movement. Not that the people have started evolving into cubes but because of the triumph of imagination.
With the advent of the camera, replicating real-life or imaginations through paint has been restricted mostly to art classrooms. The way art is brought to life has moved from canvas to a MAC. Art historians, however, developed many techniques to preserve art and keep in touch with art by studying how it has evolved over time.
One might make an argument for AI by saying that it has only augmented human creativity. But there is something more to it than mere augmentation; the human subjectivity.
Understanding how arts have evolved opens up a window into the colourful history of the journey of human thought. Not to forget the uncanny capability of the machine learning algorithms to improve through transfer learning, this approach to uncover classical art can pay huge dividends in other domain.
