Artwork authentication requires expert connoisseurs, and it quite often becomes impossible to establish authenticity amid conflicting expert opinions, subjective interpretations, and scarce evidence. Scientific tools step in here to assist with setting a painting’s age and subsurface details. But, these tools haven’t been able to crack the code to identify the artists yet. 

Convolutional neural networks are used to analyse images in facial recognition systems and self-driving vehicles; Steven Frank talks about utilising them to identify artists based on their style and technique. In this article, we discuss the AI tool created by Frank to find art forgeries. 

The Problem

Creating neural networks requires many training samples, but even the most prolific artists have not managed to produce thousands of paintings. Another issue faced with analysing paintings is the image resolution challenge. Simultaneously, high-resolution images of artworks are too large for CNNs. Smaller-sized images lack the information needed to ensure impartiality. 

The Design

Frank, an AI graduate, and his wife, Andrea, an art historian, took it upon themselves to find an AI-based solution for art forgeries. They used the ‘digitised biopsy slides’ approach used by pathologists, where practitioners have made pixelated images tractable for CNNs by creating smaller fragments or square tiles. This allows the creation of several training tiles from a single image.

Initially, the duo trained their neural network with Dutch artist Rembrandt’s portraits. The training set included randomly chosen 50 original portraits and 50 portraits by other artists. While the system could differentiate when the other 50 artists had a different style altogether, it couldn’t set apart original Rembrandt from students and admirers, much fewer forgers. 

To train the AI tool to distinguish similar work from Rembrandt’s original, the researchers compiled a dataset with non-Rembrandt entries ranging from close dupes to distinguishable paintings. To ensure that the tile contributed reliably to classification, the entropy cutoff was at least as much as that of the entire image. This tied the entropy threshold to the character of the painting. 

The selected tile size was 450 x 450 pixels, consisting of features experts rely on while judging a painting’s authorship. The dataset consisted of 76 Rembrandt and non-Rembrandt paintings that were shuffled in four ways into separate sets of 51 training and 21 test images to cross-validate the results. They successfully created a five-layer CNN that had more than 90 per cent accuracy when distinguishing Rembrandt from his students and forgers. 

Dubbing the device, ‘The A-Eye’, Frank and Andrea put it to test on Vincent van Gogh’s landscapes – this time with a larger dataset. They used 152 van Gogh and non-van Gogh paintings divided four ways into sets of 100 training and 52 test images. The team used smaller tiles to achieve high accuracy on their test set. This led them to the conclusion that the signature style of the artist’s work, i.e. the distinctive feature sizes that facilitate accuracy in CNN based classification, is particular to individual artists. 

Lastly, to predict the artisanship of the painting, the duo assigned a probability to different tiles with their pixels. Since more than one tile intercepts a pixel, they created an average of the relevant tile-level possibilities and used it to determine the value of that pixel. This resulted in a probability map of regions that are more or less likely to be an original painting of the artist. 

The A-eye managed to identify areas of doubt in the face and background of Rembrandt’s portraits of his wife. This accords with the belief held by many scholars that those regions have been overpainted later by someone else. Additionally, the duo evaluated van Gogh’s and Rembrandt’s works under controversy, and in all but one case, their classifications matched the current scholar consensus. 

Their latest experiment has left scholars with food for thought regarding the consensus on one of Rembrandt’s paintings. ‘The Man With The Golden Helmet’ was de-attributed by the Staatliche Museum’s scholars believing there was an inconsistency in the paint handling. However, contrary to that, the A-Eye has concluded that the painting is Rembrandt’s after narrowing the CNN analysis on the painting’s features. 

Probability map: Steven & Andrea’s IEEE feature

The results of the A-eye on a painting by Rembrandt (left), the painting ‘The Man With The Golden Helmet’ formerly attributed to him (middle), and Salvator Mundi (right). 

Hot colours represent areas the classifier determined had high probability of having been painted by the artist associated with the work. 

Art Recognition

Another example of this; A Swiss company, Art Recognition, leverages AI to assess the authenticity of a work by using neural networks to analyse an artist’s brush strokes and techniques in a high-resolution photographic reproduction of the painting.

In its application to assess van Gogh’s controversial self-portrait, the A-eye found the self-portrait to be 97 per cent probable authentic van Gogh. This was later confirmed by the van Gogh Museum in Amsterdam’s years worth of studies. 

“We started by training our deep convolutional neural network to learn van Gogh by using hundreds of original images from de la Faille Catalogue Raisonné,” according to the company’s case study. “To help the system better distinguish forgeries, we also fed in well-known (labelled) fakes like, for example, the famous Wacker forgeries.” The company generated a heat map to provide visual evidence on the algorithm’s evaluation, with the hotspot spectrums highlighted in red.

The use of AI in finding art forgeries or dupes is still a developing technology, but implementations like these are opening the doors to more sophisticated use of CNN with paintings.