Facebook Introduces New Image Generation Model Called Instance-Conditioned GAN

IC-GAN can be used with both labelled and unlabelled datasets.
Facebook Introduces New Image Generation Model Called Instance-Conditioned GAN

Facebook recently introduced a new image generation model called ‘Instance-Conditioned GAN (IC-GAN). This new model creates high-quality, diverse images with or without input images present in the training set. Compared to traditional methods, IC-GANs can generate realistic, unforeseen image combinations. 

The PyTorch code for Instance-Conditioned GAN is available on GitHub


GAN, or generative adversarial network, is one of the popular AI methods to create images, be it abstract collages or photorealistic pictures. However, they, too, have limitations as they can only generate images of objects or scenes closely related to the training dataset. 

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

For example, a traditional GAN trained on images of cars shows impressive results when asked to generate other images of cars, but it is most likely to fail if asked to generate images of flowers or other objects outside of the car dataset. 

That is where Facebook’s Instance-Conditioned GAN comes into the picture, where it can generate images outside of the training dataset seamlessly. The new approach exhibits exceptional transfer capabilities across various types of objects. Facebook researchers said that the researchers can use IC-GANs off the shelf with previously unseen datasets and still generate realistic-looking images without the need for annotated/labelled data. 

Download our Mobile App

Also, IC-GANs could create new visual examples to augment datasets to include diverse objects and scenes, helping artists and creators with more expansive, creative AI-generated content, and advance research in high-quality image generation. 

Here’s how IC-GAN works

Facebook’s IC-GAN can be used with both labelled and unlabelled datasets. It extends the GAN framework to model a mixture of local and overlapping data clusters. The way it works is, it takes a single image, or ‘instance,’ and then generates images that are similar to the instance’s closest neighbours in the dataset. 

The researchers said that they use neighbours as an input to the discriminator to force the generator to create samples similar to the neighbourhood samples of each instance. This way, it avoids the problem of partitioning data into small clusters as so much of the data is overlapping. As a result, it lets models use datasets more efficiently. 

After the model is trained, the researchers then test it on images it has never seen before. With the help of a single image, the model can effectively generate visually rich images similar to the closest neighbours in the dataset.

In contrast, standard methods like class conditional GANs focus on conditioning class labels, effectively partitioning the data into groups corresponding to those labels to generate more high-quality samples than their unconditional counterparts. So, instead of creating only random images, these GANs create images that fit a particular label like ‘clothing’ or ‘car.’ At the same time, they rely on annotated/labelled data that may be unfeasible to obtain. 

Previously, label-free learning approaches — using no labelled data — to image generation have been promising, but their result is usually of poor quality when trained to model complex datasets (ImageNet). Plus, they either use coarse, nonoverlapping data partitions or fine partitions that tend to deteriorate output as the clusters contain limited data points. 

The image below depicts an IC-GAN instance on the left to generate the corresponding images shown on the right. Here, no class label is provided. 

Facebook Introduces New Image Generation Model Called Instance-Conditioned GAN
(Source: Facebook)

Interestingly, the IC-GAN can be transferred to other datasets not seen during the training for both class-conditional scenarios and where there are no labels. 

Further, in IC-GAN, the researchers said they do this by swapping out the conditioning instances at inference time. In the case of class-conditional IC-GAN, they can swap either the instance conditioning or the class label. Combining instances and the class labels, the class-conditional IC-GAN can create unusual scenes that aren’t present or are very rare in current datasets.  

For example, with the image of a snowplough surrounded by snow and a class label ‘camel,’ which does not appear in the instance conditioning, the user can generate camels surrounded by snow, thereby eliminating the bias that camels live only in the desert. (see the image below) 

Facebook Introduces New Image Generation Model Called Instance-Conditioned GAN
(Source: Facebook

Another example (as shown below) shows that the IC-GAN uses the graphics of a house to create a more realistic-looking building.

Unlimited Possibilities

Facebook Introduces New Image Generation Model Called Instance-Conditioned GAN
(Source: Facebook)

In a nutshell, IC-GAN can augment data and include objects commonly found in the training data. As a result, this new approach can help generate more diverse training data for object recognition models as it works across various domains. 

For instance, traditional GAN models would not generate images of zebras standing in urban areas, as its training data would likely only contain images of zebras in grasslands. Here, the IC-GAN can augment data and include objects that are not commonly found in standard datasets. 

“In the future, we hope to explore ways to bring even more control to this model,” said Facebook, stating that it will no longer be just about the object at the centre and the background. “We want to explore how more objects can be placed in the background and determine where the items are placed, creating complex, picture-perfect scenes,” said the researchers.

Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Amit Raja Naik
Amit Raja Naik is a seasoned technology journalist who covers everything from data science to machine learning and artificial intelligence for Analytics India Magazine, where he examines the trends, challenges, ideas, and transformations across the industry.

Our Upcoming Events

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023

21 Jul, 2023 | New York
MachineCon USA 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

The Great Indian IT Reshuffling

While both the top guns of TCS and Tech Mahindra are reflecting rather positive signs to the media, the reason behind the resignations is far more grave.

OpenAI, a Data Scavenging Company for Microsoft

While it might be true that the investment was for furthering AI research, this partnership is also providing Microsoft with one of the greatest assets of this digital age, data​​, and—perhaps to make it worse—that data might be yours.