Top 8 Resources To Learn Self-Supervised Learning In 2021

Did you know that almost 90% of the world’s data has been created in the last two years alone, and nearly 2.5 quintillion bytes of data are produced by humans every day? 

On social media platforms like Instagram, close to 95 million photos and videos are shared, while 500 million tweets are sent out every day. Not to mention the swathes of visual data being churned out daily.

Self-supervised learning or unsupervised semantic feature learning has become the go-to option to build scalable computer vision models. Self-supervised learning is the ability of a system to learn without manual annotation. 

Previously, for a system to learn high-level semantic image features, it required a massive amount of manually labelled data, which is time-consuming, expensive and impractical to scale. Thanks to self-supervised learning — unlabelled images, videos and other data — can now be trained seamlessly. 

Lately, there have been several developments in self-supervised learning. Many researchers from large tech companies, including Google, Facebook and Microsoft, are developing scalable machine learning models.

Keeping in mind the advancements in the computer vision landscape, we have curated a list of resources to get the hang of self-supervised learning:

Introduction to Self-Supervised Learning

In a series of videos, Anuj Shah introduces self-supervised representation learning and why it is crucial. He also discusses the entire pipeline for performing self-supervised learning. In addition to this, he speaks about various categories of self-supervised learning and explores research works in the areas of self-supervised methods like MoCo, SimCLT, SwAV and others.  

Self-Supervised Learning

In 122 PowerPoint slides, DeepMind’s Andrew Zisserman captures the essence of self-supervised learning perfectly, touching upon its implementation on unlabelled image, videos and audio files, alongside discussing various parameters, functions and challenges to findings. 

The slides could be a good start for beginners and amateurs looking to understand the basics of self-supervised learning. Click here to view the presentation slides.    

Going Meta: Learning Algorithms & The Self-Supervised Machine 

In this Microsoft Research podcast, Dr Philip Bachman — a researcher at MSR Montreal — gives an overview of the machine learning space and talks about the clutter in successfully gathering helpful information. Further, he cites the ongoing work on Deep InfoMax, a novel approach to self-supervised learning, and delve into machine learning classification problems. 

Check out the podcast here

Self-Supervised Learning: The Dark Matter Of Intelligence

Facebook VP and chief AI scientist Yann LeCun, alongside Ishan Misra, a research scientist, in a blog post, explains the self-supervised learning model in detail, along with its impact, categories and various architecture used (siamese networks or joint embedding architecture). 

Further, the post highlights self-supervised CV model SEER, used on a billion random, unlabeled and uncurated public Instagram images and supervised fine-tuning on ImageNet. The self-supervised systems saw 84.2 percent accuracy on ImageNet. 

Self-Supervised Representation Learning

In this blog post, OpenAI’s Lilian Weng explains the nuances of self-supervised learning and covers interesting ideas in self-supervised learning tasks on images, videos, and control problems. 

Unsupervised Learning: Introduction

In this tutorial, tech evangelist Andrew Ng discusses effective machine learning techniques. Ng speaks about Silicon Valley’s best practices in the AI and the machine learning landscape, alongside providing a broad introduction to machine learning, data mining, statistical pattern recognition, supervised learning and real-time use cases.   

Topics covered include: 

  1. Supervised learning (parametric/non-parametric algorithms, kernels, neural networks, etc.)
  2. Unsupervised learning (clustering, dimensionality reduction, deep learning and more)
  3. Best practices in machine learning (variance theory/bias; innovation process in ML and AI)

Unsupervised Learning With Python 

The book written by Vaibhav Verdhan teaches you how to apply unsupervised learning algorithms. The book covers many bases, from kmeans and hierarchical clustering to advanced neural networks like GANs and restricted Boltzmann machines. The book also gives multiple examples, where each new algorithm is introduced with a case study for retail, banking, aviation sectors. 

More than anything, the book bridges the gap between complex math and practical implementations, covering all the model developments to production deployment. Also, after the end of each chapter, you’ll find quizzes, practice datasets, and research paper links for better comprehension.

Introduction To Semi-Supervised Learning

Zhu Xiaojin and Andrew B Goldberg highlight various semi-supervised learning aspects (the combination of labelled and unlabeled data) in this book. It talks about popular semi-supervised learning models like self-training, mixture models, co-training, graph-based methods and more.

The book further discusses the basic mathematical formulation, assumptions and limitations of each model. It is a good starting point for understanding the nuances of semi-supervised learning before experimenting with self-supervised learning.

More Great AIM Stories

Amit Raja Naik
Amit Raja Naik is a seasoned technology journalist who covers everything from data science to machine learning and artificial intelligence for Analytics India Magazine, where he examines the trends, challenges, ideas, and transformations across the industry.

More Stories


8th April | In-person Conference | Hotel Radisson Blue, Bangalore

Organized by Analytics India Magazine

View Event >>

30th Apr | Virtual conference

Organized by Analytics India Magazine

View Event >>

Yugesh Verma
A guide to self-supervised learning with graph data

graph structure has much additional information with them like node attributes, and label information of nodes. Using this source of information, we can have unprecedented opportunities to design advanced level self-supervised pretext tasks

Yugesh Verma
A Beginner’s Guide to Self-Supervised Classification

in the self supervised learning process we are mainly focused about making the data workable to the downstream algorithms. but when using the self-supervised learning we make the data specifically for classification we can say the process is self-supervised classification.

The End Of ImageNet
Amit Raja Naik
The End Of ImageNet

PASS does not include any humans and can be used for high-quality pre-training while significantly reducing privacy concerns.

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM