Guide to Self-Supervised Depth Estimation for Semantic Segmentation

we have learned how self-supervised depth estimation (SDE) can be used to improve semantic segmentation, in both semis and fully supervised configuration.

Previously, training deep learning networks for semantic segmentation use cases required large amounts of data(labelled), that was most of the user’s problem that they don’t have a large dataset and still want to get good accuracy from their deep learning models. To solve this issue researchers like Lukas Hoyer, Dengxin Dai, Yuhua Chen, Adrian Koring, Suman Saha, and Luc Van Gool purposed a new way for semi-supervised semantic segmentation, that can be enhanced by monocular depth estimation from the unlabeled dataset(images), their paper named “Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation(Read here) specifically purposed three methods to really make difference in semantic segmentation outputs.

Three key insights of the paper were:

  1. They transferred the knowledge from learned features during self-supervised depth estimation.
  2. Implemented a strong data augmentation technique by integrating images with labels using input image scene structure. 
  3. Utilized depth feature diversity to select the most reliable sets to be labelled for semantic segmentation.

The model is validated on the cityscape dataset, where it achieved significant performance gains, and SOTA results for semantic segmentation and they also open-sourced the technique on GitHub here.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.
example input output gif

1. Unsupervised Data Selection for Annotation

Firstly, This method uses Self-supervised Depth Estimation(SDE) learned on the K (unlabeled data) sequences to select NA images out of the N images for data annotation (see Algorithm 1 below)

It automatically selects the most useful set to be annotated in order to maximize the gain. The selection is driven by two criteria: diversity and uncertainty. Both of them are conducted by a novel use of SDE in this context.

To improve the appearance of semantic segmentation First, researchers learned that if they take depth estimation as an auxiliary task for semantic segmentation and present both transfer learning and multi-task learning, it will precisely improve the performance of the semantic segmentation model. 

2. DepthMix Data Augmentation

DepthMix creates virtual training data sets from pairs of labelled images. It blends images as well as their labels according to the structure of the scenes obtained from SDE, DepthMix explicitly considers the geometric structure of the scenes and generates lesser artefacts (see below figure)

3. Semi-Supervised Semantic Segmentation

It trains a semantic segmentation model by utilizing the annotated image dataset, the unlabeled image dataset, and K unlabeled image sequences. It first sees, how to exploit SDE on the image sequences to improve our semantic segmentation. We then show how to improve the performance.

Below is the architecture of semantic segmentation with SDE as an auxiliary task. See that the structure also allows data with a single picture or without segmentation ground truth. That means the SDE or the cross-entropy loss are not enabled.

Network Architecture

The network is trained on a shared ResNet101 encoder and one decoder for segmentation and one for Self-Supervised Depth Estimation(SDE). The decoder consists of an ASPP block to aggregate features from multiple scales and for SDE, the upsampling pieces have a disparity for output at the individual system. For efficient multi-task lore, researchers additionally follow PAD-Net architecture.


The source code for the above discussed is open-source here. Let’s see how to reproduce the result and what kind of framework the researcher has used.

Clone and Setup Environment

!git clone

%cd improving_segmentation_with_selfsupervised_depth

!pip install -r requirements.txt -f

Download Dataset and Setup directories

For reproducing the research paper results you have to download the Cityscapes dataset from here:, more specifically we need these three files:


And arrange the dataset as follows:

Note: to download the dataset you must have a work email and after that, you need to wait for a couple of days for approval by the cityscapes dataset website admin.


To train in the two phases of the pretraining (first 300,0000 iterations with frozen encoder and 50,000 iterations with ImageNet feature loss), you can run the below command:

!python --machine ws --config configs/cityscapes_monodepth_highres_dec5_crop.yml

!python --machine ws --config configs/cityscapes_monodepth_highres_dec6_crop.yml

Running Experiments

!python --machine ws --exp EXP_ID

To learn more about Framework structure you can follow resources here


In this demonstration, we have learned how self-supervised depth estimation (SDE) can be used to improve semantic segmentation, in both semis and fully supervised configuration. We saw the three effective strategies capable of leveraging the knowledge learned from SDE to get State of the art semantic segmentation result on images and video. To learn more about the approach you can follow these resources:

Mohit Maithani
Mohit is a Data & Technology Enthusiast with good exposure to solving real-world problems in various avenues of IT and Deep learning domain. He believes in solving human's daily problems with the help of technology.

Download our Mobile App


AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIM Research

Pioneering advanced AI market research

Request Customised Insights & Surveys for the AI Industry


Strengthen Critical AI Skills with Trusted Corporate AI Training

Our customized corporate training program on Generative AI provides a unique opportunity to empower, retain, and advance your talent.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox