Video segmentation, or the partitioning of video frames into multiple segments or objects, is important in a variety of practical applications, including visual effect assistance in movies, autonomous driving scene understanding, and video conferencing virtual background creation, to name a few. In order to fully comprehend this concept, we will look at what video object segmentation is and how it is used in this post. Below is a list of the main points to be discussed.

Table Of Contents

1. What is Video Object Segmentation?
2. Methods of Video Object Segmentation
   1. Unsupervised
   2. Semi-Supervised
   3. Spatio-Temporal Graph
   4. Convolutional Neural Network
3. Video Object Segmentation in Python

Let us start the discussion by understanding what is Video object segmentation.

What is Video Object Segmentation

Segmenting and tracking the objects of interest in the video is critical for effectively analyzing and using video big data. Computer vision requires two basic tasks – segmenting and tracking video objects. The object segmentation mask is made by dividing the pixels in the video frame into two groups: foreground target and background region. In behaviour recognition and video retrieval, this is the crux of the problem.

Object tracking determines the exact location of the target in the video image and generates the object bounding box, which is required for intelligent monitoring, big data video analysis, and other applications. The segmentation and tracking of video objects appear to be separate issues, but they are actually intertwined. 

That is, solving one problem usually necessitates solving another problem, either implicitly or explicitly. Clearly, solving the object segmentation problem makes solving the object tracking problem simple. On the one hand, accurate segmentation results provide reliable object observations for tracking, which can help solve issues like occult tracking.

Accurate object tracking results, on the other hand, can be used to guide the segmentation algorithm in determining the object position, reducing the impact of object fast movement, complex backgrounds, similar objects, and other factors, and improving object segmentation performance. 

Many studies have found that processing object segmentation and tracking problems at the same time can help overcome their respective difficulties and improve performance. Video object segmentation (VOS) and video object tracking are two major tasks that are related to each other (VOT).

Methods of Video Object Segmentation

Video object segmentation and tracking methods are divided into two categories in this section: unsupervised and semi-supervised video object segmentation methods. Let’s look at each one individually.

Unsupervised Video Object Segmentation

During the test period, unsupervised methods assume no human input on the video. They want to extract the most important Spatio-temporal object tube by grouping pixels that are consistent in appearance and motion. They assume the objects to be segmented and tracked have different motions or appear frequently in the sequence of images in general. 

Early video segmentation techniques were primarily geometric in nature, and they were limited to specific motion backgrounds. The traditional background subtraction method simulates the appearance of each pixel’s background while treating rapidly changing pixels as foreground. A moving object is represented by any significant change in the image and background model. The pixels that make up the changed region are flagged to be processed further.

The connected region corresponding to the object is estimated using a connected component algorithm. As a result, the process described above is known as background subtraction. Video object segmentation is accomplished by creating a background model of the scene and then looking for deviations from the model for each input frame.

Semi-Supervised Video Object Segmentation

Semi-supervised methods begin with human input, such as a pixel-accurate mask, clicks, or scribbles, and then propagate the information to subsequent frames. The use of superpixels, the creation of graphical models, the use of object proposals, and the use of optical flow and long-term trajectories are all highlighted in existing approaches. 

The architecture of these methods is typically based on semantic segmentation networks, and each video frame is processed individually. Spatio-temporal graph and CNN-based semi-supervised VOS are the two main categories in which they can be studied.

Spatio-Temporal Graph

Early methods solved some Spatio-temporal graphs with hand-crafted feature representation, including appearance, boundary, and optical flows, and propagated the foreground region throughout the video in recent years. Object representation of graph structure and Spatio-temporal connections are typically used in these methods.

The task is typically formulated as a spatiotemporal label propagation problem, and these methods approach the problem by constructing graph structures over the object representation of I pixels, (ii) superpixels, or (iii) object patches to infer the labels for subsequent frames.

Convolutional Neural Network

With the success of convolutional neural networks for static image segmentation, CNN-based methods for video object segmentation show overwhelming power. Motion-based and detection-based techniques for temporal motion information can be categorized into two categories.

In general, motion-based methods make use of the temporal coherence of object motion to formulate the problem of mask propagation from the first frame or a given annotated frame to subsequent frames.

Without using temporal information, some methods learn an appearance model to perform pixel-level detection and segmentation of the object at each frame. To fine-tune a deep network, they rely on the first frame annotation of a given test sequence.

Video Object Segmentation in Python

In this segment, we will implement the video object segmentation task using a PixelLib. PixelLib is a library that allows us to segment images and videos with just a few lines of code. It’s a versatile library designed to make image and video segmentation easy to integrate into software solutions. 

Import all the dependencies that are needed. 

pip install pixellib 
import pixellib
from pixellib.semantic import semantic_segmentation

Now we need to create a class from the Pixellib module to perform semantic segmentation, and we can refer to the class by the variable segment video.

segment_video = semantic_segmentation()

Now we need to use the pixellib function to load the xception model that was trained on the Pascal voc dataset. Pascal VOC is a dataset collection for object detection. You must first download the pre-trained xception model, which you can do in Colab by following the steps below.

# Download the model
!wget https://github.com/ayoolaolafenwa/PixelLib/releases/download/1.1/deeplabv3_xception_tf_dim_ordering_tf_kernels.h5
 
segment_video.load_pascalvoc_model("/content/deeplabv3_xception_tf_dim_ordering_tf_kernels.h5")

We can now use the model to infer the model from any video we want now that we’ve loaded it. The image segmentation is done in the Pascalvoc color format, and this is the line of code that does it. Two parameters are required for this function:

• video path: the location of the segmented video file.
• frames per second: Determines how many frames per second the output video file will have. It is set to 15 in this case, implying that the saved video file will have 15 frames per second.
• output video name: the name of the segmented video that was saved. The finished video will be saved in the same folder as your current working directory.

segment_video.process_video_pascalvoc("/content/mixkit-children-playing-with-a-dancing-fountain-3469.mp4", overlay = True, frames_per_second= 15, output_video_name="segmented.mp4")

Here is the original video that we are going to perform segmentation.

And here is the output of the inference.  

For our particular video, the model has identified a total of 296 frames and took nearly 3.5 mins to perform segmentation. 

Final Words

Through this post, we have seen video object segmentation and tracking. We have also seen some of its potential methods used by practitioners and developers. And lastly, we have practically implemented it using the Pixllib python based library where we have seen how we can obtain SOTA objected segmentation hat too with a few lines of code and we can say that the results are pretty impressive.

References:

• Video Object Segmentation and Tracking: A Survey
• PixelLib Documentation
• Link for above Codes