Now Reading
Guide To VIBE: Video Inference for 3D Human Body Pose and Shape Estimation

Guide To VIBE: Video Inference for 3D Human Body Pose and Shape Estimation

Pose estimation is now a greater research area. Until now developments have been made based on human body 2D keypoint annotations. Most of the solutions have been around a single image or 2D motion and significantly less on the 3D motion as it involves more challenges. The primary challenge being less ground truth training 3D annotated data. Some of these 3D motion researches that have come across are not satisfactory and suffer from many drawbacks. Also, these methods are mostly frame-based, which increase error rates. 

In February 2020 (later updated in April) PhD students Muhammed Kocabas, Nikos Athanasiou, and director Michael J. Black at Max Planck Institute for Intelligent Systems represented their paper to CVPR named “VIBE: Video Inference for Human Body Pose and Shape Estimation”. VIBE uses CNNs, RNNs(GRU) and GANs along with a self-attention layer to achieve its state-of-the-art results. A monocular video is analysed into video sequences. They have used both 2D keypoint annotated data and AMASS (Archive of Motion Capture as Surface Shapes) dataset of unpaired static 3D human motion containing shapes and poses. The model is tested upon 3DPW and MPI-INF-3DHP datasets and has produced new benchmark results.

Deep Learning DevCon 2021 | 23-24th Sep | Register>>

Shown above is a state-of-the-art video-pose-estimation approach, failing to produce accurate 3D body poses. To address these limitations, a large-scale motion-capture dataset is used to train a motion discriminator using an adversarial approach. VIBE can produce realistic and accurate pose and shape, beating previous methods on standard benchmarks. Below gif shows results achieved by VIBE.

VIBE uses CNNs to extract image features. The output from the CNN is fed as input to the recurrent neural network, which processes the sequential nature of human motion. Then a temporal encoder and regressor are used to predict the body parameters for the whole input sequence. This whole part is referred to as the Generator(G) model. Now with the help of the AMASS dataset 3D, realistic human motion is achieved for adversarial training and build a motion discriminator(Dm). The motion discriminator takes in both predicted pose sequences along with pose sequences sampled from AMASS. The discriminator tries to differentiate between the fake and real motions by providing a real/fake probability for each input sequence which helps in producing realistic motion. The output of this method is a standard SMPL body model format consisting sequence of pose and shape parameters.

Code Snippet

Looking for a job change? Let us help you.

Source Code – https://github.com/mkocabas/VIBE

The code is implemented in PyTorch and underneath is the train.py file illustration. 

# importing libraries

import torch
import pprint
import random
import numpy as np
from torch.utils.tensorboard import SummaryWriter
from lib.core.loss import VIBELoss
from lib.core.trainer import Trainer
from lib.core.config import parse_args
from lib.utils.utils import prepare_output_dir
from lib.models import VIBE, MotionDiscriminator
from lib.dataset.loaders import get_data_loaders
from lib.utils.utils import create_logger, get_optimizer

# Dataloaders

data_loaders = get_data_loaders(cfg)

# compiling loss

loss = VIBELoss(
       e_loss_weight=cfg.LOSS.KP_2D_W,
       e_3d_loss_weight=cfg.LOSS.KP_3D_W,
       e_pose_loss_weight=cfg.LOSS.POSE_W,
       e_shape_loss_weight=cfg.LOSS.SHAPE_W,
       d_motion_loss_weight=cfg.LOSS.D_MOTION_LOSS_W,
   )

# Initializing networks – CNN used is ResNet-50, T = 16 (after experimenting different values this gave best results) as the sequence length minibatch size 32, the temporal encoder has 2-layer GRU with a hidden size of 1024, regressor two fully-connected layers with 1024 neurons each, followed by a final layer

generator = VIBE(
       n_layers=cfg.MODEL.TGRU.NUM_LAYERS,
       batch_size=cfg.TRAIN.BATCH_SIZE,
       seqlen=cfg.DATASET.SEQLEN,
       hidden_size=cfg.MODEL.TGRU.HIDDEN_SIZE,
       pretrained=cfg.TRAIN.PRETRAINED_REGRESSOR,
       add_linear=cfg.MODEL.TGRU.ADD_LINEAR,
       bidirectional=cfg.MODEL.TGRU.BIDIRECTIONAL,
       use_residual=cfg.MODEL.TGRU.RESIDUAL,
   ).to(cfg.DEVICE)

# initializing optimizers – Adam optimizer with a learning rate of 5  10*5 and 110*4 for the G and DM respectively

gen_optimizer = get_optimizer(
       model=generator,
       optim_type=cfg.TRAIN.GEN_OPTIM,
       lr=cfg.TRAIN.GEN_LR,
       weight_decay=cfg.TRAIN.GEN_WD,
       momentum=cfg.TRAIN.GEN_MOMENTUM,)

# initializing discriminator – contains a sequence of GRUs and self-attention to amplify distinctive frames. 2 MLP layers containing 1024 neurons each with tanh activation function.

motion_discriminator = MotionDiscriminator(
       rnn_size=cfg.TRAIN.MOT_DISCR.HIDDEN_SIZE,
       input_size=69,
       num_layers=cfg.TRAIN.MOT_DISCR.NUM_LAYERS,
       output_size=1,
       feature_pool=cfg.TRAIN.MOT_DISCR.FEATURE_POOL,
       attention_size=None if cfg.TRAIN.MOT_DISCR.FEATURE_POOL !='attention' else cfg.TRAIN.MOT_DISCR.ATT.SIZE,
       attention_layers=None if cfg.TRAIN.MOT_DISCR.FEATURE_POOL !='attention' else cfg.TRAIN.MOT_DISCR.ATT.LAYERS,
       attention_dropout=None if cfg.TRAIN.MOT_DISCR.FEATURE_POOL !='attention' else cfg.TRAIN.MOT_DISCR.ATT.DROPOUT
   ).to(cfg.DEVICE)

dis_motion_optimizer = get_optimizer(
       model=motion_discriminator,
       optim_type=cfg.TRAIN.MOT_DISCR.OPTIM,
       lr=cfg.TRAIN.MOT_DISCR.LR,
       weight_decay=cfg.TRAIN.MOT_DISCR.WD,
       momentum=cfg.TRAIN.MOT_DISCR.MOMENTUM
   )

# initializing lr_schedulers

motion_lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
       dis_motion_optimizer,
       mode='min',
       factor=0.1,
       patience=cfg.TRAIN.LR_PATIENCE,
       verbose=True,
   )
   lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
       gen_optimizer,
       mode='min',
       factor=0.1,
       patience=cfg.TRAIN.LR_PATIENCE,
       verbose=True,
   )

# Training model – Both the generator and discriminator are trained together and thus helps in reducing loss.

Trainer(
       data_loaders=data_loaders,
       generator=generator,
       motion_discriminator=motion_discriminator,
       criterion=loss,
       dis_motion_optimizer=dis_motion_optimizer,
       dis_motion_update_steps=cfg.TRAIN.MOT_DISCR.UPDATE_STEPS,
       gen_optimizer=gen_optimizer,
       start_epoch=cfg.TRAIN.START_EPOCH,
       end_epoch=cfg.TRAIN.END_EPOCH,
       device=cfg.DEVICE,
       writer=writer,
       debug=cfg.DEBUG,
       logdir=cfg.LOGDIR,
       lr_scheduler=lr_scheduler,
       motion_lr_scheduler=motion_lr_scheduler,
       resume=cfg.TRAIN.RESUME,
       num_iters_per_epoch=cfg.TRAIN.NUM_ITERS_PER_EPOCH,
       debug_freq=cfg.DEBUG_FREQ,
   ).fit()

Benchmark results 

Below are the results achieved by SOTA models on 3DPW, MPI-INF-3DHP, and Human3.6M datasets.

This is taken from the paper. VIBE (direct comp.) is trained on video datasets like others, while VIBE is trained with extra data from the 3DPW training set. Vibe outperforms all.

Limitations – Vibe fails in heavy occlusion, fast motion, and multi-person occlusion.

End Notes

3D pose estimation is necessary to understand human behaviour. Vibe has introduced many methods that are clubbed together to achieve the state-of-art results. In future releases, we can expect supervision on single-frame methods by fine-tuning the HMR features, extending experiments to optical flow, and resolve the multi-person and occlusion problem. Also, now in the era of transformers, the authors have plans to explore it and enhance more to showcase better performances.

Video Demonstration – https://youtu.be/rIr-nX63dUA

Explained talk by the authors – https://twimlai.com/thats-a-vibe-ml-for-human-pose-and-shape-estimation-with-nikos-athanasiou-muhammed-kocabas-michael-black/

Code Demo – https://colab.research.google.com/drive/1dFfwxZ52MN86FA6uFNypMEdFShd2euQA

What Do You Think?

Join Our Discord Server. Be part of an engaging online community. Join Here.


Subscribe to our Newsletter

Get the latest updates and relevant offers by sharing your email.

Copyright Analytics India Magazine Pvt Ltd

Scroll To Top