ActivityNet is an enormous video dataset that covers exercises that are generally pertinent to how people invest their energy in their everyday living. It was developed in 2015 by the researchers: Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanemand Juan Carlos Niebles1. This video dataset gives tests from 203 movement classes with a normal of 137 untrimmed recordings per class and 1.41 movement occurrences per video, for an aggregate of 849 video hours.

The number and length of every video are increased in comparison to other ones. The dataset furnishes a rich action pecking order within any event four levels of depth. For example, the activity cleaning windows falls under the fourth tier category dressing, washing belongs to the second category Grooming interior cleaning falls under Grooming oneself and finally, household activities belong to the category Personal care.

Here, we will discuss data contained in this dataset, how it was gathered, and provide some benchmark models that gave high accuracy on this dataset. Further, we will implement the ActivityNet dataset using Pytorch and Keras Library.

Data Collection

Usually Gathering data using the conventional approach takes a lot of time. To resolve this issue modern tools are employed to save scientist hours. The first step was to look through the web to recover recordings identified with every movement. Utilizing the text-based queries the researchers had searched the information in platforms like YouTube and Google recordings. They confirmed all recordings recovered and eliminated those not related to the movement at hand. Next Amazon Mechanical Turk was used to survey each video and decide whether it contains a planned movement class. Because of the error of text-based inquiries, numerous recordings that are not related to any of the expected movement classes were eliminated.

Loading the dataset using Pytorch

The dataset can be downloaded from the following link.

Import all the libraries required for this project.

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
import torch.optim as optim
import transforms.spatial_transforms as spatial
import transforms.temporal_transforms as temporal
from dataset.activitynet_train import ActivityNetCaptions_Train

# transforms
spa = spatial.Compose([spt.CornerCrop(size=args.imsize), spt.ToTensor()])
tpa = temporal.Compose([tpt.TemporalRandomCrop(args.clip_len), tpt.LoopPadding(args.clip_len)])

#Loading the dataset
train = ActivityNetCaptions_Train(args.root_path, ann_path='train_fps.json', sample_duration=args.clip_len, spatial_transform=spa, temporal_transform=tpa)

Let’s define the parameters in the ActivityNet dataset Class:

·   root  – It is the root directory of the UCF101 Dataset.

·   annotation_path – It contains the split files.

·   sample_duration – Duration of sample that captures a human activity.

·   spatial_transform and temporal_transform – Transforms the data.For example it resizes an image or crops an image. 

dataloader = DataLoader(train, batch_size=args.batch_size, shuffle=True, num_workers=args.n_cpu, drop_last=True, timeout=100)

Training Results

Loading the dataset in Keras

Install the video generator using the pip command. We have used an Image generator for data augmentation.

import os
import glob
import keras
from keras_video import VideoFrameGenerator

Let’s define the parameters so that we can pass it to the model for training.

classes = [i.split(os.path.sep)[1] for i in glob.glob('videos/*')]
classes.sort()
# Parameters
Size = (112, 112)
channel = 3
Nbframe = 5
Batch_size = 32
# Data augmentation
data_augment = keras.preprocessing.image.ImageDataGenerator(
    zoom_range=.1,
    horizontal_flip=True,
    rotation_range=8,
    width_shift_range=.2,
    height_shift_range=.2)

# Create video frame generator
load_data = VideoFrameGenerator('data/train/',
    classes=classes, 
    nb_frames=Nbframe,
    split=.33, 
    shuffle=True,
    batch_size=Batch_size,
    target_shape=Size,
    nb_channel=channel,
    transformation=data_augment ,
    use_frame_cache=True)

State of the Art

The current state of the art on ActivityNet dataset is G-TAD. The model gave an accuracy of 34.09%. BMN is a close-by competitor with an accuracy of around 34%.

Conclusion

In this article, we have discussed the details and implementation of a huge human movement dataset, ActivityNet. It is made conceivable by an enormous and nonstop video assortment that is effectively adaptable to bigger quantities of exercises and bigger examples per movement, at a low cost. It additionally contains a bigger number of classes and tests per classification than customary activity datasets.