The recent advances in Deep Learning and Deep Learning models have been largely connected to the quantity and diversity of data gathered. Data augmentation enables practitioners to significantly increase the diversity of data available for training the models without actually collecting new data. Data augmentation is a technique that helps to create new training data from the existing training data artificially. This is done by applying specific techniques to training data to create new and different training examples. The most commonly used Data augmentation techniques to train large neural networks are cropping, padding, and horizontal flipping. Most approaches used in training neural networks only use basic types of augmentation. Although neural network architectures have been investigated in-depth, strong types of data augmentation and data augmentation policies that capture data invariances are yet to be discovered.
During training, even after using a proper model, we do not get satisfactory results. In the end, it all comes to the data used to train the network. Having a large dataset is crucial for the performance of a deep learning model. Lack of quantity and diversity of data thereby hampers the model performance. Data Augmentation helps us increase the size of the dataset and introduce variability in the dataset. During Data Augmentation, the neural network treats each data as a distinct data point. Data Augmentation also helps reduce the problem of overfitting.
Our dataset may have images taken in a limited set of conditions, but we might fall short in various conditions; the modified/augmented data helps deal with such scenarios. We only need to make minor alterations to our existing training data to get more data, especially when talking about Image Data Augmentation. The alterations might include flipping the image horizontally, vertically, padding, cropping, rotating, scaling and few other translations of the input image. For machine learning and deep learning models, collecting and labelling data can be exhausting and costly. Transformations in datasets by using data augmentation techniques allow companies to reduce these extra and unwanted operational costs. Companies also need to build evaluation systems for the quality of augmented datasets. As the use of data augmentation methods increases, assessing the quality of their output will be highly required.
A more ambitious data augmentation technique consists of leveraging segmentation annotations, either obtained manually or from an automatic segmentation system, and creating new images with objects placed at various positions in existing scenes.
What is AugLy?
AugLy, a new open source Python library that helps AI researchers to make use and create data augmentations to improve the robustness of their created machine learning models. Augmentations can include a wide variety of modifications to a dataset, ranging from cropping a photo to changing the pitch of a voice recording. In addition, AugLy provides sophisticated data augmentation tools to create samples to train and test different systems. AugLy currently supports four modalities, namely audio, image text and video and over 100 other augmentations. Each modality’s augmentations are contained within its sub-library. These sub-libraries can help perform both function-based, and class-based transforms, composition operators and are also available with an option to provide metadata about the transformation applied.
AugLy is a great library to utilize for augmenting data during model training.It was designed to include and avail many specific data augmentations that users perform in real life on internet platforms like Facebook, for example, turning an image into a meme, overlaying text and emojis on images and videos or reposting a screenshot from social media. It has four sub-libraries, each corresponding to a different modality, following the same interface. AugLy can also generate useful metadata to help you understand how your data was transformed.
Getting Started with the Code
This article will try to explore the different augmentation services available in the AugLy library developed by Facebook’s AI Research Team. We will perform augmentations on image and video data, making use of the components from the library. The following implementation is partially inspired by the creators of AugLy, whose GitHub repository can be accessed using the link here.
Augmentation on Image Data: Installing the Library
We will first perform image augmentation using Augly. To do so, we will first have to install the required library for image-based augmentation. It can be done using the following code,
!pip install -U augly
!sudo apt-get install python3-magicImporting Dependencies
Now that our library is installed, we will import some further dependencies,
import os import augly.image as imaugs import augly.utils as utils from IPython.display import display
Setting Path and Image Size
Now we will be setting the image path and resizing it so that the image, when displayed, does not take up the whole screen where we cannot see the changes.
# Setting image path, scaling it down input_img_path = "/content/856136_new-black-and-white-2016-drake-4k-wallpapers_1200x675_h.png" input_img = imaugs.scale(input_img_path, factor=0.5) # We can use the AugLy scale augmentation input_img = imaugs.scale(input_img_path, factor=0.7) display(input_img)
Output :
Performing Augmentations
Let’s perform some augmentations and explore what the library can offer,
# applying various augmentations to the scaled image
#creating a meme
display(
imaugs.meme_format(
input_img,
caption_height=100,
meme_bg_color=(20, 19, 21),
text_color=(255, 255, 255),
)
)
#degrading image pixels meta = [] display(imaugs.shuffle_pixels(input_img, factor=0.3, metadata=meta)) meta
[{'dst_height': 420,
'dst_width': 560,
'factor': 0.3,
'intensity': 30.0,
'name': 'shuffle_pixels',
'output_path': None,
'seed': 10,
'src_height': 420,
'src_width': 560}]
# Image Desizing
meta = []
aug = imaugs.PerspectiveTransform(sigma=20.0)
display(aug(input_img, metadata=meta))
meta
#Changing Aspect Ratio
meta = []
aug = imaugs.RandomAspectRatio()
display(aug(input_img))
meta
# Applying several transformations together to create a new image
aug = imaugs.Compose(
[
imaugs.Saturation(factor=0.7),
imaugs.OverlayOntoScreenshot(
template_filepath=os.path.join(
utils.SCREENSHOT_TEMPLATES_DIR, "mobile.png"
),
),
imaugs.Scale(factor=0.9),
]
)
display(aug(input_img))
Augmentation on Video Data
Just Like Images, similar Augmentations can also be performed on video data. To do so, we’ll follow the same process. First, install the video augmentation library and its dependencies, then we’ll use different sets of codes to perform augmentation on the input video.
#instaling the AugLy Video Library
!pip install -U augly[av]
!sudo apt-get install python3-magic
# Installing ffmpeg for the video module of augly
!sudo add-apt-repository ppa:jonathonf/ffmpeg-4
!apt install ffmpeg
#setting video hyperparameters
from IPython.display import display, HTML
from base64 import b64encode
def display_video(path):
mp4 = open(path,'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
display(
HTML(
"""
<video width=400 controls>
<source src="%s" type="video/mp4">
</video>
""" % data_url
)
)
import os
import augly.utils as utils
import augly.video as vidaugs
# Set video path and trim to first 3 seconds
input_video = os.path.join(
utils.TEST_URI, "video", "inputs", "input_1.mp4"
)
input_vid_path = "/tmp/in_video.mp4"
out_vid_path = "/tmp/aug_video.mp4"
# We can use the AugLy trim augmentation, and save the trimmed video
vidaugs.trim(input_video, output_path=input_vid_path, start=0, end=3)
display_video(input_vid_path)
Performing Different Video Augmentations
Let us now perform different types of augmentations on the input video,
# Overlaying the video with random text
vidaugs.overlay_text(input_vid_path, out_vid_path)
display_video(out_vid_path)
#code for looping video repeatedly
meta = []
vidaugs.loop(
input_vid_path,
out_vid_path,
num_loops=1,
metadata=meta,
)
display_video(out_vid_path)
meta
#displaying randomly generated emoji in video
meta = []
aug = vidaugs.RandomEmojiOverlay()
aug(input_vid_path, out_vid_path, metadata=meta)
display_video(out_vid_path)
meta
# Degrading image quaity using blur
aug = vidaugs.Compose(
[
vidaugs.AddNoise(),
vidaugs.Blur(sigma=5.0),
vidaugs.OverlayDots(),
]
)
aug(input_vid_path, out_vid_path)
display_video(out_vid_path)
EndNotes
In this article, we tried to understand what exactly image augmentation is and how it works. We also dug deep into the uses of image augmentation and explored the AugLy library, which allows us to easily perform Image, Text, Audio, and Video Augmentation. The above implementations can be found as a colab notebook, using the following links:
