Kornia: An OpenCV-inspired PyTorch Framework

Kornia is an open-source Python library inspired by OpenCV designed to handle generic Computer Vision tasks. It was introduced by Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee and Gary Bradski in October, 2019 (research paper). 

Kornia leverages PyTorch library at its backend in terms of model’s efficiency and reverse-mode auto-differentiation for defining and computing complex functions’ gradients. It comprises a subset of packages having operators that act as an input to neural networks for performing a wide range of tasks such as image transformations, depth estimation, epipolar geometry, filtering and edge-detection applicable on high-dimensional tensors. Unlike conventional CPU-based CV libraries such as torchvision and scikit-image, several standard deep learning functions can be implemented on GPUs using Kornia. 

Kornia bridges the gap between two simple yet powerful libraries namely, OpenCV and PyTorch. Though solely based on traditional CV solutions like torchvision, tf.image, PIL and skimage, it enables differentiable programming for CV applications by utilizing the crucial properties of PyTorch like GPU hardware acceleration, differentiability and distributed data-flows.

The following table compares Kornia with some important CV libraries/modules:

Library/Module	CPU supported	GPU supported	Batch Processing	Differentia-ble	Distributed	Supports multidimensional array
torchvision	Yes	No	No	No	No	No
scikit-image	Yes	No	No	No	No	No
opencv	Yes	Yes	No	No	No	No
tf.image	Yes	Yes	Yes	Yes	Yes	Yes
Kornia	Yes	Yes	Yes	Yes	Yes	Yes

Source of above summary: Research paper

Components of the library

• kornia.augmentation - module for GPU-based image augmentation
• kornia.utils - image to tensor conversion utilities and metrics
• kornia.color - module for color space conversions
• kornia.morphology - module for morphological image transformations
• kornia.losses - collection of several loss functions
• kornia.geometry - for geometric CV tasks like image conversion using various camera models
• kornia.enhance - module for intensity transformations and normalization
• kornia.filters - for edge detection and image filtering use cases
• kornia.feature - for feature setection

Practical implementation

Here, we demonstrate three use cases of Kornia - blurring a custom image, changing its color space, and adjusting its colors. The code is test on Google colab with Python 3.7.10 and Kornia 0.4.1 versions.

We have used the following image for demonstration:

Image source

Step-wise explanation of the code is as follows:

First, install the library using pip command.

!pip install kornia

Import required libraries and modules

 import torch
 import torchvision
 import kornia
 import cv2
 import numpy as np
 import matplotlib.pyplot as plt 

Image blurring using Kornia

1. Read the input image 

 #Read the image using OpenCV and convert it to a numpy array
 input_image: np.ndarray = cv2.imread('img2.jpg')
 #Convert the image color space from BGR to RGB
 input_image = cv2.cvtColor(input_image, cv2.COLOR_BGR2RGB) 

2. Convert the image to a torch 4D tensor 

tensor_image: torch.tensor = kornia.image_to_tensor(input_image, keepdim=False)

3. Create an operator that blurs the tensor image using Gaussian filter 

gaussian = kornia.filters.GaussianBlur2d((11, 11), (10.5, 10.5))

Where, (11,11) is the size of the kernel and (1.05,10.5) is the standard deviation of the kernel

4. Convert the tensor image to float type and apply the gaussian operator defined in previous step to blur the image

blur_image: torch.tensor = gaussian(tensor_image.float())

5. Convert the blurred tensor image back to a numpy array

final_blur_image: np.ndarray = kornia.tensor_to_image(blur_image.byte())

6. Plot the input and output images

 fig, ax = plt.subplots(1, 2, figsize=(16, 10))
 ax = ax.ravel() #flatten the axis to a 1D array
 #Original image
 ax[0].axis('off')  #turn off the axis lines and labels
 ax[0].set_title('Original image')  #Title of image
 ax[0].imshow(input_image)   #Display the image
 #Blurred image
 ax[1].axis('off')    #turn off the axis lines and labels
 ax[1].set_title('Blurred image')   #Title of image
 ax[1].imshow(final_blur_image)  #Display the image 

Output:

Color space conversion using Kornia

1. Read the input colored image and convert it to a numpy array

bgr_image: np.ndarray = cv2.imread('img2.jpg', cv2.IMREAD_COLOR)

2. Convert the image to a torch tensor

tensor_bgr: torch.Tensor = kornia.image_to_tensor(bgr_image, keepdim=False)

3. Define functions to flip the image horizontally and vertically and rotate it by 180 degrees.

 def horizontal_flip(input: torch.Tensor) -> torch.Tensor:
     return torch.flip(input, [-1])
 #torch.flip() reverses the order of the input tensor image along given axis 
 def vertical_flip(input: torch.Tensor) -> torch.Tensor:
     return torch.flip(input, [-2])
 def rotate_180(input: torch.Tensor) -> torch.Tensor:
     return torch.flip(input, [-2, -1]) 

4. Define a function to plot a batch of images

 def imshow(input: torch.Tensor):
 #Create a grid with 2 rows
 output: torch.Tensor = torchvision.utils.make_grid(input, 
 nrow=2, padding=5)
 #Convert the grid of images to numpy ndarray
       output_np: np.ndarray = kornia.tensor_to_image(output)
       plt.imshow(output_np)  #Plot the grid
       plt.axis('off')  #Turn off the axis lines and labels
       plt.show()   #Display the grid 

5. Create a batch of images from BGR image

#Apply the functions defined in step (3) to the input tensor image and then #concatenate resulting tensors
 batch_bgr = torch.cat([tensor_bgr, horizontal_flip(tensor_bgr), vertical_flip(tensor_bgr), rotate_180(tensor_bgr)])
 #Display the batch of images
 imshow(batch_bgr)   

Output:

6. BGR to RGB color space conversion

 batch_rgb = kornia.bgr_to_rgb(batch_bgr)  #conversion
 imshow(batch_rgb) #display the resulting  

Output:

7. RGB to Grayscale color space conversion

 batch_gray = kornia.rgb_to_grayscale(batch_rgb.float() / 255.) 
 imshow(batch_gray)  #display the grayscale image 

Output:

8. RGB to HSV (Hue, Saturation, Value) color space conversion

 batch_hsv = kornia.rgb_to_hsv(batch_rgb.float() / 255.)
 imshow(batch_hsv[:, 2:3])  #display HSV image 

Output:

Color adjustment

1. Read the input image in BGR format

bgr_image: np.ndarray = cv2.imread('img2.jpg', cv2.IMREAD_COLOR)

2. Convert input image to torch tensor

tensor_bgr: torch.Tensor = kornia.image_to_tensor(bgr_image)

Convert the image from BGR to RGB

tensor_rgb: torch.Tensor = kornia.bgr_to_rgb(tensor_bgr)

3. Expand the image 

 tensor_rgb = tensor_rgb.expand(4, -1, -1, -1)  
 # 4 x Channels x Height x Width
 tensor_rgb = tensor_rgb.float() / 255. #Normalize the expanded image 

Define a function to create a batch of images

 def imshow(input: torch.Tensor):
 #create grid having 2 rows of images
     output: torch.Tensor = torchvision.utils.make_grid(input, nrow=2, 
     padding=5)
 #Convert tensor images to numpy ndarray
     output_np: np.ndarray = kornia.tensor_to_image(output)
 #Plot the grid of images
     plt.imshow(output_np)  
     plt.axis('off')
     plt.show() 

Display the batch of RGB images

imshow(tensor_rgb)

Output:

4. Adjust brightness of the batch images

 tensor_brightness: torch.Tensor = kornia.adjust_brightness(tensor_rgb, 0.6)
 #Where, 0.6 is the factor to adjust brightness of each element in the batch
 imshow(tensor_brightness)  #display the resulting batch 

Output:

5. Adjust contrast of the batch images

 tensor_contrast: torch.Tensor = kornia.adjust_contrast(tensor_rgb, 0.2)
 #0.2 is the contrast adjustment factor per batch element
 imshow(tensor_contrast)  #dispaly the resulting batch 

Output:

6. Adjust gamma correction of the images

 tensor_gamma: torch.Tensor = kornia.adjust_gamma(tensor_rgb, gamma=3., gain=1.5)
 #where, ‘gamma’ is a non-negative number and ‘gain’ is the constant multiplier
 imshow(tensor_gamma)  #display resulting batch 

Output:

7. Adjust saturation level of the batch images

 tensor_saturated: torch.Tensor = kornia.adjust_saturation(tensor_rgb, 0.2)
 #where 0.2 is the saturation factor
 imshow(tensor_saturated)  #display the resulting batch 

Output:

8. Adjust hue of the images

 tensor_hue: torch.Tensor = kornia.adjust_hue(tensor_rgb, 0.5)
 #where 0.5 gives measure of how much to shift the hue channel
 imshow(tensor_hue)  #display the resulting batch 

Output:

• Google colab notebook of the above-explained implementation is available here.

References

To get in-depth understanding of the Kornia library, refer to the following web links:

• Research paper
• Official documentation
• GitHub repository