A deep dive into MeanIoU - An evaluation metric for object detection

Object detection is a computer vision approach that allows items in an image or video to be identified and located. Object detection may count items in a scene and establish and monitor their precise positions while precisely labelling them using this type of identification and localisation. Item detection is just recognising an object in an image using a bounding box, whereas image classification simply classifies whether an object is in the picture or not based on the likelihood. Intersection over union is a fundamental evaluation method for evaluating the accuracy of the computer vision algorithm. This article will be focusing on the working of the Mean Intersection over union (MeanIoU) and its implementation. Following are the topics to be covered.

Table of contents

1. What is object localization?
2. What is the Mean Intersection of Union?
3. How does it work?
4. Evaluating the detection with MeanIoU

Object detection combines localization and classification tasks to detect one or more objects in an image. Let’s understand object localization.

What is object localization?

The job of identifying an instance of a certain item category in a picture, often by defining a narrowly cropped bounding box centred on the instance, is known as object localization. A candidate bounding box is specified in an object proposal, and an object proposal is considered to be a correct localization if it substantially overlaps a human-labelled "ground-truth" bounding box for the provided item. Simply said, object localization seeks to identify the principal or most apparent object in an image, whereas object detection seeks to identify all objects and their borders.

A model for image classification or image recognition essentially determines the likelihood of an item in a picture. Item localization, on the other hand, relates to determining the position of an object in a picture. An object localization algorithm will return the coordinates of an object's position with the picture. The most common technique in computer vision to localize an item in an image is to use bounding boxes to indicate its location.

Are you looking for a complete repository of Python libraries used in data science, check out here.

What is the Mean Intersection of Union?

The intersection over union (IOU) method is one of the most fundamental methods used in machine learning to compare data samples. The Jaccard Index is also known as IoU in statistics. Object detection, object tracking, semantic segmentation, and other computer vision tasks use IoU as an evaluation method.

In general, IoU calculates how closely two sets of elements overlap. Using a bounding box regression, where A represents the ground truth box and B represents the prediction box, we can form the IoU based on the coordinates of the left-top corner, width, and height of A's bounding box.

How does it work?

MeanIoU calculates the ratio of area overlapped by the two bounding boxes to the area of their union. The two bounding boxes are the real observation and the prediction. Any algorithm which returns the prediction bounding box is eligible for this metric. Its mathematical formulation is:

Let’s understand this with an example, below is an image of some cars. The primary single object is the grey car. The red bounded box is the prediction made by the machine learning algorithm and the green bounded box is the real observation. For calculating the IoU two things are required according to the formula. 

• The total area covered by both the bounding boxes (union)
• The area common between the bounding boxes (intersection)

By dividing the intersection and the union of the bounding boxes the IoU could be found. The higher the value of the MeanIoU higher is the accuracy of the model.

Evaluating the detection with MeanIoU

This article uses a basic example to understand the metric by drawing rectangular bounding boxes around the object in the image, the same as the result of any object recognition algorithm. Two boxes would be drawn in the image one represents the ground truth and the other prediction.

Let’s start with importing the necessary libraries

from collections import namedtuple
import numpy as np
import cv2
from google.colab.patches import cv2_imshow
import matplotlib.pyplot as plt

If using Google Colab then must install cv2_imshow because the OpenCV's ‘imshow’ function crashes the runtime.

For this article using a custom build Mean IoU function.

def mean_intersection_over_union(boxA, boxB):
  xA = max(boxA[0], boxB[0])
  yA = max(boxA[1], boxB[1])
  xB = min(boxA[2], boxB[2])
  yB = min(boxA[3], boxB[3])
  interArea = (xB - xA) * (yB - yA)
 
  boxAArea = (boxA[2] - boxA[0] + 1) * (boxA[3] - boxA[1] + 1)
  boxBArea = (boxB[2] - boxB[0] + 1) * (boxB[3] - boxB[1] + 1)
 
  iou = interArea / float(boxAArea + boxBArea - interArea)
  return iou

The function is based on the formula discussed above, calculating the intersection and union of the bounding boxes and then dividing them.

Applying the formula to a first image

dect=[Detection("/content/drive/MyDrive/Datasets/object_detection_img/IMG_9929.jpg", [430, 160, 110, 280], [405, 110, 120, 290])]
for i in dect:
  image = cv2.imread(i.image_path)
  resized = cv2.resize(image, (800, 448))
  cv2.rectangle(resized, tuple(i.ground_truth[:2]), 
    tuple(i.ground_truth[2:]), (0, 255, 0), 2)
  cv2.rectangle(resized, tuple(i.prediction[:2]), 
    tuple(i.prediction[2:]), (0, 0, 255), 2)
  iou = mean_intersection_over_union(i.ground_truth, i.prediction)
  cv2.putText(resized, "IoU: {:.4f}".format(iou), (10, 30),
    cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
  print("{}: {:.4f}".format(i.image_path, iou))
  cv2_imshow(resized)

Two bounding boxes have been drawn using the ‘cv2.rectangle’ and then applied the custom MeanIoU function on the image and got a score of 0.7441 which is fine. The red bounded box is the prediction and the green bounded box is the grounded truth box. This tells that there is a difference of 0.2559 between the prediction and ground truth bounding box. The prediction bounding box doesn’t overlap completely with the ground truth box.

Applying the formula to a second image

dect_1=[Detection("dc-Cover.jpeg", [760, 40, 190, 200], [750, 40, 180, 210])]
for i in dect_1:
  image_1 = cv2.imread(i.image_path)
  resized_1 = cv2.resize(image_1, (800, 448))
  cv2.rectangle(resized_1, tuple(i.ground_truth[:2]), 
    tuple(i.ground_truth[2:]), (0, 255, 0), 2)
  cv2.rectangle(resized_1, tuple(i.prediction[:2]), 
    tuple(i.prediction[2:]), (0, 0, 255), 2)
  iou = mean_intersection_over_union(i.ground_truth, i.prediction)
  cv2.putText(resized_1, "IoU: {:.4f}".format(iou), (10, 30),
    cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
  print("{}: {:.4f}".format(i.image_path, iou))
  cv2_imshow(resized_1)

Similar to the above image two bounding boxes are drawn marking the primary single object. The red bounded box is the prediction and the green bounded box is the grounded truth box. This time a score of 0.96 is returned, by this it could be said that the two boxes are almost overlapping and the model has performed well.

Conclusion

The mean intersection of union is a common method for evaluating image segmentation and object detection. It uses the true positive, false negative, and false positive to calculate the score. With this article, we have understood the concept of the Mean Intersection of union and its implementation.

References

• Link to the above code
• Tensorflow documentation about MeanIoU