How I Created The Credit Card Reader Using OpenCV?

Optical character recognition has seen many applications in machine learning and deep learning. One such practical application of OCR is for identification of credit card type and number on the card. This could be of great help of banks and other financial institutions for digitally recognising the card numbers and type of card. It can also be useful for UPI services where the user can just scan the card instead of typing in the details. OCR is usually handled by a library called Tesseract. But, here we will use OpenCV to read the credit card and identify the type and number on the card. 

The purpose of this article is to build a credit card reader with OpenCV and machine learning techniques to identify the card number and the card type. Let us get started. 

Getting a Font Reference

On a typical credit card, you can see that the numbers are in a slightly different font. For these numbers to be read by the machine, we need to show the machine and save the font of these numbers. The font on the cards is called MICR which stands for magnetic ink character recognition code. So, let us now write the code in OpenCV and perform a template matching so that this font is understood for recognition later. 

Import the Required Libraries

import cv2
import numpy as np
from matplotlib import pyplot as plt

Get the reference image

The image used for this is shown below. 

This is the type of font that is used in the cards. You can download this image here. Once this is done, let us read the image and perform some basic image processing on this. We will convert the image to greyscale and apply a binary inversion threshold on it.

Download our Mobile App

template = cv2.imread('numbers.png')
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
template = cv2.threshold(template, 10, 255, cv2.THRESH_BINARY_INV)

The output of this is as shown below. 

Getting the contours for the image

Next, we need to find contours in the above image and after locating each digit we will sort them and put them in a dictionary for further reference. To contour the image we will do the following steps. 

get_contour = cv2.findContours(template.copy(), cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
get_contour = imutils.grab_contours(get_contour)
get_contour = contours.sort_contours(get_contour, method="left-to-right")
numbers = {}

Next, we will find a bounding box for these digits by looping over them and then finding the region of interest and then appending them to the dictionary created above. We will also define two kennels one rectangle and one square to get the structure of the boxes.

for (key,val) in enumerate(get_contour):
    (s1, s2, s3, s4) = cv2.boundingRect(c)
    region= template[s2:s2 + h, s1:s1 + s3]
    region= cv2.resize(region, (57, 88))
    numbers[key] = region
kernel_1 = cv2.getStructuringElement(cv2.MORPH_RECT, (10, 5))
kernel_2 = cv2.getStructuringElement(cv2.MORPH_RECT, (4, 4))

The dictionary contains keys as the numbers for 0 to 9 and the values are arrays which represent these numbers.

Now that we have the template matching ready for the card, we can go ahead and select a credit card image and read that. I have selected an image from google. You can download the image by clicking on this link.

Image processing in OpenCV

Let us now read this image and perform some basic pre-processing on the image. 

card = cv2.imread('master.png')
card = imutils.resize(card, width=300)
card_grey = cv2.cvtColor(card, cv2.COLOR_BGR2GRAY)

The image in greyscale looks like this

Now we need to separate the light-coloured numbers from the dark background so that they can be detected. To do this, we will use a morphological operation called top-hat. Then, we will use a gradient called Sobel gradient and scale the image. 

Morphology and scaling

morph = cv2.morphologyEx(card_grey, cv2.MORPH_TOPHAT, kernel_1)
sobel = cv2.Sobel(morph, ddepth=cv2.CV_32F, dx=1, dy=0, ksize=-1)
sobel = np.absolute(sobel)
(minimum, maximum) = (np.min(sobel), np.max(sobel))
sobel = (255 * ((sobel - minimum) / (maximum - minimum)))

The morphed image looks like this:

The scaled image is as follows

As you can see above, there are gaps that exist between the numbers and it can become a hurdle for the machine as they consider even the gaps as characters. To close them we need to use a technique called Otsu threshold. 

otsu = cv2.morphologyEx(sobel, cv2.MORPH_CLOSE, kernel_1)
out = cv2.threshold(otsu, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
out = cv2.morphologyEx(out, cv2.MORPH_CLOSE, kernel_2)

The next step is to group the digits and find their locations and apply the bounding boxes around them. To do this we will have to loop through the contours and apply the boxes. 

Bounding boxes

contours = cv2.findContours(out.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = imutils.grab_contours(contours)
locations = []
for (key, vals) in enumerate(contours):
    (s1, s2, s3, s4) = cv2.boundingRect(vals)
    arr = s3 / float(s4)
    if arr > 2.5 and arr < 4.0:
        if (s3 > 40 and s3 < 55) and (s4 > 10 and s4 < 20):
            locations.append((s1, s2, s3, s4))
locations = sorted(locations, key=lambda s1:s1[0])
final_locs = []

Now that we got the locations and applied the boxes around the numbers, we now need to perform the final part which is the OCR on these 4 groups of numbers. 

Extracting the numbers

To do this phase we will go over the groups of digits and collect the region of interests from the bounding boxes. Then we will apply contours on them and sort the numbers from left to right. Next, we will extract the numbers in the bounding boxes and then compare them with our template to identify the digits. Finally, we will apply a correlation-based template matching and store the outputs. 

for (idx, (group1, group2, group3, group4)) in enumerate(locations):
    detection = []
get_group = gray[group2 - 5:group2 + group4 + 5, group1 - 5:group1 + group3 + 5]
    get_group = cv2.threshold(get_group, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
    nums = cv2.findContours(get_group.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    nums = imutils.grab_contours(nums)
    nums = contours.sort_contours(nums, method="left-to-right")[0]
    for val in nums:
        (s1, s2, s3, s4) = cv2.boundingRect(val)
        regions = get_group[s2:s2 + s4, s1:s1 + s3]
        regions = cv2.resize(regions, (57, 88))
        final_vals = []
        for (num, numROI) in numbers.items():
            res = cv2.matchTemplate(regions, numROI, cv2.TM_CCOEFF)
            (_, mark, _, _) = cv2.minMaxLoc(res)
            final_vals.append(mark)
        detection.append(str(np.argmax(final_vals)))
    cv2.rectangle(image, (group1 - 5, group2 - 5),
        (group1 + group3 + 5, group2 + group4 + 5), (0, 0, 255), 2)
    cv2.putText(image, "".join(detection), (group1, group2 - 15),
        cv2.FONT_HERSHEY_SIMPLEX, 0.65, (0, 0, 255), 2)
    final_locs.extend(detection)

Lastly, for the detection of the type of card, we need a dictionary that contains the first numbers as key and name of the card as values. Usually, in Indian cards, visa cards start with 4 and MasterCard start with 5.

starting_digit = {
    "4": "Visa",
    "5": "MasterCard"
}

Now we can print the output

print("Credit Card Type: {}".format(starting_digit[final_locs[0]]))
print("Credit Card digits: {}".format("".join(final_locs)))
plt.imshow(cv2.cvtColor(card, cv2.COLOR_BGR2RGB))
plt.title('card')
plt.show()

I will show the output for a visa card as well. I have just replaced the image and ran the program again.

Conclusion

In this article, we saw how to build a credit card reader using OpenCV and OCR. We also identified the type of card with the help of the first number. With OpenCV, building an optical character recognition is simple and is accurate as well. Similar to cards this can be applied on cheques as well.