Once upon a time, when we had simpler questions like “What is the problem?”, structured datasets were used to report numbers. Fast forward a few decades, we have more complex questions to answer like “Why is this problem happening?” and with complex problems comes complex datasets or unstructured datasets. To rescue us from all this complexity comes neural network, making machines learn and resolve all those complexities for us, scaling out with each level of complexity.

One of such complex problem is handwriting recognition, imagine handwriting on a paper or a tablet and getting that translated into a computer typed text, no more redo! Imagine not wracking your brains into deciphering a doctor’s handwriting. Imagine a child with dysgraphia, a condition that results in poor handwriting, not struggling in the classroom. 

All this can happen with the handwriting recognition tool, which classifies text from an image. This tool has a Graphical User Interface, where inside a canvas, a user can write any English word in freehand style, and the model sitting at the backend will be able to recognize the word. For this tool, Multi-Layer Perceptron (MLP) classifier with Adam solver and sigmoid function has been used to achieve significant results.

Dataset

Dataset used was created by the National Institute of Standards and Technology (NIST). The NIST Special Database 19 consists of roughly 0.7 million sample png images. The current model has been trained only for uppercase letters (A-Z).  The following table highlights the number of observations per character: 

Table 1: Number of Observations Per Characters

Preprocessing

Each character in the original dataset occupies 128×128 pixels per raster (Fig 1 a), to avoid heavy computation the size of the image was reduced to 56 x 56 pixels (Fig 1 b). Furthermore, canvas size was reduced to 28×28 pixels by removing the padding (Fig 1 c) which resulted in a 784 feature configuration dataset. Each character was labelled sequentially from “A”- “Z”. 

The package ‘tkinter’ was used to create the canvas-like user interface. Once a user writes a word in the canvas, the tool converts the image into a NumPy 2-D array and then traverses the array column-wise looking for a filled pixel to mark the beginning of a letter. For words, the model continues to traverse and look for a column where there is significant relative blank space to mark the beginning of the second character. The tool is intelligent enough to differentiate a break in letters versus the beginning of a second letter.

Challenges

While designing and creating this tool, several challenges were faced highlighted below:

#1: The original dataset demanded heavy computational power to hyper tune the model for different combinations. To overcome this limitation with the personal computer that was used to build this model, the dataset was split into batches of five characters (i.e. letters A-E and F-I, etc.), and the model was trained and tested using these batches. 

#2: An imbalance in the number of observations among the characters in the dataset, resulted in complications during the testing and training phase (i.e. for example 10k and 2.5k images for J and K respectively). To overcome this a script was created that divided the test and train dataset for each character individually, and later merged them to ensure a well-balanced dataset.

#3: English alphabets contained letters that appeared very similar to each other (i.e. B and P, D and O). While training and testing, the model struggled to classify these letters accurately. To minimize misclassification for these letters, the model was trained and hyper tuned for these letters separately with a larger dataset. 

Neural Network Model Configuration

For this tool, Multi-Layer Perceptron (MLP) classifier has been trained using backpropagation to achieve significant results. Below is the configuration of the neural network:

• Hidden Layer Size: (100,100,100) i.e., 3 hidden layers with 100 neurons in each
• Activation Function: logistic sigmoid, returns f(x) = 1 / (1 + exp(-x))
• Solver for weight optimization: stochastic gradient-based optimizer (“Adam”)
• Early Stopping (to avoid overfitting): True

Figure 4: Model Architecture

Results

Table 2: Results Summary

Future Expansion 

While this tool serves as a base model in bridging the communication gap, there is more work that needs to be done. Currently, the model can decrypt letters and words, but it is capable of processing phrases and paragraphs with proper expansion. Additionally, the UI/ UX can be further developed to be leveraged by a wider user-audience.