Now Reading
Improving Deep Learning Model Robustness By Adding Noise Using Keras

Improving Deep Learning Model Robustness By Adding Noise Using Keras

Amal Nair

It is practically impossible to remove noise completely from a dataset which justifies the high usage of probability and statistics in machine learning models. Probability helps to quantify uncertainty caused due to noise in a prediction. Overfitting is a major problem as far as any machine learning algorithm is concerned. And it has been proven that adding noise can regularise and reduce overfitting to a certain level.

With this tutorial, we will take a look at how noise can help achieve better results in ML with the help of the Keras framework. This tutorial has referenced and was inspired by  Jason Brownlee’s tutorial on How to Improve Deep Learning Model Robustness by Adding Noise.

Gaussian Noise Layer

The Gaussian Noise Layer in Keras enables us to add noise to models. The layer requires the standard deviation of the noise to be specified as a parameter as given in the example below:



The Gaussian Noise Layer will add noise to the inputs of a given shape and the output will have the same shape with the only modification being the addition of noise to the values.

Ways Of Fitting Noise To A Neural Network

Fitting to input Layer

 

 

Between hidden layers in the model

Before the activation function.

 

         

After the activation function.

 

Noise Regularisation

Let us understand the effect of noise on a binary classification problem with the help of a simple example. We will consider a binary classification problem that defines two 2D concentric circles of observations. We will use the sklearn.datasets.make_circles function to generate our sample dataset.

Code:

The dataset would look like this:

The dataset consists of 2 independent variables ‘x’ and ‘y’ of the same scale and a dependent variable ‘label’ with values either 0 or 1.

Plotting the dataset:

Code:

Output:

We can see that the data points collectively results in almost 2 circle-like shapes, red and blue with the same centre. Hence the name “circles” dataset.

Now that we have our sample dataset ready, we will create Machine Learning models to analyse the result of noises in a dataset.

Overfitting a Multilayer Perceptron

We will first create a traditional Multilayer Perceptron model. Adding more nodes than what may be required to solve the problem and training the model for a long time can ensure overfitting.

From the 100 samples, we generated We will use 30 observations to train our model and 70 observations to validate or test our model’s performance.

Code:

Output

Train: 1.000, Test: 0.757

Observation

  • The model achieved a 75% accuracy on the test set.
  • Test accuracy of the overfit model  increases to a point and then begins to decrease again.

MLP With Input Layer Noise

On a small dataset inducing noises can have the effect of creating more samples or resampling the domain, making the structure of the input space artificially smother. This will help in making the problem easier to learn and improves generalisation performance.

See Also
Deep Autoencoder in PyTorch

We will now add a Gaussian Noise layer as the input layer and Gaussian Noise with a mean of 0.0 and a standard deviation 0f 0.01.

Code:

Output       

       Train: 1.000, Test: 0.771

Observation

  • The model achieved a 77% accuracy on the test set.
  • The noise causes the accuracy of the model to jump around during training, possibly due to the noise-introducing points that conflict with true points from the training dataset.
  • Test accuracy of the model increases to a point and then begins to decrease again showing the presence of overfitting.

MLP With Hidden Layer Noise

We will now try the alternate approach of adding the Gaussian Noise Layer as the Hidden Layer.

Code:

Output

Train: 0.967, Test: 0.814

Observation

  • The model achieved an 81% accuracy on the test set.
  • The model no longer appears to show the properties of overfitting by keeping an almost constant accuracy after a threshold similar to the behaviour of the training set data.

Note:

  • We can also add the noise after the activation function.
  • Your values in the results will vary, given both the stochastic nature of the learning algorithm and the stochastic nature of the noise being added to the model

Provide your comments below

comments


If you loved this story, do join our Telegram Community.


Also, you can write for us and be one of the 500+ experts who have contributed stories at AIM. Share your nominations here.

Copyright Analytics India Magazine Pvt Ltd

Scroll To Top