Troubleshoot 101: How to fix your neural network

AI pioneer Geoffrey Hinton and his students, Alex Krizhevsky and Ilya Sutskever, published a paper on convolutional neural networks in 2012 at NeurIPS 2012. The team proposed a mathematical system mimicking brain neurons; machine learning hasn’t been the same since. The neural network changed the game, and suddenly even AGI seemed to be in sight.

Now, a lot of researchers are getting into the thick of neural networks. But, building a neural network from scratch is no small task and the margin for error is small.

We made a list of common mistakes developers make when building neural networks and ways to fix them:

Dataset issues

• Start by checking your input data. Print/display a couple of input and target output batches to ensure the results are fine. You can also try to input random data to see if the error continues. If it does, it means the neural network is turning data into garbage. So you have to start debugging layer by layer/op by op to locate the issue.
• Even if your data is fine, the code that pushes the input to the neural network might be broken. Hence, it’s important to check the input of the first layer. Further, there are chances that the input is not connected to the output. So you'd need to check a few samples to see if they are labelled correctly and work the same way for output labels.
• In many cases, the relationship between input and output could be too random or not sufficiently related. Unfortunately, there isn't a standard way to detect or address this issue, as it depends on the nature of the data.
• Noisy datasets are a common problem. However, the only way to rectify it is by manually checking the labels. If noise is not the problem, maybe your dataset hasn’t been shuffled properly or is ordered by labels. Either way, it could negatively impact the learning. You can avoid this by shuffling input and labels together and ensuring your batches don’t contain a single label.
• If it’s none of the above, you might either be using a very large batch that hinders the generalisation ability of the model or lack good training samples, to begin with.

Data Normalization/Augmentation

• You need to standardise your input to have zero mean and unit variance.
• Augmentation has a regularising effect. Excessive data augmentation with other forms of regularisation can lead to underfitting.
• Check the pre-processing for training/validation/testing dataset.

Implementation issues

• Initialise with small parameters without regularisation and then increase the regularisation strength.
• Check the loss function for bugs and add unit tests. The network performance will hurt if the loss is incorrect. If you are using a loss function from a certain framework, make sure it is passed through what’s expected. For example, in PyTorch, the NLLLoss requires a softmax input, and CrossEntropyLoss does not. You might also have to adjust loss weights or try testing different combinations of loss weights. However, loss is not the best indicator of network performance. Try monitoring other metrics like accuracy.
• If you’ve implemented any of the layers yourself, double-check to see if they are working properly. There are high chances you could’ve frozen layers/variables by unintentionally disabling gradient updates. And if Gradient Descents are implemented by hand, ensure the backpropagation works.
• Hidden dimension errors and network size could cause issues.

Training issues

• Try different initialisation methods if the weights initialisation leads you to a bad local minimum. For example, He initialisation is meant for layers with ReLu activation, and Xavier works best for layers with sigmoid activation.
• Try changing your hyperparameters. Chances are you might have used a bad set. If you are unsure, try a grid search.
• Use visualisation libraries like Tensorboard and Crayon to monitor each layer's activations, weights, and updates and make sure their magnitudes match.
• Consider using a different optimiser. You can reduce the training time with the right optimiser for a task.
• Look for exploding / vanishing gradients. An ideal standard deviation for the activations should range from 0.5 to 2.0; anything beyond this indicates vanishing or exploding activations.
• Evaluate your network layer by layer and see where the NaNs (Non-a-Number) appear. Try decreasing the learning rate if NaNs appear in the first 100 iterations.
• Frameworks have layers like Dropout, Batch Norm, etc., that behave differently during training and testing. Try switching the mode to see if the network can predict properly.