How Synthetic Gradients Are Used To Optimise Training of Large Neural Networks

The recent developments in neural networks have accelerated in a fast pace, so much so that to a beginner it may seem confounding to restrict himself/herself to just one specific area of expertise. As we go forward, researchers try to find ways in which there is a constant improvement in the science of neural networks. In this article we explore a recent improvement in neural network called ‘synthetic gradients’. This is research coming out of Google’s DeepMind which has a different and interesting take on backpropagation in neural networks. Synthetic gradients have proved to improve communication between multiple neural networks. This is a good sign since it makes the complications involved in deep learning much easier. Backpropagation: How neural networks learn from data Neural