Is Biologically-Inspired Deep Learning Scalable?

Artificial Intelligence has taken much inspiration from biology and nature. Geoffrey Hinton, the British cognitive psychologist and computer scientist, also cited the human brain as the primary motivator for his work in neural networks and the backpropagation of error algorithm.

However, many experts are of the firm opinion that it is very hard to implement the back-propagation of error algorithm in the real human brain. The huge success of deep learning in recent years has prompted many experts to look more closely into the brain to see if any other ideas can be ported back into the real world. There are explorations into how the real human brain can be a playground for a grand back-propagation algorithm experiment.

Now a team led by Hinton and Sergey Bartunov along with Adam Santoro, Blake A Richards and Timothy P Lillicrap have presented research that delves into how one can scale deep learning systems that are largely biologically-motivated. The researchers presented the performance and the results on the MNIST, CIFAR10, and ImageNet datasets. The researchers set out to explore many varieties of algorithms such as target-propagation (TP) and feedback alignment (FA). One of the aims of the research is also to establish baselines that will be very important for biologically-motivated deep learning systems going forward. The results and implementation details laid out by the team will act as a base for many researchers.

Discussions And Debate Over The Backpropagation Algorithm

Even though the backpropagation algorithm has been the method that underlies most of the advances we are seeing in the AI field today, there is very less evidence that the brain performs backpropagation. Montreal Institute for Learning Algorithms (MILA) found out that backpropagation is purely linear, whereas biological neurons interleave linear as well as non-linear operations. Hinton is now very sceptical about the algorithm and went on to say that he wants to throw away his old beliefs and “start again.”

The debate around the algorithm is nothing new. There have been some weak objections about BP which were based on the design of the structure of artificial neural networks. According to the paper, more serious concerns are:

  1. “The need for the feedback connections carrying the gradient to have the same weights as the corresponding feedforward connections”
  2. “The need for a distinct form of information propagation (error feedback) that does not influence neural activity, and hence does not conform to known biological feedback mechanisms underlying neural communication.”

Scaling Learning Algorithms

The researchers observed that humans learn tasks which are not related to evolution. They therefore deduced that the human brain has a very generalised learning and very powerful algorithm for shaping behaviour. There are many algorithms that are somehow based on the biological phenomena in the brain. There are three major concerns that are stated by the researchers:

  1. The only algorithm, target propagation (TP) explored empirically depends on gradient computation via backpropagation.
  2. The algorithms have not been rigorously tested on datasets more difficult than MNIST.
  3. The algorithms have not been used in modern architectures that are way more complicated than old multi-layer perceptrons (MLPs).

The researchers said that the second point is the most worrying one. It is the most important aspect because it may tell us about whether the approach may work in a brain-like scenario. Accuracy and performance on a very small number of ML tasks with a model that does not have adaptive neural phenomena such as the varieties of plasticity, evolutionary priors, etc tell us about the suitability of an algorithm as well.

It tells us how important it is that researchers invent and discover algorithms that are biologically plausible and also scale on large datasets. Having a great and powerful learning algorithm is really important. The research activity proposed by them therefore focuses on:

  1. The sufficiency of a learning algorithm
  2. The impact of biological constraints in a network

Experiments And Results

The researchers wanted to understand the limits and constraints of some biologically-inspired algorithms. They manually searched for architectures well suited to such algorithms and went on to tweak those architectures for BP and FA (feedback alignment algorithm) versions and ran independent hyperparameter searches for each learning method.

Recent research has put an onus on using back-propagation algorithm to understand the brain which was lost in some of the last decades. In this study, the researchers studied TP and FA and composed and present a simple version of the DTP algorithm that didn’t need gradient propagation and weight transport. The researchers demonstrated that networks trained with SDTP without any weight sharing, i.e. weight transport in the backward pass or weight tying in convolutions, perform much worse than DTP — because of impoverished output targets.

More Great AIM Stories

Abhijeet Katte
As a thorough data geek, most of Abhijeet's day is spent in building and writing about intelligent systems. He also has deep interests in philosophy, economics and literature.

More Stories


8th April | In-person Conference | Hotel Radisson Blue, Bangalore

Organized by Analytics India Magazine

View Event >>

30th Apr | Virtual conference

Organized by Analytics India Magazine

View Event >>

Yugesh Verma
All you need to know about Graph Embeddings

Embeddings can be the subgroups of a group, similarly, in graph theory embedding of a graph can be considered as a representation of a graph on a surface, where points of that surface are made up of vertices and arcs are made up of edges

Yugesh Verma
A beginner’s guide to Spatio-Temporal graph neural networks

Spatio-temporal graphs are made of static structures and time-varying features, and such information in a graph requires a neural network that can deal with time-varying features of the graph. Neural networks which are developed to deal with time-varying features of the graph can be considered as Spatio-temporal graph neural networks. 

Yugesh Verma
A guide to explainable named entity recognition

Named entity recognition (NER) is difficult to understand how the process of NER worked in the background or how the process is behaving with the data, it needs more explainability. we can make it more explainable.

Yugesh Verma
10 real-life applications of Genetic Optimization

Genetic algorithms have a variety of applications, and one of the basic applications of genetic algorithms can be the optimization of problems and solutions. We use optimization for finding the best solution to any problem. Optimization using genetic algorithms can be considered genetic optimization

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM