Silicon Brains: Designing Self Organising Neural Networks

A healthy child’s brain under development is capable of adding nearly 250,000 neurons every minute! At birth, a brain has almost all the neurons that it will ever have. The brain continues to grow for a few years after a person is born and by the age of 2 years old.

Thanks to the glial cells, the brain continues to grow. Glia continues to divide and multiply and is responsible for carrying out many important functions including insulating nerve cells with myelin.

The phenomenon that is intelligence emerges across many species in various ecosystems and is linked to the nervous system.

Though we have figured out what part of our body is responsible for intelligence, the ‘HOW’ part is still ambiguous. And since the world today is driven by artificial logical machines that try to mimic intelligence humans, it is quite necessary to crack the enigma that is intelligence.

In an attempt to investigate if we could replicate this kind of self-organised growth in algorithms, researchers from Caltech propose a biologically inspired developmental algorithm that can ‘grow’ a functional, layered neural network from a single initial cell. 

The objective here is to grow a neural network that autonomously grows from a single computational “cell" followed by self-organization of its architecture.

Overview Of The Model

The process begins with the growth of a layered neural network followed by self-organization of its inner connections to form defined ‘pools’ or receptive fields.

The algorithm made to learn to organize inter-layer connections to construct a convolutional pooling layer.

The act of self-organization typically consists of the following two key elements:

• A spatiotemporal wave generator in the first layer and 
• A local learning rule in the second layer to learn the “underlying” pattern of activity generated in the first layer.

In the above illustration, the authors in their paper, show how nodes form with time and training. The red-nodes indicate active-nodes (firing), black nodes refer to silent nodes and the arrows denote the direction of time.

The learning rule here is where the processing units are modelled as rectified linear units (ReLU) and are implemented as a modified Hebbian rule for tuning the inter-layer weights to achieve the same. The Hebbian Learning Rule is a learning rule that specifies how much the weight of the connection between two units should be increased or decreased in proportion to the product of their activation. After this, the individual ReLU units compete with one another in a winner take all fashion.

Initially, every processing unit in the second layer is connected to all input-sensor nodes in the first layer. 

Having coupled the spontaneous spatiotemporal wave generator and the local learning rule, an initially fully connected two-layer network becomes a pooling architecture.

Here the spatiotemporal wave generator is a dynamical system that is modelled around a bunch of mathematical expressions as follows:

The process of growing a layered neural network involves two major sub-processes:

1. Every ‘node’ can divide horizontally to produce nodes that populate the same layer; 
2. Every node can divide vertically to produce processing units that migrate upwards to populate higher layers.

To test the functionality of these networks, the two-layered network is coupled with a linear classifier that is trained to classify hand-written digits from MNIST on the basis of the representation.

The results show that self-organized networks classify with a 90% test accuracy, are statistically similar to hand-crafted pooling networks and are statistically better than random networks.

Key Takeaways

• The algorithm is adaptable to a wide-range of input-layer geometries, robust to malfunctioning and can be used to successfully grow and self-organize pooling architectures of different pool-sizes and shapes. 
• It also provides a procedure for constructing layered neural networks through self-organization. 
• Broadly, this work shows that biologically inspired developmental algorithms can be applied to autonomously grow functional ‘brains’ in-silico. 

The authors in this work, state that their algorithm is robust and is capable of countering the key challenges accompanying growth. 

They also demonstrate that ‘grown’ networks are functionally similar to that of hand-programmed pooling networks on conventional image classification tasks. And since CNN’s represent a model class of deep networks, the authors believe that this technique can be broadly implemented for the self-organisation of intelligent systems.

Know more about this work here.