A deep neural network changes its form constantly as it gets trained. The distribution of each layer’s inputs change along with parameters of previous layers. This change increases latency in learning and it gets harder to train as the model embraces nonlinearities.

Since 2012, the capabilities of computer vision systems have improved greatly due to (a) deeper models with high complexity, (b) increased computational power and (c) availability of large-scale labeled data.

The room to refine neural networks still exist as they sometimes fumble and end up using brute force for light weight tasks. To address this, the researchers at Google, have come up with MorphNet.

The objective of the MorphNet is to provide more resources where necessary by using neural networks for tasks which use up more layers and sizing down on low end tasks.

MorphNet offers new solutions for parameter optimisation by inducing better sparsification.

MorphNet’s Targeted Pruning

Source: Google

With every pass, MorphNet learns the number of neurons per layer. When a layer has zero neurons, then that part of the network is cut off. This changes the topology of the network as the residual blocks in the network are removed.

The training can be carried out in a single run. And, MorphNet can be scaled to be ready for application on larger networks.

This technique also equips with the network with portability i.e, there is no need for keeping a track of the checkpoints that arise during training.

This iterative approach of expanding and shrinking the neural networks gives better control over the usage of computational powers and time.

A resource usage index could be done by evaluating the change in FLOPs per inference or model size. These can be markedly different depending on the application domain and corresponding constraints.

FLOPs(Floating point operations per second) are units of measure of performance of a computational operation. A processor with higher range of FLOPs is considered to be more powerful.

Allocating FLOPs(resource) during algorithmic operations(neural networks) is key to the time taken on training and other such fundamental operations. This is where Morphnet claims to make a difference, by expanding and shrinking the layers.

A neural network is nothing but a bunch of matrix multiplication. If thought in these terms, shrinking can be something like skipping the null elements in the matrices which offer no significance but have to be multiplied in traditional multiplication procedure.

For shrinking, a resource weighted sparsifying regulariser on activations is used and for expanding, a uniform multiplicative factor(width multiplier) is used.

Putting MorphNet To Use

The researchers list down a sequence of steps to deploy MorphNet on a sample CNN for image classification. Given an existing model (the “seed network”) and a target criterion, MorphNet will propose a new model by adjusting the number of output channels in each convolution layer.

Deploying MorphNet in Few Steps:

1. Choose a regularizer from morphnet.network_regularizers. The choice is based on
2. Enable the scale parameters (“gamma variables”), i.e., by setting scale=True if you are using tf.keras.layers.BatchNormalization.
3. Initialize the regularizer with a threshold and the output ops of your model (e.g., logits for classification).
4. Add the regularization term to loss.
5. Train the model.
6. Save the proposed model structure with the StructureExporter.The exported files are in JSON format.
7. Modify the model using the StructureExporter output.
8. Retrain the model from scratch without the MorphNet regularizer.
   

Know more about Morphnet here.

Check how to deploy MorphNet with TensorFlow here