In machine learning, bigger may not always be better. As the datasets and the machine learning models keep expanding, researchers are racing to build state-of-the-art benchmarks. However, larger models can be detrimental to the budget and the environment.
Over time, researchers have developed several ways to shrink the deep learning models while optimizing training datasets. In particular, three techniques–pruning, quantization, and transfer learning–have been instrumental in making models run faster and more accurately at lesser compute power.
In a 2019 study, Lottery Ticket Hypothesis, MIT researchers showed it was possible to remove a few unnecessary connections in neural networks and still achieve good or even better accuracy.
What Is Lottery Ticket Hypothesis
In machine learning and neural networks, pruning (introduced in the early 90s) refers to compressing the model by removing weights. The studies have demonstrated pruning could drastically remove parameter counts, sometimes by more than 90 percent. This helps decrease the model size and the energy consumption of the trained networks, which makes inference more efficient.
The question “if a network can be reduced in size, why do we not train this smaller architecture instead in the interest of making training more efficient as well?” led to the development of the Lottery Ticket Hypothesis. Authors Jonathan Frankle and Michael Karbin observed that architectures on which pruning is performed are harder to train from the start and reach lower accuracy than the original networks.
“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations,” according to the paper.
Pruning helps in uncovering trainable subnetworks from fully-connected and convolutional feed-forward networks. If these subnetworks have the appropriate combination of weights and connections capable of learning, they are designated as ‘winning tickets’.
As the network size increases, the number of possible subnetworks and the probability of finding the ‘lucky subnetwork’ also increase. As per the lottery ticket hypothesis, if we find this lucky subnetwork, we can train small and sparsified networks to give higher performance even when 90 percent of the full network’s parameters are removed.
The winning ticket is identified by training a network and pruning its smallest magnitude weights. What remains are the unpruned connections that consist of the architecture of the winning ticket. Each unpruned connection’s value is then reset to its initial value from the original network before it was trained. The steps involved include:
- Randomly initialize a neural network
- Train the network for a set number of iterations to arrive at parameters
- A percentage of these parameters obtained are pruned
- The remaining parameters are reset to the initial value of the original network, creating a winning ticket.
The researchers found the winning tickets were less than 10-20 percent the size of convolutional feed-forward architectures for MNIST and CIFAR10. The winning tickets could learn faster than the original network and achieve higher accuracy.
Benefits
- Since winning tickets can be trained in isolation from the beginning, we can potentially design training schemes to search for winning tickets and start pruning at the earliest.
- Winning tickets are a combination of sparse architectures and initializations which are adept at learning. These winning tickets can help in designing newer architectures and initialization schemes with similar properties conducive to learning.
- The study allows a deeper understanding of neural networks concepts such as random-initialized feed-forward networks and optimization.
Wrapping Up
In November 2019, Facebook AI said it found the first definitive evidence that lottery tickets can be generalized across distinct yet, related datasets. This can be extended to reinforcement learning and natural language processing. Facebook also introduced a theoretical framework on the formation of lottery tickets to understand lucky initializations better.
