The Lottery Ticket Hypothesis: Can You Find Winning Ticket Neurons?
Neural networks have shown remarkable success in various domains, ranging from image classification to natural language processing. Despite their impressive performance, deep neural networks are known to be computationally expensive and require significant hardware resources to train and deploy. To address this issue, researchers have explored the concept of neural network pruning, which involves removing a subset of weights or neurons from the network without significantly affecting its accuracy. One of the most recent and promising approaches to neural network pruning is the Lottery Ticket Hypothesis.
The Lottery Ticket Hypothesis was proposed by Jonathan Frankle and Michael Carbin in their 2018 paper, “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”. The hypothesis states that a dense neural network contains a subnetwork, called a “winning ticket,” that can be pruned down to a smaller size without compromising its performance. This winning ticket is a subset of the original network that is initialized with the same weights as the dense network and is able to achieve comparable accuracy after training for a much shorter period.
The idea behind the Lottery Ticket Hypothesis is that during the random weight initialization of a neural network, some connections are assigned weights…
