Few-shot learning (FSL) or low-shot learning (LSL) is a type of meta-learning. Unlike traditional AI/ML models that require a large amount of data to produce good results, AI/ML models with few-shot learning capability require only a small amount of training data with limited information.
In few-shot learning, a learner is trained on various related tasks during the meta-training phase to generalise well to unseen tasks with just a few instances during the meta-testing phase.
Few-shot classification is a method where a classifier must be adapted to accommodate new classes that are not seen in training and given only a few examples of each of these classes. In recent years, AI researchers have utilised this method for various applications in computer vision, natural language processing (NLP), acoustic signal processing and more. However, approaches such as re-training the model on new data can lead to overfitting of the model.
To mitigate such issues in few-shot classification, a team of researchers from Twitter and the University of Toronto has introduced prototypical networks. Prototypical networks learn a metric space where classification can be executed by computing distances to prototype representations of each class.
What are prototypical networks?
Prototypical networks are based on the concept that there exists an embedding in which several points cluster around a single prototype representation for each class. It aims to learn per-class prototypes based on sample averaging in the feature space.
To be more specific, prototypical networks compute each class’s M-dimensional representation or prototype through an embedding function with learnable parameters. Also, each prototype is the mean vector of the embedded support points belonging to its class.
Prototypical networks are more efficient than the recent meta-learning algorithms, making them an appealing approach to few-shot and zero-shot learning. They learn a metric space in which classification can be performed by computing distances to prototype representations of each class. Prototypical networks are related to the neural statistician from the generative modelling literature that extends the variational autoencoder to learn generative models of datasets rather than just the individual points.
Prototypical networks vs matching networks
In recent years, approaches such as matching networks have made significant progress in few-shot learning. Matching networks use an attention mechanism over a learned embedding of the labelled set of examples to predict classes for the unlabelled points.
Both prototypical networks and matching networks were introduced to mitigate the various shortcomings of the few-shot learning technique. The differences between both the networks are as follows-
- Compared to the recent approaches for few-shot learning, such as matching networks, prototypical networks reflect a more simplistic inductive bias, which is beneficial in the limited-data regime and achieve exceptional results.
- Prototypical networks vary from matching networks in the few-shot case with equivalence in the one-shot scenario. This means that, in the case of one-shot learning, since there is only one support point per class, matching networks and prototypical networks become equivalent.
- Matching networks produce a weighted nearest neighbour classifier given the support set, while prototypical networks produce a linear classifier when squared Euclidean distance is used.
Wrapping up
Prototypical Networks is a simple method for few-shot learning based on the idea that one can represent each class by the mean of its examples in a representation space learned by a neural network. The approach is more efficient and far simpler than recent meta-learning approaches and produces state-of-the-art results even without sophisticated extensions developed for matching networks.
