A Series On Underrated But Interesting Machine Learning Concepts

There are some riveting concepts in machine learning that aren't talked about as much as they should. With this article, we look at some of them, including manifold alignment, Quickprop, skill chaining, and FastICA. 

Manifold Alignment

The basic concepts of manifold alignment are to use the links between instances within each dataset to improve the understanding of the relationships across datasets and, eventually, to map originally diverse datasets to shared latent space. 

In several areas of machine learning and data mining, one is frequently confronted with data distributed over many dimensions. Dealing with such high-dimensional data directly is usually impossible, although the manifold structure underlying the data may have a low intrinsic dimensionality in many circumstances. Chang Wang et al. states that, by matching the underlying manifolds of two or more distinct data sets, manifold alignment creates links between them and allows knowledge transfer.

Benefits:

Jingliang Hu concluded that data alignment-based (data concatenation) manifold techniques are less capable of fusing hyperspectral images and PolSAR data. Manifold alignment-based approaches, on the other hand, are more capable of fusing optical and SAR pictures. Semi-supervised methods, among the manifold alignment-based manifold techniques, are able to efficiently exploit both the structure of data and the existing label information.

---

Quickprop

Fahlman states that QuickProp is a second-order optimization technique that accelerates optimization by using a simple estimate of the Hessian's diagonal, and hence belongs to the class of Quasi-Newton algorithms. In the event of a large number of iterations, weights, and samples, the learning process is exceedingly sluggish (or never reaches the target). According to Windra Swastika, the Quickprop approach is one way to speed up the learning process. The objective is to reduce the error gradient (E').

A number of algorithms attempt to improve on gradient descent by predicting an ideal learning rate at each update step, including AdaDelta, AdaGrad, and Adam. QuickProp, on the other hand, assumes that the output is a quadratic function of each variable and calculates the best position directly.

Benefits:

Windra Swastika et al. states that, when compared to Back Propagation, the Quickprop approach successfully reduces the learning process. For 40 input patterns, the Quickprop approach required only 20 iterations to obtain an error of less than 0.1, but the Back Propagation method required 2000 iterations. Both the methods have a prediction accuracy of more than 90%.

---

Skill Chaining

Skill chaining is a continual reinforcement learning skill discovery method. The associated deep skill chaining technique has expanded it to high-dimensional continuous domains. As per Singh et al., skill chaining creates a series of skills that lead to one set of designated target events, which can be as simple as the episode's end-of-episode event or as complex as intrinsically intriguing events. The skills are built so that completing each option in the chain allows the agent to move closer to its ultimate goal by executing another option. While skill chaining could uncover skills in continuous state spaces, it could only be used in state spaces with discrete actions that were relatively low in dimension. 

Benefits:

George. state that the main advantage of skill chaining is that it alleviates the stress of describing the task's value function. Eylem Tekin. concludes that skill chaining strategies are useful in various systems, including those with many products and those with a parallel or network flow structure, such as cells with automated equipment. In such situations, more research is needed to establish the possible benefits of chaining.

---

FastICA

FastICA is a time-saving independent component analysis algorithm. Erkki Oja. expresses that the instantaneous noise-free ICA model was the inspiration for FastICA. PierreComon. state that an observed random vector is transformed into statistically independent components using Independent Component Analysis (ICA).

Vicente Zarzoso. said that FastICA has only been compared against neural-based adaptive algorithms like principal component analysis (PCA), which are known to outperform most ICA algorithms. As well as its simplicity, the method's popularity can be attributed to its good performance in various applications. According to P. Chevalier, FastICA fails for weak or highly spatially correlated sources, which is the first significant attempt.

Benefits:

According to Vendetta, FastICA is the most generally utilised method for blind source separation problems because it is computationally efficient and uses less memory than other blind source separation algorithms, such as infomax. Another benefit is that separate components can be calculated one by one, reducing computing load once again. The only drawback is that if the noise is non-uniform and correlated noise vectors, this method will fail to determine sources correctly.