The general consensus in the machine learning community is that making a model smaller would lead to a larger training error, while a bigger model would result in a larger generalisation gap. That is why developers usually hunt for that sweet spot between errors and generalisation. 

However, the best test error is often achieved by the largest model, which is counterintuitive.

As one increases the model complexity past the point where the model can perfectly fit the training data (Interpolation Regime), test error continues to drop! 

The inner training dynamics of the neural networks have long been a mystery and unlocking this would lead to a better understanding of the predictions. 

In order to meet these ends, a paper titled Neural Tangent Kernel (NTK) was submitted at the prestigious NIPS conference last year and has been making noise ever since.

It also occupied a majority of the talks at the recently concluded Workshop on Theory of Deep Learning at the Institute for Advanced Study.

Authors of this paper, Arthur Jacot and his colleagues at the Swiss Federal Institute of Technology Lausanne introduced NTK as a new tool to study ANNs, which describes the local dynamics of an ANN during gradient descent. 

“This led to a new connection between ANN training and kernel methods,” wrote the authors in their paper.

In the infinite-width limit, an ANN can be described in the function space directly by the limit of the NTK, an explicit constant kernel, which only depends on its depth, nonlinearity and parameter initialisation variance. 

The practical significance of the NTK theory is that you can actually compute the kernel for infinite width networks exactly, and treat that as a new kernel machine.

The whole idea behind NTK is to find a parallel between the convergence of neural networks and kernel methods, then defining a new kernel to describe the generalisation of neural networks. 

Hence a connection between large models with best test error and kernel methods is established, which in turn paves a way to assessing the performance of neural networks. 

Because at initialisation, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit thus can be likened to kernel methods.

Why Explore Kernel Regime

Source: Rajat’s blog

A kernel method can be thought of as a trick developed to learn patterns or correlations in high dimensional data without the need to learn from a fixed set of parameters. This comes handy in case of complex parameter scenarios or when the data is unlabeled. Kernel methods learn through instances, remembering weights and giving similarity scores without the need to know the whole picture. This makes the training computationally cheap.

Now, in the case of infinite width networks, a neural tangent kernel or NTK consists of the pairwise inner products between the feature maps of the data points at initialisation.

And since the tangent kernel stays constant during training, the training dynamics is now reduced to a simple linear ordinary differential equation.

The authors in the seminal paper insist that the limit of the NTK is a powerful tool to understand the generalisation properties of neural networks, and it allows one to study the influence of the depth and nonlinearity on the learning abilities of the network. 

The key difference between the NTK and previously proposed kernels is that the NTK is defined through the inner product between the gradients of the network outputs with respect to the network parameters. 

Another fruitful direction is to “translate” different tricks of neural networks to kernels and to check their practical performance. 

Researchers at Carnegie Mellon University hope that tricks like batch normalisation, dropout, max-pooling, etc. can also benefit kernels since it has been established that the global average pooling can significantly boost the performance of kernels.

Similarly, they posit that one can try to translate other architectures like recurrent neural networks, graph neural networks, and transformers, to kernels as well.

Drawing insights from the inner workings of over-parameterised deep neural networks is still a challenging theoretical question. However, with the recent developments such as the ones discussed above, the researchers are optimistic that we can now have a better understanding of ultra-wide neural networks that can be captured by neural tangent kernels.