The Importance Of No Free Lunch Theorems In Deep Learning

“The no free lunch theorem calls for prudency when solving ML problems by requiring that you test multiple algorithms and solutions with a clear mind and without prejudice.” In a paper titled, ‘The Lack of A Priori Distinctions Between Learning Algorithms’, that dates back to 1996,  David Wolpert explored the following questions: Can we actually get something for nothing in supervised learning? Can we get useful, caveat-free theoretical results that link the training set and the learning algorithm to generalisation error, without making assumptions concerning the target? Are there useful practical techniques that require no such assumptions?  He showed that for any two algorithms, A and B, there are as many scenarios where A will perform worse than B as