The ongoing episode between the pioneers of AI, Juergen Schmidhuber and Geoff Hinton, only gets worse as Dr Schmidhuber responds to Dr Hinton getting awarded the Honda prize back in November 2019.
The Honda Prize, established in 1980 and awarded once every year, is an international award that recognises the work of individuals or groups generating new knowledge to drive the next generation, from the standpoint of eco-technology.
In the press release, the Honda prize authorities applauded Dr Hinton for creating several technologies that have enabled the broader application of AI, including the backpropagation algorithm that forms the basis of the deep learning approach to AI.
However, the backpropagation part of the above statement did not go down well with Dr Schmidhuber, and he explained in his blog everything that is wrong with Dr Hinton’s latest accolades.
Though Dr Schmidhuber’s argument for not crediting the original creators is quite old, there seems to be no change in the actions across the AI community. And Dr Schmidhuber does not look like he is going to stop anytime soon. And he quotes Elvis Presley:
“Truth is like the sun. You can shut it out for a time, but it ain’t goin’ away.”
Critique Of Hinton
Dr Schmidhuber, in his latest blog post, began his critique of Dr Hinton by admitting that Hinton and his co-workers have made significant contributions to deep learning, but he discarded the claim of creating the backprop algorithm. He explained, once again, how they have failed to cite Paul Werbos, who proposed the same method three years before Dr Hinton’s seminal paper of 1985.
They even failed to mention Seppo Linnainmaa, the inventor of this famous algorithm for credit assignment in networks.
Dr Schmidhuber
Dr Schmidhuber also explained why Hinton’s interesting unsupervised pre-training for deep NNs was irrelevant for the current deep learning revolution, and that, it is actually his team who showed that deep feedforward NNs (FNNs) can be trained by plain backpropagation and do not at all require unsupervised pre-training for important applications. This, he says, was achieved by greatly accelerating traditional FNNs on highly parallel graphics processing units called GPUs.
Hinton Responds
Dr Hinton, in response to this criticism, has responded on a popular machine learning channel on Reddit.
Dr Hinton, who has been mostly silent on Schmidhuber’s consistent criticism, said that having a public debate with Schmidhuber about academic credit is not advisable and that is why he has chosen Reddit to respond. But he tried to end the debate once and for all by stating that he never claimed to have invented backpropagation.
Dr Hinton, in his response, also claimed that Dr Schmidhuber has even resorted to tricks like having multiple aliases in Wikipedia to make it look as if other people are agreeing with what he says. The page on his website, says Dr Hinton, about Alan Turing is a nice example of how he goes about trying to diminish other people’s contributions.
“I have never claimed that I invented backpropagation.”
Dr Hinton
David Rumelhart, wrote Dr Hinton, invented it independently long after people in other fields had invented it. He also admitted to failing to cite them owing to lack of knowledge of history.
It is true that many people in the press have said I invented backpropagation and I have spent a lot of time correcting them
“Lots of different people invented different versions of backpropagation before David Rumelhart. They were mainly independent inventions, and it’s something I feel I have got too much credit for. I’ve seen things in the press that say that I invented backpropagation, and that is completely wrong. It’s one of these rare cases where an academic feels he has got too much credit for something! My main contribution was to show how you can use it for learning distributed representations, so I’d like to set the record straight on that.”
Dr Schmidhuber was quick to retaliate to Dr Hinton’s response. He added to his original blog by saying that Dr Hinton has failed to respond accurately and has only resorted to ad hominem attacks on him.
“To summarise, Dr Hinton’s comments and ad hominem arguments diverge from the contents of my post and do not challenge any of the facts presented. The facts still stand.”
Dr Schmidhuber
Why Is This Important?
At the heart of this whole event, one cannot help but notice where Dr Schmidhuber is coming from. He believes in crediting the original creators and that it establishes a healthy ecosystem in the ML community. There is no doubt that Dr Hinton and his colleagues have contributed tremendously to the advancement of AI, but popularising few people can affect any domain in its initial stages.
The celebrated trio of the modern AI scene, Yann Lecun, Yoshua Bengio and Geoffrey Hinton, have been at the forefront of every major AI breakthrough news that has come over in the past decade. However, the trio has been accused of circular citations by Dr.Schmidhuber in his 2015 post titled, “Deep Learning Conspiracy”, in which he explained in detail how LBH have been ignorant of the original inventors.
While the community is divided over Dr Schmidhuber’s polemics, they all can agree upon one thing – if there is an individual who can administer an authority to challenge the likes of Dr Hinton, then it is him.
