“He [Schmidhuber] has an unusual idea of credit attribution. He believes that if someone has the germ of an idea, they should get the credit for everything that comes after. That’s not how things work,” LeCun told AIM previously.
Opposing LeCun’s idea of credit, Schmidhuber recently said that that’s not how science works. Science has a well-established practise of dealing with plagiarism and priority disputes based on facts, such as time stamps of publications and patents.
“Einstein did not build the GPS based on his ideas”. Referring to ‘100 Authors against Einstein,” Schmidhuber explained that science is not democratic. If a hundred people claim one thing and only one person claims the opposite but can back it up through facts, then they win. “It does not matter that LeCun has posted 100 times more tweets than me. The only things that matter in science are facts and truth.”
In a previous critique, Schmidhuber wrote that the inventor should get the credit. However, they may not always be the ones who popularise it. In which case, the populariser should get credit for popularising the invention in accordance with the standard elementary principles of credit assignment. LeCun wants more than the credit due for popularising the inventions of others; he also wants the inventor’s credit.
Credit Denied
Even with the overwhelming commercial triumph of his LSTMs (the current standard for speech translation in the industry), Schmidhuber has been wrongfully denied proper credit as per experts. Schmidhuber recently told AIM, “Although it’s he [LeCun] who conducted plagiarism (which may be unintentional or not)”.
At NIPS in 2016, Schmidhuber declared that his previous work was very similar to Ian Goodfellow’s GANs (Generative Adversarial Networks). The community largely sided with Goodfellow. But Schmidhuber said, “Not the part of the community that counts! There is a peer-reviewed publication showing the correctness of my claim and this work remains unchallenged.”
In 1997, MIT rejected Schmidhuber’s LSTM paper but it turned out to be a key concept in the field of deep learning years later. Tech behemoths, such as Google, Amazon and Apple, use this approach to power their voice recognition systems like Alexa and Siri. It could have gotten more recognition if Schmidhuber had had his way. In a 2015 article, he complained that celebrity “Canadian” trio—Geoffrey Hinton, Yann LeCun and Yoshua Bengio—”heavily cite each other” but “fail to credit the pioneers of the field”.
Attention is All They Need
Schmidhuber, who is away from Silicon Valley in Switzerland, is one of many who need more attention for their work. The lead author behind the NLP revolution, Ashish Vaswani, proposed and tested a new design for a neural network. There are many different ways of configuring these networks. Vaswani and his team called their novel configuration a transformer. However, he doesn’t like to take credit as he believes, “This isn’t about me”.
In response Schmidhuber said, “However, Vaswani did not mention that over 30 years ago I published what’s now called ‘Transformers with linearized self-attention’ (1992-92). They are equivalent to my so-called neural fast weight programmers (apart from normalization), separating storage and control. Key/value was called FROM/TO. The attention terminology was introduced at ICANN 1993.” Here is a recent well-known tweet on this:
