‘Attention is All You Need’ Gets No Attention

In 2017, when former Google researchers published their work ‘Attention is All You Need’, little did we know it was going to transform the AI landscape quintessentially. This also includes the ChatGPT phenomenon, which has also been built on the groundworks of this research paper. 

Ironically, the same folks who laid the foundation of modern AI breakthroughs with their research work previously, are getting little or no attention at all. Let’s take a glance at some of their most recent projects, and what they are doing today.  

The story of ‘Attention is All You Need’ 

Aidan Gomez, a researcher at Google Brain, was exploring ways for artificial intelligence to read huge chunks of data to understand the degrees at which languages function. In 2017, the team struck gold with ‘Transformers’.

After working alongside AI luminary Geoffrey Hinton at Google, Gomez co-founded Cohere, an artificial intelligence company in 2019. The company makes large language models accessible for businesses. Gomez believes the technology developed within huge companies remains trapped within; he wanted to make LLMs available to all. “For every ML expert, there’s a thousand non-developers. We need to give those 1,000 the capability to build,” Gomez stated. 

Transformers operate with “attention,” which asks the model to determine the important and likely-to-be connected words. ChatGPT, Dalle.E, and AlphaFold are some of the breakthroughs based on the Transformers model. 

The Thinkers 

Jakob Uszkoreit, a senior software engineer, thought of the name Transformer. In 2021, he co-founded Inceptive to use deep learning and learn life's languages. At Google Brain, he built the language understanding team of the Google Assistant and worked on Google Translate during its early days. He transitioned to life sciences in 2021 to start a company that applies artificial intelligence to RNA biology for RNA therapeutics. 

The team behind Transformers sparked the NLP revolution. Ashish Vaswami was the lead author for ‘Attention is All You Need’ but doesn’t like to take credit for the advancement. After his stint at Google, Vaswani authored 19 papers and neural networks is the common subject which Vaswami has been pursuing since 2011. His company, Adept emerged in early 2022 with $65 million in initial venture capital funding. The team thinks the same basic idea of Transformers can create general assistants with high capability for business tasks. 

Vaswami works alongside David Luan, a former VP of Engineering at OpenAI, and the creator of ChatGPT. 

The Doers 

ChatGPT is not a new word on the internet. But there’s another tool named Character.AI that’s contending for its place. Though currently in beta mode, the company aims to bring the dream of open-ended conversations and collaborations with machines to life. Unlike ChatGPT, the model is built only for entertainment. Character is a full-stack AGI company founded by Noam Shazeer and Daniel de Freitas, who had previously worked in Google Brain. 

Shazeer was a part of the team developing the novel configuration of Transformers and developed Mesh-Tensorflow in 2018 which was the first practical system for training giant Transformers on supercomputers. Freitas led the LaMDA project at Google brain. The result of Freitas’ previous research was a chatbot called Meena. The effectiveness of the model led Google to hire him and an official research effort for Language Model for Dialogue Applications began. 

The duo left the tech giant’s research arm last year after the project created a ruckus when another Google engineer, Blake Lemoine, claimed that LaMDA was sentient. Google was hesitant to release the technology, as it could damage the company’s image. Hence, Character.AI was founded as Shazeer and Freitas was determined to bring that technology to the public. 

The Quitters

Google enhances existing products, instead of creating entirely new product categories, said Niki Parmar, CTO and Co-founder of Adept and former staff research scientist who was part of the ‘Attention is all You Need’ team, which led the foundation of transformer models. 

Besides the Transformer team, several notable personalities have also left their roles at the research arm to become a part of the start-up culture. For instance, Joe Fenton left his senior product manager role in 2022 to become a member of the founding team at Inflection. Rewon Child, former researcher, and Maarten Bosma, former research engineer at Google recently joined Inflection.