‘Attention is All You Need’ Gets No Attention

What are ‘Attention is All You Need’ authors doing right now – everything about the thinkers, doers, and quitters

In 2017, when former Google researchers published their work ‘Attention is All You Need’, little did we know it was going to transform the AI landscape quintessentially. This also includes the ChatGPT phenomenon, which has also been built on the groundworks of this research paper. 

Ironically, the same folks who laid the foundation of modern AI breakthroughs with their research work previously, are getting little or no attention at all. Let’s take a glance at some of their most recent projects, and what they are doing today.  

The story of ‘Attention is All You Need’ 

Aidan Gomez, a researcher at Google Brain, was exploring ways for artificial intelligence to read huge chunks of data to understand the degrees at which languages function. In 2017, the team struck gold with ‘Transformers’.

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

After working alongside AI luminary Geoffrey Hinton at Google, Gomez co-founded Cohere, an artificial intelligence company in 2019. The company makes large language models accessible for businesses. Gomez believes the technology developed within huge companies remains trapped within; he wanted to make LLMs available to all. “For every ML expert, there’s a thousand non-developers. We need to give those 1,000 the capability to build,” Gomez stated. 

Transformers operate with “attention,” which asks the model to determine the important and likely-to-be connected words. ChatGPT, Dalle.E, and AlphaFold are some of the breakthroughs based on the Transformers model. 

Download our Mobile App

The Thinkers 

Jakob Uszkoreit, a senior software engineer, thought of the name Transformer. In 2021, he co-founded Inceptive to use deep learning and learn life’s languages. At Google Brain, he built the language understanding team of the Google Assistant and worked on Google Translate during its early days. He transitioned to life sciences in 2021 to start a company that applies artificial intelligence to RNA biology for RNA therapeutics. 

The team behind Transformers sparked the NLP revolution. Ashish Vaswami was the lead author for ‘Attention is All You Need’ but doesn’t like to take credit for the advancement. After his stint at Google, Vaswani authored 19 papers and neural networks is the common subject which Vaswami has been pursuing since 2011. His company, Adept emerged in early 2022 with $65 million in initial venture capital funding. The team thinks the same basic idea of Transformers can create general assistants with high capability for business tasks. 

Vaswami works alongside David Luan, a former VP of Engineering at OpenAI, and the creator of ChatGPT. 

The Doers 

ChatGPT is not a new word on the internet. But there’s another tool named Character.AI that’s contending for its place. Though currently in beta mode, the company aims to bring the dream of open-ended conversations and collaborations with machines to life. Unlike ChatGPT, the model is built only for entertainment. Character is a full-stack AGI company founded by Noam Shazeer and Daniel de Freitas, who had previously worked in Google Brain. 

Shazeer was a part of the team developing the novel configuration of Transformers and developed Mesh-Tensorflow in 2018 which was the first practical system for training giant Transformers on supercomputers. Freitas led the LaMDA project at Google brain. The result of Freitas’ previous research was a chatbot called Meena. The effectiveness of the model led Google to hire him and an official research effort for Language Model for Dialogue Applications began. 

The duo left the tech giant’s research arm last year after the project created a ruckus when another Google engineer, Blake Lemoine, claimed that LaMDA was sentient. Google was hesitant to release the technology, as it could damage the company’s image. Hence, Character.AI was founded as Shazeer and Freitas was determined to bring that technology to the public. 

The Quitters

Google enhances existing products, instead of creating entirely new product categories, said Niki Parmar, CTO and Co-founder of Adept and former staff research scientist who was part of the ‘Attention is all You Need’ team, which led the foundation of transformer models. 

Besides the Transformer team, several notable personalities have also left their roles at the research arm to become a part of the start-up culture. For instance, Joe Fenton left his senior product manager role in 2022 to become a member of the founding team at Inflection. Rewon Child, former researcher, and Maarten Bosma, former research engineer at Google recently joined Inflection. 

Sign up for The AI Forum for India

Analytics India Magazine is excited to announce the launch of AI Forum for India – a community, created in association with NVIDIA, aimed at fostering collaboration and growth within the artificial intelligence (AI) industry in India.

Tasmia Ansari
Tasmia is a tech journalist at AIM, looking to bring a fresh perspective to emerging technologies and trends in data science, analytics, and artificial intelligence.

Our Upcoming Events

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023

21 Jul, 2023 | New York
MachineCon USA 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

GPT-4: Beyond Magical Mystery

The OpenAI CEO believes that by ingesting human knowledge, the model is acquiring a form of reasoning capability that could be additive to human wisdom in some senses.