Google overlooking the transformers paper in 2017 has cost them a staggering $6.2 billion – a collective valuation of all the AI startups built on transformers by the foundation team. Now, the last of the eight authors of the pivotal paper is (also) leaving the Big Tech to start his own company. Llion Jones said he will be joining his fellow ex-Google research scientist David Ha in the new endeavour to start Sakana AI. 

The paper titled ‘Attention Is All You Need’ was ‘born’ when the authors were, at Google’s meal time, discussing ways to make computers generate text efficiently. After running tests for the next five months, the team wrote the paper having no idea of what’s coming next. Their work made the once-dominant architecture Recurrent Neural Nets (RNNs) in language processing largely obsolete. 

Notably, the transformer architecture represents the T in ChatGPT and chatbots akin. The 2017 paper laid the foundation to the generative AI wave initiated by OpenAI after making GPT models accessible to the public (and enterprises). Jones, along with the co-authors, are finally getting the due credit for their billion dollar paper that Google misjudged six years ago. 

“It’s only recently that I’ve felt … famous?” Jones recently told The Washington Post. “No one knows my face or my name, but it takes five seconds to explain: ‘I was on the team that created the ‘T’ in ChatGPT’.”

The $6.2 billion-worth underdogs

One standout is ‘Cohere’, founded in 2019 by Aidan N Gomez, Ivan Zhang and Nick Frosst. This NLP-focused enterprise startup has soared to a valuation of $2.2 billion. Reports reveal that Tiger Global, having invested initially, will sell a stake, reaping a 40% markup on the current valuation — reflecting on the startup’s rise.

‘Adept’, another fledgling startup with just a year under its belt, has not only secured a valuation of $1 billion but has also managed to raise an impressive $350 million in venture capital this year. Founded by lead author Ashish Vaswani and Niki Parmar, the startup’s minimalist workforce of 25 belies its profound impact. The duo’s entrepreneurship goes beyond Adept. Their undisclosed startup, still in stealth mode, has raked in $8 million in funding led by Thrive Capital.

Noam Shazeer, another author of the transformative paper, co-founded ‘Character.AI’, a chatbot startup that rose to a valuation of $1 billion. A $150 million funding round led by Andreessen Horowitz solidified the company’s status as a unicorn earlier in 2023.

The narrative doesn’t end here. Illia Polosukhin founded the high-performance blockchain called ‘NEAR Protocol’ in 2017, which has a valuation of $2 billion. The latest fundraising tally of $350 million was the second nine-figure raise within four months by Polosukhin’s brainchild. 

Jakob Uszkoreit, who came up with the name Transformer, founded Inceptive in 2021 to use deep learning and learn life’s languages. During his 13-year stint as a senior software engineer, he built the language understanding team of the Google Assistant and worked on Google Translate during its early days. Inceptive’s value currently remains undisclosed. As recognition finally trickles in, it is a fitting closure to see Lukasz Kaiser, one of the eight author of the paper, find his place as a member of the technical staff at OpenAI two years ago. 

Attention was all they needed

Back in 2017, least did Google know that its paper would eventually be cited around 90,000 times by other researchers, and the transformer architecture would underpin ChatGPT, Midjourney and the rest. While Google shared the discovery with the world, the company itself didn’t put the content generation technology to use straight away. Google kept it in hibernation till CEO Sundar Pichai took the stage at Google I/O 2021 to announce LaMDA. The company did not strike gold unlike OpenAI due to the ‘Is AI sentient?’ controversy that summer. 

The Washington Post pointed out that the company went wrong with its scaling strategy. Out of the estimated 140,000 employees, over 7000 people are dedicatedly working on AI as per Bloomberg Opinion. Comparatively, OpenAI has a much smaller workforce — about 150 AI researchers out of the approximately 375 employees in 2023. 

The company’s gigantic workforce added to certain layers to sign off on ideas half-a-decade ago, several former employees have confessed. Furthermore, the search giant has a very high bar for turning ideas into products, Polosukhin said in an interview. The company’s inertia and intolerance towards experiments with dead ends and iterations is another reason the transformers team decided to walk through the exit door. Moreover, there were more exciting innovations happening outside the Mountain View HQ which led to the talent exodus. 

While the company has been experimenting with models like PaLM, LaMDA and Bard, there is still no hint of a significant product. While Google stashed it away, startups capitalised on the technology to remove language barriers between humans and machines. The ground-breaking paper overlooked by the company has made Google a victim of its own success.