Amazon Unveils New AI Language Model that Beats GPT-3

The new language model outperformed OpenAI’s GPT-3 and Google’s PaLM on various NLP benchmarks
Listen to this story

Amazon Alexa AI researchers recently unveiled Alexa Teacher Models (AlexaTM 20B) that beats GPT-3 on NLP benchmarks. The 20-billion-parameter sequence-to-sequence (seq2seq) language model showcases SOTA capabilities on few-shot learning. The model is yet to be released publicly. 

Check out the GitHub repository here

Unlike OpenAI’s GPT-3 or Google’s PaLM, which are decoder-only models, AlexaTM 20B is a seq2seq model that contains an encoder and a decoder allowing better performance on machine translation (MT) and summarization. 

Sequence-to-sequence model is a special class of recurrent neural network architecture, typically used to solve complex language problems, including machine translation, creating chatbots, question answering, text summarisation, etc. 

With 1/8 number of parameters, the new language model by Amazon outperformed GPT-3 on SQuADv2 and SuperGLUE benchmarks. The multilingual model achieves excellent performance on few-shot MT tasks, even on low-resource languages, on the Flores-101 dataset. 

On several other benchmarks like MLSum, AlexaTM outperformed all other models for 1-shot summarization in Spanish, German, French and most language pairs on 1-shot MT tasks. On low-resourced languages like Tamil, Telugu, and Marathi, the improvement was significant. On English-based languages, the model outperformed GPT-3 on MT tasks but came second to the larger PaLM model.

Saleh Soltan, senior applied scientist on Amazon, said that, “the proposed style of pretraining enables seq2seq models that outperform much larger decoder-only LLMs across different tasks, both in a few-shot setting and fine-tuning.”

Download our Mobile App

Subscribe to our newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day.
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Our Recent Stories

Our Upcoming Events

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox