Recent advancements in natural language processing (NLP) have touched many heights over the past few years. Pre-trained high-capacity language models such as ELMo and BERT have gained popularity in NLP.
Language modelling has been implementing in a number of applications such as machine translation, speech recognition, question answering and sentiment analysis, among others. Basically, the language model is similar to the grammar in any language as it provides the likelihood of the next positive word. It is one of the integral parts of modern natural language processing. It has various advantages, for instance, it requires no human supervision, easy to extend to more data, allows querying about open-class relations, etc. In simple words, the better the language model, the lower the error rate of the speech recognizer.
Recently, the researchers at Facebook AI Research and University College London introduced LAnguage Model Analysis (LAMA) which consists of a set of knowledge sources where a pre-trained language model can successfully predict masked objects in cloze sentences. In simple words, the LAMA dataset includes a large corpus of sentences where each of them is missing a key fact. The model is basically built to test the factual and commonsense knowledge in language models. It was built on facts extracted from Google-RE (facts from Wikipedia), T-REx (facts aligned with Wikipedia text), ConceptNet (a semantic network), and SQuAD (questions and answers).
This model helps in identifying the most accurate working pre-trained model. LAMA tests the popular pre-trained models such as BERT, Transformer-XL, and ELMo by asking these models to fill in a missing subject or object. The outcome proved that BERT-Large provided the most accurate reply as compared to the other pre-trained NLP models.
Popular Use Cases of Language Model
Aristo
Recently, Allen Institute for Artificial Intelligence announced a breakthrough for a BERT-based model that did pass an eighth-grade New York Regents science exam. The system is known as Aristo which is a GPU-accelerated system and has the ability to read, learn, and reason about science. The system answered more than 90% of the questions on an eighth-grade science exam correctly. Furthermore, the Aristo system, building upon the success of recent language models, exceeded 83% on the corresponding Grade 12 science exam questions.
The model answered multiple-choice questions without diagrams, and operates only in the domain of science and the work as a whole integrates multiple AI-based technologies which include natural language processing (NLP), information extraction, commonsense knowledge, knowledge representation and reasoning, and diagram understanding.
Olivia Taters
Olivia Taters can be said as one of the best bots communicating in Tweeter. This bot is the creation of Rob Dubbin. According to a report, Tater is a young teenage girl who may not always communicate in complete sentences but she is convincing enough that teenagers actually converse with her. The bot replies to the people who follow her and often keeps tweeting profoundly.
Outlook
The word-embedding NLP pre-trained methods such as BERT, ELMo and RoBERTa have set a benchmark with a remarkable rate. These sophisticated pre-trained models can provide insights and information to various missed out sentences or words. The language models will provide better enhancement to the various language-agnostic tasks and are able to accomplish much more in the domain of speech recognition, handwriting recognition, spelling correction, machine translation, sentiment analysis, among others. Also, areas where the input is ambiguous in some way or other, the language model will provide a helping hand to get the most likely input.
