Complete Guide on Language Modelling: Unigram Using Python

Language modelling is the speciality of deciding the likelihood of a succession of words. These are useful in many different Natural Language Processing applications like Machine translator, Speech recognition, Optical character recognition and many more.In recent times language models depend on neural networks, they anticipate precisely a word in a sentence dependent on encompassing words. However, in this project, we will discuss the most classic of language models: the n-gram models.

In natural language processing, an n-gram is an arrangement of n words. For example “Python” is a unigram (n = 1), "Data Science" is a bigram (n = 2), "Natural language preparing" is a trigram (n = 3) etc.Here our focus will be on implementing the unigrams(single words) models in python.

Assumptions For a Unigram Model

1.  It depends on the occurrence of the word among all the words in the dataset.

2.  Probability of a word is independent of all the words before its occurrence.

Code Implementation

Import all the libraries required for this project.

import nltk
nltk.download('reuters')
from nltk.corpus import reuters
nltk.download('punkt')

Reuters dataset consists of 10788 documents from the Reuters financial newswire services.

Store the words in a list.

words = list(reuters.words())
words

len(words)

We will start by creating a class and defining every function in it. The idea is to generate words after the sentence using the n-gram model. Predicting the next word with Bigram or Trigram will lead to sparsity problems. To solve this issue we need to go for the unigram model as it is not dependent on the previous words.

Let’s calculate the unigram probability of a sentence using the Reuters corpus.

class NGrams:
    def __init__(self, words, sentence):
        self.words = words
        self.sentence = sentence
        self.tokens = sentence.split()
    def get_tokens(self):
        return self.tokens
    def add_tokens(self,value):
        temp = self.tokens
        temp.append(value)
        self.tokens = temp
        return self.tokens
    def unigram_model(self):
        self.next_words = np.random.choice(words, size=3)
        return self.next_words

Here we need to calculate the probabilities for all the various words present in the results of the over unigram model. Select the top three words based on probabilities.

   def get_top_3_next_words(self,next_words):
        next_words_dict = dict()
        for word in next_words:
            if not word in next_words_dict.keys():
                next_words_dict[word] = 1
            else:
                next_words_dict[word] += 1
          for i,j in next_words_dict.items():
              next_words_dict[i] = np.round(j/len(next_words),2)
        return sorted(next_words_dict.items(), key = lambda k:(k[1], k[0]), reverse=True)[:3]
    def model_selection(self):
            top_words = self.unigram_model()
            print("unigram-model")
            return top_words

model = NGrams(words=words, sentence=start_sent)

import numpy as np
for i in range(5):
    values = model.model_selection()
    print(values)
    value = input()
    model.add_tokens(value)

The model generates the top three words. We can select a word from it that will succeed in the starting sentence. Repeat the process up to 5 times. The result is displayed below.

print(model.get_tokens())

Final step is to join the sentence that is produced from the unigram model.

print(" ".join(model.get_tokens()))

Final Thoughts

In this article, we have discussed the concept of the Unigram model in Natural Language Processing. Further, we can research on the topic of Bi-gram and Trigram to generate words after the sentences. Finally, I hope this article is useful to you.