Advertisement

Explained: Stemming vs lemmatization in NLP

Stemming is a faster process than lemmatization.

Stemming and lemmatization are text normalisation techniques used in NLP. Essentially, both the techniques break down the words into their root forms. What’s the significance of stemming and lemmatization in NLP and how do they differ? Let’s find out.

Lemmatization

Lemmatization entails reducing a word to its canonical or dictionary form. The root word is called a ‘lemma’.The method entails assembling the inflected parts of a word in a way that can be recognised as a single element. The process is similar to stemming but the root words have meaning.

Lemmatization has applications in:

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.
  1. Biomedicine: Using lemmatization to parse biomedicine literature may increase the efficiency of data retrieval tasks.
  2. Search engines
  3. Compact indexing: Lemmatization is an efficient method for storing data in the form of index values.

For example, NLTK provides WordNetLemmatizer class– a slim cover wrapped around the wordnetCorpus. This class makes use of a function called Morphy() to the WordNetCorpusReader class to find a root word/lemma. 

Source: Linkedin


Download our Mobile App



Lemmatizers need a lot more data on the structure of a language, which makes the creation of lemmatizers a harder process than making a stemming algorithm. 

Stemming

Stemming is a rule-based approach that produces variants of a root/base word. In simple words, it reduces a base word to its stem word. This heuristic process is the simpler of the two as the process involves indiscriminate cutting of the ends of the words. Stemming helps to shorten the look-up and normalise the sentences for a better understanding. The process has two main challenges:

  • Over stemming: The inflected word is cut off so much that the resultant stem is nonsensical. Over stemming can also result in different words with different meanings having the same stem. For example, “universal”, “university” and “universe” is reduced to “univers”. Here, even though these three words are etymologically related, their modern meanings are widely different. Treating them as synonyms in a search engine will lead to inferior search results.
  • Understemming: Here, various inflected words have the same stem despite different meanings. The issue crops up when we have several words that actually are forms of one another. An example of understemming in the Porter stemmer is “alumnus” → “alumnu”, “alumni” → “alumni”, “alumna”/”alumnae” → “alumna”. The English word has Latin morphology, and so these near-synonyms are not combined..

Lemmatization versus stemming

Both procedures involve same methodologies; that is reducing the inflectional forms of each word into a common base or root. However, the main difference is in the way they work and, therefore, the result each returns.

Source: ResearchGate

  • Stemming is a faster process than lemmatization as stemming chops off the word irrespective of the context, whereas the latter is context-dependent.
  • Stemming is a rule-based approach, whereas lemmatization is a canonical dictionary-based approach. 
  • Lemmatization has higher accuracy than stemming. 
  • Lemmatization is preferred for context analysis, whereas stemming is recommended when the context is not important.

To assess the performance of the processes, stemming and lemmatization were compared against the baseline technique used in NLP models provided with the CACM collection. Mean Average Precision (MAP) was used to evaluate document relevancy at the top 10 and 20 document levels.

Source: Inse.org

Both stemming and lemmatization outperformed better than the baseline technique at both the document levels. This indicates that when queries are processed using language modeling techniques, they yield documents that are more relevant compared to queries which are not processed.

While stemmers are quick to create and run, lemmatizers provide a better quality of results or lower margins of error.  

The key applications of these methodologies include:

  1. Information retrieval: Stemming and lemmatization can be used to map documents to general topics and provide search results by indexing.
  2. Document clustering: Stemming and lemmatization reduce the number of tokens to facilitate the transfer of same information. Here, features are estimated by determining the frequency of each token, and then clustering methods are applied.
  3. Sentiment analysis

More Great AIM Stories

Kartik Wali
A writer by passion, Kartik strives to get a deep understanding of AI, Data analytics and its implementation on all walks of life. As a Senior Technology Journalist, Kartik looks forward to writing about the latest technological trends that transform the way of life!

AIM Upcoming Events

Regular Passes expire on 3rd Mar

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Early Bird Passes expire on 17th Feb

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, Virtual
Deep Learning DevCon 2023
27 May, 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox
AIM TOP STORIES

What went wrong with Meta?

Many users have opted out of Facebook and other applications tracking their activities now that they must explicitly ask for permission.