Advertisement

Explained: Stemming vs lemmatization in NLP

Stemming is a faster process than lemmatization.

Stemming and lemmatization are text normalisation techniques used in NLP. Essentially, both the techniques break down the words into their root forms. What’s the significance of stemming and lemmatization in NLP and how do they differ? Let’s find out.

Lemmatization

Lemmatization entails reducing a word to its canonical or dictionary form. The root word is called a ‘lemma’.The method entails assembling the inflected parts of a word in a way that can be recognised as a single element. The process is similar to stemming but the root words have meaning.

Lemmatization has applications in:

  1. Biomedicine: Using lemmatization to parse biomedicine literature may increase the efficiency of data retrieval tasks.
  2. Search engines
  3. Compact indexing: Lemmatization is an efficient method for storing data in the form of index values.

For example, NLTK provides WordNetLemmatizer class– a slim cover wrapped around the wordnetCorpus. This class makes use of a function called Morphy() to the WordNetCorpusReader class to find a root word/lemma. 

Source: Linkedin

Lemmatizers need a lot more data on the structure of a language, which makes the creation of lemmatizers a harder process than making a stemming algorithm. 

Stemming

Stemming is a rule-based approach that produces variants of a root/base word. In simple words, it reduces a base word to its stem word. This heuristic process is the simpler of the two as the process involves indiscriminate cutting of the ends of the words. Stemming helps to shorten the look-up and normalise the sentences for a better understanding. The process has two main challenges:

  • Over stemming: The inflected word is cut off so much that the resultant stem is nonsensical. Over stemming can also result in different words with different meanings having the same stem. For example, “universal”, “university” and “universe” is reduced to “univers”. Here, even though these three words are etymologically related, their modern meanings are widely different. Treating them as synonyms in a search engine will lead to inferior search results.
  • Understemming: Here, various inflected words have the same stem despite different meanings. The issue crops up when we have several words that actually are forms of one another. An example of understemming in the Porter stemmer is “alumnus” → “alumnu”, “alumni” → “alumni”, “alumna”/”alumnae” → “alumna”. The English word has Latin morphology, and so these near-synonyms are not combined..

Lemmatization versus stemming

Both procedures involve same methodologies; that is reducing the inflectional forms of each word into a common base or root. However, the main difference is in the way they work and, therefore, the result each returns.

Source: ResearchGate

  • Stemming is a faster process than lemmatization as stemming chops off the word irrespective of the context, whereas the latter is context-dependent.
  • Stemming is a rule-based approach, whereas lemmatization is a canonical dictionary-based approach. 
  • Lemmatization has higher accuracy than stemming. 
  • Lemmatization is preferred for context analysis, whereas stemming is recommended when the context is not important.

To assess the performance of the processes, stemming and lemmatization were compared against the baseline technique used in NLP models provided with the CACM collection. Mean Average Precision (MAP) was used to evaluate document relevancy at the top 10 and 20 document levels.

Source: Inse.org

Both stemming and lemmatization outperformed better than the baseline technique at both the document levels. This indicates that when queries are processed using language modeling techniques, they yield documents that are more relevant compared to queries which are not processed.

While stemmers are quick to create and run, lemmatizers provide a better quality of results or lower margins of error.  

The key applications of these methodologies include:

  1. Information retrieval: Stemming and lemmatization can be used to map documents to general topics and provide search results by indexing.
  2. Document clustering: Stemming and lemmatization reduce the number of tokens to facilitate the transfer of same information. Here, features are estimated by determining the frequency of each token, and then clustering methods are applied.
  3. Sentiment analysis

Download our Mobile App

Kartik Wali
A writer by passion, Kartik strives to get a deep understanding of AI, Data analytics and its implementation on all walks of life. As a Senior Technology Journalist, Kartik looks forward to writing about the latest technological trends that transform the way of life!

Subscribe to our newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day.
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Our Upcoming Events

15th June | Online

Building LLM powered applications using LangChain

17th June | Online

Mastering LangChain: A Hands-on Workshop for Building Generative AI Applications

Jun 23, 2023 | Bangalore

MachineCon 2023 India

26th June | Online

Accelerating inference for every workload with TensorRT

MachineCon 2023 USA

Jul 21, 2023 | New York

Cypher 2023

Oct 11-13, 2023 | Bangalore

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox
MOST POPULAR