Invaluable support for artificial intelligence (AI), natural language processing (NLP) helps in establishing effective communication between computers and human beings. In recent years, there have been significant breakthroughs in empowering computers to understand human language using NLP. However, the complex diversity and dimensionality characteristics of the data sets, make this simple implementation a challenge in some cases.

The Case For NLP

As text and voice-based data, as well as their practical applications, vary widely, NLP needs to include several different techniques for interpreting human native language. These could range from statistical and machine learning methods to rules-based and algorithmic. NLP has immense potential in real-life application areas such as understanding complete sentences and finding synonyms of matching words, speech recognition, speech translation, and writing complete, grammatically correct sentences, and this need has now become amplified.

While background, domain knowledge and frameworks (e.g. algorithms and tools) are the critical components of the NLP system, it is not a simple and easy task of making machines to understand natural human language. The process includes several activities such as pre-processing, tokenisation, normalisation, correction of typographical errors, Named Entity Reorganization (NER), and dependency parsing. To attain high-quality models, NLP performs an in-depth analysis of user inputs like lexical analysis, syntactic analysis, semantic analysis, discourse integration, and pragmatic analysis, etc.  

Challenges for NLP implementation

Data challenges

The main challenge is information overload, which poses a big problem to access a specific, important piece of information from vast datasets. Semantic and context understanding is essential as well as challenging for summarisation systems due to quality and usability issues. Also, identifying the context of interaction among entities and objects is a crucial task, especially with high dimensional, heterogeneous, complex and poor-quality data.

Data ambiguities add more challenges to contextual understanding. Semantics are important to find the relationship among entities and objects. Entities and object extraction from text and visual data could not provide accurate information unless the context and semantics of interaction are identified. Also, the currently available search engines can search for things (objects or entities) rather than keyword-based search. Semantic search engines are needed because they better understand user queries usually written in natural language.  

The next challenge is the extraction of the relevant and correct information from unstructured or semi-structured data using Information Extraction (IE) techniques. It is necessary to understand the competency and limitations of the existing IE techniques related to data pre-processing, data extraction and transformation, and representations for vast volumes of multidimensional unstructured data. Higher efficiency and accuracy of these IE systems are very important. But, the complexity of big and real-time data brings challenges for ML-based approaches, which are dimensionality of data, scalability, distributed computing, adaptability, and usability. Effectively handling sparse, imbalance and high dimensional datasets are complex.

Another challenge is that a user expects more accurate and specific results from Relational Databases (RDB) for their natural language queries like English. To retrieve information from RDBs for user requests in natural language, the requests have to be converted into formal database queries like SQL. They can also reuse the existing application backend services. This approach leverages NLP to understand the user requests in natural language and prepare application service request URLs to retrieve data from the connected databases.

However, in practice, translating NLP queries to formal DB queries or service request URL is quite complicated due to several factors. These could be the complex DB layouts with table names, columns, and constraints, etc., or the semantic gap between user vocabulary and DB nomenclature. NLP search over databases requires domain-specific models for intent, context, Named Entity identification and extraction. The ambiguity of texts, complex nested entities, identification of contextual information, noise in the form of homonyms, language variability, and missing data pose significant challenges in entity recognition.

Text related challenges

Large repositories of textual data are generated from diverse sources such as text steams on the web, communications through mobile and IoT devices. Though ML and NLP have emerged as the most potent and most used technology applied to the analysis of the text and text classification remains the most popular and the most used technique. Text classification could be Multi-Level (MLC) or Multi-Class (MCC). In MCC, every instance could be assigned to only one class label, whereas MLC is a classification that assigns multiple labels to a single instance.

Solving MLC problems requires an understanding of multi-label data pre-processing for big data analysis. MLC can become very complicated due to the characteristics of real-world data such as high-dimensional label space, label dependency, and uncertainty, drifting, incomplete and imbalanced. Data reduction for large dimensional datasets and classifying multi-instance data is also a challenging task.

Then there are the issues posed by a language translation. The main challenge with language translation is not in translating words, but in understanding the meaning of sentences to provide an accurate translation. Each text comes with different words and requires specific language skills. Choosing the right words depending on the context and the purpose of the content, is more complicated.

A language may not have an exact match for a certain action or object that exists in another language. Idiomatic expressions explain something by way of unique examples or figures of speech. Most importantly, the meaning of particular phrases cannot be predicted by the literal definitions of the words it contains.

Somewhat related is another challenge, that of the inability to accurately deal with new users and products that do not have any history. The user-item rating matrix is very sparse (data sparsity) because stores have many products that will not be rated by many users.

The standard challenge for all new tools, is the process, storage and maintenance. Unlike statistical machine learning, building NLP pipelines is a complex process — pre-processing, sentence splitting, tokenisation, pos tagging, stemming and lemmatisation, and the numerical representation of words. NLP requires high-end machines to build models from large and heterogeneous data sources.

NLP models are larger and consume more memory compared to statistical ML models. Several intermediate and domain-specific models have to be maintained (e.g. sentence identification, pos tagging, lemmatisation, word representation models like TF-IDF, word2vec, etc.). Rebuilding all the intermediate NLP models for new data sets may cost more.

Conclusion

Most of the challenges are due to data complexity, characteristics such as sparsity, diversity, dimensionality, etc. and the dynamic nature of the datasets. NLP is still an emerging technology, and there are a vast scope and opportunities for engineers and industries to deal with many open challenges of implementing NLP systems. 

With the special focus on addressing NLP challenges, organisations can build accelerators, robust, scalable domain-specific knowledge bases and dictionaries that bridges the gap between user vocabulary and domain nomenclature. The proficient and skilled pool of data scientists working for any product engineering services provider will be capable of building customised architectures and NLP pipeline to enable NLP search on different kind of datasets (structured & unstructured).

The quality research and adoption of state-of-the-art technologies like linked data, knowledge graph, etc., can improve the quality of data with enriched meanings, linking data sources with appropriate and meaningful relationships among them. Last but not least, developing accelerators and frameworks make complex NLP implementations more affordable and provide improved performance.

This article is a part of the AIM Writers Programme. If you wish to write for us, email us at info@analyticsindiamag.com