Boosting its NLP capabilities, IBM has launched new innovative capabilities in IBM Watson Discovery and IBM Watson Assistant, which will empower businesses to deploy and scale sophisticated AI systems. It will leverage NLP with accuracy and efficiency, all while requiring fewer data and training time. It is another significant step by the tech giant to offer advanced ability to understand the language of business. 

With an aim to bring better NLP and NLU offerings to users in its enterprise products, the company has yet again shown its drive to take NLP efforts to a newer height. 

What’s New With IBM’s NLP Game

While recent announcements by IBM focus around language, explainability, and workplace automation, the update around its language capabilities include reading comprehension, FAQ extraction and improving interactions in Watson Assistant. All these products aim to bring resilience, productivity and value for enterprises. Let’s break these features in detail.

Reading comprehension, which is currently available in beta in IBM Watson Discovery, will let users access and search enterprise documents using natural language. Built on top of the question answering system by IBM Research, it is designed to help identify more precise answers in response to natural language queries. The company stated that it also provides scores that indicate how confident the system is in each answer.

Another feature by IBM called FAQ Extraction will enable businesses to keep virtual assistants up to date more efficiently by automatically extracting relevant data from preselected sources. The idea is to keep virtual assistants updated with latest answers while reducing the need for manual intervention. It is to be noted that FAQ Extraction will use NLP technique to automate the extraction of Q&A pairs from FAQ documents.

Apart from these two, the company also announced a new intent classification model in IBM Watson Assistant, which is aimed at understanding an end user’s goal or intent behind engaging with the virtual assistant. It will then be used to train the systems accordingly while enabling greater accuracy in virtual assistants. 

Furthermore, Watson discovery has extended its capabilities in terms of language support as it now supports ten new languages — including Hindi. As the company stated, it can understand Devanagari script and extract information such as business keywords, named entities, perform sentiment analysis, among others. To bring about this capability, IBM Research India collaborated with AI Horizon Network — IIT Bombay — that works on a series of advanced research in AI, ML and NLP applications. 

Talking about the partnership, Prof Pushpak Bhattacharyya, Professor – Department of Computer Science and Engineering, IIT- Bombay said that the complexity of Indian languages makes its adoption and adaption into English-centric NLP quite difficult. He said that through this partnership, they have been able to use machine learning for India language NLP and address challenges such as low resource, understanding of Hindi language sense, intent, sentiment and more. The collaboration has witnessed some interesting developments over the last two years.  

Further stressing on their focus on NLP developments, Gargi Dasgupta –Director, IBM Research India said that they used a combination of AI, deep learning and shallow semantic understanding models to make Watson understand Hindi. 

IBM Improvising Its NLP Capabilities Since Earlier This Year

Earlier this year, IBM made some big improvements to the natural language processing capabilities of its IBM Watson platform. These innovations which were born out of IBM Research’s Project Debater led Watson to understand and analyse challenging aspects of the English language. 

With a capability to debate with humans on complex topics, Debater was involved in a popular TV show — “That’s Debatable” on the topic “Is it time to redistribute the world’s wealth?” earlier this year. Using a feature of NLP called the key point analysis, it could categorise and summarise thousands of public opinions into concrete points. The technique essentially summarises the most significant points from statements and complex documents to produce a concise list of information. 

Having worked on Project Debator’s NLP capabilities throughout the year, IBM is now bringing these into larger IBM products. Not just the language understanding and processing, but the company is building upon sentiment analysis, intent analysis, summarisation and more, to further make the models more accurate. The idea is to work continuously in these projects to improve natural language offering for clients. All these efforts combined are directed towards IBM being a leader in the NLP space, not far away in the future.