IBM Watson Natural Language Understanding API (Part 2 of 2)

Enable Chatbots to handle longer multiple intent dialog

This is a continuation of the the initial Watson NLU API article. In this article I am looking at Semantic Roles, Sentiment and Syntax.

Sytax is still marked as Experimental within the IBM documentation.

However, I find this section especially interesting and powerful for processing a large dialog.

Semantic Roles allow for parsing a sentence into a subject, action and object form. The sentence “John is running a race for his club.” returns the subject of the sentence as “John”, the tense as present and the verb as run from the text running.

Image for post
Image for post
Semantic Roles JSON Payload

The semantic roles request can be enhanced by setting “keywords” to true; the default being false. Also, entities can be displayed by setting entities to true; again, the default is false.

Here is an example of an augmented request payload.

Augmented Request Payload for Semantic Roles

Keywords returned are: John, race. Entity listed is Person: John. The action verb is run and the tense is present.

Sentiment allows for the analysis of general sentiment of the content, and also sentiment toward specific target phrases.

Image for post
Image for post
Simple Sentiment Score on a Medium Article


Tokens: Tokenization is the process of parsing text into smaller units, referred to as tokens. Tokens are a sequence of characters that are semantically meaningful units.

Here’s an example sentence with its tokens.

Image for post
Image for post
Example to tokenization

Often tokenization is the initial task performed in the NLP process. Tokens can be useful for a range of applications. For instance part of speech tagging, dependency parsing, lemmas and more.

The quality of the higher order feature you are building will ultimately depend on how good your tokenizer for the language is.

Lemma: The words you find in a dictionary are lemmas: the base form, or root form of words.

Lemmatization is commonly used in information retrieval systems or search engines while building the indexes. Words like, editing, commenting, programming can be converted to their root forms (lemmas) before adding to the search index. At query time the text is normalized and compared with the index.

Image for post
Image for post
Example of Lemmas in a Sentence

Part of Speech tags all the tokens in a text with part of speech such as noun, verb, adjective etc. In almost all languages, certain words can mean different things depending on the context and this is where part of speech tagging is very useful.

Image for post
Image for post
Part of Speech Example

Sentence Boundary Detection is essential for any higher order features in NLP/NLU. Sentence boundary detection is an important initial step in building higher order features. For example, to determine the sentiment of a paragraph with multiple sentences, you first have to identify where individual sentences start and end.

Image for post
Image for post
Full Payload for the Syntax Functionality
Image for post
Image for post
Extracted Sentences from the dialog

The sentences where extracted from the dialog as per the example to the left.

This is extremely handy should users enter a longer dialog consisting of a couple of sentences. And obviously with a whole host of intents and entities embedded in the dialog. Any developer having a desire to build a robust and resilient conversational interface, need to make provision for highly unstructured user input.

It make sense to pass the dialog through a higher order NLP layer like this, and establish some basic fundamentals pertaining to the dialog. Based on the output from the NLP, the NLU layer can be aided to accurately assess the intents and entities embedded in the dialog.

The NLP layer can be used for post process analysis also in the case of tone, sentiment and semantics.

Written by

NLP/NLU, Chatbots, Voice, Conversational UI/UX, CX Designer, Developer, Ubiquitous User Interfaces.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store