Chatbot Demo: Smart Entity Extraction

Glean multiple entities from a complex dialog

When testing a chatbot interface, the plan which is most often followed, is to stick to the dialog or script, and enter short dialogs with single intents. Typically the “happy path” which should work well. This is a scenario in conversation where dialogs are shorter in length and with single intents and or entities.

However, once a user veer off script, and the dialog become longer with multiple sentences and crowded with intents, the bot struggles to extract the intents and entities.

Image for post
Image for post
Chatbot detecting multiple entities/intents per dialog

In this case, a the stronger weighted intent is favored and the others discarded. Or, the chatbot responds with an “I don’t understand” dialog and most often the dialog does not progress beyond this point.

There are cases where users enter text with up to 4 intents in one dialog; or more. Also, the split between intents and entities are not always clear.

This problem necessitates the use of a bona fide NLP (Natural Language Processing) environment. Or at least a higher order NLP first pass for dialogs. Any solution making use of keyword spotting or fixed phrase recognition will fall short here and be caught out as a NLP/NLU impostor.

The solution…

The ideal is for the NLP system to parse the user dialog, and then in turn detect multiple intents within the dialog and capture the entities of that specific intent. This leads to a merger of intents and entities being contextualized.

Hence entities are recognized within a given intent context. This is the key ingredient, the chatbot being able to detect the context of the user utterance. Also, the relative context, within the dialog, of valuable intents and/or entities within the utterance.

As per the example above, the items of interest is picked out of the customer dialog. IBM Watson Assistant makes this particularly easy to setup and configure.

For an in-depth look at the IBM Watson Assistant’s approach to contextually parse user input, have a look at this article I wrote on the subject.

Image for post
Image for post
Example of simple contextual entity detection.

More reading and general information on the subject…

IBM Watson Assistant Contextual Intents:

Natural Language Processing:

Natural Language Understanding:

Unstructured Data:

NLP/NLU, Chatbots, Voice, Conversational UI/UX, CX Designer, Developer, Ubiquitous User Interfaces.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store