This Is How Nuance Mix Manages NLU & Training Data
How Does Nuance Approach NLU & What Makes Mix Different
Introduction
As mentioned in an introductory post on Nuance Mix, the Mix Conversational AI ecosystem is a complete end-to-end solution for developing chatbots & voicebots.
Looking at the abstracted conversational elements of the Nuance Mix Conversational AI framework, it is clearly an environment which should scale well.
Elements playing into the favour of Nuance Mix from a Conversational AI perspective are:
- The ability to leverage Azure cloud for accessing geographic regions and regional availability.
- Considering the Gartner Conversational AI report, Microsoft was disqualified for not having a complete end-to-end solution. Nuance Mix will fill the void a product like Composer might have filled. And compliment Bot Framework. Perhaps also act as an extension for Microsoft Power Virtual Agents.
- Nuance has an established presence in Synthesised Speech & Speech Recognition. Two technologies requiring specialized and large sets of training data. This is certainly one of Mix’s distinguishing factors and opens up many options for deployment to speech enabled mediums.
- Nuance Mix NLU can be used as a stand-alone API, which is useful in an enterprise setting where the NLU model can be extended for use in other parts of the business for re-use.
- NLU can be exported and imported as a JSON file, should a bulk editor or other related tools be more effective.
Enhanced Intent Management
A common trait of the Gartner Conversational AI leaders are to augment intents by adding functionality and structure to it. The aim of this functionality is to enhance the management of intents in specific.
This forms part of the merging of the four pillars of chatbots; intents, entities, dialog state management and bot messages.
Here are a few examples from Nuance Mix NLU to illustrate this principle…
As seen above, states of training examples can be set. Intent-assigned indicates that the example will only be used for intent detection. Annotation- assigned points to the sample having been assigned an intent and annotation is complete. And will be used for intent and entity detection.
Lastly, a “pause” icon indicates that the sample, although assigned an intent, is to be Excluded from the model. And subsequently can be included.
Count indicator (optional) showing the number of times the exact sample appears in the corpus. You can also increase or decrease the number of appearances.
This count feature can be useful with larger data sets to detect multiple entries of a training phrase.
Training examples can be easily moved from one intent to another, or a whole new intent can be created.
As seen above, users can test utterances directly in the NLU model. An augmented JSON file is crated with the score and assigned inntent. The training example can be added to the training data with the click of a button.
Annotating Intent training examples are straight-forward; and compound or multiple entities can be defined per intent.
NLU Development Workbench
The NLU development workbench has five sections. After a list of intents and entities are defined the workbench really comes into its own. It needs to be stated that Mix NLU works well for smaller iterations of NLU development & testing.
A: Entities linked to an intent are listed here. Entities can be linked, unlinked or edited. This is a good view into how entities are contextually relevant within intent training data.
B: Intent training data can be annotated, deleted, muted, weights added etc.
C: The entity type and relationships are managed here. It might very well be as a chatbot scales, that entity types and relationships change.
D: Testing NLU in isolation is helpful, and makes for NLU being used in isolation or combined with a complete chatbot environment.
E: The JSON payload based on the current model. This reminds of Cognigy quite a bit and valuable information can be gleaned from this payload. And supplementary information can be used to improve the conversational experience.
NLU Optimise Workbench
The next step is the NLU optimisation workbench. Key features here is a bulk import feature of training data. Different filters can be applied to select training data on defined criteria.
A: All the intents are listed here split out according to the training data assigned. This might seem like a clumsy way of presenting the data, but if a NLU developer is familiar with their data, scrolling down while inspecting the data will be helpful in seeing any aberrations or anomalies.
B: The Status of the training example is shown here, annotated, not annotated, paused or in use.
C: Training data with annotations are visible, annotations can be viewed verbose or shortened.
D: A feature which I found to be useful is to filter training examples based on example length. This is especially relevant when training examples are uploaded in bulk. Training examples which are too short, or too long can create problems in the model. This is a feature I have not come across in the past, but is indeed a handy tool to keep utterances to an optimal size in length.
Automation In NLU Data Optimisation
Auto-intent performs an analysis of UNASSIGNED_SAMPLES intent group, suggesting intents for these samples.
Each previously unassigned sample is tentatively labeled with one of a small number of auto-detected intents present within the set of unassigned samples.
There are two options for Auto-intent:
- Group samples by existing intents. This is the default option.
- Identify new intents and group samples by existing or new intents.
If a sample is recognized as fitting the pattern of an already defined intent, Auto-intent suggests this existing intent.
In the second option, for groups of samples that appear related to each other, but which do not appear to fit the pattern of an existing intent, the samples are labeled generically as:
- AUTO_INTENT_01,
- AUTO_INTENT_02,
- et cetera.
General Observations
It does seem like the ASR and NLU shares a combined data pack as a default. However, custom data packs are available based on organisational settings.
Intents
- Intents in Nuance Mix NLU does not have any inherent structure like in the case of Cognigy or Kore.ai.
- Two newly identified intents can be merged.
- Intents and entities can have relationships. In the example below, the intent iBookFlight is linked to four entities. The link can be removed or edited. Entities to be derived from the intent can have a relationship of isA or hasA.
- This tight feedback-loop between intents and entities are really good in improving accuracy. The idea if intentities is something Microsoft is also implementing.
- The intent for a selected sample of data can be changed.
Entities
- Nuance Mix make provision for four entity types: List, Relationship, Freeform and Regex. List entities are helpful when there is a finite list of entity types you want to extract.
- The Freeform entity is defined and identified contextually within the user utterance. Catching Freeform Entities from a conversation and responding to it adds more resilience to a chatbot.
- These freeform entities can service well for disambiguation menus and avoid not defaulting to fallback proliferation menus. The disambiguation menu can be contextual and ask confirmation on freeform entities detected. These entities will also be tied to an intent, contributing to contextual awareness.
- As seen above, Nuance Mix really does a good job in guiding the user along the way in building the best bot. This is also a very strong element in IBM Watson Assistant, Cognigy & Mix in general.
- The documentation is fused with the design and development environment. Help is offered in an intuitive and topical manner.
Conclusion
Nuance Mix NLU has a few astute product design features:
- Linking ASR and NLU does make sense, and this closer coupling of the two technologies to enable voice definitely make sense.
- The fact that Nuance Mix has in-house ASR and TTS must be an advantage in terms of having a complete end-to-end ecosystem and tighter management of the platform.
- Minority languages are a concern.
- Linking intents and entities with conditions is a brilliant idea.
- The management environment for data speaks to the current approach of augmenting intents and adding structure and improved management to the intents interface.
