NLU Remains Relevant For Conversational AI
The integration of LLMs into existing Conversational AI Frameworks (CAIFs) is becoming more and more prominent due to their generative and predictive capabilities. Most CAIFs have already implemented some form of LLMs or have plans to do so in the near future.
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
The main purpose of NLU within these frameworks is to make predictions, assigning user utterances to pre-defined classifications and intents, and extracting entities such as named and industry-specific entities.
NLU can also be used independently to process conversational data offline. To consider the importance of NLU in today’s and future frameworks, below are a few practical considerations.
Efficient Open-Sourced NLU Pipelines
Many CAIFs come with generic, internal NLU pipelines that are likely based on open-source software that has no licensing requirements or third-party obligations.
For instance, with Rasa, a powerful open-source NLU API can be created that supports intents with structure and various entity types. The standard NLU pipeline can be used, and it is highly configurable.
There is no need for a large amount of training data or computing power, and training time is relatively quick. Moreover, there are several lightweight local installation options.
Built-In Efficiencies For Intents & Entities
Intents and entities have become increasingly efficient and organised over time. Many Gartner CAIFs have implemented nested intents or sub-intents, and intents can be split or merged with a drag-and-drop UI.
Entities are only associated with certain intents, and this pairing of an intent with specific entities requires two checks to be performed before the chatbot can respond.
Structures for entities include Compound Contextual Entities, Entity Decomposition, entity groups, roles, etc.
Accurate entity detection is essential for the successful completion of an order, as it prevents the need to ask a user for data they have already provided.
Training Time & No-Code Environments
LLMs require data formatting and transformation to be done through a pro-code environment, which can be tedious and tricky. NLU, however, only needs a few training examples and is usually managed through a no-code studio environment.
Frameworks such as Rasa and Cognigy have made incremental training possible, and IBM Watson Assistant has recently been able to reduce NLU training time significantly.
Comparable Results between LLMs & NLU
When NLU-like classification tasks are performed with a LLM, the results from NLU engines and LLMs are often comparable, with NLU results typically being more reliable and predictable.
However, this is only the case when the LLM is used to its full capacity and NLU is optimised for constructing a classification model with minimal data.
Consistency with NLU
When the same data is submitted, NLU produces consistent and predictable results with minimal to no variation in results.
When testing different LLM providers from a zero to few shot learning perspective, OpenAI appears to yield the best results, followed by AI21 and Cohere.
Unfortunately, it has been difficult to generate consistent and accurate content with LLMs such as Goose AI and Bloom.
Data Entry Is Easy
The majority of chatbot development frameworks have a user-friendly graphical user interface (GUI) for entering natural language understanding (NLU) data.
This point-and-click approach eliminates the need for formatting data into a JSON or CSV structure before importing it, thus avoiding potential errors.
Additionally, the lightweight nature of NLU Engine data exports makes the process of migrating between chatbot frameworks and NLU engines much less intimidating.
Intents are essentially just classes, and the necessity of classifying text and assigning it to one or more labels/classes/intents will never cease.
Recent advancements in terms of intents include the following:
Entities Have Evolved
Accurate entity detection is essential in avoiding the need to re-prompt the user, as it allows for the gathering of entities from highly unstructured input on the first pass. Intents can be seen as verbs, while entities can be seen as nouns.
Read more here.
Easy & Local Installations
If you are interested in exploring natural language understanding, there are several easy and accessible options available.
Rasa is an open-source NLU API that is easy to configure and train, while other NLU tools such as Spacy, Snips, and Cisco MindMeld are also available.
Moreover, these tools are lightweight, open-sourced, and can be installed in any environment, even workstations, without the need for expensive computing, LLMs, or highly technical ML pipelines.
⭐️ Please follow me on LinkedIn for updates on Conversational AI ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
