Conversational AI & Intent Driven Development
And The Approach Of Oracle Digital Assistant, AWS Lex V2 & Nuance Mix
Introduction
There are a few basic principles for measuring the performance of a conversational agent.
However, when it comes to creating a chatbot and implementing continuous improvements, there are few challenges. The first being training data…
CDD & IDD
Conversation Driven Development
- To some extend Rasa has pioneered Conversation Driven Development with Rasa X. However, this is a principle which can be gleaned and used anywhere. The dialog tree can be built-out via a conversation, which as a more natural and intuitive way of getting started with conversational development.
- The second part of this CDD approach is to have an NLU Inbox, where user utterances can be annotated and assigned to intents.
- Of course with CDD user conversations are used for development. User conversations can be gleaned from live agent chat, transcribed call centre conversations, etc.
Another approach to chatbot development, which perhaps precedes Conversational Driven Development is Intent Driven Development.
This example from HumanFirst shows how thousands of user utterances can be clustered according to similarity with adjustable granularity and cluster size.
This is a form of semantic search of user utterances where the aim is not to only find keywords, but determine the contextual meaning of the words and sentences.
Intent Driven Development
The advantages of Intent Driven Development:
- Intents are being extended, with more functionality, settings and structure being introduced to intents.
- Intents are not being deprecated, but rather developed.
- Intents should be determined by existing user utterances and not based on products, services and business priorities.
- Intent lists must be flexible, and continuously updated based on user speech. New clusters of utterances should be detected in rapid fashion and made provision for.
- Semantic search and determining similarities are integral to Large Language Models.
- IDD leverages the power of LLM, semantic search and determining the similarity cluster sizes to understand the importance of intents.
- IDD follows the user conversations and leads the direction of conversational agent development.
Current Intent Detection
AWS Lex V2, Nuance Mix and Oracle Digital Assistant all have functionality for intent detection. Later in this article, we will be looking at these implementations in more detail. However, here are an initial observations:
- These frameworks are considering intent detection options, which is a move in the right direction.
- Attention is given to Intent Driven Development and it is part and parcel of the assistant development process.
- The processes are too rigid and the granularity, cluster size and other metrics cannot be set on the fly to tweak the data.
- Confidence scores are not given.
- In the case of Nuance Mix, the process of detection takes a few hours and no progress for or completion notification is sent. This does not contribute to a quick iterative improvement.
- An advantage of Nuance Mix is the simplistic format of training data, and the bulk nature of the upload.
- The ODA approach make provision for PDF and URL endpoint uploads. The different document formats made provision for can be increased.
- Guidance on optimal document formatting will be an advantage.
- ODA generation of similar utterances based on questions submitted, as seen below, reminds of a LLM implementation type.
- The two big disadvantages of Lex V2 intent detection implementation is data size, 10,000 records are required. Added to this, data must be in a Contact Lens output files JSON format.
- With ODA and Nuance Mix, the process is not streamlined, automated or suited for implementation at scale, at this stage. (The functionality might be incorporated into the API, but I could not find it.)
Nuance Mix
Nuance Mix auto-intent functionality is a piece of automation which analyse and group semantically similar sentences. These clusters need to be accepted or rejected by inspection.
Below, the list of generically named intents ca be seen, AUTO_INTENT_N, with the assigned intent, the status (accepted or reject) and the phrase.
The process of naming the intent, accepting or rejecting the intent with a filter which can be set on the right.
Oracle Digital Assistant
Read more about the intent detection functionality of Oracle here.
Below the auto-generated intent name is seen, with the question provided, and the answer. On the right of the page is a list of auto-generated utterances for each intent.
The intent can edited as seen above.
The intent submitted and incorporated into the skill…
Conclusion
There s a move amongst the three chatbot development frameworks mentioned here to incorporating LLM-like models for semantic search of user sentences for clustering and intent detection.
Not only new intents, but also matches for existing intents.
Added to this, ODA has auto-completion functionality, where example utterances are given for ODA to generate auto-completion options for user input. This is a subtle form of disambiguation, a form of Natural Language Generation.
Lastly, the generated intent utterances which are similar to the uploaded questions is also a form of Natural Language Generation.
