Intents Need To Be Ground-Truthed Prior To Chatbot Deployment
Reaching desired thresholds of intent recognition are rapidly accelerated by ground-truthing intents prior to launch as opposed to a process of reactive correction.
Introduction
Each organisation has a benchmark or percentage of successful intent recognition for their chatbot or voicebot. The percentage of successful intent recognition, or the percentage of none-intents are often part of the main dashboard and tracked metrics.
Firstly, it needs to be noted that there will always be a certain percentage of none-intents.
Something to keep in mind, digital assistants are linked to an organisation and addresses a finite number of products and services. Hence out-of-domain queries will occur and might register as none-intents.
For instance, at a large mobile operator, our aim was to limit the percentage of none intents to < 10%.
The Challenge
Classification of text and creating labels are very much a standard procedure in the AI world. The challenge though when it comes to digital assistants, is that the classification of the user utterances cannot be an asynchronous process, but needs to be synchronous.
The live conversations need to be classified (assigned to intents) in real-time as the conversation unfolds.
Hence the chatbot needs to have the classifications/intents preloaded, having a good sense of what the ambit of user conversations might entail.
Ground-Truthed Intents
Intent classification is best performed by using a corpus of text data. The text data should ideally be customer conversations, or utterances. And the text data can also be transcribed audio.
This data is then grouped in semantically similar clusters, each of these clusters constitute an intent and can be assigned a label (also referred to as an intent name).
These labeled intents can be considered as ground truth in terms of intents when it comes to coverage. This is also an effective way for solving for the long tail of intent distribution.
Subsequently a machine-learning process can be used for a “weak supervision” approach where new text data are automatically assigned to the ground-truthed intents.
Considering the image above, key elements of data labelling are:
- Human-In-The-Loop methodology
- Accelerated AI-Assisted latent space
- Intelligent intent detection and management at scale
- Intent splitting, merging, hierarchal or nested intents, deactivation of intents
- Detecting intent confusion and disambiguation
- Setting intent granularity and cluster sizes
New utterances which are not related to an existing intent are clustered in separate groupings and marked as new, and hence constitutes a new intent.
This process can also be referred to as Intent Driven Design & Development.
Reactive Approach
Unfortunately most chatbot implementations do not follow a Data Centric approach of NLU Design with intents being deduced from business intents and not real-world customer conversations.
Added to this, often training data is synthetically produced or thought-up.
Subsequent to the chatbot launch, a catch-up process ensues where focus is placed on none-intents.
This negative approach misplaces the focus on none-intents (the conversation customers do not want to have), instead of placing the focus where it should be, establishing ground-truthed intents; hence the conversation customers want to have.
⭐️ Please follow me on LinkedIn for updates on Conversational AI ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
