More On Chatbots, Irrelevance & Out-Of-Domain Detection…
And How Does HumanFirst Approach This Problem
Introduction
To teach your assistant about subjects it should ignore, you can review your user conversation logs to mark utterances that discuTss off-topic subjects as irrelevant.
Utterances that are marked as irrelevant should be saved as counterexamples, and included as part of the training data. This data teach your assistant to explicitly not answer utterances of this type.
Often there are subjects that you expect customers to ask and that you want your assistant to address eventually, but that you aren’t ready to fully implement yet.
If customers ask about one of these topics in the meantime, the out of scope node is triggered to explain that the assistant can’t help with the current request, but can help them with other things.
Setup challenges, including training data and maintenance, were among the top reasons for not implementing chatbots in enterprises, according to a recent Deloitte survey.
Would the ideal not be for the chatbot to inform the user a request is out of domain? In a nice way. 🙂
According to Deloitte 20% of patents in conversational AI relate to improving the training process. Innovations focus on automating and accelerating the training process to better understand users’ inputs and improve the quality of responses.
This article takes a look at how HumanFirst approaches this challenge…
Building A Sustainable Conversational AI Ecosystems
The initial maturing of the Conversational AI landscape has also lead to the proliferation of various tools. These tools could be divided into three categories:
- Pre-Conversation
- In-Conversation &
- Post-Conversation.
HumanFirst cover both pre and post conversation scenarios. Again, traditionally in chatbot development and maintenance, the focus was always on what happens in conversation. However, there are specific vertical tools available to address the pre and post conversation areas. Of which HumanFirst is one.
As seen above, HumanFirst has an utterance view and also a conversations view. Hence addressing the pre and post conversation scenario.
The HumanFirst interface allows for per utterance view of user input strings or intents. This is analogous to how most NLU interfaces works. With the marked difference that in traditional NLU interfaces the user needs to decide/guess on what intents need to be serviced. And subsequently possible or plausible user utterances associated with this intent need to be crafted.
With HumanFirst customer utterances can be added and with clustering intents are defined automatically.
The conversations tab allows for past dialogs to be interrogated and utterances labeled.
Manually processing user input strings or dialogs, with traditional manual approaches is error prone, inefficient and not feasible long-term.
The HumanFirst Approach To Irrelevance
HumanFirst has interface options available in the form of:
- Web based UI
- API Interface
- CLI
- Pipelines
Here we will only focus on the web UI in general, and intents in specific.
Develop For User Input, & Not For What Is Relevant To Pre-Designed Journeys
The HumanFirst approach is to take any customer utterances or conversations, and generate intents from those utterances, automatically. A user can set granularity and cluster size, and HumanFirst auto-detect groups of utterances. This is with completely raw data, with no annotation, no predefined tags etc. And no predefined entities!
The intersection of the Venn diagram is where the chatbot is effective, this overlap needs to be grown to such an extent that the two circles largely merge.
The aim of HumanFirst is to not have makers start with pre-defined journeys and intents, and continuously retrofit those on actual customer behaviour.
But rather work from customer experiences forward. Below you see that an utterance which is unlabelled, can be moved to the out of scope intent. Notice the icon is different. Out of scope plays a role in the process of training the model.
Above you see that an utterance which is unlabelled, can be moved to the out of scope intent. Notice the icon is different. Out of scope plays a role in the process of training the model.
HumanFirst does not require formatted data from specific NLU solution, and it is not solely a tool of analysing & exported data from NLU solutions. Raw user utterances & conversations are organized in seconds and irrelevance from utterances can quickly be detected.
Whilst setting similarity, suggestions are made from utterance data and these suggestions which are scored, can be added to an intent, or moved to out of scope.
As seen above, the Coverage tool is very helpful in visualising the coverage in terms of intent confidence.
Utterances associated with an intent can be organized according to Date, Margin Score, Uncertainty, Entropy or Similarity.
Utterances associated with an intent can be organised according to Date, Margin Score, Uncertainty, Entropy or Similarity. The associated new utterances to the intent can the throttled with the confidence slider.
Lastly, the newly trained model can be compared to previous models based on five metrics.
To read more about the meaning of the metrics, below is a very good explanation.
Conclusion
Democratising Access To Conversational AI
As seen below, HumanFirst allows for the upload of utterances or whole conversations. Existing data sets can be re-used or public data can be used. The option to use public data is quite fun for experimenting and prototyping.
The idea, that with HumanFirst the ability exist to easily interrogate data, creating structure from utterance data is core to its purpose. Users can even work their way back from conversation data.
Above, HumanFirst has got a few interesting public data sets which can be used for experimentation and framework exploration. Working with familiar data helps with demystifying some core NLP concepts.
There is a definite place for Intent-Driven Development, and out-of-scope or irrelevance can be defined from the onset. This is where user utterances and conversations dictate the journeys which are available.
There are exceptions of course. For instance, in the case of new products & services, the bot might need to pro-actively suggested or prompt the user. Obviously introducing something new to the user which they were not aware of.
