IBM Watson Assistant Is Focusing On Enhancing Irrelevance Detection In New Releases
…And Across Multiple Languages
Introduction
In IBM Watson Assistant Dialog Skills can still be used, if pre-existing. However, new instances of Watson Assistant can only be created in Action Skills and not Dialog Skills.
From 9 February 2022 all new instances of IBM Watson Assistant (WA) defaults to the new interface. This new interface or experience is Actions based and not independent NLU/Dialog Skill based.
Seemingly, from the new instances, the previous interface is inaccessible. However, for existing implementations in the previous interface, no work will be lost when toggling or switching between the two environments.
For existing implementations, you can switch back to the classic/previous experience at any time by clicking Switch to classic experience from the account menu.
This prompted doubt on the future existence of Dialog Skills. Judging on all the new releases IBM Watson Assistant has, especially on Irrelevance Detection, and across different languages; the future of Dialog Skills seem secure.
On 25 March 2022, improved irrelevance detection for Dutch was introduced. Irrelevance detection for Dutch disregards any punctuation in an input sentence.
For example, you can now expect the same confidence score for the following two inputs: ik ben een kleine krijger? and ik ben een kleine krijger. In this example, the question mark (?) doesn't affect the confidence score.
Also, on 1 March 2022, IBM enhanced Watson Assistant’s irrelevance detection.
The detection classification algorithm was enhanced to use any provided counter examples as part of the training.
The assurance is given that existing workspaces sans counter examples will not be affected.
The update is relevant to the languages: English, French, Spanish, Italian. And very importantly, the general universal language model.
A few key considerations:
- Irrelevance detection is a key and distinguishing feature of IBM Watson Assistant. Enhancements on it is key and this development is surely a good sign.
- This feature is only available in classic IBM Watson Assistant.
- All new instances of Watson Assistant points to the new WA. No new instance of WA can be created in the classic WA (with Dialog Skills).
- Hence this feature is only available in existing classic WA instances.
- Hopefully soon Dialog Skills will be available in new WA. Or access to new classic instances granted.
Utterances with an assigned intent can be marked as irrelevant and saved as counter examples in the workspace. And hence included as part of the training data. This teaches the chatbot to explicitly not answer utterances of this nature.
Defining Irrelevance With Watson Assistant
In BM Watson Assistant you can teach your dialog skill to recognize when input is about topics which is OOD.
User conversations can be reviewed and marked as off-topic subjects and thus irrelevant.
These user utterances marked as irrelevant are saved as counterexamples and included as part of the training data.
Hence training the assistant to explicitly not answer utterances of this type.
While testing your dialog, you can mark an intent, based on a user input, as irrelevant directly from the Try it out pane.
Care should be taken during this process…
- There is no way to access or change the inputs from the user interface later.
- The only way to reverse the identification of an input as being irrelevant is to use the same input in a test integration channel, and then explicitly assign it to an intent.
When you set Irrelevance Detection to enabled, an alternative method for evaluating the relevance of a newly submitted utterance is triggered in addition to the standard method.
To switch this feature on for IBM Watson Assistant:
- From the Skills page, open your skill.
- From the skill menu, click Options.
- On the Irrelevance detection page, choose Enhanced.
This supplemental method examines the structure of the new utterance and compares it to the structure of the user example utterances in your training data.
This approach help chatbots that have few or no counterexamples, recognize irrelevant utterances.
Looking a the image above, the skill has no intent for account status. Hence with irrelevance detection switched off, it defaults to the intent #Balances.
With the feature switched on, the utterance rightly goes to a Irrelevant status.
To build a chatbot that provides a more customized experience, you want it to use information from data that is derived from within the application’s domain.
And by adding your own counterexamples. Even if only a few.
How Does it Work?
Understanding what your users say is based on two pillars:
- Intents you need the chatbot to address. Examples of this for a courier company might me order tracking, package collection etc. Training takes place by defining intents and adding example user utterances. These user utterances are grouped according to intent based on what users might say.
- Defining counter examples which should be deemed as irrelevant or which needs to be ignored.
Time can be spent to understand the target audience’s domain and specific intentions. And subsequently craft or source training data accordingly.
Counter examples should be part of this training data.
The aim of enhanced irrelevance detection is to mitigate any vulnerability in counter examples.
According to IBM:
When enabled, an alternative method for evaluating the relevance of a newly submitted utterance is triggered in addition to the standard method.
The best approach is to add in domain examples, and also counter examples in an iterative fashion with continued monitoring.
Negate False Intent Assignment
Often, instead of stating the intent is out of scope, in a desperate attempt to handle the user utterance, the chatbot assigns the best fit intent to the user; often wrong.
Alternatively the chatbot continues to inform the user it does not understand; and having the user continuously rephrasing the input. Instead of the chatbot merely stating the question is not part of its domain.
A handy design element is to have two or three sentences serve as an intro for first-time users; sketching the gist of the chatbot domain.
The traditional approaches are:
- Many “out-of-scope” examples are dreamed up and entered. Which is hardly ever successful.
- Attempts are made to disambiguate the user input.
But actually, the chatbot should merely state that the query is outside of its domain and give the user guidance.
Conclusion
In general chatbots are designed and developed for a specific domain. These domains are narrow and applicable to to the concern of the organization they serve. Hence chatbots are custom and purpose built as an extension of the organization’s operation, usually to allow customers to self-service.
As an added element to make the chatbot more interactive and lifelike, and to anthropomorphize the interface, small talk is introduced. Also referred to as chitchat.
But what happens if a user utterance falls outside this narrow domain? With most implementations the highest scoring intent is assigned to the users utterance, in a frantic attempt to field the query.
