How To Migrate Your Chatbot From IBM Watson Assistant To Rasa
Considerations When Moving Your Chatbot To Rasa From Watson Assistant
Introduction
IBM Watson Assistant has a very minimalist design & development interface, yet very efficient. The four chatbot pillars are well defined with a top layer of advanced functionality…
It will hardly ever be a seamless transition when you migrate from one chatbot platform to another. As there are fundamental design and data differences between environments.
Most chatbot platforms are constituted by the same four basic pillars:
- Intents
- Entities
- State Machine / Dialog Management
- Script
There are of course variations between systems:
- The extend to which the core and the NLU elements are separated.
- The propriety structures and data types introduced to entities.
- To which extend entities and intents are merged.
- Special support features.
Watson Assistant Structure
The functionality available In the Watson Assistant console is clearly defined and intuitively structured. It is easy to learn and become familiar with the environment.
Intents
IBM Watson Assistant (WA), at its core has a basic intent and entity structure. Intents are as minimalist as can be.
During the intent creation process, there are two features which aid in the defining of intents. Intent Recommendations and Intent Conflict Resolution.
Bot of these features translate into better defined intents, and translates nicely into the JSON export file.
Hence the leverage these functions lend to the intent creation process is not lost.
It must be noted that entities alone can be exported, in CSV format. But all data structures are lost and only the entity utterance examples are exported.
Entities
Entities within WA does not have any structure attached to them.
This simplifies the transition process.
Synonym Recommendations is a feature which improves the quality of the entities created.
Fortunately, this translates into the export of the skill.
Annotation is key in establishing truly contextual entities.
This is part and parcel of the export and translates well into Rasa.
Entities can also be exported in CSV format. But again, data structures are lost.
Dialog (Conversational Nodes)
This is where it becomes tricky. It must be noted, that IBM Watson does have a NLU API function within their cloud offering.
But for WA, the chat environment is combined with the NLU portion.
This is a completely opposite approach compared to AWS Lex or Microsoft LUIS/Bot Framework.
The console is completely integrated with all of this functionality made available via an feature rich API.
The good news is that in the chatbot JSON export file creates a clear distinction between the intents/entities and the dialog flow.
Hence the manipulation of the file is easy as the segmentation of components are clearly defined.
Watson Specific Functionality
At the very top of the Basic architectural structure of IBM Watson Assistant diagram you will see a set of Watson specific features.
These are Webhooks, Disambiguation, Auto Learning, Auto Correction, Irrelevance Detection, System Entities & Fuzzy Matching.
These features really do augment the chatbot performance and drives down the user effort.
Unfortunately, on migration, these functions will be lost. But, it needs to be mentioned that these functions do not pollute or clutter the skill export. The export file is clean and readable. It is almost as if IBM separates skills into two segments.
The one is the user crafting the chatbot elements, and the Watson augmenting these elements.
Watson Assistant Export Compared To Rasa nlu.md Format
Approach
The general approach I will following migrating to Rasa from Watson Assistant would be to export the WA skill into a JSON formatted file.
From there you can create a conversion program to convert the JSON export into the Rasa nlu.md file format.
The lack of entity structures within WA bodes well for the migration process.
Intents
The format of intent exports in the JSON file from Watson Assistant:
{
"intents": [
{
"intent": "InformationLoan",
"examples": [
{
"text": "How can I get information on loans?"
},
{
"text": "How can I get more information on loans?"
},
{
"text": "how does applying for a loan work"
},
{
"text": "How does loans work"
},
{
"text": "Information on loans please"
},
{
"text": "where do I go for info on loans"
}
],
"description": ""
},Compared to Rasa’s NLU Format:
## intent:ask_transfer_charge
- Will I be charged for transferring money
- do transfers cost something?
- is there a transfer charge?
- Is there a charge
- will i be charged for a transaction?
- do xfers cost something?
- is there a transfer fee
- is there a xfer fee
- how much is the transfer feeForm this comparison creating intent training data from the WA export via text manipulation is straight forward.
Entities
I prefer chatbot development environments where the entities can be annotated within the intent. in many environments the intents and entities are too far removed; but here there is a convergence.
Below is an WA export example, with the intent name, with a text example. You will also see the entity name and the location of the entity.
{
"intent": "TransferMoney",
"examples": [
{
"text": "Can we transfer money from my savings account to my bond account",
"mentions": [
{
"entity": "Money",
"location": [
16,
21
]
},
{
"entity": "ToAccount",
"location": [
52,
56
]
},
{
"entity": "FromAccount",
"location": [
30,
37
]
}
]
},Elsewhere the entity is defined in detail:
{
"entity": "Money",
"values": [
{
"type": "synonyms",
"value": "money",
"synonyms": []
},
{
"type": "synonyms",
"value": "R500",
"synonyms": [
"10000",
"400",
"R500"
]
}
],
"fuzzy_match": true
},Comparing this to the nlu.md file format:
- I want to pay the [current balance](payment_amount) on my [credit card](account_type)From the WA example the nlu.md format can be generated. The only impediment, within WA entities can be defined in isolation. Adding these to Rasa might proof problematic.
Dialog / Stories
Rasa’s approach to dialog management cannot be fore juxtaposed to IBM’s approach. WA has a very rigid dialog management environment. Elements like disambiguation, digression, multiple conditioned responses and a huge amount of business logic are vested in the dialog, and makes for a complex environment.
Slots are also defined in the dialog. Most probably the best approach will be to create user stories from the different permutations which exist within the defined WA dialog. This will be a manual process; with the added advantage of being able to visually inspect the dialog within WA.
Conclusion
To constitute the Rasa NLU portion will be very much straight forward; the NLU API if you like. Most, if not all, of this portion can be automated with a text manipulation program. The Rasa Core portion will take manual intervention.
However, by stepping through the application to extracting the user stories can also be used as an opportunity to optimize and improve the chatbot dialog.
