Measure Chatbot Customer Effort Using Disambiguation & Auto Learning

Gauge Expended Customer Effort For Problem Solving

Cobus Greyling
6 min readSep 2, 2020



  • What is Customer Effort, and how can it be measured in chatbot conversations?
  • And, how can Disambiguation improve Customer Effort?
  • Can Automatic Learning be employed to improve Customer Effort over time?

Here we will look at three elements, which combined, will have a huge impact on the conversational experience you are building:

  • Disambiguation
  • Automatic Learning
  • Customer Effort

What Is Disambiguation?

At the bottom of this page there are links to in-depth articles on disambiguation. However, in short…

When a user enters an ambiguous utterance, instead of defaulting to the intent with the highest score; the chatbot checks the differences between the top intent scores.

If there is no clear winner, the top 3 or 4 intents are presented to the user. This presentation takes the form of a menu.

The user is presented with a few options which might be applicable to an ambiguous statement.

Allowing the user to disambiguate their own utterance.

This is a far more elegant solution than sending the user off on a conversational path which was not intended.

In other words, disambiguation allows the chatbot to request clarification from the user.

A list of related options should be pretested to the user, allowing the user to disambiguate the dialog by selecting an option from the list.

The list presented should be relevant, even if only vaguely, to the context of the utterance. Hence only contextual options must be presented.

What Is Auto Learning?

It is really a win-win situation when the feedback from the user can be used to improve your NLU model via automatic learning on the fly; as this is invaluable training data vetted by the users.

For the same utterance “paying loan” the Option Pay automatically moves up in the Disambiguation menu.

As the users enter utterances, which are deemed ambiguous by chatbot’s NLU model, a short related menu is presented. When the user selects one of these options within the disambiguation model, a relation is established.

A practical example of a disambiguation menu.

When this link is enforced, the chatbot can learn from this. In future, when similar utterances are entered by the user, the disambiguation menu can be re-ordered to present the more likely selected item first.

Items which are never selected, can be dropped completely from the disambiguation menu.

Hence the chatbot learns automatically from user input what the likely option might be.

From the prototype example above, a loans chatbot skill was created. These intents are very similar and are really ambiguous.

If the utterance paying loan is entered, and Pay is continuously selected, the option moves up in the menu.

Thus creating a menu tailor-made from the utterance, ranked according to likelihood of being selected.

Disambiguation enables chatbots to request help from the user when more than one dialog node might apply to the user’s query.

The combination of disambiguation and automatic learning within a chatbot allows for measuring customer effort.

What Is Customer Effort?

Customer effort is an extremely convenient metric to measure your chatbots performance.

A chatbot’s customer effort measurement boils down to its ease of use. You can see it as the effort exerted or expended by the user to achieve their objective.

Should the customer effort (friction) be too high, the user will revert to other mediums and in this instance the chatbot would have failed in its mission.

From this graph it is evident that the Customer Effort to reach the Pay Node/Dialog is much higher than the Take Loan node.

Here is a practical example of customer effort.

  • If your customer chooses the third option in a list of choices, the effort expended is considered to be higher than the effort expended to choose the first option.
  • Likewise, if a customer chooses None of the above, to signify that none of the options address a need, then the effort metric is even higher.
  • For example, IBM Watson Assistant uses a notebook to plot the Customer Effort metric values graphically, so you have a visual indication of how the metric changes over time.
  • You can also see related information such as disambiguation volume and which dialog nodes are most frequently included as disambiguation list options.
This graph shows the number of clicks expended by the user to achieve their objective.

Customer effort can be calculated and displayed per node and over time. This helps you to identify which nodes are yielding negative NPS and where improvement is required.

Customer Effort can be displayed per dialog node over time.

Above you can see the customer effort for the Take Loan node or dialog is much lower than Pay node. Hence it is clear where the friction needs to be reduced.

Customer Effort over time decreasing during the time auto learning was applied.

The graph above shows the total (red line) and average (blue line) customer effort over time. The shaded area indicates the time auto learning was applied for. There is a considerable drop in customer effort during the time auto learning was applied.

The heat-map above shows the co-occurrence count of the top N node pairs in disambiguation lists. Moving the cursor on each square to view count information.

You can see users showing interest in a loan gets to the node of taking a loan. Showing interest and asking about payment is also co-occurring. Taking a loan and payment also goes together.

It is evident that payment is paramount in users interest.


Combining these three elements:

  • Disambiguation
  • Automatic Learning
  • Customer Effort

makes for a measured and user focused continuous improvement of the user experience.



Cobus Greyling

I explore and write about all things at the intersection of AI & language; LLMs/NLP/NLU, Chat/Voicebots, CCAI.