Considerations When Measuring Chatbot Success
And What Principles You Should Implement…
Performance measures are important to organizations wanting to track their investment in a conversational interface…
But standards & metrics differ by industry and obviously by companies in each industry. Due to the nascent nature of the technology companies are also eager to learn from one another.
With some overestimate the importance and impact of their chatbot, and other heaving discounting the significant impact their conversational interface is having…
Industry Type Matters
Obviously chatbots are implemented across a vast array of industries. These industries use different parameters. Parameters which they deem as crucial to the measuring of success in their environment.
Banking and financial sectors use chatbots to perform existing tasks faster. And an important driver is lower call volumes and how much savings are incurred from call deflection.
The most common and probably important chatbot performance metric is conversation length and structure. In most cases conversation transcripts are reviewed and manually classed in order so points of improvement noted.
Organisations are aiming for shorter conversations and simplified dialog structures. A conversation or specific dialog always have a happy path which developers hope the user will find and stick to.
A rudimentary and simplistic approach would be to have a repair path, or a few. Paths which intends to bring the conversation back to the happy path from points of digression. Hence ‘repairing’ it.
This approach might lead to a situation called fallback proliferation.
For others, channel retention is an important metric in measuring theirs chatbot’s success. Hence customers or users returning to be serviced via the chatbot channel.
If a company aims to replace traditional communication channels (especially telephone calls) the goal is to have retention in the chatbot medium. The argument is that if the experience was good, to exceptional, the user is bound to return.
Added to this, the call center savings are very tangible.
The caution here is that there are other self-help channels too, other than calling an agent. User interactions must be viewed in its totality in terms of mediums available to the user.
An additional measure of chatbot performance is the ability and degree to which the experience is personalized. Only relevant information should be surfaced to the user.
It may again vary based on industry. Banks look at relevant options to the specific user.
Retail might focus on ways to get users to shop. With the chatbot presenting option as a shot assistant would.
Brands transform themselves into personalized shopping assistant.
Pillars of Trust
In general, a user experience trust manifests as a firm belief in reliability, truth or the ability of someone or something to act for the their good. Among us as humans, trust is seated in credibility and confidence in another’s judgement.
Also part of this is predictability of behavior.
- The belief that the bot can deliver on expectations and the extend to which the bot is seen as effective.
- The ability to not only survive but compete in a marketplace of mediums and channels.
- This is defined as user held belief that that organization is fair and just.
- That the organization acts consistently and dependably.
The emphasis on trust is heightened in certain environments.
Environments such as financial care, heath care and other fields demanding sensitive data by which the user may be exposed to physical, financial or psychological harm. User will not share personal information if they are unsure about security.
A crucial part of trust is related to anthropomorphizezation. Ascribing human attributes to a chatbot. This manifest in the chatbot having a name, a “voice”, personality etc.
We as humans have been placing social expectations on computers for a while. And, athropomorphization increases users’ trust in computer agents.
High quality automation may lead to more fruitful interaction because the machine seems to be more objective and rational that a human.
Humans tend to be less sympathetic toward chatbots; because humans are expected to be imperfect while the opposite is true for automation.
An important point to note is that anthropomorphization is not about creating superficial human-like characteristics. But rather a chatbot which is human like-minded.
I would like end with this quote from Zadrozny et al.
Penned twenty years ago…
Customers have the initiative to express their interests, wishes, or queries directly and naturally, by speaking, typing, and pointing.
The computer system provides intelligent answers or asks relevant questions because it has a model of the domain and a user model.
The business goal of such computerized systems is to create the marketplace of one.
In essence, improved discourse models can enable better one-to-one context for each individual.
Even though we build NL systems, we realize this goal cannot be fully achieved due to limitations of science, technology, business knowledge, and programming environments.