Dialog Flow Generation To Constrain LLM-Based Chatbots
This paper presents an unsupervised method for automatically generating domain-specific dialog flows to guide and constrain LLM-based chatbots.
Introduction
The chart below highlights a key tension in chatbot development: the balance between flexibility and control.
When more control is embedded into the conversation flow, it often leads to tighter constraints, making the interaction less flexible.
Conversely, as the conversation becomes more flexible from the user’s perspective, there is less control over guiding the dialogue toward achieving specific business objectives.
This trade-off is a critical consideration in designing effective chatbots.
Hence, astute chatbot development frameworks strive to find an optimal balance between flexibility and control.
By doing so, they aim to provide users with a more natural and adaptable conversational experience while still guiding interactions toward desired business outcomes.
These frameworks often incorporate adaptive learning mechanisms that allow chatbots to adjust their responses dynamically without sacrificing the overall direction of the conversation.
The goal is to ensure that chatbots remain user-friendly while effectively driving business goals.
Ultimately, this balance enhances both user satisfaction and the chatbot’s effectiveness in fulfilling its intended purpose.
Chatbots are Domain Specific
Yes, there are general-purpose conversational UIs like ChatGPT, HuggingChat, Cohere Coral, Alexa, and Google Home.
These systems constantly strive to establish context by understanding the user and their specific domain. By grasping this context, they can better comprehend and address the conversation, ultimately enhancing the quality and relevance of their responses.
An unsupervised method for automatically generating domain-specific dialog flows that can constrain LLM-based chatbots. Source
However, most chatbots are domain specific, servicing the products and services of a specific organisation or domain.
OK, so considering domain together with flexibility/control, there is obviously an immense opportunity to have LLM-driven dialogs and really the power that LLMs bring…
Recent research addresses the challenge of keeping LLM-driven dialog systems focused within specialised domains, a crucial need in applications like healthcare and customer service.
Due to their broad generalisation capabilities, these chatbots often risk providing inaccurate or irrelevant responses. To tackle this, the authors propose an unsupervised method for automatically generating domain-specific dialog flows that can constrain LLM-based chatbots.
They present two variations of these dialog flows, depending on the availability of in-domain conversation data.
Through both human and automated evaluations across various dialog domains, the study shows that these data-guided dialog flows offer better domain coverage, reducing the need for extensive manual crafting.
The design of dialog schemas thus has significant manual overhead for developers, resulting in scalability and coverage limitations. Source
The figure illustrates the process of intrinsic flow induction. It begins with the generation of an initial flow, which is then refined through a series of feedback, update, and clean-up stages. This is achieved via detailed prompts.
Clusters
Considering the image below, something which is note worth, is the fact that utterances are grouped in clusters which are semantically similar.
Intents
By merging semantically similar clusters that show significant overlap in action intent — identified through cosine similarity between their labels — this approach reduces redundancy in action labels by consolidating clusters with similar actions.
Hence this process of detecting strong signals, surfaces the relevant intents for this conversation.
Flow Generation
The process for generating dialog flows closely mirrors the intrinsic dialog flow generation.
Initially, representative action labels and sample conversations from the domain are included in the prompt for the initial flow generation. Subsequently, feedback, updates, and clean-up steps are applied to refine the dialog flow.
The intrinsic flow method produces broad, general dialog flows but may lack domain-specific patterns observed in real-world conversations.
Conversely, relying solely on domain-specific dialog instances might limit the flow’s scope due to their variability. To address this, a hybrid approach is used, combining the broad coverage of intrinsic flows with the detailed focus from domain-specific data.
This merging process involves prompting GPT-4 to retain distinctive features from the intrinsic flow while eliminating redundant elements, resulting in what is termed the data-guided flow.
Intrinsic Flow Example
Below is an example of an intrinsic flow called city_info_code.txt . Notice how the user intent is defined and encoded. Then each of these intents are mapped to a bot flow which is also coded.
For example U1 goes to B1, and so on.
B0[Bot: Greeting and Asks for City Information Inquiry] --> U1[User: Wants to Know About Attractions]
B0 --> U2[User: Inquires About City Services]
B0 --> U3[User: Asks About Local Regulations]
B0 --> U4[User: Interested in City Events]
B0 --> U45[User: Asks for Personalized Recommendations]
B0 --> U46[User: Inquires About Local Culture and History]
B0 --> U47[User: Wants Info on Dining and Shopping]
B0 --> U48[User: Seeks Accessibility and Accommodation Info]
U1 --> B1[Bot: Provides Information on Attractions]
U2 --> B2[Bot: Details City Services]
U3 --> B3[Bot: Explains Local Regulations]
U4 --> B4[Bot: Shares Information on Upcoming Events]
U45 --> B5[Bot: Gives Personalized Recommendations]
U46 --> B6[Bot: Discusses Local Culture and History]
U47 --> B7[Bot: Informs About Dining and Shopping]
U48 --> B8[Bot: Advises on Accessibility and Accommodation]
B1 --> U9[User: Requests Further Details or Ready to Explore]
B2 --> U9
B3 --> U9
B4 --> U9
B5 --> U9
B6 --> U9
B7 --> U9
B8 --> U9
U9 --> B9[Bot: Provides Additional Info or Encourages Exploration]Data Flow Example
Below is an example of a data flow, called alarm set code, which should be a familiar conversation to many.
Again, notice the intents being defined upfront, and how the bot dialogs are assigned to different user intents. There is also the notion of sub or nested intents.
B0[Bot: Greeting] --> U1[User: Express need for alarm assistance]
U1 --> B1[Bot: Prompt for alarm action]
B1 --> U2[User: Set alarm]
B1 --> U3[User: Edit alarm]
B1 --> U4[User: Delete alarm]
B1 --> U5[User: Copy alarm]
U2 --> B2[Bot: Set Alarm]
U3 --> B3[Bot: Ask for alarm details]
U4 --> B4[Bot: Ask which alarm to delete]
U5 --> B9[Bot: Ask details for copying alarm]
B2 --> U6[User: Confirm alarm set]
B2 --> U7[User: Request to set repeat]
U6 --> B7[Bot: Confirm Completion]
U7 --> B8[Bot: Set repeat and confirm]
B3 --> U8[User: Specify alarm edit details]
B4 --> U9[User: Specify alarm to delete]
B9 --> U14[User: Confirm copy details]
U8 --> B5[Bot: Perform edit and confirm]
U9 --> B6[Bot: Delete alarm and confirm]
U14 --> B10[Bot: Copy alarm and confirm]
B5 --> U10[User: Confirm edit]
B6 --> U11[User: Request additional deletion]
B6 --> U12[User: End conversation]
B10 --> U15[User: Request further changes]
B10 --> U16[User: End conversation]
U10 --> B7
U11 --> B6
U12 --> B11[Bot: Expressing gratitude and farewell]
U15 --> B3
U16 --> B11
B8 --> U13[User: End Conversation]
U13 --> B11Example Conversations
Below is a short snippet of the example conversations used to generate the alarm code reset flow. The complete training file has about 700 conversations. Each line is a conversation defined by the utterances, the session id, domain and task.
{"session_id": "298663b0", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "i need to edit an alarm", "ok which one?", "the \"walk dog\"", "what would you like to do with it?", "change to \"feed cat\"", "changing. do you want to apply this to other alarms with this name?", "yes", "ok", "thanks", "yes"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "9fefc2ac", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I would like to edit a feature on my alarm.", "Alright, what would you like to edit?", "I would like to cancel all my alarms for the next week, but could you first list them all before I do anything?", "Sure. You have 5 alarms for next week. Monday at 8 AM, Tuesday at 9 AM, Wednesday at 10 AM, Thursday at 11 AM, and Friday at 12 PM. Would you like to cancel all of the alarms for next week?", "Alright, thanks. Could you cancel all my alarms?", "Sure. This action cannot be undone once completed. Would you like to proceed with deleting all of your alarms for next week?", "I would like to confirm my cancellation of all my alarms for next week.", "Your alarms for next week have been deleted.", "Very good. Thank you very much!", "You're welcome!"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "63fb88cb", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I needed to change one of my alarms", "I can help you with that, which alarm would you like to change?", "I need to change my 10:30 alarm \"Walk Dog\"", "Ok, what would you like to change it to?", "Let's change it to \"Feed Cat\"", "Ok. You also have \"walk dog\" set for 7PM each day. Would you like to change those alarms as well?", "Erase them. I'm not walking the dog anymore. I hired a kid.", "I can do that, is there anything else I can help you with", "Nope, that's it. Thanks, bot!", "Have a great day"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "8c8066e6", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to set an alarm", "Ok, what day and time?", "Please set an alarm for Monday June 11th at 8am", "Ok, should it be on vibrate or sound?", "Set this alarm to vibrate", "Ok, what should it be called?", "Please Title this alarm 'Pay Bills'", "Ok, anything else?", "No, that's all. Thanks!", "Welcome!"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "dbfc73d9", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello, how may I help you?", "Set an alarm for 6:30pm", "Okay. Do you want it on sound or on vibrate?", "Sound Please", "Okay. What sound should it make? Ringing or do you want a personalized ringtone?", "I want a speaking voice tone i mean", "Okay. What speaking tone would you like?", "A gorgeous lady like", "Okay. When should the alarm go off?", "6:30pm", "What day?", "Everyday", "Okay. Alarm set successfully."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "481aec04", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to move an alarm", "which alarm", "I need my Monday alarm at three changed", "whats the new time and day", "Friday at four please", "do you want to change the name of the alarm", "Please change to ding dong", "done. anything else", "No thanks!", "yw"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "ee342fd6", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need some help with my alarms.", "It would be my pleasure to assist", "I want to set a new alarm.", "I can do that for you. Would you like to title the alarm?", "Wake-up alarm. Set it for 7:00 AM.", "Creating \"Wake-up\" alarm for 7:00am. Would you like any particular sound to wake to?", "Great. Now set it to have an air horn sound.", "Setting to blow horn sound Anything else I can set for you today?", "Excellent. That's all I need. Thank you.", "You're most welcome"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "328d3657", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I want to change the settings for my 7:30pm alarm", "i think you mean at 8. there is no 7:30 alarm", "That's right.", "what do you want to change?", "I want to change the sound. What sound is currently set?", "sound is set at birds chirp right now", "Change that to a chime", "changing", "Thanks", "no problem"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "d22e3a13", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to modify my alarms", "I can help, what would you like to change?", "I want an alarm every Friday at 8 pm", "Every Friday at 8 PM alarm is that correct?", "yes", "Alarm noise or Vibrate?", "both", "What Alarm sound would you like?", "old phone tone", "Ok I have set an alarm for Friday at 8 PM", "thanks"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER"]}
{"session_id": "5cb1b497", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to change an alarm please.", "I can help you with that. What alarm would you like to edit?", "I would like to change my 10:30 am alarm to 11:30 am for Monday.", "Alright, I've made that change for you. Would you like to enable the vibrate feature on this alarm?", "I would like to set this alarm to vibrate.", "Enable vibrate for this alarm. You also have the option of enabling vibrate with sound, would you like to have sound play for this alarm, or only vibrate?", "Only vibrate.", "Ok, I've set this alarm up for you. Would you like to add a title?", "Yes. Add \"Milk Meeting\" as my title.", "I've made this change for you. Anything else I can do?", "That's all."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER"]}
{"session_id": "b479a288", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need some help managing my alarms", "Sure, how can I help with alarms today", "I need an alarm set for ever 2 minutes between 6am and 8am.", "What sound settings do you want for these alarms?", "Shrieking baby sound", "And for what day specifically?", "Monday and Thursday", "Alarms set, anything else I can assist with?", "That is all, thanks bot.", "No problem"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "620925cc", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I want you to set an alarm for me", "What time would you like your alarm set for?", "7am for tomorrow please", "Alright. Your alarm is set for 7 AM.", "And can you set an alarm for Saturday at 3pm?", "Sure. A second alarm is set for Saturday, 3 PM.", "Okay, thank you so much", "You are welcome! May I help you with anything else today?", "No, that was all, goodbyte", "Goodbye. Have a nice day."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "b5b3d0b0", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Hello. i wanted help with setting some alarms.", "I can help with that. What alarms would you like to set?", "Starting at 6 AM tomorrow, I want to have alarms go off every 30 minutes until 8 AM.", "Done. Your alarms starting at 6 AM tomorrow have been set. Would you like to add more alarms?", "No, but can a choose the sound the alarms will make?", "Yes. Do you want the same sound settings for all alarms ?", "Yes. I want a loud sound, similar to a siren.", "Loud siren has been set to your alarms.", "Thank you. You have been very helpful.", "You are welcome."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "c9fb5fc5", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to rename an alarm please", "which alarm", "Please rename the alarm titled 'walk dog' to 'feed cat'", "done. do you want to apply it to similarly named alarms", "Yes. Please change all alarms titled 'walk dog' to 'feed cat'", "are you sure", "Yes, I'm positive", "changes done", "Great! Thanks so much", "yw"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "84adfea3", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to delete an alarm", "Okay. Which alarm do you want to delete?", "I am not entirely sure. I know I have 2 alarms at night, one is for 9pm and the other is for 10pm but I can't remember which is which. What are their titles?", "They both have the same name, actually: Nighttime Alarm.", "Oh man, Do I have an alarm named \"pick Jess up\"?", "Yes, you have several alarms under that name too.", "Okay, just delete all of those please", "Alright, I've deleted all of the alarms with that name.", "You rock! Thanks", "You are welcome. Have a good day."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "d6077f73", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Can you pull up my list of alarms?", "Sure, here is you alarm list.", "What alarm do I have next week on Tuesday?", "You have three meetings for Tuesday.", "Are there alarms for those? I'd like to delete the earliest one", "Three alarms named \"Meetings\" for Tuesday. The earliest alarm is for 8AM, shall I delete that alarm?", "Yes please, delete that", "Alarm \"Meetings\" set for 8AM deleted.", "Great, thanks for that!", "You're welcome"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "c2d70b5c", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need help editing an alamr", "Ok, which alarm?", "My Monday 3pm alarm move it to Friday at 4pm", "Ok, same sound on that alarm?", "yes please", "and same name?", "Yes keep everything the same", "Anything else I can do for you?", "That's all thank you", "Great"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "2e27a20b", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Cancel all my alarms", "You have five alarms set for next week, are you sure you want to cancel all of them?", "Yeah but I want you to list them before cancelling", "Ok, your first alarm is for Monday at 3pm. Do you want to cancel that one?", "Yes. but kindly list them", "Your second alarm is for Tuesday at 1pm.", "Yes", "Your third alarm is set for Wednesday at 1pm.", "Yes", "Your fourth alarm is set for Wednesday at 3pm."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "1763c59a", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I'd like to edit an alarm.", "Sure! Which alarm would you like to edit?", "My alarm for tomorrow at 10 AM.", "Got it! What would you like to change it to?", "I'd like to have it play a sound.", "No problem! Which sound would you like this alarm to play?", "\"Waterfalls\".", "All done! This alarm will now play \"Waterfalls\". Is there anything else I can help you with?", "That's all, thank you!", "You are welcome! Have a great day."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "741540d2", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I want my Saturday alarm set for 7:00 am. to vibrate and make a bell sound.", "Sure. I can help you to set your alarm for 7:00AM and make it vibrate and make a bell sound.", "Thank you. Could you also set up a Monday to Friday alarm for 5:00 am.?", "Sure. I can set you an alarm from Monday to Friday for 5:00AM. What sound would you like that alarm to make?", "I want it to play thunder sounds.", "All done. Your alarm now makes thunder sounds.", "Could you also set up an alarm for 9:00 pm. everyday?", "Yes. I can also set up an alarm for 9:00PM everyday. What sound would you like it to play?", "Horse sounds, please.", "All done! Your 9:00PM alarm will make horse sounds!"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "0b03aafa", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Hi! I need help with my device.", "I can help you set and edit alarms. Do you need assistance with that?", "I want to set my alarm for 7:30 PM to vibrate only.", "You do not currently have an alarm set for 7:30 PM. You do have an alarm set for 8 PM.", "Oh okay. please change the setting for that alarm instead.", "To confirm, you want to set your 8 PM alarm to vibrate only. Correct?", "Yes.", "I will make that change. Do you need to make any other changes?", "No. That's all. Thanks!", "I am here to serve"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "063c3d96", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "i need to set an alarm", "What alarm do you want to set?", "i need to set several alarms", "What are the alarms that you want to set?", "i need to set alarms 15 mins apart between 6am and 8 am", "Okay. Do want the alarms every day of the week?", "yes", "Okay. I have set alarms from Mon-Sun 6 to 8 at 15 minute intervals", "great ty", "You are welcome. Can I help with anything else?"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}My Observations
- The process of flow creation starts with generating an initial flow by using a large language model (LLM). The level of automation can work and serves well for solving for the blank canvas problem or blank canvas paralysis.
- However, starting out with existing conversations is a much more astute approach. Making use of existing dialog turns between customers and service representatives leverages impactful and meaningful conversations.
- I guess someone can follow the same methodology and only replace the initial LLM dialog generation with real-world conversations.
- The aim of this approach is premised on only involving humans in a weak supervision role, hence elevating the burden on designers and builders.
- The dialogs are also premised on a graph approach with nodes and conditional edges.
- This tool is ideal for making use of existing conversations, creating semantically similar clusters, and creating or generating flows which can be used as a starting point when developing a chatbot.
- The flows can be used as a starting point for designing and building chatbots and creating prototype flows to build from.
- I’m not sure what the best why is to implement the generated flows…having a tool to convert the ASCII into a visual representation or some kind of GUI will be helpful.
- Or some kind of development framework which can interpret and ingest the text based design.
✨✨ Follow me on LinkedIn for updates ✨✨
I’m currently the Chief Evangelist @ Kore AI. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.
