Turning Conversations Into Workflows
This research from Salesforce aims to create structured workflows from customer conversations; a new angle to Agentic Workflows….
Automation still does seem to depend on well-structured workflows to deliver consistent and accurate responses to customer inquiries.
There has been research from OpenAI where they make use of documentation within an organisation to create workflows or what can be referred to as graph data. Essentially nodes and edges which act as decision points.
But what to do if there is no documentation or training manuals to use as a source?
In this research, Salesforce AI Research looks at the potential for the automatic extraction of flows from conversations or past customer interactions.
The extraction process they propose involves two primary stages:
1. A retrieval phase that identifies relevant conversations by focusing on key procedural elements,
2. A structured workflow generation phase that employs a question-answer-based chain-of-thought (QA-CoT) prompting technique.
To thoroughly evaluate the quality of the extracted workflows, they present a simulation framework featuring an automated AI Agent and customer bots, which assesses the workflows’ effectiveness in addressing customer issues.
Extensive testing conducted on datasets shows that the QA-CoT approach enhances workflow extraction.
Also, , the evaluation method they develop proves to align closely with human judgments, offering a dependable and scalable framework for future investigations.
Procedural Elements — Intents & Entities
The image below is an example of procedural elements extracted from a conversation by the GPT-4o mini LLM. Notice the intent, and the slots are analogous to entities. This data format really harken back to the basic building blocks of Chatbots underpinned by an NLU engine managing intents and entities.
The dialog workflow extraction task simplifies and organises the steps required to address customer issues into clear guidelines.
For instance, it includes a workflow for customers who complain about a bill they didn’t purchase (illustrated below), accompanied by a corresponding state machine diagram that outlines various scenarios.
The process emphasises two primary objectives to enhance dialog workflow extraction:
Retrieval, which involves selecting the most relevant past conversations, and
Extraction, which employs a structured question-and-answer approach to create workflows from those conversations.
QA-CoT (Question-Answer Chain-of-Thought)
- Start with a Conversation: The AI is given a historical chat, like a customer saying, “I got billed for something I didn’t buy,” and the agent’s responses.
- Pose Specific Questions: The AI is prompted with questions like: “What is the customer’s main issue?” “What information did the customer provide (e.g., name, account number)?” “What conditions apply (e.g., membership level, system errors)?” “What steps did the agent take to resolve it?”
- Answer Step-by-Step: For each question, the AI provides an answer based on the conversation. For example: Main issue: “Incorrect billing. Info provided: “Account number.” Conditions: “Gold member, no system error.” Steps: “Verify account, check purchase history, issue refund.”
- Build the Workflow: These answers are then stitched together into a clear sequence of steps — an actionable workflow that covers the scenario.
Lastly
The framework presented marks a significant step forward in refining how AI-powered service agents operate.
Transforming messy, real-world conversations into clear, actionable workflows, the authors demonstrate a practical solution to a long-standing problem in customer service automation.
Their focus on retrieval and QA-CoT extraction not only improves accuracy but also sets a foundation for scalable, adaptable AI systems.
As businesses increasingly rely on automation to handle customer interactions, this research offers a blueprint for making those interactions more efficient and trustworthy.
Ultimately, it’s a compelling reminder that the key to better AI lies in learning from the past, turning everyday dialogues into the building blocks of tomorrow’s solutions.
Chief Evangelist @ Kore.ai | I’m passionate about exploring the intersection of AI and language. From Language Models, AI Agents to Agentic Applications, Development Frameworks & Data-Centric Productivity Tools, I share insights and ideas on how these technologies are shaping the future.
