Agentic Discovery
AI Agents are agentic software applications which primarily live within a digital environment. Agentic applications are premised on basic LLM related skills like decomposing a problem into smaller sub-tasks, and leveraging reasoning to move from task to task, by observing and assessing each task. The Self-Discover approach allows Large Language Models Automatically Compose Their Own Reasoning Structures.
18 min readSep 9, 2024
What is exciting, is that LangChain has created an implementation based on the self-Discovery research, which they call Self Discover Agent. I have the complete working code at the end of this article, and I ran the agent making use of gpt-4o-mini.
Introduction
The study on Self-Discovery: Large Language Models Self-Compose Reasoning Structures investigates how Large Language Models (LLMs) can autonomously generate and apply problem-solving strategies.
This is done without the need for prior training on specific tasks.
Instead, the model uses its own reasoning processes to uncover and implement solutions based on a set of high-level heuristics.
In simple terms, Self-Discovery operates in two stages:
Stage 1: The model examines a given task and selects useful problem-solving strategies (called reasoning modules as seen below), which are structured like a step-by-step plan.
These reasoning modules cover cognitive techniques such as critical thinking, breaking down problems, and analysing risks.
Stage 2: The model uses the structured reasoning process from Stage 1 to solve specific instances of the task, filling in the steps based on the task details.
This process is an example of agentic discovery, where the model actively selects and adapts its approach to the task, rather than following pre-programmed rules.
It uses 39 reasoning modules that mimic human cognitive approaches to problem-solving, such as considering alternative perspectives or identifying key assumptions.
These modules work like interchangeable tools that the model combines to create a tailored solution for each task.
Some Observations
- Self-Contained Environment: The agent in the study operates within a contained environment and does not integrate external APIs or libraries. Instead, it relies on reasoning modules that guide it to generate solutions to complex tasks.
- Self-Discovery Process: The self-discovery process indeed involves three key steps: selecting relevant reasoning modules, adapting those modules to fit the specific task, and then implementing a structured reasoning plan. This stepwise approach is core to how the agent tackles problems.
- Modules as Prompts: The modules are essentially prompts or guiding questions rather than sophisticated tools. These modules serve as high-level problem-solving heuristics, such as use critical thinking or think step by step. They are not advanced systems but are used to guide the reasoning process.
- No External Systems: This approach does not currently extend to integrating external systems, but the process could potentially be adapted for that in the future.
- So it’s important to note that the current framework is focused on internal reasoning without external system integration.
- However, the framework could be adapted to incorporate external tools in future iterations, which will make it an incredibly powerful framework with web or mobile OS discovery capabilities.
- Consider here the work done by Ferrit-UI (Apple), WebVoyager, Microsoft and others in terms of web discovery.
- The idea of extending this to interact with third-party systems or APIs could enhance the agent’s ability to leverage external data or computation for more complex tasks.
Advantages
This approach offers several advantages:
- It benefits from combining multiple reasoning modules rather than relying on a single method.
- It is computationally efficient, requiring only three additional inference steps at the task level.
- It performs better than methods that use more computational resources, such as self-consistency.
- The reasoning structure is intrinsic to the task, providing greater interpretability than optimised prompts.
Inspired by Humans
SELF-DISCOVER is inspired by how humans use prior knowledge and skills to solve problems. When faced with a new challenge, humans typically first draw on their past experience to identify relevant knowledge and skills, then apply them to the task, and finally connect various skills to solve the problem. SELF-DISCOVER mirrors this process through two distinct stages.
In Stage 1, the framework works to uncover the intrinsic reasoning structure of a task using meta-reasoning. This involves guiding language models (LLMs) through three meta-prompts to select, adapt, and implement a reasoning structure. The reasoning structure is formatted in key-value pairs, similar to JSON, to enhance interpretability and quality.
Stage 1 only needs to be run once for each task, as it operates on the task level.
In Stage 2, the discovered reasoning structure is used to solve every instance of the task, with models following the structure step-by-step to reach the final answer.
Considering the image below, an illustration of the three actions of SELF-DISCOVER: language models (LMs) are used to compose a coherent reasoning structure by selecting relevant modules, adapting them to task-specific descriptions, and implementing the reasoning structure in a JSON format.
Below, self-discovery is compared to other approaches.
More On Self-Discover
The SELF-DISCOVER approach is designed to uncover the unique reasoning structure of each task while maintaining high computational efficiency.
It is inspired by how humans internally develop reasoning processes to solve problems.
The method works by selecting atomic reasoning modules, such as break down into subtasks and critical thinking.
In Stage 1, SELF-DISCOVER creates a coherent reasoning structure for the task using examples without labels.
In Stage 2, the model follows the discovered reasoning structure to solve instances of the task.
SELF-DISCOVER performs best on tasks that require diverse world knowledge.
LangChain Implementation
Below is a complete working example of the LangChain implementation…
pip install -q langchain_openai langchain
####
from langchain_openai import ChatOpenAI
####
import os
os.environ['OPENAI_API_KEY'] = str("<Your API Key Goes Here>")
####
model = ChatOpenAI(temperature=0, model="gpt-4o-mini")
####
from langchain import hub
from langchain_core.prompts import PromptTemplate
####
select_prompt = hub.pull("hwchase17/self-discovery-select")
####
select_prompt.pretty_print()
####
'''
Select several reasoning modules that are crucial to utilize in order to solve the given task:
All reasoning module descriptions:
{reasoning_modules}
Task: {task_description}
Select several modules are crucial for solving the task above:
'''
####
adapt_prompt = hub.pull("hwchase17/self-discovery-adapt")
####
adapt_prompt.pretty_print()
####
'''
Rephrase and specify each reasoning module so that it better helps solving the task:
SELECTED module descriptions:
{selected_modules}
Task: {task_description}
Adapt each reasoning module description to better solve the task:
'''
####
structured_prompt = hub.pull("hwchase17/self-discovery-structure")
####
structured_prompt.pretty_print()
####
'''
Operationalize the reasoning modules into a step-by-step reasoning plan in JSON format:
Here's an example:
Example task:
If you follow these instructions, do you return to the starting point? Always face forward. Take 1 step backward. Take 9 steps left. Take 2 steps backward. Take 6 steps forward. Take 4 steps forward. Take 4 steps backward. Take 3 steps right.
Example reasoning structure:
{
"Position after instruction 1":
"Position after instruction 2":
"Position after instruction n":
"Is final position the same as starting position":
}
Adapted module description:
{adapted_modules}
Task: {task_description}
Implement a reasoning structure for solvers to follow step-by-step and arrive at correct answer.
Note: do NOT actually arrive at a conclusion in this pass. Your job is to generate a PLAN so that in the future you can fill it out and arrive at the correct conclusion for tasks like this
'''
####
reasoning_prompt = hub.pull("hwchase17/self-discovery-reasoning")
####
reasoning_prompt.pretty_print()
####
'''
Follow the step-by-step reasoning plan in JSON to correctly solve the task. Fill in the values following the keys by reasoning specifically about the task given. Do not simply rephrase the keys.
Reasoning Structure:
{reasoning_structure}
Task: {task_description}
'''
####
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
####
select_chain = select_prompt | model | StrOutputParser()
####
adapt_chain = adapt_prompt | model | StrOutputParser()
####
structure_chain = structured_prompt | model | StrOutputParser()
####
reasoning_chain = reasoning_prompt | model | StrOutputParser()
####
overall_chain = (
RunnablePassthrough.assign(selected_modules=select_chain)
.assign(adapted_modules=adapt_chain)
.assign(reasoning_structure=structure_chain)
.assign(answer=reasoning_chain)
)
####
reasoning_modules = [
"1. How could I devise an experiment to help solve that problem?",
"2. Make a list of ideas for solving this problem, and apply them one by one to the problem to see if any progress can be made.",
# "3. How could I measure progress on this problem?",
"4. How can I simplify the problem so that it is easier to solve?",
"5. What are the key assumptions underlying this problem?",
"6. What are the potential risks and drawbacks of each solution?",
"7. What are the alternative perspectives or viewpoints on this problem?",
"8. What are the long-term implications of this problem and its solutions?",
"9. How can I break down this problem into smaller, more manageable parts?",
"10. Critical Thinking: This style involves analyzing the problem from different perspectives, questioning assumptions, and evaluating the evidence or information available. It focuses on logical reasoning, evidence-based decision-making, and identifying potential biases or flaws in thinking.",
"11. Try creative thinking, generate innovative and out-of-the-box ideas to solve the problem. Explore unconventional solutions, thinking beyond traditional boundaries, and encouraging imagination and originality.",
# "12. Seek input and collaboration from others to solve the problem. Emphasize teamwork, open communication, and leveraging the diverse perspectives and expertise of a group to come up with effective solutions.",
"13. Use systems thinking: Consider the problem as part of a larger system and understanding the interconnectedness of various elements. Focuses on identifying the underlying causes, feedback loops, and interdependencies that influence the problem, and developing holistic solutions that address the system as a whole.",
"14. Use Risk Analysis: Evaluate potential risks, uncertainties, and tradeoffs associated with different solutions or approaches to a problem. Emphasize assessing the potential consequences and likelihood of success or failure, and making informed decisions based on a balanced analysis of risks and benefits.",
# "15. Use Reflective Thinking: Step back from the problem, take the time for introspection and self-reflection. Examine personal biases, assumptions, and mental models that may influence problem-solving, and being open to learning from past experiences to improve future approaches.",
"16. What is the core issue or problem that needs to be addressed?",
"17. What are the underlying causes or factors contributing to the problem?",
"18. Are there any potential solutions or strategies that have been tried before? If yes, what were the outcomes and lessons learned?",
"19. What are the potential obstacles or challenges that might arise in solving this problem?",
"20. Are there any relevant data or information that can provide insights into the problem? If yes, what data sources are available, and how can they be analyzed?",
"21. Are there any stakeholders or individuals who are directly affected by the problem? What are their perspectives and needs?",
"22. What resources (financial, human, technological, etc.) are needed to tackle the problem effectively?",
"23. How can progress or success in solving the problem be measured or evaluated?",
"24. What indicators or metrics can be used?",
"25. Is the problem a technical or practical one that requires a specific expertise or skill set? Or is it more of a conceptual or theoretical problem?",
"26. Does the problem involve a physical constraint, such as limited resources, infrastructure, or space?",
"27. Is the problem related to human behavior, such as a social, cultural, or psychological issue?",
"28. Does the problem involve decision-making or planning, where choices need to be made under uncertainty or with competing objectives?",
"29. Is the problem an analytical one that requires data analysis, modeling, or optimization techniques?",
"30. Is the problem a design challenge that requires creative solutions and innovation?",
"31. Does the problem require addressing systemic or structural issues rather than just individual instances?",
"32. Is the problem time-sensitive or urgent, requiring immediate attention and action?",
"33. What kinds of solution typically are produced for this kind of problem specification?",
"34. Given the problem specification and the current best solution, have a guess about other possible solutions."
"35. Let’s imagine the current best solution is totally wrong, what other ways are there to think about the problem specification?"
"36. What is the best way to modify this current best solution, given what you know about these kinds of problem specification?"
"37. Ignoring the current best solution, create an entirely new solution to the problem."
# "38. Let’s think step by step."
"39. Let’s make a step by step plan and implement it with good notation and explanation.",
]
task_example = "Lisa has 10 apples. She gives 3 apples to her friend and then buys 5 more apples from the store. How many apples does Lisa have now?"
task_example = """This SVG path element <path d="M 55.57,80.69 L 57.38,65.80 M 57.38,65.80 L 48.90,57.46 M 48.90,57.46 L
45.58,47.78 M 45.58,47.78 L 53.25,36.07 L 66.29,48.90 L 78.69,61.09 L 55.57,80.69"/> draws a:
(A) circle (B) heptagon (C) hexagon (D) kite (E) line (F) octagon (G) pentagon(H) rectangle (I) sector (J) triangle"""
####
reasoning_modules_str = "\n".join(reasoning_modules)
####
####
overall_chain.invoke(
{"task_description": task_example, "reasoning_modules": reasoning_modules_str}
)
####And the output:
{'task_description': 'This SVG path element <path d="M 55.57,80.69 L 57.38,65.80 M 57.38,65.80 L 48.90,57.46 M 48.90,57.46 L\n45.58,47.78 M 45.58,47.78 L 53.25,36.07 L 66.29,48.90 L 78.69,61.09 L 55.57,80.69"/> draws a:\n(A) circle (B) heptagon (C) hexagon (D) kite (E) line (F) octagon (G) pentagon(H) rectangle (I) sector (J) triangle',
'reasoning_modules': '1. How could I devise an experiment to help solve that problem?\n2. Make a list of ideas for solving this problem, and apply them one by one to the problem to see if any progress can be made.\n4. How can I simplify the problem so that it is easier to solve?\n5. What are the key assumptions underlying this problem?\n6. What are the potential risks and drawbacks of each solution?\n7. What are the alternative perspectives or viewpoints on this problem?\n8. What are the long-term implications of this problem and its solutions?\n9. How can I break down this problem into smaller, more manageable parts?\n10. Critical Thinking: This style involves analyzing the problem from different perspectives, questioning assumptions, and evaluating the evidence or information available. It focuses on logical reasoning, evidence-based decision-making, and identifying potential biases or flaws in thinking.\n11. Try creative thinking, generate innovative and out-of-the-box ideas to solve the problem. Explore unconventional solutions, thinking beyond traditional boundaries, and encouraging imagination and originality.\n13. Use systems thinking: Consider the problem as part of a larger system and understanding the interconnectedness of various elements. Focuses on identifying the underlying causes, feedback loops, and interdependencies that influence the problem, and developing holistic solutions that address the system as a whole.\n14. Use Risk Analysis: Evaluate potential risks, uncertainties, and tradeoffs associated with different solutions or approaches to a problem. Emphasize assessing the potential consequences and likelihood of success or failure, and making informed decisions based on a balanced analysis of risks and benefits.\n16. What is the core issue or problem that needs to be addressed?\n17. What are the underlying causes or factors contributing to the problem?\n18. Are there any potential solutions or strategies that have been tried before? If yes, what were the outcomes and lessons learned?\n19. What are the potential obstacles or challenges that might arise in solving this problem?\n20. Are there any relevant data or information that can provide insights into the problem? If yes, what data sources are available, and how can they be analyzed?\n21. Are there any stakeholders or individuals who are directly affected by the problem? What are their perspectives and needs?\n22. What resources (financial, human, technological, etc.) are needed to tackle the problem effectively?\n23. How can progress or success in solving the problem be measured or evaluated?\n24. What indicators or metrics can be used?\n25. Is the problem a technical or practical one that requires a specific expertise or skill set? Or is it more of a conceptual or theoretical problem?\n26. Does the problem involve a physical constraint, such as limited resources, infrastructure, or space?\n27. Is the problem related to human behavior, such as a social, cultural, or psychological issue?\n28. Does the problem involve decision-making or planning, where choices need to be made under uncertainty or with competing objectives?\n29. Is the problem an analytical one that requires data analysis, modeling, or optimization techniques?\n30. Is the problem a design challenge that requires creative solutions and innovation?\n31. Does the problem require addressing systemic or structural issues rather than just individual instances?\n32. Is the problem time-sensitive or urgent, requiring immediate attention and action?\n33. What kinds of solution typically are produced for this kind of problem specification?\n34. Given the problem specification and the current best solution, have a guess about other possible solutions.35. Let’s imagine the current best solution is totally wrong, what other ways are there to think about the problem specification?36. What is the best way to modify this current best solution, given what you know about these kinds of problem specification?37. Ignoring the current best solution, create an entirely new solution to the problem.39. Let’s make a step by step plan and implement it with good notation and explanation.',
'selected_modules': "To solve the task of identifying the shape drawn by the given SVG path element, the following reasoning modules would be crucial:\n\n1. **Critical Thinking (Module 10)**: Analyzing the SVG path data requires logical reasoning to interpret the coordinates and lines described in the path. This involves evaluating the evidence provided by the path commands and understanding how they connect to form a shape.\n\n2. **Break Down the Problem (Module 9)**: Breaking down the SVG path into its individual segments will help in visualizing how the lines connect and what shape they form. This involves examining each command in the path data to understand the structure.\n\n3. **Identify the Core Issue (Module 16)**: Understanding that the core issue is to determine the geometric shape represented by the path will help focus the analysis on the characteristics of the shape.\n\n4. **Evaluate Relevant Data (Module 20)**: Analyzing the specific coordinates and commands in the SVG path data will provide insights into the shape's properties, such as the number of sides and angles.\n\n5. **Consider Alternative Perspectives (Module 7)**: Exploring different interpretations of the path data can help ensure that all possible shapes are considered before arriving at a conclusion.\n\n6. **Simplify the Problem (Module 4)**: Simplifying the path by visualizing or sketching it can make it easier to identify the shape without getting lost in the complexity of the path commands.\n\nBy utilizing these reasoning modules, one can effectively analyze the SVG path and determine the shape it represents.",
'adapted_modules': 'To effectively identify the shape represented by the given SVG path element, the following reasoning modules have been rephrased and specified to enhance their utility in solving the task:\n\n1. **Critical Analysis of Path Commands (Module 10)**: This module focuses on systematically interpreting the SVG path commands and their associated coordinates. It involves assessing how each command (e.g., Move, Line) contributes to the overall structure of the shape, ensuring a clear understanding of how the segments connect to form a complete figure.\n\n2. **Segment Decomposition (Module 9)**: This module emphasizes the importance of breaking down the SVG path into distinct line segments. By analyzing each segment individually, one can visualize the connections and relationships between them, which aids in recognizing the overall geometric shape formed by the path.\n\n3. **Core Shape Identification (Module 16)**: This module centers on pinpointing the primary objective of the task: to classify the geometric shape represented by the SVG path. By focusing on the characteristics such as the number of vertices and the arrangement of lines, one can streamline the analysis towards identifying the correct shape.\n\n4. **Data Evaluation for Shape Properties (Module 20)**: This module involves a detailed examination of the coordinates and commands within the SVG path data. By evaluating these elements, one can derive critical properties of the shape, such as the total number of sides, angles, and any symmetry, which are essential for accurate shape classification.\n\n5. **Exploration of Shape Variations (Module 7)**: This module encourages considering various interpretations of the SVG path data. By exploring different potential shapes that could be formed from the path commands, one can ensure that all possibilities are accounted for, leading to a more comprehensive conclusion.\n\n6. **Visualization and Simplification (Module 4)**: This module advocates for visualizing or sketching the SVG path to simplify the analysis process. By creating a visual representation of the path, one can more easily identify the shape without becoming overwhelmed by the complexity of the commands, facilitating a clearer understanding of the geometric figure.\n\nBy applying these refined reasoning modules, one can effectively analyze the SVG path and accurately determine the shape it represents from the provided options.',
'reasoning_structure': 'Here’s a step-by-step reasoning plan in JSON format for analyzing the given SVG path element to identify the shape it represents:\n\n```json\n{\n "Step 1": {\n "Module": "Critical Analysis of Path Commands",\n "Description": "Interpret the SVG path commands and their coordinates. Identify the starting point and the sequence of commands."\n },\n "Step 2": {\n "Module": "Segment Decomposition",\n "Description": "Break down the SVG path into individual line segments. List each segment and its endpoints."\n },\n "Step 3": {\n "Module": "Core Shape Identification",\n "Description": "Analyze the number of distinct segments and their connections to determine the overall shape structure."\n },\n "Step 4": {\n "Module": "Data Evaluation for Shape Properties",\n "Description": "Evaluate the coordinates and commands to derive properties such as the number of sides and angles."\n },\n "Step 5": {\n "Module": "Exploration of Shape Variations",\n "Description": "Consider different interpretations of the path data. Identify any alternative shapes that could be formed."\n },\n "Step 6": {\n "Module": "Visualization and Simplification",\n "Description": "Create a visual representation of the SVG path to simplify the analysis and enhance understanding of the shape."\n },\n "Step 7": {\n "Module": "Final Shape Classification",\n "Description": "Based on the analysis and visual representation, classify the shape according to the provided options."\n }\n}\n```\n\nThis structured plan outlines the reasoning process that can be followed to analyze the SVG path and identify the shape it represents. Each step corresponds to a specific module and provides a clear description of the actions to be taken.',
'answer': '```json\n{\n "Step 1": {\n "Module": "Critical Analysis of Path Commands",\n "Description": "The SVG path starts at point (55.57, 80.69) and consists of a series of line commands (L) connecting various points. The sequence of commands indicates a closed shape as it returns to the starting point."\n },\n "Step 2": {\n "Module": "Segment Decomposition",\n "Description": "The path can be broken down into the following segments: 1) (55.57, 80.69) to (57.38, 65.80), 2) (57.38, 65.80) to (48.90, 57.46), 3) (48.90, 57.46) to (45.58, 47.78), 4) (45.58, 47.78) to (53.25, 36.07), 5) (53.25, 36.07) to (66.29, 48.90), 6) (66.29, 48.90) to (78.69, 61.09), and finally 7) (78.69, 61.09) back to (55.57, 80.69)."\n },\n "Step 3": {\n "Module": "Core Shape Identification",\n "Description": "The path consists of 7 distinct segments that connect back to the starting point, indicating a closed shape with 7 vertices."\n },\n "Step 4": {\n "Module": "Data Evaluation for Shape Properties",\n "Description": "The shape has 7 sides and 7 angles, which corresponds to the definition of a heptagon."\n },\n "Step 5": {\n "Module": "Exploration of Shape Variations",\n "Description": "While the path primarily suggests a heptagon, variations in the angles and lengths of the segments could lead to different visual representations, but they would still conform to the heptagon classification."\n },\n "Step 6": {\n "Module": "Visualization and Simplification",\n "Description": "Visualizing the path confirms that it forms a closed shape with 7 sides, reinforcing the identification of a heptagon."\n },\n "Step 7": {\n "Module": "Final Shape Classification",\n "Description": "Based on the analysis and visual representation, the shape represented by the SVG path is classified as a heptagon."\n }\n}\n```'}{
"Step1": {
"Module": "Critical Analysis of Path Commands",
"Description": "The SVG path starts at point(55.57,
80.69)and consists of a series of line commands(L)connecting various points.
The sequence of commands indicates a closed shape as it returns to the starting point."
},
"Step2": {
"Module": "Segment Decomposition",
"Description": "The path can be broken down into the following segments: 1)(55.57,
80.69)to(57.38,
65.80),
2)(57.38,
65.80)to(48.90,
57.46),
3)(48.90,
57.46)to(45.58,
47.78),
4)(45.58,
47.78)to(53.25,
36.07),
5)(53.25,
36.07)to(66.29,
48.90),
6)(66.29,
48.90)to(78.69,
61.09),
and finally7)(78.69,
61.09)back to(55.57,
80.69)."
},
"Step3": {
"Module": "CoreShapeIdentification",
"Description": "The path consists of 7 distinct segments that connect back to the starting point,
indicating a closed shape with 7 vertices."
},
"Step4": {
"Module": "Data Evaluation for Shape Properties",
"Description": "The shape has 7 sides and 7 angles,
which corresponds to the definition of a heptagon."
},
"Step5": {
"Module": "Exploration of Shape Variations",
"Description": "While the path primarily suggests a heptagon,
variations in the angles and lengths of the segments could lead to different visual representations,
but they would still conform to the heptagon classification."
},
"Step6": {
"Module": "Visualization and Simplification",
"Description": "Visualizing the path confirms that it forms a closed shape with 7 sides,
reinforcing the identification of a heptagon."
},
"Step7": {
"Module": "Final Shape Classification",
"Description": "Based on the analysis and visual representation,
the shape represented by the SVG path is classified as a heptagon."
}
}✨✨ Follow me on LinkedIn for updates ✨✨
I’m currently the Chief Evangelist @ Kore AI. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.
