Self-Critique LLM Chain Using LangChain & SmartLLMChain
SmartLLMChain is a LangChain implementation of the self-critique chain principle. It is useful for particularly complex question answering; following a cycle of ideation, critique and resolve.
15 min readSep 15, 2023
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, data productivity suites & more.
Considering the question:
If it takes one man three days to dig a hole of one cubic metres, how long will it take 30 men to dig 30 such holes?
It is obvious to us as humans that the answer to the question is three days.
However, when posing the question to gpt-3.5-turbo the incorrect answer is given; as seen below.
This problem can be solved by accessing the same LLM, but following a procedure where the gpt-3.5-turbo model is used to ideate [1] on the problem, critique [2] the findings and present a resolution [3].
Considering the code below where the chain is defined, different models can be defined for each of the steps in the chain. the ideation_llm, critique_llm and resolver_llm can be defined separately.
chain = SmartLLMChain(
ideation_llm=ChatOpenAI(temperature=0.9, model_name="gpt-3.5-turbo"),
critique_llm=ChatOpenAI(temperature=0.9, model_name="gpt-3.5-turbo"),
resolver_llm=ChatOpenAI(temperature=0.9, model_name="gpt-3.5-turbo"),
prompt=prompt,
n_ideas=3,
verbose=True,
)What is convenient is that different models can be assigned to each of the three steps; hence a particular model can be assigned to ideation, critique an resolver.
Consider the correct and thorough response generated by the SmartLLMChain below, by referencing the same model in gpt-3.5-turbo . The enhanced response is astounding in its thoroughness and correctness taking various variables into account. This response is in stark contrast with the response from the OpenAI playground where a direct prompting approach was followed.
This again illustrates the versatility of LLMs and how important prompt engineering is, in creating an astute prompt pipeline.
Here is the output from the chain:
Step 3: The improved answer taking into account the modifier factor is as
follows:
Let's assume that the rate of digging scales with the number of people,
but not linearly. Let's introduce a modifier factor of K to account for
this. This means that the rate of digging for N men can be represented as
K * (1/3) cubic meters per day.
Now, if 30 men work together, their combined rate of digging can be
calculated as 30 * (K * 1/3) = 10K cubic meters per day.
Since we want to dig 30 holes, we need to determine how many days it would
take to dig those 30 holes at a rate of 10K cubic meters per day. To find
this, we divide the total volume (30 cubic meters) by the rate of digging
(10K cubic meters per day):
30 / (10K) = 3 / K
Therefore, it will take 30 men 3/K days to dig 30 holes of one cubic meter
each, considering the modifier factor K.
Please note that without additional information or context, we cannot
determine the specific value of the modifier factor K. It would depend on
factors such as the efficiency, coordination, and skill level of the
individuals involved.
Please output the full answer from Step 3.Breaking down the process followed by the SmartLLMChain:
Considering the image below, [1] ideation step gets a predefined number of output proposals (ideas) from the LLM.
A LLM then [2] critiques all of the ideas looking for possible flaws in the proposals and picking the most appropriate suggestion.
Lastly, in [3] resolve, the LLM tries to improve the best idea from the [2] critique step. The output here constitutes the final answer.
It needs to be noted that SmartLLMChains use multiple LLM passes, which can make it a costly chain to run, with higher latency.
The LLM referenced needs to have a chat endpoint (in the case of OpenAI) and must be capable of reflection, which some models are not capable of doing.
Below is the the complete output from the Colab notebook, notice the idea section in blue, followed by the critique section in yellow. And finally the resolution in green.
Below is the simplest form possible of the application, you can copy this and past it into a notebook.
You will need to enter your own OpenAI API key.
pip install langchain
pip install openai
pip install langchain_experimental
import os
os.environ["OPENAI_API_KEY"] = "xxxxxxxxxxxxxxxx"
from langchain.prompts import PromptTemplate
from langchain.chat_models import ChatOpenAI
from langchain_experimental.smart_llm import SmartLLMChain
hard_question = "If it takes one man three days to dig a hole of one cubic metres, how long will it take 30 men to dig 30 such holes?"
prompt = PromptTemplate.from_template(hard_question)
llm = ChatOpenAI(temperature=0, model_name="gpt-4")
chain = SmartLLMChain(llm=llm, prompt=prompt, n_ideas=3, verbose=True)
chain.run({})And the output from the code:
> Entering new SmartLLMChain chain...
Prompt after formatting:
If it takes one man three days to dig a hole of one cubic metres, how long will it take 30 men to dig 30 such holes?
Idea 1:
Step 1: If one man can dig a hole of one cubic meter in three days, it means that the rate of digging for one man is 1/3 cubic meters per day.
Step 2: Now, we need to find the total rate of digging for 30 men. Since each man has a rate of 1/3 cubic meters per day, the total rate for 30 men would be 30 times the rate of one man.
Total rate for 30 men = 30 * (1/3) = 10 cubic meters per day.
Step 3: Finally, we need to find how long it will take 30 men to dig 30 holes, each of one cubic meter. Since the total rate of digging for 30 men is 10 cubic meters per day, we can divide the total number of holes (30) by the rate to find the time.
Time = Total number of holes / Total rate of digging
Time = 30 / 10 = 3 days.
Therefore, it will take 30 men 3 days to dig 30 holes, each of one cubic meter.
Idea 2:
Step 1: If one man can dig a hole of one cubic meter in three days, it means that the rate of digging for one man is 1/3 cubic meters per day.
Step 2: Now, we need to find the total rate of digging for 30 men. Since each man has a rate of 1/3 cubic meters per day, the total rate for 30 men would be 30 times the rate of one man.
Total rate for 30 men = 30 * (1/3) = 10 cubic meters per day.
Step 3: Finally, we need to find how long it will take 30 men to dig 30 holes, each of one cubic meter. Since the total rate of digging for 30 men is 10 cubic meters per day, we can divide the total number of holes (30) by the rate to find the time.
Time = Total number of holes / Total rate of digging
Time = 30 / 10 = 3 days.
Therefore, it will take 30 men 3 days to dig 30 holes, each of one cubic meter.
Idea 3:
Step 1: If one man can dig a hole of one cubic meter in three days, it means that the rate of digging for one man is 1/3 cubic meters per day.
Step 2: Now, we need to find the total rate of digging for 30 men. Since each man has a rate of 1/3 cubic meters per day, the total rate for 30 men would be 30 times the rate of one man.
Total rate for 30 men = 30 * (1/3) = 10 cubic meters per day.
Step 3: Finally, we need to find how long it will take 30 men to dig 30 holes, each of one cubic meter. Since the total rate of digging for 30 men is 10 cubic meters per day, we can divide the total number of holes (30) by the rate to find the time.
Time = Total number of holes / Total rate of digging
Time = 30 / 10 = 3 days.
Therefore, it will take 30 men 3 days to dig 30 holes, each of one cubic meter.
Critique:
Idea 1:
Flaws and faulty logic:
1. The answer assumes that the rate of digging for one man remains constant regardless of the number of men working. However, in reality, the rate of digging may vary depending on factors such as coordination, efficiency, and physical limitations.
2. The answer assumes that the time taken to dig one hole is the same as the time taken to dig 30 holes. This may not be accurate as the efficiency of digging may decrease when multiple holes are being dug simultaneously.
3. The answer does not consider any potential limitations or constraints that may arise when multiple men are working together, such as space constraints or coordination issues.
Idea 2:
Flaws and faulty logic:
1. The answer assumes that the rate of digging for one man remains constant regardless of the number of men working. However, in reality, the rate of digging may vary depending on factors such as coordination, efficiency, and physical limitations.
2. The answer assumes that the time taken to dig one hole is the same as the time taken to dig 30 holes. This may not be accurate as the efficiency of digging may decrease when multiple holes are being dug simultaneously.
3. The answer does not consider any potential limitations or constraints that may arise when multiple men are working together, such as space constraints or coordination issues.
Idea 3:
Flaws and faulty logic:
1. The answer assumes that the rate of digging for one man remains constant regardless of the number of men working. However, in reality, the rate of digging may vary depending on factors such as coordination, efficiency, and physical limitations.
2. The answer assumes that the time taken to dig one hole is the same as the time taken to dig 30 holes. This may not be accurate as the efficiency of digging may decrease when multiple holes are being dug simultaneously.
3. The answer does not consider any potential limitations or constraints that may arise when multiple men are working together, such as space constraints or coordination issues.
In conclusion, all three answer options have flaws and faulty logic as they do not account for potential variations in the rate of digging and the efficiency of multiple men working together. They also overlook any limitations or constraints that may affect the overall time taken to dig the holes.
Resolution:
Step 1: Finding the best answer option chosen by the researcher.
Step 2: Improving the chosen answer option.
Step 3: Printing the full answer.
Answer:
It seems that the researcher did not provide any indication of which answer option they considered to be the best. Therefore, we cannot determine the chosen answer option.
However, based on the flaws and faulty logic identified in all three answer options, it is necessary to provide an improved and accurate answer.
Improved Answer:
To determine the time it will take for 30 men to dig 30 holes, we need to consider the rate of digging and the efficiency of multiple men working together.
Given that one man can dig a hole of one cubic meter in three days, we can calculate the rate of digging for one man as 1/3 cubic meters per day.
To find the total rate of digging for 30 men, we multiply the rate of one man by the number of men:
Total rate for 30 men = 30 * (1/3) = 10 cubic meters per day.
Now, we need to find the time it will take for 30 men to dig 30 holes, each of one cubic meter. Since the total rate of digging for 30 men is 10 cubic meters per day, we can divide the total number of holes (30) by the rate to find the time:
Time = Total number of holes / Total rate of digging
Time = 30 / 10 = 3 days.
Therefore, it will take 30 men 3 days to dig 30 holes, each of one cubic meter.
This answer takes into account the rate of digging, the efficiency of multiple men working together, and provides a clear and accurate calculation of the time required.
> Finished chain.
Step 1: Finding the best answer option chosen by the researcher.\n\nStep 2: Improving the chosen answer option.\n\nStep 3: Printing the full answer.\n\nAnswer: \n\nIt seems that the researcher did not provide any indication of which answer option they considered to be the best. Therefore, we cannot determine the chosen answer option.\n\nHowever, based on the flaws and faulty logic identified in all three answer options, it is necessary to provide an improved and accurate answer.\n\nImproved Answer:\n\nTo determine the time it will take for 30 men to dig 30 holes, we need to consider the rate of digging and the efficiency of multiple men working together. \n\nGiven that one man can dig a hole of one cubic meter in three days, we can calculate the rate of digging for one man as 1/3 cubic meters per day.\n\nTo find the total rate of digging for 30 men, we multiply the rate of one man by the number of men:\n\nTotal rate for 30 men = 30 * (1/3) = 10 cubic meters per day.\n\nNow, we need to find the time it will take for 30 men to dig 30 holes, each of one cubic meter. Since the total rate of digging for 30 men is 10 cubic meters per day, we can divide the total number of holes (30) by the rate to find the time:\n\nTime = Total number of holes / Total rate of digging\nTime = 30 / 10 = 3 days.\n\nTherefore, it will take 30 men 3 days to dig 30 holes, each of one cubic meter.\n\nThis answer takes into account the rate of digging, the efficiency of multiple men working together, and provides a clear and accurate calculation of the time required.As mentioned earlier in the article, different models can be defined for each step of the chain:
chain = SmartLLMChain(
ideation_llm=ChatOpenAI(temperature=0.9, model_name="gpt-4"),
critique_llm=ChatOpenAI(temperature=0.9, model_name="gpt-3.5-turbo-16k-0613"),
resolver_llm=ChatOpenAI(temperature=0.9, model_name="gpt-3.5-turbo"),
prompt=prompt,
n_ideas=3,
verbose=True,
)And the chain can be run again:
chain.run({})With the output:
at the same rate as the one man. However, it is possible that the efficiency or skill level of the additional men may differ, resulting in a different overall rate of digging.
Idea 2:
- The answer assumes that the rate of digging remains the same when 30 men work together. However, as mentioned before, the rate of digging may not scale linearly with the number of people, and there could be diminishing returns or coordination issues.
- The answer divides the total volume (30 cubic meters) by the rate of digging (10 cubic meters per day) to determine the number of days. This assumes that the 30 men will work continuously without any breaks or variations in their rate. In reality, there could be limitations or variations in their working hours or productivity.
Idea 3:
- The answer assumes that the rate of digging remains constant regardless of the number of people involved, similar to the first idea. Again, this may not reflect reality.
- The answer states that 30 men can dig 30 times the amount that one man can dig in one day. However, this assumes that the efficiency or skill level of the additional men is the same as the one man. As mentioned before, there could be variations in productivity among the individuals, leading to a different overall rate of digging.
In summary, all three answer options make assumptions about the rate of digging and the efficiency of the people involved without considering potential variations or limitations in real-life scenarios. Therefore, they may not accurately determine the time required for 30 men to dig 30 holes.
Resolution:
Step 1: The researcher thought that Idea 3 was the best answer option.
Step 2: To improve Idea 3, we need to consider the potential variations in the rate of digging when multiple people are involved. We can account for this by assuming that the rate of digging scales with the number of people, but not necessarily linearly. We can introduce a modifier factor to reflect this.
Step 3: The improved answer taking into account the modifier factor is as follows:
Let's assume that the rate of digging scales with the number of people, but not linearly. Let's introduce a modifier factor of K to account for this. This means that the rate of digging for N men can be represented as K * (1/3) cubic meters per day.
Now, if 30 men work together, their combined rate of digging can be calculated as 30 * (K * 1/3) = 10K cubic meters per day.
Since we want to dig 30 holes, we need to determine how many days it would take to dig those 30 holes at a rate of 10K cubic meters per day. To find this, we divide the total volume (30 cubic meters) by the rate of digging (10K cubic meters per day):
30 / (10K) = 3 / K
Therefore, it will take 30 men 3/K days to dig 30 holes of one cubic meter each, considering the modifier factor K.
Please note that without additional information or context, we cannot determine the specific value of the modifier factor K. It would depend on factors such as the efficiency, coordination, and skill level of the individuals involved.
Please output the full answer from Step 3.
> Finished chain.
Step 1: The researcher thought that Idea 3 was the best answer option.\n\nStep 2: To improve Idea 3, we need to consider the potential variations in the rate of digging when multiple people are involved. We can account for this by assuming that the rate of digging scales with the number of people, but not necessarily linearly. We can introduce a modifier factor to reflect this.\n\nStep 3: The improved answer taking into account the modifier factor is as follows:\n\nLet's assume that the rate of digging scales with the number of people, but not linearly. Let's introduce a modifier factor of K to account for this. This means that the rate of digging for N men can be represented as K * (1/3) cubic meters per day.\n\nNow, if 30 men work together, their combined rate of digging can be calculated as 30 * (K * 1/3) = 10K cubic meters per day.\n\nSince we want to dig 30 holes, we need to determine how many days it would take to dig those 30 holes at a rate of 10K cubic meters per day. To find this, we divide the total volume (30 cubic meters) by the rate of digging (10K cubic meters per day):\n\n30 / (10K) = 3 / K\n\nTherefore, it will take 30 men 3/K days to dig 30 holes of one cubic meter each, considering the modifier factor K.\n\nPlease note that without additional information or context, we cannot determine the specific value of the modifier factor K. It would depend on factors such as the efficiency, coordination, and skill level of the individuals involved.\n\nPlease output the full answer from Step 3.I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, data productivity suites & more.
⭐️ Follow me on LinkedIn for updates on Large Language Models ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.
