LLM Apps
Large Language Model Applications
In a previous post I wrote about the ability to create a prompt pipeline where a prompt template is used and relevant data is retrieved on the fly and injected into the template.
Prompt Pipelines add flexibility to LLM Apps and extend the discipline of prompt engineering by making prompts highly programable.
Prompt chains (chaining) are useful to create a predefined sequence of events. Where a predefined sequences do not exist or use-cases could not be envisaged, the flexibility of agents can fulfil user intent.
There is an emergence of Visual Programming tools facilitating the chaining of large language model prompts into an application; which is mostly a conversational UI.
Below is the current available technology stack for developing LLM Apps.
I covered the concepts of Prompt Pipelines, Prompt Chaining and LangChain based Agents.
This article considers how Haystack has implemented LLM based agents to create LLM Apps. It seems like the terms LLM Apps and Generative Apps or Gen Apps are used interchangeably.
Haystack recently launched their framework to creating Agents which is very similar to the LangChain approach to Agents.
Agents can create highly flexible and dynamic chains on the fly.
Chains (prompt chaining) can also have agents imbedded into a pre-defined chain.
Haystack has the advantage of being steeped in experience regarding semantic search, document search, Prompt Pipelines and Agents. The advent of LLMs seemingly fits perfectly into their roadmap.
Considering Haystack’s offering, is prompt chaining functionality soon to follow?
Considering the image below, the graphic details the loop (sequence of events) of a LangChain Agent and the example output is also shown.
⭐️ Please follow me on LinkedIn for updates on LLMs ⭐️
The sequence of events listed below as followed by the Haystack Agent is not too dissimilar. The Haystack agent also cycles through a set number of tools at its disposal. There is also a Thought step and then an Output or Observation step.
The distinct advantage Haystack has, when it comes to Agents are the ability to leverage existing pipelines and nodes.
Below is a list of pipelines which could be referenced:
- WebQAPipeline
- ExtractiveQAPipeline
- DocumentSearchPipeline
- GenerativeQAPipeline
- SearchSummarizationPipeline
- FAQPipeline
- TranslationWrapperPipeline
- QuestionGenerationPipeline
Nodes or Components are building blocks of pipelines, and these can also be referenced individually by an agent. All of this contributes to a rich ecosystem where components can be repurposed.
To prevent the agent from using the tools infinitely, the number of iterations is limited to eight by default.
— Haystack.
It does not seem like Haystack has different agent types like LangChain. The documentation advises that the text-davinci-003 completion model should be made use of.
Below is the prompt template used for this example:
PromptTemplate(
name="zero-shot-react",
prompt_text="You are a helpful and knowledgeable agent. To achieve your goal of answering complex questions "
"correctly, you have access to the following tools:\n\n"
"{tool_names_with_descriptions}\n\n"
"To answer questions, you'll need to go through multiple steps involving step-by-step thinking and "
"selecting appropriate tools and their inputs; tools will respond with observations. When you are ready "
"for a final answer, respond with the `Final Answer:`\n\n"
"Use the following format:\n\n"
"Question: the question to be answered\n"
"Thought: Reason if you have the final answer. If yes, answer the question. If not, find out the missing information needed to answer it.\n"
"Tool: pick one of {tool_names} \n"
"Tool Input: the input for the tool\n"
"Observation: the tool will respond with the result\n"
"...\n"
"Final Answer: the final answer to the question, make it short (1-5 words)\n\n"
"Thought, Tool, Tool Input, and Observation steps can be repeated multiple times, but sometimes we can find an answer in the first pass\n"
"---\n\n"
"Question: {query}\n"
"Thought: Let's think step-by-step, I first need to ",
),And lastly, the complete working code example of the Haystack agent:
%%bash
pip install --upgrade pip
pip install farm-haystack[colab]
pip install datasets
pip install openai
pip install google-search-results
import logging
logging.basicConfig(format="%(levelname)s - %(name)s - %(message)s", level=logging.WARNING)
logging.getLogger("haystack").setLevel(logging.DEBUG)
from haystack import Pipeline, BaseComponent, Answer
from haystack.nodes import PromptNode, PromptTemplate
# For the pipeline to work, you'd need to import the Reader, Retriever, and DocumentStore
# This example skips the pipeline configuration steps
from haystack.pipelines import WebQAPipeline
from haystack.nodes.retriever.web import WebRetriever
from haystack.agents import Agent, Tool
# First, configure the PromptNode, WebRetriever, and WebQAPipeline that will act as Agent's tools
openai_key= 'xxxxxxxxxx'
prompt_node = PromptNode(
"text-davinci-003",
api_key=openai_key,
max_length=256,
default_prompt_template="question-answering-with-document-scores")
search_key = 'xxxxxxxxxxx'
web_retriever = WebRetriever(api_key=search_key)
pipeline = WebQAPipeline(retriever=web_retriever, prompt_node=prompt_node)
# Let's create the prompt for the Agent's PromptNode
# This prompt is simplified, you could improve it by adding examples for the model
# See the full example at https://github.com/deepset-ai/haystack/blob/main/examples/agent_multihop_qa.py
few_shot_prompt = """
You are a helpful and knowledgeable agent. To achieve your goal of answering complex questions correctly, you have access to the following tools:
Search: useful for when you need to Google questions. You should ask targeted questions, for example, Who is Anthony Dirrell's brother?
To answer questions, you'll need to go through multiple steps involving step-by-step thinking and selecting appropriate tools and their inputs; tools will respond with observations. When you are ready for a final answer, respond with the `Final Answer:`
Examples:
##
Question: Anthony Dirrell is the brother of which super middleweight title holder?
Thought: Let's think step by step. To answer this question, we first need to know who Anthony Dirrell is.
Tool: Search
Tool Input: Who is Anthony Dirrell?
Observation: Boxer
Thought: We've learned Anthony Dirrell is a Boxer. Now, we need to find out who his brother is.
Tool: Search
Tool Input: Who is Anthony Dirrell brother?
Observation: Andre Dirrell
Thought: We've learned Andre Dirrell is Anthony Dirrell's brother. Now, we need to find out what title Andre Dirrell holds.
Tool: Search
Tool Input: What is the Andre Dirrell title?
Observation: super middleweight
Thought: We've learned Andre Dirrell title is super middleweight. Now, we can answer the question.
Final Answer: Andre Dirrell
##
Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?
Thought: Let's think step by step. To answer this question, we first need to know who won the 1971 San Francisco mayoral election.
Tool: Search
Tool Input: Who won the 1971 San Francisco mayoral election?
Observation: Joseph Alioto
Thought: We've learned Joseph Alioto won the 1971 San Francisco mayoral election. Now, we need to find out what party he belongs to.
Tool: Search
Tool Input: What party does Joseph Alioto belong to?
Observation: Democratic Party
Thought: We've learned Democratic Party is the party of Joseph Alioto. Now, we need to find out when the Democratic Party was founded.
Tool: Search
Tool Input: When was the Democratic Party founded?
Observation: 1828
Thought: We've learned the Democratic Party was founded in 1828. Now, we can answer the question.
Final Answer: 1828
##
Question: Right Back At It Again contains lyrics co-written by the singer born in what city?
Thought: Let's think step by step. To answer this question, we first need to know what song the question is referring to.
Tool: Search
Tool Input: What is the song Right Back At It Again?
Observation: "Right Back at It Again" is the song by A Day to Remember
Thought: We've learned Right Back At It Again is a song by A Day to Remember. Now, we need to find out who co-wrote the song.
Tool: Search
Tool Input: Who co-wrote the song Right Back At It Again?
Observation: Jeremy McKinnon
Thought: We've learned Jeremy McKinnon co-wrote the song Right Back At It Again. Now, we need to find out what city he was born in.
Tool: Search
Tool Input: Where was Jeremy McKinnon born?
Observation: Gainsville, Florida
Thought: We've learned Gainsville, Florida is the city Jeremy McKinnon was born in. Now, we can answer the question.
Final Answer: Gainsville, Florida
##
Question: {query}
Thought:
"""
few_shot_agent_template = PromptTemplate("few-shot-react", prompt_text=few_shot_prompt)
prompt_node = PromptNode(
"text-davinci-003", api_key=os.environ.get("OPENAI_API_KEY"), max_length=512, stop_words=["Observation:"]
)
# Let's configure Search as the Agent's tool
# Each tool needs to have a description that helps the Agent decide when to use it
web_qa_tool = Tool(
name="Search",
pipeline_or_node=pipeline,
description="useful for when you need to Google questions.",
output_variable="results",
)
# Time to initialize the Agent specifying the PromptNode to use and the Tools
agent = Agent(
prompt_node=prompt_node,
prompt_template=few_shot_agent_template,
tools=[web_qa_tool],
final_answer_pattern=r"Final Answer\s*:\s*(.*)",
)
hotpot_questions = [
"What year was the father of the Princes in the Tower born?",
"Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.",
"Where was the actress who played the niece in the Priest film born?",
"Which author is English: John Braine or Studs Terkel?",
]
for question in hotpot_questions:
result = agent.run(query=question)
print(f"\n{result}")Below is the output of only the last question:
“Which author is English: John Braine or Studs Terkel?”
⭐️ Please follow me on LinkedIn for updates on LLMs ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
