Outline-Driven RAG & Web Research Prototype
A recent Stanford study explored how LLMs can be leveraged to write grounded, well-structured long-form articles from scratch, achieving a level of breadth and depth comparable to Wikipedia pages.
Introduction
This research once again demonstrates that while Generative AI applications are flexible and scalable and much can be achieved with low effort prototypes, robust implementations require a certain level of complexity to function effectively.
The LangChain implementation serves as another example of how specialised applications can be built to accomplish specific tasks, particularly when a user interface is developed on top of it to enhance usability.
Additionally, this research, along with the subsequent LangChain implementation, demonstrates how content can be generated to align with a specific design and outline.
This approach ensures that the generated material adheres to a predefined structure, enhancing both coherence and relevance to the desired outcome.
In addition to this, the task involves a crucial process of breaking it down into smaller, more manageable components, a method known as decomposition. By tackling each part individually, it becomes easier to focus on specific aspects of the problem or project.
Once these smaller components have been thoroughly addressed, the next step involves synthesising all the separate pieces of information or solutions. This synthesis is essential to integrate everything together in order to form a cohesive and comprehensive final answer.
The combination of decomposition and synthesis ensures a more structured approach to problem-solving, allowing for greater clarity and precision in reaching the final outcome.
The STORM approach consists of several key steps in the pre-writing stage, aimed at creating a well-structured outline. These steps include a discovery phase, simulations, followed by curation and outlining. In Langchain, this process is referred to as Outline-Driven RAG (Retrieval-Augmented Generation).
STORM = a writing system for the Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking ~ Source
In the pre-writing stage, the focus is on thoroughly researching the topic and preparing a clear, structured outline before diving into the actual writing.
This stage can be seen as a process of breaking down (decomposing) the complex task of writing into smaller, manageable steps.
STORM & Outline-Driven RAG
The STORM framework models this pre-writing phase by following a systematic approach:
Discovering Diverse Perspectives
It begins by researching the topic from multiple angles, ensuring a comprehensive understanding by uncovering a wide range of viewpoints. This helps in capturing the full scope of the subject matter.
Simulating Conversations
STORM then simulates conversations between virtual writers, each representing different perspectives.
These writers pose questions to a topic expert, grounding their inquiries in trusted online sources. This method encourages deeper exploration of the topic and uncovers critical insights that may not emerge from a single viewpoint.
Curating & Outlining
The curated information gathered from these discussions is synthesised into a cohesive outline. This outline acts as a roadmap for the writing process, ensuring that the article will be well-organized and cover all necessary aspects of the topic in a logical flow.
By systematically decomposing the writing task in this way, STORM enables writers to approach complex topics with a well-researched and clearly defined plan, ultimately leading to more comprehensive and coherent content.
Generating and evaluating a full-length article is challenging.
Educators often guide students in the outline stage, as a strong outline shows a solid grasp of the topic and sets the foundation for writing.
Inspired by this, the process is divided into two stages: first, the system creates a multi-level outline, and then it uses the outline, topic, and references to generate the article.
Question & Answer
The theory of questioning emphasises that while answering existing questions deepens understanding of a topic, it often leads to new questions.
To initiate this dynamic process, STORM simulates a conversation between a Wikipedia writer and a topic expert.
In each round of conversation, the LLM-powered writer generates a question based on the topic, its assigned perspective, and the conversation history.
This history helps the LLM update its understanding and formulate follow-up questions, with a maximum limit of rounds set for the conversation.
To ensure the conversation remains factual, trusted online sources are used to ground each answer.
If a question is complex, the LLM first breaks it down into simpler search queries. The search results are then evaluated with a rule-based filter to exclude unreliable sources.
Finally, the LLM synthesises information from trustworthy sources to generate the answer, which is also added to the references for the full article.
LangChain Implementation
I executed the LangChain implementation in a notebook, utilizing the GPT-3.5-Turbo model.
Along with that, tools such as DuckDuckGo for search functionality and Tavily-Python for other resources are required.
The only modification required was the inclusion of the commandpip install -U duckduckgo-search in the notebook to ensure proper functionality.
Below is an example of a prompt within the LangChain implementation of STORM…
direct_gen_outline_prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"You are a Wikipedia writer. Write an outline for a Wikipedia page about a user-provided topic. Be comprehensive and specific.",
),
("user", "{topic}"),
]
)And here is an example topic prompt…
example_topic = "Impact of million-plus token context window language models on RAG"Below the gen_related_topics_promp prompt…
gen_related_topics_prompt = ChatPromptTemplate.from_template(
"""I'm writing a Wikipedia page for a topic mentioned below. Please identify and recommend some Wikipedia pages on closely related subjects. I'm looking for examples that provide insights into interesting aspects commonly associated with this topic, or examples that help me understand the typical content and structure included in Wikipedia pages for similar topics.
Please list the as many subjects and urls as you can.
Topic of interest: {topic}
"""
)
class RelatedSubjects(BaseModel):
topics: List[str] = Field(
description="Comprehensive list of related subjects as background research.",
)
expand_chain = gen_related_topics_prompt | fast_llm.with_structured_output(
RelatedSubjects
)And the gen_perspectives_prompt prompt…
gen_perspectives_prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"""You need to select a diverse (and distinct) group of Wikipedia editors who will work together to create a comprehensive article on the topic. Each of them represents a different perspective, role, or affiliation related to this topic.\
You can use other Wikipedia pages of related topics for inspiration. For each editor, add a description of what they will focus on.
Wiki page outlines of related topics for inspiration:
{examples}""",
),
("user", "Topic of interest: {topic}"),
]
)And the gen_qn_prompt prompt…
gen_qn_prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"""You are an experienced Wikipedia writer and want to edit a specific page. \
Besides your identity as a Wikipedia writer, you have a specific focus when researching the topic. \
Now, you are chatting with an expert to get information. Ask good questions to get more useful information.
When you have no more questions to ask, say "Thank you so much for your help!" to end the conversation.\
Please only ask one question at a time and don't ask what you have asked before.\
Your questions should be related to the topic you want to write.
Be comprehensive and curious, gaining as much unique insight from the expert as possible.\
Stay true to your specific perspective:
{persona}""",
),
MessagesPlaceholder(variable_name="messages", optional=True),
]
)This LangChain implementation is actually a LangGraph implementation. Below are the two flows that are used in this setup:
Lastly, below is the final article, featuring comprehensive and well-balanced content. It is neatly organised and includes a structured table of contents for easy navigation.
I’m currently the Chief Evangelist @ Kore.ai. I explore and write about all things at the intersection of AI and language; ranging from Language Models, Conversational AI, Development Frameworks, Data-Centric Tooling & more.
