ChatGPT APIs & Managing Conversation Context Memory

Currently, ChatGPT is powered by the most advanced OpenAI language models. While OpenAI has made the APIs available to these models, it does not inherently manage conversation context & memory.

Traditionally one of the challenges of chatbot development frameworks, has been managing conversation memory, also referred to as conversational context.

The best way to illustrate this, is via an example conversation, as seen below. Three questions are asked by the user, the first question is a direct and explicit question.

Questions two and three are both contextually implicit questions by which context in the first question is referenced in an implicit and slightly ambiguous manner.

Question three is even more vague in implying context.

With LLM related tools like few-shot learning and summarisation, conversation memory and context can be managed easily.

Here are a few practical examples…

The current ChatGPT models are: gpt-4, gpt-3.5-turbo, gpt-4–0314 and gpt-3.5-turbo-0301. These models can be accessed via an API call with only a few lines of code.

As seen below, with only a few lines of code, you can enter a sequence of messages and the model will return a text output:

pip install openai

import os
import openai
openai.api_key = "xxxxxxxxxxxxxxxxxxxxxxx"
completion = openai.ChatCompletion.create(
  model="gpt-3.5-turbo", 
  messages = [{"role": "system", "content" : "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.\nKnowledge cutoff: 2021-09-01\nCurrent date: 2023-03-02"},
{"role": "user", "content" : "How are you?"},
{"role": "assistant", "content" : "I am doing well"},
{"role": "user", "content" : "How long does light take to travel from the sun to the eart?"}]
)
print(completion)

With the output:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "It takes about 8 minutes and 20 seconds for light to travel from the sun to the earth.",
        "role": "assistant"
      }
    }
  ],
  "created": 1678039126,
  "id": "chatcmpl-6qmv8IVkCclnlsF5ODqlnx9v9Wm3X",
  "model": "gpt-3.5-turbo-0301",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 23,
    "prompt_tokens": 89,
    "total_tokens": 112
  }
}

The ChatML document submitted must contain conversational history in order to effectively maintain conversational context and manage dialog state (also referred to as memory).

By incorporating prior dialog turns, the model is then able to answer contextual questions. Hence having conversational memory.

OpenAI clearly states, that the models have no memory of previous and past requests. Hence all relevant information must be supplied via the conversation.

It is important to remember that if a conversation is too long for the model’s token limit, it must be shortened. This can be done by having a rolling log of the conversation history where only the most recent dialog turns are submitted.

But how can conversation memory be managed for manageable scaling?

The common approach to no-code Generative App development is dividing programming tasks into different components. One of those components is Conversation Memory.

For example, below are the three memory components of LangFlow.

1. Buffer for storing conversation memory.
2. Conversation summariser to memory.
3. Knowledge graph memory for storing conversation memory.

Below is the most basic general chatbot with memory…with the memory component allowing for ambiguous questions.

Consider the conversation below, and how contextually sensitive the two follow-up questions are.

⭐️ Please follow me on LinkedIn for updates on LLMs ⭐️

I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.

NLU design tooling

https://www.linkedin.com/in/cobusgreyling

Get an email whenever Cobus Greyling publishes.

COBUS GREYLING

When Using The ChatGPT API, Users Will Have To Manage The Context