Mar 6
When Using The ChatGPT API, Users Will Have To Manage The Context
Currently ChatGPT is powered by gpt-3.5-turbo-0301, the most advanced OpenAI language model. Whilst OpenAI has made the API available to this model, the API does not inherently perform conversation context management…
Obviously gpt-3.5-turbo-0301 does well at single-turn tasks without any conversational dialog turns.
However, most implementations will solicit a follow-up question from users and hence a ChatGPT implementation needs to be able to maintain conversational context and chain of reasoning…
The ChatGPT models hold no memory of past requests, all relevant information must be supplied via the conversation.
In order to maintain conversational context and manage dialog state, conversational history will have to be included in the ChatML document submitted, in order for the model to be able answer contextual questions. These contextual questions can be answered by the model by leveraging prior dialog turns.
OpenAI clearly states, that the models have no memory of previous and past requests. Hence all relevant information must be supplied via the conversation.
Keep in mind, If a conversation cannot fit within the model’s token limit, it will need to be shortened in some way. This can be achieved by having a type of rolling log for conversational history, where only the last n amount of dialog turns are re-submitted.
Below are a few practical examples. You can enter a sequence of messages and the model will return a text output, as seen below:
pip install openai
import os
import openai
openai.api_key = "xxxxxxxxxxxxxxxxxxxxxxx"
completion = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages = [{"role": "system", "content" : "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.\nKnowledge cutoff: 2021-09-01\nCurrent date: 2023-03-02"},
{"role": "user", "content" : "How are you?"},
{"role": "assistant", "content" : "I am doing well"},
{"role": "user", "content" : "How long does light take to travel from the sun to the eart?"}]
)
print(completion)
With the output:
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "It takes about 8 minutes and 20 seconds for light to travel from the sun to the earth.",
"role": "assistant"
}
}
],
"created": 1678039126,
"id": "chatcmpl-6qmv8IVkCclnlsF5ODqlnx9v9Wm3X",
"model": "gpt-3.5-turbo-0301",
"object": "chat.completion",
"usage": {
"completion_tokens": 23,
"prompt_tokens": 89,
"total_tokens": 112
}
}⭐️ Please follow me on LinkedIn for updates on Conversational AI ⭐️
Below is the ChatML document submitted to the gpt-3.5-turbo-0301 model:
[{"role": "system", "content" : "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.\nKnowledge cutoff: 2021-09-01\nCurrent date: 2023-03-02"},
{"role": "user", "content" : "How are you?"},
{"role": "assistant", "content" : "I am doing well"},
{"role": "user", "content" : "When was the last Formula One championship in South Africa?"}]With the correct response given to the last question:
[
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "The last Formula One championship race held in South Africa was on October 17, 1993.",
"role": "assistant"
}
}
]However, when a follow-up question is asked with no context provided in the ChatML document…as seen below:
[{"role": "system", "content" : "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.\nKnowledge cutoff: 2021-09-01\nCurrent date: 2023-03-02"},
{"role": "user", "content" : "How are you?"},
{"role": "assistant", "content" : "I am doing well"},
{"role": "user", "content" : "Who won the race in South Africa?"}]The context is lost and the ChatGPT API states that fact.
{
"finish_reason": null,
"index": 0,
"message": {
"content": "I'm sorry, but I don't have access to up-to-date information on current events. My training data only goes up until September 2021, and I am not capable of browsing the internet or accessing real-time data.",
"role": "assistant"
}
}The correct way to ask the follow up question, is by providing the contextual reference to the system:
[{"role": "system", "content" : "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.\nKnowledge cutoff: 2021-09-01\nCurrent date: 2023-03-02"},
{"role": "user", "content" : "How are you?"},
{"role": "assistant", "content" : "I am doing well"},
{"role": "user", "content" : "When was the last Formula One championship in South Africa?"},
{"role": "assistant", "content" : "The last Formula One championship race held in South Africa was on October 17, 1993."},
{"role": "user", "content" : "Who won the race in South Africa?"}]And the correct response is given:
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "The 1993 South African Grand Prix was won by Alain Prost of France, driving for the Williams-Renault team.",
"role": "assistant"
}
}In Closing
Additional to context management, OpenAI also states that gpt-3.5-turbo-0301 does not pay strong attention to the system message, and extra focus is required for instructions in a user message.
If model generated output is unsatisfactory, iteration and experimenting are required to yield improvements. For instance:
- By making the instructions more explicit
- Specifying the answer format
- Ask the model to think step by step or sequentially
Thirdly, and lastly, no fine-tuning is available for gpt-3.5-turbo models; only base GPT-3 models can be fine-tuned.
⭐️ Please follow me on LinkedIn for updates on Conversational AI ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
