OpenAI GPT-3.5 Turbo Model With 16k Context Window
OpenAI has launched a new model called gpt-3.5-turbo-16k & I was able to submit a 14 page document to the model for summarisation.
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
Yesterday OpenAI made a new model available with the name gpt-3.5-turbo-16k. The astounding thing about this model is the size of its context window.
The image below shows the document submitted. The document consists of 14 pages and > 12,000 words, for summarisation, with success!
Something I found interesting is that the gpt-3.5-turbo-16k model is a chat only model and is not supported by the completions endpoint. Below is the error message received:
InvalidRequestError: This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?
Hence I had to opt for the chat endpoint and the ChatML notation for input.
Below a working Python application accessing the 16k model:
import os
import openai
openai.api_key = "xxxxxxxxxxxxxxxxxxxx"
completion = openai.ChatCompletion.create(
model="gpt-3.5-turbo-16k",
messages = [{"role": "system", "content" : "You are a chatbot which can search text and provide a summarised answer."},
{"role": "user", "content" : "How are you?"},
{"role": "assistant", "content" : "I am doing well"},
{"role": "user", "content" : "What is the distance between New York and Montreal?"}
]
)
print(completion)When I submitted a very large document which exceeded the context window, the error message received back is very informative in terms of what is possible from a context window perspective:
InvalidRequestError: This model’s maximum context length is 16385 tokens. However, your messages resulted in 18108 tokens. Please reduce the length of the messages.
gpt-3.5-turbo-16koffers 4 times the context length ofgpt-3.5-turboat twice the price: $0.003 per 1K input tokens and $0.004 per 1K output tokens. 16k context means the model can now support ~20 pages of text in a single request. ~ OpenAI
The summary consists of 256 words, a one sentence brief summary and 12 key points.
Below is the response from the model, after processing the 14 page document
The usage detail at the bottom in terms token usage is helpful.
{
"id": "chatcmpl-7RF4lzwU4q3PLSRPCarcXuezErEHI",
"object": "chat.completion",
"created": 1686727163,
"model": "gpt-3.5-turbo-16k-0613",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The text provides information on various aspects of South Africa, including its geography, population, economy, government, tourism, transport, and energy sector. Here are the key points:\n\n- South Africa is the southernmost country in Africa and has diverse geographical features, including coastlines along the South Atlantic and Indian Oceans, mountains, plateaus, and savannahs.\n- It has a population of about 60 million people, with diverse ethnic groups and 11 official languages.\n- The country has a mixed economy, with mining and tourism being significant sectors. It has the second-largest economy in Africa.\n- Crime, poverty, and inequality are challenges faced by South Africa, although progress has been made in some areas.\n- The government is a parliamentary republic, with a president as the head of state and government. The country has a three-tier system of governance, with provinces and local municipalities.\n- The South African National Defence Force is responsible for national security.\n- South Africa has a well-developed road and rail network, with the road network being the largest in Africa.\n- The country is known for its tourism industry, with attractions such as national parks, beaches, and cultural heritage sites.\n- South Africa has a significant energy sector, with coal being the primary source of electricity generation. It is also the only African country with a nuclear power plant.\n- Eskom, the state-owned utility, is the largest producer of electricity in Africa but faces challenges such as debt and power shortages.\n- South Africa has made significant scientific and technological contributions, such as the first human-to-human heart transplant and development of a yellow fever vaccine."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 15986,
"completion_tokens": 325,
"total_tokens": 16311
}
}One of the ailments of LLMs which has been lamented for quite a while has been context window size of models. And that data has to be pre-processed to some degree to accommodate the limited context window size.
The sheer context which can be managed by this new OpenAI model is astounding, together with the speed and accuracy.
⭐️ Please follow me on LinkedIn for updates on Conversational AI ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.
