Does Submitting Long Context Solve All LLM Contextual Reference Challenges?
Large Language Models (LLMs) are known to hallucinate. Hallucination is when a LLM generates a highly succinct and highly plausible answer; but factually incorrect. Hallucination can be negated by injecting prompts with contextually relevant data which the LLM can reference.
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.
Growing LLM context size has the allure that large swaths of contextual reference data can merely be submitted to the LLM to act as reference data.
Reference data which will create a contextual reference for the LLM and in turn negate hallucination…
Below is a view of the Vercel playground, for each of the LLMs available the context window is shown.
A recent study examined the performance of LLMs on two tasks:
- One involving the identification of relevant information within input contexts.
- A second involving multi-document question answering and key-value retrieval.
The study found that LLMs perform better when the relevant information is located at the beginning or end of the input context.
However, when relevant context is in the middle of longer contexts, the retrieval performance is degraded considerably. This is also the case for models specifically designed for long contexts.
Extended-context models are not necessarily better at using input context. Source
Other considerations to keep in mind in terms of submitting large volumes of data is inference time (latency) and also token costs in terms of input and output.
Making use of a RAG (Retrieval Augmented Generation) a chunk of data is injected into the prompt at inference. The paragraph or snippet of text is typically retrieved from a Vector Store/Database via semantic search. The text is presented to the LLM at inference time. Read more here.
⭐️ Follow me on LinkedIn for updates on Large Language Models ⭐️
I’m currently the Chief Evangelist @ HumanFirst. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.
