Is RAG Really Dead? Testing Multi Fact Retrieval & Reasoning in GPT4-128k

From the video description

One of the most popular benchmarks for long context LLM retrieval is @GregKamradt's Needle in A Haystack: a fact (needle) is injected into a (haystack) of context (e.g., Paul Graham essays) and the LLM is asked a question related to this fact. But, it only evaluates a single needle. RAG typically requires retrieval and reasoning over multiple chunks. In order to use long context LLM in place of RAG systems, it is critical to understand how well long context LLMs can retrieve multiple facts and reason over them. We recently updated Greg's repo to work with multi-needles and use LangSmith for evaluation and logging.