Meta Releases New RAG Benchmark
Meta has launched the Comprehensive Retrieval Augmented Generation Benchmark (CRAG), a new benchmark for factual question-answering. CRAG includes 4,409 question-answer pairs and mock APIs for simulating retrieval, designed to evaluate Large Language Models (LLMs) in retrieval-augmented generation (RAG) tasks. The benchmark features tasks such as Retrieval Summarization, Knowledge Graph and Web Retrieval Augmentation, and End-to-end RAG. Key insights include CRAG's ability to assess only the "Generator" in the RAG pipeline, performance metrics across five domains and eight question categories, and significant performance improvements with RAG products. Meta aims to maintain and update CRAG, although the dataset has not yet been released.