EfficientRAG: Efficient Retriever for Multi-Hop Question Answering (AI summary)

Key Points

- EfficientRAG is introduced as an efficient retriever for multi-hop question answering without the need for large language model (LLM) calls at each iteration and filters out irrelevant information.

- The paper highlights limitations of current iterative retrieval approaches, including the need for multiple LLM calls and dedicated prompting and few-shot examples that might need updating across different scenarios.

- EfficientRAG consists of a Labeler and a Filter to iteratively generate new queries for retrieval and filter out irrelevant information, enhancing efficiency compared to other RAG methods.

- The study evaluates EfficientRAG on three multi-hop question-answering datasets and demonstrates the model's high recall and promising accuracy on subsequent question-answering tasks.

- Synthetic data is used to train EfficientRAG Labeler for token labeling and chunk filtering, and the model is fine-tuned based on DeBERTa-v3-large with 304M parameters.

- The model's performance is assessed using the Recall@K metric across three datasets and yields notably high recall scores on HotpotQA and 2WikiMQA datasets, with minimal number of retrieved chunks.

- EfficientRAG's speed is equivalent to direct retrieval methods and three times faster than LLM-based baselines while maintaining a similar number of iterations.

- The use of GPT-3.5 as a generator enhances the end-to-end performance of both the baselines and EfficientRAG, with the latter continuing to deliver exceptional results.

- EfficientRAG is shown to have remarkable transferability across diverse datasets and exhibits high efficiency compared to other iterative methods, achieving significant improvements in time efficiency.

Summary

The research paper introduces EfficientRAG, which addresses the limitations of current retrieval-augmented generation (RAG) methods in handling complex, multi-hop questions. It proposes a solution that iteratively generates new queries without the need for multiple calls of large language models at each iteration and filters out irrelevant information. The paper demonstrates the effectiveness of EfficientRAG through experimental results, showing its superior performance compared to existing RAG methods on open-domain multi-hop question-answering datasets.

The paper explains that pre-trained large-language models (LLMs) have limitations in dealing with complex, multi-hop questions, especially in domain-specific settings, and face issues of hallucinations. Retrieval-augmented generation (RAG) techniques have been proposed to ground generated responses by retrieving knowledge from external resources. Previous RAG methods often use one-round retrieval, which can fail in complex, multi-hop questions. To address this, recent works proposed iterative retrieval approaches, but these have limitations in terms of latency and updating of few-shot examples.

To address these limitations, the paper proposes EfficientRAG, consisting of a Labeler and a Filter to iteratively generate new queries for retrieval and filter out irrelevant information. The Labeler & Tagger produces outputs from separate heads within the same model to annotate useful information and tag chunks as helpful or irrelevant. The Filter then constructs new queries for the next round of retrieval. The paper details the empirical study conducted to assess the performance of EfficientRAG and presents the results across multiple datasets and comparing them with various baselines. The evaluations demonstrate the high recall and promising question-answering accuracy of EfficientRAG on three benchmark datasets.

Furthermore, the paper evaluates the efficiency of EfficientRAG and demonstrates its speed equivalent to direct retrieval methods and three times faster than LLM-based baselines while maintaining a similar number of iterations. It discusses the potential of EfficientRAG to adapt to different scenarios without further downstream training and presents out-of-domain experiments showcasing its transferability. The paper provides comprehensive details on the methodologies used during the evaluation, including the training, data synthesis, and implementation details.

In conclusion, the paper introduces the EfficientRAG retriever as a novel approach for multi-hop question retrieval and demonstrates its high recall, promising question-answering accuracy, and efficiency. It highlights the potential of the EfficientRAG framework to outperform traditional retrieval-augmented generation methods, especially in the context of complex, multi-hop question-answering scenarios, and suggests its adaptability to other models.

Reference: https://arxiv.org/abs/2408.04259