Making Large Language Models A Better Foundation For Dense Retrieval (AI summary)

Key Points

- The paper introduces a novel approach called LLaRA, which aims to adapt large language models (LLMs) for dense retrieval application, addressing the challenge of adapting LLMs, pre-trained via text generation tasks, as the backbone encoder for dense retrieval.

- LLaRA consists of two pretext tasks, EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), which adapt LLMs to effectively initialize the backbone encoder for dense retrieval.

- The paper discusses the importance of dense retrieval in real-world applications such as web search and open-domain question answering, and how the quality of dense retrieval is influenced by the capacity of its backbone encoder.

- The study evaluates the effectiveness of LLaRA and demonstrates its impact on retrieval accuracy after fine-tuning and its generality across different retrieval scenarios, showing notable improvement in retrieval quality.

- LLaRA is applied to the adaptation of LLaMA-2-7B with the Wikipedia corpus, resulting in state-of-the-art performances on various dense retrieval benchmarks such as MSMARCO and BEIR.

- The proposed LLaRA approach is highlighted as a simple but effective solution to adapt LLMs for dense retrieval application, providing a significant improvement in retrieval capability without the need for additional labeled data.

Summary

Introduction to LLaRA and Its Technical Contributions
The paper proposes an innovative approach called LLaRA (LLM adapted for dense Retrieval) to adapt large language models (LLMs) for dense retrieval. LLaRA consists of two pretext tasks, EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), designed to adapt LLMs for handling diverse semantic matching scenarios. The study demonstrates the application of LLaRA to adapt LLaMA-2-7B with the Wikipedia corpus, resulting in significant improvements in retrieval quality. The technical contributions of the work include the simplicity and effectiveness of LLaRA and the plan to make the model and source code publicly available for future research in this area.

Challenges in Leveraging LLMs for Dense Retrieval
The paper also discusses the challenges in leveraging LLMs for dense retrieval due to their pre-training by text generation tasks, which is different from representing texts as embeddings. To address this, the novel approach LLaRA is introduced as a post-hoc adaptation of LLMs to improve their usability for dense retrieval. This involves performing pretext tasks to enhance LLMs' capability to generate text embeddings for the semantic representation of global context, ultimately improving retrieval accuracy and generality across different scenarios.

Verification of LLaRA's Effectiveness through Experimental Studies
The study verifies the effectiveness of LLaRA through experimental studies, demonstrating its impact on retrieval accuracy after fine-tuning and its generality across different scenarios. The results show notable improvements in retrieval performance across various evaluation scenarios, including passage and document retrieval on MS MARCO and zero-shot retrieval on BEIR benchmark. LLaRA achieves state-of-the-art performances, indicating the substantial improvement in the LLM's text embedding capability and its effectiveness for dense retrieval.

Reference: https://arxiv.org/abs/2312.15503v1