Key Points

1. The article introduces GNN-RAG, a novel method for combining the language understanding abilities of Large Language Models (LLMs) with the reasoning abilities of Graph Neural Networks (GNNs) in a retrieval-augmented generation (RAG) style.

2. GNN-RAG first uses a GNN to reason over a dense KG subgraph and retrieve answer candidates, and then extracts the shortest paths in the KG that connect question entities and GNN-based answers to represent useful KG reasoning paths. These reasoning paths are then verbalized and provided as input for LLM reasoning with RAG.

3. The GNN in GNN-RAG acts as a dense subgraph reasoner to extract useful graph information, while the LLM leverages its natural language processing ability for the ultimate KGQA task.

4. GNN-RAG achieves state-of-the-art performance on two widely used KGQA benchmarks (WebQSP and CWQ), outperforming or matching GPT-4 performance with a 7B tuned LLM.

5. GNN-RAG excels on multi-hop and multi-entity questions, outperforming competing approaches by 8.9–15.5% points at answer F1.

6. The authors develop a retrieval augmentation (RA) technique that further boosts KGQA performance with GNN-RAG by combining the strengths of GNN-based and LLM-based retrievers.

7. GNN-RAG improves the reasoning abilities of LLMs without incurring additional LLM calls, unlike existing RAG systems for KGQA.

8. The authors analyze the retrieval and reasoning capabilities of GNNs and LLMs, and provide theoretical insights on why GNNs are effective at handling complex graph information for KGQA.

9. Experimental results demonstrate the effectiveness, faithfulness, and efficiency of the proposed GNN-RAG framework.

Summary

The paper introduces GNN-RAG, a novel method for combining the language understanding abilities of large language models (LLMs) with the reasoning capabilities of graph neural networks (GNNs) for question answering over knowledge graphs (KGQA). The core idea of GNN-RAG is to leverage GNNs to retrieve relevant information from the knowledge graph (KG) and then use a language model in a retrieval-augmented generation (RAG) framework to reason over the retrieved information and generate the final answer.

First, the GNN component of GNN-RAG reasons over a dense KG subgraph to retrieve answer candidates for a given question. The GNN leverages its ability to handle complex graph structures and multi-hop reasoning to fetch useful information from the KG. Second, GNN-RAG extracts the shortest paths in the KG that connect the question entities to the GNN-retrieved answer candidates. These reasoning paths are then verbalized and provided as input to the LLM component, which uses its natural language understanding capabilities to reason over the retrieved information and generate the final answer.

Retrieval Augmentation Technique
The paper also introduces a retrieval augmentation (RA) technique that combines the retrievals from the GNN-based approach with an LLM-based retrieval method. This allows GNN-RAG to leverage the strengths of both approaches - the GNN's ability to handle complex graph structures for multi-hop questions and the LLM's natural language understanding for simple, single-hop questions.

Experimental results show that GNN-RAG outperforms or matches state-of-the-art methods on two KGQA benchmarks, WebQSP and CWQ. Specifically, GNN-RAG achieves a 6.5-15.5% improvement in F1 score over competing methods on multi-hop and multi-entity questions, demonstrating the effectiveness of its GNN-based retrieval in handling complex reasoning over knowledge graphs. Furthermore, the retrieval augmentation technique boosts performance by an additional 0.8-2.6% F1 points, showing the complementary benefits of combining GNN and LLM-based retrievals.

Key Contributions
The key contributions of the paper are: (1) the GNN-RAG framework that effectively combines LLM and GNN capabilities for KGQA, (2) the retrieval augmentation technique that further enhances KGQA performance, and (3) state-of-the-art results on standard KGQA benchmarks, especially for complex, multi-hop questions. Overall, the paper demonstrates how integrating the strengths of LLMs and GNNs can lead to significant improvements in knowledge-intensive language understanding tasks.

Reference: https://arxiv.org/abs/2405.201...