Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models (AI summary)

Key Points

1. Large Language Models (LLMs) have shown remarkable capabilities in various NLP tasks, but their performance in complex logical reasoning tasks is limited.

2. The Chain-of-Thought (CoT) prompting method has been proposed to enhance LLMs' logical reasoning abilities by adding intermediate steps in the reasoning process. Subsequent research has extended this approach into more complex reasoning topologies, such as Tree-of-Thoughts (ToT) and Graph-of-Thoughts (GoT).

3. Neuro-symbolic methods have been proposed to integrate LLMs with symbolic reasoning to address unfaithful reasoning and information loss issues in the reasoning process.

4. The Logic-of-Thought (LoT) prompting method is proposed to address information loss by employing propositional logic to generate expanded logical information from input context, which enhances the LLMs' logical reasoning capability. LoT can be seamlessly integrated with existing prompting methods.

5. LoT consists of three phases: Logic Extraction, Logic Extension, and Logic Translation. In the Logic Extraction phase, LLMs are used to extract formal logic expressions from the input context. In the Logic Extension phase, logical expressions are expanded through Python-implemented logical rules. In the Logic Translation phase, the expanded logical expressions are translated into natural language descriptions.

6. LoT significantly boosts the performance of various prompting methods across five logical reasoning tasks, achieving substantial improvements in accuracy for tasks like ReClor, LogiQA, and ProofWriter datasets.

7. The study uses five logical reasoning datasets: ReClor, LogiQA, RuleTaker, ProofWriter, and FOLIO, along with various prompting methods and pre-trained models for extensive experiments.

8. The experiments demonstrate that integrating LoT with existing prompting methods consistently achieves the best performance, highlighting the superiority of LoT in enhancing logical reasoning abilities.

9. Comparative analysis with neuro-symbolic methods like SatLM and prompting methods like ToT further showcases LoT's effectiveness in boosting logical reasoning capabilities of LLMs, achieving significant enhancements in accuracy and outperforming standalone prompting methods in some cases.

10. The study provides a detailed and systematic evaluation of the proposed LoT prompting method and its integration with existing prompting techniques, highlighting its potential to enhance the logical reasoning capabilities of Large Language Models.

Summary

The paper proposes a new prompting method called Logic-of-Thought (LoT) to enhance the logical reasoning abilities of Large Language Models (LLMs). Existing prompting methods, such as Chain-of-Thought, can improve reasoning to some extent, but they suffer from an "unfaithful issue" where the derived conclusions may not align with the generated reasoning chain. Additionally, potential omissions in the extraction of logical expressions in previous neuro-symbolic methods can lead to information loss and incorrect results.

Key Phases of LoT Method
To address these issues, the LoT method employs propositional logic to generate expanded logical information from the input context. It then utilizes this expanded logical information as additional augmentation to the input prompts, enhancing the LLM's capability for logical reasoning. The LoT method consists of three key phases: 1. Logic Extraction: LLMs are used to extract propositions and logical relations from the input context, representing them using propositional symbols and logical expressions. 2. Logic Extension: The extracted logical expressions are further expanded using logical reasoning laws implemented in a Python program. 3. Logic Translation: The expanded logical expressions are translated back into natural language descriptions, which are then incorporated into the original input prompt.

Experiment Results
The authors conduct extensive experiments to evaluate the effectiveness of LoT in boosting the performance of various prompting methods, including Chain-of-Thought (CoT), Self-Consistency (SC), CoT with Self-Consistency (CoT-SC), and Tree-of-Thoughts (ToT), across five logical reasoning datasets. The results demonstrate that LoT can significantly enhance the performance of these existing prompting methods.

Performance Improvements
Specifically, LoT improves the performance of CoT on the ReClor dataset by +4.35%, boosts SC's performance on ReClor by +6.52%, and enhances CoT-SC's accuracy on LogiQA by +5%. Additionally, LoT elevates the performance of ToT on the ProofWriter dataset by +8%. The authors also provide detailed analyses of the strengths and limitations of LoT compared to other neuro-symbolic approaches, such as SatLM. The key contributions of this work are: 1) the proposal of the novel LoT prompting method to address the information loss issue in existing neuro-symbolic approaches, 2) the seamless integration of LoT with a variety of prompting techniques, and 3) the extensive experimental validation of LoT's effectiveness in boosting logical reasoning capabilities across multiple datasets.

Reference: https://arxiv.org/abs/2409.17539