Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought (AI summary)

Key Points

1. The paper introduces a novel framework, LM-Guided Chain-of-Thought (LM-Guided CoT), which leverages a lightweight language model (LM) to guide a larger language model in reasoning tasks. The lightweight LM generates a rationale for each input instance, and the large LM predicts a task output based on this rationale.

2. The approach is resource-efficient as it only requires training the lightweight LM and optimizes the model through knowledge distillation and reinforcement learning from rationale-oriented and task-oriented reward signals.

3. The implementation of LM-Guided CoT on multi-hop extractive question answering (QA) benchmarks, HotpotQA, and 2WikiMultiHopQA, yielded improved answer prediction accuracy compared to baselines.

4. Conventional Chain-of-Thought (CoT) prompting has limitations in terms of performance gains with large language models (LMs) and may generate low-quality rationales. Hence, the paper proposes LM-Guided CoT as a solution to alleviate these limitations.

5. The proposed framework consists of two LMs: a lightweight model for rationale generation and a large black-box model for answer prediction.

6. The paper also explores methods for rationale distillation, evaluation, and refinement, as well as the use of reinforcement learning for rationale refinement.

7. Experimental results show that the model with knowledge distillation outperforms the original CoT prompting, and reinforcement learning contributes to improvements in rationale qualities and task performance.

8. The results also indicate that top-quality rationales for the large LM do not consistently improve task performance, prompting the need to explore a more harmonious balance between LM-generated rationale utilities and overall task performance.

9. The paper discusses the annotation details, analysis of relationships between aspect types and task performance, and compares two methods for providing rewards for reinforcement learning, contributing to the understanding and effectiveness of the proposed framework.

Summary

The research paper introduces a novel framework, LM-Guided CoT, which aims to improve the reasoning abilities of large language models (LMs) by leveraging a lightweight language model to guide a black-box large LM in reasoning tasks. The lightweight LM first generates a rationale for each input instance, and then the large LM is prompted to predict a task output based on this generated rationale. The approach is resource-efficient, requiring training only the lightweight LM, and is optimized through knowledge distillation and reinforcement learning from rationale-oriented and task-oriented reward signals.

Experimental Results and Quality Evaluation
The experimental results using multi-hop extractive question answering benchmarks, HotpotQA and 2WikiMultiHopQA, demonstrate that the LM-Guided CoT approach outperforms all baselines regarding answer prediction accuracy. It is found that reinforcement learning helps the model to produce higher-quality rationales and improve question-answering performance. The research paper also highlights the importance of aspects like factuality, logicality, coherence, fluency, naturalness, and readability in evaluating the quality of the generated rationales and their impact on task performance.

The proposed LM-Guided CoT framework decomposes the conventional CoT prompting into two steps using two models: rationale generation and answer prediction. The results reveal that the approach outperforms all baselines and improves the overall quality of the generated rationales and the task performance while maintaining resource efficiency. However, the study acknowledges the need for further exploration and generalizability to other reasoning tasks.

Detailed Analyses and Conclusion
In addition to the experimental findings, the paper includes detailed analyses of relationships between aspect types and task performance, evaluation methods for rationale quality, and insights into training and evaluation setups for the proposed framework. The paper concludes with insights into the suitability of different methods for providing rewards for reinforcement learning and provides a comprehensive description of the hyperparameters utilized in the model training process.

Reference: https://arxiv.org/abs/2404.034...