Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (AI summary)

Key Points

1. Introduction of Self-Play fine-tuning method (SPIN) for Large Language Models (LLMs).

2. Comparison of SPIN with traditional fine-tuning methods and evaluation of its effectiveness on various benchmark datasets.

3. Theoretical analysis proving the convergence of the optimization process of SPIN when the LLM's distribution aligns with the target data distribution.

4. Discussion of limitations and future directions for research on LLM fine-tuning.

Summary

The paper introduces a novel fine-tuning method called Self-Play fIne-tuNing (SPIN) to empower a weak Large Language Model (LLM) to improve itself without additional human-annotated data. The proposed SPIN method utilizes self-play, eliminating the need for human or advanced LLM guidance, and presents a two-player game approach to fine-tuning the LLM. The paper provides theoretical proof of convergence and evaluates the SPIN method on several benchmark datasets, including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. The experimental results show that SPIN can significantly improve the LLM’s performance across a variety of benchmarks and even outperform models trained through direct preference optimization. The paper compares the SPIN method with other approaches in the field, including supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF).

Additionally, the paper addresses the limitations of the SPIN method and proposes future directions for research, such as exploring the dynamically changing target data distribution and reducing the volume of required synthetic data. Overall, the paper presents a promising method for fine-tuning LLMs and demonstrates its effectiveness in improving model performance.

SPIN Method Evaluation and Comparison
The paper proposes a novel fine-tuning method called Self-Play fIne-tuNing (SPIN) to strengthen a weak Large Language Model (LLM) without the need for additional human annotated data. The SPIN method utilizes self-play, eliminating the need for human or advanced LLM guidance, and presents a two-player game approach to fine-tuning the LLM. The paper provides a theoretical proof of convergence for the SPIN method and presents experimental results on a fine-tuned LLM. Comparisons with other approaches in the field are also discussed. The SPIN method is demonstrated to be effective in strengthening weak LLMs, showcasing its potential for empowering LLMs to improve themselves without additional human annotated data.

Reference: https://arxiv.org/abs/2401.01335