Synthesizing Text-to-SQL Data from Weak and Strong LLMs (AI summary)

Key Points

1. The paper proposes a synthetic data approach that combines strong data generated by larger, more powerful models (strong models) with weak data produced by smaller, less well-aligned models (weak models) for improving the text-to-SQL task.

2. The strong data aims to enhance data diversity and facilitate cross-domain generalization, while the weak data, combined with an executor, helps the model learn from errors and feedback.

3. The paper evaluates the effectiveness of the proposed approach by fine-tuning the CodeLLaMA open-source model to create a specialized text-to-SQL model named SENSE, which achieves state-of-the-art results on the SPIDER and BIRD benchmarks.

4. Experiments show that SENSE narrows the performance gap between open-source and closed-source models in text-to-SQL tasks, including challenging settings like the BIRD benchmark.

5. SENSE also demonstrates advantages in robustness, outperforming previous methods on various robustness datasets such as SYN, REALISTIC, and DK.

6. The paper conducts an in-depth analysis on the impact of synthetic data, revealing that strong data enhances data diversity and cross-domain generalization, while weak data improves the model's ability to learn from errors.

7. The study also examines the transferability of the proposed approach, showing that it works equally well when using homogeneous models with the same pre-training data.

8. Beyond text-to-SQL, the paper evaluates SENSE on general and broad tasks, demonstrating its robust versatility and generalization capabilities across various benchmarks.

9. The paper aims to contribute to the advancement of the text-to-SQL community by making the SENSE data and models publicly available.

Summary

Synthetic Data Approach in Text-to-SQL Modeling
The paper introduces a synthetic data approach that combines strong data generated by larger, more potent language models (LLMs) with weak data produced by smaller, less well-aligned models. This approach contributes to the improvement of domain generalization in text-to-SQL models and investigates the potential of weak data supervision through preference learning. The researchers utilized the synthetic data approach for instruction tuning on open-source LLMs, resulting in a specialized text-to-SQL model called SENSE. The SENSE model achieved state-of-the-art results on the SPIDER and BIRD benchmarks, thereby helping to close the performance gap between open-source models and the methods derived from closed-source models.

Two-Phase Approach for Improving Text-to-SQL Capabilities
The paper evaluates both open-source and closed-source LLMs on text-to-SQL benchmarks using a standardized prompt and finds that the text-to-SQL capabilities of open-source models were significantly inferior. The researchers propose a two-phase approach to enhance the text-to-SQL capabilities of open-source base models. In the first phase, they enhance the base model's text-to-SQL capabilities through Supervised Fine-tuning (SFT). This process focuses on the diversity and quality of data, using strong data synthesized by larger models. In the second phase, the model is introduced to incorrect SQL queries intentionally generated by weaker LLMs. Through Preference Learning, the model is encouraged to discern between correct and incorrect SQL, effectively learning from its mistakes. The paper provides an extensive experimental analysis, evaluating the performance of SENSE on various text-to-SQL benchmarks, robustness datasets, and challenging environments. The authors compare SENSE to baseline methods and fine-tuned models and find that the SENSE model demonstrated superior performance, even surpassing closed-source models. The researchers also address the limitations of their approach and emphasize the potential of synthetic data in text-to-SQL parsing. They propose that the release of SENSE data and models can contribute to the advancement of the text-to-SQL community.

Showcasing the Potential of Synthetic Data in Text-to-SQL Parsing
The paper showcases the potential of synthetic data in text-to-SQL parsing and highlights the effectiveness of the proposed SENSE model in narrowing the performance gap between open-source and closed-source models. The researchers also acknowledge the limitations of their study, particularly related to computational resources and the scope of their evaluation. Finally, the paper acknowledges the support of various research and development programs and emphasizes the potential implications of their work for the broader NLP community.

Reference: https://arxiv.org/abs/2408.03256