Tuning Language Models by Proxy (AI summary)

Key Poionts

1. Large pretrained language models benefit from additional finetuning to achieve desired behaviors such as instruction-following, specific domains of interest, or particular tasks.

2. Tuning these models has become increasingly resource-intensive, or impossible when model weights are private, leading to a challenge of efficiently customizing larger language models for diverse users and applications.

3. The study introduces a lightweight decoding-time algorithm called proxy-tuning, which operates on top of black-box language models to achieve the result of directly tuning the model without accessing its internal weights, only its predictive distributions over the output vocabulary.

4. Proxy-tuning tunes a smaller language model and contrasts the prediction of the small tuned model and its untuned version to guide the larger base model by applying the difference between the predictions of the small tuned and untuned models to shift the original predictions of the base model in the direction of tuning, while retaining the benefits of larger-scale pretraining.

5. Proxy-tuning experiments with large pretrained language models (e.g., L LAMA 2-70B) using smaller, cheaper-to-tune experts, showcasing its ability to close a significant portion of the performance gap compared to directly tuned models across various knowledge, reasoning, and safety benchmarks.

6. When tested on TruthfulQA, proxy-tuned models were more truthful than directly tuned models, suggesting that decoding-time guidance better retains the model’s factual knowledge.

7. The study demonstrates the generality of proxy-tuning by applying it for domain adaptation on code and task-specific finetuning on question-answering and math problems.

8. Proxy-tuning influences the probability distribution at the token level, promoting reasoning and stylistic tokens, and providing users with a hyperparameter to control the amount of guidance exerted at runtime and tradeoff between different desired attributes of generations.

9. The work provides an efficient method for customizing large pretrained language models through decoding-time guidance, offering a promising alternative to directly updating model weights and addressing the challenge of adapting proprietary models to diverse use cases.

Summary

The research paper introduces a method called proxy-tuning, which is a lightweight decoding-time algorithm designed to efficiently customize large pretrained language models (LMs) without accessing their internal weights. The method tunes a smaller LM and then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the base model in the direction of tuning. This allows for the customization of large, potentially proprietary LMs through decoding-time guidance.

The paper evaluates the effectiveness of proxy-tuning across various tasks including instruction-following, domain adaptation on code, and task-specific finetuning on question-answering and math problems. The experiments demonstrate that proxy-tuning is able to close a significant portion of the performance gap between the base model and its directly tuned version when applied to various tasks. Proxy-tuned models are found to be more truthful than directly tuned models, and the method also consistently improves performance on coding tasks and specific language tasks.

The paper also provides insights into the mechanism of proxy-tuning, showcasing how it influences the probability distribution at the token level, particularly in promoting reasoning and stylistic tokens. Additionally, it explores the use of a hyperparameter to control the strength of tuning, providing more granular control over the steering of larger language models.

The study concludes that proxy-tuning is a promising method for efficiently customizing large LMs for diverse use cases, addressing the challenge of adapting proprietary models. It offers a potential alternative to traditional finetuning approaches, providing access to large LMs for users who lack the resources for extensive model training. The authors emphasize the need for organizations to share output probabilities from their models to enable the use of methods like proxy-tuning, and highlight the potential for further exploration of customizable, algorithmic, decoding-time "tuning."

Reference: https://arxiv.org/abs/2401.08565