Key Points

1. Language models (LMs) are currently limited by their tokenizers, which map raw text to a sequence of tokens. This restricts their flexibility and efficiency in dealing with languages other than English and domains like code.

2. The paper introduces the problem of Zero-Shot Tokenizer Transfer (ZeTT), which focuses on swapping the original LM tokenizer with an arbitrary one on the fly without degrading performance.

3. Prior heuristics and approaches for transferring LMs to new tokenizers have shown limitations, resulting in a performance gap with the original LM.

4. The paper proposes a new solution using a hypernetwork to train a diverse distribution of tokenizers to predict the embedding parameters for any given tokenizer. Through empirical demonstration, the hypernetwork generalizes to new tokenizers for both encoder and decoder LMs.

5. The proposed method brings performance close to the original models while reducing the length of the tokenized sequence. It is also shown that the remaining performance gap can be quickly closed by continued training on a small amount of extra tokens.

6. Results demonstrate that the hypernetwork can be transferred to fine-tuned variants of the base LM without extra training, providing a state-of-the-art solution to n-shot tokenizer transfer and establishing a competitive baseline to the zero-shot tokenizer transfer problem.

7. The paper provides detailed procedures for converting tokenizers to byte-level and UnigramLM, and discusses the effect of amortizing over the tokenization function and the computational overhead of the hypernetwork.

8. Evaluation of the proposed method shows promising results in zero-shot transfer of language-specific tokenizers, cross-lingual transfer, and transferring fine-tuned models to a new tokenizer using a hypernetwork trained for the base model.

9. Overall, the paper makes substantial strides towards detaching LMs from their tokenizer, increasing their flexibility, and reusability.

Summary

Limitations of Language Models and the Introduction of Zero-Shot Tokenizer Transfer (ZeTT)
The paper discusses the limitations of language models (LMs) due to their tokenizer and introduces the new problem of Zero-Shot Tokenizer Transfer (ZeTT). It proposes a solution through the training of a hypernetwork to address this limitation. The study demonstrates the generalization of the hypernetwork to new tokenizers and its effectiveness in cross-lingual and coding tasks. The paper begins by highlighting that LMs are usually bound to their tokenizer, which restricts their flexibility. The researchers define the Zero-Shot Tokenizer Transfer (ZeTT) problem to swap the original LM tokenizer with an arbitrary one without degrading performance. The challenge at the core of ZeTT is finding embeddings for the tokens in the vocabulary of the new tokenizer. The authors propose a solution by training a hypernetwork that takes a tokenizer as input and predicts the corresponding embeddings. The paper empirically demonstrates that the hypernetwork generalizes to new tokenizers both with encoder and decoder LMs. The method is shown to come close to the original models’ performance in cross-lingual and coding tasks while markedly reducing the length of the tokenized sequence.

Methods for Addressing Tokenizer Inefficiencies and Developing New Techniques for Tokenizer Transfer
The paper also discusses the issue that existing LMs are usually trained primarily on the English language, and tokenizers often encode languages besides English or other domains, such as code, less efficiently. This leads to disparities in the inference cost between English and non-English text. The paper addresses these issues by developing methods to equip an LM with a new tokenizer by retraining the embedding parameters and optionally continuing to train the entire model. The paper also delves into heuristic-free tokenizer transfer, forward- and backward-propagating through a subset of the model layers, and regularly resetting the embedding parameters during pretraining. Additionally, the researchers introduce the concept of Hypernetworks, which are networks that predict the parameters of another network. The paper presents a detailed training loop for Zero-Shot Tokenizer Transfer, providing a step-by-step algorithm and discussing the distributions over texts and tokenizers used in the process. The paper also discusses the effects of amortizing over the tokenization function and analyzes the computational overhead of the hypernetwork.

Results and Findings of the Study
The paper also presents the results and findings of the study. It demonstrates the effectiveness of the hypernetwork approach in transferring language models to new tokenizers, preserving accuracy, and reducing the length of tokenized sequences. The results show that the proposed method outperforms prior heuristics for embedding initialization and provides a stronger baseline for Zero-Shot Tokenizer Transfer. Furthermore, the study shows that the hypernetwork can be successfully applied to fine-tuned versions of the base model, demonstrating its potential applicability in practical scenarios.

Overall, the research makes substantial strides toward detaching LMs from their tokenizer, increasing their flexibility and reusability. The paper concludes with acknowledgments for the support received during the research.

Reference: https://arxiv.org/abs/2405.078...