Transformers Can Do Arithmetic with the Right Embeddings (AI summary)

Key Points

1. Transformers struggle to perform complex multi-step and algorithmic reasoning tasks in a zero-shot setting without access to tools. This difficulty stems from their inability to clearly represent the exact position of a digit within a long sequence of digits.

2. The authors propose a new positional embedding called "Abacus Embeddings" that directly addresses this issue by encoding the position of each digit relative to the start of the number.

3. When combined with Abacus Embeddings, architectural modifications like input injection and recurrent layers further improve performance.

4. With these enhancements, the authors can train on only 20-digit numbers and achieve 99% accuracy on 100-digit addition problems, demonstrating strong length generalization.

5. The gains in numeracy from these methods also transfer to other multi-step reasoning tasks like sorting and multiplication.

6. The authors push length generalization beyond prior work, achieving 6x extrapolation compared to the previous state-of-the-art of 2.5x.

7. Combining Abacus Embeddings with other relative position embeddings like FIRE further improves performance and robustness.

8. Looped transformer architectures with recurrent layers outperform standard transformers, especially on out-of-distribution examples.

9. The authors demonstrate the potential of their methods to enable transformers to perform a variety of complex algorithmic reasoning tasks without relying on external tools.

Summary

The paper addresses the poor performance of transformers on arithmetic tasks, particularly multi-digit addition. The authors find that this difficulty stems largely from transformers' inability to clearly represent the exact position of each digit within a long sequence of digits. To address this, the authors propose a novel positional embedding called "Abacus Embeddings" which encodes the position of each digit relative to the start of the number.

The Abacus Embeddings provide a significant boost in performance on addition tasks compared to prior positional embedding methods like FIRE and no positional embeddings (NoPE). When combined with standard transformer architectures, the Abacus Embeddings enable near-perfect in-distribution accuracy on addition problems with up to 100 digit operands, representing a 6x extrapolation beyond the maximum operand size seen during training.
The authors also explore architectural modifications like input injection and looped transformer models, finding that these further improve performance when combined with the Abacus Embeddings. The looped transformer models, in particular, achieve state-of-the-art results, reaching 99% accuracy on 100 digit addition problems after training on just 20 digit numbers for a single GPU day.

Finally, the authors show that the improvements afforded by the Abacus Embeddings extend beyond just addition, also boosting performance on more complex algorithmic reasoning tasks like multiplication and sorting of variable length number arrays. This highlights the broader applicability of the Abacus Embedding approach for enhancing the numerical and algorithmic capabilities of transformer models.

Overall, the work demonstrates that explicitly modeling the positional significance of digits is a crucial component for enabling transformers to perform robust multi-step numerical reasoning, paving the way for further advancements in this domain.

Reference: https://arxiv.org/abs/2405.173...