Summary
This scientific article introduces Code Llama, a family of large language models designed specifically for code-based tasks. These models show state-of-the-art performance in various aspects such as code generation, infilling (predicting missing parts of code), and instruction following for programming tasks.
Code Llama comes in different flavors to cover a wide range of applications, including a foundational model (Code Llama), Python specialization (Code Llama - Python), and instruction-following model (Code Llama - Instruct).
The models are trained on large amounts of code data and achieve impressive results on code benchmarks. They are released under a permissive license, allowing for both research and commercial use.
Keypoints
1. Code Llama is a family of large language models for code generation and infilling, based on Llama 2.
2. Code Llama models provide state-of-the-art performance among open models, with improvements in infilling capabilities, support for large input contexts, and instruction following ability.
3. Code Llama comes in different flavors for different applications, including foundation models, Python specializations, and instruction-following models.
4. The models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens.
5. Code Llama achieves state-of-the-art performance on several code benchmarks, outperforming other publicly available models.
6. Code Llama is released under a permissive license, allowing for both research and commercial use.
7. Specialization and increasing the capabilities of the models are achieved through a cascade of training and fine-tuning steps.
8. Code Llama features infilling capabilities, supporting autoregressive and causal infilling prediction.
9. Code Llama is fine-tuned to handle long input contexts, extending the maximum context length from 4,096 tokens to 100,000 tokens.
Reference: https://arxiv.org/abs/2308.129...