What is the Role of Small Models in the LLM Era: A Survey (AI summary)

Key Points

1. Large language models (LLMs) like GPT and BERT have revolutionized natural language processing through the pre-train and fine-tune paradigm, demonstrating exceptional performance across various tasks.

2. As LLMs continue to grow in size and capability, reaching "artificial general intelligence" levels, their computational costs and resource demands have also escalated exponentially.

3. Small models (SMs), while generally lagging behind LLMs in overall performance, can sometimes achieve comparable results through techniques like knowledge distillation.

4. LLMs offer superior generality and can handle a broad spectrum of tasks with minimal training, while SMs tend to be more specialized and can outperform LLMs on specific domains or tasks.

5. LLMs require substantial computational resources for training and inference, making them less practical for real-time applications or resource-constrained environments, whereas SMs have lower resource demands.

6. Smaller, shallower models are generally more interpretable than their larger, deeper counterparts, which is important in fields like healthcare, finance, and law where model decisions must be easily understandable.

7. LLMs and SMs can collaborate to strike a balance between power and efficiency, with SMs enhancing LLMs through techniques like data curation and weak-to-strong generalization, while LLMs can also improve SMs through knowledge distillation and data augmentation.

8. SMs possess distinct advantages, such as simplicity, lower cost, and greater interpretability, and have a niche market, so it is crucial to carefully assess the trade-offs between LLMs and SMs based on the specific requirements of the task or application.

9. The role of small models in the era of LLMs is an important and underexplored topic that merits further research to optimize the use of computational resources and develop cost-effective, efficient, and interpretable AI systems.

Summary

The paper examines the relationship between large language models (LLMs) and small models (SMs) and how they can collaborate and compete in various settings. LLMs like GPT-4 and LLaMA-405B have made significant advances in artificial general intelligence, but their large size leads to high computational costs and energy consumption, making them impractical for many researchers and businesses. In contrast, SMs are frequently used in practical applications, though their significance is currently underestimated.

The paper presents two main perspectives on the role of SMs in the era of LLMs. First, LLMs and SMs can collaborate to strike a balance between performance and efficiency. SMs can enhance LLMs through techniques like data curation, where small proxy models are used to evaluate and select high-quality training data. Conversely, LLMs can enhance SMs through knowledge distillation, where the knowledge of a large model is transferred to a smaller one. This enables the development of cost-effective yet powerful models.

Second, LLMs and SMs compete in certain environments and scenarios. In computation-constrained settings like edge devices, SMs are preferred due to their lower resource demands and faster inference speed. For specialized tasks where sufficient training data is scarce, SMs can outperform LLMs by leveraging domain-specific knowledge. Additionally, in high-stakes decision-making contexts that require interpretability, such as healthcare and finance, smaller and simpler models are often preferred as their internal reasoning is more transparent and understandable.

The paper emphasizes the importance of carefully evaluating the trade-offs between LLMs and SMs when selecting the appropriate model for a given task or application. While LLMs offer superior performance, SMs have notable advantages in terms of accessibility, simplicity, lower cost, and interoperability. The authors hope this study provides valuable insights for practitioners, encouraging further research on resource optimization and the development of cost-effective systems that leverage the strengths of both LLMs and SMs.

Reference: https://arxiv.org/abs/2409.068...