From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future (AI summary)

Key Points

1. The paper presents a comprehensive survey of the current state, challenges, and future directions in applying large language models (LLMs) and LLM-based agents to various software engineering tasks.

2. The survey covers six key areas: requirement engineering, code generation and software development, autonomous learning and decision-making, software design and evaluation, software test generation, and software security and maintenance.

3. LLMs have achieved remarkable success in areas like code generation and vulnerability detection, but they also exhibit limitations such as lack of autonomy and self-improvement.

4. LLM-based agents, which combine LLMs as the core for decision-making and action-taking, have the potential to address some of the inherent limitations of LLMs.

5. The survey examines the differences and similarities between LLMs and LLM-based agents in terms of tasks, benchmarks, and evaluation metrics across the six software engineering domains.

6. The paper discusses the models and benchmarks used in LLM and LLM-based agent research, providing a comprehensive analysis of their applications and effectiveness.

7. The survey highlights the transformative potential of LLM-based agents in software engineering, particularly in areas like autonomous debugging, code refactoring, and adaptive test generation.

8. The paper aims to shed light on pushing the boundaries of LLM-based agents in software engineering for future research. 9. The paper presents, to the best of the authors' knowledge, the first comprehensive survey outlining the integration and transformation of LLMs to LLM-based agents in the domain of software engineering.

Summary
This paper investigates the applications of large language models (LLMs) and LLM-based agents in software engineering. The paper aims to differentiate between LLMs and LLM-based agents and provides a comprehensive analysis of their applications and effectiveness in six key areas: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance.

Limitations of LLMs and the Potential of LLM-based Agents
The survey discusses the limitations and shortcomings of LLMs, such as lack of autonomy and self-improvement, and proposes LLM-based agents as a technology with potential for Artificial General Intelligence (AGI) to address these limitations. It highlights the need for a unified standard and benchmarking to qualify an LLM solution as an LLM-based agent in its domain.

Applications in Requirement Engineering
In requirement engineering, LLMs have shown promise in automating tasks like requirement classification, generation, and assessment. However, LLM-based agents exhibit higher levels of autonomy and collaboration, enabling more efficient and accurate requirement engineering processes.

Use in Code Generation and Software Development
For code generation and software development, LLMs excel at tasks like code completion and translation, while LLM-based agents demonstrate advantages in handling complex workflows through task decomposition and multi-agent collaboration. This allows them to generate higher-quality code and improve development efficiency.

Autonomous Learning and Decision-Making Capabilities
In autonomous learning and decision-making, LLMs focus on enhancing specific functions, while LLM-based agents exhibit greater flexibility and adaptability, leveraging experiential learning and real-time feedback to optimize their strategies and responses. LLMs are effective in automating software design and evaluation tasks, but LLM-based agents can handle more complex and dynamic workflows by integrating with various tools and systems, leading to improved reliability and security.

Application in Software Test Generation, Security, and Maintenance
The paper also discusses the application of LLMs and LLM-based agents in software test generation, security, and maintenance. LLMs are adept at tasks like vulnerability detection and bug reproduction, while LLM-based agents showcase proactive defense capabilities and autonomous decision-making. Overall, the survey highlights the significant potential of LLM-based agents in addressing the limitations of traditional LLMs and advancing software engineering practices. The paper provides a comprehensive overview of the current state of research, challenges, and future directions in this rapidly evolving field.

Reference: https://arxiv.org/abs/2408.02479