Key Points

1. The paper introduces L UMOS, a unified and modular training framework for open-source language agents. L UMOS aims to address the issues associated with closed-source agents, such as affordability, transparency, and reproducibility. It features a learnable, unified, and modular architecture with a planning module, grounding module, and execution module, allowing for modular upgrades and wider applicability to diverse interactive tasks.

2. L UMOS exhibits several key advantages over existing open-source agents:
- Outperforms multiple larger open-source agents on held-out datasets for each task type, even surpassing GPT agents on QA and web tasks.
- Outperforms open-source agents produced by chain-of-thoughts and unmodularized integrated training.
- Demonstrates effective generalization to unseen tasks, outperforming 33B-scale agents and domain-specific agents.

3. L UMOS is designed to execute actions and interact with external tools or environments to solve complex interactive tasks, such as QA, web tasks, math, and multimodal reasoning. It primarily relies on open-data and large language models (LLMs).

4. The paper emphasizes the importance of a unified format for complex interactive tasks and high-quality training annotations to enhance generalization on unseen tasks with new environments and actions.

5. L UMOS has two formulations for developing agents: L UMOS-OnePass (L UMOS -O) and L UMOS-Iterative (L UMOS -I). L UMOS -O is an efficient formulation that enables one-pass inference, while L UMOS -I is an adaptive formulation that helps agents flexibly plan based on the execution feedback.

6. The proposed agent framework provides improved or comparable performance with GPT-based or larger open-source agents across various complex interactive tasks, including QA, web, math, and multimodal tasks. The evaluation demonstrates the competitive performance of L UMOS, and its potential benefits for a wide spectrum of language agent applications.

7. The paper introduces a novel annotation conversion method to obtain high-quality annotations for training L UMOS, leveraging existing benchmarks' ground-truth rationales and converting them into a universally applicable format consistent with the modular design of L UMOS.

8. The significant performance improvements of L UMOS compared to various other open-source agents and GPT-based agents are shown across different interactive tasks, demonstrating the effectiveness of the proposed training framework.

9. The paper highlights the competitive advantages of L UMOS, including its ability to surpass larger open-source agents, its effectiveness in generalizing to unseen tasks, and the importance of high-quality training annotations in training and evaluating language agents.

Summary

The paper introduces L UMOS, an open-source framework for training language agents, addressing the limitations of closed-source agents such as affordability, transparency, and reproducibility. L UMOS features a unified and modular architecture comprising a planning module, grounding module, and execution module, enabling the agent to effectively interact with external tools or environments. The paper also discusses the approach for gathering high-quality training annotations and demonstrates that L UMOS outperforms existing open-source agents on various complex interactive tasks. The framework exhibits enhanced performance on held-out datasets, surpassing GPT-based agents in tasks like QA and web tasks. The paper also highlights the advantages of L UMOS over closed-source agents in terms of affordability, transparency, and reproducibility.

Development and formulations within L UMOS framework
Additionally, the paper presents two formulations for developing agents within the L UMOS framework - L UMOS -I and L UMOS -O, demonstrating the ability of these formulations to promote collaboration among agent modules to solve complex tasks. It also discusses the use of LLMs to transform reasoning steps in existing benchmarks into a unified format applicable within the L UMOS framework.

Future plans and enhancements for L UMOS framework
Furthermore, the paper mentions plans for expanding the scope of annotations to encompass a wider variety of task types and enhancing the framework with advanced mechanisms that enable the agents to recognize and rectify planning errors. It also outlines the intention to transition to fully open-source QA frameworks that leverage models such as LLAMA-2-70B.

Overall, the paper proposes L UMOS as a significant advancement in the development of open-source language agents, offering a valuable resource for the development of new models and research in this field.

Reference: https://arxiv.org/abs/2311.056...