Key Points
1. Function "get user current location()" retrieves the user's current city information.
2. Function "get all projects in the Todoist account" returns a list of all projects in the user's Todoist account.
3. Function "update project(project id, is favorite)" allows updating a project, including setting it as a favorite or not.
4. Function "get all tasks for a given project" retrieves all tasks within a specified project, with parameters for content, completion status, priority, and due date.
5. Function "get task description(task id)" fetches the description of a specific task in the Todoist account, using unique task identifiers and task names.
6. Function "get task duration(task id)" retrieves the duration of a specific task, with parameters for task name and duration in the format of 'amount(unit)'.
7. Function "complete task(task id)" deletes a specific task from the Todoist account using the task's unique identifier.
8. Function "get supported actions for current tool" returns the supported actions for the current tool or environment.
9. A set of examples and goals are provided, along with the expected actions and inputs for completing tasks and answering questions within the Todoist account.
Summary
The paper presents AGENTBOARD, a benchmark designed for evaluating generalist Large Language Models (LLMs) as agents. The benchmark includes 9 unique tasks covering various agent tasks such as embodied AI, web agents, game agents, and tool agents. It offers 1013 exemplary environments with multi-round interactions and partially-observable characteristics. AGENTBOARD features an open-source evaluation toolkit for comprehensive and interactive analysis of LLM agents, shedding light on their capabilities and limitations.
The evaluation framework introduces a fine-grained progress rate metric that captures incremental advancements and provides an analytical web panel for interactive visualization. The paper emphasizes the importance of systematically and analytically evaluating LLM agents and demonstrates the effectiveness of AGENTBOARD in recognizing advancements and guiding the development of stronger LLM agent models.
The paper also assesses a variety of proprietary and open-weight LLM agents using AGENTBOARD. The evaluation reveals that proprietary models, particularly GPT-4, outperform open-weight models across various tasks. The analysis also shows that open-weight LLMs demonstrate weaknesses in grounding, world modeling, and self-reflection abilities.
Overall, AGENTBOARD presents a significant advancement in the evaluation of LLM agents, offering a comprehensive and standardized framework for assessing their capabilities and guiding further progress in the field.
Reference: https://arxiv.org/abs/2401.13178v1