[Roadmap] Research papers and tools integration

[dandansamax]

Agent Side

Includes models, methods, and tools that improve agent performance.

Visual-prompt

Tools that help locate or identify GUI elements.

• [Feature Request] OmniParser visual prompt #31

Policy

Prompting methods that work on any general purpose MLMs.

Static Training method/model

Public models with static parameters or training methods that rely on static datasets.

• Octopus-v2

Dynamic training method/model

Training methods that require dynamic environments. (Reinforcement learning)

Benchmark & Environment Side

Tools

Tools that support general purpose control.

• appium: UI automation of many app platforms, including mobile (iOS, Android, Tizen), browser (Chrome, Firefox, Safari), desktop (macOS, Windows), TV (Roku, tvOS, Android TV, Samsung), and more!
• selenium: Automates browsers.

Environments

Specific environments that packed in virtual machine, docker, or code.

Datasets

Task datasets that do not rely on a specific environment.

End-to-end Benchmarks

Benchmarks that include all requried things, including tools, environments, task datasets and so on.

• WebArena: https://arxiv.org/abs/2307.13854