From f64aa8112bcc128b455114266cc57ec331fa37a2 Mon Sep 17 00:00:00 2001 From: Ikechukwu Uchendu Date: Tue, 20 Aug 2024 13:29:35 -0400 Subject: [PATCH] tutorial for adding custom domains to a2perf --- docs/content/circuit_training/index.md | 1 + docs/content/tutorials/add_domain.ipynb | 143 +++++++++++++++++++ docs/content/tutorials/generalization.ipynb | 21 +++ docs/content/tutorials/inference.md | 4 +- docs/content/tutorials/metrics/index.md | 7 + docs/content/tutorials/metrics/metrics.ipynb | 37 +++++ docs/content/tutorials/training.ipynb | 4 +- docs/index.md | 4 +- 8 files changed, 216 insertions(+), 5 deletions(-) create mode 100644 docs/content/tutorials/add_domain.ipynb create mode 100644 docs/content/tutorials/generalization.ipynb create mode 100644 docs/content/tutorials/metrics/index.md create mode 100644 docs/content/tutorials/metrics/metrics.ipynb diff --git a/docs/content/circuit_training/index.md b/docs/content/circuit_training/index.md index 33aca69..8368c6c 100644 --- a/docs/content/circuit_training/index.md +++ b/docs/content/circuit_training/index.md @@ -47,3 +47,4 @@ CircuitTraining-Ariane-v0 CircuitTraining-ToyMacroStdcell-v0 +``` diff --git a/docs/content/tutorials/add_domain.ipynb b/docs/content/tutorials/add_domain.ipynb new file mode 100644 index 0000000..96f96b4 --- /dev/null +++ b/docs/content/tutorials/add_domain.ipynb @@ -0,0 +1,143 @@ +{ + "cells": [ + { + "metadata": {}, + "cell_type": "markdown", + "source": [ + "# Adding Custom Domains\n", + "\n", + "This tutorial will demonstrate how to add additional domains to A2Perf.\n", + "\n", + "## Path to A2Perf Domains\n", + "\n", + "Domains are stored in `A2Perf/a2perf/domains`. Currently, you'll find the main domains in this folder such as `circuit_training`, `quadruped_locomotion`, and `web_navigation`. There are also additional modules such as `tfa` that is there for interfacing with Tensorflow Agents.\n", + "\n", + "
\n", + "\n", + "---\n", + "\n", + "\n", + "## Creating a new domain\n", + "\n", + "To add a new domain:\n", + "\n", + "1. Create a new folder in the domains directory. For the purposes of this tutorial, we will create a new domain called \"my_domain\":\n", + "\n", + "\n", + "```bash\n", + "cd A2Perf/a2perf/domains\n", + "mkdir my_domain\n", + "``` \n", + "\n", + "2. Inside of `my_domain`, create the following files:\n", + "- `__init__.py`\n", + "- [any other files needed for your domain]\n", + "\n", + "3. Inside of `__init__.py`, you will need to register your domain with gymnasium. Here is an example of how we do this for the web navigation domain:\n", + "\n", + "```python\n", + "import gymnasium as gym\n", + "import pkg_resources\n", + "\n", + "base_url_path = pkg_resources.resource_filename('a2perf.domains.web_navigation',\n", + " 'gwob')\n", + "base_url = f'file://{base_url_path}/'\n", + "data_dir = pkg_resources.resource_filename(\n", + " 'a2perf.domains.web_navigation.environment_generation', 'data')\n", + "\n", + "gym.envs.register(\n", + " id='WebNavigation-v0',\n", + " entry_point=(\n", + " 'a2perf.domains.web_navigation.gwob.CoDE.environment:WebNavigationEnv'\n", + " ),\n", + " apply_api_compatibility=False,\n", + " disable_env_checker=False,\n", + " kwargs=dict(\n", + " use_legacy_step=False,\n", + " use_legacy_reset=False,\n", + " data_dir=data_dir,\n", + " base_url=base_url),\n", + ")\n", + "```\n", + "\n", + "Note: The `base_url_path`, `base_url`, and `data_dir` are specific to web navigation and are not necessary for your domain.\n", + "\n", + "
\n", + "\n", + "---\n", + "\n", + "\n", + "\n", + "## Additional configurations\n", + "\n", + "\n", + "When adding a new domain, you may need to configure additional settings:\n", + "\n", + "1. Docker configurations:\n", + " In `a2perf/launch/docker_utils.py`, add entries for your new domain:\n", + "\n", + " a. In the `get_docker_instructions` function:\n", + " ```python\n", + " docker_instructions = {\n", + " # ... existing entries ...\n", + " BenchmarkDomain.MY_DOMAIN.value: common_setup + [\n", + " # Add your domain-specific Docker instructions here\n", + " # E.g., installing dependencies, setting up the environment\n", + " ],\n", + " }\n", + " ```\n", + " \n", + " b. In the `get_entrypoint` function:\n", + " ```python\n", + " entrypoints = {\n", + " # ... existing entries ...\n", + " BenchmarkDomain.MY_DOMAIN.value: xm.CommandList([\n", + " # Add your domain-specific entrypoint commands here\n", + " ]),\n", + " }\n", + "\n", + "2. Update constants:\n", + " In `a2perf/constants.py`, add your new domain to the `BenchmarkDomain` enum:\n", + " ```python\n", + " @gin.constants_from_enum\n", + " class BenchmarkDomain(enum.Enum):\n", + " QUADRUPED_LOCOMOTION = \"QuadrupedLocomotion-v0\"\n", + " WEB_NAVIGATION = \"WebNavigation-v0\"\n", + " CIRCUIT_TRAINING = \"CircuitTraining-v0\"\n", + " MY_DOMAIN = \"MyDomain-v0\"\n", + " ```\n", + " \n", + "These configurations ensure A2Perf can properly set up and run experiments for your new domain.\n", + "\n" + ], + "id": "e706fe6999459158" + }, + { + "metadata": {}, + "cell_type": "markdown", + "source": "", + "id": "8b69291819f7204f" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 2 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython2", + "version": "2.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/content/tutorials/generalization.ipynb b/docs/content/tutorials/generalization.ipynb new file mode 100644 index 0000000..071c914 --- /dev/null +++ b/docs/content/tutorials/generalization.ipynb @@ -0,0 +1,21 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": "# Benchmarking Generalization" + }, + { + "metadata": {}, + "cell_type": "markdown", + "source": "This tutorial is coming soon. Stay tuned!" + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/docs/content/tutorials/inference.md b/docs/content/tutorials/inference.md index 6229f98..29025ef 100644 --- a/docs/content/tutorials/inference.md +++ b/docs/content/tutorials/inference.md @@ -1,9 +1,9 @@ --- layout: "contents" -title: Tutorial on Benchmarking Inference Code +title: Benchmarking Inference firstpage: --- -# Tutorial on Benchmarking Inference Code +# Benchmarking Inference This tutorial is coming soon. Stay tuned! diff --git a/docs/content/tutorials/metrics/index.md b/docs/content/tutorials/metrics/index.md new file mode 100644 index 0000000..3ecb397 --- /dev/null +++ b/docs/content/tutorials/metrics/index.md @@ -0,0 +1,7 @@ +```{toctree} +:hidden: +:maxdepth: 2 +:caption: Collecting Metrics + +metrics +``` diff --git a/docs/content/tutorials/metrics/metrics.ipynb b/docs/content/tutorials/metrics/metrics.ipynb new file mode 100644 index 0000000..1c91f4f --- /dev/null +++ b/docs/content/tutorials/metrics/metrics.ipynb @@ -0,0 +1,37 @@ +{ + "cells": [ + { + "metadata": {}, + "cell_type": "markdown", + "source": "# Collecting Metrics", + "id": "3033b875439e1060" + }, + { + "metadata": {}, + "cell_type": "markdown", + "source": "This tutorial is coming soon. Stay tuned!", + "id": "e2cfbfc9d1617443" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 2 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython2", + "version": "2.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/content/tutorials/training.ipynb b/docs/content/tutorials/training.ipynb index 6f33d28..53dcf89 100644 --- a/docs/content/tutorials/training.ipynb +++ b/docs/content/tutorials/training.ipynb @@ -3,7 +3,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "# Tutorial on Benchmarking Training Code" + "source": "# Benchmarking Training" }, { "metadata": {}, @@ -77,7 +77,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Submission folder\n", + "## Submission folder\n", "In order to benchmark your reinforcement learning algorithms, your submission should have the following structure: \n", "```\n", "├── my_submission\n", diff --git a/docs/index.md b/docs/index.md index e61b5bd..1b07065 100644 --- a/docs/index.md +++ b/docs/index.md @@ -20,7 +20,6 @@ A2Perf is a benchmarking suite for evaluating agents on sequential decision-maki This library contains a collection of environments from domains spanning computer chip-floorplanning, web navigation, and quadruped locomotion. - The Gymnasium interface allows users to initialize and interact with the A2Perf environments as follows: @@ -69,6 +68,9 @@ content/web_navigation/index content/tutorials/training content/tutorials/inference +content/tutorials/generalization +content/tutorials/metrics/index +content/tutorials/add_domain ``` ```{toctree}