tutorial for adding custom domains to a2perf

docs/content/circuit_training/index.md

@@ -47,3 +47,4 @@
 
 CircuitTraining-Ariane-v0
 CircuitTraining-ToyMacroStdcell-v0
+```

docs/content/tutorials/add_domain.ipynb

@@ -0,0 +1,143 @@
+{
+ "cells": [
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "# Adding Custom Domains\n",
+    "\n",
+    "This tutorial will demonstrate how to add additional domains to A2Perf.\n",
+    "\n",
+    "## Path to A2Perf Domains\n",
+    "\n",
+    "Domains are stored in `A2Perf/a2perf/domains`. Currently, you'll find the main domains in this folder such as `circuit_training`, `quadruped_locomotion`, and `web_navigation`. There are also additional modules such as `tfa` that is there for interfacing with Tensorflow Agents.\n",
+    "\n",
+    "<br>\n",
+    "\n",
+    "---\n",
+    "\n",
+    "\n",
+    "## Creating a new domain\n",
+    "\n",
+    "To add a new domain:\n",
+    "\n",
+    "1. Create a new folder in the domains directory. For the purposes of this tutorial, we will create a new domain called \"my_domain\":\n",
+    "\n",
+    "\n",
+    "```bash\n",
+    "cd A2Perf/a2perf/domains\n",
+    "mkdir my_domain\n",
+    "``` \n",
+    "\n",
+    "2. Inside of `my_domain`, create the following files:\n",
+    "- `__init__.py`\n",
+    "- [any other files needed for your domain]\n",
+    "\n",
+    "3. Inside of `__init__.py`, you will need to register your domain with gymnasium. Here is an example of how we do this for the web navigation domain:\n",
+    "\n",
+    "```python\n",
+    "import gymnasium as gym\n",
+    "import pkg_resources\n",
+    "\n",
+    "base_url_path = pkg_resources.resource_filename('a2perf.domains.web_navigation',\n",
+    "                                                'gwob')\n",
+    "base_url = f'file://{base_url_path}/'\n",
+    "data_dir = pkg_resources.resource_filename(\n",
+    "    'a2perf.domains.web_navigation.environment_generation', 'data')\n",
+    "\n",
+    "gym.envs.register(\n",
+    "    id='WebNavigation-v0',\n",
+    "    entry_point=(\n",
+    "        'a2perf.domains.web_navigation.gwob.CoDE.environment:WebNavigationEnv'\n",
+    "    ),\n",
+    "    apply_api_compatibility=False,\n",
+    "    disable_env_checker=False,\n",
+    "    kwargs=dict(\n",
+    "        use_legacy_step=False,\n",
+    "        use_legacy_reset=False,\n",
+    "        data_dir=data_dir,\n",
+    "        base_url=base_url),\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "Note: The `base_url_path`, `base_url`, and `data_dir` are specific to web navigation and are not necessary for your domain.\n",
+    "\n",
+    "<br>\n",
+    "\n",
+    "---\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Additional configurations\n",
+    "\n",
+    "\n",
+    "When adding a new domain, you may need to configure additional settings:\n",
+    "\n",
+    "1. Docker configurations:\n",
+    "   In `a2perf/launch/docker_utils.py`, add entries for your new domain:\n",
+    "\n",
+    "   a. In the `get_docker_instructions` function:\n",
+    "   ```python\n",
+    "   docker_instructions = {\n",
+    "       # ... existing entries ...\n",
+    "       BenchmarkDomain.MY_DOMAIN.value: common_setup + [\n",
+    "           # Add your domain-specific Docker instructions here\n",
+    "           # E.g., installing dependencies, setting up the environment\n",
+    "       ],\n",
+    "   }\n",
+    "   ```\n",
+    "   \n",
+    "   b. In the `get_entrypoint` function:\n",
+    "   ```python\n",
+    "   entrypoints = {\n",
+    "       # ... existing entries ...\n",
+    "       BenchmarkDomain.MY_DOMAIN.value: xm.CommandList([\n",
+    "           # Add your domain-specific entrypoint commands here\n",
+    "       ]),\n",
+    "   }\n",
+    "\n",
+    "2. Update constants:\n",
+    "   In `a2perf/constants.py`, add your new domain to the `BenchmarkDomain` enum:\n",
+    "   ```python\n",
+    "      @gin.constants_from_enum\n",
+    "      class BenchmarkDomain(enum.Enum):\n",
+    "        QUADRUPED_LOCOMOTION = \"QuadrupedLocomotion-v0\"\n",
+    "        WEB_NAVIGATION = \"WebNavigation-v0\"\n",
+    "        CIRCUIT_TRAINING = \"CircuitTraining-v0\"\n",
+    "        MY_DOMAIN = \"MyDomain-v0\"\n",
+    "      ```\n",
+    "   \n",
+    "These configurations ensure A2Perf can properly set up and run experiments for your new domain.\n",
+    "\n"
+   ],
+   "id": "e706fe6999459158"
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "",
+   "id": "8b69291819f7204f"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/content/tutorials/generalization.ipynb

@@ -0,0 +1,21 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "# Benchmarking Generalization"
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "This tutorial is coming soon. Stay tuned!"
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/content/tutorials/inference.md

@@ -1,9 +1,9 @@
 ---
 layout: "contents"
-title: Tutorial on Benchmarking Inference Code
+title: Benchmarking Inference
 firstpage:
 ---
 
-# Tutorial on Benchmarking Inference Code
+# Benchmarking Inference
 
 This tutorial is coming soon. Stay tuned!

docs/content/tutorials/metrics/index.md

@@ -0,0 +1,7 @@
+```{toctree}
+:hidden:
+:maxdepth: 2 
+:caption: Collecting Metrics
+
+metrics
+```

docs/content/tutorials/metrics/metrics.ipynb

@@ -0,0 +1,37 @@
+{
+ "cells": [
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "# Collecting Metrics",
+   "id": "3033b875439e1060"
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "This tutorial is coming soon. Stay tuned!",
+   "id": "e2cfbfc9d1617443"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/content/tutorials/training.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Tutorial on Benchmarking Training Code"
+   "source": "# Benchmarking Training"
   },
   {
    "metadata": {},
@@ -77,7 +77,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Submission folder\n",
+    "## Submission folder\n",
     "In order to benchmark your reinforcement learning algorithms, your submission should have the following structure: \n",
     "```\n",
     "├── my_submission\n",

docs/index.md

@@ -20,7 +20,6 @@ A2Perf is a benchmarking suite for evaluating agents on sequential decision-maki
 This library contains a collection of environments from domains spanning
 computer chip-floorplanning, web navigation, and quadruped locomotion.
 
-
 The Gymnasium interface allows users to initialize and interact with the A2Perf
 environments as follows:
 
@@ -69,6 +68,9 @@ content/web_navigation/index
 
 content/tutorials/training
 content/tutorials/inference
+content/tutorials/generalization
+content/tutorials/metrics/index
+content/tutorials/add_domain
 ```
 
 ```{toctree}