diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml new file mode 100644 index 0000000..62e728a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -0,0 +1,48 @@ +name: Bug Report +description: Submit a bug report +title: "[Bug Report] Bug title" +labels: ["bug"] +body: + - type: textarea + id: description + attributes: + label: Describe the bug + description: A clear and concise description of what the bug is. + validations: + required: true + + - type: textarea + id: code-example + attributes: + label: Code example + description: | + Please try to provide a minimal example to reproduce the bug. Error messages and stack traces are also helpful. + This will be automatically formatted into code, so no need for backticks. + render: shell + + - type: textarea + id: system-info + attributes: + label: System info + description: | + Describe the characteristic of your environment: + * Describe how CogmentLab was installed (pip, docker, source, ...) + * Version of `cogment_lab` (by `cogment_lab.__version__`) + * What OS/version of Linux you're using. Note that while we will accept PRs to improve Window's support, we do not officially support it. + * Python version + + - type: textarea + id: additional-context + attributes: + label: Additional context + description: Add any other context about the problem here. + + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: > + I have checked that there is no similar [issue](https://github.com/cogment/cogment_lab/issues) in + the repo + required: true diff --git a/.github/ISSUE_TEMPLATE/proposal.yml b/.github/ISSUE_TEMPLATE/proposal.yml new file mode 100644 index 0000000..0e6fe10 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/proposal.yml @@ -0,0 +1,49 @@ +name: Proposal +description: Propose changes that are not fixing bugs +title: "[Proposal] Proposal title" +labels: ["enhancement"] +body: + - type: textarea + id: proposal + attributes: + label: Proposal + description: A clear and concise description of the proposal. + validations: + required: true + + - type: textarea + id: motivation + attributes: + label: Motivation + description: | + Please outline the motivation for the proposal. + Is your feature request related to a problem? e.g.,"I'm always frustrated when [...]". + If this is related to another GitHub issue, please link here too. + + - type: textarea + id: pitch + attributes: + label: Pitch + description: A clear and concise description of what you want to happen. + + - type: textarea + id: alternatives + attributes: + label: Alternatives + description: A clear and concise description of any alternative solutions or features you've considered, if any. + + - type: textarea + id: additional-context + attributes: + label: Additional context + description: Add any other context or screenshots about the feature request here. + + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: > + I have checked that there is no similar [issue](https://github.com/cogment/cogment_lab/issues) in + the repo + required: true diff --git a/.github/ISSUE_TEMPLATE/question.yml b/.github/ISSUE_TEMPLATE/question.yml new file mode 100644 index 0000000..9725bc4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/question.yml @@ -0,0 +1,21 @@ +name: Question +description: Ask a question +title: "[Question] Question title" +labels: ["question"] +body: + - type: markdown + attributes: + value: > + If you're a beginner and have basic questions, please ask on + [r/reinforcementlearning](https://www.reddit.com/r/reinforcementlearning/) or in the + [RL Discord](https://discord.com/invite/xhfNqQv) (if you're new please use the beginners channel). + Basic questions that are not bugs or feature requests will be closed without reply, because GitHub + issues are not an appropriate venue for these. Advanced/nontrivial questions, especially in areas where + documentation is lacking, are very much welcome. + - type: textarea + id: question + attributes: + label: Question + description: Your question + validations: + required: true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..df1dcee --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,46 @@ +# Description + +Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. + +Fixes # (issue) + +## Type of change + +Please delete options that are not relevant. + +- [ ] Bug fix (non-breaking change which fixes an issue) +- [ ] New feature (non-breaking change which adds functionality) +- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) +- [ ] This change requires a documentation update + +### Screenshots + +Please attach before and after screenshots of the change if applicable. + + + +# Checklist: + +- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `pre-commit run --all-files` (see `CONTRIBUTING.md` instructions to set it up) +- [ ] I have commented my code, particularly in hard-to-understand areas +- [ ] I have made corresponding changes to the documentation +- [ ] My changes generate no new warnings +- [ ] I have added tests that prove my fix is effective or that my feature works +- [ ] New and existing unit tests pass locally with my changes + + diff --git a/.github/stale.yml b/.github/stale.yml new file mode 100644 index 0000000..9df5640 --- /dev/null +++ b/.github/stale.yml @@ -0,0 +1,62 @@ +# Configuration for probot-stale - https://github.com/probot/stale + +# Number of days of inactivity before an Issue or Pull Request becomes stale +daysUntilStale: 60 + +# Number of days of inactivity before an Issue or Pull Request with the stale label is closed. +# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale. +daysUntilClose: 14 + +# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled) +onlyLabels: + - more-information-needed + +# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable +exemptLabels: + - pinned + - security + - "[Status] Maybe Later" + +# Set to true to ignore issues in a project (defaults to false) +exemptProjects: true + +# Set to true to ignore issues in a milestone (defaults to false) +exemptMilestones: true + +# Set to true to ignore issues with an assignee (defaults to false) +exemptAssignees: true + +# Label to use when marking as stale +staleLabel: stale + +# Comment to post when marking as stale. Set to `false` to disable +markComment: > + This issue has been automatically marked as stale because it has not had + recent activity. It will be closed if no further activity occurs. Thank you + for your contributions. + +# Comment to post when removing the stale label. +# unmarkComment: > +# Your comment here. + +# Comment to post when closing a stale Issue or Pull Request. +# closeComment: > +# Your comment here. + +# Limit the number of actions per hour, from 1-30. Default is 30 +limitPerRun: 30 + +# Limit to only `issues` or `pulls` +only: issues + +# Optionally, specify configuration settings that are specific to just 'issues' or 'pulls': +# pulls: +# daysUntilStale: 30 +# markComment: > +# This pull request has been automatically marked as stale because it has not had +# recent activity. It will be closed if no further activity occurs. Thank you +# for your contributions. + +# issues: +# exemptLabels: +# - confirmed diff --git a/.github/workflows/build-docs.yml b/.github/workflows/build-docs.yml new file mode 100644 index 0000000..46c27d1 --- /dev/null +++ b/.github/workflows/build-docs.yml @@ -0,0 +1,46 @@ +name: Build main branch documentation website +on: + push: + branches: [main] +permissions: + contents: write +jobs: + docs: + name: Generate Website + runs-on: ubuntu-latest + env: + SPHINX_GITHUB_CHANGELOG_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v3 + + - uses: actions/setup-python@v4 + with: + python-version: '3.9' + + - name: Install dependencies + run: pip install -r docs/requirements.txt + + - name: Install Gymnasium + run: pip install mujoco && pip install .[box2d] + + - name: Build Envs Docs + run: python docs/scripts/gen_mds.py && python docs/scripts/gen_envs_display.py + + - name: Build + run: sphinx-build -b dirhtml -v docs _build + + - name: Move 404 + run: mv _build/404/index.html _build/404.html + + - name: Update 404 links + run: python docs/scripts/move_404.py _build/404.html + + - name: Remove .doctrees + run: rm -r _build/.doctrees + + - name: Upload to GitHub Pages + uses: JamesIves/github-pages-deploy-action@v4 + with: + folder: _build + target-folder: main + clean: false diff --git a/.github/workflows/build-publish.yml b/.github/workflows/build-publish.yml new file mode 100644 index 0000000..dd96506 --- /dev/null +++ b/.github/workflows/build-publish.yml @@ -0,0 +1,68 @@ +# This workflow will build and (if release) publish Python distributions to PyPI +# For more information see: +# - https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions +# - https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ +# +# derived from https://github.com/Farama-Foundation/PettingZoo/blob/e230f4d80a5df3baf9bd905149f6d4e8ce22be31/.github/workflows/build-publish.yml +name: build-publish + +on: + push: + branches: [main] + pull_request: + branches: [main] + release: + types: [published] + +jobs: + build-wheels: + runs-on: ${{ matrix.os }} + strategy: + matrix: + include: + - os: ubuntu-latest + python: 37 + platform: manylinux_x86_64 + - os: ubuntu-latest + python: 38 + platform: manylinux_x86_64 + - os: ubuntu-latest + python: 39 + platform: manylinux_x86_64 + - os: ubuntu-latest + python: 310 + platform: manylinux_x86_64 + - os: ubuntu-latest + python: 311 + platform: manylinux_x86_64 + + steps: + - uses: actions/checkout@v3 + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: '3.x' + - name: Install dependencies + run: python -m pip install --upgrade pip setuptools build + - name: Build sdist and wheels + run: python -m build + - name: Store wheels + uses: actions/upload-artifact@v3 + with: + path: dist + + publish: + runs-on: ubuntu-latest + needs: + - build-wheels + if: github.event_name == 'release' && github.event.action == 'published' + steps: + - name: Download dists + uses: actions/download-artifact@v3 + with: + name: artifact + path: dist + - name: Publish + uses: pypa/gh-action-pypi-publish@release/v1 + with: + password: ${{ secrets.PYPI_API_TOKEN }} diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 0000000..88b3e92 --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,37 @@ +name: build +on: [pull_request, push] + +permissions: + contents: read # to fetch code (actions/checkout) + +jobs: + build-all: + runs-on: ubuntu-latest + strategy: + matrix: + python-version: ['3.8', '3.9', '3.10'] + steps: + - uses: actions/checkout@v3 + - run: | + docker build -f bin/all-py.Dockerfile \ + --build-arg PYTHON_VERSION=${{ matrix.python-version }} \ + --tag cogment_lab-all-docker . + - name: Start background services + run: docker run -d --name cogment_lab-test cogment_lab-all-docker /usr/local/bin/cogmentlab launch base + + - name: Run tests + run: docker run cogment_lab-all-docker pytest tests/* + + + build-necessary: + runs-on: + ubuntu-latest + steps: + - uses: actions/checkout@v3 + - run: | + docker build -f bin/necessary-py.Dockerfile \ + --build-arg PYTHON_VERSION='3.10' \ + --tag cogment_lab-necessary-docker . + - name: Run tests + run: | + docker run cogment_lab-necessary-docker pytest tests diff --git a/.github/workflows/docs-manual-versioning.yml b/.github/workflows/docs-manual-versioning.yml new file mode 100644 index 0000000..2156b5c --- /dev/null +++ b/.github/workflows/docs-manual-versioning.yml @@ -0,0 +1,71 @@ +name: Manual Docs Versioning +on: + workflow_dispatch: + inputs: + version: + description: 'Documentation version to create' + required: true + commit: + description: 'Commit used to build the Documentation version' + required: false + latest: + description: 'Latest version' + type: boolean + +permissions: + contents: write +jobs: + docs: + name: Generate Website for new version + runs-on: ubuntu-latest + env: + SPHINX_GITHUB_CHANGELOG_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v3 + if: inputs.commit == '' + + - uses: actions/checkout@v3 + if: inputs.commit != '' + with: + ref: ${{ inputs.commit }} + + - uses: actions/setup-python@v4 + with: + python-version: '3.9' + + - name: Install dependencies + run: pip install -r docs/requirements.txt + + - name: Install CogmentLab + run: pip install .[atari,accept-rom-license,box2d] + + - name: Build Envs Docs + run: python docs/scripts/gen_mds.py && python docs/scripts/gen_envs_display.py + + - name: Build + run: sphinx-build -b dirhtml -v docs _build + + - name: Move 404 + run: mv _build/404/index.html _build/404.html + + - name: Update 404 links + run: python docs/scripts/move_404.py _build/404.html + + - name: Remove .doctrees + run: rm -r _build/.doctrees + + - name: Upload to GitHub Pages + uses: JamesIves/github-pages-deploy-action@v4 + with: + folder: _build + target-folder: ${{ inputs.version }} + clean: false + + - name: Upload to GitHub Pages + uses: JamesIves/github-pages-deploy-action@v4 + if: inputs.latest + with: + folder: _build + clean-exclude: | + *.*.*/ + main diff --git a/.github/workflows/docs-versioning.yml b/.github/workflows/docs-versioning.yml new file mode 100644 index 0000000..54b50e0 --- /dev/null +++ b/.github/workflows/docs-versioning.yml @@ -0,0 +1,59 @@ +name: Docs Versioning +on: + push: + tags: + - 'v?*.*.*' +permissions: + contents: write +jobs: + docs: + name: Generate Website for new version + runs-on: ubuntu-latest + env: + SPHINX_GITHUB_CHANGELOG_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v3 + + - uses: actions/setup-python@v4 + with: + python-version: '3.9' + + - name: Get tag + id: tag + uses: dawidd6/action-get-tag@v1 + + - name: Install dependencies + run: pip install -r docs/requirements.txt + + - name: Install CogmentLab + run: pip install .[atari,accept-rom-license,box2d] + + - name: Build Envs Docs + run: python docs/scripts/gen_mds.py && python docs/scripts/gen_envs_display.py + + - name: Build + run: sphinx-build -b dirhtml -v docs _build + + - name: Move 404 + run: mv _build/404/index.html _build/404.html + + - name: Update 404 links + run: python docs/scripts/move_404.py _build/404.html + + - name: Remove .doctrees + run: rm -r _build/.doctrees + + - name: Upload to GitHub Pages + uses: JamesIves/github-pages-deploy-action@v4 + with: + folder: _build + target-folder: ${{steps.tag.outputs.tag}} + clean: false + + - name: Upload to GitHub Pages + uses: JamesIves/github-pages-deploy-action@v4 + with: + folder: _build + clean-exclude: | + *.*.*/ + main diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml new file mode 100644 index 0000000..80ce02a --- /dev/null +++ b/.github/workflows/pre-commit.yml @@ -0,0 +1,21 @@ +# https://pre-commit.com +# This GitHub Action assumes that the repo contains a valid .pre-commit-config.yaml file. +name: pre-commit +on: + pull_request: + push: + branches: [main] + +permissions: + contents: read # to fetch code (actions/checkout) + +jobs: + pre-commit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: actions/setup-python@v4 + - run: python -m pip install pre-commit + - run: python -m pre_commit --version + - run: python -m pre_commit install + - run: python -m pre_commit run --all-files diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..86adbeb --- /dev/null +++ b/.gitignore @@ -0,0 +1,45 @@ +*.swp +*.pyc +*.py~ +.DS_Store +.cache +.pytest_cache/ +__pycache__/ + +# Setuptools distribution and build folders. +/dist/ +/build +/wheels +/wheelhouse + +# Virtualenv +/env +/venv + +# Python egg metadata, regenerated from source files by setuptools. +/*.egg-info + +*.sublime-project +*.sublime-workspace + +logs/ + +.ipynb_checkpoints +ghostdriver.log + +junk +MUJOCO_LOG.txt + +rllab_mujoco + +tutorial/*.html + +# IDE files +.eggs +.tox + +# PyCharm project files +.idea +vizdoom.ini + +lib/ \ No newline at end of file diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..a4bd6cd --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,9 @@ +stages: + - lint + +licenses_checker: + stage: lint + image: registry.gitlab.com/ai-r/cogment/license-checker:latest + script: + - license-checker + diff --git a/.license.yaml b/.license.yaml new file mode 100644 index 0000000..b71f6b0 --- /dev/null +++ b/.license.yaml @@ -0,0 +1,6 @@ +license: + copyright_year: 2024 + author: "AI Redefined Inc. " + ignore: + - "**/.venv" + - "**/cog_settings.py" diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..6714a99 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,68 @@ +# See https://pre-commit.com for more information +# See https://pre-commit.com/hooks.html for more hooks +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.4.0 + hooks: + - id: check-symlinks + - id: destroyed-symlinks + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + - id: check-toml + - id: check-ast + - id: check-added-large-files + - id: check-merge-conflict + - id: check-executables-have-shebangs + - id: check-shebang-scripts-are-executable + - id: detect-private-key + - id: debug-statements + - repo: https://github.com/codespell-project/codespell + rev: v2.2.4 + hooks: + - id: codespell + args: + - --ignore-words-list=reacher,ure,referenc,wile + - repo: https://github.com/PyCQA/flake8 + rev: 6.0.0 + hooks: + - id: flake8 + args: + - --ignore=E203,W503,E741 + - --max-complexity=30 + - --max-line-length=456 + - --show-source + - --statistics + - repo: https://github.com/asottile/pyupgrade + rev: v3.3.1 + hooks: + - id: pyupgrade + args: ["--py37-plus"] + - repo: https://github.com/PyCQA/isort + rev: 5.12.0 + hooks: + - id: isort + - repo: https://github.com/python/black + rev: 23.3.0 + hooks: + - id: black + - repo: https://github.com/pycqa/pydocstyle + rev: 6.3.0 + hooks: + - id: pydocstyle + args: + - --source + - --explain + - --convention=google + additional_dependencies: ["tomli"] + - repo: local + hooks: + - id: pyright + name: pyright + entry: pyright + language: node + pass_filenames: false + types: [python] + additional_dependencies: ["pyright"] + args: + - --project=pyproject.toml diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..cd482d8 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, +and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by +the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all +other entities that control, are controlled by, or are under common +control with that entity. For the purposes of this definition, +"control" means (i) the power, direct or indirect, to cause the +direction or management of such entity, whether by contract or +otherwise, or (ii) ownership of fifty percent (50%) or more of the +outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity +exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, +including but not limited to software source code, documentation +source, and configuration files. + +"Object" form shall mean any form resulting from mechanical +transformation or translation of a Source form, including but +not limited to compiled object code, generated documentation, +and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or +Object form, made available under the License, as indicated by a +copyright notice that is included in or attached to the work +(an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object +form, that is based on (or derived from) the Work and for which the +editorial revisions, annotations, elaborations, or other modifications +represent, as a whole, an original work of authorship. For the purposes +of this License, Derivative Works shall not include works that remain +separable from, or merely link (or bind by name) to the interfaces of, +the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including +the original version of the Work and any modifications or additions +to that Work or Derivative Works thereof, that is intentionally +submitted to Licensor for inclusion in the Work by the copyright owner +or by an individual or Legal Entity authorized to submit on behalf of +the copyright owner. For the purposes of this definition, "submitted" +means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, +and issue tracking systems that are managed by, or on behalf of, the +Licensor for the purpose of discussing and improving the Work, but +excluding communication that is conspicuously marked or otherwise +designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity +on behalf of whom a Contribution has been received by Licensor and +subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of +this License, each Contributor hereby grants to You a perpetual, +worldwide, non-exclusive, no-charge, royalty-free, irrevocable +copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the +Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of +this License, each Contributor hereby grants to You a perpetual, +worldwide, non-exclusive, no-charge, royalty-free, irrevocable +(except as stated in this section) patent license to make, have made, +use, offer to sell, sell, import, and otherwise transfer the Work, +where such license applies only to those patent claims licensable +by such Contributor that are necessarily infringed by their +Contribution(s) alone or by combination of their Contribution(s) +with the Work to which such Contribution(s) was submitted. If You +institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work +or a Contribution incorporated within the Work constitutes direct +or contributory patent infringement, then any patent licenses +granted to You under this License for that Work shall terminate +as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the +Work or Derivative Works thereof in any medium, with or without +modifications, and in Source or Object form, provided that You +meet the following conditions: + +(a) You must give any other recipients of the Work or +Derivative Works a copy of this License; and + +(b) You must cause any modified files to carry prominent notices +stating that You changed the files; and + +(c) You must retain, in the Source form of any Derivative Works +that You distribute, all copyright, patent, trademark, and +attribution notices from the Source form of the Work, +excluding those notices that do not pertain to any part of +the Derivative Works; and + +(d) If the Work includes a "NOTICE" text file as part of its +distribution, then any Derivative Works that You distribute must +include a readable copy of the attribution notices contained +within such NOTICE file, excluding those notices that do not +pertain to any part of the Derivative Works, in at least one +of the following places: within a NOTICE text file distributed +as part of the Derivative Works; within the Source form or +documentation, if provided along with the Derivative Works; or, +within a display generated by the Derivative Works, if and +wherever such third-party notices normally appear. The contents +of the NOTICE file are for informational purposes only and +do not modify the License. You may add Your own attribution +notices within Derivative Works that You distribute, alongside +or as an addendum to the NOTICE text from the Work, provided +that such additional attribution notices cannot be construed +as modifying the License. + +You may add Your own copyright statement to Your modifications and +may provide additional or different license terms and conditions +for use, reproduction, or distribution of Your modifications, or +for any such Derivative Works as a whole, provided Your use, +reproduction, and distribution of the Work otherwise complies with +the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, +any Contribution intentionally submitted for inclusion in the Work +by You to the Licensor shall be under the terms and conditions of +this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify +the terms of any separate license agreement you may have executed +with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade +names, trademarks, service marks, or product names of the Licensor, +except as required for reasonable and customary use in describing the +origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or +agreed to in writing, Licensor provides the Work (and each +Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +implied, including, without limitation, any warranties or conditions +of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A +PARTICULAR PURPOSE. You are solely responsible for determining the +appropriateness of using or redistributing the Work and assume any +risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, +whether in tort (including negligence), contract, or otherwise, +unless required by applicable law (such as deliberate and grossly +negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, +incidental, or consequential damages of any character arising as a +result of this License or out of the use or inability to use the +Work (including but not limited to damages for loss of goodwill, +work stoppage, computer failure or malfunction, or any and all +other commercial damages or losses), even if such Contributor +has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing +the Work or Derivative Works thereof, You may choose to offer, +and charge a fee for, acceptance of support, warranty, indemnity, +or other liability obligations and/or rights consistent with this +License. However, in accepting such obligations, You may act only +on Your own behalf and on Your sole responsibility, not on behalf +of any other Contributor, and only if You agree to indemnify, +defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason +of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + +To apply the Apache License to your work, attach the following +boilerplate notice, with the fields enclosed by brackets "[]" +replaced with your own identifying information. (Don't include +the brackets!) The text should be enclosed in the appropriate +comment syntax for the file format. We also recommend that a +file or class name and description of purpose be included on the +same "printed page" as the copyright notice for easier +identification within third-party archives. + +Copyright [yyyy] [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..66398af --- /dev/null +++ b/README.md @@ -0,0 +1,97 @@ +![cog-lab](https://github.com/RedTachyon/cogment_lab/assets/19414946/165557d0-fdf0-4d0a-99f1-3fc321fa194c) + +# Human + AI = ❤️ + + +## Docs | Blog | Discord + + +[![Package version](https://img.shields.io/pypi/v/cogment-lab?color=%23007ec6&label=pypi%20package)](https://pypi.org/project/cogment-lab) +[![Downloads](https://pepy.tech/badge/cogment-lab)](https://pepy.tech/project/cogment-lab) +[![Supported Python versions](https://img.shields.io/pypi/pyversions/cogment-lab.svg)](https://pypi.org/project/cogment-lab) +[![License - Apache 2.0](https://img.shields.io/badge/license-Apache_2.0-green)](https://github.com/cogment-lab/blob/main/LICENSE) +[![Follow @AI_Redefined](https://img.shields.io/twitter/follow/nestframework.svg?style=social&label=Follow%20@AI_Redefined)](https://twitter.com/AI_Redefined) +[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://pre-commit.com/) + +# Introduction + +Cogment Lab is a toolkit for doing HILL RL -- that is human-in-the-loop learning, with an emphasis on reinforcement learning. +It is based on [cogment](https://cogment.ai), a low-level framework for exchanging messages between +environments, AI agents and humans. +It's the perfect tool for when you want to interact with your environment yourself, and maybe even trained AI agents. + +# Cogment interaction model + +While it typically isn't necessary to interact with Cogment directly to use Cogment Lab, it is useful to understand the principles on which it operates. + +Cogment exchanges messages between environments and actor. These messages contain the observations, actions, rewards, and anything +else that you might want to keep track of. + +Interactions are split into Trials, which correspond to the typical notion of an episode in RL. Each trial has a unique ID, and + +## Cogment Lab at a glance + +Cogment Lab (as well as Cogment in general) follows a microservice-based architecture. +Each environment, agent, and human interface (collectively: service) is launched as a subprocess, and exchanges messages with the orchestrator, +which in turn ensures synchronization and correct routing of messages. + +Generally speaking, you don't need to worry about any of that - Cogment Lab conveniently covers up all the rough edges, +allowing you to do your research without worries. + +Cogment Lab is inherently asynchronous - but if you're not familiar with async python, don't worry about it. +The only things you need to remember are: +- Wrap your code in `async def main()` +- Run it with `asyncio.run(main())` +- When calling certain functions use the `await` keyword, e.g. `data = await cog.get_episode_data(...)` + +If you are familiar with async programming, there's a lot of interesting things you can do with it - go crazy. + + +## Terminology + +- A `service` is anything that interacts with the Cogment orchestrator. It can be an environment or an actor, including human actors. +- An `actor` in particular is the service that interacts with an environment, and ofter wraps an `agent`. The internal structure of an actor is entirely up to the user +- An `agent` is what we typically think of as an agent in RL - something that perceives its environment and acts upon it. We do not attempt to solve the agent foundation problem in this documentation. +- An `agent` is simultaneously the part of the environment that's taking an action - multiagent environments may have several agents, so we need to assign an actor to each agent. + + +## Known rough edges + +- When running the web UI, you can open the tab only once per launched process. So if you open the UI, you can run however many trials you want, as long as you don't close it. If you do close it, you should kill the process and start a new one. + + +## Local installation + +- Requires Python 3.10 +- Install requirements in a virtual env with somthing similar to the following + + ```console + $ python -m venv .venv + $ source .venv/bin/activate + $ pip install -r requirements.txt + $ pip install -e . + ``` +- For the examples you'll need to install the additional `examples_requirements.txt`. + + +### Apple silicon installation + +To run on M1/2/3 macs, you'll need to perform those additional steps + +``` +pip uninstall grpcio grpcio-tools +export GRPC_PYTHON_LDFLAGS=" -framework CoreFoundation" +pip install grpcio==1.48.2 grpcio-tools==1.48.2 --no-binary :all: +``` + + +## Usage + +Run `cogmentlab launch base`. + +Then, run whatever scripts or notebooks. + +Terminology: +- Model: a relatively raw PyTorch (or other?) model, inheriting from `nn.Module` +- Agent: a model wrapped in some utility class to interact with np arrays +- Actor: a cogment service that may involve models and/or actors diff --git a/TODO b/TODO new file mode 100644 index 0000000..fa6f4e4 --- /dev/null +++ b/TODO @@ -0,0 +1,19 @@ +Idea: +- Support native gymnasium VectorEnv when we finish the API +- Support launching the same environment in several processes as cogment-based vectorization + + + +TODO: +- Improve robustness of the UI? (reconnecting, etc.) +- Better UI builder (define buttons, sliders, etc.) +- More QoL functions (are any missing?) +- Clean up logging (probably need to define a logger somewhere in top level of the library? maybe in each file?) +- Enable running services in the main process (async; actors work, do the same for envs and web_ui) +- Maybe rename web_ui to human? +- Add more conversions (observer, teacher for PettingZoo) +- Add support for vectorized envs, vectorized actors +- Add support for parallel envs +- Do batched evaluation with vectorized envs, homogeneous pettingzoo parallel envs +- Connect4 integration +- Instead of a fancy BaseEnv/BaseActor, use Gymnasium EZPickle (or reimplement, it's super simple) \ No newline at end of file diff --git a/bin/all-py.Dockerfile b/bin/all-py.Dockerfile new file mode 100644 index 0000000..037a76f --- /dev/null +++ b/bin/all-py.Dockerfile @@ -0,0 +1,44 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ARG PYTHON_VERSION +FROM python:$PYTHON_VERSION + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +RUN apt-get -y update \ + && apt-get install --no-install-recommends -y \ + unzip \ + libglu1-mesa-dev \ + libgl1-mesa-dev \ + libosmesa6-dev \ + xvfb \ + patchelf \ + ffmpeg cmake \ + && apt-get autoremove -y \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +COPY . /usr/local/cogment_lab/ +WORKDIR /usr/local/cogment_lab/ + +RUN pip install -r requirements.txt --no-cache-dir +RUN pip install .[all] --no-cache-dir + +ENV PATH="/usr/local/bin:${PATH}" + + +RUN /usr/local/bin/cogmentlab install + +ENTRYPOINT ["/usr/local/cogment_lab/bin/docker_entrypoint"] diff --git a/bin/docker_entrypoint b/bin/docker_entrypoint new file mode 100755 index 0000000..a5fe2c9 --- /dev/null +++ b/bin/docker_entrypoint @@ -0,0 +1,26 @@ +#!/bin/bash +# This script is the entrypoint for our Docker image. + +set -ex + +# Set up display; otherwise rendering will fail +Xvfb -screen 0 1024x768x24 & +export DISPLAY=:0 + +# Wait for the file to come up +display=0 +file="/tmp/.X11-unix/X$display" +for i in $(seq 1 10); do + if [ -e "$file" ]; then + break + fi + + echo "Waiting for $file to be created (try $i/10)" + sleep "$i" +done +if ! [ -e "$file" ]; then + echo "Timing out: $file was not created" + exit 1 +fi + +exec "$@" diff --git a/bin/necessary-py.Dockerfile b/bin/necessary-py.Dockerfile new file mode 100644 index 0000000..f9f709f --- /dev/null +++ b/bin/necessary-py.Dockerfile @@ -0,0 +1,38 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ARG PYTHON_VERSION +FROM python:$PYTHON_VERSION + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +RUN apt-get -y update \ + && apt-get install --no-install-recommends -y \ + unzip \ + libglu1-mesa-dev \ + libgl1-mesa-dev \ + libosmesa6-dev \ + xvfb \ + patchelf \ + ffmpeg cmake \ + && apt-get autoremove -y \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +COPY . /usr/local/cogment_lab/ +WORKDIR /usr/local/cogment_lab/ + +RUN pip install .[dev] --no-cache-dir + +ENTRYPOINT ["/usr/local/cogment_lab/bin/docker_entrypoint"] diff --git a/cogment_lab/__init__.py b/cogment_lab/__init__.py new file mode 100644 index 0000000..c0d2dc3 --- /dev/null +++ b/cogment_lab/__init__.py @@ -0,0 +1,26 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations +import logging + +from cogment_lab.process_manager import Cogment +from cogment.utils import logger + +logger.addHandler(logging.NullHandler()) + +__version__ = "0.0.1" + + +__all__ = ["Cogment"] diff --git a/cogment_lab/actors/__init__.py b/cogment_lab/actors/__init__.py new file mode 100644 index 0000000..e3c0650 --- /dev/null +++ b/cogment_lab/actors/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .simple import RandomActor, ConstantActor diff --git a/cogment_lab/actors/configs.py b/cogment_lab/actors/configs.py new file mode 100644 index 0000000..782e122 --- /dev/null +++ b/cogment_lab/actors/configs.py @@ -0,0 +1,25 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import TypedDict + + +class AgentConfig(TypedDict): + agent_name: str + agent_type: str # "cogment" or "custom" + + cogment_actor: str + + agent_class: str + agent_kwargs: dict diff --git a/cogment_lab/actors/nn_actor.py b/cogment_lab/actors/nn_actor.py new file mode 100644 index 0000000..ed64c67 --- /dev/null +++ b/cogment_lab/actors/nn_actor.py @@ -0,0 +1,86 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import numpy as np +import torch +from torch import nn +from torch.nn import functional as F + +from cogment_lab.core import CogmentActor +from coltra import Observation, DAgent, CAgent +from coltra.models import BaseModel + + +class ColtraActor(CogmentActor): + def __init__(self, model: BaseModel): + super().__init__(model) + self.model = model + self.agent = DAgent(self.model) if self.model.discrete else CAgent(self.model) + + async def act(self, observation: np.ndarray, rendered_frame=None): + obs = Observation(vector=observation) + action, _, _ = self.agent.act(obs) + return action.discrete + + +class NNActor(CogmentActor): + def __init__(self, network: nn.Module, device: str = "cpu"): + super().__init__(network=network) + self.network = network + self.device = device + self.eps = 0.0 + self.num_actions: int | None = None + self.rng = np.random.default_rng(0) + + async def act(self, observation: np.ndarray, rendered_frame=None) -> int: + if self.num_actions is None: + observation = observation.copy() + obs = torch.from_numpy(observation).float().to(self.device) + [act_probs] = self.network(obs) + self.num_actions = act_probs.shape[0] + + if self.eps > 0.0 and self.rng.random() < self.eps: + return self.rng.integers(0, self.num_actions) + + observation = observation.copy() + obs = torch.from_numpy(observation).float().to(self.device) + [act_probs] = self.network(obs) + return act_probs.detach().cpu().numpy().argmax() + + def set_eps(self, eps: float): + self.eps = eps + + +class BoltzmannActor(CogmentActor): + def __init__(self, network: nn.Module, device: str = "cpu"): + super().__init__(network=network) + self.network = network + self.device = device + self.temperature = 1.0 + self.num_actions: int | None = None + self.rng = np.random.default_rng(0) + + async def act(self, observation: np.ndarray, rendered_frame=None) -> int: + observation = observation.copy() + obs = torch.from_numpy(observation).float().to(self.device) + with torch.no_grad(): + [act_vals] = self.network(obs) + act_probs = F.softmax(act_vals / self.temperature, dim=0) + + action = torch.multinomial(act_probs, 1).item() + + return action + + def set_temperature(self, temperature: float): + self.temperature = temperature diff --git a/cogment_lab/actors/runner.py b/cogment_lab/actors/runner.py new file mode 100644 index 0000000..8bc42e4 --- /dev/null +++ b/cogment_lab/actors/runner.py @@ -0,0 +1,53 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +from multiprocessing import Queue + +import cogment + +from cogment_lab.generated import cog_settings +from cogment_lab.utils.runners import setup_logging +from cogment_lab.core import BaseActor + + +async def register_actor(actor: BaseActor, actor_name: str, queue: Queue, port: int = 9002): + context = cogment.Context(cog_settings=cog_settings, user_id="cogment_lab") + + context.register_actor(impl=actor.impl, impl_name=actor_name, actor_classes=["player"]) + + serve = context.serve_all_registered(cogment.ServedEndpoint(port=port)) + + queue.put(True) + + await serve + + +def actor_runner( + actor_class: type, + actor_args: tuple, + actor_kwargs: dict, + actor_name: str, + signal_queue: Queue, + port: int = 9002, + log_file: str | None = None, +): + """Given an actor, runs it""" + if log_file: + setup_logging(log_file) + actor = actor_class(*actor_args, **actor_kwargs) + + asyncio.run(register_actor(actor, actor_name, signal_queue, port)) diff --git a/cogment_lab/actors/simple.py b/cogment_lab/actors/simple.py new file mode 100644 index 0000000..dd7cbc6 --- /dev/null +++ b/cogment_lab/actors/simple.py @@ -0,0 +1,41 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from typing import Any + +import gymnasium as gym +import numpy as np + +from cogment_lab.core import CogmentActor + + +class RandomActor(CogmentActor): + def __init__(self, action_space: gym.spaces.Space): + super().__init__(action_space) + self.gym_action_space = action_space + + async def act(self, observation: Any, rendered_frame=None): + return self.gym_action_space.sample() + + +class ConstantActor(CogmentActor): + def __init__(self, action: Any): + super().__init__(action) + if isinstance(action, list): + action = np.array(action) + self.action = action + + async def act(self, observation: Any, rendered_frame=None): + return self.action diff --git a/cogment_lab/cli/__init__.py b/cogment_lab/cli/__init__.py new file mode 100644 index 0000000..08d7dad --- /dev/null +++ b/cogment_lab/cli/__init__.py @@ -0,0 +1,14 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + diff --git a/cogment_lab/cli/actor.py b/cogment_lab/cli/actor.py new file mode 100644 index 0000000..e656980 --- /dev/null +++ b/cogment_lab/cli/actor.py @@ -0,0 +1,66 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import logging + +import cogment +import yaml + +from cogment_lab.actors.configs import AgentConfig +from cogment_lab.core import BaseActor, NativeActor +from cogment_lab.generated import cog_settings +from cogment_lab.utils import import_object +from cogment_lab.utils.yaml_utils import gym_space_constructors + + +log = logging.getLogger(__name__) + + +def get_actor(agent_config: AgentConfig) -> BaseActor: + agent_type = agent_config.get("agent_type") + if agent_type is None: + raise ValueError("agent_type is not provided in config") + + if agent_type == "cogment": + impl_path = agent_config["cogment_actor"] + impl = import_object(impl_path) + agent = NativeActor(impl=impl) + elif agent_type == "custom": + agent_class = agent_config["agent_class"] + agent_kwargs = agent_config["agent_kwargs"] + cls = import_object(agent_class) + agent = cls(**agent_kwargs) + else: + raise NotImplementedError(f"Invalid agent_type: {agent_type}") + + return agent + + +async def create_agents(agent_configs: list[AgentConfig], port: int): + context = cogment.Context(cog_settings=cog_settings, user_id="cogment_lab") + + for config in agent_configs: + agent = get_actor(config) + context.register_actor(impl=agent.impl, impl_name=config["agent_name"]) + + await context.serve_all_registered(cogment.ServedEndpoint(port=port)) + + +def actor_main(config_path: str): + gym_space_constructors() + with open(config_path) as config_file: + config = yaml.load(config_file, Loader=yaml.Loader) + log.info(config) + asyncio.run(create_agents(agent_configs=config["agents"], port=config["port"])) diff --git a/cogment_lab/cli/cli.py b/cogment_lab/cli/cli.py new file mode 100644 index 0000000..a32d3a6 --- /dev/null +++ b/cogment_lab/cli/cli.py @@ -0,0 +1,107 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import argparse +import logging +import os +import subprocess +import sys + +TEAL = "\033[36m" +RESET = "\033[0m" + +custom_format = f"%(asctime)s {TEAL}[%(levelname)s] [%(name)s]{RESET} %(message)s [thread:%(thread)d]" + +formatter = logging.Formatter(custom_format, datefmt="%Y-%m-%dT%H:%M:%S%z") + +handler = logging.StreamHandler() +handler.setFormatter(formatter) + +logging.basicConfig(level=logging.INFO, handlers=[handler]) + +sys.path.insert(0, "..") + + +def install_cogment(path: str | None = None): + try: + subprocess.run( + [ + "curl", + "--silent", + "-L", + "https://raw.githubusercontent.com/cogment/cogment/main/install.sh", + "--output", + "install-cogment.sh", + ], + check=True, + ) + subprocess.run(["chmod", "+x", "install-cogment.sh"], check=True) + cmd = ["sudo", "./install-cogment.sh"] + if path: + cmd += ["--install-dir", path] + cmd += ["--version", "2.19.1"] + if os.getenv("GITHUB_ACTIONS") == "true": + cmd = cmd[1:] # Remove sudo for github actions + subprocess.run(cmd, check=True) + logging.info("Cogment installed successfully.") + except subprocess.CalledProcessError as e: + logging.error(f"Installation failed: {e}") + finally: + if os.path.exists("install-cogment.sh"): + os.remove("install-cogment.sh") + logging.info("Cleanup completed.") + + +def main(): + parser = argparse.ArgumentParser(description="Cogment Lab CLI") + subparsers = parser.add_subparsers(dest="command") + + # launch subcommand + parser_launch = subparsers.add_parser("launch") + parser_launch.add_argument("file") + + # env subcommand + parser_env = subparsers.add_parser("env") + parser_env.add_argument("file") + + parser_actor = subparsers.add_parser("actor") + parser_actor.add_argument("file") + + parser_install = subparsers.add_parser("install") + parser_install.add_argument("path", nargs="?") + + args = parser.parse_args() + + if args.command == "install": + install_cogment(args.path) + elif args.command == "launch": + from cogment_lab.cli import launch + + launch.launch_main(args.file) + elif args.command == "env": + from cogment_lab.cli import env + + env.env_main(args.file) + elif args.command == "actor": + from cogment_lab.cli import actor + + actor.actor_main(args.file) + else: + print("Invalid command. Use 'launch', 'env', 'actor' or `install`.") + + +if __name__ == "__main__": + main() diff --git a/cogment_lab/cli/env.py b/cogment_lab/cli/env.py new file mode 100644 index 0000000..a7bb253 --- /dev/null +++ b/cogment_lab/cli/env.py @@ -0,0 +1,63 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import logging + +import cogment +import yaml + +from cogment_lab.core import BaseEnv, NativeEnv +from cogment_lab.envs.configs import EnvConfig +from cogment_lab.envs.environment import GymEnvironment +from cogment_lab.generated import cog_settings +from cogment_lab.utils import import_object + + +# log = logging.getLogger(__name__) + + +def get_environment(config: EnvConfig) -> BaseEnv: + """Given a config, generates an impl for a cogment environment""" + env_type = config.get("env_type") + if env_type is None: + raise ValueError("env_type is not provided in config") + + if env_type == "gymnasium": + env = GymEnvironment(config) + elif env_type == "cogment": + impl_path = config["cogment_env"] + impl = import_object(impl_path) + env = NativeEnv(impl=impl) + else: + raise NotImplementedError(f"Invalid env_type: {env_type}") + + return env + + +async def create_envs(env_configs: list[EnvConfig], port: int): + context = cogment.Context(cog_settings=cog_settings, user_id="cogment_lab") + + for config in env_configs: + env = get_environment(config) + context.register_environment(impl=env.impl, impl_name=config["env_name"]) + + await context.serve_all_registered(cogment.ServedEndpoint(port=port)) + + +def env_main(config_path: str): + with open(config_path) as config_file: + config = yaml.load(config_file, Loader=yaml.Loader) + logging.info(config) + asyncio.run(create_envs(env_configs=config["environments"], port=config["port"])) diff --git a/cogment_lab/cli/launch.py b/cogment_lab/cli/launch.py new file mode 100644 index 0000000..22723f1 --- /dev/null +++ b/cogment_lab/cli/launch.py @@ -0,0 +1,53 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import subprocess + + +def launch_service(service_name: str): + try: + process = subprocess.Popen(["cogment", "services", service_name]) + logging.info(f"{service_name} launched successfully. PID: {process.pid}") + return process + except Exception as e: + logging.error(f"Failed to launch {service_name}: {e}") + return None + + +def launch_main(command: str): + services = command.split() + processes = [] + + if "base" in services or "all" in services: + if len(services) > 1: + raise ValueError("Cannot combine 'base' or 'all' with other services") + + if "base" in services: + services_to_run = ["orchestrator", "trial_datastore"] + elif "all" in services: + services_to_run = ["orchestrator", "trial_datastore", "model_registry", "directory", "web_proxy"] + else: + services_to_run = services + + for service in services_to_run: + process = launch_service(service) + if process: + processes.append(process) + + # Optional: Wait for all subprocesses to complete + for process in processes: + process.wait() + + return processes diff --git a/cogment_lab/constants.py b/cogment_lab/constants.py new file mode 100644 index 0000000..b6e9d85 --- /dev/null +++ b/cogment_lab/constants.py @@ -0,0 +1,15 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +DEFAULT_RENDERED_WIDTH = 1024 diff --git a/cogment_lab/core.py b/cogment_lab/core.py new file mode 100644 index 0000000..df71bc1 --- /dev/null +++ b/cogment_lab/core.py @@ -0,0 +1,269 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import abc +import copy +import logging +from typing import Awaitable, Callable, Generic, TypeVar + +import cogment +import numpy as np +from cogment.actor import ActorSession +from cogment.environment import EnvironmentSession +from cogment.session import RecvEvent + +from cogment_lab.session_helpers import ActorSessionHelper, EnvironmentSessionHelper +from cogment_lab.specs import ( + AgentSpecs +) + +Action = TypeVar("Action") +Actions = dict[str, Action] + +Observation = TypeVar("Observation") +Observations = dict[str, Observation] + +Rewards = dict[str, float] + +Dones = dict[str, bool] + + +class State: + """Class to hold state information.""" + + def __setattr__(self, name, value): + self.__dict__[name] = value + + def __getattr__(self, name): + """Get attribute from internal dictionary.""" + return self.__dict__.get(name, None) + + +class BaseEnv(abc.ABC): + """Base environment class.""" + + agent_specs: dict[str, AgentSpecs] + + def __init__(self, *args, **kwargs): + """Initialize with arguments.""" + self.args = copy.deepcopy(args) + self.kwargs = copy.deepcopy(kwargs) + + async def impl(self, environment_session: EnvironmentSession): + """Abstract method to implement environment logic.""" + raise NotImplementedError() + + def get_constructor(self): + """Get constructor for this environment.""" + cls = self.__class__ + return lambda: cls(*self.args, **self.kwargs) + + +class CogmentEnv(BaseEnv, abc.ABC, Generic[Observation, Action]): + """Base Cogment environment class.""" + + environment_session: EnvironmentSession + session_helper: EnvironmentSessionHelper + agent_specs: dict[str, AgentSpecs] + actor_name: str + + def __init__(self, *args, **kwargs): + """Initialize.""" + super().__init__(*args, **kwargs) + + async def initialize(self, state: State, environment_session: EnvironmentSession): + """Initialize state and session.""" + state.environment_session = environment_session + return state + + @abc.abstractmethod + async def reset(self, state: State) -> tuple[State, Observations]: + """Reset environment state. + + Returns: + State: Updated state. + Observations: Initial observations. + """ + raise NotImplementedError() + + @abc.abstractmethod + async def step(self, state: State, action: Actions) -> tuple[State, Observations, Rewards, Dones, Dones, dict]: + """Take a step in the environment. + + Args: + state: Current state. + action: Actions from actors. + + Returns: + State: Updated state. + Observations: New observations. + Rewards: Rewards for each actor. + Dones: Whether each actor is done. + Dones: Whether each actor is truncated. + dict: Additional info. + """ + raise NotImplementedError() + + async def end(self, state: State): + """Clean up when done.""" + pass + + async def read_actions(self, state: State, event: RecvEvent): + """Read actions from event.""" + player_action = state.session_helper.get_action(event, state.actor_name) + return player_action.value + + async def impl(self, environment_session: EnvironmentSession): + """Implement environment logic.""" + state = State() + state = await self.initialize(state, environment_session) + state, observations = await self.reset(state) + + observations = list(observations.items()) + + logging.info(f"Starting environment session") + + environment_session.start(observations) + + async for event in environment_session.all_events(): + event: RecvEvent + if event.actions: + actions = await self.read_actions(state, event) + state, observations, rewards, terminateds, truncateds, info = await self.step(state, actions) + + dones = {actor_name: terminateds[actor_name] or truncateds[actor_name] for actor_name in terminateds} + + logging.info(f"Adding rewards: {rewards}") + for actor_name in state.actors: + if actor_name not in rewards: + rewards[actor_name] = float("nan") + for actor_name, reward in rewards.items(): + environment_session.add_reward(value=reward, to=[actor_name], confidence=1.0) + + observations = list(observations.items()) + + if all(dones.values()): + logging.info(f"Logging dones=True") + environment_session.end(observations) + # elif event.type != cogment.EventType.ACTIVE: + # logging.info("Logging event.type!=ACTIVE") + # environment_session.end(observations) + else: + logging.info(f"Logging a normal observation") + environment_session.produce_observations(observations) + + await self.end(state) + + +class BaseActor(abc.ABC): + """Base actor class.""" + + def __init__(self, *args, **kwargs): + """Initialize.""" + self.args = args + self.kwargs = kwargs + + async def impl(self, actor_session: ActorSession): + """Abstract method to implement actor logic.""" + raise NotImplementedError() + + +class NativeActor(BaseActor): + """Native actor wrapping a function.""" + + def __init__(self, impl: Callable[[ActorSession], Awaitable]): + """Initialize with implementation function.""" + super().__init__(impl) + self._impl = impl + + async def impl(self, actor_session: ActorSession): + """Call implementation function.""" + await self._impl(actor_session) + + +class CogmentActor(BaseActor, abc.ABC, Generic[Observation, Action]): + """Base Cogment actor class.""" + + actor_session: ActorSession + current_event: RecvEvent + session_helper: ActorSessionHelper + + def __init__(self, *args, **kwargs): + """Initialize.""" + super().__init__(*args, **kwargs) + + async def initialize(self, actor_session: ActorSession): + """Initialize session and helpers.""" + self.actor_session = actor_session + self.actor_session.start() + + self.session_helper = ActorSessionHelper(actor_session, None) + self.action_space = self.session_helper.get_action_space() + + @abc.abstractmethod + async def act(self, observation: Observation, rendered_frame: np.ndarray | None = None) -> Action: + """Choose an action based on observation. + + Args: + observation: Current observation. + rendered_frame: Optional rendered frame. + + Returns: + Action to take. + """ + raise NotImplementedError() + + async def on_reward(self, rewards: list): + """Handle received rewards.""" + pass + + async def on_message(self, messages: list): + """Handle received messages.""" + pass + + async def end(self): + """Clean up when done.""" + pass + + async def impl(self, actor_session: ActorSession): + """Implement actor logic.""" + await self.initialize(actor_session) + async for event in actor_session.all_events(): + event: RecvEvent + self.current_event = event + if event.type != cogment.EventType.ACTIVE: + logging.info(f"Skipping event of type {event.type}") + continue + + if event.observation: + observation = self.session_helper.get_observation(event) + logging.info(f"Got observation: {observation}") + + if not observation.active: + action = None + elif not observation.alive: + action = None + else: + action = await self.act(observation.value, observation.rendered_frame) + logging.info(f"Got action: {action} with action_space: {self.action_space.gym_space}") + cog_action = self.action_space.create_serialize(action) + actor_session.do_action(cog_action) + if event.rewards: + await self.on_reward(event.rewards) + if event.messages: + await self.on_message(event.messages) + + await self.end() diff --git a/cogment_lab/envs/__init__.py b/cogment_lab/envs/__init__.py new file mode 100644 index 0000000..4f53414 --- /dev/null +++ b/cogment_lab/envs/__init__.py @@ -0,0 +1,16 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from cogment_lab.envs.gymnasium import GymEnvironment +from cogment_lab.envs.pettingzoo import AECEnvironment diff --git a/cogment_lab/envs/configs.py b/cogment_lab/envs/configs.py new file mode 100644 index 0000000..e087942 --- /dev/null +++ b/cogment_lab/envs/configs.py @@ -0,0 +1,65 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Module for environment configuration. + +Defines data structures for specifying environment configurations. +""" + +from __future__ import annotations + +from typing import Any, TypedDict + + +class EnvConfig(TypedDict, total=False): + """ + Configuration for a single environment. + + Attributes: + env_type: Type of the environment. + env_name: Name of the environment. + cogment_env: Cogment environment ID to use, instead of env_id/registration/make_args. + env_id: Environment ID. + registration: Environment registration details. + make_args: Arguments for make() to construct the environment. + reset_options: Reset options for the environment. + render: Whether to render the environment. + """ + + env_type: str + env_name: str + + # Either this... + cogment_env: str | None + + # ...or all of this + env_id: str | None + registration: str | None + make_args: dict[str, Any] + reset_options: dict[str, Any] + render: bool + + +class EnvRunnerConfig(TypedDict): + """ + Configuration for an environment runner. + + Attributes: + envs: List of EnvConfig, one for each environment. + port: Port number for the runner. + """ + + envs: list[EnvConfig] + port: int diff --git a/cogment_lab/envs/conversions/__init__.py b/cogment_lab/envs/conversions/__init__.py new file mode 100644 index 0000000..08d7dad --- /dev/null +++ b/cogment_lab/envs/conversions/__init__.py @@ -0,0 +1,14 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + diff --git a/cogment_lab/envs/conversions/observer.py b/cogment_lab/envs/conversions/observer.py new file mode 100644 index 0000000..5f8a66f --- /dev/null +++ b/cogment_lab/envs/conversions/observer.py @@ -0,0 +1,157 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from typing import Any + +import gymnasium as gym +from pettingzoo import AECEnv, ParallelEnv +from pettingzoo.utils.agent_selector import agent_selector + + +class GymObserverAEC(AECEnv): + metadata = {"render_modes": ["rgb_array"], "name": "GymWithObserverEnv"} + + def __init__(self, gym_env_name: str, gym_make_kwargs: dict = {}, render_mode: str | None = None): + super().__init__() + logging.info( + f"Creating GymObserverAEC with gym_env_name={gym_env_name}, gym_make_kwargs={gym_make_kwargs}, render_mode={render_mode}" + ) + self.gym_env = gym.make(gym_env_name, render_mode=render_mode, **gym_make_kwargs) + self.possible_agents = ["gym", "observer"] + self.observation_spaces = { + "gym": self.gym_env.observation_space, + "observer": gym.spaces.Dict( + {"observation": self.gym_env.observation_space, "action": self.gym_env.action_space} + ), + } + self.action_spaces = { + "gym": self.gym_env.action_space, + "observer": self.gym_env.action_space, # Even though actions are ignored + } + self._agent_selector = agent_selector(self.possible_agents) + # self.reset() + + def reset(self, seed: int | None = None, options: dict | None = None): + logging.info(f"Resetting GymObserverAEC with seed={seed}, options={options}") + self.agents = self.possible_agents[:] + self.agent_selection = self._agent_selector.reset() + self._cumulative_rewards = {agent: 0 for agent in self.agents} + self.rewards = {agent: 0 for agent in self.agents} + self.terminations = {agent: False for agent in self.agents} + self.truncations = {agent: False for agent in self.agents} + self._last_observation, info = self.gym_env.reset() + self.infos = {"gym": info, "observer": info} + self._gym_action = None + + def step(self, action): + logging.info(f"Stepping GymObserverAEC with action={action}") + current_agent = self.agent_selection + + if current_agent == "gym": + # Execute main agent's action directly + observation, reward, terminated, truncated, info = self.gym_env.step(action) + self._last_observation = observation + self.rewards["gym"] = float(reward) + self.terminations["gym"] = terminated + self.truncations["gym"] = truncated + self.infos["gym"] = info + + self.rewards["observer"] = self.rewards["gym"] + self.terminations["observer"] = self.terminations["gym"] + self.truncations["observer"] = self.truncations["gym"] + self.infos["observer"] = self.infos["gym"] + + self._gym_action = action + + # Observer agent's turn; actions are ignored + elif current_agent == "observer": + pass + + self.agent_selection = self._agent_selector.next() + + def observe(self, agent): + if agent == "gym": + return self._last_observation + elif agent == "observer": + return {"observation": self._last_observation, "action": self._gym_action} + + def render(self): + return self.gym_env.render() + + def close(self): + self.gym_env.close() + + def observation_space(self, agent: str): + return self.observation_spaces[agent] + + def action_space(self, agent: str): + return self.action_spaces[agent] + +class GymObserverParallel(ParallelEnv): + metadata = {"render_modes": ["rgb_array"], "name": "GymWithObserverEnv"} + + def __init__(self, gym_env_name: str, gym_make_kwargs: dict = {}, render_mode: str | None = None): + super().__init__() + logging.info( + f"Creating GymObserverParallel with gym_env_name={gym_env_name}, gym_make_kwargs={gym_make_kwargs}, render_mode={render_mode}" + ) + self.gym_env = gym.make(gym_env_name, render_mode=render_mode, **gym_make_kwargs) + self.possible_agents = ["gym", "observer"] + self.observation_spaces = { + "gym": self.gym_env.observation_space, + "observer": self.gym_env.observation_space + } + self.action_spaces = { + "gym": self.gym_env.action_space, + "observer": self.gym_env.action_space, # Even though actions are ignored + } + # self.reset() + + def reset(self, seed: int | None = None, options: dict | None = None): + logging.info(f"Resetting GymObserverParallel with seed={seed}, options={options}") + self.agents = self.possible_agents[:] + + obs, info = self.gym_env.reset(seed=seed, options=options) + + infos = {"gym": info, "observer": info} + observations = {"gym": obs, "observer": obs} + + return observations, infos + + + def step(self, action: dict[str, Any]): + logging.info(f"Stepping GymObserverParallel with action={action}") + + obs, reward, terminated, truncated, info = self.gym_env.step(action["gym"]) + + observations = {"gym": obs, "observer": obs} + rewards = {"gym": reward, "observer": reward} + terminations = {"gym": terminated, "observer": terminated} + truncations = {"gym": truncated, "observer": truncated} + infos = {"gym": info, "observer": info} + + return observations, rewards, terminations, truncations, infos + + def render(self): + return self.gym_env.render() + + def close(self): + self.gym_env.close() + + def observation_space(self, agent: str): + return self.observation_spaces[agent] + + def action_space(self, agent: str): + return self.action_spaces[agent] diff --git a/cogment_lab/envs/conversions/teacher.py b/cogment_lab/envs/conversions/teacher.py new file mode 100644 index 0000000..abd7eb7 --- /dev/null +++ b/cogment_lab/envs/conversions/teacher.py @@ -0,0 +1,181 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import gymnasium as gym +from pettingzoo import AECEnv, ParallelEnv +from pettingzoo.utils.agent_selector import agent_selector +import warnings +import numpy as np + + +class GymTeacherAEC(AECEnv): + metadata = {"render_modes": ["rgb_array"]} + + def __init__(self, gym_env_name: str, gym_make_kwargs: dict = {}, render_mode: str | None = None): + super().__init__() + self.gym_env = gym.make(gym_env_name, render_mode=render_mode, **gym_make_kwargs) + self.possible_agents = ["gym", "teacher"] + self.observation_spaces = { + "gym": self.gym_env.observation_space, + "teacher": gym.spaces.Dict( + {"observation": self.gym_env.observation_space, "action": self.gym_env.action_space} + ), + } + teacher_action_space = gym.spaces.Dict({"active": gym.spaces.Discrete(2), "action": self.gym_env.action_space}) + self.action_spaces = {"gym": self.gym_env.action_space, "teacher": teacher_action_space} + self._agent_selector = agent_selector(self.possible_agents) + self.override = False + # self.reset() + + def reset(self, seed: int | None = None, options: dict | None = None): + self.agents = self.possible_agents[:] + self.agent_selection = self._agent_selector.reset() + self._cumulative_rewards = {agent: 0 for agent in self.agents} + self.rewards = {agent: 0 for agent in self.agents} + self.terminations = {agent: False for agent in self.agents} + self.truncations = {agent: False for agent in self.agents} + self._last_observation, info = self.gym_env.reset(seed=seed, options=options) + self.infos = {"gym": info, "teacher": info} + self._gym_action = None + + def step(self, action): + current_agent = self.agent_selection + next_agent = self._agent_selector.next() + + if current_agent == "gym": + self._cumulative_rewards["gym"] = 0 + self._cumulative_rewards["teacher"] = 0 + + self._gym_action = action + + elif current_agent == "teacher": + if action["active"] == 1: + self.override = True + real_action = action["action"] + else: + self.override = False + real_action = self._gym_action + observation, reward, terminated, truncated, info = self.gym_env.step(real_action) + self._last_observation = observation + self.rewards["gym"] = float(reward) + self.rewards["teacher"] = float(reward) + + self.terminations["gym"] = terminated + self.terminations["teacher"] = terminated + + self.truncations["gym"] = truncated + self.truncations["teacher"] = truncated + + self.infos["gym"] = info + self.infos["teacher"] = info + + self._accumulate_rewards() + self.agent_selection = next_agent + + def observe(self, agent): + if agent == "gym": + return self._last_observation + elif agent == "teacher": + return {"observation": self._last_observation, "action": self._gym_action} + + def render(self): + img = self.gym_env.render() + if self.override: + W, H, _ = img.shape + N = 5 + + # Set the borders to red + img[:N, :, :] = [255, 0, 0] # Top border + img[-N:, :, :] = [255, 0, 0] # Bottom border + img[:, :N, :] = [255, 0, 0] # Left border + img[:, -N:, :] = [255, 0, 0] # Right border + return img + + def close(self): + self.gym_env.close() + + def observation_space(self, agent: str): + return self.observation_spaces[agent] + + def action_space(self, agent: str): + return self.action_spaces[agent] + + +class GymTeacherParallel(ParallelEnv): + metadata = {"render_modes": ["rgb_array"]} + + def __init__(self, gym_env_name: str, gym_make_kwargs: dict = {}, render_mode: str | None = None): + super().__init__() + self.gym_env = gym.make(gym_env_name, render_mode=render_mode, **gym_make_kwargs) + self.possible_agents = ["gym", "teacher"] + self.observation_spaces = { + "gym": self.gym_env.observation_space, + "teacher": self.gym_env.observation_space + } + + teacher_action_space = gym.spaces.Dict({"active": gym.spaces.Discrete(2), "action": self.gym_env.action_space}) + self.action_spaces = {"gym": self.gym_env.action_space, "teacher": teacher_action_space} + self._agent_selector = agent_selector(self.possible_agents) + self.override = False + # self.reset() + + def reset(self, seed: int | None = None, options: dict | None = None): + self.agents = self.possible_agents[:] + + obs, info = self.gym_env.reset(seed=seed, options=options) + + infos = {"gym": info, "teacher": info} + observations = {"gym": obs, "teacher": obs} + + return observations, infos + + def step(self, action): + if action["teacher"]["active"] == 1: + self.override = True + real_action = action["teacher"]["action"] + else: + self.override = False + real_action = action["gym"] + + observation, reward, terminated, truncated, info = self.gym_env.step(real_action) + + observations = {"gym": observation, "teacher": observation} + rewards = {"gym": reward, "teacher": reward} + terminations = {"gym": terminated, "teacher": terminated} + truncations = {"gym": truncated, "teacher": truncated} + infos = {"gym": info, "teacher": info} + + return observations, rewards, terminations, truncations, infos + + def render(self): + img = self.gym_env.render() + if self.override: + W, H, _ = img.shape + N = 5 + + # Set the borders to red + img[:N, :, :] = [255, 0, 0] # Top border + img[-N:, :, :] = [255, 0, 0] # Bottom border + img[:, :N, :] = [255, 0, 0] # Left border + img[:, -N:, :] = [255, 0, 0] # Right border + return img + + def close(self): + self.gym_env.close() + + def observation_space(self, agent: str): + return self.observation_spaces[agent] + + def action_space(self, agent: str): + return self.action_spaces[agent] diff --git a/cogment_lab/envs/gymnasium.py b/cogment_lab/envs/gymnasium.py new file mode 100644 index 0000000..59a6eb1 --- /dev/null +++ b/cogment_lab/envs/gymnasium.py @@ -0,0 +1,265 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import importlib +import logging +import os +from typing import Any, Callable + +import gymnasium as gym +from cogment.environment import EnvironmentSession + +from cogment_lab.core import CogmentEnv, State +from cogment_lab.session_helpers import EnvironmentSessionHelper +from cogment_lab.specs import AgentSpecs + +log = logging.getLogger(__name__) + +# configure pygame to use a dummy video server to be able to render headlessly +os.environ["SDL_VIDEODRIVER"] = "dummy" + + +class GymEnvironment(CogmentEnv): + """ + Gymnasium integration for Cogment. + + Exposes a Gymnasium environment as a Cogment environment. + """ + + session_helper: EnvironmentSessionHelper + actor_name = "gym" + + def __init__( + self, + env_id: str | Callable[..., gym.Env], + registration: str | None = None, + make_kwargs: dict[str, Any] | None = None, + reset_options: dict[str, Any] | None = None, + render: bool = False, + reinitialize: bool = False, + dry: bool = False, + sub_dry: bool = True, + ): + """ + Initialize the GymEnvironment. + + Args: + env_id: The Gym environment ID. + registration: Optional Gym registration string. + make_kwargs: Optional args to pass to gym.make(). + reset_options: Optional reset options to pass to env.reset(). + render: Whether to render the environment. + reinitialize: Whether to reinitialize the environment each session. + dry: Whether to abstain from initializing the environment in this process. + sub_dry: Whether to abstain from initializing the environment in the initializer of the subprocess. + """ + super().__init__( + env_id=env_id, + registration=registration, + make_kwargs=make_kwargs, + reset_options=reset_options, + render=render, + reinitialize=reinitialize, + dry=sub_dry, + sub_dry=sub_dry, + ) + + self.env_id = env_id + self.registration = registration + self.make_kwargs = make_kwargs or {} + self.reset_options = reset_options or {} + self.render = render + self.reinitialize = reinitialize + self.dry = dry + + if "render_mode" in self.make_kwargs: + raise ValueError("render_mode cannot be set in make_kwargs") + + if self.render: + self.make_kwargs["render_mode"] = "rgb_array" + + if isinstance(self.env_id, Callable): + self.env_maker = self.env_id + else: + self.env_maker = lambda **kwargs: gym.make(self.env_id, **kwargs) + + if self.registration: + importlib.import_module(self.registration) + + if not self.dry: + self.env = self.env_maker(**self.make_kwargs) + + self.agent_specs = { + "gym": AgentSpecs.create_homogeneous( + observation_space=self.env.observation_space, + action_space=self.env.action_space, + ) + } + + self.initialized = True + else: + self.env = None + self.agent_specs = {} + self.initialized = False + + self._is_closed = False + + def get_implementation_name(self): + """ + Get the name of the Gym environment. + + Returns: + The Gym environment ID. + """ + return self.env_id + + def get_agent_specs(self): + """ + Get the agent specs. + + Returns: + The agent specs dict. + """ + return self.agent_specs + + async def initialize(self, state: State, environment_session: EnvironmentSession): + """ + Initialize the environment session. + + Args: + state: The Cogment state. + environment_session: The Cogment environment session. + + Returns: + The updated state. + """ + + logging.info("Initializing environment session") + + if not self.initialized and not self.reinitialize: + self.env = self.env_maker(**self.make_kwargs) + self.agent_specs = { + "gym": AgentSpecs.create_homogeneous( + observation_space=self.env.observation_space, + action_space=self.env.action_space, + ) + } + + state.env = self.env + state.agent_specs = self.agent_specs + elif self.initialized and not self.reinitialize: + state.env = self.env + state.agent_specs = self.agent_specs + elif self.reinitialize: + state.env = self.env_maker(**self.make_kwargs) + state.agent_specs = { + "gym": AgentSpecs.create_homogeneous( + observation_space=state.env.observation_space, + action_space=state.env.action_space, + ) + } + + state.environment_session = environment_session + state.session_helper = EnvironmentSessionHelper(environment_session, state.agent_specs) + state.session_cfg = state.environment_session.config + state.actors = state.session_helper.actors + state.actor_name = state.session_helper.actors[0] + + self.initialized = True + + return state + + async def reset(self, state: State): + """ + Reset the environment. + + Args: + state: The Cogment state. + + Returns: + A tuple with the updated state and a dict of observations. + """ + + logging.info("Resetting environment") + + obs, _info = state.env.reset(seed=state.session_cfg.seed, options=state.session_cfg.reset_args) # THIS + + state.observation_space = state.session_helper.get_observation_space(self.actor_name) + frame = state.env.render() if state.session_cfg.render else None + observation = state.observation_space.create_serialize(value=obs, rendered_frame=frame, active=True, alive=True) + + return state, {"*": observation} + + async def read_actions(self, state: State, event): + """ + Read the agent action from the event. + + Args: + state: The Cogment state. + event: The event from Cogment. + + Returns: + The agent action value. + """ + player_action = state.session_helper.get_action(tick_data=event, actor_name=self.actor_name) + return player_action.value + + async def step(self, state: State, action): + """ + Step the environment. + + Args: + state: The Cogment state. + action: The agent action. + + Returns: + A tuple with the updated state, observations dict, rewards dict, + terminateds dict, truncateds dict, and info dict. + """ + + logging.info("Stepping environment") + + obs, reward, terminated, truncated, info = state.env.step(action) + logging.info(f"Step returned {obs=}, {reward=}, {terminated=}, {truncated=}, {info=}") + + observation = state.observation_space.create_serialize( + value=obs, rendered_frame=state.env.render() if state.session_cfg.render else None, active=True, alive=True + ) + + # observations = [("*", observation)] + observations = {"*": observation} + rewards = {self.actor_name: reward} + terminateds = {self.actor_name: terminated} + truncateds = {self.actor_name: truncated} + + return state, observations, rewards, terminateds, truncateds, info + + async def end(self, state: State): + """ + End the environment session. + + Args: + state: The Cogment state. + + Returns: + The updated state. + """ + + logging.info("Ending environment") + state.env.close() + if self.env is not None and not self._is_closed: + self.env.close() + self._is_closed = True diff --git a/cogment_lab/envs/pettingzoo.py b/cogment_lab/envs/pettingzoo.py new file mode 100644 index 0000000..597910d --- /dev/null +++ b/cogment_lab/envs/pettingzoo.py @@ -0,0 +1,526 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from typing import TypedDict, Any + +import numpy as np +from cogment.session import RecvEvent +from pettingzoo import AECEnv, ParallelEnv + +from cogment.environment import EnvironmentSession + +from cogment_lab.core import CogmentEnv, State +from cogment_lab.session_helpers import EnvironmentSessionHelper +from cogment_lab.specs import AgentSpecs +from cogment_lab.specs.observation_space import ObservationSpace +from cogment_lab.utils import import_object +from cogment_lab.generated.data_pb2 import ( + Observation as PbObservation, +) +import pettingzoo + + +class PZConfig(TypedDict): + """Configuration for a PettingZoo environment.""" + + env_path: str + make_args: dict + reset_options: dict + + +class AECEnvironment(CogmentEnv): + """Cogment environment wrapper for PettingZoo AEC environments.""" + + def __init__( + self, + env_path: str, + make_kwargs: dict | None = None, + reset_options: dict = None, + render: bool = False, + reinitialize: bool = False, + dry: bool = False, + sub_dry: bool = True, + ): + """ + Initialize the AECEnvironment. + + Args: + env_path: Path to the PettingZoo environment class. + make_kwargs: Arguments to pass to the environment constructor. + reset_options: Options to pass to reset(). + render: Whether to render the environment. + reinitialize: Whether to reinitialize the environment each session. + dry: Whether to abstain from initializing the environment in this process. + sub_dry: Whether to abstain from initializing the environment in the initializer of the subprocess + """ + super().__init__( + env_path=env_path, + make_kwargs=make_kwargs, + reset_options=reset_options, + render=render, + reinitialize=reinitialize, + dry=sub_dry, + sub_dry=sub_dry, + ) + self.env_path = env_path + self.make_args = make_kwargs or {} + self.reset_options = reset_options or {} + self.render = render + self.reinitialize = reinitialize + self.dry = dry + + logging.info( + f"Creating AECEnvironment with {env_path=}, {make_kwargs=}, {reset_options=}, {render=}, {reinitialize=}, {dry=}, {sub_dry=}" + ) + + self.env_maker = import_object(self.env_path) + + assert callable(self.env_maker), f"Environment class at {self.env_path} is not callable" + + if render: + self.make_args["render_mode"] = "rgb_array" + + if not self.dry: + self.env: AECEnv = self.env_maker(**self.make_args) + self.agent_specs = self.create_agent_specs(self.env) + else: + self.env = None + self.agent_specs = {} + + self.initialized = False + + async def initialize(self, state: State, environment_session: EnvironmentSession): + """ + Initialize the environment session. + + Args: + state: The Cogment state. + environment_session: The Cogment environment session. + + Returns: + The updated state. + """ + logging.info("Initializing environment session") + if not self.initialized and not self.reinitialize: + self.env = self.env_maker(**self.make_args) + self.agent_specs = self.create_agent_specs(self.env) + + state.env = self.env + state.agent_specs = self.agent_specs + elif self.initialized and not self.reinitialize: + state.env = self.env + state.agent_specs = self.agent_specs + elif self.reinitialize: + state.env = self.env_maker(**self.make_args) + state.agent_specs = self.create_agent_specs(state.env) + + self.initialized = True + + state.environment_session = environment_session + state.session_helper = EnvironmentSessionHelper(environment_session, state.agent_specs) + state.session_cfg = state.environment_session.config + state.actors = state.session_helper.actors + + state.observation_spaces = {agent: state.session_helper.get_observation_space(agent) for agent in state.actors} + + return state + + async def reset(self, state: State): + """ + Reset the environment. + + Args: + state: The Cogment state. + + Returns: + A tuple of (state, observations) + """ + logging.info("Resetting environment") + + state.env.reset(seed=state.session_cfg.seed) + + obs, reward, term, trunc, info = state.env.last() + agent = state.env.agent_selection + + state.actor_name = agent + + frame = state.env.render() if state.session_cfg.render else None + logging.info(f"Creating observation from {obs=}") + observation = state.observation_spaces[agent].create_serialize( + value=obs, rendered_frame=frame, active=True, alive=True + ) + if frame is not None: + logging.info(f"Frame shape at reset: {frame.shape}") + else: + logging.info("Frame at reset is None") + + observations = {agent: observation} + observations = self.fill_observations_(state=state, observations=observations, frame=frame) + + return state, observations + + async def step(self, state: State, action: Any): + """ + Take an environment step. + + Args: + state: The Cogment state. + action: The action to take. + + Returns: + A tuple of (state, observations, rewards, terminated, truncated, info) + """ + logging.info("Stepping environment") + + state.env.step(action) + obs, reward, terminated, truncated, info = state.env.last() + agent = state.env.agent_selection + + frame = state.env.render() if state.session_cfg.render else None + + if frame is not None: + logging.info(f"Frame shape at step: {frame.shape}") + else: + logging.info("Frame at step is None") + + observation = state.observation_spaces[agent].create_serialize( + value=obs, + rendered_frame=frame, + active=True, + alive=not (terminated or truncated), + ) + + # observations = [(agent, observation)] + observations = {agent: observation} + rewards = {agent: reward} + terminateds = state.env.terminations + truncateds = state.env.truncations + + state.actor_name = agent + + observations = self.fill_observations_(state, observations, frame=frame) + + return state, observations, rewards, terminateds, truncateds, info + + async def end(self, state: State): + """ + End the environment session. + + Args: + state: The Cogment state. + """ + logging.info("Ending environment") + state.env.close() + + @staticmethod + def fill_observations_( + state: State, observations: dict[str, PbObservation], frame: np.ndarray + ) -> dict[str, PbObservation]: + """ + Fill in any missing observations with the default observation. Mutates the observations dict. + + Args: + state: The Cogment state. + observations: The observations dict. + + Returns: + The filled observations dict. + """ + if "*" in observations: + return observations + for actor_name in state.actors: + if actor_name not in observations: + observations[actor_name] = state.observation_spaces[actor_name].create_serialize( + rendered_frame=frame, active=False + ) + + return observations + + @staticmethod + def create_agent_specs(env: AECEnv): + """ + Create the agent specs from a PettingZoo AEC environment. + + Args: + env: The PettingZoo AEC environment. + + Returns: + The agent specs dict. + """ + num_agents = len(env.possible_agents) + + # Check all observation and action spaces + + is_homogeneous = True + observation_space = env.observation_space(env.possible_agents[0]) + action_space = env.action_space(env.possible_agents[0]) + for agent in env.possible_agents: + if env.observation_space(agent) != observation_space: + is_homogeneous = False + break + if env.action_space(agent) != action_space: + is_homogeneous = False + break + + if is_homogeneous: + one_agent_specs = AgentSpecs.create_homogeneous( + observation_space=observation_space, + action_space=action_space, + ) + agent_specs = {agent: one_agent_specs for agent in env.possible_agents} + else: + agent_specs = { + agent: AgentSpecs.create_homogeneous( + observation_space=env.observation_space(agent), + action_space=env.action_space(agent), + ) + for agent in env.possible_agents + } + + return agent_specs + + +class ParallelEnvironment(CogmentEnv): + """Cogment environment wrapper for PettingZoo Parallel environments.""" + + def __init__( + self, + env_path: str, + make_kwargs: dict | None = None, + reset_options: dict = None, + render: bool = False, + reinitialize: bool = False, + dry: bool = False, + sub_dry: bool = True, + ): + """ + Initialize the AECEnvironment. + + Args: + env_path: Path to the PettingZoo environment class. + make_kwargs: Arguments to pass to the environment constructor. + reset_options: Options to pass to reset(). + render: Whether to render the environment. + reinitialize: Whether to reinitialize the environment each session. + dry: Whether to abstain from initializing the environment in this process. + sub_dry: Whether to abstain from initializing the environment in the initializer of the subprocess + """ + super().__init__( + env_path=env_path, + make_kwargs=make_kwargs, + reset_options=reset_options, + render=render, + reinitialize=reinitialize, + dry=sub_dry, + sub_dry=sub_dry, + ) + self.env_path = env_path + self.make_args = make_kwargs or {} + self.reset_options = reset_options or {} + self.render = render + self.reinitialize = reinitialize + self.dry = dry + + logging.info( + f"Creating ParallelEnvironment with {env_path=}, {make_kwargs=}, {reset_options=}, {render=}, {reinitialize=}, {dry=}, {sub_dry=}" + ) + + self.env_maker = import_object(self.env_path) + + assert callable(self.env_maker), f"Environment class at {self.env_path} is not callable" + + if render: + self.make_args["render_mode"] = "rgb_array" + + if not self.dry: + self.env: ParallelEnv = self.env_maker(**self.make_args) + self.agent_specs = self.create_agent_specs(self.env) + else: + self.env = None + self.agent_specs = {} + + self.initialized = False + + async def initialize(self, state: State, environment_session: EnvironmentSession): + """ + Initialize the environment session. + + Args: + state: The Cogment state. + environment_session: The Cogment environment session. + + Returns: + The updated state. + """ + logging.info("Initializing environment session") + if not self.initialized and not self.reinitialize: + self.env: ParallelEnv = self.env_maker(**self.make_args) + self.agent_specs = self.create_agent_specs(self.env) + + state.env = self.env + state.agent_specs = self.agent_specs + elif self.initialized and not self.reinitialize: + state.env: ParallelEnv = self.env + state.agent_specs = self.agent_specs + elif self.reinitialize: + state.env: ParallelEnv = self.env_maker(**self.make_args) + state.agent_specs = self.create_agent_specs(state.env) + + self.initialized = True + + state.environment_session = environment_session + state.session_helper = EnvironmentSessionHelper(environment_session, state.agent_specs) + state.session_cfg = state.environment_session.config + state.actors = state.session_helper.actors + + state.observation_spaces = {agent: state.session_helper.get_observation_space(agent) for agent in state.actors} + + return state + + async def reset(self, state: State): + """ + Reset the environment. + + Args: + state: The Cogment state. + + Returns: + A tuple of (state, observations) + """ + logging.info("Resetting environment") + + obs, info = state.env.reset(seed=state.session_cfg.seed) + + frame = state.env.render() if state.session_cfg.render else None + logging.info(f"Creating observation from {obs=}") + + + + if frame is not None: + logging.info(f"Frame shape at reset: {frame.shape}") + else: + logging.info("Frame at reset is None") + + observations = { + agent: + state.observation_spaces[agent].create_serialize( + value=obs[agent], rendered_frame=frame, active=True, alive=True + ) + for agent in obs + } + + # observations = {agent: observation} + # observations = self.fill_observations_(state=state, observations=observations, frame=frame) + + return state, observations + + async def step(self, state: State, action: dict[str, Any]): + """ + Take an environment step. + + Args: + state: The Cogment state. + action: The action to take. + + Returns: + A tuple of (state, observations, rewards, terminated, truncated, info) + """ + logging.info("Stepping environment") + + state.env.step(action) + obs, rewards, terminated, truncated, info = state.env.step(action) + + frame = state.env.render() if state.session_cfg.render else None + + if frame is not None: + logging.info(f"Frame shape at step: {frame.shape}") + else: + logging.info("Frame at step is None") + + + observations = { + agent: + state.observation_spaces[agent].create_serialize( + value=obs[agent], + rendered_frame=frame, + active=True, + alive=not (terminated[agent] or truncated[agent]) + ) + for agent in obs + } + + + return state, observations, rewards, terminated, truncated, info + + async def end(self, state: State): + """ + End the environment session. + + Args: + state: The Cogment state. + """ + logging.info("Ending environment") + state.env.close() + + + async def read_actions(self, state: State, event: RecvEvent): + """Read actions from event.""" + player_actions = {agent: state.session_helper.get_action(event, agent).value + for agent in state.actors} + return player_actions + + + @staticmethod + def create_agent_specs(env: ParallelEnv): + """ + Create the agent specs from a PettingZoo AEC environment. + + Args: + env: The PettingZoo AEC environment. + + Returns: + The agent specs dict. + """ + num_agents = len(env.possible_agents) + + # Check all observation and action spaces + + is_homogeneous = True + observation_space = env.observation_space(env.possible_agents[0]) + action_space = env.action_space(env.possible_agents[0]) + for agent in env.possible_agents: + if env.observation_space(agent) != observation_space: + is_homogeneous = False + break + if env.action_space(agent) != action_space: + is_homogeneous = False + break + + if is_homogeneous: + one_agent_specs = AgentSpecs.create_homogeneous( + observation_space=observation_space, + action_space=action_space, + ) + agent_specs = {agent: one_agent_specs for agent in env.possible_agents} + else: + agent_specs = { + agent: AgentSpecs.create_homogeneous( + observation_space=env.observation_space(agent), + action_space=env.action_space(agent), + ) + for agent in env.possible_agents + } + + return agent_specs diff --git a/cogment_lab/envs/runner.py b/cogment_lab/envs/runner.py new file mode 100644 index 0000000..862c06e --- /dev/null +++ b/cogment_lab/envs/runner.py @@ -0,0 +1,74 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Registers an environment with Cogment and runs the Cogment server""" + +from __future__ import annotations + +import asyncio +import logging +from multiprocessing import Queue + +import cogment + +from cogment_lab.core import BaseEnv +from cogment_lab.generated import cog_settings +from cogment_lab.utils.runners import setup_logging + + +async def register_env(env: BaseEnv, env_name: str, signal_queue: Queue, port: int = 9001): + """Registers an environment with Cogment and runs the Cogment server + + Args: + env (BaseEnv): The environment to register + env_name (str): The name to register the environment under + signal_queue (Queue): A queue to signal when server has started + port (int, optional): The port for the Cogment server. Defaults to 9001. + + """ + context = cogment.Context(cog_settings=cog_settings, user_id="cogment_lab") + + context.register_environment(impl=env.impl, impl_name=env_name) + logging.info(f"Registered environment {env_name} with cogment") + + serve = context.serve_all_registered(cogment.ServedEndpoint(port=port)) + signal_queue.put(True) + await serve + + +def env_runner( + env_class: type, + env_args: tuple, + env_kwargs: dict, + env_name: str, + signal_queue: Queue, + port: int = 9001, + log_file: str | None = None, +): + """Given an environment, runs it + + Args: + env_class (type): The environment class to instantiate + env_args (tuple): Positional arguments for the environment + env_kwargs (dict): Keyword arguments for the environment + env_name (str): The name to register the environment under + signal_queue (Queue): A queue to signal when server has started + port (int, optional): The port for the Cogment server. Defaults to 9001. + log_file (str | None, optional): File path to write logs to. Defaults to None. + """ + if log_file: + setup_logging(log_file) + env = env_class(*env_args, **env_kwargs) + + asyncio.run(register_env(env, env_name, signal_queue, port)) diff --git a/cogment_lab/generated/__init__.py b/cogment_lab/generated/__init__.py new file mode 100644 index 0000000..08d7dad --- /dev/null +++ b/cogment_lab/generated/__init__.py @@ -0,0 +1,14 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + diff --git a/cogment_lab/generated/cog_settings.py b/cogment_lab/generated/cog_settings.py new file mode 100644 index 0000000..0d61b2d --- /dev/null +++ b/cogment_lab/generated/cog_settings.py @@ -0,0 +1,35 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import cogment as _cog +from types import SimpleNamespace + +import cogment_lab.generated.data_pb2 as data_pb +import cogment_lab.generated.ndarray_pb2 as ndarray_pb +import cogment_lab.generated.spaces_pb2 as spaces_pb + + +_player_class = _cog.actor.ActorClass( + name="player", + config_type=data_pb.AgentConfig, + action_space=data_pb.PlayerAction, + observation_space=data_pb.Observation, + ) + + +actor_classes = _cog.actor.ActorClassList(_player_class) + +trial = SimpleNamespace(config_type=data_pb.TrialConfig) + +environment = SimpleNamespace(config_type=data_pb.EnvironmentConfig) diff --git a/cogment_lab/generated/data_pb2.py b/cogment_lab/generated/data_pb2.py new file mode 100644 index 0000000..64100ad --- /dev/null +++ b/cogment_lab/generated/data_pb2.py @@ -0,0 +1,61 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# source: data.proto +"""Generated protocol buffer code.""" +from google.protobuf.internal import builder as _builder +from google.protobuf import descriptor as _descriptor +from google.protobuf import descriptor_pool as _descriptor_pool +from google.protobuf import symbol_database as _symbol_database +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + +import cogment_lab.generated.ndarray_pb2 as ndarray__pb2 +import cogment_lab.generated.spaces_pb2 as spaces__pb2 + + +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\ndata.proto\x12\x0b\x63ogment_lab\x1a\rndarray.proto\x1a\x0cspaces.proto\"\xd7\x01\n\x10\x45nvironmentSpecs\x12\x16\n\x0eimplementation\x18\x01 \x01(\t\x12\x12\n\nturn_based\x18\x02 \x01(\x08\x12\x13\n\x0bnum_players\x18\x03 \x01(\x05\x12\x34\n\x11observation_space\x18\x04 \x01(\x0b\x32\x19.cogment_lab.spaces.Space\x12/\n\x0c\x61\x63tion_space\x18\x05 \x01(\x0b\x32\x19.cogment_lab.spaces.Space\x12\x1b\n\x13web_components_file\x18\x06 \x01(\t\"s\n\nAgentSpecs\x12\x34\n\x11observation_space\x18\x01 \x01(\x0b\x32\x19.cogment_lab.spaces.Space\x12/\n\x0c\x61\x63tion_space\x18\x02 \x01(\x0b\x32\x19.cogment_lab.spaces.Space\"Y\n\x05Value\x12\x16\n\x0cstring_value\x18\x01 \x01(\tH\x00\x12\x13\n\tint_value\x18\x02 \x01(\x05H\x00\x12\x15\n\x0b\x66loat_value\x18\x03 \x01(\x02H\x00\x42\x0c\n\nvalue_type\"\xf1\x01\n\x11\x45nvironmentConfig\x12\x0e\n\x06run_id\x18\x01 \x01(\t\x12\x0e\n\x06render\x18\x02 \x01(\x08\x12\x14\n\x0crender_width\x18\x03 \x01(\x05\x12\x0c\n\x04seed\x18\x04 \x01(\r\x12\x0f\n\x07\x66latten\x18\x05 \x01(\x08\x12\x41\n\nreset_args\x18\x06 \x03(\x0b\x32-.cogment_lab.EnvironmentConfig.ResetArgsEntry\x1a\x44\n\x0eResetArgsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12!\n\x05value\x18\x02 \x01(\x0b\x32\x12.cogment_lab.Value:\x02\x38\x01\"/\n\nHFHubModel\x12\x0f\n\x07repo_id\x18\x01 \x01(\t\x12\x10\n\x08\x66ilename\x18\x02 \x01(\t\"\xa4\x01\n\x0b\x41gentConfig\x12\x0e\n\x06run_id\x18\x01 \x01(\t\x12,\n\x0b\x61gent_specs\x18\x02 \x01(\x0b\x32\x17.cogment_lab.AgentSpecs\x12\x0c\n\x04seed\x18\x03 \x01(\r\x12\x10\n\x08model_id\x18\x04 \x01(\t\x12\x17\n\x0fmodel_iteration\x18\x05 \x01(\x05\x12\x1e\n\x16model_update_frequency\x18\x06 \x01(\x05\"\r\n\x0bTrialConfig\"\x88\x01\n\x0bObservation\x12*\n\x05value\x18\x01 \x01(\x0b\x32\x1b.cogment_lab.nd_array.Array\x12\x0e\n\x06\x61\x63tive\x18\x02 \x01(\x08\x12\r\n\x05\x61live\x18\x03 \x01(\x08\x12\x1b\n\x0erendered_frame\x18\x04 \x01(\x0cH\x00\x88\x01\x01\x42\x11\n\x0f_rendered_frame\":\n\x0cPlayerAction\x12*\n\x05value\x18\x01 \x01(\x0b\x32\x1b.cogment_lab.nd_array.Arrayb\x06proto3') + +_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals()) +_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'data_pb2', globals()) +if _descriptor._USE_C_DESCRIPTORS == False: + + DESCRIPTOR._options = None + _ENVIRONMENTCONFIG_RESETARGSENTRY._options = None + _ENVIRONMENTCONFIG_RESETARGSENTRY._serialized_options = b'8\001' + _ENVIRONMENTSPECS._serialized_start=57 + _ENVIRONMENTSPECS._serialized_end=272 + _AGENTSPECS._serialized_start=274 + _AGENTSPECS._serialized_end=389 + _VALUE._serialized_start=391 + _VALUE._serialized_end=480 + _ENVIRONMENTCONFIG._serialized_start=483 + _ENVIRONMENTCONFIG._serialized_end=724 + _ENVIRONMENTCONFIG_RESETARGSENTRY._serialized_start=656 + _ENVIRONMENTCONFIG_RESETARGSENTRY._serialized_end=724 + _HFHUBMODEL._serialized_start=726 + _HFHUBMODEL._serialized_end=773 + _AGENTCONFIG._serialized_start=776 + _AGENTCONFIG._serialized_end=940 + _TRIALCONFIG._serialized_start=942 + _TRIALCONFIG._serialized_end=955 + _OBSERVATION._serialized_start=958 + _OBSERVATION._serialized_end=1094 + _PLAYERACTION._serialized_start=1096 + _PLAYERACTION._serialized_end=1154 +# @@protoc_insertion_point(module_scope) diff --git a/cogment_lab/generated/ndarray_pb2.py b/cogment_lab/generated/ndarray_pb2.py new file mode 100644 index 0000000..6f63473 --- /dev/null +++ b/cogment_lab/generated/ndarray_pb2.py @@ -0,0 +1,41 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# source: ndarray.proto +"""Generated protocol buffer code.""" +from google.protobuf.internal import builder as _builder +from google.protobuf import descriptor as _descriptor +from google.protobuf import descriptor_pool as _descriptor_pool +from google.protobuf import symbol_database as _symbol_database +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + + + +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\rndarray.proto\x12\x14\x63ogment_lab.nd_array\"\xb8\x01\n\x05\x41rray\x12*\n\x05\x64type\x18\x01 \x01(\x0e\x32\x1b.cogment_lab.nd_array.DType\x12\r\n\x05shape\x18\x02 \x03(\r\x12\x10\n\x08raw_data\x18\x03 \x01(\x0c\x12\x10\n\x08npy_data\x18\x04 \x01(\x0c\x12\x13\n\x0b\x64ouble_data\x18\x05 \x03(\x01\x12\x12\n\nint32_data\x18\x06 \x03(\x11\x12\x12\n\nint64_data\x18\x07 \x03(\x12\x12\x13\n\x0buint32_data\x18\x08 \x03(\r*\x83\x01\n\x05\x44Type\x12\x11\n\rDTYPE_UNKNOWN\x10\x00\x12\x11\n\rDTYPE_FLOAT32\x10\x01\x12\x11\n\rDTYPE_FLOAT64\x10\x02\x12\x0e\n\nDTYPE_INT8\x10\x03\x12\x0f\n\x0b\x44TYPE_INT32\x10\x04\x12\x0f\n\x0b\x44TYPE_INT64\x10\x05\x12\x0f\n\x0b\x44TYPE_UINT8\x10\x06\x62\x06proto3') + +_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals()) +_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'ndarray_pb2', globals()) +if _descriptor._USE_C_DESCRIPTORS == False: + + DESCRIPTOR._options = None + _DTYPE._serialized_start=227 + _DTYPE._serialized_end=358 + _ARRAY._serialized_start=40 + _ARRAY._serialized_end=224 +# @@protoc_insertion_point(module_scope) diff --git a/cogment_lab/generated/spaces_pb2.py b/cogment_lab/generated/spaces_pb2.py new file mode 100644 index 0000000..9af8bef --- /dev/null +++ b/cogment_lab/generated/spaces_pb2.py @@ -0,0 +1,52 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# source: spaces.proto +"""Generated protocol buffer code.""" +from google.protobuf.internal import builder as _builder +from google.protobuf import descriptor as _descriptor +from google.protobuf import descriptor_pool as _descriptor_pool +from google.protobuf import symbol_database as _symbol_database +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + +import cogment_lab.generated.ndarray_pb2 as ndarray__pb2 + + +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0cspaces.proto\x12\x12\x63ogment_lab.spaces\x1a\rndarray.proto\"$\n\x08\x44iscrete\x12\t\n\x01n\x18\x01 \x01(\x05\x12\r\n\x05start\x18\x02 \x01(\x05\"Z\n\x03\x42ox\x12(\n\x03low\x18\x02 \x01(\x0b\x32\x1b.cogment_lab.nd_array.Array\x12)\n\x04high\x18\x03 \x01(\x0b\x32\x1b.cogment_lab.nd_array.Array\"5\n\x0bMultiBinary\x12&\n\x01n\x18\x01 \x01(\x0b\x32\x1b.cogment_lab.nd_array.Array\":\n\rMultiDiscrete\x12)\n\x04nvec\x18\x01 \x01(\x0b\x32\x1b.cogment_lab.nd_array.Array\"|\n\x04\x44ict\x12\x31\n\x06spaces\x18\x01 \x03(\x0b\x32!.cogment_lab.spaces.Dict.SubSpace\x1a\x41\n\x08SubSpace\x12\x0b\n\x03key\x18\x01 \x01(\t\x12(\n\x05space\x18\x02 \x01(\x0b\x32\x19.cogment_lab.spaces.Space\"\x89\x02\n\x05Space\x12\x30\n\x08\x64iscrete\x18\x01 \x01(\x0b\x32\x1c.cogment_lab.spaces.DiscreteH\x00\x12&\n\x03\x62ox\x18\x02 \x01(\x0b\x32\x17.cogment_lab.spaces.BoxH\x00\x12(\n\x04\x64ict\x18\x03 \x01(\x0b\x32\x18.cogment_lab.spaces.DictH\x00\x12\x37\n\x0cmulti_binary\x18\x04 \x01(\x0b\x32\x1f.cogment_lab.spaces.MultiBinaryH\x00\x12;\n\x0emulti_discrete\x18\x05 \x01(\x0b\x32!.cogment_lab.spaces.MultiDiscreteH\x00\x42\x06\n\x04kindb\x06proto3') + +_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals()) +_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'spaces_pb2', globals()) +if _descriptor._USE_C_DESCRIPTORS == False: + + DESCRIPTOR._options = None + _DISCRETE._serialized_start=51 + _DISCRETE._serialized_end=87 + _BOX._serialized_start=89 + _BOX._serialized_end=179 + _MULTIBINARY._serialized_start=181 + _MULTIBINARY._serialized_end=234 + _MULTIDISCRETE._serialized_start=236 + _MULTIDISCRETE._serialized_end=294 + _DICT._serialized_start=296 + _DICT._serialized_end=420 + _DICT_SUBSPACE._serialized_start=355 + _DICT_SUBSPACE._serialized_end=420 + _SPACE._serialized_start=423 + _SPACE._serialized_end=688 +# @@protoc_insertion_point(module_scope) diff --git a/cogment_lab/humans/__init__.py b/cogment_lab/humans/__init__.py new file mode 100644 index 0000000..08d7dad --- /dev/null +++ b/cogment_lab/humans/__init__.py @@ -0,0 +1,14 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + diff --git a/cogment_lab/humans/actor.py b/cogment_lab/humans/actor.py new file mode 100644 index 0000000..819e848 --- /dev/null +++ b/cogment_lab/humans/actor.py @@ -0,0 +1,212 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +import base64 +import io +import logging +import multiprocessing as mp +import os +from typing import Any +import json + +from jinja2 import Template + +import cogment +import numpy as np +import uvicorn +from PIL import Image +from fastapi import FastAPI, WebSocket, WebSocketDisconnect +from fastapi.responses import FileResponse, HTMLResponse +from fastapi.staticfiles import StaticFiles + +from cogment_lab.core import CogmentActor +from cogment_lab.generated import cog_settings + + +def image_to_msg(img: np.ndarray | None) -> str | None: + if img is None: + return None + img = Image.fromarray(img) + img_byte_array = io.BytesIO() + img.save(img_byte_array, format="PNG") + base64_encoded_result_bytes = base64.b64encode(img_byte_array.getvalue()) + base64_encoded_result_str = base64_encoded_result_bytes.decode("ascii") + return f"data:image/png;base64,{base64_encoded_result_str}" + + +def msg_to_action(data: str, action_map: list[str] | dict[str, int]) -> int: + if isinstance(action_map, list): + action_map = {action: i for i, action in enumerate(action_map)} + + if data.startswith("{"): + # This is a JSON object + action = json.loads(data) + elif data not in action_map: + action = action_map["no-op"] + else: + action = action_map[data] + + logging.info(f"Processed action {action} from {data} with action_map {action_map}") + return action + + +async def start_fastapi( + port: int, + send_queue: asyncio.Queue, + recv_queue: asyncio.Queue, + actions: list[str] | dict[str, Any] | None = None, + fps: float = 30.0, + html_override: str | None = None, + file_override: str | None = None, + jinja_parameters: dict[str, Any] | None = None, +): + app = FastAPI() + + if actions is None: + actions = ["no-op", "ArrowLeft", "ArrowRight"] + + if jinja_parameters is None: + jinja_parameters = {} + + @app.get("/") + async def get(): + logging.info("Serving index.html") + if html_override is not None: + # Render HTML from string + template = Template(html_override) + rendered_html = template.render(**jinja_parameters) + return HTMLResponse(rendered_html) + elif file_override is not None and os.path.isfile(file_override): + # Render HTML from file + with open(file_override, "r") as file: + file_content = file.read() + template = Template(file_content) + rendered_html = template.render(**jinja_parameters) + return HTMLResponse(rendered_html) + else: + # Fallback option: Serve static file + static_directory_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), "static") + return FileResponse(os.path.join(static_directory_path, "index.html")) + + @app.websocket("/ws") + async def websocket_endpoint(websocket: WebSocket): + logging.info("Waiting for socket connection") + logging.info(f"Setting last_action_data") + last_action_data = "no-op" + logging.info(f"Set {last_action_data=}") + await websocket.accept() + logging.info("Client connected") + while True: + try: + logging.info("Waiting for frame") + frame: np.ndarray = await recv_queue.get() + if not isinstance(frame, np.ndarray): + logging.warning(f"Got frame of type {type(frame)}") + continue + logging.info(f"Got frame with shape {frame.shape}") + msg = image_to_msg(frame) + if msg is not None: + await websocket.send_text(msg) + + try: + action_data = await asyncio.wait_for(websocket.receive_text(), timeout=1.0 / fps) + last_action_data = action_data + logging.info(f"Got action {action_data}, updated {last_action_data=}") + except asyncio.TimeoutError: + logging.info(f"Timed out waiting for action, using {last_action_data=}") + action_data = last_action_data + + action = msg_to_action(action_data, actions) + + await send_queue.put(action) + except WebSocketDisconnect: + logging.info("Client disconnected, waiting for new connection.") + await websocket.close() + await websocket.accept() # Accept a new WebSocket connection + except Exception as e: + logging.error("An error occurred: %s", e) + break # Break the loop in case of non-WebSocketDisconnect exceptions + + current_file_path = os.path.abspath(os.path.dirname(__file__)) + static_directory_path = os.path.join(current_file_path, "static") + app.mount("/static", StaticFiles(directory=static_directory_path), name="static") + + config = uvicorn.Config(app, host="0.0.0.0", port=port) + server = uvicorn.Server(config) + + await server.serve() + + +class HumanPlayer(CogmentActor): + def __init__(self, send_queue: asyncio.Queue, recv_queue: asyncio.Queue): + super().__init__(send_queue, recv_queue) + self.send_queue = send_queue + self.recv_queue = recv_queue + + async def act(self, observation: Any, rendered_frame: np.ndarray | None = None) -> int: + logging.info( + f"Getting an action with {observation=}" + f" and {rendered_frame.shape=}" + if rendered_frame is not None + else "no frame" + ) + await self.send_queue.put(rendered_frame) + action = await self.recv_queue.get() + return action + + +async def run_cogment_actor(port: int, send_queue: asyncio.Queue, recv_queue: asyncio.Queue, signal_queue: mp.Queue): + context = cogment.Context(cog_settings=cog_settings, user_id="cogment_lab") + + human_player = HumanPlayer(send_queue, recv_queue) + + logging.info("Registering actor") + + context.register_actor(impl=human_player.impl, impl_name="web_ui", actor_classes=["player"]) + logging.info("Serving actor") + + serve = context.serve_all_registered(cogment.ServedEndpoint(port=port)) + + signal_queue.put(True) + + await serve + + +async def shutdown(): + tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()] + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + asyncio.get_event_loop().stop() + + +def signal_handler(sig, frame): + asyncio.create_task(shutdown()) + + +async def main(app_port: int = 8000, cogment_port: int = 8999): + app_to_actor = asyncio.Queue() + actor_to_app = asyncio.Queue() + fastapi_task = asyncio.create_task(start_fastapi(port=app_port, send_queue=app_to_actor, recv_queue=actor_to_app)) + cogment_task = asyncio.create_task( + run_cogment_actor(port=cogment_port, send_queue=actor_to_app, recv_queue=app_to_actor) + ) + + await asyncio.gather(fastapi_task, cogment_task) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/cogment_lab/humans/runner.py b/cogment_lab/humans/runner.py new file mode 100644 index 0000000..15f9566 --- /dev/null +++ b/cogment_lab/humans/runner.py @@ -0,0 +1,118 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +import signal +from multiprocessing import Queue +from typing import Any + +from cogment_lab.humans.actor import start_fastapi, run_cogment_actor +from cogment_lab.utils.runners import setup_logging + + +# def human_actor_runner( +# app_port: int = 8000, +# cogment_port: int = 8999, +# log_file: str | None = None +# ): +# """Runs the human actor along with the FastAPI server""" +# if log_file: +# setup_logging(log_file) +# +# # Queues for communication between FastAPI and Cogment actor +# app_to_actor = asyncio.Queue() +# actor_to_app = asyncio.Queue() +# +# # Asyncio tasks for the FastAPI server and Cogment actor +# fastapi_task = start_fastapi(port=app_port, send_queue=app_to_actor, recv_queue=actor_to_app) +# cogment_task = asyncio.create_task(run_cogment_actor(port=cogment_port, send_queue=actor_to_app, recv_queue=app_to_actor)) +# +# # Run the asyncio event loop +# asyncio.run(asyncio.gather(fastapi_task, cogment_task)) + + +async def shutdown(): + tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()] + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + asyncio.get_event_loop().stop() + + +def signal_handler(sig, frame): + asyncio.create_task(shutdown()) + + +async def human_actor_main( + app_port: int, + cogment_port: int, + signal_queue: Queue, + actions: list[str] | None = None, + fps: float = 30, + html_override: str | None = None, + file_override: str | None = None, + jinja_parameters: dict[str, Any] | None = None, +): + app_to_actor = asyncio.Queue() + actor_to_app = asyncio.Queue() + fastapi_task = asyncio.create_task( + start_fastapi( + port=app_port, + send_queue=app_to_actor, + recv_queue=actor_to_app, + actions=actions, + fps=fps, + html_override=html_override, + file_override=file_override, + jinja_parameters=jinja_parameters, + ) + ) + cogment_task = asyncio.create_task( + run_cogment_actor( + port=cogment_port, send_queue=actor_to_app, recv_queue=app_to_actor, signal_queue=signal_queue + ) + ) + + await asyncio.gather(fastapi_task, cogment_task) + + +def human_actor_runner( + app_port: int, + cogment_port: int, + signal_queue: Queue, + log_file: str | None = None, + actions: list[str] | None = None, + fps: float = 30, + html_override: str | None = None, + file_override: str | None = None, + jinja_parameters: dict[str, Any] | None = None, +): + if log_file: + setup_logging(log_file) + + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + for sig in [signal.SIGINT, signal.SIGTERM]: + loop.add_signal_handler(sig, lambda s=sig, frame=None: signal_handler(s, frame)) + + try: + loop.run_until_complete( + human_actor_main( + app_port, cogment_port, signal_queue, actions, fps, html_override, file_override, jinja_parameters + ) + ) + finally: + loop.close() diff --git a/cogment_lab/humans/static/index.html b/cogment_lab/humans/static/index.html new file mode 100644 index 0000000..6f86525 --- /dev/null +++ b/cogment_lab/humans/static/index.html @@ -0,0 +1,60 @@ + + + + + + + RL Env Interface + + +

RL Env Interface

+ Cogment UI
+ + + + diff --git a/cogment_lab/process_manager.py b/cogment_lab/process_manager.py new file mode 100644 index 0000000..38d4697 --- /dev/null +++ b/cogment_lab/process_manager.py @@ -0,0 +1,491 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +import datetime +import logging +import os +from asyncio import Task +from collections.abc import Sequence +from multiprocessing import Process, Queue +from typing import Callable, Any, Coroutine + +import cogment + +from cogment_lab.core import BaseEnv, BaseActor +from cogment_lab.generated import cog_settings, data_pb2 +from cogment_lab.humans.runner import human_actor_runner +from cogment_lab.actors.runner import actor_runner +from cogment_lab.envs.runner import env_runner + +import multiprocessing as mp +import atexit + +from cogment_lab.utils.trial_utils import get_actor_params, format_data_multiagent, TrialData + +ORCHESTRATOR_ENDPOINT = f"grpc://localhost:9000" +ENVIRONMENT_ENDPOINT = f"grpc://localhost:9001" +RANDOM_AGENT_ENDPOINT = f"grpc://localhost:9002" +HUMAN_AGENT_ENDPOINT = f"grpc://localhost:8999" +DATASTORE_ENDPOINT = f"grpc://localhost:9003" + + +AgentName = str +ImplName = str +TrialName = str + + +class Cogment: + """Main Cogment class for managing experiments""" + + def __init__( + self, + user_id: str = "cogment_lab", + torch_mode: bool = False, + log_dir: str | None = None, + mp_method: str | None = None, + ): + """Initializes the Cogment instance + + Args: + user_id (str, optional): User ID. Defaults to "cogment_lab". + torch_mode (bool, optional): Whether to use PyTorch multiprocessing. Defaults to False. + log_dir (str, optional): Directory to store logs. Defaults to "logs". + mp_method (str | None, optional): Multiprocessing method to use. Defaults to None. + """ + self.processes: dict[ImplName, Process] = {} + self.tasks: dict[ImplName, Task] = {} + + self._register_shutdown_hook() + self.torch_mode = torch_mode + self.log_dir = log_dir + + self.envs: dict[ImplName, BaseEnv] = {} + self.actors: dict[ImplName, BaseActor] = {} + + self.context = cogment.Context(cog_settings=cog_settings, user_id=user_id) + self.controller = self.context.get_controller(endpoint=cogment.Endpoint(ORCHESTRATOR_ENDPOINT)) + self.datastore = self.context.get_datastore(endpoint=cogment.Endpoint(DATASTORE_ENDPOINT)) + + self.env_ports: dict[ImplName, int] = {} + self.actor_ports: dict[ImplName, int] = {} + + self.mp_ctx = mp.get_context(mp_method) if mp_method else mp.get_context() + + self.trial_envs: dict[TrialName, ImplName] = {} + + def _add_process( + self, target: Callable, args: tuple, name: ImplName, use_torch: bool | None = None, force: bool = False + ): + """Adds a process to the list of processes + + Args: + target (Callable): The process target function + args (tuple): Arguments for the process target + name (ImplName): Name of the process + use_torch (bool | None, optional): Whether to use PyTorch multiprocessing. Defaults to None. + force (bool, optional): Whether to force adding the process if it already exists. Defaults to False. + + Raises: + ValueError: If the process already exists and force is False + """ + if use_torch is None: + use_torch = self.torch_mode + + if name in self.processes and not force: + raise ValueError(f"Process {name} already exists") + + if use_torch: + from torch.multiprocessing import Process as TorchProcess + + p = TorchProcess(target=target, args=args) + else: + p = self.mp_ctx.Process(target=target, args=args) + p.start() + self.processes[name] = p + + def _add_task(self, target: Coroutine, name: ImplName) -> Task: + """Adds a task to the list of tasks + + Args: + target (Coroutine): The task target function + name (ImplName): Name of the task + + Returns: + Task: The task instance + """ + if name in self.tasks: + raise ValueError(f"Task {name} already exists") + + task = asyncio.create_task(target) + self.tasks[name] = task + return task + + def run_env( + self, env: BaseEnv, env_name: ImplName, port: int = 9001, log_file: str | None = None + ) -> Coroutine[bool]: + """Given an environment, runs it in a subprocess + + Args: + env (BaseEnv): The environment instance + env_name (ImplName): Name for the environment + port (int, optional): Port to run the environment on. Defaults to 9001. + log_file (str | None, optional): Log file path. Defaults to None. + + Returns: + bool: Whether the environment startup succeeded + """ + env_class = type(env) + env_args = env.args + env_kwargs = env.kwargs + + signal_queue = Queue(1) + + if self.log_dir is not None and log_file: + log_file = os.path.join(self.log_dir, log_file) + + self._add_process( + target=env_runner, + name=env_name, + args=(env_class, env_args, env_kwargs, env_name, signal_queue, port, log_file), + ) + logging.info(f"Started environment {env_name} on port {port} with log file {log_file}") + + self.envs[env_name] = env + self.env_ports[env_name] = port + + return self.is_ready(signal_queue) + + def run_actor( + self, actor: BaseActor, actor_name: ImplName, port: int = 9002, log_file: str | None = None + ) -> Coroutine[bool]: + """Given an actor, runs it + + Args: + actor (BaseActor): The actor instance + actor_name (ImplName): Name for the actor + port (int, optional): Port to run the actor on. Defaults to 9002. + log_file (str | None, optional): Log file path. Defaults to None. + + Returns: + bool: Whether the actor startup succeeded + """ + actor_class = type(actor) + actor_args = actor.args + actor_kwargs = actor.kwargs + + signal_queue = Queue(1) + + if self.log_dir is not None and log_file: + log_file = os.path.join(self.log_dir, log_file) + + self._add_process( + target=actor_runner, + name=actor_name, + args=(actor_class, actor_args, actor_kwargs, actor_name, signal_queue, port, log_file), + ) + logging.info(f"Started actor {actor_name} on port {port} with log file {log_file}") + + self.actors[actor_name] = actor + self.actor_ports[actor_name] = port + + return self.is_ready(signal_queue) + + def run_local_actor( + self, actor: BaseActor, actor_name: ImplName, port: int = 9002, log_file: str | None = None + ) -> Task: + """Given an actor, runs it locally + + Args: + actor (BaseActor): The actor instance + actor_name (ImplName): Name for the actor + port (int, optional): Port to run the actor on. Defaults to 9002. + log_file (str | None, optional): Log file path. Defaults to None. + + Returns: + bool: Whether the actor startup succeeded + """ + + if self.log_dir is not None and log_file: + log_file = os.path.join(self.log_dir, log_file) + + self.context.register_actor(impl=actor.impl, impl_name=actor_name, actor_classes=["player"]) + + serve = self._add_task(self.context.serve_all_registered(cogment.ServedEndpoint(port=port)), actor_name) + + logging.info(f"Started actor {actor_name} on port {port} with log file {log_file}") + + self.actors[actor_name] = actor + self.actor_ports[actor_name] = port + + return serve + + def run_web_ui( + self, + app_port: int = 8000, + cogment_port: int = 8999, + actions: list[str] | dict[str, Any] = [], + log_file: str | None = None, + fps: int = 30, + html_override: str | None = None, + file_override: str | None = None, + jinja_parameters: dict[str, Any] | None = None, + ) -> Coroutine[bool]: + """Runs the human actor in a separate process + + Args: + app_port (int, optional): Port for web UI. Defaults to 8000. + cogment_port (int, optional): Port for Cogment connection. Defaults to 8999. + actions (list[str], optional): Allowed actions. Defaults to []. + log_file (str | None, optional): Log file path. Defaults to None. + fps (int, optional): Frames per second for environment. Defaults to 30. + html_override (str | None, optional): HTML override file path. Defaults to None. + file_override (str | None, optional): File override file path. Defaults to None. + jinja_parameters (dict[str, Any] | None, optional): Jinja parameters for the HTML override. Defaults to None. + + Returns: + bool: Whether the web UI startup succeeded + """ + + signal_queue = Queue(1) + + if self.log_dir is not None and log_file: + log_file = os.path.join(self.log_dir, log_file) + + self._add_process( + target=human_actor_runner, + name="web_ui", + args=( + app_port, + cogment_port, + signal_queue, + log_file, + actions, + fps, + html_override, + file_override, + jinja_parameters, + ), + ) + logging.info(f"Started web UI on port {app_port} with log file {log_file}") + + self.actor_ports["web_ui"] = cogment_port + + return self.is_ready(signal_queue) + + def stop_service(self, name: ImplName, timeout: float = 1.0): + """Stops a process or a task. + + Args: + name (str): Name of the process or task to stop + timeout (float, optional): How long (in seconds) to wait for the process to stop before killing it. Defaults to 1.0. + """ + if name in self.processes: + self._stop_process(name, timeout) + elif name in self.tasks: + self._stop_task(name) + else: + raise ValueError(f"Service {name} does not exist") + + def _stop_process(self, name: ImplName, timeout: float = 1.0): + """Stops a process + + Args: + name (str): Name of the process to stop + timeout (float, optional): How long (in seconds) to wait for the process to stop before killing it. Defaults to 1.0. + """ + if name not in self.processes: + raise ValueError(f"Process {name} does not exist") + logging.info(f"Stopping process {name}") + process = self.processes[name] + if timeout == 0.0: + process.kill() + process.join() + else: + process.terminate() + process.join(timeout=timeout) + if process.is_alive(): + process.kill() + process.join() + del self.processes[name] + + def _stop_task(self, name: ImplName): + """Stops a task + + Args: + name (str): Name of the task to stop + """ + if name not in self.tasks: + raise ValueError(f"Task {name} does not exist") + logging.info(f"Stopping task {name}") + task = self.tasks[name] + task.cancel() + del self.tasks[name] + + def stop_all_services(self, timeout: float = 1.0): + """Stops all processes and tasks + + Args: + timeout (float, optional): How long (in seconds) to wait for the processes to stop before killing them. Defaults to 1.0. + """ + for name in list(self.processes.keys()): + self._stop_process(name, timeout) + for name in list(self.tasks.keys()): + self._stop_task(name) + + def _cleanup_wrapper(self): + try: + loop = asyncio.get_event_loop() + except RuntimeError: + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + asyncio.run(self.cleanup()) + loop.close() + + async def cleanup(self, timeout: float = 1.0): + """Cleans up all processes. Idempotent.""" + tasks = list(self.tasks.keys()) + processes = list(self.processes.keys()) + + for name in tasks: + try: + self._stop_task(name) + except Exception as e: + logging.warning(f"Failed to stop task {name}: {e}") + + for name in processes: + try: + self._stop_process(name, timeout) + except Exception as e: + logging.warning(f"Failed to stop process {name}: {e}") + + try: + await self.context._grpc_server.stop(None) + except asyncio.exceptions.CancelledError: + logging.info("Server already stopped") + except AttributeError: + logging.info("Server not started") + + def _register_shutdown_hook(self): + """Registers the cleanup method to run on shutdown""" + atexit.register(self._cleanup_wrapper) + + async def start_trial( + self, + env_name: ImplName, + actor_impls: dict[AgentName, ImplName] | ImplName, + session_config: dict[str, Any] = {}, + trial_name: TrialName | None = None, + ): + """Starts a new trial + + Args: + env_name (ImplName): Name of the environment implementation + actor_impls (dict[AgentName, ImplName] | ImplName): Actor implementations mapped to agent names + session_config (dict[str, Any]): kwargs for the environment session + trial_name (str | None, optional): Trial name. Defaults to None. + + Returns: + str: The trial ID + """ + if trial_name is None: + trial_name = f"{env_name}-{datetime.datetime.now().isoformat()}" + + if isinstance(actor_impls, str): + actor_impls = {"gym": actor_impls} + + env = self.envs[env_name] + actor_params = [ + get_actor_params( + name=agent_name, + implementation=actor_impl, + agent_specs=env.agent_specs[agent_name], + endpoint=f"grpc://localhost:{self.actor_ports[actor_impl]}", + ) + for agent_name, actor_impl in actor_impls.items() + ] + + env_config = data_pb2.EnvironmentConfig(**session_config) + + trial_params = cogment.TrialParameters( + cog_settings, + environment_name=env_name, + environment_endpoint=f"grpc://localhost:{self.env_ports[env_name]}", + environment_config=env_config, + actors=actor_params, + environment_implementation=env_name, + datalog_endpoint=DATASTORE_ENDPOINT, + ) + + trial_id = await self.controller.start_trial(trial_id_requested=trial_name, trial_params=trial_params) + + logging.info(f"Started trial {trial_id} with name {trial_name}") + + self.trial_envs[trial_id] = env_name + + return trial_id + + async def get_trial_data( + self, + trial_id: str, + env_name: str | None = None, + fields: Sequence[str] = ( + "observations", + "actions", + "rewards", + "done", + "next_observations", + "last_observation", + ), + ) -> dict[str, TrialData]: + """Gets trial data from the datastore, formatting it appropriately.""" + if env_name is None: + env_name = self.trial_envs[trial_id] + env = self.envs[env_name] + agent_specs = env.agent_specs + + data = await format_data_multiagent(self.datastore, trial_id, agent_specs, fields) + + return data + + async def get_trial(self, trial_id: str): + """Gets a trial by ID + + Args: + trial_id (str): The trial ID + + Returns: + Trial: The trial instance + """ + [trial] = await self.datastore.get_trials(ids=[trial_id]) + return trial + + def __del__(self): + """Cleanup on delete""" + self.stop_all_services() + + async def is_ready(self, queue: Queue): + """Waits for a readiness signal on a queue + + Args: + queue (Queue): The queue to wait on + + Returns: + Any: The object that was put on the queue + """ + while queue.empty(): + await asyncio.sleep(0.1) + return queue.get() diff --git a/cogment_lab/protos/cogment.yaml b/cogment_lab/protos/cogment.yaml new file mode 100644 index 0000000..3712420 --- /dev/null +++ b/cogment_lab/protos/cogment.yaml @@ -0,0 +1,20 @@ +import: + proto: + - ndarray.proto + - spaces.proto + - data.proto + +environment: + config_type: cogment_lab.EnvironmentConfig + +trial: + config_type: cogment_lab.TrialConfig + +# Static configuration +actor_classes: + - name: player + action: + space: cogment_lab.PlayerAction + observation: + space: cogment_lab.Observation + config_type: cogment_lab.AgentConfig diff --git a/cogment_lab/protos/data.proto b/cogment_lab/protos/data.proto new file mode 100644 index 0000000..b622e7f --- /dev/null +++ b/cogment_lab/protos/data.proto @@ -0,0 +1,79 @@ +// Copyright 2024 AI Redefined Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +syntax = "proto3"; + +package cogment_lab; + +import "ndarray.proto"; +import "spaces.proto"; + + +message EnvironmentSpecs { + string implementation = 1; + bool turn_based = 2; + int32 num_players = 3; + spaces.Space observation_space = 4; + spaces.Space action_space = 5; + string web_components_file = 6; +} + +message AgentSpecs { + spaces.Space observation_space = 1; + spaces.Space action_space = 2; +} + +message Value { + oneof value_type { + string string_value = 1; + int32 int_value = 2; + float float_value = 3; + } +} +message EnvironmentConfig { + string run_id = 1; + bool render = 2; + int32 render_width = 3; + uint32 seed = 4; + bool flatten = 5; + map reset_args = 6; +} + +message HFHubModel { + string repo_id = 1; + string filename = 2; +} + +message AgentConfig { + string run_id = 1; + AgentSpecs agent_specs = 2; + uint32 seed = 3; + string model_id = 4; + int32 model_iteration = 5; + int32 model_update_frequency = 6; +} + +message TrialConfig { +} + +message Observation { + nd_array.Array value = 1; + bool active = 2; + bool alive = 3; + optional bytes rendered_frame = 4; +} + +message PlayerAction { + nd_array.Array value = 1; +} \ No newline at end of file diff --git a/cogment_lab/protos/ndarray.proto b/cogment_lab/protos/ndarray.proto new file mode 100644 index 0000000..24dc3a9 --- /dev/null +++ b/cogment_lab/protos/ndarray.proto @@ -0,0 +1,38 @@ +// Copyright 2024 AI Redefined Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +syntax = "proto3"; + +package cogment_lab.nd_array; + +enum DType { + DTYPE_UNKNOWN = 0; + DTYPE_FLOAT32 = 1; + DTYPE_FLOAT64 = 2; + DTYPE_INT8 = 3; + DTYPE_INT32 = 4; + DTYPE_INT64 = 5; + DTYPE_UINT8 = 6; +} + +message Array { + DType dtype = 1; + repeated uint32 shape = 2; + bytes raw_data = 3; + bytes npy_data = 4; + repeated double double_data = 5; + repeated sint32 int32_data = 6; + repeated sint64 int64_data = 7; + repeated uint32 uint32_data = 8; +} diff --git a/cogment_lab/protos/spaces.proto b/cogment_lab/protos/spaces.proto new file mode 100644 index 0000000..04d4686 --- /dev/null +++ b/cogment_lab/protos/spaces.proto @@ -0,0 +1,55 @@ +// Copyright 2024 AI Redefined Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +syntax = "proto3"; + +import "ndarray.proto"; + +package cogment_lab.spaces; + +message Discrete { + int32 n = 1; + int32 start = 2; +} + +message Box { + nd_array.Array low = 2; + nd_array.Array high = 3; +} + +message MultiBinary { + nd_array.Array n = 1; +} + +message MultiDiscrete { + nd_array.Array nvec = 1; +} + +message Dict { + message SubSpace { + string key = 1; + Space space = 2; + } + repeated SubSpace spaces = 1; +} + +message Space { + oneof kind { + Discrete discrete = 1; + Box box = 2; + Dict dict = 3; + MultiBinary multi_binary = 4; + MultiDiscrete multi_discrete = 5; + } +} diff --git a/cogment_lab/py.typed b/cogment_lab/py.typed new file mode 100644 index 0000000..e69de29 diff --git a/cogment_lab/session_helpers.py b/cogment_lab/session_helpers.py new file mode 100644 index 0000000..26fa7cc --- /dev/null +++ b/cogment_lab/session_helpers.py @@ -0,0 +1,173 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +from typing import Optional, Any + +from cogment.actor import ActorSession +from cogment.environment import EnvironmentSession +from cogment.model_registry_v2 import ModelRegistry +from cogment.session import RecvEvent, ActorInfo + +from cogment_lab.specs import AgentSpecs +from cogment_lab.specs.action_space import ActionSpace, Action +from cogment_lab.specs.observation_space import ObservationSpace, Observation + + +class ActorSessionHelper: + """ + Cogment Verse actor session helper + + Provides additional methods to the regular Cogment actor session. + """ + + def __init__(self, actor_session: ActorSession, model_registry: Optional[ModelRegistry]): + self.actor_session = actor_session + self.agent_specs = AgentSpecs.deserialize(self.actor_session.config.agent_specs) + self.action_space = self.agent_specs.get_action_space(seed=self.actor_session.config.seed) + self.observation_space = self.agent_specs.get_observation_space() + self.model_registry = model_registry + + def get_action_space(self) -> ActionSpace: + return self.action_space + + def get_observation_space(self) -> ObservationSpace: + return self.observation_space + + def get_observation(self, event: RecvEvent) -> Observation: + """ + Return the cogment verse observation for the current event. + + If the event does not contain an observation, return None. + """ + if not event.observation: + return None + + return self.observation_space.deserialize(event.observation.observation) + + def get_render(self, event: RecvEvent) -> bytes: + """ + Return the render for the current event. + + If the event does not contain a render, return None. + """ + if not event.observation: + return None + + return event.observation.render + + +class EnvironmentSessionHelper: + """ + A session helper for environments. + """ + + actor_infos: list[ActorInfo] + + def __init__(self, environment_session: EnvironmentSession, agent_specs: dict[str, AgentSpecs]): + self.actor_infos = environment_session.get_active_actors() + + assert set(agent_specs.keys()) == set( + actor_info.actor_name for actor_info in self.actor_infos + ), f"Agent specs and active actors do not match. {agent_specs.keys()} != {self.actor_infos}" + + # Mapping actor_name to actor_idx + self.actor_idxs = {actor_info.actor_name: actor_idx for (actor_idx, actor_info) in enumerate(self.actor_infos)} + # Mapping actor_idx to actor_info + self.actors = [actor_info.actor_name for actor_info in self.actor_infos] + + if isinstance(agent_specs, AgentSpecs): + agent_specs = {actor_name: agent_specs for actor_name in self.actors} + + self.agent_specs = agent_specs + self.observation_spaces = { + agent_id: specs.get_observation_space(environment_session.config.render_width) + for (agent_id, specs) in agent_specs.items() + } + + def get_observation_space(self, actor_name: str) -> ObservationSpace: + return self.observation_spaces[actor_name] + + def get_action_space(self, actor_name: str) -> ActionSpace: + return self.agent_specs[actor_name].get_action_space() + + def _get_actor_idx(self, actor_name: str) -> int: + actor_idx = self.actor_idxs.get(actor_name) + + if actor_idx is None: + raise RuntimeError(f"No actor with name [{actor_name}] found!") + + return actor_idx + + def get_action(self, tick_data: Any, actor_name: str) -> Action | None: + # For environments, tick_datas are events + event: RecvEvent = tick_data + + if not event.actions or not event.actions: + return None + + actor_idx = self._get_actor_idx(actor_name) + action_space = self.get_action_space(actor_name) + + return action_space.deserialize( + event.actions[actor_idx].action, + ) + + def get_observation(self, tick_data: Any, actor_name: str): + """ + Return the cogment verse observation of a given actor at a tick. + + If no observation, returns None. + """ + raise NotImplementedError + + def get_player_actions(self, tick_data: Any, actor_name: str) -> Action | None: + """ + Return the cogment verse player action of a given actor at a tick. + + If only a single player actor is present, no `actor_name` is required. + + If no action, returns None. + """ + event = tick_data + if not event.actions: + return None + + actions = [ + self.get_action(actor_name, tick_data) + for player_actor_name in self.actors + if player_actor_name == actor_name + ] + if len(actions) == 0: + raise RuntimeError(f"No player actors having name [{actor_name}]") + return actions[0] + + def get_player_observations(self, tick_data: Any, actor_name: str): + if actor_name is None: + observations = [self.get_observation(tick_data, actor_name) for player_actor_name in self.actors] + if len(observations) == 0: + raise RuntimeError("No player actors") + if len(observations) > 1: + raise RuntimeError("More than 1 player actor, please provide an actor name") + return observations[0] + + observations = [ + self.get_observation(tick_data, actor_name) + for player_actor_name in self.actors + if player_actor_name == actor_name + ] + if len(observations) == 0: + raise RuntimeError(f"No player actors having name [{actor_name}]") + return observations[0] diff --git a/cogment_lab/specs/__init__.py b/cogment_lab/specs/__init__.py new file mode 100644 index 0000000..4279b84 --- /dev/null +++ b/cogment_lab/specs/__init__.py @@ -0,0 +1,16 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .encode_rendered_frame import encode_rendered_frame +from .environment_specs import AgentSpecs diff --git a/cogment_lab/specs/action_space.py b/cogment_lab/specs/action_space.py new file mode 100644 index 0000000..180e3f2 --- /dev/null +++ b/cogment_lab/specs/action_space.py @@ -0,0 +1,135 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import gymnasium as gym + +from cogment_lab.generated.data_pb2 import ( # pylint: disable=import-error + PlayerAction, +) +from .ndarray_serialization import deserialize_ndarray, serialize_ndarray + + +# pylint: disable=attribute-defined-outside-init +class Action: + """ + Cogment Verse actor action + + Properties: + flat_value: + The action value, as a flat numpy array. + value: + The action value, as a numpy array. + """ + + def __init__(self, gym_space: gym.Space, pb_action=None, value=None): + """ + Action constructor. + Shouldn't be called directly, prefer the factory function of ActionSpace. + """ + self._gym_space = gym_space + + if pb_action is not None: + assert value is None + self._pb_action = pb_action + return + + self._value = value + + def _compute_flat_value(self): + if hasattr(self, "_value"): + value = self._value + if value is None: + return None + return gym.spaces.flatten(self._gym_space, self._value) + + if not self._pb_action.value != b"": + # This happens whenever value is None + return None + + return deserialize_ndarray(self._pb_action.value) + + @property + def flat_value(self): + if not hasattr(self, "_flat_value"): + self._flat_value = self._compute_flat_value() + return self._flat_value + + def _compute_value(self): + flat_value = self.flat_value + if flat_value is None: + return None + return gym.spaces.unflatten(self._gym_space, flat_value) + + @property + def value(self): + if not hasattr(self, "_value"): + self._value = self._compute_value() + return self._value + + +class ActionSpace: + """ + Cogment Verse action space + + Properties: + gym_space: + Wrapped Gym space for the action values (cf. https://www.gymlibrary.dev/api/spaces/) + actor_class: + Class of the actor for which this space will serialize Action probobug messages + seed: + Random seed used when generating random actions + """ + + def __init__(self, gym_space: gym.Space, seed: int = None): + self.gym_space = gym_space + + if seed: + self.gym_space.seed(int(seed)) + + def create(self, value=None): + """ + Create an Action + """ + return Action(self.gym_space, value=value) + + def sample(self, mask=None): + """ + Generate a random Action + """ + return Action(self.gym_space, value=self.gym_space.sample(mask=mask)) + + def serialize( + self, + action, + ): + """ + Serialize an Action to an Action protobuf message + """ + if action.value is None: + return PlayerAction() + + serialized_value = serialize_ndarray(action.flat_value) + return PlayerAction(value=serialized_value) + + def deserialize(self, pb_action): + """ + Deserialize an Action from an Action protobuf message + """ + return Action(self.gym_space, pb_action=pb_action) + + def create_serialize(self, value=None): + """ + Create and serialize an Action to an Action protobuf message + """ + return self.serialize(self.create(value)) diff --git a/cogment_lab/specs/encode_rendered_frame.py b/cogment_lab/specs/encode_rendered_frame.py new file mode 100644 index 0000000..4d6aad8 --- /dev/null +++ b/cogment_lab/specs/encode_rendered_frame.py @@ -0,0 +1,68 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import cv2 +import numpy as np + +MAX_RENDERED_WIDTH = 2048 + + +def encode_rendered_frame(rendered_frame: np.ndarray, max_size: int = MAX_RENDERED_WIDTH) -> bytes: + if max_size <= 0: + max_size = MAX_RENDERED_WIDTH + # gRPC max message size hack + height, width = rendered_frame.shape[:2] + if max(height, width) > max_size: + if height > width: + new_height = max_size + new_width = int(new_height / height * width) + else: + new_width = max_size + new_height = int(height / width * new_width) + rendered_frame = cv2.resize(rendered_frame, (new_width, new_height), interpolation=cv2.INTER_AREA) + + # note rgb -> bgr for cv2 + result, encoded_frame = cv2.imencode(".jpg", rendered_frame[:, :, ::-1]) + assert result + + return encoded_frame.tobytes() + + +def decode_rendered_frame(encoded_frame: bytes) -> np.ndarray: + """ + Decode the rendered frame from bytes to a NumPy array. + + Args: + encoded_frame (bytes): The encoded frame as a byte array. + + Returns: + np.ndarray: The decoded rendered frame as a NumPy array. + """ + if encoded_frame is None or len(encoded_frame) == 0: + return None + + # Convert the byte array back to a NumPy array + encoded_frame_np = np.frombuffer(encoded_frame, dtype=np.uint8) + + # Decode the image from the byte array + decoded_frame = cv2.imdecode(encoded_frame_np, cv2.IMREAD_COLOR) + + # Check if the decoding was successful + if decoded_frame is None: + raise ValueError("Failed to decode the rendered frame.") + + # Convert from BGR to RGB + decoded_frame = decoded_frame[:, :, ::-1] + + return decoded_frame diff --git a/cogment_lab/specs/environment_specs.py b/cogment_lab/specs/environment_specs.py new file mode 100644 index 0000000..04bfc35 --- /dev/null +++ b/cogment_lab/specs/environment_specs.py @@ -0,0 +1,93 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import gymnasium as gym + +from cogment_lab.generated.data_pb2 import ( + AgentSpecs as PbAgentSpecs, +) +from .action_space import ActionSpace +from .ndarray_serialization import SerializationFormat +from .observation_space import ObservationSpace +from .spaces_serialization import deserialize_space, serialize_gym_space +from ..constants import DEFAULT_RENDERED_WIDTH + + +class AgentSpecs: + """ + Representation of the specification of an agent within Cogment Lab. + """ + + def __init__(self, agent_specs_pb: PbAgentSpecs): + """ + AgentSpecs constructor. + Shouldn't be called directly, prefer the factory function such as AgentSpecs.deserialize or AgentSpecs.create_homogeneous. + """ + self._pb = agent_specs_pb + + def get_observation_space(self, render_width: int = DEFAULT_RENDERED_WIDTH) -> ObservationSpace: + """ + Build an instance of the observation space for this environment + + Parameters: + render_width: optional + maximum width for the serialized rendered frame in observation + + NOTE: In the future we'll want to support different observation space per agent role + """ + return ObservationSpace(deserialize_space(self._pb.observation_space), render_width) + + def get_action_space(self, seed: int | None = None) -> ActionSpace: + """ + Build an instance of the action space for this environment + + Parameters: + seed: optional + the seed used when generating random actions + + NOTE: In the future we'll want to support different action space per agent roles + """ + return ActionSpace(deserialize_space(self._pb.action_space), seed) + + @classmethod + def create_homogeneous( + cls, + observation_space: gym.Space, + action_space: gym.Space, + serialization_format: SerializationFormat = SerializationFormat.STRUCTURED, + ): + """ + Factory function building a homogenous EnvironmentSpecs, ie with all actors having the same action and observation spaces. + """ + return cls.deserialize( + PbAgentSpecs( + observation_space=serialize_gym_space(observation_space, serialization_format), + action_space=serialize_gym_space(action_space, serialization_format), + ) + ) + + def serialize(self): + """ + Serialize to a EnvironmentSpecs protobuf message + """ + return self._pb + + @classmethod + def deserialize(cls, agent_specs_pb: PbAgentSpecs): + """ + Factory function building an EnvironmentSpecs instance from a EnvironmentSpecs protobuf message + """ + return cls(agent_specs_pb) diff --git a/cogment_lab/specs/ndarray_serialization.py b/cogment_lab/specs/ndarray_serialization.py new file mode 100644 index 0000000..5178058 --- /dev/null +++ b/cogment_lab/specs/ndarray_serialization.py @@ -0,0 +1,145 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import io +from enum import Enum + +import numpy as np + +from cogment_lab.generated.ndarray_pb2 import ( + DTYPE_FLOAT32, + DTYPE_FLOAT64, + DTYPE_INT8, + DTYPE_INT32, + DTYPE_INT64, + DTYPE_UINT8, + DTYPE_UNKNOWN, + Array, +) + + +PB_DTYPE_FROM_DTYPE = { + "float32": DTYPE_FLOAT32, + "float64": DTYPE_FLOAT64, + "int8": DTYPE_INT8, + "int32": DTYPE_INT32, + "int64": DTYPE_INT64, + "uint8": DTYPE_UINT8, +} + +DTYPE_FROM_PB_DTYPE = { + DTYPE_FLOAT32: np.dtype("float32"), + DTYPE_FLOAT64: np.dtype("float64"), + DTYPE_INT8: np.dtype("int8"), + DTYPE_INT32: np.dtype("int32"), + DTYPE_INT64: np.dtype("int64"), + DTYPE_UINT8: np.dtype("uint8"), +} + +DOUBLE_DTYPES = frozenset(["float32", "float64"]) +UINT32_DTYPES = frozenset(["uint8"]) +INT32_DTYPES = frozenset(["int8", "int32"]) +INT64_DTYPES = frozenset(["int64"]) + + +class SerializationFormat(Enum): + RAW = 1 + NPY = 2 + STRUCTURED = 3 + + +def serialize_ndarray( + nd_array: np.ndarray, + serialization_format: SerializationFormat = SerializationFormat.RAW, +) -> Array: + str_dtype = str(nd_array.dtype) + pb_dtype = PB_DTYPE_FROM_DTYPE.get(str_dtype, DTYPE_UNKNOWN) + + # SerializationFormat.RAW + if serialization_format is SerializationFormat.RAW: + return Array( + shape=nd_array.shape, + dtype=pb_dtype, + raw_data=nd_array.tobytes(order="C"), + ) + + # SerializationFormat.NPY + if serialization_format is SerializationFormat.NPY: + buffer = io.BytesIO() + np.save(buffer, nd_array, allow_pickle=False) + return Array( + shape=nd_array.shape, + dtype=pb_dtype, + npy_data=buffer.getvalue(), + ) + + # SerializationFormat.STRUCTURED: + if str_dtype in DOUBLE_DTYPES: + return Array( + shape=nd_array.shape, + dtype=pb_dtype, + double_data=nd_array.ravel(order="C").tolist(), + ) + if str_dtype in UINT32_DTYPES: + return Array( + shape=nd_array.shape, + dtype=pb_dtype, + uint32_data=nd_array.ravel(order="C").tolist(), + ) + if str_dtype in INT32_DTYPES: + return Array( + shape=nd_array.shape, + dtype=pb_dtype, + int32_data=nd_array.ravel(order="C").tolist(), + ) + if str_dtype in INT64_DTYPES: + return Array( + shape=nd_array.shape, + dtype=pb_dtype, + int64_data=nd_array.ravel(order="C").tolist(), + ) + + raise RuntimeError( + f"[{str_dtype}] is not a supported numpy dtype for serialization format [{serialization_format}]" + ) + + +def deserialize_ndarray(pb_array: Array) -> np.ndarray | None: + dtype = DTYPE_FROM_PB_DTYPE.get(pb_array.dtype) + str_dtype = str(dtype) + shape = tuple(pb_array.shape) + + if len(pb_array.raw_data) > 0: + return np.frombuffer(pb_array.raw_data, dtype=dtype).reshape(shape, order="C") + + if len(pb_array.npy_data) > 0: + buffer = io.BytesIO(pb_array.npy_data) + return np.load(buffer, allow_pickle=False) + + # SerializationFormat.STRUCTURED + if str_dtype in DOUBLE_DTYPES: + return np.array(pb_array.double_data, dtype=dtype).reshape(shape, order="C") + if str_dtype in UINT32_DTYPES: + return np.array(pb_array.uint32_data, dtype=dtype).reshape(shape, order="C") + if str_dtype in INT32_DTYPES: + return np.array(pb_array.int32_data, dtype=dtype).reshape(shape, order="C") + if str_dtype in INT64_DTYPES: + return np.array(pb_array.int64_data, dtype=dtype).reshape(shape, order="C") + + return None + # raise RuntimeError( + # f"[{str_dtype}] is not a supported numpy dtype for serialization format [SerializationFormat.STRUCTURED]" + # ) diff --git a/cogment_lab/specs/observation_space.py b/cogment_lab/specs/observation_space.py new file mode 100644 index 0000000..b768409 --- /dev/null +++ b/cogment_lab/specs/observation_space.py @@ -0,0 +1,227 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import gymnasium as gym +import numpy as np + +from cogment_lab.constants import DEFAULT_RENDERED_WIDTH +from cogment_lab.generated.data_pb2 import ( + Observation as PbObservation, +) +from .encode_rendered_frame import encode_rendered_frame, decode_rendered_frame +from .ndarray_serialization import deserialize_ndarray, serialize_ndarray + + +# pylint: disable=attribute-defined-outside-init +class Observation: + """ + Cogment Verse actor observation + + Properties: + flat_value: + The observation value, as a flat numpy array. + value: + The observation value, as a numpy array. + active: optional + Boolean indicating if the object is active. + alive: optional + Boolean indicating if the object is alive. + rendered_frame: optional + Environment's rendered frame as a numpy array of RGB pixels. + """ + + def __init__( + self, + gym_space: gym.Space, + pb_observation=None, + value=None, + active=None, + alive=None, + rendered_frame=None, + ): + """ + Observation constructor. + Shouldn't be called directly, prefer the factory function of ObservationSpace. + """ + + self._gym_space = gym_space + + if pb_observation is not None: + assert value is None + assert active is None + assert alive is None + assert rendered_frame is None + self._pb_observation = pb_observation + return + + self._value = value + self._active = active + self._alive = alive + self._rendered_frame = rendered_frame + + self._pb_observation = PbObservation( + active=active, + alive=alive, + ) + + def _compute_flat_value(self): + if hasattr(self, "_value"): + return gym.spaces.flatten(self._gym_space, self._value) + + if not self._pb_observation.value != b"" or self._pb_observation.value is None: + return None + + return deserialize_ndarray(self._pb_observation.value) + + @property + def flat_value(self): + if not hasattr(self, "_flat_value"): + self._flat_value = self._compute_flat_value() + return self._flat_value + + def _compute_value(self): + return gym.spaces.unflatten(self._gym_space, self.flat_value) if self.flat_value is not None else None + + @property + def value(self): + if not hasattr(self, "_value"): + self._value = self._compute_value() + return self._value + + def _deserialize_rendered_frame(self): + if not self._pb_observation.rendered_frame != b"": + return None + return decode_rendered_frame(self._pb_observation.rendered_frame) + + @property + def rendered_frame(self): + if not hasattr(self, "_rendered_frame"): + self._rendered_frame = self._deserialize_rendered_frame() + return self._rendered_frame + + @property + def active(self): + return self._pb_observation.active if self._pb_observation.active != b"" else self._active + + @property + def alive(self): + return self._pb_observation.alive if self._pb_observation.alive != b"" else self._alive + + def __repr__(self): + return f"Observation(value={self.value.shape if isinstance(self.value, np.ndarray) else self.value}, active={self.active}, alive={self.alive}, rendered_frame={self.rendered_frame.shape if self.rendered_frame is not None else 'None'})@{hex(id(self))}" + + def __str__(self): + return self.__repr__() + + +class ObservationSpace: + """ + Cogment Verse observation space + + Properties: + gym_space: + Wrapped Gym space for the observation values + render_width: + Maximum width for the serialized rendered frame in observations + """ + + def __init__(self, space: gym.Space, render_width: int = DEFAULT_RENDERED_WIDTH): + """ + ObservationSpace constructor. + Shouldn't be called directly, prefer the factory function of EnvironmentSpecs. + """ + if isinstance(space, gym.spaces.Dict) and ("action_mask" in space.spaces): + # Check the observation space defines an action_mask "component" (like PettingZoo does) + assert "observation" in space.spaces + assert len(space.spaces) == 2 + + self.gym_space = space.spaces["observation"] + self.action_mask_gym_space = space.spaces["action_mask"] + else: + # "Standard" observation space, no action_mask + self.gym_space = space + self.action_mask_gym_space = None + + # Other configuration + self.render_width = render_width + + def create( + self, + value=None, + active=None, + alive=None, + rendered_frame=None, + ) -> Observation: + """ + Create an Observation + """ + return Observation( + self.gym_space, + value=value, + active=active, + alive=alive, + rendered_frame=rendered_frame, + ) + + def serialize( + self, + observation: Observation, + ) -> PbObservation: + """ + Serialize an Observation to an Observation protobuf message + """ + + serialized_value = None + if observation.value is not None: + flat_value = gym.spaces.flatten(self.gym_space, observation.value) + serialized_value = serialize_ndarray(flat_value) + + serialized_rendered_frame = None + if observation.rendered_frame is not None: + serialized_rendered_frame = encode_rendered_frame( + rendered_frame=observation.rendered_frame, max_size=self.render_width + ) + + return PbObservation( + value=serialized_value, + active=observation.active, + alive=observation.alive, + rendered_frame=serialized_rendered_frame, + ) + + def deserialize(self, pb_observation: PbObservation) -> Observation: + """ + Deserialize an Observation from an Observation protobuf message + """ + + return Observation(self.gym_space, pb_observation=pb_observation) + + def create_serialize( + self, + value=None, + active=None, + alive=None, + rendered_frame=None, + ) -> PbObservation: + """ + Create a serialized Observation + """ + return self.serialize( + self.create( + value=value, + active=active, + alive=alive, + rendered_frame=rendered_frame, + ) + ) diff --git a/cogment_lab/specs/spaces_serialization.py b/cogment_lab/specs/spaces_serialization.py new file mode 100644 index 0000000..f09c2dc --- /dev/null +++ b/cogment_lab/specs/spaces_serialization.py @@ -0,0 +1,100 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import gymnasium as gym +import numpy as np + +from cogment_lab.generated.spaces_pb2 import ( # pylint: disable=import-error + Box, + Dict, + Discrete, + MultiBinary, + MultiDiscrete, + Space, +) + +from .ndarray_serialization import ( + SerializationFormat, + deserialize_ndarray, + serialize_ndarray, +) + + +def serialize_gym_space(space: gym.Space, serialization_format=SerializationFormat.STRUCTURED) -> Space: + if isinstance(space, (gym.spaces.Discrete, gym.spaces.Discrete)): + return Space(discrete=Discrete(n=space.n, start=space.start)) + if isinstance(space, gym.spaces.Box): + low = space.low + high = space.high + return Space( + box=Box( + low=serialize_ndarray(low, serialization_format=serialization_format), + high=serialize_ndarray(high, serialization_format=serialization_format), + ), + ) + + if isinstance(space, gym.spaces.MultiBinary): + if isinstance(space.n, np.ndarray): + size = space.n + elif isinstance(space.n, int): + size = np.array([space.n], dtype=np.dtype("int32")) + else: + size = np.array(space.n, dtype=np.dtype("int32")) + return Space( + multi_binary=MultiBinary(n=serialize_ndarray(size, serialization_format=serialization_format)), + ) + + if isinstance(space, gym.spaces.MultiDiscrete): + nvec = space.nvec + return Space( + multi_discrete=MultiDiscrete(nvec=serialize_ndarray(nvec, serialization_format=serialization_format)), + ) + + if isinstance(space, gym.spaces.Dict): + spaces = [] + for key, gym_sub_space in space.spaces.items(): + spaces.append(Dict.SubSpace(key=key, space=serialize_gym_space(gym_sub_space))) + return Space(dict=Dict(spaces=spaces)) + raise RuntimeError(f"[{type(space)}] is not a supported space type") + + +def deserialize_space(pb_space: Space) -> gym.Space: + space_kind = pb_space.WhichOneof("kind") + if space_kind == "discrete": + discrete_space_pb = pb_space.discrete + return gym.spaces.Discrete(n=discrete_space_pb.n, start=discrete_space_pb.start) + if space_kind == "box": + box_space_pb = pb_space.box + low = deserialize_ndarray(box_space_pb.low) + high = deserialize_ndarray(box_space_pb.high) + return gym.spaces.Box(low=low, high=high, shape=low.shape, dtype=low.dtype) + if space_kind == "multi_binary": + multi_binary_space_pb = pb_space.multi_binary + size = deserialize_ndarray(multi_binary_space_pb.n) + if size.size > 1: + return gym.spaces.MultiBinary(n=size) + return gym.spaces.MultiBinary(n=size[0]) + if space_kind == "multi_discrete": + multi_discrete_space_pb = pb_space.multi_discrete + nvec = deserialize_ndarray(multi_discrete_space_pb.nvec) + return gym.spaces.MultiDiscrete(nvec=nvec) + if space_kind == "dict": + dict_space_pb = pb_space.dict + spaces = [] + for sub_space in dict_space_pb.spaces: + spaces.append((sub_space.key, deserialize_space(sub_space.space))) + + return gym.spaces.Dict(spaces=spaces) + + raise RuntimeError(f"[{space_kind}] is not a supported space kind") diff --git a/cogment_lab/utils/__init__.py b/cogment_lab/utils/__init__.py new file mode 100644 index 0000000..7b0b8d1 --- /dev/null +++ b/cogment_lab/utils/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .import_class import import_object diff --git a/cogment_lab/utils/coltra_utils.py b/cogment_lab/utils/coltra_utils.py new file mode 100644 index 0000000..32ecd9b --- /dev/null +++ b/cogment_lab/utils/coltra_utils.py @@ -0,0 +1,63 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import numpy as np +import torch +from coltra import Agent +from coltra.models import BaseModel + +from cogment_lab.utils.trial_utils import TrialData +from coltra.buffers import OnPolicyRecord, Observation, Action + + +def convert_trial_data_to_coltra(trial_data: TrialData, agent: Agent) -> OnPolicyRecord: + """Convert TrialData to OnPolicyRecord. + + Args: + trial_data (TrialData): TrialData instance + model (BaseModel): Model instance to evaluate values + + Returns: + OnPolicyRecord: Converted OnPolicyRecord instance + """ + obs = trial_data.observations + action = trial_data.actions + reward = trial_data.rewards + done = trial_data.done + # state = None # Assuming 'state' is not provided in TrialData + + # last_value = agent.act(Observation(vector=trial_data.last_observation), get_value=True)[2]["value"] + # value = agent.act(Observation(vector=trial_data.observations), get_value=True)[2]["value"] + + last_value, _ = agent.value(Observation(vector=trial_data.last_observation), ()) + value, _ = agent.value(Observation(vector=trial_data.observations), ()) + + last_value = last_value.detach().squeeze(-1).cpu().numpy() + value = value.detach().squeeze(-1).cpu().numpy() + + # Check if required fields are not None + if obs is None or action is None or reward is None or done is None: + raise ValueError("Missing required fields in TrialData for conversion") + + # Create an OnPolicyRecord instance with the mapped fields + on_policy_record = OnPolicyRecord( + obs=Observation(vector=obs).tensor(), + action=Action(discrete=action).tensor(), + reward=torch.tensor(reward.astype(np.float32)), + value=torch.tensor(value.astype(np.float32)), + done=torch.tensor(done.astype(np.float32)), + last_value=torch.tensor(last_value.astype(np.float32)), + ) + + return on_policy_record diff --git a/cogment_lab/utils/grpc.py b/cogment_lab/utils/grpc.py new file mode 100644 index 0000000..2a8a2d2 --- /dev/null +++ b/cogment_lab/utils/grpc.py @@ -0,0 +1,101 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +from cogment_lab.core import Observation +from cogment_lab.generated import data_pb2 +from google.protobuf.json_format import ParseDict + + +def extend_actor_config( + actor_config_template: dict, + run_id: str, + agent_specs: data_pb2.AgentSpecs, + seed: int, +) -> data_pb2.AgentConfig: + """Extends an actor configuration template with additional parameters. + + Args: + actor_config_template: Template for actor configuration, possibly None. + run_id: Identifier for the run. + agent_specs: Specifications for the environment. + seed: Seed for random number generation. + + Returns: + An instance of AgentConfig with extended configuration. + """ + config = data_pb2.AgentConfig() + if actor_config_template: + ParseDict(actor_config_template, config) + config.run_id = run_id + config.agent_specs.CopyFrom(agent_specs) + config.seed = seed + return config + + +def create_value(val: str | int | float) -> data_pb2.Value: + """Creates a Value protobuf message from a Python value. + + Args: + val: The input string, integer, or float. + + Returns: + A Value protobuf message containing the input value. + """ + value_message = data_pb2.Value() + if isinstance(val, str): + value_message.string_value = val + elif isinstance(val, int): + value_message.int_value = val + elif isinstance(val, float): + value_message.float_value = val + else: + raise ValueError("Unsupported type") + return value_message + + +def get_env_config( + run_id: str | None = None, + render: bool | None = None, + render_width: int | None = None, + seed: int | None = None, + flatten: bool | None = None, + reset_args_dict: dict[str, str | int | float] | None = None, +) -> data_pb2.EnvironmentConfig: + """Generates an EnvironmentConfig protobuf message. + + Args: + run_id: Identifier for the run. + render: Whether to render the environment. + render_width: Render width if rendering. + seed: Random seed. + flatten: Whether to flatten the observation. + reset_args_dict: Dictionary of reset argument values. + + Returns: + An EnvironmentConfig protobuf message. + """ + env_config = data_pb2.EnvironmentConfig() + + env_config.run_id = run_id + env_config.render = render + env_config.render_width = render_width + env_config.seed = seed + env_config.flatten = flatten + + for key, val in reset_args_dict.items(): + env_config.reset_args[key].CopyFrom(create_value(val)) + + return env_config diff --git a/cogment_lab/utils/import_class.py b/cogment_lab/utils/import_class.py new file mode 100644 index 0000000..4cb4ddf --- /dev/null +++ b/cogment_lab/utils/import_class.py @@ -0,0 +1,29 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from importlib import import_module + + +def import_object(class_name: str): + """Imports an object from a module based on a string + + Args: + class_name (str): The full path to the object e.g. "package.module.Class" + + Returns: + object: The imported object + """ + module_path, class_name = class_name.rsplit(".", 1) + module = import_module(module_path) + return getattr(module, class_name) diff --git a/cogment_lab/utils/runners.py b/cogment_lab/utils/runners.py new file mode 100644 index 0000000..b42ac6e --- /dev/null +++ b/cogment_lab/utils/runners.py @@ -0,0 +1,61 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import logging +import os +import subprocess +import sys + + +def setup_logging(log_file: str): + """ + Set up logging to file and stdout/stderr. + + Args: + log_file: Path to log file + """ + # Redirect stdout and stderr to log file + dirname = os.path.dirname(log_file) + if dirname: + os.makedirs(dirname, exist_ok=True) + with open(log_file, "a") as f: + os.dup2(f.fileno(), sys.stdout.fileno()) + os.dup2(f.fileno(), sys.stderr.fileno()) + + # Configure logging + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", + handlers=[logging.FileHandler(log_file)], # , logging.StreamHandler()], + ) + + +def process_cleanup(): + """ + Clean up any leftover processes related to cogment_lab. + """ + try: + pid = os.getpid() + + command = ( + f"ps aux | grep 'cogment-lab' | grep 'multiprocessing' | grep -v grep | " + f"awk '{{if ($3 != {pid}) print $2}}' | xargs -r kill -9" + ) + + subprocess.run(command, shell=True, check=True) + print("Processes terminated successfully.") + except subprocess.CalledProcessError as e: + print(f"An error occurred: {e}") diff --git a/cogment_lab/utils/trial_utils.py b/cogment_lab/utils/trial_utils.py new file mode 100644 index 0000000..b087cb8 --- /dev/null +++ b/cogment_lab/utils/trial_utils.py @@ -0,0 +1,415 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import dataclass, field +from typing import Literal, Any + +import cogment +import numpy as np +from cogment import ActorParameters +from cogment.datastore import Datastore, DatastoreSample +import gymnasium as gym + +from cogment_lab.core import CogmentEnv +from cogment_lab.generated import cog_settings +from cogment_lab.specs import AgentSpecs +from cogment_lab.utils.grpc import extend_actor_config + + +def get_actor_params( + name: str, + implementation: str, + agent_specs: AgentSpecs, + endpoint: str = "grpc://localhost:9002", + base_params: dict | None = None, + run_id: str = "run_id", + seed: int = 0, +) -> ActorParameters: + """ + Create and return actor parameters for a given actor. + + Args: + name (str): The name of the actor. + implementation (str): The implementation type of the actor. + agent_specs (AgentSpecs): The agent specifications. + endpoint (str): The endpoint URL, defaults to "grpc://localhost:9002". + base_params (dict | None): Base parameters for the actor, optional. + run_id (str): The run ID, defaults to "run_id". + seed (int): The seed for randomness, defaults to 0. + + Returns: + ActorParameters: The configured actor parameters. + """ + if base_params is None: + base_params = {} + params = ActorParameters( + cog_settings, + name=name, + class_name="player", + endpoint=endpoint, + implementation=implementation, + config=extend_actor_config(base_params, run_id, agent_specs.serialize(), seed), + ) + return params + + +@dataclass +class TrialData: + """ + Dataclass to store structured trial data for reinforcement learning. + + Attributes: + observations (np.ndarray | dict[str, np.ndarray] | None): Observations from the trial. + actions (np.ndarray | dict[str, np.ndarray] | None): Actions taken during the trial. + rewards (np.ndarray | None): Rewards received during the trial. + done (np.ndarray | None): Done flags for each step of the trial. + next_observations (np.ndarray | dict[str, np.ndarray] | None): Next observations after each step. + last_observation (np.ndarray | dict[str, np.ndarray] | None): The final observation of the trial. + """ + + observations: np.ndarray | dict[str, np.ndarray] | None = field(default=None) + actions: np.ndarray | dict[str, np.ndarray] | None = field(default=None) + rewards: np.ndarray | None = field(default=None) + done: np.ndarray | None = field(default=None) + next_observations: np.ndarray | dict[str, np.ndarray] | None = field(default=None) + last_observation: np.ndarray | dict[str, np.ndarray] | None = field(default=None) + + +def initialize_buffer(space: gym.Space | None, length: int) -> np.ndarray | dict[str, np.ndarray]: + """ + Initializes a buffer based on the given gym space and length. + + Args: + space (gym.Space | None): The gym space to base the buffer on. If None, an empty buffer is created. + length (int): The length of the buffer. + + Returns: + Union[np.ndarray, Dict[str, np.ndarray]]: The initialized buffer, either as an ndarray or a dictionary of ndarrays. + """ + if space is None: + return np.empty((length,), dtype=np.float32) + elif isinstance(space, gym.spaces.Dict): + return {key: np.empty((length,) + space[key].shape, dtype=space[key].dtype) for key in space.spaces.keys()} + elif isinstance(space, gym.spaces.Tuple): + return {i: np.empty((length,) + space[i].shape, dtype=space[i].dtype) for i in range(len(space.spaces))} + else: # Simple space + return np.empty((length,) + space.shape, dtype=space.dtype) + + +def write_to_buffer( + buffer: np.ndarray | dict[str, np.ndarray], + data: np.ndarray | dict[str, Any], + idx: int, +): + """ + Writes data to a specified index in the buffer. + + Args: + buffer (np.ndarray | dict[str, np.ndarray]): The buffer to write data to. + data (np.ndarray | dict[str, Any]): The data to write. + idx (int): The index at which to write the data. + + """ + if isinstance(buffer, dict): + for key in buffer.keys(): + buffer[key][idx] = data[key] + else: + buffer[idx] = data + + +async def format_data( + datastore: Datastore, + trial_id: str, + fields: Sequence[str] = ( + "observations", + "actions", + "done", + "next_observations", + "last_observation", + ), + agent_specs: AgentSpecs | None = None, + env: CogmentEnv | None = None, +) -> TrialData: + """ + Formats trial data from a Cogment trial into a structured format for reinforcement learning. + + Args: + datastore (Datastore): The datastore to fetch trial data from. + trial_id (str): The identifier of the trial. + fields (List[str]): The list of fields to include in the formatted data. + agent_specs (Optional[EnvironmentSpecs]): The environment specifications, optional. + env (Optional[CogmentEnv]): A Cogment environment, optional. + + Returns: + TrialData: The formatted trial data. + + Raises: + AssertionError: If both agent_specs and env are None. + """ + assert agent_specs is not None or env is not None, "Either agent_specs or env must be provided" + assert agent_specs is None or env is None, "Only one of agent_specs and env can be provided" + + if agent_specs is None: + agent_specs = env.agent_specs + + trials = await datastore.get_trials([trial_id]) + samples = [] + async for sample in datastore.all_samples(trials): + samples.append(sample) + if sample.trial_state == cogment.TrialState.ENDED: + break + # if len(samples) >= sample_count: + # break + + data = extract_data_from_samples(samples, fields, agent_specs) + + return data + + +def extract_data_from_samples( + samples: list[DatastoreSample], + fields: Sequence[str] = ( + "observations", + "actions", + "rewards", + "done", + "next_observations", + "last_observation", + ), + agent_specs: AgentSpecs | None = None, + actor_name: str = "gym", +) -> TrialData: + """ + Extracts trial data into a TrialData instance from a list of DatastoreSamples. + + Args: + samples (list[DatastoreSample]): The samples to extract data from. + fields (Sequence[str]): The fields to extract into the TrialData. + agent_specs (AgentSpecs | None): The environment specifications. + actor_name (str): The name of the actor to extract data for. + + Returns: + TrialData: The extracted trial data. + """ + sample_count = len(samples) + if sample_count == 0: + raise ValueError("No samples provided") + + cog_observation_space = agent_specs.get_observation_space() + observation_space = cog_observation_space.gym_space + + cog_action_space = agent_specs.get_action_space() + action_space = cog_action_space.gym_space + + data = TrialData() + if "observations" in fields: + data.observations = initialize_buffer(observation_space, sample_count - 1) + if "actions" in fields: + data.actions = initialize_buffer(action_space, sample_count - 1) + if "rewards" in fields: + data.rewards = initialize_buffer(None, sample_count - 1) + if "done" in fields: + data.done = initialize_buffer(None, sample_count - 1) + data.done[-1] = True + if "next_observations" in fields: + data.next_observations = initialize_buffer(observation_space, sample_count - 1) + if "last_observation" in fields: + data.last_observation = initialize_buffer(observation_space, 1) + + for i, sample in enumerate(samples[:-1]): + if "observations" in fields: + obs = cog_observation_space.deserialize(sample.actors_data[actor_name].observation).value + write_to_buffer(data.observations, obs, i) + if "actions" in fields: + action = cog_action_space.deserialize(sample.actors_data[actor_name].action).value + write_to_buffer(data.actions, action, i) + if "rewards" in fields: + write_to_buffer(data.rewards, sample.actors_data[actor_name].reward, i) + if "done" in fields and i < sample_count - 2: + write_to_buffer(data.done, False, i) + if "next_observations" in fields: + next_obs = cog_observation_space.deserialize(samples[i + 1].actors_data[actor_name].observation).value + write_to_buffer(data.next_observations, next_obs, i) + if "last_observation" in fields: + last_obs = agent_specs.get_observation_space().deserialize(samples[-1].actors_data[actor_name].observation).value + write_to_buffer(data.last_observation, last_obs, 0) + + return data + + +def extract_rewards_from_samples( + samples: list[DatastoreSample], + agent_specs: AgentSpecs | None = None, + actor_name: str = "gym", +) -> TrialData: + """ + Extracts rewards from trial samples into a TrialData instance. + + Args: + samples (list[DatastoreSample]): The samples to extract rewards from. + agent_specs (AgentSpecs | None): The environment specifications. + actor_name (str): The name of the actor to extract rewards for. + + Returns: + TrialData: The extracted rewards. + """ + sample_count = len(samples) + + rewards = initialize_buffer(None, sample_count - 1) + + for i, sample in enumerate(samples[:-1]): + write_to_buffer(rewards, sample.actors_data[actor_name].reward, i) + + return rewards + + +def concat_trial_field( + field_data: list[np.ndarray | dict[str, np.ndarray] | None], +) -> np.ndarray | dict[str, np.ndarray] | None: + """ + Concatenates a list of fields (either np.ndarray or dict of np.ndarrays) from TrialData instances. + Filters out None values before concatenation. + + Args: + field_data: List of fields to be concatenated. + + Returns: + Concatenated field as np.ndarray or dict of np.ndarrays, or None if all elements are None. + """ + # Filter out None values + valid_field_data = [data for data in field_data if data is not None] + + if not valid_field_data: + return None + + if all(isinstance(data, np.ndarray) for data in valid_field_data): + return np.concatenate(valid_field_data, axis=0) + elif all(isinstance(data, dict) for data in valid_field_data): + keys = valid_field_data[0].keys() + return {key: np.concatenate([data[key] for data in valid_field_data], axis=0) for key in keys} + else: + raise TypeError("Inconsistent field types in TrialData list.") + + +def concatenate(trial_data_list: list[TrialData]) -> TrialData: + """ + Concatenates a list of TrialData instances into a single TrialData instance. + + Args: + trial_data_list: List of TrialData instances to be concatenated. + + Returns: + A single concatenated TrialData instance. + """ + observations = concat_trial_field([trial.observations for trial in trial_data_list]) + actions = concat_trial_field([trial.actions for trial in trial_data_list]) + rewards = concat_trial_field([trial.rewards for trial in trial_data_list]) + done = concat_trial_field([trial.done for trial in trial_data_list]) + next_observations = concat_trial_field([trial.next_observations for trial in trial_data_list]) + + # Handle 'last_observation' separately + last_trial = trial_data_list[-1] + last_observation = ( + last_trial.last_observation if last_trial.last_observation is not None else last_trial.next_observations[-1] + ) + + return TrialData( + observations=observations, + actions=actions, + rewards=rewards, + done=done, + next_observations=next_observations, + last_observation=last_observation, + ) + + +async def format_data_multiagent( + datastore: Datastore, + trial_id: str, + actor_agent_specs: dict[str, AgentSpecs], + fields: Sequence[str] = ( + "observations", + "actions", + "rewards", + "done", + "next_observations", + "last_observation", + ), +) -> dict[str, TrialData]: + """ + Formats trial data from a multiagent Cogment trial into structured formats for reinforcement learning. + + Args: + datastore (Datastore): The datastore to fetch trial data from. + trial_id (str): The identifier of the trial. + actor_agent_specs (dict[str, EnvironmentSpecs]): A dictionary mapping actor IDs to their environment specifications. + fields (List[str]): The list of fields to include in the formatted data. + + Returns: + dict[str, TrialData]: A dictionary mapping actor IDs to their formatted trial data. + """ + trials = [] + while len(trials) == 0: + try: + trials = await datastore.get_trials([trial_id]) + except cogment.CogmentError: + continue + + # Initialize a dictionary to store samples for each actor + actor_samples = {actor_id: [] for actor_id in actor_agent_specs.keys()} + actor_reward_samples = {actor_id: [] for actor_id in actor_agent_specs.keys()} + + # Get all samples + all_samples = [] + async for sample in datastore.all_samples(trials): + all_samples.append(sample) + + # Sort according to tick_id -- this might not be necessary with some version of cogment + all_samples.sort(key=lambda x: x.tick_id) + + for sample in all_samples: + for actor_id in actor_agent_specs.keys(): + # Add the sample to the list for an actor if the observation for that actor is not None + if ( + sample.actors_data.get(actor_id) + and sample.actors_data[actor_id].observation is not None + and sample.actors_data[actor_id].observation.value is not None + and sample.actors_data[actor_id].observation.value.raw_data != b"" + ): + actor_samples[actor_id].append(sample) + + if ( + sample.actors_data.get(actor_id) + and sample.actors_data[actor_id].reward is not None + and sample.actors_data[actor_id].reward == sample.actors_data[actor_id].reward # Check for NaN + ): + actor_reward_samples[actor_id].append(sample) + + # Extract data for each actor + actor_data = {} + + for actor_id, samples in actor_samples.items(): + actor_data[actor_id] = extract_data_from_samples( + samples, fields, actor_agent_specs[actor_id], actor_name=actor_id + ) + + for actor_id, reward_samples in actor_reward_samples.items(): + actor_data[actor_id].rewards = extract_rewards_from_samples( + reward_samples, actor_agent_specs[actor_id], actor_name=actor_id + ) + + return actor_data diff --git a/cogment_lab/utils/yaml_utils.py b/cogment_lab/utils/yaml_utils.py new file mode 100644 index 0000000..c57ec0a --- /dev/null +++ b/cogment_lab/utils/yaml_utils.py @@ -0,0 +1,181 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import gymnasium as gym +import numpy as np +import yaml + + +def gym_space_constructors(): + """Registers YAML constructors for Gym spaces. + + This allows Gym spaces to be created automatically from YAML files + by registering constructors for each space type. + """ + + def box_constructor(loader, node): + """YAML constructor for Box spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A Box space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + return gym.spaces.Box( + low=np.array(values.get("low", -np.inf)), + high=np.array(values.get("high", np.inf)), + shape=values.get("shape", None), + dtype=values.get("dtype", np.float32), + seed=values.get("seed", None), + ) + + def discrete_constructor(loader, node): + """YAML constructor for Discrete spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A Discrete space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + return gym.spaces.Discrete(n=values["n"], seed=values.get("seed", None), start=values.get("start", 0)) + + def multibinary_constructor(loader, node): + """YAML constructor for MultiBinary spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A MultiBinary space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + return gym.spaces.MultiBinary(n=values["n"], seed=values.get("seed", None)) + + def multidiscrete_constructor(loader, node): + """YAML constructor for MultiDiscrete spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A MultiDiscrete space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + return gym.spaces.MultiDiscrete( + nvec=np.array(values["nvec"]), + dtype=values.get("dtype", np.int64), + seed=values.get("seed", None), + start=values.get("start", None), + ) + + def text_constructor(loader, node): + """YAML constructor for Text spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A Text space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + return gym.spaces.Text( + max_length=values["max_length"], + min_length=values.get("min_length", 1), + charset=values.get("charset", "alphanumeric"), + seed=values.get("seed", None), + ) + + def dict_constructor(loader, node): + """YAML constructor for Dict spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A Dict space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + spaces = values.pop("spaces", None) + seed = values.pop("seed", None) + return gym.spaces.Dict(spaces=spaces, seed=seed, **values) + + def tuple_constructor(loader, node): + """YAML constructor for Tuple spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A Tuple space constructed from the YAML node. + """ + values = loader.construct_sequence(node) + spaces = values.pop("spaces", None) + seed = values.pop("seed", None) + return gym.spaces.Tuple(spaces=spaces, seed=seed) + + def sequence_constructor(loader, node): + """YAML constructor for Sequence spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A Sequence space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + space = values.get("space") + seed = values.get("seed", None) + stack = values.get("stack", False) + return gym.spaces.Sequence(space=space, seed=seed, stack=stack) + + def graph_constructor(loader, node): + """YAML constructor for Graph spaces. + + Args: + loader: The YAML loader. + node: The YAML node. + + Returns: + A Graph space constructed from the YAML node. + """ + values = loader.construct_mapping(node) + node_space = values.pop("node_space") + edge_space = values.pop("edge_space", None) + seed = values.pop("seed", None) + return gym.spaces.Graph(node_space=node_space, edge_space=edge_space, seed=seed) + + # Register constructors + yaml.add_constructor("!Box", box_constructor) + yaml.add_constructor("!Discrete", discrete_constructor) + yaml.add_constructor("!MultiBinary", multibinary_constructor) + yaml.add_constructor("!MultiDiscrete", multidiscrete_constructor) + yaml.add_constructor("!Text", text_constructor) + + yaml.add_constructor("!Dict", dict_constructor) + yaml.add_constructor("!Tuple", tuple_constructor) + yaml.add_constructor("!Graph", graph_constructor) + yaml.add_constructor("!Sequence", sequence_constructor) + yaml.add_constructor("!Tuple", tuple_constructor) diff --git a/examples/demos/active-lunar/lunar-active.py b/examples/demos/active-lunar/lunar-active.py new file mode 100644 index 0000000..3c19a47 --- /dev/null +++ b/examples/demos/active-lunar/lunar-active.py @@ -0,0 +1,201 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import datetime +import random + +import numpy as np +import torch +import wandb +from coltra.models import FCNetwork +from torch import optim +from tqdm import trange +from typarse import BaseParser + +from cogment_lab.actors import ConstantActor +from cogment_lab.actors.nn_actor import NNActor, BoltzmannActor +from cogment_lab.envs import AECEnvironment, GymEnvironment +from cogment_lab.process_manager import Cogment +from cogment_lab.utils.runners import process_cleanup +from shared import ReplayBuffer, get_current_eps, dqn_loss + + +class Parser(BaseParser): + wandb_project: str = "test" + wandb_name: str = "test" + + env_name: str = "LunarLander-v2" + + batch_size: int = 128 + gamma: float = 0.99 + replay_buffer_capacity: int = 50000 + learning_rate: float = 6.3e-4 + num_episodes: int = 500 + seed: int = 0 + + human_episodes: int = 100 + + +async def main(): + args = Parser() + + process_cleanup() + + torch.manual_seed(args.seed) + np.random.seed(args.seed) + random.seed(args.seed) + + wandb.init(project=args.wandb_project, name=args.wandb_name) + + wandb.config.batch_size = args.batch_size + wandb.config.gamma = args.gamma + wandb.config.replay_buffer_capacity = args.replay_buffer_capacity + wandb.config.learning_rate = args.learning_rate + wandb.config.num_episodes = args.num_episodes + wandb.config.seed = args.seed + wandb.config.env_name = args.env_name + + logpath = f"logs/logs-{datetime.datetime.now().isoformat()}" + + cog = Cogment(log_dir=logpath) + + cenv = AECEnvironment( + env_path="cogment_lab.envs.conversions.teacher.GymTeacherAEC", + make_kwargs={"gym_env_name": args.env_name, "gym_make_kwargs": {}, "render_mode": "rgb_array"}, + render=True, + reinitialize=True, + ) + + await cog.run_env(cenv, "lunar", port=9021, log_file="env-aec.log") + + obs_len = cenv.env.observation_space("gym").shape[0] + + cenv_single = GymEnvironment(env_id=args.env_name, reinitialize=True, render=True) + + await cog.run_env(cenv_single, "lunar-single", port=9022, log_file="env-gym.log") + + # Create and run the learner network + + replay_buffer = ReplayBuffer(args.replay_buffer_capacity, obs_len) + + # Run the agent + network = FCNetwork( + input_size=obs_len, output_sizes=[cenv.env.action_space("gym").n], hidden_sizes=[256, 256], activation="tanh" + ) + + actor = NNActor(network, "cpu") + optimizer = optim.Adam(network.parameters(), lr=args.learning_rate) + + cog.run_local_actor(actor, "dqn", port=9012, log_file="dqn.log") + + # Run the human teacher + + # # Lunar lander actions + actions = { + "no-op": {"active": 0, "action": 0}, + "ArrowDown": {"active": 1, "action": 0}, + "ArrowRight": {"active": 1, "action": 1}, + "ArrowUp": {"active": 1, "action": 2}, + "ArrowLeft": {"active": 1, "action": 3}, + } + + await cog.run_web_ui(actions=actions, log_file="human.log", fps=60) + + print(f"Launched web UI at http://localhost:8000") + + total_timesteps = 0 + ep_rewards = [] + + for episode in (pbar := trange(args.num_episodes)): + actor.set_eps(get_current_eps(episode)) + if episode == args.human_episodes: + cog.stop_service("lunar") + + if episode < args.human_episodes: + trial_id = await cog.start_trial( + env_name="lunar", + actor_impls={"gym": "dqn", "teacher": "web_ui"}, + session_config={"render": True, "seed": episode}, + ) + else: + trial_id = await cog.start_trial( + env_name="lunar-single", actor_impls={"gym": "dqn"}, session_config={"render": True, "seed": episode} + ) + + trial_data_task = asyncio.create_task(cog.get_trial_data(trial_id)) + + gradient_updates = 0 + + + trial_data = await trial_data_task + + + # Logging + dqn_data = trial_data["gym"] + + total_reward = dqn_data.rewards.sum() + pbar.set_description(f"mean_reward: {total_reward:.3}") + ep_rewards.append(total_reward) + + total_timesteps += len(dqn_data.rewards) + + # Add data to replay buffer + + for t in range(len(dqn_data.done)): + state = dqn_data.observations[t] + dqn_action = dqn_data.actions[t] + try: + human_data = trial_data["teacher"] + human_active = human_data.actions["active"][t] + human_action = human_data.actions["action"][t] + except (IndexError, KeyError): + human_active = 0 + human_action = 0 + + action = human_action if human_active == 1 else dqn_action + + + reward = dqn_data.rewards[t] + next_state = dqn_data.next_observations[t] + done = dqn_data.done[t] + + replay_buffer.push(state, action, reward, next_state, done) + + # Train, once per datapoint + + if len(replay_buffer) > args.batch_size: + batch = replay_buffer.sample(args.batch_size) + + loss = dqn_loss(network, batch, args.gamma) + + optimizer.zero_grad() + loss.backward() + optimizer.step() + + gradient_updates += 1 + + log_dict = { + "episode": episode, + "reward": total_reward, + "ep_length": len(dqn_data.rewards), + "total_timesteps": total_timesteps, + "gradient_updates": gradient_updates, + } + + wandb.log(log_dict) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/demos/active-lunar/lunar-base.py b/examples/demos/active-lunar/lunar-base.py new file mode 100644 index 0000000..fee5683 --- /dev/null +++ b/examples/demos/active-lunar/lunar-base.py @@ -0,0 +1,157 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import datetime +import random + +import numpy as np +import torch +import wandb +from coltra.models import FCNetwork +from torch import optim +from tqdm import trange +from typarse import BaseParser + +from cogment_lab.actors import ConstantActor +from cogment_lab.actors.nn_actor import NNActor, BoltzmannActor +from cogment_lab.envs import AECEnvironment, GymEnvironment +from cogment_lab.process_manager import Cogment +from cogment_lab.utils.runners import process_cleanup +from shared import ReplayBuffer, get_current_eps, dqn_loss + + +class Parser(BaseParser): + wandb_project: str = "test" + wandb_name: str = "test" + + env_name: str = "LunarLander-v2" + + batch_size: int = 128 + gamma: float = 0.99 + replay_buffer_capacity: int = 50000 + learning_rate: float = 6.3e-4 + num_episodes: int = 500 + seed: int = 0 + +async def main(): + args = Parser() + + process_cleanup() + + torch.manual_seed(args.seed) + np.random.seed(args.seed) + random.seed(args.seed) + + wandb.init(project=args.wandb_project, name=args.wandb_name) + + wandb.config.batch_size = args.batch_size + wandb.config.gamma = args.gamma + wandb.config.replay_buffer_capacity = args.replay_buffer_capacity + wandb.config.learning_rate = args.learning_rate + wandb.config.num_episodes = args.num_episodes + wandb.config.seed = args.seed + wandb.config.env_name = args.env_name + + logpath = f"logs/logs-{datetime.datetime.now().isoformat()}" + + cog = Cogment(log_dir=logpath) + + + cenv = GymEnvironment(env_id=args.env_name, reinitialize=True, render=True) + + obs_len = cenv.env.observation_space.shape[0] + + await cog.run_env(cenv, "lunar", port=9021, log_file="env-gym.log") + + # Create and run the learner network + + replay_buffer = ReplayBuffer(args.replay_buffer_capacity, obs_len) + + # Run the agent + network = FCNetwork( + input_size=obs_len, output_sizes=[cenv.env.action_space.n], hidden_sizes=[256, 256], activation="tanh" + ) + + actor = NNActor(network, "cpu") + optimizer = optim.Adam(network.parameters(), lr=args.learning_rate) + + cog.run_local_actor(actor, "dqn", port=9012, log_file="dqn.log") + + total_timesteps = 0 + ep_rewards = [] + + for episode in (pbar := trange(args.num_episodes)): + actor.set_eps(get_current_eps(episode)) + if episode == args.human_episodes: + cog.stop_service("lunar") + + + trial_id = await cog.start_trial( + env_name="lunar", actor_impls={"gym": "dqn"}, session_config={"render": True, "seed": episode} + ) + + trial_data_task = asyncio.create_task(cog.get_trial_data(trial_id)) + + gradient_updates = 0 + + trial_data = await trial_data_task + + # Logging + dqn_data = trial_data["gym"] + + total_reward = dqn_data.rewards.sum() + pbar.set_description(f"mean_reward: {total_reward:.3}") + ep_rewards.append(total_reward) + + total_timesteps += len(dqn_data.rewards) + + # Add data to replay buffer + + for t in range(len(dqn_data.done)): + state = dqn_data.observations[t] + action = dqn_data.actions[t] + + reward = dqn_data.rewards[t] + next_state = dqn_data.next_observations[t] + done = dqn_data.done[t] + + replay_buffer.push(state, action, reward, next_state, done) + + # Train, once per datapoint + + if len(replay_buffer) > args.batch_size: + batch = replay_buffer.sample(args.batch_size) + + loss = dqn_loss(network, batch, args.gamma) + + optimizer.zero_grad() + loss.backward() + optimizer.step() + + gradient_updates += 1 + + log_dict = { + "episode": episode, + "reward": total_reward, + "ep_length": len(dqn_data.rewards), + "total_timesteps": total_timesteps, + "gradient_updates": gradient_updates, + } + + wandb.log(log_dict) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/demos/active-lunar/shared.py b/examples/demos/active-lunar/shared.py new file mode 100644 index 0000000..0a3330e --- /dev/null +++ b/examples/demos/active-lunar/shared.py @@ -0,0 +1,144 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import numpy as np +import torch +from torch import nn +import hashlib + +from cogment_lab import Cogment + +EPS_INIT = 0.9 +EPS_FINAL = 0.001 +EPS_DECAY = 300 + + +class ReplayBuffer: + def __init__(self, capacity: int, obs_size: int, seed: int = 0): + self.capacity = capacity + self.buffer_counter = 0 + + self.rng = np.random.default_rng(seed) + + # Pre-allocate memory + self.states = np.zeros((capacity, obs_size), dtype=np.float32) + self.actions = np.zeros(capacity, dtype=np.int32) + self.rewards = np.zeros(capacity, dtype=np.float32) + self.next_states = np.zeros((capacity, obs_size), dtype=np.float32) + self.dones = np.zeros(capacity, dtype=np.float32) + + def push(self, state: np.ndarray, action: int, reward: float, next_state: np.ndarray, done: float): + index = self.buffer_counter % self.capacity + + self.states[index] = state + self.actions[index] = action + self.rewards[index] = reward + self.next_states[index] = next_state + self.dones[index] = done + + self.buffer_counter += 1 + + def sample(self, batch_size: int) -> tuple[np.ndarray, ...]: + max_buffer_size = min(self.buffer_counter, self.capacity) + batch_indices = self.rng.choice(max_buffer_size, batch_size, replace=False) + + return ( + self.states[batch_indices], + self.actions[batch_indices], + self.rewards[batch_indices], + self.next_states[batch_indices], + self.dones[batch_indices], + ) + + def __len__(self): + return min(self.buffer_counter, self.capacity) + + +def get_current_eps( + current_step: int, eps_start: float = EPS_INIT, eps_final: float = EPS_FINAL, eps_decay_duration: int = EPS_DECAY +) -> float: + """ + Calculate the epsilon value for epsilon-greedy exploration in DQN. + + Parameters: + current_step (int): The current step index. + eps_start (float): The starting value of epsilon. + eps_final (float): The final value of epsilon. + eps_decay_duration (int): The number of steps over which epsilon is decayed linearly. + + Returns: + float: The current epsilon value. + """ + current_step = min(current_step, eps_decay_duration) + + decay_rate = (eps_start - eps_final) / eps_decay_duration + + current_epsilon = eps_start - decay_rate * current_step + + current_epsilon = max(current_epsilon, eps_final) + + return current_epsilon + + +def hash_model(model: nn.Module): + model_state = model.state_dict() + model_weights = [] + + for key, value in model_state.items(): + model_weights.append(value.cpu().numpy().tobytes()) + + model_hash = hashlib.sha256(b"".join(model_weights)).hexdigest() + + return model_hash + + +def dqn_loss(model: nn.Module, batch: tuple[np.ndarray, ...], γ: float) -> torch.Tensor: + states, actions, rewards, next_states, dones = batch + + states = torch.from_numpy(states) + actions = torch.from_numpy(actions).to(torch.int64) + rewards = torch.from_numpy(rewards) + next_states = torch.from_numpy(next_states) + dones = torch.from_numpy(dones) + + current_q_values = model(states)[0].gather(1, actions.unsqueeze(1)).squeeze(1) + next_q_values = model(next_states)[0].max(1)[0] + expected_q_values = rewards + γ * next_q_values * (1 - dones) + + loss = nn.MSELoss()(current_q_values, expected_q_values.detach()) + + return loss + + +async def evaluate_model( + cog: Cogment, env_name: str, actor_impls: dict[str, str], num_episodes: int = 10 +) -> tuple[float, float]: + total_rewards = [] + episode_lengths = [] + + for ep in range(num_episodes): + trial_id = await cog.start_trial( + env_name=env_name, actor_impls=actor_impls, session_config={"render": False, "seed": 10_000 + ep} + ) + + trial_data = await cog.get_trial_data(trial_id) + dqn_data = trial_data["gym"] + + total_rewards.append(dqn_data.rewards.sum()) + episode_lengths.append(len(dqn_data.rewards)) + + mean_reward = np.mean(total_rewards) + mean_length = np.mean(episode_lengths) + + return mean_reward, mean_length diff --git a/examples/docs_example.py b/examples/docs_example.py new file mode 100644 index 0000000..b2ea9f0 --- /dev/null +++ b/examples/docs_example.py @@ -0,0 +1,87 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import datetime + +from cogment_lab import Cogment +from cogment_lab.actors import RandomActor, ConstantActor +from cogment_lab.envs import GymEnvironment +from cogment_lab.utils.runners import process_cleanup + +LUNAR_LANDER_ACTIONS = ["no-op", "ArrowRight", "ArrowUp", "ArrowLeft"] + + +async def main(): + # Create the global process manager + process_cleanup() + + logpath = f"logs/logs-{datetime.datetime.now().isoformat()}" + + cog = Cogment(log_dir=logpath) + + # Launch the environment + env = GymEnvironment( + env_id="LunarLander-v2", # ID of a Gymnasium environment + render=True, # True if we want to see the rendering at some point + ) + + await cog.run_env( + env=env, + env_name="lunar", + port=9011, # Typically, we use ports 901x for environments and 902x for actors + log_file="env.log", + ) + + # Launch a constant actor + constant_actor = ConstantActor(0) + + await cog.run_actor(actor=constant_actor, actor_name="constant", port=9021, log_file="random.log") + + # Launch a random actor + random_actor = RandomActor(env.env.action_space) + + await cog.run_actor(actor=random_actor, actor_name="random", port=9022, log_file="constant.log") + + # Launch an episode + episode_id = await cog.start_trial( + env_name="lunar", # Which environment + actor_impls={"gym": "random"}, # Which actor(s) will act + ) + + # Compute the total reward of the episode + data = await cog.get_trial_data(trial_id=episode_id) + random_reward = data["gym"].rewards.sum() + + print(f"Random agent's reward: {random_reward}") + + # Launch a human actor UI + await cog.run_web_ui(actions=LUNAR_LANDER_ACTIONS, log_file="human.log", fps=60) + + episode_id = await cog.start_trial(env_name="lunar", actor_impls={"gym": "web_ui"}, session_config={"render": True}) + + print("Go to http://localhost:8000 in your browser and see how well you do!") + + data = await cog.get_trial_data(trial_id=episode_id) + + human_reward = data["gym"].rewards.sum() + + if human_reward > random_reward: + print(f"Good job! You beat a random agent with a score of {human_reward}!") + else: + print(f"Awkward... You lost with a score of {human_reward}...") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/gymnasium/bc-training.ipynb b/examples/gymnasium/bc-training.ipynb new file mode 100644 index 0000000..732e3b4 --- /dev/null +++ b/examples/gymnasium/bc-training.ipynb @@ -0,0 +1,746 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T17:26:58.848013Z", + "start_time": "2023-12-13T17:26:56.922017Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import torch\n", + "import torch.nn.functional as F\n", + "from coltra import HomogeneousGroup\n", + "from coltra.buffers import Observation\n", + "from coltra.models import MLPModel\n", + "from coltra.policy_optimization import CrowdPPOptimizer\n", + "from tqdm import trange\n", + "\n", + "from cogment_lab.actors import ColtraActor\n", + "from cogment_lab.envs.gymnasium import GymEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.coltra_utils import convert_trial_data_to_coltra\n", + "from cogment_lab.utils.runners import process_cleanup\n", + "from cogment_lab.utils.trial_utils import concatenate\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:26:58.911299Z", + "start_time": "2023-12-13T17:26:58.848302Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:26:59.807004Z", + "start_time": "2023-12-13T17:26:59.288309Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T18:26:59.286916\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:27:03.568016Z", + "start_time": "2023-12-13T17:27:01.302459Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# We'll train on \n", + "\n", + "cenv = GymEnvironment(\n", + " env_id=\"MountainCar-v0\",\n", + " render=True,\n", + " make_kwargs={\"max_episode_steps\": 401},\n", + ")\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"mcar\",\n", + " port=9011, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:27:07.411552Z", + "start_time": "2023-12-13T17:27:05.008940Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Create a model using coltra\n", + "\n", + "model = MLPModel(\n", + " config={\n", + " \"hidden_sizes\": [64, 64],\n", + " }, \n", + " observation_space=cenv.env.observation_space, \n", + " action_space=cenv.env.action_space\n", + ")\n", + "\n", + "# Put the model in shared memory so that the actor can access it\n", + "model.share_memory()\n", + "actor = ColtraActor(model=model)\n", + "\n", + "\n", + "await cog.run_actor(\n", + " actor=actor,\n", + " actor_name=\"coltra\",\n", + " port=9021,\n", + " log_file=\"actor.log\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "outputs": [ + { + "data": { + "text/plain": "{'mcar': ,\n 'coltra': }" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:27:09.742973Z", + "start_time": "2023-12-13T17:27:09.738867Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:27:15.628829Z", + "start_time": "2023-12-13T17:27:13.486915Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "MOUNTAIN_CAR_ACTIONS = [\"no-op\", \"ArrowLeft\", \"ArrowRight\"]\n", + "\n", + "actions = MOUNTAIN_CAR_ACTIONS\n", + "\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=30)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 10/10 [00:04<00:00, 2.18it/s]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean_reward: -400.1\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "# Estimate random agent performance\n", + "\n", + "episodes = []\n", + "for i in trange(10):\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id)\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + "mean_reward = np.mean([sum(e.rewards) for e in episodes])\n", + "print(f\"mean_reward: {mean_reward}\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:27:27.047222Z", + "start_time": "2023-12-13T17:27:22.445172Z" + } + }, + "id": "b7cde51d7dc0c3a9" + }, + { + "cell_type": "code", + "execution_count": 11, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "mean_reward: -4e+02: 100%|██████████| 10/10 [00:14<00:00, 1.47s/it]\n" + ] + } + ], + "source": [ + "# Train for a bit with PPO\n", + "\n", + "ppo = CrowdPPOptimizer(HomogeneousGroup(actor.agent), config={\n", + " \"gae_lambda\": 0.95,\n", + " \"minibatch_size\": 128,\n", + "})\n", + "\n", + "all_rewards = []\n", + "\n", + "for t in (pbar := trange(10)):\n", + " num_steps = 0\n", + " episodes = []\n", + " while num_steps < 1000:\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id, env_name=\"mcar\")\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + " num_steps += len(data.rewards)\n", + " \n", + " all_data = concatenate(episodes)\n", + " \n", + " record = convert_trial_data_to_coltra(all_data, actor.agent)\n", + " metrics = ppo.train_on_data({\"crowd\": record}, shape=(1,) + record.reward.shape)\n", + " \n", + " mean_reward = metrics[\"crowd/mean_episode_reward\"]\n", + " all_rewards.append(mean_reward)\n", + " pbar.set_description(f\"mean_reward: {mean_reward:.3}\")\n", + " \n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:27:55.773514Z", + "start_time": "2023-12-13T17:27:41.057103Z" + } + }, + "id": "fd1d49a788c06eb4" + }, + { + "cell_type": "code", + "execution_count": 12, + "outputs": [ + { + "data": { + "text/plain": "[]" + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkkAAAGdCAYAAAAGx+eQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYSklEQVR4nO3dfXRT15kv/u+RZEmW8bsdHMc2fknACe/BrcdkIO0agtNhssadWSQFpqlTQkp/0EBNQ3CSErhJyw0EEghpGDqXC0lDITQdeqcDFF9Yacjg3BSCm5IJaYINNmAbY4zlV0mWzu8P+RxbsfwiW9J50fezllYi6Ujaso85z9772c8WRFEUQUREREQ+DEo3gIiIiEiNGCQRERER+cEgiYiIiMgPBklEREREfjBIIiIiIvKDQRIRERGRHwySiIiIiPxgkERERETkh0npBmiZx+PBtWvXEBsbC0EQlG4OERERjYAoimhra0N6ejoMhsHHixgkjcG1a9eQmZmpdDOIiIhoFOrq6pCRkTHo8wySxiA2NhaA94ccFxencGuIiIhoJOx2OzIzM+Xr+GAYJI2BNMUWFxfHIImIiEhjhkuVYeI2ERERkR8MkoiIiIj8YJBERERE5AeDJCIiIiI/GCQRERER+cEgiYiIiMgPBklEREREfjBIIiIiIvKDQRIRERGRH2EJkhwOB2bMmAFBEFBVVeXz3CeffII5c+bAarUiMzMTmzdvHvD6Q4cOIT8/H1arFVOnTsWRI0eG/cz33nsP9957LywWC+68807s3bt3wDGvv/46srOzYbVaUVhYiI8++mi0X5GIiIh0JixB0tq1a5Genj7gcbvdjvnz52PChAk4e/YstmzZgg0bNmD37t3yMadPn8aiRYuwdOlSnDt3DiUlJSgpKcH58+cH/byamhosWLAA3/zmN1FVVYXVq1fj8ccfxx/+8Af5mIMHD6KsrAzPP/88Pv74Y0yfPh3FxcW4fv16cL88ERERaZMYYkeOHBHz8/PFTz/9VAQgnjt3Tn7uF7/4hZiYmCg6HA75saefflqcNGmSfP/hhx8WFyxY4POehYWF4g9+8INBP3Pt2rXi5MmTfR575JFHxOLiYvn+17/+dXHFihXyfbfbLaanp4ubNm0a8XdrbW0VAYitra0jfg0REREpa6TX75BucNvY2Ihly5bh8OHDsNlsA56vrKzE3LlzYTab5ceKi4vx0ksvoaWlBYmJiaisrERZWZnP64qLi3H48OFBP7eyshLz5s0b8JrVq1cDAJxOJ86ePYvy8nL5eYPBgHnz5qGysnLQ93U4HHA4HPJ9u90+6LFjse3452hz9ITkvWl0kmxmLJubC2uUUemmRLyWDif2/FcN2nXwNxJlNOCRr2UiL3Wc0k2JeKIo4lf/rxbVTe1KN4W+ouyBiYi1Riny2SELkkRRRGlpKZYvX46CggJcunRpwDENDQ3IycnxeWz8+PHyc4mJiWhoaJAf639MQ0PDoJ892Gvsdju6urrQ0tICt9vt95gLFy4M+r6bNm3Cxo0bB30+WA78qQ7X2xzDH0hhNT7eiocLMpVuRsR7s/IyXjv5pdLNCJpLNzqw+9ECpZsR8f673o6fHh48jYOU88Nv5GknSFq3bh1eeumlIY/57LPPcPz4cbS1tfmM1mhdeXm5z6iW3W5HZmbwL5ql92WjQwe9ZL34sPomzl5uwReNbUo3hQD89br39zB3Yiqm3hGncGtGr6HVgXc/voIvrnPkQg2+aPT+HrKSbHho+u0Kt4b6s5lDOuk1pIA/ec2aNSgtLR3ymNzcXJw8eRKVlZWwWCw+zxUUFGDJkiXYt28f0tLS0NjY6PO8dD8tLU3+r79jpOf9Gew1cXFxiI6OhtFohNFoDPh9LRbLgO8TCv/fN+4M+WfQyL314WWcvdyC6qYOpZtCgPx7+F7RBPzd3eOHOVq9Glq78e7HV1B7sxPOHg/MJlZkUZI0zXbfnSl4qjhf4daQWgQcJKWmpiI1NXXY43bs2IEXX3xRvn/t2jUUFxfj4MGDKCwsBAAUFRXh2WefhcvlQlSUdyitoqICkyZNQmJionzMiRMn5Hwi6ZiioqJBP7uoqGhAmYD+rzGbzZg1axZOnDiBkpISAIDH48GJEyewcuXK4X8IFFFyU2IAADU3GCQpzeMRcan395DT+3vRqvFxFkRHGdHlcqOupZN5SQqr7j2vcjV+XlFwhazrkpWVhSlTpsi3iRMnAgDy8vKQkZEBAFi8eDHMZjOWLl2KTz/9FAcPHsT27dt9prRWrVqFY8eOYevWrbhw4QI2bNiAM2fO+AQz5eXlePTRR+X7y5cvR3V1NdauXYsLFy7gF7/4Bd555x38+Mc/lo8pKyvDL3/5S+zbtw+fffYZfvjDH6KjowOPPfZYqH4kpFHSxbj2Zidcbo/CrYlsjW3d6HK5YTQIyEwauBhESwRBkM+tGo5SKq5GJ8E3BZdyE30A4uPjcfz4caxYsQKzZs1CSkoK1q9fjyeeeEI+Zvbs2di/fz+ee+45PPPMM7jrrrtw+PBhTJkyRT6mvr4etbW18v2cnBz853/+J3784x9j+/btyMjIwL/927+huLhYPuaRRx5BU1MT1q9fj4aGBsyYMQPHjh0bkMxNlBZnhTXKgG6XB1dauviPqIKkYCIryYYoo/anp3JSY/Df9XaOUipMFMW+ICmVf9/UJ2xBUnZ2NkRRHPD4tGnTcOrUqSFfu3DhQixcuHDQ5/1V0/7GN76Bc+fODfm+K1eu5PQaDctgEJCTMg6f1dtRc6OdQZKC9DYlktf7PaoZJCnqepsDnU7vCGWWxkcoKbi03xUjCgPposzkbWXpbUpEGrWoucEVbkqq1tkIJQUPzwaiEchhj18VpBVIepkSyUnxJmsz+FZWdW+Qqpfgm4KHQRLRCDDBVh10N5KU7P0e19scuqggrlXS37VezisKHgZJRCPQNy3CIEkpzh4P6lq6AAC5KfpYLh9vi0JyjHdbpks8txSjt+CbgodBEtEISDlJDfZuVkNXSF1LJ9weETazEePjQl/UNVw4lau8Gp0tCKDgYZBENAIJNjOSpB5/My9mSug/JSIIgsKtCZ7cVE7lKsnl9qD2ZicAIJcFPekrGCQRjVAOV7gpSq/JtXLyNle4KaLuZid6PCKio/Q1QknBwSCJaIRyuD2JovQ6JcLzSln985H0NEJJwcEgiWiEeDFTljSCp5fl/5L+023+Cu5SaLHSNg2FQRLRCOUywVZRfT1+feWNZCXZIAhAm6MHN9qdSjcn4uitijsFF4MkohGSywA0tbPHH2btjh5cb3MA0F9OkjXKiDsSogFwlFIJrJFEQ2GQRDRC2ckxEATA3t2Dmx3s8YeTVEMoZZwZ8dFRCrcm+KRVVdyeJPzkXDeubCM/GCQRjZA1yoj0eG+Pn1Nu4XWxSZ8r2yTcG1AZHY4eNNi7AfRVPyfqj0ESUQBY00YZeq+IzIKSypDOq+QYM+Jt+huhpLFjkEQUAF7MlKHXpG0JV04qQ+/BN40dgySiAPRdzJg7Ek56v5hJ3+tycwfcHi4KCBe9n1c0dgySiALAHn/4iaIoT2/m6rSWTXpCNMwmA1xuEVd7N/Gl0GONJBoOgySiAOT1roC51NzJHn+Y3Gh3os3RA0EAJiTblG5OSBgNgpw4zO1JwqevRpI+p3Fp7BgkEQUgPSEaZqMBzh4Prt1ijz8cqntXtmUkRsNiMircmtDh3oDhJYqifG7pdYSSxo5BElEAjAZBHs1g8nZ46D1pWyIXK+V5FRbNHU60dXtHKLOS9DlCSWPHIIkoQHJeUhOnRcJBrxvbfhXz3cJL+jnfkRANa5R+RyhpbBgkEQWIPf7wqo6QFUi5DJLCituR0EgwSCIKEDe6Da9IWaYtfb+rt7rQ7XIr3Br948a2NBIMkogC1LfPFoOkUHN7RFxu1vfyf0lSTN++dJeaeW6FmlTrjHu20VAYJBEFiD3+8LnS0gmXW4TZZJD3zdMrQRC4wi2MqjndRiPAIIkoQMkxZsRaTRBF4HJzp9LN0TU5Hyk5BgaDoHBrQo95SeHhHaH0/u0ySKKhMEgiCpAgCP0uZlzhFkqRllzLkaTwuHarC063xztCmaDvEUoaGwZJRKPAjW7DI9K2jehbOcngO5Skv9vsZBuMETBCSaPHIIloFOTkbfb4QypSaiRJpO0xON0WWlKNM25HQsNhkEQ0Ciz8Fx5ykBQhI0nZKd7Kzy2dLrR0OBVujX5F2ggljR6DJKJR4HRb6HU53bjauz+e3rckkdjMJtwebwXAcyuUIqVAKY0dgySiUZD+cb3Z4cStTvb4Q0GqFRQfHYVEW5TCrQkfjlKGnpQYHynTuDR6DJKIRiHGYsL4OAsAXsxCpX+lbUGInOTaHK6cDKlulxvXWqURSgZJNDQGSUSjxB5/aEVa0raE51VoXW7uhCgCcVYTkmLMSjeHVI5BEtEocXuS0JKnRCIsuTav97xiraTQ6L8dSSSNUNLoMEgiGqVcFv4Lqerei1mkJG1LpJGkS80d8HhEhVujPxeZj0QBYJBENEpc4RZaNRG6AikjMRomg4Bulwf19m6lm6M7kXpe0egwSCIaJbnHf4M9/mBr6XDiVqcLQF/toEhhMhqQlez9zixWGnyskUSBYJBENEqZSd4tDbpcbjS2sccfTNLo3O3xVtjMJoVbE37cGzB0OJJEgWCQRDRKUUYDspLY4w+FSL+QcSo3NG51OnGzt5J5pJ5bFJiwBEkOhwMzZsyAIAioqqryee6TTz7BnDlzYLVakZmZic2bNw94/aFDh5Cfnw+r1YqpU6fiyJEjw37me++9h3vvvRcWiwV33nkn9u7d6/P8hg0bIAiCzy0/P38sX5MiUC4vZiHRtwIpMi9kXDkZGjURPkJJgQtLkLR27Vqkp6cPeNxut2P+/PmYMGECzp49iy1btmDDhg3YvXu3fMzp06exaNEiLF26FOfOnUNJSQlKSkpw/vz5QT+vpqYGCxYswDe/+U1UVVVh9erVePzxx/GHP/zB57jJkyejvr5evn3wwQfB+9IUEXK4wi0kpJ9npK1sk/C8Co2+8yoyg28KXMhD6aNHj+L48eN49913cfToUZ/n3n77bTidTuzZswdmsxmTJ09GVVUVtm3bhieeeAIAsH37djz44IN46qmnAAAvvPACKioqsHPnTuzatcvvZ+7atQs5OTnYunUrAODuu+/GBx98gFdeeQXFxcXycSaTCWlpaaH42hQhpORP5o4EV6QWkpRI3/tKSyccPW5YTEaFW6QPkT6NS4EL6UhSY2Mjli1bhrfeegs228AVKpWVlZg7dy7M5r6qp8XFxfj888/R0tIiHzNv3jyf1xUXF6OysnLQzx3pa7744gukp6cjNzcXS5YsQW1t7ZDfx+FwwG63+9wosrE6cvB5PGLEX8xSYy2IMRvhEYG6m51KN0c3Iv28osCFLEgSRRGlpaVYvnw5CgoK/B7T0NCA8ePH+zwm3W9oaBjyGOn5QN7Xbrejq8u7Z09hYSH27t2LY8eO4Y033kBNTQ3mzJmDtra2Qd9306ZNiI+Pl2+ZmZmDHkuRIbd3OqiupQvOHo/CrdGHens3HD0emAwCMhKjlW6OIgRBkEcpOeUWPFLuYKTmulHgAg6S1q1bNyDh+au3Cxcu4LXXXkNbWxvKy8tD0e4x+9a3voWFCxdi2rRpKC4uxpEjR3Dr1i288847g76mvLwcra2t8q2uri6MLSY1Gh9ngc1shNsjoq6FPf5gkFYKZiXbYDJG7gJcKR+Lo5TB4fGIuHQjsnPdKHAB5yStWbMGpaWlQx6Tm5uLkydPorKyEhaLxee5goICLFmyBPv27UNaWhoaGxt9npfuS7lCgx0zVC7RYK+Ji4tDdLT/nmlCQgImTpyIL7/8ctD3tVgsA74PRTZBEJCTEoNPr9lR09Qh77tFoyevbIvwC1kup3KDqrGtG10uN0wGAZkROkJJgQs4SEpNTUVqauqwx+3YsQMvvviifP/atWsoLi7GwYMHUVhYCAAoKirCs88+C5fLhaioKABARUUFJk2ahMTERPmYEydOYPXq1fJ7VVRUoKioaNDPLioqGlAmYLjXtLe34+LFi/jud7877Hcj6k8Kkrx7jY0f9nga2sUI3dj2q3I53RZU1RyhpFEI2ZmSlZWFKVOmyLeJEycCAPLy8pCRkQEAWLx4McxmM5YuXYpPP/0UBw8exPbt21FWVia/z6pVq3Ds2DFs3boVFy5cwIYNG3DmzBmsXLlSPqa8vByPPvqofH/58uWorq7G2rVrceHCBfziF7/AO++8gx//+MfyMT/5yU/wxz/+EZcuXcLp06fx7W9/G0ajEYsWLQrVj4R0ij3+4GJyrRcLSgZXdYSvmKTRUTScjo+Px/Hjx1FTU4NZs2ZhzZo1WL9+vbz8HwBmz56N/fv3Y/fu3Zg+fTp+85vf4PDhw5gyZYp8TH19vc/KtJycHPznf/4nKioqMH36dGzduhX/9m//5rP8/8qVK1i0aBEmTZqEhx9+GMnJyfjwww9HNEpG1B8TbIOLQZJXdu/3v9HugL3bpXBrtK+GNZJoFMJWcjQ7OxuiOHAT0GnTpuHUqVNDvnbhwoVYuHDhoM9/tZo2AHzjG9/AuXPnBn3NgQMHhvxMopFigm3wOHrcuNKbAB/pPf44axRSxllwo92BSzc6MC0jQekmaZqU68akbQoEJ2aJxkjqmV5vc6Dd0aNwa7St7mYnPCIQYzYiNZaLJDiVGzwcoaTRYJBENEbx0VFIGectiHqJF7MxqZaTtsdBEASFW6M8Jm8Hh7PHg7oWb428vAhfEECBYZBEFARS7/RiE7cnGYtq9vZ9MHk7OGpvdsLtETlCSQFjkEQUBNyeJDiYXOur77xi8D0W8lRbagxHKCkgDJKIgoDJ28FRw20jfEg/h5qmDr8LX2hkmLRNo8UgiSgIOJIUHJxu85WZZINBADqcbjS1OZRujmYxaZtGi0ESURDkscc/ZvZuF260ewMBXsy8LCYjMpNsAJiXNBZS4juTtilQDJKIgiAr2QZBANocPbjR7lS6OZokrQxMjbUg1hqlcGvUg6OUY8eRJBotBklEQWAxGZHRu2lmNVe4jUo1k7b9kle48bwalbZuF673TlVm89yiADFIIgoSJm+PDffW8o8FJcfm0g1vBfeUcRbEcYSSAsQgiShIeDEbG06J+CcF38xJGp3q3pVtDL5pNBgkEQUJC/+NTd8ybV7M+pM2UK5t7kSP26Nwa7SHwTeNBYMkoiCRa9owSAqYKIpyIcncVNay6e/2OCusUQb0eERc6d1ag0aOtbdoLBgkEQWJ1FO93NzBHn+Arrc50OF0wyAAWb1L3snLYBCQnSyNUjJ5O1BcEEBjwSCJKEjS46NhNhngcou4eos9/kBIF7LMJBvMJv6z9FXc6HZ0RFHkSBKNCf81IgoSg0FATjLzkkaDeSNDY62k0Wlqd6Dd0QODALkoJ1EgGCQRBZF8MWOPPyBM2h4ay0uMjvR3mJFog8VkVLg1pEUMkoiCKIfJ26NSwxpJQ+JI0uhwhJLGikESURCxVtLoyIUkubLNL2nPsfrWbnQ6exRujXYwH4nGikESURD1JdhyFdJIudwe1DZ7qyKzx+9fgs2MRJu3WjQD8JG72MQRShobBklEQSTljlxr7UaX061wa7ThSksXejwirFEGpMVZlW6OanHKLXB9uW4coaTRYZBEFESJtijER3t7/JeaeTEbCelClp0cA4NBULg16iUnb3NRwIj0uD2ovdk7QsnpNholBklEQSQIAnv8AapuYt7ISLCie2Cu3uqCyy3CYjLgdo5Q0igxSCIKMiZvB4YrkEaGewMGprrfecURShotBklEQcbqyIHpW/7PvJGh9F8UIIqiwq1RvxqOUFIQMEgiCjIpd4T7bI2MvLcWL2ZDkvZvs3f34GaHU+HWqF81C5RSEDBIIgoy5iSNXIejBw32bgBcpj0ca5QRdyREA+C5NRJ907gcoaTRY5BEFGTZKd49om51utDCHv+QpBWAibYoJNjMCrdG/ZiXNHLSdBtHkmgsGCQRBZnNbMLt8d7VNLyYDY1J24HhKOXIdDnduNbKEUoaOwZJRCHAi9nI9PX2OSUyEtxAeWSkEcoEWxQSYzhCSaPHIIkoBPpq2jB5eyjcWyswrJU0MtwwmYKFQRJRCMgr3NjjH9JFXswCIpVJqGnugNvDMgCDkfZO5AgljRWDJKIQYEHJ4YmiiBrpYsaRpBG5IzEaUUYBzh4Prt3qUro5qlXNEUoKEgZJRCHQPyfJwx6/Xzc7nLB39wDoqwFEQzMaBExIZgA+HC4IoGBhkEQUAhm9PX5Hjwf1vXWAyJd0IbsjIRrWKKPCrdEOLgoYHoMkChYGSUQhYDIakJXkrZfElUj+cUpkdJi8PbSWDidudboAMEiisWOQRBQiUtIoV7j5x97+6OSyoOSQqjlCSUHEIIkoRKQe/0WOJPnVtwKJQVIg+lZOMvj2h+cVBRODJKIQYe7I0DiSNDrSz+vqrS50u9wKt0Z9eF5RMIU8SHI4HJgxYwYEQUBVVZXPc5988gnmzJkDq9WKzMxMbN68ecDrDx06hPz8fFitVkydOhVHjhwZ8vPq6+uxePFiTJw4EQaDAatXr/Z7XKDvSxQoBkmDc3tEXGruBNBX+4dGJmWcGbEWE0QRqL3ZqXRzVIdBEgVTyIOktWvXIj09fcDjdrsd8+fPx4QJE3D27Fls2bIFGzZswO7du+VjTp8+jUWLFmHp0qU4d+4cSkpKUFJSgvPnzw/6eQ6HA6mpqXjuuecwffp0v8eM5n2JAiXljlxp6YSjhz3+/q7d6oKzxwOz0YA7EqOVbo6mCIIg15VisdKB5CCJCwIoCEIaJB09ehTHjx/Hyy+/POC5t99+G06nE3v27MHkyZPxne98B08++SS2bdsmH7N9+3Y8+OCDeOqpp3D33XfjhRdewL333oudO3cO+pnZ2dnYvn07Hn30UcTHx/s9ZjTvSxSo1FgLxllM8IhAHXv8PqQL2YRkG4wGQeHWaA+Llfrn8YjyzySPI5QUBCELkhobG7Fs2TK89dZbsNlsA56vrKzE3LlzYTb3bT5YXFyMzz//HC0tLfIx8+bN83ldcXExKisrx9S20b6vw+GA3W73uRENRhAEecifPX5fnBIZG66c9K/e3g1HjwdRRoEjlBQUIQmSRFFEaWkpli9fjoKCAr/HNDQ0YPz48T6PSfcbGhqGPEZ6frRG+76bNm1CfHy8fMvMzBxTO0j/crhc269qbkcyJpxu8086ryYkx3CEkoIioCBp3bp1EARhyNuFCxfw2muvoa2tDeXl5aFqtyLKy8vR2toq3+rq6pRuEqmcnLzNi5mPam5sOyacbvOPI5QUbKZADl6zZg1KS0uHPCY3NxcnT55EZWUlLBaLz3MFBQVYsmQJ9u3bh7S0NDQ2Nvo8L91PS0uT/+vvGOn50Rrt+1oslgHfiWgorI7sX9/FjHkjo5HdGwQ0dzjR2ulCvC1K4RapgzSyxuCbgiWgICk1NRWpqanDHrdjxw68+OKL8v1r166huLgYBw8eRGFhIQCgqKgIzz77LFwuF6KivH/gFRUVmDRpEhITE+VjTpw44bOMv6KiAkVFRYE0e4BQvS/RV3G6baBulxtXe3ewZ49/dMZZTLgt1oLrbQ7UNHdghi1B6SapAkeSKNgCCpJGKisry+f+uHHe3mJeXh4yMjIAAIsXL8bGjRuxdOlSPP300zh//jy2b9+OV155RX7dqlWrcP/992Pr1q1YsGABDhw4gDNnzviUCSgvL8fVq1fx5ptvyo9J9Zja29vR1NSEqqoqmM1m3HPPPSN+X6JgkP6xvtHugL3bhTgre/y1NzshikCs1YSUcebhX0B+5abGeIOkG+2YkZmgdHNUoUbeD5AjlBQcilXcjo+Px/Hjx1FTU4NZs2ZhzZo1WL9+PZ544gn5mNmzZ2P//v3YvXs3pk+fjt/85jc4fPgwpkyZIh9TX1+P2tpan/eeOXMmZs6cibNnz2L//v2YOXMm/v7v/z6g9yUKhlhrFFJjvVO0zEvykpJrc1NiIAhMrh2tvu1JeF4BgKPHjSst3lIbHEmiYAnJSNJXZWdnQxTFAY9PmzYNp06dGvK1CxcuxMKFCwd9fu/evQMe8/dZgb4vUbDkpMSgqc2BmhsdmM4evzz1yAvZ2HCjW1+1zZ3wiECshSOUFDzcu40oxHgx8yWNqDFpe2y4ctJXdb9K2xyhpGBhkEQUYtzDzRe3jQiOnH4rJ0cyeq53TNqmUGCQRBRifUESqyMD/ZJreTEbk8xE75YuXS43Gu0OpZujuL4RSp5XFDwMkohCTFppU9PEHn9rpwvNHU4AvJiNldlkQFaSd8unagbgXNlGIcEgiSjEspJsMAhAh9ON622R3eOXLubj4yyIsYRl3YiucW/APtK5xRFKCiYGSUQhZjYZkCn1+CP8Ysa8keBivptXa5cLN9q9I5TZPLcoiBgkEYUBL2Ze3I4kuHheeV3q/f63xVowjiOUFEQMkojCgMnbXtzYNri40a0XRygpVBgkEYUBL2ZeXIEUXFIZgNqbnXC5PQq3Rjly8M2yEhRkDJKIwkBacRPJBSVFUey3AokXs2BIi7MiOsoIt0dE3c1OpZujmL6yEpzGpeBikEQUBtLISW1z5Pb4G+zd6HK5YTQIciI7jY0gCFzhhr79ADlCScHGIIkoDNLirLBGGdDjEXGlpUvp5ihCmmrLSrIhysh/eoKlf+XtSNR/hJJV3CnY+C8VURgYDAKykyM7eZsb24ZGpO8NeL3NgU5n7whlIkcoKbgYJBGFiZSHE6nTIlyBFBqRvnJS+nvKTIyG2cRLGgUXzyiiMJGSSiN1WoRJ26Ehb3sT8ecVk7Yp+BgkEYVJpBf+40hSaOT0TuM22h3ocPQo3Jrwk0bQeF5RKDBIIgqTnAiebnP2eFDbu0Sdy7SDK94WheQYM4DIDMCrWXuLQohBElGYSAm2DfbuiOvx17V0wu0RER1lxPg4i9LN0Z2cCE7ermEVdwohBklEYZJgMyPRFgUAuNQcWRez/pW2BUFQuDX6I0/lRtgopcvdN0LJ5f8UCgySiMIoUvOSWMcmtPpqJUXWCrcrLV3okUYoY61KN4d0iEESURjJK5EirMcvTQPlcUokJCJ15WT/pG2DgSOUFHwMkojCKHJHknovZhxJCgm5BteNDoiiqHBrwkdO2uZ5RSHCIIkojKTk0osRFiT1rUDiyrZQyEqyQRCAtu4e3Gh3Kt2csKlm0jaFGIMkojCSc0ea2iOmx9/u6MH1NgeAvpo+FFzWKCPuSIgGEFmjlDVc/k8hxiCJKIyk/dvs3T242REZPf5LvRft5Bgz4ntX91HwReL2JCxQSqHGIIkojCKxx8+NbcMj0ja67XD0oMHeDYAFSil0GCQRhVn/JNtIIE2JcM+20Iq0lZNSrTGOUFIoMUgiCjO5OnKEXMyq5WXa7O2HUqRV3eZ2JBQODJKIwizSckeYNxIe0s/3cnMH3B79LwrgeUXhwCCJKMwiqVaSKIqcbguT9IRomE0GuNwirrZ0Kd2ckGMVdwoHBklEYSYlmV5q7tR9j/9GuxNtjh4IgreWD4WO0SAgO9n7M66OgFFK1kiicGCQRBRmdyRGw2w0wNnjwbVb+u7xS739OxKiYY0yKtwa/YuUUUrvCCVz3Sj0GCQRhZnRIGBCb49f7xczKe9KWnlFoSWvcNP5eXWzwwl7t3eEUvpbIgoFBklECuhb4abvaRFpBRKnRMIjUlZOVnOEksKEQRKRAuTtSXTe42chyfDKjZDpNm5HQuHCIIlIAZFSHZnLtMNL+jlfvdWFbpdb4daEDpO2KVwYJBEpQEo21XOP3+0RcbmZQVI4JcWYEWc1AeirSK1HNXKBUp5XFFoMkogUEAk9/qstXXC5RZhNBqT37ldHoSUIAnIiYHuSvhpJXBBAocUgiUgBKePMiLWaIIpA7c1OpZsTEvJ2JMkxMBoEhVsTOfJ0PpXr9oi41Oz9m+F0G4VayIMkh8OBGTNmQBAEVFVV+Tz3ySefYM6cObBarcjMzMTmzZsHvP7QoUPIz8+H1WrF1KlTceTIkSE/r76+HosXL8bEiRNhMBiwevXqAcfs3bsXgiD43KxW61i+JlFABEHoy0vS6Qo37q2lDL2vcLt2qwvOHg9HKCksQh4krV27Funp6QMet9vtmD9/PiZMmICzZ89iy5Yt2LBhA3bv3i0fc/r0aSxatAhLly7FuXPnUFJSgpKSEpw/f37Qz3M4HEhNTcVzzz2H6dOnD3pcXFwc6uvr5dvly5fH9kWJAqT3DUm5bYQy+lZO6jT47j2vspNtHKGkkDOF8s2PHj2K48eP491338XRo0d9nnv77bfhdDqxZ88emM1mTJ48GVVVVdi2bRueeOIJAMD27dvx4IMP4qmnngIAvPDCC6ioqMDOnTuxa9cuv5+ZnZ2N7du3AwD27NkzaNsEQUBaWlowvibRqMjJ2zrt8XNlmzL0XnW7r9I2zysKvZCNJDU2NmLZsmV46623YLMNrIhaWVmJuXPnwmw2y48VFxfj888/R0tLi3zMvHnzfF5XXFyMysrKMbevvb0dEyZMQGZmJv7xH/8Rn3766ZjfkygQeq+VVMNl2orITvb+vFs6XWjpcCrcmuDrC76ZtE2hF5IgSRRFlJaWYvny5SgoKPB7TENDA8aPH+/zmHS/oaFhyGOk50dr0qRJ2LNnD373u9/hV7/6FTweD2bPno0rV64M+TqHwwG73e5zIxotPRf+63a5cbV3Xzr2+MMrxmJCWpw3x7JGh2UAWCOJwimgIGndunUDEp6/ertw4QJee+01tLW1oby8PFTtHpOioiI8+uijmDFjBu6//3789re/RWpqKv71X/91yNdt2rQJ8fHx8i0zMzNMLSY9koKH5g4nWjtdCrcmuKQaPfHRUUiKMQ9zNAVbrjRKqcOpXHmEkrluFAYB5SStWbMGpaWlQx6Tm5uLkydPorKyEhaLxee5goICLFmyBPv27UNaWhoaGxt9npfuS7lCgx0T7FyiqKgozJw5E19++eWQx5WXl6OsrEy+b7fbGSjRqMVYTBgfZ0Gj3YHqG+2YmZWodJOCpv/KNkFgcm245aTE4PTFZrkMg15whJLCLaAgKTU1FampqcMet2PHDrz44ovy/WvXrqG4uBgHDx5EYWEhAO9ozrPPPguXy4WoqCgAQEVFBSZNmoTExET5mBMnTvgs46+oqEBRUVEgzR6W2+3GX/7yF/z93//9kMdZLJYBgR/RWOSkxKDR7kDNjQ5dBUnMR1KWXpO3Lzd3QhSBOKuJI5QUFiFZ3ZaVleVzf9w4b4JdXl4eMjIyAACLFy/Gxo0bsXTpUjz99NM4f/48tm/fjldeeUV+3apVq3D//fdj69atWLBgAQ4cOIAzZ874lAkoLy/H1atX8eabb8qPSfWY2tvb0dTUhKqqKpjNZtxzzz0AgP/xP/4H/uZv/gZ33nknbt26hS1btuDy5ct4/PHHQ/HjIBpUTso4fFh9U3cXM9ZIUpY0FaW3WknydiSp4zhCSWER0hIAQ4mPj8fx48exYsUKzJo1CykpKVi/fr28/B8AZs+ejf379+O5557DM888g7vuuguHDx/GlClT5GPq6+tRW1vr894zZ86U///s2bPYv38/JkyYgEuXLgEAWlpasGzZMjQ0NCAxMRGzZs3C6dOn5SCKKFz0utFt38WMQZISpJVfl5o74PGIMOiknhCTtincwhIkZWdnQxTFAY9PmzYNp06dGvK1CxcuxMKFCwd9fu/evQMe8/dZ/b3yyis+I1ZEStFrgm3fdBuXaSshMzEaJoOAbpcHDfZu3VSmlv5OGCRRuHDvNiIF9c8dGS6414qWDidaelfrZacMrJFGoWcyGpCV7P3Z62kql1XcKdwYJBEpKDPJu7VCl8uNBnu30s0JCmlK5PZ4K2xmxWb0I54e9wasZhV3CjMGSUQKijIakJXU2+PXyZQbtyNRB73tDXir04mbvRXEpariRKHGIIlIYXq7mMlJ2wySFCXvDaib88r7PdLirIixcISSwoNBEpHC9FbThiNJ6sDzimjsGCQRKSxXZxvdSrV58lK5sk1Jeb3nVd3NTjh7PAq3Zuy4HQkpgUESkcL01OP3eER53zb2+JWVGmtBjNkIjwjU3uxUujljxqRtUgKDJCKFSbWEanXQ46+3d6Pb5YHJICAjUR+1ebRKEAR5qbweVrhJI5QcSaJwYpBEpLDxcRZERxnh9oioa9F2j19aoZeVbIPJyH9elKaX5G2PR8QleSSJ07gUPvxXjEhhgiD0TblpvAyAtLKNFZHVQS9TuY1t3ehyuTlCSWHHIIlIBXJ0krzNvBF10cvegPIIZZINURyhpDDi2UakAnl6uZjJK5A4JaIGelk5Wc2VbaQQBklEKqCXBFspuZYjSeqQ3ft7aGpzoK3bpXBrRo/nFSmFQRKRCughwdbR48aV3sRz5iSpQ5w1CinjLAC0fW71VXHnCCWFF4MkIhXI6d2L6nqbA+2OHoVbMzp1NzvhEYEYsxGpsRalm0O9cnWQvM1q26QUBklEKhBvi0JyjBkA5KXOWiNPiaTGQBAEhVtDEnlvQI2unHT2eFDX0gWAOUkUfgySiFRC6xvd1rCOjSppfeVkXUsn3B4RMWYjbuMIJYUZgyQilZBXImm0xy+vbOOUiKpofbqthiOUpCAGSUQqIY3AVN/Q5go3bhuhTrn9Vk6KoqhwawJXzaRtUhCDJCKV0Hp1ZBaSVKfMJBsMAtDhdKOpzaF0cwLGpG1SEoMkIpXoP92mtR6/vduFG+3eC3A2L2aqYjEZkZFoA6DNfDd5hJLnFSmAQRKRSmQl2SAIQJujBzfanUo3JyDSiryUcRbEWaMUbg19lZZHKTmSREpikESkEtYoI+5I8G7eqbWLGZO21U2rQVK7owfXe6cIc5jrRgpgkESkItKeZzUaS95m0ra65aVqs1YSRyhJaQySiFQkV6OF/5i0rW5aXTl5sXcvQ45QklIYJBGpiFYLSvbtrcWLmRpJU1W1zZ3ocXsUbs3IMR+JlMYgiUhFtJg7IoqiXPCP023qdHucFRaTAT0eEVd6t/jQAjlI4nlFCmGQRKQiUpB0ubkDbo82ygA0tTnQ4XTDIHhr8pD6GAyCJgNwjiSR0hgkEanIHQnRMJsMcLlFXNVIj1+aGsxMssFiMircGhqMXHlbI0FS/xHKPI4kkUIYJBGpiMEgICdZuphpI8mWvX1t6BtJ0sZ5daPdiTZHD0coSVEMkohUJkdjK9yqm5i0rQXyCjeNnVcZiRyhJOUwSCJSGSlJVSu5IywkqQ1ay0niCCWpAYMkIpXR2sWsr0YSd2lXMymIrW/tRqezR+HWDI9BEqkBgyQilcnVUJDU4/agtrkTAJdpq11ijBkJNm/V6ks3OhVuzfCk4JtlJUhJDJKIVEbamuTqrS50u9wKt2ZoV1q60OMRYY0y4PY4q9LNoWFoKQDvm8blCCUph0ESkcok2qIQH93b429W98VMupBlJ8fAYBAUbg0NR5oSVfsKN7dHxOVmFpIk5TFIIlIZQRA0s8JN3luLFzJNyNXIRrdXWjrhcouwmDhCScpikESkQlqZFmFyrbZoZW/A/hsmc4SSlMQgiUiFtDKSVMOVbZrSd161QxTVu+2NVGmbwTcpjUESkQr11UpSd+4IR5K0Jbu3mru9uwctnS6FWzM4nlekFmEJkhwOB2bMmAFBEFBVVeXz3CeffII5c+bAarUiMzMTmzdvHvD6Q4cOIT8/H1arFVOnTsWRI0eG/Lzf/va3eOCBB5Camoq4uDgUFRXhD3/4w4DjXn/9dWRnZ8NqtaKwsBAfffTRmL4nUbDkygm26h1J6nT2oL61GwD31tKKaLMRdyREA1B3AC6vbEvlCCUpKyxB0tq1a5Genj7gcbvdjvnz52PChAk4e/YstmzZgg0bNmD37t3yMadPn8aiRYuwdOlSnDt3DiUlJSgpKcH58+cH/bz3338fDzzwAI4cOYKzZ8/im9/8Jh566CGcO3dOPubgwYMoKyvD888/j48//hjTp09HcXExrl+/HtwvTzQK2SnevapaOl1o6XAq3Br/pAtZoi0KCTazwq2hkZJGZy6qeCqXW92QWoQ8SDp69CiOHz+Ol19+ecBzb7/9NpxOJ/bs2YPJkyfjO9/5Dp588kls27ZNPmb79u148MEH8dRTT+Huu+/GCy+8gHvvvRc7d+4c9DNfffVVrF27Fl/72tdw11134ec//znuuusu/Md//Id8zLZt27Bs2TI89thjuOeee7Br1y7YbDbs2bMnuD8AolGwmU24Pd67qketSbacEtEmtVd073K6ca13hJJb3ZDSQhokNTY2YtmyZXjrrbdgsw3cxbmyshJz586F2dzXCy0uLsbnn3+OlpYW+Zh58+b5vK64uBiVlZUjbofH40FbWxuSkpIAAE6nE2fPnvV5X4PBgHnz5g35vg6HA3a73edGFCpqv5j1JddySkRL5PNKpSNJUm2wBFsUEmM4QknKClmQJIoiSktLsXz5chQUFPg9pqGhAePHj/d5TLrf0NAw5DHS8yPx8ssvo729HQ8//DAA4MaNG3C73QG/76ZNmxAfHy/fMjMzR9wGokD1BUnqzB2p4bYRmqT2DZQ5QklqEnCQtG7dOgiCMOTtwoULeO2119DW1oby8vJQtHvE9u/fj40bN+Kdd97BbbfdNqb3Ki8vR2trq3yrq6sLUiuJBlL7SFI1L2aaJNfgau6Ax6O+MgAMkkhNTIG+YM2aNSgtLR3ymNzcXJw8eRKVlZWwWCw+zxUUFGDJkiXYt28f0tLS0NjY6PO8dD8tLU3+r79jpOeHcuDAATz++OM4dOiQz9RaSkoKjEZjwO9rsVgGfB+iUMnrXdmjxlpJoijKybUcSdKWjEQboowCnD0eXGvtQkbiwFQIJUnnex5XtpEKBBwkpaamIjU1ddjjduzYgRdffFG+f+3aNRQXF+PgwYMoLCwEABQVFeHZZ5+Fy+VCVJR3r6qKigpMmjQJiYmJ8jEnTpzA6tWr5feqqKhAUVHRkJ//61//Gt///vdx4MABLFiwwOc5s9mMWbNm4cSJEygpKQHgzVs6ceIEVq5cOex3IwoHqSd9qbfHr6bKwzc7nLB39wDoq71D2mA0CJiQHIMvr7ejuqlDfUHSDa5sI/UIWU5SVlYWpkyZIt8mTpwIAMjLy0NGRgYAYPHixTCbzVi6dCk+/fRTHDx4ENu3b0dZWZn8PqtWrcKxY8ewdetWXLhwARs2bMCZM2d8gpny8nI8+uij8v39+/fj0UcfxdatW1FYWIiGhgY0NDSgtbVVPqasrAy//OUvsW/fPnz22Wf44Q9/iI6ODjz22GOh+pEQBSQjMRomg4Bulwf19m6lm+NDmhK5IyEa1iijwq2hQKl5KpfTbaQmilbcjo+Px/Hjx1FTU4NZs2ZhzZo1WL9+PZ544gn5mNmzZ2P//v3YvXs3pk+fjt/85jc4fPgwpkyZIh9TX1+P2tpa+f7u3bvR09ODFStW4Pbbb5dvq1atko955JFH8PLLL2P9+vWYMWMGqqqqcOzYsQHJ3ERKMRkNyEr29vLVthKJ+Ujapta9AVs6nLjVWwmcI5SkBgFPt41Wdna2372Cpk2bhlOnTg352oULF2LhwoWDPr93716f+++9996I2rRy5UpOr5Gq5abEoLqpAzU32vG3d6Uo3RwZe/vaptaNbqX2pMdbEW3mCCUpj3u3EamYWi9m3IBU29RaXkIOvrkYgFSCQRKRikl7V6ltWoQ1krRNOq+utHTB0eNWuDV9pKAtlwVKSSUYJBGpmDySpKKcJLdHRE1vVWRezLQpZZwZsRYTRBG43NypdHNk1RyhJJVhkESkYlKC7ZWWTtX0+K/d6oKzx4Moo4A7EqOVbg6NgiAI8pSWmgJwTreR2jBIIlKx1FgLYsxGeESg7qY6evzShWxCcgyMKqrdRIFRWxkAj0fsm8blSBKpBIMkIhVTY4+fK9v0QW3J2/X2bjikEcoEjlCSOjBIIlI5Ke9HLT1+Jm3rg9oWBUgrJickx8Bk5KWJ1IFnIpHKqW1apJpTIrqgtoKSNdyOhFSIQRKRyuWqbLpN2tg2hyvbNC27Nxi50e5Ea5dL4dYAF5sYfJP6MEgiUjk1FZTsdrlx9VYXAPb4tW6cxYTbYi0A1DGaxFw3UiMGSUQq19fjd8DerWyPv/ZmJ0QRiLWYkDLOrGhbaOzUlLzNIInUiEESkcrFWaOQMs7b47+kcI9fLvaXGgNB4PJ/rZOmcpXeQNnR48aVFm+JC9ZIIjVhkESkAfLFTOEgiXVs9EVaOan0VG7dzU54ekcoU3s7BERqwCCJSANyVbI9Sd8KJCZt64FaVk5yhJLUikESkQaoJXm7/8WMtC+n3wilKIqKtaOa+UikUgySiDRALQm2nG7Tl8xEG4wGAZ1ONxrtDsXaUcONbUmlGCQRaUD/BFulevytnS40dzgB9K24I20zmwzI7N2kuFrBAJwr20itGCQRaUBmkg0GAehwutHUpkyPv6bZeyG7LdaCcRaTIm2g4FNDXlJfFXfmupG6MEgi0gCLyYjMJBsA5fKSpKk+7tmmL/IebgotCrB3u3Cj3Rv4M9eN1IZBEpFGKN3j78sbYW9fT5Q+r6TaXxyhJDVikESkEfIKtyZlckcuMmlbl3IVXjlZzaRtUjEGSUQaofSu7VyBpE/SFFftzU643J6wf76cj8SpNlIhBklEGpGjYHVkURT7ViDxYqYr42OtiI4ywu0RUXezM+yfz5VtpGYMkog0Qu7xN3eiJ8w9/ka7A10uN4wGAZmJtrB+NoWWwSDIJR2UGKVkFXdSMwZJRBpxe5wV1igDejwirrR0hfWzpRo6WUk2mE38Z0NvlNobUBRFeRqX022kRvzXjkgjDAYB2clSkm14k7eZXKtvUr7bxTCXAbje5kCHkyOUpF4Mkog0ROpth3ujW+aN6JtS295I53FmYjRHKEmVeFYSaYhSNW0YJOkbzysi/xgkEWmIlNyq1MWMNZL0SQpSGu0OdDh6wva5TNomtWOQRKQhSvT4XW4PanuXhnP5vz4l2MxIijEDCO+5xbISpHYMkog0JK/3YlLf2o1OZ3h6/HU3O+H2iIiOMiItzhqWz6TwU6JYqVTzK48jlKRSDJKINCTBZkaiLQpA+C5m/Ve2CYIQls+k8Ovb9iY855XL7UFtM0coSd0YJBFpTLin3DglEhlyUsO7wu1KSxd6ekcox8dyhJLUiUESkcbIydth6vFXM2k7IoR7uk0KxrJTYmAwcISS1IlBEpHGhLs6ct8KJAZJetZ/b0BRFEP+edK0HoNvUjMGSUQaI11UwrXRrbz8P5XLtPVsQrINggC0dfegucMZ8s/rO68YJJF6MUgi0pgcuep2e8h7/B2OHjTaHd7PTebFTM+sUUbckRANIDyjlCwkSVrAIIlIY6T92+zdPbgZ4h6/dCFLjjEjvndVHelX3wq30Cdvcz9A0gIGSUQaE84efzV7+xElXFO5HY4eNNi7AfDcInULS5DkcDgwY8YMCIKAqqoqn+c++eQTzJkzB1arFZmZmdi8efOA1x86dAj5+fmwWq2YOnUqjhw5MuTn/fa3v8UDDzyA1NRUxMXFoaioCH/4wx98jtmwYQMEQfC55efnj/m7EoVDTpguZjXs7UcUubxEiFdOXmr2vn9SjBkJNnNIP4toLMISJK1duxbp6ekDHrfb7Zg/fz4mTJiAs2fPYsuWLdiwYQN2794tH3P69GksWrQIS5cuxblz51BSUoKSkhKcP39+0M97//338cADD+DIkSM4e/YsvvnNb+Khhx7CuXPnfI6bPHky6uvr5dsHH3wQvC9NFELhqpUkr2xjcm1EyEkNz96AzEcirTCF+gOOHj2K48eP491338XRo0d9nnv77bfhdDqxZ88emM1mTJ48GVVVVdi2bRueeOIJAMD27dvx4IMP4qmnngIAvPDCC6ioqMDOnTuxa9cuv5/56quv+tz/+c9/jt/97nf4j//4D8ycOVN+3GQyIS0tLYjflig85DIAIe7x921sy5VtkUCabrvc7N2Kxhii+kU1XP5PGhHSkaTGxkYsW7YMb731Fmw224DnKysrMXfuXJjNfcOtxcXF+Pzzz9HS0iIfM2/ePJ/XFRcXo7KycsTt8Hg8aGtrQ1JSks/jX3zxBdLT05Gbm4slS5agtrZ2yPdxOByw2+0+NyIlhGMkSRTFvkKSHEmKCOkJ0TCbDHC6Pbh2qytkn8Mq7qQVIQuSRFFEaWkpli9fjoKCAr/HNDQ0YPz48T6PSfcbGhqGPEZ6fiRefvlltLe34+GHH5YfKywsxN69e3Hs2DG88cYbqKmpwZw5c9DW1jbo+2zatAnx8fHyLTMzc8RtIAomaWSnprkDbk9oygDcaHeirbsHggBkJQ3s5JD+GA0CspO9v+uLIVzhdpFV3EkjAg6S1q1bNyDh+au3Cxcu4LXXXkNbWxvKy8tD0e4R279/PzZu3Ih33nkHt912m/z4t771LSxcuBDTpk1DcXExjhw5glu3buGdd94Z9L3Ky8vR2toq3+rq6sLxFYgGuCMxGlFGAc6e0PX4pd7+HQnRsEYZQ/IZpD6hHqUURRE1TVIVd07jkroFnJO0Zs0alJaWDnlMbm4uTp48icrKSlgsFp/nCgoKsGTJEuzbtw9paWlobGz0eV66L+UKDXbMSHKJDhw4gMcffxyHDh0aMGX3VQkJCZg4cSK+/PLLQY+xWCwDvg+REowGAROSY/Dl9XbU3OhAZghGergdSWTyBi6NIQuSbnY4Ye8doZyQzBFKUreAg6TU1FSkpqYOe9yOHTvw4osvyvevXbuG4uJiHDx4EIWFhQCAoqIiPPvss3C5XIiK8haqq6iowKRJk5CYmCgfc+LECaxevVp+r4qKChQVFQ35+b/+9a/x/e9/HwcOHMCCBQuGbW97ezsuXryI7373u8MeS6QGOSl9QdLcicP/TQaKG9tGplBvdCu9b3o8RyhJ/UKWk5SVlYUpU6bIt4kTJwIA8vLykJGRAQBYvHgxzGYzli5dik8//RQHDx7E9u3bUVZWJr/PqlWrcOzYMWzduhUXLlzAhg0bcObMGaxcuVI+pry8HI8++qh8f//+/Xj00UexdetWFBYWoqGhAQ0NDWhtbZWP+clPfoI//vGPuHTpEk6fPo1vf/vbMBqNWLRoUah+JERBFeqNbuUVSNyzLaLkytvehOa84mIA0hJFK27Hx8fj+PHjqKmpwaxZs7BmzRqsX79eXv4PALNnz8b+/fuxe/duTJ8+Hb/5zW9w+PBhTJkyRT6mvr7eZ2Xa7t270dPTgxUrVuD222+Xb6tWrZKPuXLlChYtWoRJkybh4YcfRnJyMj788MMRjZIRqUGoqyOzlk1kkn7f11q70O1yB/39azhCSRoS8jpJkuzsbL+bcU6bNg2nTp0a8rULFy7EwoULB31+7969Pvffe++9Ydtz4MCBYY8hUjMp6TUU+2y5PSIuN3f2fg4vZpEkKcaMOKsJ9u4eXGruQH5aXFDfv7qJuW6kHdy7jUijpIvM1VvB7/FfbemC0+2B2WRAeu8+cRQZBEHoq7wdgim3vhpJnMYl9WOQRKRRKePMiLWYIIpA7c3OoL53de/KtuxkW8iqLpN6hWoq1+0Rcal3hJLTbaQFDJKINMrb4w9Nki3zkSJbqGolXbvVBWcPRyhJOxgkEWlYqJZry8m1nBKJSKFaOSm9H0coSSsYJBFpWKiSt6WRKY4kRSbp9x7884pJ26QtDJKINCwnxD1+5o1Epuxk7++9pdOFlg5n0N63bxqXI5SkDQySiDQsFNNt3S43rvbuB8cef2SKsZiQFmcF4N1EOVhYxZ20hkESkYZl915smjucaO10BeU9L/VeFOOsJiTFmIPynqQ9cvJ2EBcF9C3/Z5BE2sAgiUjDxllMuC3Wu+lysHr80kUxJ3UcBIHJtZEq2FO5HKEkLWKQRKRxfSuRgpNkK02J5PFCFtGCPZVbe7MTougdoUzmCCVpBIMkIo3rW+EWnIsZV7YR0Bd8XwzSCjd5ZRtHKElDGCQRaVywqyNLI1LMG4lsUvB9qbkDHs/AfTcDxaRt0iIGSUQaF+wEW1bbJgDISIyGySCg2+VBg717zO9XwxFK0iAGSUQa1z/BVhTH1uNv6XCipXeVnFQrhyJTlNGArCQbgODkJTH4Ji1ikESkcZmJ3i0eulxuNNodY3ovaYVcWpwVMRZTMJpHGpYTxKlcBkmkRQySiDTObOrr8VePcYWbNCWSy3wkQr+Vk2Ocym3tdKG5t3I3gyTSEgZJRDrQt9fW2C5mUpDFCxkB/VZOjjH4ll7PEUrSGgZJRDqQE6SaNpwSof54XlGkY5BEpAPBuphVc7qN+pHOg7qbnXD2eEb9PtyOhLSKQRKRDgSjOrLHI8r7tnGXdgKA22ItiDEb4RG9FbNHizWSSKsYJBHpQG6qN6ipvdkJl3t0Pf4Geze6XR6YDAIyE6OD2TzSKEEQgrKHGxcEkFYxSCLSgfFxFkRHGeH2iKgbZY9fughmJdtgMvKfBvKSRhVHuzegKIr9cpI4Qknawn8JiXRAEIQxr3CT9tbilAj1N9bzqsHejS6XGyaDgAyOUJLGMEgi0omxTotUcwUS+THWvQGlqbasJBuiOEJJGsMzlkgnxnwx45QI+THWlZMMvknLGCQR6UTfxWx0uSOsZUP+SCOUTW0OtHW7An49zyvSMgZJRDohrXAbTY/f2eORE77zuAKJ+omzRiFlnAUAcOlG4IsCpPNROj+JtIRBEpFO5CR7g5tGuwMdjp6AXlt7sxMeEYgxG5EaawlF80jD+qZyAx+l5EgSaRmDJCKdiLdFITnGDCDw0SRpZVtOagwEQQh620jbRrvCzdnjkYtQskYSaRGDJCIdyRll8jaTtmkoo105WdfSCbdHhM1sxG0coSQNYpBEpCNy8naAPX5OidBQRrvCTToPc1I4QknaxCCJSEf6evyB5Y5wby0aSv+9AUVRHPHrGHyT1jFIItKR3JTRrXDrW4HEixkNlJVsg0EA2h09aGp3jPh11VzZRhrHIIlIR6Qgp7pp5D3+tm4Xmtq8F75s9vjJD4vJiIxEG4DAkre51Q1pHYMkIh3JSrJBEIA2Rw9utDtH9BppFCllnAVx1qhQNo80bDR5SZxuI61jkESkI9YoI+5I8G4iOtKLWQ3zkWgEAg2S2h09uM4RStI4BklEOhPo9iTVTezt0/D6T+WOxCV5hNKM+GiOUJI2MUgi0plAN7qVp0SYtE1DCDj45lQb6QCDJCKdkfdwG2GPn9NtNBLSeVV7sxM9bs+wx0vnXy4LlJKGMUgi0plAqm6Loti3AokjSTSE2+OssJgMcLlFXGnpGvZ4aZ83jlCSloUlSHI4HJgxYwYEQUBVVZXPc5988gnmzJkDq9WKzMxMbN68ecDrDx06hPz8fFitVkydOhVHjhwZ8vM++OAD3HfffUhOTkZ0dDTy8/PxyiuvDDju9ddfR3Z2NqxWKwoLC/HRRx+N6XsSqYEUJF1u7oDbM3QZgKY2BzqcbhgEIDPJFo7mkUYZDEJAydtc2UZ6EJYgae3atUhPTx/wuN1ux/z58zFhwgScPXsWW7ZswYYNG7B79275mNOnT2PRokVYunQpzp07h5KSEpSUlOD8+fODfl5MTAxWrlyJ999/H5999hmee+45PPfccz7ve/DgQZSVleH555/Hxx9/jOnTp6O4uBjXr18P7pcnCrP0hGiYe3v8V4fp8UujTRmJNlhMxnA0jzRspKOUoij2m25jkETaFfIg6ejRozh+/DhefvnlAc+9/fbbcDqd2LNnDyZPnozvfOc7ePLJJ7Ft2zb5mO3bt+PBBx/EU089hbvvvhsvvPAC7r33XuzcuXPQz5w5cyYWLVqEyZMnIzs7G//yL/+C4uJinDp1Sj5m27ZtWLZsGR577DHcc8892LVrF2w2G/bs2RPcHwBRmBkNArKTewv/DZNky94+BWKkyds32p1oc/RAELzVuom0KqRBUmNjI5YtW4a33noLNtvAP5TKykrMnTsXZrNZfqy4uBiff/45Wlpa5GPmzZvn87ri4mJUVlaOuB3nzp3D6dOncf/99wMAnE4nzp496/O+BoMB8+bNG/J9HQ4H7Ha7z41IjUY6LcIgiQIR6HmVkRjNEUrStJAFSaIoorS0FMuXL0dBQYHfYxoaGjB+/Hifx6T7DQ0NQx4jPT+UjIwMWCwWFBQUYMWKFXj88ccBADdu3IDb7Q74fTdt2oT4+Hj5lpmZOWwbiJQgr3Ab5mIm1bzJY3ItjcBIV05KI01c2UZaF3CQtG7dOgiCMOTtwoULeO2119DW1oby8vJQtHtETp06hTNnzmDXrl149dVX8etf/3pM71deXo7W1lb5VldXF6SWEgWXnDsyzMVMXoHEixmNgJRfdK21G53OnkGPY4FS0gtToC9Ys2YNSktLhzwmNzcXJ0+eRGVlJSwWi89zBQUFWLJkCfbt24e0tDQ0Njb6PC/dT0tLk//r7xjp+aHk5OQAAKZOnYrGxkZs2LABixYtQkpKCoxGY8Dva7FYBnwfIjXKHcG0SI/bg9rmTgBcpk0jkxhjRoItCrc6Xbh0oxP3pMf5PU5K7GZZCdK6gEeSUlNTkZ+fP+TNbDZjx44d+POf/4yqqipUVVXJy/YPHjyIn/3sZwCAoqIivP/++3C5XPL7V1RUYNKkSUhMTJSPOXHihE8bKioqUFRUFFC7PR4PHA7vPkJmsxmzZs3yeV+Px4MTJ04E/L5EaiT14K/e6kK3y+33mCstXejxiLCYDLg9zhrO5pGGjSQviblupBcBjySNVFZWls/9ceO8w/l5eXnIyMgAACxevBgbN27E0qVL8fTTT+P8+fPYvn27T02jVatW4f7778fWrVuxYMECHDhwAGfOnPFZzl9eXo6rV6/izTffBOCtf5SVlYX8/HwAwPvvv4+XX34ZTz75pPyasrIyfO9730NBQQG+/vWv49VXX0VHRwcee+yx0PxAiMIoKcaMOKsJ9u4eXGruQH7awB5//wuZwSCEu4mkUTkpMThXe2vQFW5uj4jLzQySSB9CFiSNRHx8PI4fP44VK1Zg1qxZSElJwfr16/HEE0/Ix8yePRv79+/Hc889h2eeeQZ33XUXDh8+jClTpsjH1NfXo7a2Vr7v8XhQXl6OmpoamEwm5OXl4aWXXsIPfvAD+ZhHHnkETU1NWL9+PRoaGjBjxgwcO3ZsQDI3kRYJgoDc1HGoqruFmib/QRKnRGg08nqTtwerlXS1pQsut3eEMj0+OpxNIwq6sAVJ2dnZEMWB1X+nTZvmU7/In4ULF2LhwoWDPr93716f+z/60Y/wox/9aNg2rVy5EitXrhz2OCItyk2JQVXdrUEvZjVy0jaDJBq54abbqvudVxyhJK3j3m1EOjXcCre+FUhc2UYj1/+88tfx5co20hMGSUQ6Ja1YGyx3hMm1NBrZyd7zpbXLhZZO14DneV6RnjBIItKpoaZFOp09qG/tBsC9tSgw0WYj0uO9qyH9BeAMkkhPGCQR6ZR0kWrpdKGlw+nz3KUb3vpICbYoJMaYB7yWaCjSKKW/qdwaLgggHWGQRKRTNrMJt0s9/mbfi5l8IWNvn0ZB2m7kq6OU3S43rt7q8jmGSMsYJBHpmDzl1vTVIInbkdDoDTaVe6k3GOcIJekFgyQiHZNXIn0ld0SaJuGUCI3GYNNtXNlGesMgiUjHBuvxVzO5lsZA3huwuQMeT18ZACZtk94wSCLSsVw/PX5RFFHdxEKSNHp3JEQjyijA2ePBtdYu+XF5hJLnFekEgyQiHZNyji716/G3dLpg7+4B0FfzhigQJqMBWUk2AL6jlMx1I71hkESkY5mJ0TAZBHS7PGiwe+siSReyOxKiEW02Ktk80rDc1IEr3Lj8n/SGQRKRjpmMBmQl+/b4mVxLwZD7lW1vWjqccgVujlCSXjBIItK5vouZdwSJSdsUDH0rJzt8/pseb+UIJekGgyQinfvqxayGI0kUBH0rJ9t7/9t7XnGqjXSEQRKRzuV8pToyL2YUDNL5c6WlC44ed7+kbZ5XpB8Mkoh0rn+tJI9HlLco4TJtGovUcRaMs5ggikBtc2e/Gklc2Ub6wSCJSOfyenv8dTc7cflmJ5w9HkQZBWQk2hRuGWmZIAh9dbhudLCKO+kSgyQinUuNtSDGbIRHBN77/DoAYEJyDIwGQeGWkdZJo5RfXm+X923jCCXpCYMkIp0TBEHOHznxmTdIYt4IBYN0HlVebEa3yztCeUdCtMKtIgoeBklEEUDKE/l/Nc0A2Nun4JCCJOm8ykqywWTkZYX0g2czUQSQLmYut+hzn2gscnuD777ziknbpC8MkogiwFdHjhgkUTBkp/gm/zNpm/SGQRJRBPjqxUvad4toLGKtUbgt1iLf5zQu6Q2DJKIIkN3v4hVrMSFlnFnB1pCe9B+V5Agl6Q2DJKIIEGeNQso4b48/JzUGgsDl/xQc/UcpWcWd9IZBElGEkKZC2NunYJLOp3EWE1LHWYY5mkhbGCQRRYhJabE+/yUKhklpcQCAiePHcYSSdMekdAOIKDx+9Hd3Ii81Bv80K0PpppCOzLkzBT/79hR8LTtJ6aYQBZ0giqKodCO0ym63Iz4+Hq2trYiLi1O6OURERDQCI71+c7qNiIiIyA8GSURERER+MEgiIiIi8oNBEhEREZEfDJKIiIiI/GCQREREROQHgyQiIiIiPxgkEREREfnBIImIiIjIDwZJRERERH6EJUhyOByYMWMGBEFAVVWVz3OffPIJ5syZA6vViszMTGzevHnA6w8dOoT8/HxYrVZMnToVR44cGfLzPvjgA9x3331ITk5GdHQ08vPz8corr/gcs2HDBgiC4HPLz88f83clIiIifQjLBrdr165Feno6/vznP/s8brfbMX/+fMybNw+7du3CX/7yF3z/+99HQkICnnjiCQDA6dOnsWjRImzatAn/8A//gP3796OkpAQff/wxpkyZ4vfzYmJisHLlSkybNg0xMTH44IMP8IMf/AAxMTHy+wLA5MmT8X//7/+V75tM3O+XiIiIvEK+we3Ro0dRVlaGd999F5MnT8a5c+cwY8YMAMAbb7yBZ599Fg0NDTCbzQCAdevW4fDhw7hw4QIA4JFHHkFHRwd+//vfy+/5N3/zN5gxYwZ27do14nb80z/9E2JiYvDWW28B8I4kHT58eMDIViC4wS0REZH2jPT6HdKhk8bGRixbtgyHDx+GzWYb8HxlZSXmzp0rB0gAUFxcjJdeegktLS1ITExEZWUlysrKfF5XXFyMw4cPj7gd586dw+nTp/Hiiy/6PP7FF18gPT0dVqsVRUVF2LRpE7KysgZ9H4fDAYfDId9vbW0F4P1hExERkTZI1+3hxolCFiSJoojS0lIsX74cBQUFuHTp0oBjGhoakJOT4/PY+PHj5ecSExPR0NAgP9b/mIaGhmHbkJGRgaamJvT09GDDhg14/PHH5ecKCwuxd+9eTJo0CfX19di4cSPmzJmD8+fPIzY21u/7bdq0CRs3bhzweGZm5rBtISIiInVpa2tDfHz8oM8HHCStW7cOL7300pDHfPbZZzh+/Dja2tpQXl4e6EcEzalTp9De3o4PP/wQ69atw5133olFixYBAL71rW/Jx02bNg2FhYWYMGEC3nnnHSxdutTv+5WXl/uMank8Hty8eRPJyckQBCFo7bbb7cjMzERdXR2n8VSAvw/14e9EXfj7UBf+PoYniiLa2tqQnp4+5HEBB0lr1qxBaWnpkMfk5ubi5MmTqKyshMVi8XmuoKAAS5Yswb59+5CWlobGxkaf56X7aWlp8n/9HSM9PxRplGrq1KlobGzEhg0b5CDpqxISEjBx4kR8+eWXg76fxWIZ8H0SEhKGbcdoxcXF8QRXEf4+1Ie/E3Xh70Nd+PsY2lAjSJKAg6TU1FSkpqYOe9yOHTt8coCuXbuG4uJiHDx4EIWFhQCAoqIiPPvss3C5XIiKigIAVFRUYNKkSUhMTJSPOXHiBFavXi2/V0VFBYqKigJqt8fj8ckn+qr29nZcvHgR3/3udwN6XyIiItKnkOUkfTUBety4cQCAvLw8ZGRkAAAWL16MjRs3YunSpXj66adx/vx5bN++3aem0apVq3D//fdj69atWLBgAQ4cOIAzZ85g9+7d8jHl5eW4evUq3nzzTQDA66+/jqysLLnu0fvvv4+XX34ZTz75pPyan/zkJ3jooYcwYcIEXLt2Dc8//zyMRuOgI01EREQUWRQtDBQfH4/jx49jxYoVmDVrFlJSUrB+/XqfWkazZ8/G/v378dxzz+GZZ57BXXfdhcOHD/vUSKqvr0dtba183+PxoLy8HDU1NTCZTMjLy8NLL72EH/zgB/IxV65cwaJFi9Dc3IzU1FT87d/+LT788MMRjZKFmsViwfPPPz9gao+Uwd+H+vB3oi78fagLfx/BE/I6SURERERaxL3biIiIiPxgkERERETkB4MkIiIiIj8YJBERERH5wSBJhV5//XVkZ2fDarWisLAQH330kdJNikibNm3C1772NcTGxuK2225DSUkJPv/8c6WbRb3+5//8nxAEwaeGGoXX1atX8S//8i9ITk5GdHQ0pk6dijNnzijdrIjldrvx05/+FDk5OYiOjkZeXh5eeOGFYfcno8ExSFKZgwcPoqysDM8//zw+/vhjTJ8+HcXFxbh+/brSTYs4f/zjH7FixQp8+OGHqKiogMvlwvz589HR0aF00yLen/70J/zrv/4rpk2bpnRTIlZLSwvuu+8+REVF4ejRo/jv//5vbN26VS4ETOH30ksv4Y033sDOnTvx2Wef4aWXXsLmzZvx2muvKd00zWIJAJUpLCzE1772NezcuROAt+ZTZmYmfvSjH2HdunUKty6yNTU14bbbbsMf//hHzJ07V+nmRKz29nbce++9+MUvfoEXX3wRM2bMwKuvvqp0syLOunXr8F//9V84deqU0k2hXv/wD/+A8ePH43/9r/8lP/bP//zPiI6Oxq9+9SsFW6ZdHElSEafTibNnz2LevHnyYwaDAfPmzUNlZaWCLSMAaG1tBQAkJSUp3JLItmLFCixYsMDn74TC7//8n/+DgoICLFy4ELfddhtmzpyJX/7yl0o3K6LNnj0bJ06cwF//+lcAwJ///Gd88MEHPhu6U2AUrbhNvm7cuAG3243x48f7PD5+/HhcuHBBoVYR4B3RW716Ne677z6fau8UXgcOHMDHH3+MP/3pT0o3JeJVV1fjjTfeQFlZGZ555hn86U9/wpNPPgmz2Yzvfe97SjcvIq1btw52ux35+fkwGo1wu9342c9+hiVLlijdNM1ikEQ0AitWrMD58+fxwQcfKN2UiFVXV4dVq1ahoqICVqtV6eZEPI/Hg4KCAvz85z8HAMycORPnz5/Hrl27GCQp5J133sHbb7+N/fv3Y/LkyaiqqsLq1auRnp7O38koMUhSkZSUFBiNRjQ2Nvo83tjYiLS0NIVaRStXrsTvf/97vP/++/LmzBR+Z8+exfXr13HvvffKj7ndbrz//vvYuXMnHA4HjEajgi2MLLfffjvuuecen8fuvvtuvPvuuwq1iJ566imsW7cO3/nOdwAAU6dOxeXLl7Fp0yYGSaPEnCQVMZvNmDVrFk6cOCE/5vF4cOLECRQVFSnYssgkiiJWrlyJf//3f8fJkyeRk5OjdJMi2t/93d/hL3/5C6qqquRbQUEBlixZgqqqKgZIYXbfffcNKInx17/+FRMmTFCoRdTZ2QmDwfeybjQa4fF4FGqR9nEkSWXKysrwve99DwUFBfj617+OV199FR0dHXjssceUblrEWbFiBfbv34/f/e53iI2NRUNDAwAgPj4e0dHRCrcu8sTGxg7IB4uJiUFycjLzxBTw4x//GLNnz8bPf/5zPPzww/joo4+we/du7N69W+mmRayHHnoIP/vZz5CVlYXJkyfj3Llz2LZtG77//e8r3TTNYgkAFdq5cye2bNmChoYGzJgxAzt27EBhYaHSzYo4giD4ffx//+//jdLS0vA2hvz6xje+wRIACvr973+P8vJyfPHFF8jJyUFZWRmWLVumdLMiVltbG37605/i3//933H9+nWkp6dj0aJFWL9+Pcxms9LN0yQGSURERER+MCeJiIiIyA8GSURERER+MEgiIiIi8oNBEhEREZEfDJKIiIiI/GCQREREROQHgyQiIiIiPxgkEREREfnBIImIiIjIDwZJRERERH4wSCIiIiLyg0ESERERkR//P0wh55WTiGzrAAAAAElFTkSuQmCC" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(all_rewards)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:28:10.099466Z", + "start_time": "2023-12-13T17:28:09.937126Z" + } + }, + "id": "e48689f2a72e24dc" + }, + { + "cell_type": "code", + "execution_count": 13, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Reinitialize the agent\n", + "\n", + "cog.stop_service(\"coltra\")\n", + "\n", + "model = MLPModel(\n", + " config={\n", + " \"hidden_sizes\": [64, 64],\n", + " }, \n", + " observation_space=cenv.env.observation_space, \n", + " action_space=cenv.env.action_space\n", + ")\n", + "\n", + "# Put the model in shared memory so that the actor can access it\n", + "model.share_memory()\n", + "actor = ColtraActor(model=model)\n", + "\n", + "\n", + "await cog.run_actor(\n", + " actor=actor,\n", + " actor_name=\"coltra\",\n", + " port=9021,\n", + " log_file=\"actor.log\"\n", + ")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:28:21.263799Z", + "start_time": "2023-12-13T17:28:19.093753Z" + } + }, + "id": "5c1585be28fdae6c" + }, + { + "cell_type": "code", + "execution_count": 15, + "outputs": [], + "source": [ + "# Get some human episodes\n", + "episodes = []\n", + "for i in range(1):\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"gym\": \"web_ui\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id, env_name=\"mcar\")\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + " \n", + "all_data = concatenate(episodes)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:30:08.137344Z", + "start_time": "2023-12-13T17:29:54.830169Z" + } + }, + "id": "8f1381b80d4c8799" + }, + { + "cell_type": "code", + "execution_count": 16, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean_reward: -151.0\n", + "rewards: [-151.0]\n" + ] + } + ], + "source": [ + "mean_reward = np.mean([sum(e.rewards) for e in episodes])\n", + "print(f\"mean_reward: {mean_reward}\")\n", + "print(f\"rewards: {[sum(e.rewards) for e in episodes]}\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:30:11.105847Z", + "start_time": "2023-12-13T17:30:11.099839Z" + } + }, + "id": "73d139b8e5d005d8" + }, + { + "cell_type": "code", + "execution_count": 17, + "outputs": [], + "source": [ + "cog.stop_service(\"web_ui\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:30:20.978109Z", + "start_time": "2023-12-13T17:30:19.950558Z" + } + }, + "id": "73c751b525abf31e" + }, + { + "cell_type": "code", + "execution_count": 18, + "outputs": [], + "source": [ + "all_obs = Observation(vector=all_data.observations).tensor()\n", + "all_actions = torch.tensor(all_data.actions)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:30:28.238655Z", + "start_time": "2023-12-13T17:30:28.234181Z" + } + }, + "id": "cdcef32e17f9afc1" + }, + { + "cell_type": "code", + "execution_count": 23, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "loss: 0.28: 100%|██████████| 500/500 [00:00<00:00, 1018.11it/s] \n" + ] + } + ], + "source": [ + "losses = []\n", + "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n", + "for t in (pbar := trange(500)):\n", + " preds = model(all_obs)[0].logits\n", + " loss = F.cross_entropy(preds, all_actions)\n", + " \n", + " optimizer.zero_grad()\n", + " loss.backward()\n", + " optimizer.step()\n", + " \n", + " pbar.set_description(f\"loss: {loss.item():.3}\")\n", + " \n", + " losses.append(loss.item())\n", + " " + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:30:58.439176Z", + "start_time": "2023-12-13T17:30:57.943699Z" + } + }, + "id": "9a1cb51957e9f672" + }, + { + "cell_type": "code", + "execution_count": 24, + "outputs": [ + { + "data": { + "text/plain": "[]" + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABGtklEQVR4nO3de1xUdf4/8NdcmBluM4DAcHEU8a4oJCphN1sxdNvU/e2FyrKobNesb0WXjW+bbtk3tt3v+m3b3GzdTK39ltW325ZRNl7KRCy8oSKKgIgww30GBpiBmfP7Y2BsEtRB4AzD6/l4nAd6zucc3nM0eXU+lyMRBEEAERERkReTil0AERER0aUwsBAREZHXY2AhIiIir8fAQkRERF6PgYWIiIi8HgMLEREReT0GFiIiIvJ6DCxERETk9eRiF9AfHA4HqqqqEBwcDIlEInY5REREdBkEQUBzczNiYmIglV78GYpPBJaqqirodDqxyyAiIqI+OHv2LEaOHHnRNj4RWIKDgwE4P7BarRa5GiIiIrocZrMZOp3O9XP8YnwisHR3A6nVagYWIiKiIeZyhnNw0C0RERF5PQYWIiIi8noMLEREROT1GFiIiIjI6zGwEBERkddjYCEiIiKvx8BCREREXo+BhYiIiLweAwsRERF5PQYWIiIi8noMLEREROT1GFiIiIjI6zGwXERTqw2v7T6NJ98/LHYpREREwxoDy0XYHQJezD2Bd7+vxLmmNrHLISIiGrYYWC5iRJASM0eHAQC2HzOIXA0REdHwxcByCTdN1QIAvjxuFLkSIiKi4YuB5RJumhIFAMgva0BTq03kaoiIiIYnBpZLGDUiAJOigmF3CNhxokbscoiIiIYlBpbLcNNU51OWLziOhYiISBQMLJfhpinOcSy7T9aizWYXuRoiIqLhh4HlMkyNUSM2xB/tHQ7sKakTuxwiIqJhh4HlMkgkEszvesryJbuFiIiIBh0Dy2Xqnt78VZERnXaHyNUQERENLwwsl2l2XBg0/n5obO1AwZlGscshIiIaVhhYLpNcJsW8yZEAuIgcERHRYGNg8UD3InJfHjdAEASRqyEiIho+GFg8cMOECKj8pDjb0Iai6maxyyEiIho2GFg84K+Q4brxEQCcT1mIiIhocDCweOgm1/RmjmMhIiIaLH0KLOvWrUNcXBxUKhVSUlKwf//+i7ZvamrCypUrER0dDaVSiQkTJmDbtm2u43/4wx8gkUjctkmTJvWltAE3b7IWUglwvNqMsw2tYpdDREQ0LHgcWLZu3YqsrCysXr0aBw4cQGJiItLT01FT0/OLAW02G+bPn4/y8nK8//77KC4uxoYNGxAbG+vWburUqaiurnZte/bs6dsnGmBhgQrMigsDAGznbCEiIqJBIff0hLVr12L58uXIzMwEAKxfvx6fffYZNm7ciKeeeuqC9hs3bkRDQwP27t0LPz8/AEBcXNyFhcjliIqK8rQcUdw0NQr5ZQ348rgB91w7RuxyiIiIfJ5HT1hsNhsKCgqQlpZ2/gJSKdLS0pCXl9fjOZ988glSU1OxcuVKaLVaJCQk4IUXXoDd7v4SwVOnTiEmJgbx8fFYunQpKioqeq3DarXCbDa7bYOpexzL/rIGNFpsg/q9iYiIhiOPAktdXR3sdju0Wq3bfq1WC4Oh51kzpaWleP/992G327Ft2zY888wz+Mtf/oLnn3/e1SYlJQWbNm1Cbm4uXn31VZSVleG6665Dc3PPU4dzcnKg0Whcm06n8+RjXDFdWAAmR6vhEAD9iZ67woiIiKj/DPgsIYfDgcjISPzjH/9AcnIyMjIy8PTTT2P9+vWuNgsXLsSvfvUrTJ8+Henp6di2bRuamprw7rvv9njN7OxsmEwm13b27NmB/hgX6H7K8gVfhkhERDTgPAos4eHhkMlkMBrdB5sajcZex59ER0djwoQJkMlkrn2TJ0+GwWCAzdZzd0pISAgmTJiAkpKSHo8rlUqo1Wq3bbClT3V+3m9O1aLNZr9EayIiIroSHgUWhUKB5ORk6PV61z6HwwG9Xo/U1NQez7nmmmtQUlICh+P8G45PnjyJ6OhoKBSKHs9paWnB6dOnER0d7Ul5g2pydDBGhvqjvcOBr0/Vil0OERGRT/O4SygrKwsbNmzA5s2bUVRUhBUrVsBisbhmDS1btgzZ2dmu9itWrEBDQwMefvhhnDx5Ep999hleeOEFrFy50tXm8ccfx+7du1FeXo69e/fi5z//OWQyGW677bZ++IgDQyKRnH+3EBeRIyIiGlAeT2vOyMhAbW0tVq1aBYPBgKSkJOTm5roG4lZUVEAqPZ+DdDodvvjiCzz66KOYPn06YmNj8fDDD+N3v/udq01lZSVuu+021NfXIyIiAtdeey327duHiIiIfviIA+emqVps/LYM+hNGdNodkMu4cDAREdFAkAg+8Nphs9kMjUYDk8k0qONZOu0OzPqvr9DY2oG3l1+N1LEjBu17ExERDXWe/PzmI4ErIJdJMW9y17uF+DJEIiKiAcPAcoV++DJEH3hYRURE5JUYWK7QdeMjoPKT4lxTG45XD+6Ku0RERMMFA8sV8lfIcO045+BgfRFXvSUiIhoIDCz9IG1yJABAX8TpzURERAOBgaUf/GSSM7AcrjShxtwucjVERES+h4GlH0SqVUgcqQEA7ODLEImIiPodA0s/6Z7e/BXHsRAREfU7BpZ+ktYVWPaU1KK9gy9DJCIi6k8MLP1kcnQwYjQqtHc48G1JndjlEBER+RQGln4ikUjYLURERDRAGFj60byu6c07TnDVWyIiov7EwNKPro4fgQCFDEazFUfPcdVbIiKi/sLA0o9UfjJcP9656u12LiJHRETUbxhY+tk8rnpLRETU7xhY+tmNkyIhkQDHqsyoNrWJXQ4REZFPYGDpZ+FBSlylCwHAlyESERH1FwaWAdA9vZndQkRERP2DgWUAdK96++3perTaOkWuhoiIaOhjYBkAE7RB0IX5w9bpwDenuOotERHRlWJgGQASiQTzJrFbiIiIqL8wsAyQ7m6hHSdq4XBw1VsiIqIrwcAyQGaPCUOwUo66FisOVzaJXQ4REdGQxsAyQBRyKa6f4Fz1ltObiYiIrgwDywBKm+Jc9fYrjmMhIiK6IgwsA2juhEhIJcAJQzMqG1vFLoeIiGjIYmAZQKGBCswcHQaA3UJERERXgoFlgHW/DJHdQkRERH3HwDLAupfpzy9tQIuVq94SERH1BQPLABsbEYgx4YGw2R345mSt2OUQERENSQwsA8y56q2zW2g7u4WIiIj6pE+BZd26dYiLi4NKpUJKSgr2799/0fZNTU1YuXIloqOjoVQqMWHCBGzbtu2KrjmUdHcL7SquhZ2r3hIREXnM48CydetWZGVlYfXq1Thw4AASExORnp6OmpqeZ8HYbDbMnz8f5eXleP/991FcXIwNGzYgNja2z9ccambGhUKtkqPBYsPBikaxyyEiIhpyJIIgePS//CkpKZg1axZeeeUVAIDD4YBOp8NDDz2Ep5566oL269evx5///GecOHECfn5+/XLNHzObzdBoNDCZTFCr1Z58nEHzH28fxCeHq/DbG8biqYWTxC6HiIhIdJ78/PboCYvNZkNBQQHS0tLOX0AqRVpaGvLy8no855NPPkFqaipWrlwJrVaLhIQEvPDCC7Db7X2+5lCUNoVvbyYiIuoruSeN6+rqYLfbodVq3fZrtVqcOHGix3NKS0uxY8cOLF26FNu2bUNJSQkeeOABdHR0YPXq1X26ptVqhdVqdf3ebDZ78jFEccOECMilEpyqacGZegtGjwgUuyQiIqIhY8BnCTkcDkRGRuIf//gHkpOTkZGRgaeffhrr16/v8zVzcnKg0Whcm06n68eKB4bG3w+z4pyr3n7FVW+JiIg84lFgCQ8Ph0wmg9Ho3q1hNBoRFRXV4znR0dGYMGECZDKZa9/kyZNhMBhgs9n6dM3s7GyYTCbXdvbsWU8+hmi6V71ltxAREZFnPAosCoUCycnJ0Ov1rn0OhwN6vR6pqak9nnPNNdegpKQEDofDte/kyZOIjo6GQqHo0zWVSiXUarXbNhSkdU1v3l/WAHN7h8jVEBERDR0edwllZWVhw4YN2Lx5M4qKirBixQpYLBZkZmYCAJYtW4bs7GxX+xUrVqChoQEPP/wwTp48ic8++wwvvPACVq5cednX9BVx4YEYGxGIToeA3cVc9ZaIiOhyeTToFgAyMjJQW1uLVatWwWAwICkpCbm5ua5BsxUVFZBKz+cgnU6HL774Ao8++iimT5+O2NhYPPzww/jd73532df0JWlTtDi9uxT6IiNuSYwRuxwiIqIhweN1WLzRUFiHpdv35Q345fo8BKvkKPj9fCjkfDsCERENTwO2DgtduatGhSI8SInm9k7sPV0ndjlERERDAgPLIJNJJUif6uzqyj1qELkaIiKioYGBRQQLE6IBAF8eN6LT7rhEayIiImJgEUFKfBhCAvzQYLFhf3mD2OUQERF5PQYWEfjJpJg/md1CREREl4uBRSQLpzlX8c09aoDDMeQnahEREQ0oBhaRXDMuHMFKOWqarTh4tlHscoiIiLwaA4tIlHIZftL1bqHPC9ktREREdDEMLCJamODsFvr8qAE+sH4fERHRgGFgEdENEyLh7yfDuaY2HD1nFrscIiIir8XAIiJ/hQxzJ0YAAD4/Wi1yNURERN6LgUVkCxLOzxZitxAREVHPGFhE9pNJkVDIpCits+CksUXscoiIiLwSA4vIglV+uG58OAB2CxEREfWGgcULLJzmfLcQV70lIiLqGQOLF5g/WQu5VIIThmaU1VnELoeIiMjrMLB4AU2AH1LHjgDAbiEiIqKeMLB4iYUJ7BYiIiLqDQOLl7hpqhZSCXCk0oTKxlaxyyEiIvIqDCxeIjxIiVlxYQD4lIWIiOjHGFi8yMIfLCJHRERE5zGweJEFXeNYCioaUWNuF7kaIiIi78HA4kWiNCpcNSoEggB8cYxPWYiIiLoxsHiZ7m6hbYUMLERERN0YWLxM9/Tm/LJ61LdYRa6GiIjIOzCweBldWACmxqjhEIDtx41il0NEROQVGFi8UHe30OecLURERASAgcUrdc8W2nu6Dqa2DpGrISIiEh8DixcaFxmE8ZFB6LAL0BexW4iIiIiBxUuxW4iIiOg8BhYv1d0t9PXJWlisnSJXQ0REJC4GFi81OToYo0cEwNrpwM7iGrHLISIiElWfAsu6desQFxcHlUqFlJQU7N+/v9e2mzZtgkQicdtUKpVbm7vvvvuCNgsWLOhLaT5DIpFgAbuFiIiIAPQhsGzduhVZWVlYvXo1Dhw4gMTERKSnp6OmpvenAGq1GtXV1a7tzJkzF7RZsGCBW5u3337b09J8TvcicjtP1KC9wy5yNUREROLxOLCsXbsWy5cvR2ZmJqZMmYL169cjICAAGzdu7PUciUSCqKgo16bVai9oo1Qq3dqEhoZ6WprPSRypQYxGhVabHV+frBW7HCIiItF4FFhsNhsKCgqQlpZ2/gJSKdLS0pCXl9freS0tLRg9ejR0Oh0WL16MY8eOXdBm165diIyMxMSJE7FixQrU19f3ej2r1Qqz2ey2+SKJRIL0rm6hXHYLERHRMOZRYKmrq4Pdbr/gCYlWq4XB0PMP1IkTJ2Ljxo34+OOP8dZbb8HhcGDOnDmorKx0tVmwYAG2bNkCvV6PF198Ebt378bChQtht/fcDZKTkwONRuPadDqdJx9jSOnuFtpeZISt0yFyNUREROKQD/Q3SE1NRWpqquv3c+bMweTJk/Haa69hzZo1AIBbb73VdXzatGmYPn06xo4di127dmHevHkXXDM7OxtZWVmu35vNZp8NLcmjQxEepERdixV7T9dh7sRIsUsiIiIadB49YQkPD4dMJoPR6L76qtFoRFRU1GVdw8/PD1dddRVKSkp6bRMfH4/w8PBe2yiVSqjVarfNV8mkEixIcD7R2lZYLXI1RERE4vAosCgUCiQnJ0Ov17v2ORwO6PV6t6coF2O321FYWIjo6Ohe21RWVqK+vv6ibYaTm6fFAHCOY2G3EBERDUcezxLKysrChg0bsHnzZhQVFWHFihWwWCzIzMwEACxbtgzZ2dmu9s899xy+/PJLlJaW4sCBA7jjjjtw5swZ3HfffQCcA3KfeOIJ7Nu3D+Xl5dDr9Vi8eDHGjRuH9PT0fvqYQ9vsMWGIDFbC3N6Jb05xthAREQ0/Ho9hycjIQG1tLVatWgWDwYCkpCTk5ua6BuJWVFRAKj2fgxobG7F8+XIYDAaEhoYiOTkZe/fuxZQpUwAAMpkMR44cwebNm9HU1ISYmBjcdNNNWLNmDZRKZT99zKFNJpXgp9OisWlvOf59uArzJl84LZyIiMiXSQRBEMQu4kqZzWZoNBqYTCafHc9ScKYRv3h1LwIVMhQ8Mx8qP5nYJREREV0RT35+811CQ8SMUSGIDfGHxWbHzhN8txAREQ0vDCxDhEQiwc+mOwchf3qEs4WIiGh4YWAZQn423TlbSH/CCIu1U+RqiIiIBg8DyxCSEKtG3IgAtHc48FWR8dInEBER+QgGliHE2S3kfMrCbiEiIhpOGFiGmFsSnYFld3EtTG0dIldDREQ0OBhYhpiJUcEYHxkEm92B7cfZLURERMMDA8sQ1P2U5d+Hq0SuhIiIaHAwsAxB3dObvy2pQ4PFJnI1REREA4+BZQiKjwjC1Bg1Oh0Cco8axC6HiIhowDGwDFHnZwuxW4iIiHwfA8sQ1d0ttK+0HjXN7SJXQ0RENLAYWIYoXVgAknQhcAjA54XsFiIiIt/GwDKEdT9l4WwhIiLydQwsQ9jPpsdAIgG+P9OIqqY2scshIiIaMAwsQ1iURoVZo8MAAJ9xqX4iIvJhDCxD3C2Jzm4hzhYiIiJfxsAyxC1IiIZUAhyuNOFMvUXscoiIiAYEA8sQFxGsxJyx4QD4BmciIvJdDCw+oHu2EAMLERH5KgYWH7AgIQpyqQRF1WaU1LSIXQ4REVG/Y2DxASEBClw3vrtbiINviYjI9zCw+IhbEp3vFvr34SoIgiByNURERP2LgcVHzJ+ihUIuxelaC04YmsUuh4iIqF8xsPiIYJUfbpwYAYBL9RMRke9hYPEhP5vu7Bb69Eg1u4WIiMinMLD4kHmTI+HvJ0NFQyuOVJrELoeIiKjfMLD4kACFHPMmRwLgbCEiIvItDCw+pnu20KdHquFwsFuIiIh8AwOLj7lhQgSClXJUm9pxoKJR7HKIiIj6BQOLj1H5yTB/qhYAZwsREZHvYGDxQbd0zRb6rNAAO7uFiIjIB/QpsKxbtw5xcXFQqVRISUnB/v37e227adMmSCQSt02lUrm1EQQBq1atQnR0NPz9/ZGWloZTp071pTQCcM24cGj8/VDXYkV+ab3Y5RAREV0xjwPL1q1bkZWVhdWrV+PAgQNITExEeno6ampqej1HrVajurratZ05c8bt+J/+9Ce8/PLLWL9+PfLz8xEYGIj09HS0t7d7/okICrkUCxOiAAD/5huciYjIB3gcWNauXYvly5cjMzMTU6ZMwfr16xEQEICNGzf2eo5EIkFUVJRr02q1rmOCIOCll17C73//eyxevBjTp0/Hli1bUFVVhY8++qhPH4rOLyL3+dFq2DodIldDRER0ZTwKLDabDQUFBUhLSzt/AakUaWlpyMvL6/W8lpYWjB49GjqdDosXL8axY8dcx8rKymAwGNyuqdFokJKS0us1rVYrzGaz20buro4Pg1atRFNrBz4/yqcsREQ0tHkUWOrq6mC3292ekACAVquFwWDo8ZyJEydi48aN+Pjjj/HWW2/B4XBgzpw5qKysBADXeZ5cMycnBxqNxrXpdDpPPsawIJdJcfvs0QCALXlnLtGaiIjIuw34LKHU1FQsW7YMSUlJuOGGG/DBBx8gIiICr732Wp+vmZ2dDZPJ5NrOnj3bjxX7jttm6yCXSlBwphHHqrhUPxERDV0eBZbw8HDIZDIYjUa3/UajEVFRUZd1DT8/P1x11VUoKSkBANd5nlxTqVRCrVa7bXShSLUKC7oG377JpyxERDSEeRRYFAoFkpOTodfrXfscDgf0ej1SU1Mv6xp2ux2FhYWIjo4GAIwZMwZRUVFu1zSbzcjPz7/sa1LvlqXGAQA+OnQOptYOcYshIiLqI4+7hLKysrBhwwZs3rwZRUVFWLFiBSwWCzIzMwEAy5YtQ3Z2tqv9c889hy+//BKlpaU4cOAA7rjjDpw5cwb33XcfAOcMokceeQTPP/88PvnkExQWFmLZsmWIiYnBkiVL+udTDmOz4kIxKSoY7R0OvFfArjMiIhqa5J6ekJGRgdraWqxatQoGgwFJSUnIzc11DZqtqKiAVHo+BzU2NmL58uUwGAwIDQ1FcnIy9u7diylTprjaPPnkk7BYLLj//vvR1NSEa6+9Frm5uRcsMEeek0gkuDN1NJ7+8Cje2ncG91wzBlKpROyyiIiIPCIRBGHIr91uNpuh0WhgMpk4nqUHFmsnrn5Bj2ZrJzbfMxs3TIgQuyQiIiKPfn7zXULDQKBSjl8kjwQAvJlXLm4xREREfcDAMkzcmepck0V/ogZnG1pFroaIiMgzDCzDxNiIIFw7LhyCALyVzynOREQ0tDCwDCPLup6yvPvdWbR32EWuhoiI6PIxsAwj8yZrERvij8bWDvz7cJXY5RAREV02BpZhRCaV4I6rnU9ZNueVwwcmiBER0TDBwDLMZMzSQSGX4ug5Mw6ebRK7HCIiosvCwDLMhAUqsCgxBgCweW+5uMUQERFdJgaWYeiurvcLbSusRk1zu7jFEBERXQYGlmFo2kgNrhoVgg67gHf28/1CRETk/RhYhqm758QBAP6VfwYddoe4xRAREV0CA8swtTAhGuFBShjNVnx5zCh2OURERBfFwDJMKeRS3D5bB8A5xZmIiMibMbAMY7enjIZMKsH+sgYUVZvFLoeIiKhXDCzDWJRGhQVTowAAW/L4fiEiIvJeDCzDXPf7hT48WImmVpvI1RAREfWMgWWYmz0mDJOigtHe4cC733OKMxEReScGlmFOIpEg85o4AM5uIbuD7xciIiLvw8BCWJwUi5AAP1Q2tkFfxCnORETkfRhYCCo/GW6dNQoA8M89ZSJXQ0REdCEGFgLgHHzrJ3NOcf6+vEHscoiIiNwwsBAAICbEH7+YMRIA8MrOEpGrISIicsfAQi4r5o6FVALsKq5FYaVJ7HKIiIhcGFjIZfSIQCxOigUAvLLzlMjVEBERncfAQm4emDsWEgnwxTEjig3NYpdDREQEgIGFfmS8NhgLE5zL9a/dXixyNURERE4MLHSBR9MmQNr1lKXgDGcMERGR+BhY6ALjtcH49UwdAOCFbScgCFz9loiIxMXAQj16dP4EqPykKDjTiC+Pc/VbIiISFwML9UirVuG+a+MBAC/mnkCn3SFyRURENJwxsFCvfnNDPMICFSittWAr3+RMREQiYmChXgWr/PAfPxkHAHjpq1NosXaKXBEREQ1XfQos69atQ1xcHFQqFVJSUrB///7LOu+dd96BRCLBkiVL3PbffffdkEgkbtuCBQv6Uhr1s9tTRiNuRABqm614ZQeX7CciInF4HFi2bt2KrKwsrF69GgcOHEBiYiLS09NRU1Nz0fPKy8vx+OOP47rrruvx+IIFC1BdXe3a3n77bU9LowGgkEvxzM+mAABe31OKsjqLyBUREdFw5HFgWbt2LZYvX47MzExMmTIF69evR0BAADZu3NjrOXa7HUuXLsWzzz6L+Pj4HtsolUpERUW5ttDQUE9LowHyk0mRmDsxAh12AWs+PS52OURENAx5FFhsNhsKCgqQlpZ2/gJSKdLS0pCXl9frec899xwiIyNx77339tpm165diIyMxMSJE7FixQrU19f32tZqtcJsNrttNHAkEgme+dkU+Mkk2HGiBjtOcJozERENLo8CS11dHex2O7Rardt+rVYLg8HQ4zl79uzB66+/jg0bNvR63QULFmDLli3Q6/V48cUXsXv3bixcuBB2u73H9jk5OdBoNK5Np9N58jGoD8ZGBOGea8YAAJ7793FYO3v+syEiIhoIAzpLqLm5GXfeeSc2bNiA8PDwXtvdeuutWLRoEaZNm4YlS5bg008/xXfffYddu3b12D47Oxsmk8m1nT3LKbeD4cGfjENEsBLl9a14fU+Z2OUQEdEw4lFgCQ8Ph0wmg9Ho3iVgNBoRFRV1QfvTp0+jvLwct9xyC+RyOeRyObZs2YJPPvkEcrkcp0+f7vH7xMfHIzw8HCUlPc9KUSqVUKvVbhsNvGCVH7IXTgIAvLKjBAZTu8gVERHRcOFRYFEoFEhOToZer3ftczgc0Ov1SE1NvaD9pEmTUFhYiEOHDrm2RYsW4cYbb8ShQ4d67cqprKxEfX09oqOjPfw4NNCWJMVixqgQtNrs+K9tRWKXQ0REw4THXUJZWVnYsGEDNm/ejKKiIqxYsQIWiwWZmZkAgGXLliE7OxsAoFKpkJCQ4LaFhIQgODgYCQkJUCgUaGlpwRNPPIF9+/ahvLwcer0eixcvxrhx45Cent6/n5aumFQqwbOLEiCVAP8+XIU9p+rELomIiIYBjwNLRkYG/vu//xurVq1CUlISDh06hNzcXNdA3IqKClRXV1/29WQyGY4cOYJFixZhwoQJuPfee5GcnIxvvvkGSqXS0/JoEEwbqcGdV48GAKz6+CgH4BIR0YCTCIIgiF3ElTKbzdBoNDCZTBzPMkhMbR2Y95fdqGux4rH5E/DQvPFil0REREOMJz+/+S4h6hONvx+e+dlkAMArO0tQUd8qckVEROTLGFiozxYlxmDO2BGwdjqw+pOj8IGHdURE5KUYWKjPJBIJnlucAD+ZBDuLa/HFsZ4XDyQiIrpSDCx0RcZFBuE3148FADz77+OwWDtFroiIiHwRAwtdsZU3jsPIUH9Um9rxV/0pscshIiIfxMBCV8xfIcNzi6cCAF7fU4YTBr6MkoiI+hcDC/WLn0zSIn2qFnaHgN9/eBQOBwfgEhFR/2FgoX6z6pap8PeT4fszjXj/QKXY5RARkQ9hYKF+Exvij0fSnAvIPf/pcVQ1tYlcERER+QoGFupX91w7BokjNTC3d+KRrYdgZ9cQERH1AwYW6ld+Milevu0qBCpk2F/WgHU7S8QuiYiIfAADC/W70SMCsWZJAgDgr/pTKDjTIHJFREQ01DGw0ID4+VWxWJwUA7tDwMPvHIKprUPskoiIaAhjYKEBIZFI8PySBOjC/FHZ2Ian/u8I3zVERER9xsBCAyZY5Ye/3TYDfjIJPj9qwBvflotdEhERDVEMLDSgknQhePqnkwEAL2wrwoGKRpErIiKioYiBhQbcXXPicPO0aHQ6BDz4rwNotNjELomIiIYYBhYacBKJBH/8xTSMCQ9Elakdj757iEv3ExGRRxhYaFAEq/zw96UzoJRLsau4Fq/uPi12SURENIQwsNCgmRytdq3P8pcvi7GruEbkioiIaKhgYKFB9euZOtw2WweHADz09kGU1VnELomIiIYABhYadH9YNBXJo0PR3N6J5Vu+R3M7F5UjIqKLY2ChQaeUy/DqHTMQpVahpKYFj249zEG4RER0UQwsJIrIYBVeuzMZCrkUXxUZ8dJXJ8UuiYiIvBgDC4kmUReCnJ9PAwC8vKMEuUerRa6IiIi8FQMLieoXySNx77VjAABZ7x5GUbVZ5IqIiMgbMbCQ6LIXTsK148LRarPjnk3fwWhuF7skIiLyMgwsJDq5TIp1t8/A2IhAVJvacc+m72CxdopdFhEReREGFvIKmgA/vHH3bIwIVOBYlRn/8fZB2DlziIiIujCwkNcYNSIAG+6aCaVcCv2JGqz59LjYJRERkZdgYCGvMmNUKP4nIwkAsGlvOTbuKRO3ICIi8goMLOR1fjotGtkLJwEA1nx2HJ8eqRK5IiIiElufAsu6desQFxcHlUqFlJQU7N+//7LOe+eddyCRSLBkyRK3/YIgYNWqVYiOjoa/vz/S0tJw6tSpvpRGPuL+6+OxLHU0BAF4dOsh7DlVJ3ZJREQkIo8Dy9atW5GVlYXVq1fjwIEDSExMRHp6OmpqLv7m3fLycjz++OO47rrrLjj2pz/9CS+//DLWr1+P/Px8BAYGIj09He3tnN46XEkkEqy+ZSpunhaNDruA37z5PQorTWKXRUREIvE4sKxduxbLly9HZmYmpkyZgvXr1yMgIAAbN27s9Ry73Y6lS5fi2WefRXx8vNsxQRDw0ksv4fe//z0WL16M6dOnY8uWLaiqqsJHH33k8Qci3yGTSrA2IxFzxo6AxWbH3W/s59udiYiGKY8Ci81mQ0FBAdLS0s5fQCpFWloa8vLyej3vueeeQ2RkJO69994LjpWVlcFgMLhdU6PRICUlpddrWq1WmM1mt418k1Iuw2t3JmNqjBr1FhuWbcxHDReWIyIadjwKLHV1dbDb7dBqtW77tVotDAZDj+fs2bMHr7/+OjZs2NDj8e7zPLlmTk4ONBqNa9PpdJ58DBpiglV+2JQ5G6NHBOBsQxvueuM7mNs7xC6LiIgG0YDOEmpubsadd96JDRs2IDw8vN+um52dDZPJ5NrOnj3bb9cm7xQRrMSWe2YjPEiJomoz7t/yPdo77GKXRUREg0TuSePw8HDIZDIYjUa3/UajEVFRURe0P336NMrLy3HLLbe49jkcDuc3lstRXFzsOs9oNCI6OtrtmklJST3WoVQqoVQqPSmdfMDoEYHYlDkLt/5jH/aVNuCRdw5h3dIZkEklYpdGREQDzKMnLAqFAsnJydDr9a59DocDer0eqampF7SfNGkSCgsLcejQIde2aNEi3HjjjTh06BB0Oh3GjBmDqKgot2uazWbk5+f3eE0a3hJiNfjHsmQoZFLkHjPgmY+PQhC4hD8Rka/z6AkLAGRlZeGuu+7CzJkzMXv2bLz00kuwWCzIzMwEACxbtgyxsbHIycmBSqVCQkKC2/khISEA4Lb/kUcewfPPP4/x48djzJgxeOaZZxATE3PBei1EADBnbDheujUJK//3AP43vwIRQUo8On+C2GUREdEA8jiwZGRkoLa2FqtWrYLBYEBSUhJyc3Ndg2YrKioglXo2NObJJ5+ExWLB/fffj6amJlx77bXIzc2FSqXytDwaJn46LRprFifg9x8dxV/1pxAerMSdV48WuywiIhogEsEHnqebzWZoNBqYTCao1Wqxy6FB9D/bT+Kv+lOQSIB1t8/AT6dFX/okIiLyCp78/Oa7hGhIeyRtPJamjIIgAI+8cwh7T3MJfyIiX8TAQkOaRCLBc4sTsDAhCja7A/dvKcDRc1zCn4jI1zCw0JAnk0rwPxlJuDo+DC3WTizbuB8lNc1il0VERP2IgYV8gspPhg3LZmJarAYNFhuW/jMfFfWtYpdFRET9hIGFfEawyg9b7pmNCdogGM1WLH19HwwmvneIiMgXMLCQTwkNVOCte1MQ1/XeoaX/3Ie6FqvYZRER0RViYCGfE6lW4a37UhCjUeF0rQXLXt8PUxtflkhENJQxsJBPGhkagLfuS0F4kALHq824+439sFg7xS6LiIj6iIGFfFZ8RBDevDcFGn8/HKxownK+4ZmIaMhiYCGfNjlajc33zEagQoa9p+ux8l8H0GF3iF0WERF5iIGFfF6SLgSv3z0LSrkU+hM1yP6gkG94JiIaYhhYaFi4On4E1t+ZDKkEeL+gEu99Xyl2SURE5AEGFho2bpwYicdumggAeObjo1zCn4hoCGFgoWFlxQ1jcePECFg7Hbj7je9QXmcRuyQiIroMDCw0rEilErx061WYHK1GXYsVS/+Zz9VwiYiGAAYWGnY0/s4l/MeEB+JcUxvueD0ftc1cDZeIyJsxsNCwFBGsxJv3zka0RoWSmhb8+rU8VDW1iV0WERH1goGFhq2RoQF4e/nViA3xR1mdBb9an8cxLUREXoqBhYa1uPBAvPfbVMR3dQ/9cv1eHKxoFLssIiL6EQYWGvZiQvyx9TepmBKtRl2LDbf+Yx8+O1ItdllERPQDDCxEcI5pee+3qUibHAlrpwMr//cA1u0s4Yq4RERegoGFqEugUo7X7pyJe64ZAwD48xfFePy9I7B18t1DRERiY2Ah+gGZVIJVt0zBmiUJkEkl+L8Dlbjz9Xw0Wmxil0ZENKwxsBD14M6rR+P1u2YiSClHflkD/t+re1FS0yJ2WUREwxYDC1Ev5k6MxP+tmOOa9vzzdd9ixwmj2GUREQ1LDCxEFzExKhgfP3gNZseFodnaiXs3f8/BuEREImBgIbqE8CAl3rovBUtTRkEQnINxH3r7IFptnWKXRkQ0bDCwEF0GhVyK//r5NPzXzxMgl0rw6ZFq/PLVPFQ2topdGhHRsMDAQuSBpSmj8fb9VyM8SIHj1WYseuVb7CutF7ssIiKfx8BC5KFZcWH45MFrkRCrRoPFhjv+mY8NX5fC4eC4FiKigcLAQtQHMSH+eO83c7A4KQadDgH/ta0I927+Dg1cr4WIaED0KbCsW7cOcXFxUKlUSElJwf79+3tt+8EHH2DmzJkICQlBYGAgkpKS8Oabb7q1ufvuuyGRSNy2BQsW9KU0okHjr5DhpYwkPL8kAQq5FDuLa7Hwr1+zi4iIaAB4HFi2bt2KrKwsrF69GgcOHEBiYiLS09NRU1PTY/uwsDA8/fTTyMvLw5EjR5CZmYnMzEx88cUXbu0WLFiA6upq1/b222/37RMRDSKJRII7rh6Nj1deg7ERgTCarbh9wz6s3X6SS/oTEfUjieDhghIpKSmYNWsWXnnlFQCAw+GATqfDQw89hKeeeuqyrjFjxgzcfPPNWLNmDQDnE5ampiZ89NFHnlXfxWw2Q6PRwGQyQa1W9+kaRFeq1daJVR8fw/sFlQCAcZFByPl/0zArLkzkyoiIvJMnP789esJis9lQUFCAtLS08xeQSpGWloa8vLxLni8IAvR6PYqLi3H99de7Hdu1axciIyMxceJErFixAvX1vT9Wt1qtMJvNbhuR2AIUcvz3rxLxyu1XITxIgZKaFvxqfR6yPyiEqbVD7PKIiIY0jwJLXV0d7HY7tFqt236tVguDwdDreSaTCUFBQVAoFLj55pvxt7/9DfPnz3cdX7BgAbZs2QK9Xo8XX3wRu3fvxsKFC2G323u8Xk5ODjQajWvT6XSefAyiAfWz6TH4KusG3DrL+ffy7f0VmLd2N977/ixnEhER9ZFHXUJVVVWIjY3F3r17kZqa6tr/5JNPYvfu3cjPz+/xPIfDgdLSUrS0tECv12PNmjX46KOPMHfu3B7bl5aWYuzYsfjqq68wb968C45brVZYrVbX781mM3Q6HbuEyOvkl9Yj+8NClNZaAABTotX4/c2TMWdcuMiVERGJb8C6hMLDwyGTyWA0ur8Azmg0IioqqvdvIpVi3LhxSEpKwmOPPYZf/vKXyMnJ6bV9fHw8wsPDUVJS0uNxpVIJtVrtthF5o5T4Efj84euQvXASglVyHK824/Z/5iPzjf04XsWuTCKiy+VRYFEoFEhOToZer3ftczgc0Ov1bk9cLsXhcLg9IfmxyspK1NfXIzo62pPyiLySUi7Db24Yi91P3Ii758RBLpVgZ3EtfvryN/iPtw+irM4idolERF7P42nNWVlZ2LBhAzZv3oyioiKsWLECFosFmZmZAIBly5YhOzvb1T4nJwfbt29HaWkpioqK8Je//AVvvvkm7rjjDgBAS0sLnnjiCezbtw/l5eXQ6/VYvHgxxo0bh/T09H76mETiCwtU4A+LpuLLR6/Hz6Y7w/gnh6uQtnY3fv9RIepbeg/xRETDndzTEzIyMlBbW4tVq1bBYDAgKSkJubm5roG4FRUVkErP5yCLxYIHHngAlZWV8Pf3x6RJk/DWW28hIyMDACCTyXDkyBFs3rwZTU1NiImJwU033YQ1a9ZAqVT208ck8h7xEUF45fYZWDHXhP/+ohg7i2vx1r4KfHyoCvddG4+75oxGSIBC7DKJiLyKx+uweCOuw0JD2b7Sejz37+M4Xu0c0xKokOH2lFG477p4aNUqkasjIho4nvz8ZmAh8gJ2h4DPCqvx950lOGFoBgAoZFL8IjkWv7l+LOLCA0WukIio/zGwEA1RgiBgV3Et/r6rBN+VNwIApBLg5ukxWHHDWEyJ4d9vIvIdDCxEPuC78gb8fWcJdhbXuvbdODECD9w4jsv9E5FPYGAh8iHHq8x4dfdpfHakCt0L5c6KC8UDc8dh7sQISCQScQskIuojBhYiH1ReZ8FrX5fi/woqYbM73wQ9OVqN394Qj4UJ0VDIPV6lgIhIVAwsRD7MaG7H63vK8K99Z2CxOd+3FR6kQMYsHW6bPQojQwNErpCI6PIwsBANA02tNmzJO4N/5Z+B0excdE4qAW6cGIk7rh6N6ydEQCZldxEReS8GFqJhpMPugL7IiDf3ncG3JfWu/bowf9w+ezR+PXMkRgRxEUYi8j4MLETDVGltC/6VX4H3vj8Lc3snAOd6LgsSovDrmTrMGTsCUj51ISIvwcBCNMy12ez495Eq/GvfGRyuNLn2x2hU+EXySPxixkguRkdEomNgISKXwkoT3v3+LD4+dM711AVwTo3+ZfJI/HRaNIJVfiJWSETDFQMLEV2gvcOOr4qMeL+gEl+frHWt6aLykyJtshaLEmNww8QIKOUycQslomGDgYWILspgaseHB8/hvYKzKK21uParVXIsTIjGoqQYXB0/grOMiGhAMbAQ0WURBAFHKk345HAVPj1S5ZoeDQARwUr8bHo0bp4WjRmjQjlYl4j6HQMLEXnM7hCwv6wBnxw+h22FBpjaOlzHIoKVSJ+qxcKEaMweEwY/GVfVJaIrx8BCRFfE1unA1ydr8emRKuiLatBsPT9YNyTAD2mTtViYEIVrxoVD5ccxL0TUNwwsRNRvbJ0OfHu6Dl8cNeDL40Y0WGyuY0FKOW6cFIkFU6Mwd2IEApVyESsloqGGgYWIBkSn3YHvyhvxxTEDco8aYDC3u44p5VJcNz4C8yZHYu7ECERr/EWslIiGAgYWIhpwDoeAw5VNyO0KL2fqW92OT4oKxg0TI3DjxEgkjw7luBciugADCxENKkEQcMLQjO3HjdhVXINDZ5tc67wAQLBSjmvHh2PuxAjMnRgJrVolXrFE5DUYWIhIVI0WG74+VYvdxbXYfbIW9T8Y9wIAk6PVmNv19GXGqBDI+fSFaFhiYCEir+FwCCg8Z8LO4hrsKq7F4com/PBfnWCVHNePj8ANEyNw3fhwjn0hGkYYWIjIa9W3WPHNqTrsLK7B1ydr0dja4XY8bkQAro4fgdSxI5AaPwKR7D4i8lkMLEQ0JNi7Bu7uKq7F7uIaFJ4zuY19AYD4iECkxo/A1V1bRLBSnGKJqN8xsBDRkGRu78B3ZQ3YV1qPvNJ6HKsy48f/Qo2PDELq2PMBJixQIU6xRHTFGFiIyCeYWjuQX+YML3mn63HC0HxBm0lRwa7wcnV8GEICGGCIhgoGFiLySY0WmzPAnHaGmJPGFrfjEgkwOUrtegIzKy6UAYbIizGwENGwUNdiRX5pA/JK65B3uh6nay0XtJmgDcLMuDDMHB2KWXFhGBnqD4mEb54m8gYMLEQ0LNWY27GvrAF5p+uRX1qP0roLA4xWrcTM0WFIHh2KGaNDMSVaDYWc68AQiYGBhYgIzicwBWca8X15A74rb8TRcyZ0/mgaklIuxbRYDWaMDsWMUSGYMSqUU6mJBgkDCxFRD9psdhyubML35Q04UNGEAxWNaPrROjAAEBvij+kjNZg+MgSJIzVIGKmBWuUnQsVEvo2BhYjoMgiCgNI6Cw6cacSBiiYcrGhEsbH5gqnUgHM9mMSRIZgWq0GiToOpMRqo/GSDXzSRDxnwwLJu3Tr8+c9/hsFgQGJiIv72t79h9uzZPbb94IMP8MILL6CkpAQdHR0YP348HnvsMdx5552uNoIgYPXq1diwYQOamppwzTXX4NVXX8X48eMvqx4GFiLqL83tHSg8Z8KRShOOVDbh8FkTzjW1XdBOJpVggjYYiV1PYqaP1GBiVDDfSk3kgQENLFu3bsWyZcuwfv16pKSk4KWXXsJ7772H4uJiREZGXtB+165daGxsxKRJk6BQKPDpp5/isccew2effYb09HQAwIsvvoicnBxs3rwZY8aMwTPPPIPCwkIcP34cKtWl+5IZWIhoINW3WHHknAlHznaFmEoT6lqsF7RTyKWYEq12hZhEnQbx4UGQSjkriagnAxpYUlJSMGvWLLzyyisAAIfDAZ1Oh4ceeghPPfXUZV1jxowZuPnmm7FmzRoIgoCYmBg89thjePzxxwEAJpMJWq0WmzZtwq233nrJ6zGwENFgEgQB1aZ2HKls6noS4wwy5vbOC9oGKeWYGqNGos75FCZxZAinVhN18eTnt9yTC9tsNhQUFCA7O9u1TyqVIi0tDXl5eZc8XxAE7NixA8XFxXjxxRcBAGVlZTAYDEhLS3O102g0SElJQV5eXo+BxWq1wmo9/383ZrPZk49BRHRFJBIJYkL8ERPijwUJ0QCc/76V17e6upEKzzXh6DkzWqydyC9rQH5Zg+t8jb8fpkSrMTVGjamxakyN0SA+PBBydicR9cqjwFJXVwe73Q6tVuu2X6vV4sSJE72eZzKZEBsbC6vVCplMhr///e+YP38+AMBgMLiu8eNrdh/7sZycHDz77LOelE5ENKAkEgnGhAdiTHggFifFAgA67Q6U1LbgyFkTDlc2ofCcCUXVZpjaOpyvGyitd52vlEsxqTvExKgxJVqNydFqDuwl6uJRYOmr4OBgHDp0CC0tLdDr9cjKykJ8fDzmzp3bp+tlZ2cjKyvL9Xuz2QydTtdP1RIR9Q+5TIpJUWpMilLj17Oc/0bZOh04aWzG8SozjlWZcKzKjKJqMyw2Ow6fbcLhs02u86USYGxEUFeI0bi+agI4xZqGH48CS3h4OGQyGYxGo9t+o9GIqKioXs+TSqUYN24cACApKQlFRUXIycnB3LlzXecZjUZER0e7XTMpKanH6ymVSiiVfMU8EQ09CrkUCbEaJMRqADhDjMMhoLzegmNV5q7NhONVZtRbbDhV04JTNS346FCV6xqxIf7uISZWjSi1iuNiyKd5FFgUCgWSk5Oh1+uxZMkSAM5Bt3q9Hg8++OBlX8fhcLjGoIwZMwZRUVHQ6/WugGI2m5Gfn48VK1Z4Uh4R0ZAklUoQHxGE+Igg3JIYA8A5JsZotrqewnR/rWxsw7km5/bl8fP/8xgWqHB2Jf0gyMSNCISMM5TIR3jcJZSVlYW77roLM2fOxOzZs/HSSy/BYrEgMzMTALBs2TLExsYiJycHgHO8ycyZMzF27FhYrVZs27YNb775Jl599VUAzn7fRx55BM8//zzGjx/vmtYcExPjCkVERMONRCJBlEaFKI0K8yafH+Nnau3AsWpTV5eSM8icrrWgwWLDN6fq8M2pOldbhVyK+PBAjI0MwriIIIyNDML4yCDERwRCKefYGBpaPA4sGRkZqK2txapVq2AwGJCUlITc3FzXoNmKigpIpedHulssFjzwwAOorKyEv78/Jk2ahLfeegsZGRmuNk8++SQsFgvuv/9+NDU14dprr0Vubu5lrcFCRDScaAL8MGdsOOaMDXfta++w44Sh+QdPY8w4UW2GtdOBE4ZmnDA0u11DJnUOEJ6oDcYEbTDiwgOgCwvAqLAAjAhUsGuJvBKX5ici8kF2h4BzjW0oqW1GSU0LTtdYUFLbgpPGZjT3sF5MtwCFDLrQAOjC/KELC4Au1BlkdGHOfQGKQZmrQcME3yVEREQ96h4bU2xsxklDM04am3GmoRWVDa2oNrf3+B6lHwoPUvwoyJwPNtEaFdeSIY8wsBARkcesnXZUNbWjoqEVZ7u2ioZWnG1sRUV9a48r+f6QXOpcUO+HQWZkaAAig5WIDFZCq1YhUMknNHTegK10S0REvkspl7kWv+uJqbUDZxt/FGQa2lDZ0IrKxjbY7A5UdB3rTaBCBq1ahYhgJSLVKkQGKxEepMSIIAUiur6OCFJiRKCCi+aRGwYWIiK6LJoAP2gCuteQcedwCDA2t6OivhVnG9tQ0dXNVNnUhtpmK2rM7bDY7LDY7Cits6C0znLJ7xeslGNEkMIVaEYEOcNNeJACIwK7vnb9XuPvx8HCPo6BhYiIrphUKkG0xh/RGn+k9NKmxdqJGnM7apqtMJrbnUGm2Yq6FivqW2yot1hR1+z82mEX0GztRLO1E+X1vT+x6SaTSqBWyaH294Na5Qe1v9z5tevXwSq/Hx3/QRt/PwQqZAw8Xo6BhYiIBkWQUo6grgXyLkYQBJjbO88HmRZnqKn7Uaipb7GhtsWK5vZO2B0CGls70Nja0afapBI4Q82Pgs4PA41KIUOAnwz+ChlUfjL4+8kQqJQjQOH8GqySIzSAXVkDhYGFiIi8ikQigcbfDxp/P4yNuHR7a6cdjZYOmNs7YG5zfm1u7+z6dadrn7mt8wdtzu/vsAtwCICprQOmtg4AbVdUv8pPikCFHP4KmetrQNfmr5C7Qk+gUoYAhRz+ft3HnL8//2v381VyGaTDeOViBhYiIhrSlHIZojQyRGk8X2xUEARYOx2u8GLqJdS0Wu1o63Bu7Tbn11abHW02Oyy2TlisnWhu70SnQ0B7hwPtHTbg0sN0PCKRAIEKOQKVMoQFKjE5KhjXT4hAlEYFqUQCqcTZNaeSyxAb4g+1v9ynurk4rZmIiKgfCIKAFmsnmlo70NoVZNpszmDT+oNfO8NOJyxWZ+Bp7bCjzdbZ1e58CPph+76QSpxPq6QSICbEH1q1CqEBfhgTHoSr48MQFqhAoFKOIKXc2bXlN/hPcLgOCxERkY9wOAS0d9rRYnWGnJb2TtQ0t+O78kbkldajpb0DggDYBQEOQYDFakeDxebx95FIgICucTlqfz+MCFRAAKCQSRGlUSFao8KDPxnXr++h4josREREPkIqlXSNbZEDwd17NW4vxfyxVpuzi0oQgE6Hc32cBosNDRYbDp1twpFKE1ranV1ZFlsnHAIgCHBNPa9ptqLkR9dUyKXImj9hoD7mJTGwEBER+RhXwOkyMjTA9etlqe5tBcE57sb5BKcTLdZOmNo60GCxQSaVoNVmh9HcjjabXdQxMQwsREREw5hEIoF/18ykiGCl2OX0im+pIiIiIq/HwEJERERej4GFiIiIvB4DCxEREXk9BhYiIiLyegwsRERE5PUYWIiIiMjrMbAQERGR12NgISIiIq/HwEJERERej4GFiIiIvB4DCxEREXk9BhYiIiLyej7xtmZBEAAAZrNZ5EqIiIjocnX/3O7+OX4xPhFYmpubAQA6nU7kSoiIiMhTzc3N0Gg0F20jES4n1ng5h8OBqqoqBAcHQyKR9Ou1zWYzdDodzp49C7Va3a/XpvN4nwcP7/Xg4H0eHLzPg2cg7rUgCGhubkZMTAyk0ouPUvGJJyxSqRQjR44c0O+hVqv5H8Mg4H0ePLzXg4P3eXDwPg+e/r7Xl3qy0o2DbomIiMjrMbAQERGR12NguQSlUonVq1dDqVSKXYpP430ePLzXg4P3eXDwPg8ese+1Twy6JSIiIt/GJyxERETk9RhYiIiIyOsxsBAREZHXY2AhIiIir8fAcgnr1q1DXFwcVCoVUlJSsH//frFLGlK+/vpr3HLLLYiJiYFEIsFHH33kdlwQBKxatQrR0dHw9/dHWloaTp065damoaEBS5cuhVqtRkhICO699160tLQM4qfwfjk5OZg1axaCg4MRGRmJJUuWoLi42K1Ne3s7Vq5ciREjRiAoKAi/+MUvYDQa3dpUVFTg5ptvRkBAACIjI/HEE0+gs7NzMD+KV3v11Vcxffp018JZqamp+Pzzz13HeY8Hxh//+EdIJBI88sgjrn281/3jD3/4AyQSids2adIk13Gvus8C9eqdd94RFAqFsHHjRuHYsWPC8uXLhZCQEMFoNIpd2pCxbds24emnnxY++OADAYDw4Ycfuh3/4x//KGg0GuGjjz4SDh8+LCxatEgYM2aM0NbW5mqzYMECITExUdi3b5/wzTffCOPGjRNuu+22Qf4k3i09PV144403hKNHjwqHDh0SfvrTnwqjRo0SWlpaXG1++9vfCjqdTtDr9cL3338vXH311cKcOXNcxzs7O4WEhAQhLS1NOHjwoLBt2zYhPDxcyM7OFuMjeaVPPvlE+Oyzz4STJ08KxcXFwn/+538Kfn5+wtGjRwVB4D0eCPv37xfi4uKE6dOnCw8//LBrP+91/1i9erUwdepUobq62rXV1ta6jnvTfWZguYjZs2cLK1eudP3ebrcLMTExQk5OjohVDV0/DiwOh0OIiooS/vznP7v2NTU1CUqlUnj77bcFQRCE48ePCwCE7777ztXm888/FyQSiXDu3LlBq32oqampEQAIu3fvFgTBeV/9/PyE9957z9WmqKhIACDk5eUJguAMl1KpVDAYDK42r776qqBWqwWr1Tq4H2AICQ0NFf75z3/yHg+A5uZmYfz48cL27duFG264wRVYeK/7z+rVq4XExMQej3nbfWaXUC9sNhsKCgqQlpbm2ieVSpGWloa8vDwRK/MdZWVlMBgMbvdYo9EgJSXFdY/z8vIQEhKCmTNnutqkpaVBKpUiPz9/0GseKkwmEwAgLCwMAFBQUICOjg63ez1p0iSMGjXK7V5PmzYNWq3W1SY9PR1msxnHjh0bxOqHBrvdjnfeeQcWiwWpqam8xwNg5cqVuPnmm93uKcC/z/3t1KlTiImJQXx8PJYuXYqKigoA3neffeLlhwOhrq4Odrvd7Q8BALRaLU6cOCFSVb7FYDAAQI/3uPuYwWBAZGSk23G5XI6wsDBXG3LncDjwyCOP4JprrkFCQgIA531UKBQICQlxa/vje93Tn0X3MXIqLCxEamoq2tvbERQUhA8//BBTpkzBoUOHeI/70TvvvIMDBw7gu+++u+AY/z73n5SUFGzatAkTJ05EdXU1nn32WVx33XU4evSo191nBhYiH7Ny5UocPXoUe/bsEbsUnzRx4kQcOnQIJpMJ77//Pu666y7s3r1b7LJ8ytmzZ/Hwww9j+/btUKlUYpfj0xYuXOj69fTp05GSkoLRo0fj3Xffhb+/v4iVXYhdQr0IDw+HTCa7YDS00WhEVFSUSFX5lu77eLF7HBUVhZqaGrfjnZ2daGho4J9DDx588EF8+umn2LlzJ0aOHOnaHxUVBZvNhqamJrf2P77XPf1ZdB8jJ4VCgXHjxiE5ORk5OTlITEzEX//6V97jflRQUICamhrMmDEDcrkccrkcu3fvxssvvwy5XA6tVst7PUBCQkIwYcIElJSUeN3faQaWXigUCiQnJ0Ov17v2ORwO6PV6pKamiliZ7xgzZgyioqLc7rHZbEZ+fr7rHqempqKpqQkFBQWuNjt27IDD4UBKSsqg1+ytBEHAgw8+iA8//BA7duzAmDFj3I4nJyfDz8/P7V4XFxejoqLC7V4XFha6BcTt27dDrVZjypQpg/NBhiCHwwGr1cp73I/mzZuHwsJCHDp0yLXNnDkTS5cudf2a93pgtLS04PTp04iOjva+v9P9OoTXx7zzzjuCUqkUNm3aJBw/fly4//77hZCQELfR0HRxzc3NwsGDB4WDBw8KAIS1a9cKBw8eFM6cOSMIgnNac0hIiPDxxx8LR44cERYvXtzjtOarrrpKyM/PF/bs2SOMHz+e05p/ZMWKFYJGoxF27drlNj2xtbXV1ea3v/2tMGrUKGHHjh3C999/L6Smpgqpqamu493TE2+66Sbh0KFDQm5urhAREcFpoD/w1FNPCbt37xbKysqEI0eOCE899ZQgkUiEL7/8UhAE3uOB9MNZQoLAe91fHnvsMWHXrl1CWVmZ8O233wppaWlCeHi4UFNTIwiCd91nBpZL+Nvf/iaMGjVKUCgUwuzZs4V9+/aJXdKQsnPnTgHABdtdd90lCIJzavMzzzwjaLVaQalUCvPmzROKi4vdrlFfXy/cdtttQlBQkKBWq4XMzEyhublZhE/jvXq6xwCEN954w9Wmra1NeOCBB4TQ0FAhICBA+PnPfy5UV1e7Xae8vFxYuHCh4O/vL4SHhwuPPfaY0NHRMcifxnvdc889wujRowWFQiFEREQI8+bNc4UVQeA9Hkg/Diy81/0jIyNDiI6OFhQKhRAbGytkZGQIJSUlruPedJ8lgiAI/fvMhoiIiKh/cQwLEREReT0GFiIiIvJ6DCxERETk9RhYiIiIyOsxsBAREZHXY2AhIiIir8fAQkRERF6PgYWIiIi8HgMLEREReT0GFiIiIvJ6DCxERETk9RhYiIiIyOv9f0M/f0plXRGtAAAAAElFTkSuQmCC" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(losses)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:30:59.243854Z", + "start_time": "2023-12-13T17:30:59.170246Z" + } + }, + "id": "78ac13e102efe3a0" + }, + { + "cell_type": "code", + "execution_count": 25, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 10/10 [00:01<00:00, 5.13it/s]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean_reward: -170.4\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "# Estimate agent performance\n", + "\n", + "episodes = []\n", + "for i in trange(10):\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id, env_name=\"mcar\")\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + "mean_reward = np.mean([sum(e.rewards) for e in episodes])\n", + "print(f\"mean_reward: {mean_reward}\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:31:03.547789Z", + "start_time": "2023-12-13T17:31:01.595787Z" + } + }, + "id": "d221ebfe535f9317" + }, + { + "cell_type": "code", + "execution_count": 27, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "mean_reward: -1.81e+02: 100%|██████████| 30/30 [01:12<00:00, 2.40s/it]\n" + ] + }, + { + "data": { + "text/plain": "[]" + }, + "execution_count": 27, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGdCAYAAADqsoKGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABwHUlEQVR4nO3de3zcdZ0v/td37rlP0ubaXNq0NK30SlegVaCFSguosLgsyO5vrVYQFo+CuAp6FtA9nCogHFZdkRWoexYV8KgooFJLW7kUlNK0tDS9t0lzb9okk8vcv78/Zj7fmaS5zGS+3/le8no+HnnQJpPpt2E68573532RZFmWQURERGRyNr0vgIiIiEgNDGqIiIjIEhjUEBERkSUwqCEiIiJLYFBDRERElsCghoiIiCyBQQ0RERFZAoMaIiIisgSH3heQDdFoFG1tbSgoKIAkSXpfDhEREaVAlmX4fD5UVVXBZps8DzMtgpq2tjbU1NTofRlEREQ0BS0tLaiurp70dtMiqCkoKAAQ+6EUFhbqfDVERESUiv7+ftTU1Civ45OZFkGNOHIqLCxkUENERGQyqZaOsFCYiIiILIFBDREREVkCgxoiIiKyBAY1REREZAkMaoiIiMgSGNQQERGRJTCoISIiIktgUENERESWwKCGiIiILEHToObBBx/EqlWrkJubC6/Xe87XN2/eDEmSxvzo6upSbrd9+3ZccMEFcLvdmDdvHjZv3qzlZRMREZEJaRrUBINB3HDDDbj99tvH/PqNN96I9vb2ER/r1q3DZZddhrKyMgDA8ePHcc0112DNmjVobGzEnXfeic9//vP44x//qOWlExERkclouvvpW9/6FgCMm1nJyclBTk6O8vvu7m689tpreOqpp5TPPfHEE5gzZw6+973vAQAWLlyIN954A4899hjWrVun3cUTERGRqRiqpua//uu/kJubi7/7u79TPrdz506sXbt2xO3WrVuHnTt3jns/gUAA/f39Iz6IMhWNynj6jeNobOnV+1KIiGgMhgpqnnrqKdx8880jsjcdHR0oLy8fcbvy8nL09/djeHh4zPvZtGkTioqKlI+amhpNr5umh90tZ/Htlz7AN371vt6XQkREY0g7qLnnnnvGLe4VH01NTWlfyM6dO3HgwAFs3Lgx7e8d7d5770VfX5/y0dLSkvF9EvUMBAEAx08PQpZlna+GiIhGS7um5u6778aGDRsmvE19fX3aF/KTn/wEy5Ytw4oVK0Z8vqKiAp2dnSM+19nZicLCwhEZnWRutxtutzvtayCayFAwAgAYDkVwdiiEkjyXzldERETJ0g5qSktLUVpaqupFDAwM4Pnnn8emTZvO+drKlSvxyiuvjPjcli1bsHLlSlWvgWgyg8Gw8utTZ4cY1BARGYymNTXNzc1obGxEc3MzIpEIGhsb0djYiIGBgRG3e+655xAOh/GP//iP59zHbbfdhmPHjuFrX/sampqa8B//8R94/vnncdddd2l56UTnGApElF+fOjt2PRcREelH05bu++67Dz/96U+V3y9fvhwAsG3bNqxevVr5/FNPPYXrr79+zAF9c+bMwcsvv4y77roLjz/+OKqrq/GTn/yE7dyUdcmZmlYGNUREhqNpULN58+aUpv++9dZbE3599erV2L17t0pXRTQ1oqYGiB0/ERGRsRiqpZvIyAYDSZmaXmZqiIiMhkENUYpGZmoY1BARGQ2DGqIUjcjUnB3mrBoiIoNhUEOUouRMjS8QRv9weIJbExFRtjGoIUpRcvcTAJzqZbEwEZGRMKghSlHynBqAdTVEREbDoIYoRSJTU1XkAcBZNURERsOghihFoqZmfkUBAGZqiIiMhkENUYqG4pmahnIR1LCmhojISBjUEKUgEpXhD0UBAOfFgxoO4CMiMhYGNUQpGErqfEpkahjUEBEZCYMaohSIehqHTcKc0jwAQN9wCD5/SM/LIiKLaOrox+qHt+HZd07qfSmmxqCGKAVimnCuy458twPeXCcAHkERkToeffUQTvQM4f/uZFCTCQY1RCkQmZpcV2yx/SxvDgC2dRNR5o6fHsSWA50AgIOdPvQzAzxlDGqIUqBkatx2AEB1cSyoYV0NEWXq6TeOQ6ySk2WgsblX1+sxMwY1RCkQmZo8JVOTC4DHT0SUmbODQbywqwUAMHtG7Hll18mzel6SqTGoIUqBmCac6xqdqeGsGiKaumffOQl/KIoPVRZi40fnAADea2ZQM1UOvS+AyAzE3qc8dzxTU8yaGiLKTCAcwU/jhcG3XlqP+fFxEY3NvYhEZdhtkp6XZ0rM1BClYPxMDYMaIpqaFxvb0O0LoLLIg2uWVKKhogB5Ljt8gTAOd/n0vjxTYlBDlILRNTXVxbGz757BIIaDkXG/j4hoLLIs46nXjwMANqyaDafdBrtNwrJaLwDW1UwVgxqiFIzufirKcaIgfhTV2su6GiJKz58Pn8bBTh/yXHbcdGGt8vkVtcUAGNRMFYMaohSMztQAibqaFh5BEVGafvL6MQDAjR+uRVGOU/n8BXWxoOY9BjVTwqCGKAWjMzVAoq6GxcJElI4P2vrx+uHTsEnAZz8ye8TXlsczNSd6hnB6IKDD1ZkbgxqiFIyVqRF1NSwWJqJ0/OSNWJbmqsWVqCnJHfG1ohwn5pfnA2C2ZioY1GTg5b3t+Mav38feU716XwppbHT3E5C0KoED+IgoRZ39fvxuTxsA4JZL6se8zYr4EdQuzqtJG4OaDLy0tw0/e6cZO4/26H0ppDElU+NOztRwAB8RpWfzWycQisj48OxiLKvxjnmbC2pZVzNVDGoysGhWEQDg/dY+na+EtDY0VqaGNTVElIbBQBjPvh0btvf5cbI0QCJTs+dUH4LhaFauzSoY1GRgcTyo2cegxvJGTxQGEjU1Xb4A/CHOqiGiib3wbgv6/WHMnpGLtQvLx73dnJl5KM51IhiOYn8bX1/SwaAmAyKoOdEzxFXxFjdWTU1xrhM5ztjv2/v8ulwXEZlDJCrj6TdPAAA2fnTOhCsQJElSsjXvcWN3WhjUZKA4z6XUVTBbY20iU5Ob1P0kSRLraogoJa/u70DzmSF4c534uxU1k96e82qmhkFNhkS25v1TDGqsSpZlJVOTl5SpATirhohS85/xYXv/eFEdckY9j4xFTBZ+9+QZyLKs6bVZCYOaDLFY2PoC4Sii8eeUXPfIxfazuNiSiCax6+RZvNfcC5fdhn9aVZfS9yyp9sJhk9DZH0Abj7dTxqAmQywWtj4xTRiAUkMjJAbw8fiJiMYmViJcu6wKZQWelL4nx2XHh6oKAXAPVDoY1GSIxcLWJ2bU5Djt5xT3cQAfEU2kuWcIf9zfAWDiNu6xcF5N+hjUZKg4z6W8sDFbY01KPY373HPwah4/EdEEnn7zOKIycOn8UjRUFKT1vcpkYQY1KWNQo4Il1TyCsrLBMTqfBFFT09nv55AsIhqhbyiE599tAQDccsmctL9fBDUftPcrA0BpYgxqVJAoFu7X+UpIC2NNExZK891wO2yIykAHi/mIKMmzfzmJoWAECyoK8NF5M9P+/ipvDiqLPIhEZexp4ZvmVDCoUQGLha1tcIxpwoIkScrx46leFgsTUUwwHMXm+LC9Wy6phySNP2xvIsq8Gi63TAmDGhWIoOb46UEWC1vQRJkagG3dRHSu3+5pQ5cvgPJCNz6xtGrK9yPm1bCuJjUMalTAYmFrGxQbuseoqQE4gI+IRpJlWWnj/syq2XA5pv5SuyIpU8MhfJNjUKMSHkFZ11B8Tk3uGN1PQPKsGgY1RAS8ceQ0mjp8yHXZ8Q8XpjZsbzwfqiqEx2lD71AIx04PqnSF1sWgRiWLq1ksbFUpZ2pYU0NEAP7z9eMAgL//mxoU5Tozui+n3YYl1V4APIJKBYMalTBTY13DwYkzNUqhMDM1RNPewQ4f/nyoGzYJ+NxH0m/jHssKLrdMGYMalbBY2Lomz9TEjp86+vwIRzirxkgOdviw9tEd+O2eNr0vhaYJUUuz7vwK1M7IVeU+WSycOgY1KkkuFt7PIyhLUWpqxul+Kitww2mXEI7K6PQFsnlpNIn/fP0YjnQN4IX4ADQiLXX5/HixMRZAp7sSYSLLa70AgMNdA+gb4pvmiTCoUdFiZQhfr74XQqpSMjVjzKkBAJtNQpU4gjrDuhqjCIajys6dZv5/oSz4r7dOIhiJ4oJar3JkpIYZ+W7MmZkHAHivhdmaiTCoURGLha1psjk1ABdbGtEbR7rh88f+37WeHUYkynZY0o4sy/jFX5sBxIbtqY3LLVPDoEZFi1gsbEkT7X4SuNjSeF7a2678OhyV0d7H/zeknWOnB3F6IAi3w4YrFparfv9cbpkaBjUqYrGwNYlMTd6EmZpYQSAH8BlDIBzBlv2dAACnPTaenkdQpCWRQVk8qyijYXvjEUFNY0svGxImwKBGRSUsFrYkJVMzTk0NkJSp4awaQ3j90Gn4AmGUF7pxcf0MAMCpMww4STvvNfcCSOxqUtt5ZfkocDswFIygqcOnyZ9hBQxqVMZ5NdaTUqaGqxIM5eX3Y0dPVy+uRG1JLIvGTA1paXd84eQF8U4ltdlsEpZzueWkNAtqHnzwQaxatQq5ubnwer1j3uavf/0rrrjiCni9XhQXF2PdunXYs2fPiNvs3bsXl1xyCTweD2pqavDQQw9pdcmqSBQLM6ixCtH9lEqmpq3XjygLUnXlD0Ww5YPY0dPHlzCoIe35/CEc7IxlT0RBrxY4r2ZymgU1wWAQN9xwA26//fYxvz4wMID169ejtrYW77zzDt544w0UFBRg3bp1CIVi9Sj9/f248sorUVdXh127duHhhx/GAw88gCeffFKry84Yi4WtJRSJIhiOnV9PlKmpKPTAbpMQjETRPcBZNXp6/fBpDATCqCzyYHlNsRLUtJxlUEPaaGzphSzH3tyUFXo0+3NWMFMzqfHfemboW9/6FgBg8+bNY369qakJZ86cwbe//W3U1NQAAO6//34sWbIEJ0+exLx58/Dss88iGAzi6aefhsvlwvnnn4/GxkY8+uijuPXWW7W69IyI46dj8WLhQk9mez9IX0PxLA0wcfeTw25DRaEHrb3DOHV2COUaPrHRxF7eGxt+dvXiSthsEmpEUMNMDWnkvZO9ALTN0gDA0poi2CSg5cwwuvr9mgZQZqVbTU1DQwNmzJiBp556CsFgEMPDw3jqqaewcOFCzJ49GwCwc+dOXHrppXC5XMr3rVu3DgcPHsTZs+NHqoFAAP39/SM+soXFwtYi6mmcdmnSjga2desv+ejpmiWVAKAENacHgsr/TyI1vadxPY1Q4HGioaJwxJ9JI+kW1BQUFGD79u347//+b+Tk5CA/Px9/+MMf8Pvf/x4OR+wdcUdHB8rLR/b7i993dHSMe9+bNm1CUVGR8iEyQdmyaFbsQccjKPNLZUaNMItBje52HOrGYDCCWd4cLK/xAgCKcpwo9MT+/7WwA4pUFo3KiSJhjTqfkq2o8wJgXc140gpq7rnnHkiSNOFHU1NTSvc1PDyMjRs34iMf+QjefvttvPnmm1i0aBGuueYaDA9n9sRz7733oq+vT/loacnu3hexJt6qxcI+fwhPv3EcHX1+vS9Fc6l0PglisSWDGv28vFd0PVVAkiTl82KxIIuFSW3HTg+g3x+Gx2nDwspCzf88DuGbWFo1NXfffTc2bNgw4W3q61MbD/2zn/0MJ06cwM6dO2Gz2ZTPFRcX48UXX8RNN92EiooKdHZ2jvg+8fuKiopx79vtdsPtdqd0HVqwcrGwLMu48xeN2NrUhb2nevF/blqu9yVpKpUZNUI1VyXoyh+K4E8HxNFT1Yiv1ZbkYl9rP+tqSHWinmZJtRdOu/aHH6JuZ19rP/yhCDzOyd9wTSdpBTWlpaUoLS1V5Q8eGhqCzWYb8W5K/D4ajXWbrFy5Et/85jcRCoXgdMYKbrds2YKGhgYUF2uf5puq5GJhnz+EAgsVCz/1xnFsbeoCALx5tAeyLI/4f2g16WVqxPETXzj1sP1gF4biR09L46MVhJpiZmpIGyJjonWRsFBbkouZ+S6cHghif1sfVtSVZOXPNQvNwsrm5mY0NjaiubkZkUgEjY2NaGxsxMDAAADgYx/7GM6ePYs77rgDBw4cwP79+/HZz34WDocDa9asAQDcfPPNcLlc2LhxI/bv34/nnnsOjz/+OL7yla9oddmqGFEs3GadYuE9Lb347h8Sx4vdvgCOnR7U8Yq0J7qf0qmpaT07DFnmrJpsE7uePr6k8pxAWxQLM+AktWWrSFiQJEkJoIxyBGWk2VyaBTX33Xcfli9fjvvvvx8DAwNYvnw5li9fjnfffRcAsGDBAvzud7/D3r17sXLlSlxyySVoa2vDH/7wB1RWxroWioqK8Oqrr+L48eNYsWIF7r77btx3332GbedOJoqF3z9ljSOofn8IX/z5ewhFZFy1qAIXzYm9O3jn2Bmdr0xbSqbGPXmmprIoB5IEBMJRnB4Ian1plGQ4GMHWA7EMouh6SsYBfKSFvuEQDnfF3qhno0hYMFpdzR0/ew8ffvBP+N2eNr0vRbs5NZs3bx53Ro3wsY99DB/72McmvM2SJUvw+uuvq3hl2bF4VhH+uL/TEsXCsizj3l+9j5Yzw6guzsF3PrUET79xHO8cP4O3j/Xg5otq9b5EzaTT/eRy2FBe4EFHvx+tvcMoLdCvrmu62XawC8OhCGpKcpTj32SJWTXDlj8ypexpbOkFII6EsvfvPRHU9Bri8dzaO4xuX8AQ9T3c/aQRKxUL/+wvzXh5bzscNgnf//RyFOU4lSWB7xzvsfRRi8jU5KZQUwOwrkYvouvpmsVVYz7Bz/LGsmjDoQizaKSa905m9+hJWDSrCE67hNMDAUOMKWiLN0dUefUfBsigRiOji4XNqqmjH9/+3QcAgK+tb8Dy+Fnu8lovXHYbOvsDONFj3RfwwTRqaoBEUMPFltkzFAxja1Ni19NYXA4bKuPTV3kERWoR9TQrsnj0BAAep11547yrWd8SAH/SG4WqohxdrwVgUKOZGflu0xcLDwXDuOPZ9xAIR7G6oRSf/2iiXd/jtGNZfLjZO8d6dLpC7Q0FUq+pATiATw+vNXXBH4qibkYuzq8af04I1yWQmqJRGY3NvQCgvNnLJqMst2yPzyvLcdrhzdW/05dBjYbMPln4vhf342j3IMoL3fjeDUths41M619cHysWftvCQU36mZrYCydn1WRP4ujp3K6nZLUMakhFh7sG4AuEkeuyY0FFQdb/fGW5ZXxOjl6Sj570ru0BGNRoShxBmbFY+Ne7T+GXu07BJgH/58blmDFGEdxFSl3NGcvW1aTT/QRAyc6xpiY7BgNhvNY0ftdTshp2QJGKxNHTkuoiOLIwdG800W3V1NGPgYB+O80SQY3+R08AgxpNLTJpUHOsewDf/PU+AMCXrjgPK+fOGPN2F9QWw2mX0N7nt+wLRTrdT8DImhqrBnpGsrWpC4FwFHNm5uFDk4yoVzI1DDhJBe9leejeaOWFHlQX5yAqx2aI6aWtN3b8NItBjfUpxcLd5ikW9oci+OLPdmMoGMHF9SX4H5efN+5tc1x2LI3vubLqvJp0JgoDiXcrg8EIeofM8f/czF7eG5uLMdnREzCyrZsoU3oVCSczwrwaZmqmkRn5blQVxTouzFIs/L9fOYAP2vtRkufC4zcth9028QuFaO22al1NOrufgFgBtZhPw2JhbQ0Ewth2sBvA5EdPAFBTEnvSbesbRjAc1fTayNp6h4I42h2bpq5HkbBghMnCbX2x57nKIv3buQEGNZoz07yaP+xrx3/tPAkA+N7fL0V54eQP0ovixcJWratJN1MDJNKwrb085tDS1gOdCIajqC/NS6lQszTfDY/TBllOvLskmord8a6nOTPzUJLn0u06lGLh5rO6rSoQTRE8fpomllSbo66m5cwQvvbLvQCAL1xajzUNZSl934q6YjhsElp7hy2ZmUi3+wlIHsBnvZ+HkSi7nlI4egJiO3O42JLUII6elmd56N5oCyoKkOO0w+cP40j3QNb/fFmW0R6vqeHx0zRhhmLhUCSKL/1iN/r9YSyr8eKr6xpS/t5cl0MJ3Kx4BJXunBqAs2qywecPYYdy9FSV8vdxBxSpIbHEUr+jJwBw2G3KvDA9jqB6h0IYDsXe+FXw+Gl6EMXCx08P6tp2N5FHXj2I3c29KPA48P1PL4czzfbERF2NtYqFo1EZQ6GpZGrERmgGNVr504FOBCNRzCvLx/zy/JS/r4YdUJShSNLQPb2DGkDfYmFx9DQz322IvU8AgxrNiWJhWQb2GzBbs/1gF3684xgA4KFPLVGe9NNxUdIeKCvxhyMQZULpZGqUtm7WbWgm1YF7o3GqMGXqUKcPg8EI8lx2NOgwdG+0xBC+7Ac1bUo9jTGyNACDmqww6hFUZ78fdz+/BwDw/11ch6sWT95BMpYVdcWw2yScOjtsqaFzovNJkgCPI42ghgP4NNU3HMKfD50GkFrXU7JatnVThkRGZFmtd9Lu0GwQdT3HTg/izGB2l7WKoKbSADufBAY1WWDEycKRqIw7f9GInsEgFlYW4pvXLJzyfeW7Hcrf0UrzaoZFkbDTfs6KiImImhqfP4y+Yc6qUdufPogdPc0vz8f88vTeKbOmhjJllHoawZvrwryy2BFstrM1bX3GKhIGGNRkxSIDdkD94LUj2HmsB7kuO35w8/KMz0MTrd3WOYIajLdzpzqjRsh1OZQ2T27rVt/L74ujp9QLhAVxNNg3HGLASVOy20D1NIJYbikCrmxJ3vtkFAxqssBoxcLBcBT/sf0IAODfrl2EuaWpF1qOx4rFwlOZUSOwrkYbfUMhvH5YdD1VpP39eW4HZubHAk7W1VC6zgwGcfy0GLrn1fdikuhVLNxmsBk1AIOarJhpsGLhtt5hBMJR5DjtuP6CWarc59/UFcMmxdL6VhlsJmpqctLofBK42FIbr37QgVBExoKKAswrm1qRpuhOY1BD6dodz4TUl+bBm6vf0L3RxHLLPad6EYpkb1p2m8Fm1AAMarLGSMXCJ+NP5rUluaqtii/wOBN1NRY5glIjU8O2bnUljp6mVtQOmGexpT8UwVEdBqrR+JR9TwY6egKA+pl58OY64Q9F0dTuy8qfGYpE0emLBTWVPH6afhYbaF2CKJKcSvv2RJTWboscQaW79ymZsiqBQY1qeoeCeONwrOvp6jS7npKZpVj47uf34Irv7UCjjhuYaSRxvHOBjkssx2KzSUrR/LHT2QmEO/r8kGXAZbdhZp47K39mKhjUZImRioWbe2JnwnUz1A1qLo4XC1tlsnBmmZr4AD7uf1LNq/s7EY7KWFhZmFEdmFhs2Wzwtu79bbHnij8f6tb5SggAwpEo9rTE/p8YqUhYqIsH6yd7svOc096XyNKk0x2qNQY1WSIyNccMUCzcnHT8pKa/mV0CmwSc6BlCR/wBb2ZT2fskiLZuZmrU81L86OnjGWRpgESG8pSBMzWyLKOzPwAAzNQYRFOHD8OhCArcDpxXlnlzhdrEm9RsBTVK55OBZtQADGqyZma+G5UGKRYW71DVDmoKPU6cX2Wdupqp7H0SRFBzdiiEQQN0vJnd2cEg3jwSP3rKoJ4GSDzuT50dRkSnzcaT8QXCyk6dxpZeyLIxr3M6EUXCy2q9hspMCLUz8gAAzWcGs/LntSrt3Axqpi0jDOGTZVk5fqpV+fgJAC6aI46gzF9Xk0mmptDjRFGOEwDbutXwx/0diERlnF9ViDkz8zK6r8qiHDhsEoKRKDr7jZlR7IpnaYBYGzEnIOvvvfh8muUGPHoCsn/8ZMQVCQCDmqwyQrHwmcEgBoMRSFKiQ0dNiWJhC2RqMqipAdjWrSal6ynDoycAsNskJZNm1LburlHB1u6W7O/1oZFEkfAKgxUJC+L4qcsXUKaha0lZkcBMzfRlhGJh0c5dWeiBO419Rqm6cHYJJClWOzT6idlsMul+ApIG8LGuJiM9AwG8dTQWJGfSyp2sptjYHVCiVVZgXY2+Tg8ElMfKshqvvhczDm+uC4We2HNVNh7XRpxRAzCoySojFAu3aNTOLRTlOrGwohAA8PZxcx9BZZyp4awaVfxxfyciURmLZxWhbkZmR0+Csq3boP9vRJFwTnx9CYMafYmdSueV5SvHykYk/n2c7NG+rqatj8dP015ysfAHbf26XENzjzadT8kutsgRVOaZGtHWbcwXTjPoGwrh+XdbAKhz9CQktnUbNFMTz3JeNr8UALC/rR/BcPYmxdJI7xlw39NYRJ2k1pmafn8IPn/sTZ+RNnQDDGqyTkwW3nuqV5c/Xxw/qT2jJtlFFplXo15NDYOadPX7Q3j8T4fx0YdeQ2NLLxw2SbWjJyB5Vo0xgxpRKPzhOSUoznUiGI7iQLs+b4QoaTN3nVffC5lEtoqF2+NHT95cJ/Km+KZPKwxqskzvYmGtpgknu2hOrK7maPcgun2Byb/BoDLpfgJYUzMVA4EwfvDaYVzy3W147E+H4POH0VBegKc2fFjVx6xZMjUVhR4sjddw8AhKH6FIVHkTavRMjTKrRuPHtVIkbLAsDcCgJusW61wsLI6f1KpNGIs314WG+MhuM8+ryWRODZAIak4PBOAPad+NYGaDgTB+tP0oLvnua3jk1UPoGw5hXlk+fnDzcvz+y5coxzBqEUFNtjpF0tUVfzNQXujG8prYC6mYk0LZdaC9H/5QFIUeR0aTrLOhtiQ+q0bjmppWg7ZzA4Cx8kbTwOhi4fwspu78oQg64u8AtaypAWJ1NU0dPrxz7Aw+vqRK0z9LK5lmaopynMh3OzAQCOPU2WHMM+AUUr0NByP477dP4okdR9EzGAQQW8735bXn4eNLqmDXaMhZUY4TBW4HfIEwTp0dwnnlU9v4rYXYNOHYv9PyQg+W1XoBMFOjF1EkvLy22JBD95KJTM2ps8MIR6Jw2LXJW7T3GXPwHsBMTdbpWSws5qXkux0oztW2gt8Ke6DEO/ipZmokSUostmSx8Aj+UARPvXEclzy0DQ++cgA9g0HUzcjFo3+/FK/edSmuXTZLs4AGiP2/qTHotu7+4TAC8aLg0gI3llV7AcTWj5yNB36UPWYpEgZix5Uuhw3hqKzsZtKCUdu5AQY1ulik02Th5J1PkqTtO44L58Q6oA53DeD0gPnqaoLhKIKR2AtLrnPq2bTqYg7gS+YPRfDTt07g0oe24d9e+gCnBwKoLs7BQ3+3BFu/chmuv6Bas3eXoynFwlmawJoqMaOmKMcJj9OOolwn6uNTlBt1ajCYzsxSJAzEtnXXxJ9ztCwWNuqKBIBBjS70KhbORju3UJKXqKv5iwnn1STXWeRMsfsJ4GJLIRCO4P++fRJrHtmO+3+7H12+AGZ5c/Cd6xdj21dX4+//piZrwYxQa9BZNYmjJ7fyOTHwrTGeNaDs6PL5cersMCTJuEP3RlNm1Wi4A8qoKxIA1tToYrFObd3ZaOdOdlF9CQ52+vDOsZ6MlxBm22C8ndtlt8HlmPqLbfU0HsAXikTx3smz2HawG7/b06a8u6so9OCLl8/D3/9NTUY/20yJoMZobd1i8F55YeIFY1mtF7/a3cq6mix772QvAGB+WQEKPMYdupdMeVxrlKmJRGV0xI+2jNj9xKBGB4t0KhbWeprwaBfXz8B/7TyJd0yYqREzanKnWE8jiAF806WmptsXwI5D3djW1IU/H+5WBnQBQFmBG3esmYcbP1wDj1P9FR3pqjZoW3dX/PiprCApqIlnCfacim3s1vr4mGISR0/Gr6cRlLZujYKabl8A4agMu01CWYF78m/IMgY1OigtiBULt/f58UFbPy6Mb7bW2sksHj8BUP5eTR0+nBkMoiTPlZU/Vw1imnDeFDufBKsvtYxGZew51YttB7ux/WAX9p4aeaRakufCZfNLsbqhFOvOrzBEMCMkz6oxUqDQ1Z9o5xYWVBTC5bChdyiEEz1DGW8qp9SIzqcL4h1oZqD1rBqxHqGi0JP1I+NUMKjRyaJZRWjv8+P91r6sBDWyLCtp9mwdP83Md+O8snwc7hrAX46fwfpFFVn5c9Ugjp9yM6inARLHT12+AALhiCZLRLOtbyiEHYe7sb2pCzsOdSut2MLiWUVYs6AMaxpKsaTaq2kXUyZEwDkYjODMYBAz8o3xrjO5nVtwOWxYVFWI95p70dhylkFNFgTDUeyN1z2aKVOTPKtGi2C9TSkSNl49DcCgRjeLZxVhywedWSsW7vYFEAhHYbdJWa1Yv6i+BIe7BvD2sR5TBTVDGe59EkryXPA4bfCHomjv9WO2SV+MmnuG8Lu9bdh+sAu7Tp5FVE58rcDtwKXxbMxlDaUjjk2MzOO0o6LQg45+P1rODhswqBl5PctqimNBTXMv/nZ5tR6XNq180B7bt+VN6j4zg5qSHEhSLFjvGQxipsqP6zYDdz4BDGp0szjLbd0iFVnl9cCZxZThxfUz8N9vN5uurmYww71PgiRJqC7OxZGuAbT2DpsyqBkORnDNv78OX9Jm+YbyAqxeUIo1DWVYUVec1ceUmmpLctHR70fzmSHDdLeIQuHSUcHhslov8Cawm8XCWaEM3avxGuZoMhVuhx2VhR609flxsmdIg6DGuDNqAAY1uhHFwke7BzAcjGTUNpyKbLZzJ0vU1fSjdygIb6456mqGMpwmnGyWNwdHugZMW1dzuMsHXyCMArcDX79qAVY3lCoF0GZXXZKDv5wwTrGwLMtKofDoTM3yeNAVG9sfMVR9khXtihcJrzDR0ZNQOyMXbX1+NJ8ZVP36lRk1RcbMyJrz7ZUFlBa4UZLngizHAhutnTyjT1BTVuDB3NI8yLK55tUMZrj3KZnZ27oPd8Yen+fPKsQ/XlxnmYAGMN5iy7NDIYQisbO90lGdJdXFOZiZ70IoImN/lqeRT0e7lSJh8wU1dfG6Gi06oIx+/MSgRkfzy2O7gA52+DT/s1qUoCb7xx8X1cemC799zDxBjaqZGpMP4DvcFQtqziszzn4ktRhtVo3I0pTkuc4pKpckKTGEj0dQmuro86Otzw+bBGVLupnUztBuVo1Yv8Cghs4hJu4e6tQ+qGnWKVMDxOpqAHNt7FarpgZIzKoxa6bmSFfs8XleufUWctYYLKgR9TTjzf8wS1DTNxxSsp1mJObTNFQUIi+LS4fVolVb93C8UxBgUENjmF+RvaBGpCGz1c6d7OJ4Xc0H7f3oGwpl/c+fCrW6nwCYfqmlyNRYccu4CPLb+/wIxXd96Wmsdu5ky2piRyGNLWezdk3p6hsK4fJHtuOGJ3ZCluXJv8GAzDifJplWx09iRk2+24FCjzGDPQY1OpqvZGq0rakZCoaVpZLZmiacrKzQgzkzY3U1fz1hjiMoNTM1YsFce9+wIV440+EPRZQshhWPn0rz3XA5bIhEZbT3arfVOFVd/WKa8NiZmiU1RZAkoOXMMHoMuih2V/MZ9AwG8UF7v2lrf5RJwiaspwESx0+nBwKqZsxEPU1lkcewHWEManQ0P/4i0do7DJ9fuwyGeFEqynGiKEef/SUX18eyNW8fM8cRlJqZmpnxF86oDGVnilkc7R6ALAPFuU7MzDdH51o6krcatxigO22svU/JCj1OzC2NZcyMegTV2JIYU/FaU5eOVzI1gXAE+1pjwZgZO5+A2HO9Nzf2XK/m0arRi4QBBjW6Ksp1oiL+5KVltqZZx6Mn4aI5oq5m+mVqbDYpaV2CuY6gjiQVCRv1nVmmjFQsPN7gvWRGr6vZk3RdZgxq9rX2IxiJoiTPpetzZqbqStTfAWX0GTUAgxrdZaOupjnLiyzHclE8U7O/rQ/9Gmal1DKsYvcTkGjrNltdjWjnnmfBImHBSMXCnb54ofA4mRrA2EGNLMd2gQl7TvUa9phsPLubE/U0Zg7ka2fE1yWcGVTtPkWmZpZBVyQADGp01xB/schGUKNH55NQWZSDuhm5iMrAuyaoqxlUghp1BpyZdbHlYdH5ZMEiYcFIs2q6JykUBkYGNdGosQpxm88MoXcoBJfdhvnl+ZBlYPvBbr0vKy2inma5SetpBE0yNX3T+PjpwQcfxKpVq5Cbmwuv1zvmbbZu3YpVq1ahoKAAFRUV+PrXv45weGRR0969e3HJJZfA4/GgpqYGDz30kFaXrIvzstDWrSyy1DGoAYCL55hnXs1QUL3he0BSpsZkx09WnlEj1BgkqIlGZXT5zt3QPdqCigJ4nDb4/GEcO6394M50iOzRh6oKceWHYrvezHYE9d7JXgDmLRIWlFk1qtbUTOPjp2AwiBtuuAG33377mF/fs2cPrr76aqxfvx67d+/Gc889h9/+9re45557lNv09/fjyiuvRF1dHXbt2oWHH34YDzzwAJ588kmtLjvrxKyagx3a19TomakBEkdQ75igWHgwoO7x0ywTThUOhCPKuzwrzqgRaoqNcfx0ZiiIcFSGJGHCfT0Ou03ZHbe7uTdLV5eaPfEi4WU1Xly+sAwA8OdD3abp+mvrHUZHvx92m4SlNUV6X05G1M7UyLKctCJhGgY13/rWt3DXXXdh8eLFY379ueeew5IlS3Dfffdh3rx5uOyyy/DQQw/hhz/8IXy+WNbi2WefRTAYxNNPP43zzz8fN910E770pS/h0Ucf1eqys068WJweCGhy9hyJysoLaa3ORW9isvD7rX2adnupQcnUqFZTE/vZm6mm5vjpQUSiMgo8jnFbjK2gpiT2BH12KKTr41IUCc/Ic026INSodTWinmZpTRGWVntRkueCLxDGuyeMO1cn2a74fJqFlQWqvaHRS128pqa1V51REj2DQQTDUUgSUF5k3OcD3WpqAoEAPJ6R58Y5OTnw+/3YtWsXAGDnzp249NJL4XIlWknXrVuHgwcP4uzZ8f+RBAIB9Pf3j/gwqlyXQ8mgaNEB1dnvRzAShcMmoVLn6HqWNwc1JTmxupqTxn2Si0blxJoElY6fRE1NW+8wIgargxiPKBI+ryzf1AWTkynwOFEcb39tOaNf0NmlTBOevAhT1HsYKagJRaLY1xrL1Cyt9sJuk7B6fikAYNtBcxxBvd+ayDSZXVlBYgZTmwpvpsQcp9J89zkrPIxEt6Bm3bp1eOutt/Dzn/8ckUgEra2t+Pa3vw0AaG9vBwB0dHSgvLx8xPeJ33d0dIx735s2bUJRUZHyUVNTo9HfQh1iB5QoylSTSD1WF+fAbtP/hSlRV2PcI6jhUET5tVqZmvJCDxw2CeGorLwjN7rpUE8jKMXCOhZyp9LOLYgX3aYOn9Kpp7dDnT4EwlEUehyYHc8SrFkQO4IyS12N2MO3oKJQ5yvJnM0mqTquoNUEM2qANIOae+65B5IkTfjR1NSU0n1deeWVePjhh3HbbbfB7XZj/vz5uPrqq2MXZcss1rr33nvR19enfLS0tGR0f1qbr9TVqB/UKIssZ2R/keVYxBHUOwYuFhYzaiQJ8DjVifvtNgmV8TbIvad6TTE+3so7n0YzQrFwokh48kxNZZEHZQVuRKIy9rX1TXr7bBD1NEtrvLDF30BdOr8UdpuEI10DmixXVFsiqLFGIK9mXU2indvYQU1ab0PvvvtubNiwYcLb1NfXp3x/X/nKV3DXXXehvb0dxcXFOHHiBO69917lPioqKtDZ2Tnie8TvKyoqxr1ft9sNt9u4Z36jNWg4q+ZkfEZBbYkxHogXxfdAvd/ah4FAGPkGXBYnpgnnuRyqHrvUluSi5cwwbvvv9+DNdWJptRfLa71YXluMZdVeFOXqM+15PMqMGgu3cwtGmFUjMjUTzagRxMbuVz/oRGNzLz48u0Try5uUGLq3tNqrfK4ox4kVdcX4y/EzeK2pExs+Mkefi0tB31AIHfH/B/MtEtSo2QGVmCZs3Bk1QJpBTWlpKUpLS1W9AEmSUFVVBQD4+c9/jpqaGlxwwQUAgJUrV+Kb3/wmQqEQnM7YE/6WLVvQ0NCA4mJzt9slS87UyLKs6gtpc7xGQCw401tNSS5m5rtxeiCAY90DWJL0BGgUIlOj1owa4YtrzkMgFMX7rX3oHQphx6Fu7DiUmOFRX5qH5TXFWFbrxfIaLxZUFMAxScGoVkKRKI6fjgXE4vFpZUaYVZNYkZDaG7JltfGgxiB1NYkiYe+Iz1+xoCwW1BzsNnRQczD+prKqyINCj7HeYExVIlOT+QA+MaNG79rMyWj2Nrm5uRlnzpxBc3MzIpEIGhsbAQDz5s1Dfn7snd/DDz+M9evXw2az4Ve/+hW+853v4Pnnn4fdHnsxufnmm/Gtb30LGzduxNe//nXs27cPjz/+OB577DGtLlsX9aV5sNsk9PvD6PIFUko/p8oI04RHm5nvwumBAM4adGO3KBLOUzmLtHLuDPzy9lUIhqNo6ujH7uZeNLb0YnfzWZzoGcKx7kEc6x7E/3vvFIDY0deSWbFszrKaWEanoig775JO9gwiHJWR57KjMkt/pp6MsCqhyxevqUmhUBgwVgfUYCCsZJqXVo9shb58QRk2/b4Jbx/rwVAwbNiuooMdsYaSBotkaYBEB5Q6x0/Gn1EDaBjU3HffffjpT3+q/H758uUAgG3btmH16tUAgN///vd48MEHEQgEsHTpUrz44ou46qqrlO8pKirCq6++ijvuuAMrVqzAzJkzcd999+HWW2/V6rJ14XbYMXtGLo52D+Jgh0/doKZHHD8ZJ6gRi9Z6h4I6X8nYxFZbtTM1gsthw5JqL5ZUe/GZ+OfODAaxJx7g7G6JBTs+fxh/OXEGf0mawPyJpVX4/qeXa3JdyRLrEay78ymZmFVz6uwwolFZqQnJpsTxU2qZmiXVXkhSrICzy+dPqWtKK/ta+xCVY1mO0cdn88ryUV2cg1Nnh/HmkR587EPl49yLvkSmpsECRcJC8vFTpqcAlqypScfmzZuxefPmCW/z2muvTXo/S5Ysweuvv67SVRlXQ0UBjnYP4lCnD5fOV+eIr98fUrIhes+oSVacG2vRPztozKBGydRk8R1lSZ4LaxaUKd0i0aiMY6cH8F48m/PeybNo6vDhpb1t2HT9Ys1rkRKdT9avpwGASq8HdpuEQDiK7gF1s6WpiERldKdRKAwA+W4H5pcV4GCnD43Nvbjy/PHrDLUmjp7GOk6WJAmXLyjDf+08ideauowb1HSIoMY6j/nq4hxIUuw57fRAEKVTnDcVCEeUQnaj19Rw95NBaNEBJboNZuS5DFWQ6xVBjUGPn5RMjUozaqbCZpMwr6wAf/83Nfjff7sYf7jzUlQWeSDLUGaBaGm6BTVOu005ZtPjCKpnIICoDNik2L/XVIkjqN06H0Eldz6NRQTr25q6DNn5J8symkRQU26dTI3bYVem/2ay2LKzLxC/PxtK0nh86oFBjUE0aLADKtHObZwsDQBl0JlRj5/0yNSkQnSV7MnCC9jhzunTzi3oWSws3gXPzHenVRy+rNYLAGjUeV2CqOsZb7XAyvoZ8Dht6Oj344N24w1D7ej3w+cPw26TMLfMGE0VaqlVoa07eUaN0Y+jGdQYhGghPNw1oNrmXSNs5x5LsdEzNRp1P2VqSfwFY+8pbTM14UgUx7pj7+qmw+A9Qc9i4c4UtnOPRWRq9p7q1W1SdbcvgNbeYUgSlJ1Uo3mcdnx03kwAsWyN0YgszZyZeYaeljsVdTMyD2rM0s4NMKgxjLqSXLjsNgwFI6rtBzpp0KBGFAqfNWqmJqBN91OmlsUzNVp3uzSfGUIwEoXHaTN8UaCa9JxVk247tzC/vAC5LjsGgxEc6dJnY/feeD3NvNJ8FEzQCm3k6cKJehrrBfFqzKpp7zP+IkuBQY1BOOw2zI3XL6hVV9Ni0KBGZGp6DZqpUfY+GSxTs6i6SOl2Oa3B8lNB1NPMK8vXpQtILyKoOaXD/qd0Bu8ls9skLIm3UDe26LNPTRm6N8m+pDUNsaBmd0svzhisSeCQmCRswZlMYkZZJrNqWk3Szg0wqDGUhnj9wkGV6mpEutFwQU2ewTM1Bj1+KvQ4UT8z9gQl3h1r4cg02vmUTM/jJzGjZirb0JfV6LvcsvHUxEXCQpU3BwsqCiDLwI5DxsrWiOMnq0wSTlanQqbGLO3cAIMaQ5mv4rqEcCSqHGPVGWTvk+A1eKZmUMnUGOv4CUi8cIhuEy2IIuHpsB4hWU1x7Am70+eHP5TdJZGJ46f0axaUDigdioVlWVYyNctSmA5+uXIE1T3JLbMnHIniSHcskLfKzqdk4vjp9EAQA/HOznS1mWSZJcCgxlDml4mgJvOz8fY+PyJRGS6HbUrv/rQkjp8GAmEEw1Gdr+ZcQ/F/+Hk6tnSPR+mA0jBTI46fpsN6hGQleS7kueyQZahW15aqdDZ0j7Y83gF1qNOnjCPIlpM9Q+gbDsHlsKVUjyKCmh0HuxCOGOPf/omeIQTDUeQ47coQRisp9DiVjtOpLBWVZVkJaipZKEzpEE8KR7sGMv4HL46eaopzDFcXUZTjhOgK7B023hFUovvJyJkabTZ9R6Jy0vHT9MrUSJKkW7GwaOmeylTg8kIPKos8iMqxRbHZJILr86sK4XJM/nKyvLYY3lwn+v1h7DqpTw3QaKKGcX65dWvIauPZ+qnMqun3h5XsNQuFKS2zvDnIddkRjERxIsNdHeJJ2WhHT0CsuFEsjDPiEVRi95PxMjULKwvgtEs4OxTCqbPqZxNazw4jEI7C5bAZal9YtiSKhbMX1IQjUaXwe6qTjPXaA9U4xmbuidhtElbHJ6a/dtAYdTVW3Pk0Wl0Gs2pElqYkz4Ucg9UZjoVBjYHYbBLOU2kI38kzxtv5lEykQ424KiGx+8l4mRq3w46FlbGJp1q8gB3uij3u5pbmw27Rd60T0aNY+PRAELIce8FPZ5pwskRdTXazH0o9zSRFwsmSpwsbgRV3Po2mzKqZwuPaTDNqAAY1hiM6oDINaozazi0YeVWCUScKC+JdsRYdUNNtPcJoolg4m0GNqKcpzXdP+fhDj0xNKBLF/rZYlmOyzqdkl80vhU2K1Q6eOqvfVnRBmVFj4RoyJVjPIFNjhqMngEGN4cxXKVNj1GnCgpFXJRhh99NExFwSLTqgxHbu6RrUiE6RlizOqsmkSFhYXF0Eu01CZ39AGZSmtYMdPgTCURR6HJidxioWb64LK+pibeh6Z2uGgmEle2Hp46d4GcLJKdTUmGlGDcCgxnDUWGwpy3JiRo3B9j4JRl2VIMuy4TM14l35+619qneQHOmafjufkiXvf8rW4sVOUSScwWbwXJdDee7I1h4oUSS8tMab9j4go0wXPtI1AFmOLRGd6gZrMxDHT229foTSfM7g8RNlRLxbONEzNOVZGX3DIfj8sWyDUVsUE7NqjJWpCUaiCMd36Bg1U1Nfmo88lx3DoYgyX0MN0aicNE3Yuu9aJ1Id//fiC4TRN5ydgLtLhUwNkP0jqD1pFgknu2JBOQDgraM9GA5mdyZQMmXonoWPnoDYUEeP04ZIVEZrmg0GyooEZmpoKsoK3CjKcSISlZWlgukSWZqyArdhq9WLDbr/Sex9AoBcpzF/dnabhMXKEVSvavfb1jeMoWAETrukvLObbjxOuzLXKVt1NV1i8N4U2rmTiXk1u7MW1KQ2SXgs88vzMcubg0A4ireOnlb5ylJn5Z1PySRJSmzrTvNx3cbjJ8qEJEmYH0/9i06UdCXauY37wuTNM+bxk5hR43bY4LAb959HYgifenU1IkszZ2YenAb+u2st27NqOn1T29A92nJxLHlK/WPJ0QYCYRyKPz8trR57M/dEJEnCmgXx1m4dj6BE7aIVJwmPVhvfAdWcxg6ocCSKjngm0QwrEgAGNYaUaV2NeDI28pyREoMePyVm1BiznkZIHsKnliOd03Pn02iJuprsFNyKFQllGR4/zS3NR4HbgeFQRJWp5BPZ19oHWQaqijxTrgW6PKm1O1v1S6NZeefTaEpbdxodUF2+ACJRGU67hNJ8c9QcMagxoIYMd0A1G3SRZbLE8ZPBMjUBYy6zHE0ENQc7fKrtKTo8zYuEhWxnakRNzVSmCSez2SQsqREbu3szvawJpbqZeyIr62fC7bChrc+v2hLfdJwZDKI7XqRt9ZoaYGqzakSRcHmhxzTTlhnUGJCSqZlqUGOG4yejZ2oM2vkkVBV5MDPfhXBUVmaFZOrwNN3OPZqYVZONGSrBcBQ98QGUmRYKA8nFwtoO4UvufJqqHJcdq+bOAKDPEZTIhNeU5CDf4JlZNUxlVk1bn7nqaQAGNYYkgpqWM8NTWlBn9Bk1AFCcl1iToFfqeSxGn1EjSJKk6hA+WZYTx0/TPFOTzanC3fH1CE67pIw5yMSymtj8F+0zNfEi4Sl0PiW7fGGsC0qPeTXKeoRpkKUBErNqmtMYVyAyNWappwEY1BhSSZ4LM+Pnl2K5YKqC4Sja4i14ojDMiMQTeDgqw5flzcITGY4f5Rj9+AkAlohiYRVewDr7A/AFwrDbJMw24L6wbBLHT61nhxGJahtwJx89qZHeF5maw10D8Pm1Odrt8vnR2jsMSYLShTdVoq5m18mzWV+ZkliPMD2CmlneHNik2HOcOHabjNlm1AAMagyroSL2bjndI6jW3mHIMpDjtGNmfubv/LTicdrhccYefr2DxqmrGQyIoMb46eil8foJNTqgRD3N7Bm5KW1btrLyQg9cdhvCUVnz6bxqFQkLpQVuzPLmQJaBvSp2xiXbG8/SnFeWn/GxzSxvDhrKCxCVgT8f7lbj8lKWaOe27s6nZC6HTTlGSrWuJhHUMFNDGVLWJaTZAXWyJ7HIMt0pn9mWmCpsnLqaoXhLd56JMjXHTw+iL8OC68PsfFLYbRKqs7QDqku0c2dYJJxsWXxejVZHUEo9TYZHT4Ie04VlWVY6xKbL8ROQfgeU2VYkAAxqDKthisXCyiJLAxcJC14DBjVKpsYEhYMleS6l/mNva29G96UUCU/zehqhOmldgpbU2Ps02nJlY3evaveZrFGFzqdk4ghqx6FuzY/7hFNnhzEQCMNpl1BfOn2OW9OdVWO2ZZYAgxrDOi8e1BxOc97ESRO0cwuJpZbGOX4yU6YGSLywZHrUcDgePM+bpossR6stiT2Jaz2rJnH8pGKmJmldgtpF+LIsKzVcy1QKai6o9aIox4neoRB2N2vbtSWIcRlzS/On1aDJdNq6B5NWhbCmhjImpgp39PvTOlowQzu3YMTjJzFR2Aw1NUBimmsmRw2yLLOde5RsdUB1KoXC6mVqFs0qgsMm4fRAAK296gZlJ3qG0O8Pw+WwqVZg67DbcNn87E4Xni47n0arK0n9+EnUkxV4HCjwODW9LjUxqDGoAo9TaaM7lMa6BDNMExa8BhzAJ3Y/5Rm8pVtIZGp6p3wf3QMB9A2HYJMwrVLxExGLYFs0nlWj7H1SMVPjcdqxsDJW/Kp2XY3I0iyqKlQ1w3F5lutqpsvOp9FEWUIqwbqopzFTOzfAoMbQRLYm1XUJsiybYkaNUGzAAXxmy9ScX1UImxQ7xuiID8pKl5hPUzcjDx6DLvHMtpos1dR0qbT3aTTlCErluhq162mEy+aXwibFMihqZ5fGMp12PiUTs2rODAYnbfk3Y+cTwKDG0OanuS6hZzCIoWAEkgSle8PIDJmpCZorU5Prcigp9Km+KxdHT6ynSRBBzemB4JQGYKYiEI4oj301C4WBRFDz1xNnVL1f0fmkVj2NUJznwvLa2OBArQfxhSJRHO2OPean2/FTvtuBGfFlwpMdQYmgprLIPPU0AIMaQ5tfll5QIx6klYUeuB3Gf1E2ZKYmYK5MDYCMJwsrO58Y1CiKcpwoyokF3afOapM5EEdPLodN+bPUsmreDDjtEvac6sOOQ+rMfwlFospKjiUqtXMnS15wqaVj3YMIRWTkux2mePOntlSPoFqZqSG1ifPegx2+lLoYzNTODSRWJRipUNgsu5+SKRu7pxrUcD3CmLQuFk4cPblVnylVWZSDz6ycDQB48OUPEI5EM77Pgx0+BMNRFHocmK3Bc8yahlhQ8+bR06otaR2LGJMxvzzf8LO8tJBqsXA7a2pIbfPK8iFJseOZ0wOTv/CbqZ4GSJpTY6SJwkFz7H5KJiYL7z3Vh+gU5nwcYefTmGpKtB3Ap7Rzqzh4L9n/uPw8eHOdONQ5gOfebcn4/pLrabQIBhZWFqCyyAN/KIqdx3pUv39B2fk0zepphFplB9TEs2rEuh1makg1Hqdd2cOTyhGUmWbUAMY8flK6n0yUqZlfXgC3wwafP4zjKQ7VEnoGAugZDEKSYjM7KEHrYmEtBu8lK8p14s4rzgMAPPrqoYx3Qak9n2Y0SZIS04UPaHcEpXQ+TbN6GiGVTE00KiuZGjPNqAEY1BheOh1QieMnc7TliuF7g8EIguHM0+NqSHQ/mSdT47TbsGiWyNb0pvW9IktTXZyDHBP9nbOhVvOgRttMDQD8w8V1qC/NQ89gEP+x/WhG96X2eoSxXN6QaO1We3CgkFhkOT12Po2WyqqE04MBBCNR2CT1O/O0xqDG4ER1/uEUZtWcPJPY+2QGhR4nxGJiI2RrIlEZ/lAsuMozwZqEZEviQ/j2tKQ3WZhD98an9awardq5kzntNnzz6oUAgKfeOD7lAG0gEFYeK0tqMtvMPZFV82bA5bChtXdY+fPUNBAIK1Oip+/xU+xx3d43PO6bybbexPZ4s01cNtfVTkMiqJksU+MPRZR3fnUmCWpsNknp+jBCW7dYkQCYK1MDjByNn45EPQ2PnkZLLhTWImuQGLynzfGTcPmCMnxk3gwEw1F89w9NU7qP90/1QZZjRaNaZpZyXQ6srJ8BAPjTgU7V718c45cWuFESb22ebkrz3ch12RGVgVPjBOztSueTubI0AIMaw2tQZtUMTPjEKh6cBW6HMv/FDIy0KkF0PtltEtwOc/3TEC22H7T3p3WUd4g7n8ZV5c2BJAH+UBTdAwHV7z9RU6PtC4ckSfjm1R+CJAEv7W3HrpPp71dSjp40zNII686vAAD8bk+76vd9qGN6Dt1LJkmSErCPtwPKrO3cAIMaw5s9Iw9Ou4SBQBhtE0yMFeejNSW5pmpT9CpLLY0T1OQ67ab6GQLA7Bm5KPQ4EAxHU55ADSRv556+T/LjcTlsynbiY93pFWCnQutC4WQfqirE36+oAQD820sfpJ15EkXCWtbTCFctqoDDJuFAez+OpLEiJhVN07xIWFCykOPU1bSZtJ0bYFBjeC6HDXNmTt4BZaZFlskSmRr9j5+UwXsmaucWJElKe15N71AQ3b5YBoKZmrEtmhUrJs1kt9ZYhoMR9Ptjj7dSDY9zkt195XzkuuxobOnF7/amlwXZo9F6hLEU57lwaXzB5W8b21S9bxHwz5/GmRpg8mJhs65IABjUmIKoqzk0wTtws7VzC14DHj+ZqZ07mXgXvSfFuhpRTzPLm4N8kxVGZ4sSKKZZgD0ZUSTscdpQ6MnOz76s0IN/Xj0XAPDd3zelPOCuq9+Ptj4/bBKweJb2x08AcO2yKgDAb/e0qVrPNF13Po022awaMaPGbCsSAAY1piBSpQcnyNSYbZqwUKwcPxkgU2PCwXvJEhu7U3sB5s6nyU21AHsyXb7Edu5sHnV+/pJ6VBV50No7jKfeOJ7S9+yJP57OKyvIWlfg2oXl8DhtONEzhPdb1Qkou32JmUzTvdtvslk1zNSQplJZbGm2acJCcZ6YKmyATE188J6Z9j4lWxpv6z7U5cNACksYlfUIDGrGtXhWESQpVjgpjurUoNTTZOnoSfA47fja+gUAgP/YdiSlv1Pi6Ck7WRogNlJh7cJyAMCLKh1BiaOnupLcaT+TqS5p/9PoKeT+UESZYM+aGtKEyNQc7hxAZIwx+NGobNqgxkibukWmJs+kT3hlhR5UFnkgy8C+FN7dKossufNpXAUeJ+bFJy2neqyXCmXwXhaKhEf75NIqLK0uwmAwgke3HJr09onOJ6+2FzbKJ5fGjqBe2ts25vNeuhJD96Z3lgaIZWDsNgmBcFTJGgod8YaUHKfdVJ20AoMaE6gpyYXbYUMgHB1zeFb3QACBcBR2m2S6dKGhWrqVQmFzZmqAxBC+VApbjyjHT3ySn8iyDBeGjqUrS+3cY7HZJPzPj38IAPDcX5vRFN+FNJZoVM5q51OyyxpKUehxoLM/gL8cP5Px/SV2Pk3PScLJnHabMoPm5KjVKm1JM2rM1gUKMKgxBbtNUt5Nj1VXI85Fq7zmm/6YyNToH9QMKoXC5szUAKkXtvr8IbTH35GxpmZiSzWoqxHHT2UF2c/UAMCHZ5fg6sUViMrAgy8fGLcY90TPIPr9YbgdtqxnONwOO65aVAkgVjCcqem+82m0upJYsfDoWTVmnlEDMKgxjYk6oJR27hJz7HxKllhqqf/x05Cy98m8mZplogNqkqyCyNKUF7qVqc40NiVT09I7pS3oY+nsTxQK6+We9Qvhstvw+uHT2H6we8zbiMfR+VWFurxh+mS8C+r3+9oz2g8Xjco4FK8h4/FTjGgqGT2rRsyoETOazIZBjUlM1AHVHE8f1pisngYYualbrReMqRoUG7pN2v0EAIvix0+nzg6jZ4IpuNz5lLqGitgW9H5/GCfS3II+ns54S7ceNTVC7YxcfPYjswEA/+vlDxCKnBs0iIxftutphIvrZ6C0wI3eoRBePzx24JWKlrNDGA5F4HLYMNtkHaJaqRtnqnB7HzM1lAVKpmasoMakRcJA4vgpKgM+/+QdO1qyQqam0OPE3NJYxm6i1u4jbOdOWfIWdLXqaroNkKkBgH9eMw8leS4c7R7EL/7SfM7Xxd93mU5Bjd0m4ZrFmR9BiUnC80rz4TDZEb1WkjugkrWaeO8TwKDGNERb97HuwXPSsGadJgzEWkxznLHMiN51NVaoqQESBZ0T1YCI4JidT6lRfqbNvRnf12AgDF+8KF3voKYox4m71p4HAHjsT4fRN5w4Bg6Go9jfFiuuzXaRcDIxiG/LB50YDqY2MHA07nw6V228XKF5nEJhM7ZzAwxqTKOqyIN8twPhqHxOCtzMmRogMYBP76DGCt1PAFJalyBm1Mxn0WRKltV6AQCNKQ42nIhooc1z2Q0xyfnTF9ZiXlk+zgwG8R/bjiifP9jhQzAcRVGOU9c3TMtqvKgtycVQMDLlzd1NbOc+h6ipOTsUQr8/FszKspyoqWFQM9KJEyewceNGzJkzBzk5OZg7dy7uv/9+BIMjX7j27t2LSy65BB6PBzU1NXjooYfOua8XXngBCxYsgMfjweLFi/HKK69oddmGJUkS5osOqKRi4cFAWBmUZLZpwoLXIMXCgyZfkyAk2rr7xuxqGQyElRSzmMFCExMF2Afa+hEITy1bIGRrO3eqHHYbvnn1QgDAM2+eUApHG5Pm0+jZ2itJEj6xNHYENdVBfNz5dK58twMz82PPveL/ee9QCMPx9RkVJlyRAGgY1DQ1NSEajeLHP/4x9u/fj8ceewxPPPEEvvGNbyi36e/vx5VXXom6ujrs2rULDz/8MB544AE8+eSTym3eeustfPrTn8bGjRuxe/duXHfddbjuuuuwb98+rS7dsBrGmCwssjTeXCcKPebsYinOM0imxuRrEoSFlYVw2iWcGQzi1Nnhc75+tDuWpZmZ71YmOtPEakpyUJzrRDASRVN7ZpujRVBTqlM791hWN5TikvNmIhiJ4jt/OAAgMWxwWXX2JgmP55NLZwEAdhzqQl+ab34C4QiOn45lt3n8NFLtqHUJ4s3OzHwXPE5zPg9qFtSsX78ezzzzDK688krU19fjk5/8JL761a/iV7/6lXKbZ599FsFgEE8//TTOP/983HTTTfjSl76ERx99VLnN448/jvXr1+Nf/uVfsHDhQvzbv/0bLrjgAvzgBz/Q6tINS3SqjBXUmPXoCUheaqlvpkasSTB7psbjtGNhZWzA2FhHUFyPkL7kLeiZzqvpMkiRcDJJkvDNaxbCJgGvvN+Bv544k9XN3JNpqCjAgooChCIy/rA/vQ3jR7sGEYnKKPQ4UGGgn7kR1M0Qs2piQZ+YXWXWoycgyzU1fX19KCkpUX6/c+dOXHrppXC5Eu8W161bh4MHD+Ls2bPKbdauXTviftatW4edO3eO++cEAgH09/eP+LCCRKZmQPlciwWCmsRSS70LhUX3kznfoSQTR1BjjfZX2rlZJJyW5Hk1mUgcPxknUwMACyoKceOHawEA//qbfTgSz+gt0bFIONknliY2d6fjYKeYJFxgygm5WhKvG+L4SZkmbNIZNUAWg5ojR47g+9//Pr7whS8on+vo6EB5efmI24nfd3R0THgb8fWxbNq0CUVFRcpHTU2NWn8NXYmizhM9g/DHzz1F2tDcQY0xViUMBcVCS/MHNUuVIXznFrYeETufmKlJi5KpybCtO3lDt9F85WPzkeeyo6nDB1mOdcAY5ZhM7ILaebRHWTORioMdHLo3HlEAfnJ0UDOdMjX33HMPJEma8KOpqWnE97S2tmL9+vW44YYbcMstt6h28eO599570dfXp3y0tLRo/mdmw8x8F0ryXJDlxJwRM7dzC0Y4fpJlWQlq8gzQkZIp8QL8/qk+hEcNVTvMnU9TIgLFY92Dadd1JFNWJBgwqCktcOOf18xTfq/XfJqx1JTkYnmtF1EZeGlv6kdQ3Pk0vtGzasw+owaYQlBz991348CBAxN+1NfXK7dva2vDmjVrsGrVqhEFwABQUVGBzs6RLXri9xUVFRPeRnx9LG63G4WFhSM+rECSJOXdtajmFw9GM04TFoxw/BQIR5VNwFbI1MwtzUeey47hUEQ5RgAAfyiiPGZ4/JSekjyX8iKwt7V3yvejZGoMkgEZbeNH5ygzSowU1ACJbE06R1Dc+TQ+MaumrW8YgXBkemZqSktLsWDBggk/RI1Ma2srVq9ejRUrVuCZZ56BzTbyj1u5ciX+/Oc/IxRKvOvZsmULGhoaUFxcrNxm69atI75vy5YtWLlyZdp/WStQ6mq6fIhEZZw6a6Hjp0H9MjVDSUO9zDxRWLDbJGUK7t6k5ZZHuwcgy7FAcgY7n9KmHOtNsa5GlmXDtXSP5nHa8cQ/rsCGVbNx04XGOrq/ZkklbFKsWHv0zqKx9PtDaIsXvzKoOdfMfBdyXXbIcmy1CguFJyACmtraWjzyyCPo7u5GR0fHiFqYm2++GS6XCxs3bsT+/fvx3HPP4fHHH8dXvvIV5TZf/vKX8Yc//AHf+9730NTUhAceeADvvvsuvvjFL2p16YaWvNiyo9+PUESG0y6h0sSFXaKtWM9MzWB88J7HaYPdZo1iwmVj1IAcSdr5xKLJ9GXaATUQCCsBtJ57nyazuLoID3zyfBQYbExEWYEHq+bOBAD8bu/k2RoxSbiyyIOiXGP9XYxAkiTlDfGx7kEl4J5Wx0+p2rJlC44cOYKtW7eiuroalZWVyodQVFSEV199FcePH8eKFStw991347777sOtt96q3GbVqlX42c9+hieffBJLly7FL3/5S/zmN7/BokWLtLp0Q0vugDoZnyxcXZxr6hfixERh/TM1Zm/nTiZegPcmBTViHMA8Hj1NiRIotow92HAyYjt3gdthiYygHpQjqBQG8YmdT5ycPT5xpPqX4z2IyoDLbsPMPOMG3JPR7F/Vhg0bsGHDhklvt2TJErz++usT3uaGG27ADTfcoNKVmdv8eHFna+8wPojvZTFzPQ2QKBQeDkXgD0V0Gfo0aJHBe8lEW3dTu0/5uSrrEdj5NCXnVxXCYZNweiCAtj5/2vtxugywndvs1i2qwP/8zT4c7PShqaMfCyYoABZBPIfujU/Mqtl5rAcAUOn1wGbiN8nc/WQyRblOZb7F1gNdABIr5M2q0ONQMk16rUqwyuC9ZLO8OZiZ70I4KuOD9lgArBw/8Z3rlHicdiyojP3sprLc0oiD98ymKMeJ1Q2lACbP1ohMDdu5xyeOn8TyUjPPqAEY1JiSSKX+5cQZAOYuEgZi57reHH1XJVhp8J4gSZIyOG1PSy8C4YiyDJUzaqYuMQOoN+3vNXqRsFl8Mr65+3d728Y9BpRlObHziUH8uMTxk/gxVpq4ngZgUGNKoopftCCbdZFlMq/Om7rF3icrzKhJltytc/z0IKJyLDNmlIFqZrQsg2JhUVPD46fMXLGgHHkuO1rODGP3OP8funwB9A2HYLdJmMcgflx18bZuId0jVaNhUGNCozfNmj1TAyTauvU6fhoMWGeacLIlNYmN3crOp3J2PmVi2QSDDSfTGa+pKS8w97thveW47PjYh2KT5sc7ghJHT7Nn5Jp2OWM2VHk9cCTV0Ji5nRtgUGNKo+ctmL1QGEieKqxzpsZCNTVA0hTc04PYdTK2T41HT5mpL81HvtuB4VBEmc6cqq5+FgqrRRxBvbS3fczgMjFJmEdPE3HYbZhVnAhkGNRQ1iWnUmfmu5BvgSOTxFRhnTM1Fup+AmJTcEUmT0xhZSo+M3abNOHC0Il0slBYNZecV4riXCdODwTw9rEz53xd2flUbo2J8lpKzvbPYk0NZVue24Gaklg0bYUsDZAYwHd2kJkatYkX4DPxny07nzInZgClUywsy7LS0s3jp8w57TZctTg29+y3e1rP+Xrydm6aWPLuQDMPcgUY1JiWOIIyezu34NV5AN+gsqHbekHN6P09PH7KnDjW251GW3e/Pwx/KHZMwuMndYhBfL/f14FAOLHqJBKVlRoyBjWTE8XCRTlO0zdLMKgxqQ/PLgEALI4/uZpdolBYp0xNQHQ/Wev4CYDS1g0AeS47KouYJcjU8lovgNhwN5Hlm4yopynKcbJwVSUXzi5BRaEHPn8Y2w92K58/2TOIQDgKj9NmiUYKrdWXxoIaK/ysGNSY1MaPzsFvv/gRfGZlnd6XoopinVu6RaYmx2LdTwCwaFYhRHPDPHY+qaK80IOKQg+iMrCvtT+l70nU0zBLoxabTcInloojqEQXVPJ8GjOvkMmWS+eX4n9cPg//+vEP6X0pGWNQY1IOuw1Lqr1w2K3xv9Crc0v3sAV3Pwm5LocyfIzrEdSzNN4u39hyNqXbi8F7ZaynUdUnl84CAGw90KkspuXOp/Q47TbcfWUDLpxTovelZMwar4hkesU6t3RbcaJwso/Mi202vqCuWOcrsQ6lWLilL6Xbd3LvkyYWzSrEnJl58Iei2PJBJwDufJrOGNSQIYjjp77hEKLR9LcfZ0rZ/WTyIrnxfPXKBvz0cxfixr+p0ftSLCPdycLc+6QNSZLwCbG5O34EdZA7n6YtBjVkCOL4KSoD/f7sH0FZPVOT47Ljsvmlpt6+azSLZxVBkoDW3mF0+wKT3j7Rzs1MjdpEF9SfD3WjvW9Y2XE2elApWR+DGjIEl8OGvHhAoUdb91DQ2pkaUl+Bx4l5pbEapVSG8HHwnnbmleXj/KpChKMy/n3rEUTlWPaXO86mHwY1ZBh6rkoQBYZWzdSQNpalMYRPKRRmUKMJka154d0WALGjJ3b6TT8MasgwivPEqoTsBjXhSBSBcGwomhW7n0g7S1Osq5FlOammhtkDLXw8HtSE4zV5PHqanhjUkGEoHVCD2T1+GgolJpFabfcTaUvJ1LT0Tljg3jsUQjC+dJFHItqY5c3Bh2cnuvsaKrjzaTpiUEOGodfxk+h8ctgkuCwy94eyo6GiAG6HDf3+sFKcOhbRzl2c64TbwcBZK59cNkv5NTufpic+g5Nh6LWpO7nziWfwlA6n3YZFs+Ibuyeoq2E7d3ZcvagCbocNuS475pdz0OR0xKCGDEPvTA07n2gqxHLLxgmWW7JIODtm5Lvxwm0r8bNbLkaBx6n35ZAO+CxOhmGETA1RupbVeoE3gcZT408W7orPseGMGu0tsciSX5oaZmrIMPRalSC2LDNTQ1OxLP4ieqCtH4FwZMzbiEwNj5+ItMWghgzDq2zqznKmJn78xEwNTUVNSQ6Kc50IRqJoaveNeZtEUMNMDZGWGNSQYRQrm7p1ytRwRg1NgSRJk86rEdOES7mhm0hTDGrIMPQ6flIyNTx+oilKnlczli5maoiygkENGYY3PlHYH4rCHxq7NkELIlOT6+TxE02NkqkZo607GpXRPcCWbqJsYFBDhlHgdsAR3yKdzWyNWGbJacI0VaKt+1j3IPpG1YSdHQoiFIlNG+Y0YSJtMaghw5AkKVEsnMVVCcqGbtbU0BSV5LlQNyMXALC3tXfE10Q9zcx8F5ycWE2kKf4LI0Px6lAsrGzoZqaGMiCyNaPrasSKhDIWCRNpjkENGUqxDm3dzNSQGsbrgGKRMFH2MKghQ9FjVQInCpMalilBTR9kObGxWxw/MVNDpD0GNWQoiVUJWSwU5u4nUsH5VYVw2CScHgigrc+vfJ6D94iyh0ENGUpiVk32jp+YqSE1eJx2LKgsADByuaXY+8RllkTaY1BDhqLH8ZNSU8NMDWVIKRZOmlfTxb1PRFnDoIYMRSkUHtSh+4mZGsrQsjGKhUVNDY+fiLTHt6ZkKF4djp/Y/URqEUHN+6f6EI5EIUkSpwkTZREzNWQo2S4UlmU5UVPDOTWUofrSfOS7HRgORXC4awA9gwFEojJsEjAjz6X35RFZHoMaMpTivOxmavyhKET3LTM1lCm7TcKS6iIAsSF8XfGjpxn5bjg4TZhIc/xXRoYiup/6/SFEovIkt86cyNIAQA4XWpIKxBC+Pad60eVjOzdRNjGoIUMRu59kGegb1j5bI2bU5LrssMWXaRJlQnRA7W7uTRQJc/AeUVYwqCFDcdptKIi3VmejrTsxo4ZHT6SO5bVeAMChTh9OnB4EwBk1RNnCoIYMx5uXvWLhIQ7eI5WVF3pQUehBVAa2NnXFP8fjJ6JsYFBDhqNMFR7U/vhpMOn4iUgtS2tixcJHugYAsJ2bKFsY1JDhZHOqMKcJkxaW1RSP+H1ZATM1RNnAoIYMJzGrJguFwjx+Ig2ITI3ATA1RdjCoIcMpzmKmZpDThEkDi2cVQUpqpitjTQ1RVjCoIcMRbd3ZGMA3FOA0YVJfgceJeaX5AGID+WbkMaghygYGNWQ4IlOTje4nZmpIK2IPVGm+G3bOQCLKCgY1ZDiJTE0WCoWZqSGNiMnCbOcmyh6+PSXDSWRqstDSzUwNaeTjSyrxWlMX/nb5LL0vhWja0CxTc+LECWzcuBFz5sxBTk4O5s6di/vvvx/BYOLdt9/vx4YNG7B48WI4HA5cd911Y97X9u3bccEFF8DtdmPevHnYvHmzVpdNBpDNQmF2P5FWvLkuPL3hw/jE0iq9L4Vo2tAsqGlqakI0GsWPf/xj7N+/H4899hieeOIJfOMb31BuE4lEkJOTgy996UtYu3btmPdz/PhxXHPNNVizZg0aGxtx55134vOf/zz++Mc/anXppLPkQmFZ1nappRi+xzk1RETmp9kz+fr167F+/Xrl9/X19Th48CB+9KMf4ZFHHgEA5OXl4Uc/+hEA4M0330Rvb+859/PEE09gzpw5+N73vgcAWLhwId544w089thjWLdunVaXTzoqzotlaoLhKIZDEU33MjFTQ0RkHVktFO7r60NJSUla37Nz585zsjjr1q3Dzp07x/2eQCCA/v7+ER9kHnkuO5z2WLeI1m3drKkhIrKOrAU1R44cwfe//3184QtfSOv7Ojo6UF5ePuJz5eXl6O/vx/Dw8Jjfs2nTJhQVFSkfNTU1U75uyj5JkhKrEga1rath9xMRkXWkHdTcc889kCRpwo+mpqYR39Pa2or169fjhhtuwC233KLaxY/n3nvvRV9fn/LR0tKi+Z9J6srWqoQhZmqIiCwj7Wfyu+++Gxs2bJjwNvX19cqv29rasGbNGqxatQpPPvlk2hdYUVGBzs7OEZ/r7OxEYWEhcnJyxvwet9sNt5uzIcwsW0stB+M1NXnM1BARmV7aQU1paSlKS0tTum1rayvWrFmDFStW4JlnnoHNlv5p18qVK/HKK6+M+NyWLVuwcuXKtO+LzCORqdH6+CmWqdGyGJmIiLJDs2fy1tZWrF69GnV1dXjkkUfQ3d2tfK2iokL59QcffIBgMIgzZ87A5/OhsbERALBs2TIAwG233YYf/OAH+NrXvobPfe5zeO211/D888/j5Zdf1urSyQASs2q0O34KhqMIRqIA2P1ERGQFmgU1W7ZswZEjR3DkyBFUV1eP+Fry7JGrr74aJ0+eVH6/fPnyEbeZM2cOXn75Zdx11114/PHHUV1djZ/85Cds57a4bBw/DcfraQBmaoiIrECzZ/INGzZMWnsDxCYPT2b16tXYvXt35hdFppGNQuGhUKyexmmX4HJwDRoRkdnxmZwMKRurEgZZT0NEZCkMasiQklclaEVME85jPQ0RkSUwqCFDEqsStOx+UjI13PtERGQJDGrIkERNjZYThZmpISKyFgY1ZEii+6nfH0Y43natNrH3iTU1RETWwKCGDMmb41R+3TesTV2N2PvEacJERNbAoIYMyWG3ocATy6BoVSzMTA0RkbUwqCHDEm3dWhULM1NDRGQtDGrIsIo1butmpoaIyFoY1JBhab0qgd1PRETWwqCGDEvrTd2cU0NEZC0MasiwvBpv6mamhojIWhjUkGFpXSjMmhoiImthUEOGVZwnpgprO6cml5kaIiJLYFBDhqV1obCSqWFNDRGRJTCoIcNKFAqzpoaIiCbHoIYMq1jzlm7W1BARWQmDGjIsb1KmRpZl1e+fE4WJiKyFQQ0ZlsjUBCNRJauilmhUxlCImRoiIithUEOGleuyw+WIPUTVPoLyhyMQyR9maoiIrIFBDRmWJEmaFQuLacKSBHgcDGqIiKyAQQ0ZmlbFwqLzKddph80mqXrfRESkDwY1ZGhejTZ1c+8TEZH1MKghQ9NqVQJn1BARWQ+DGjI0ZaqwyqsSuPeJiMh6GNSQoRUrx08qZ2o4o4aIyHIY1JChaVUozEwNEZH1MKghQ9OqUFipqWGmhojIMhjUkKFpVSisdD8xU0NEZBkMasjQivM0qqlh9xMRkeUwqCFDE91PvWp3P8UzNTnM1BARWQaDGjI0cfzkC4QRikRVu19maoiIrIdBDRlaUY4TUnyLgZr7n5TuJ04UJiKyDAY1ZGh2m4RCj1hqqV5dzTAzNURElsOghgyvWIO2bu5+IiKyHgY1ZHheDQbwsaaGiMh6GNSQ4YlMjZrHT5woTERkPQxqyPASqxLUO37i7iciIuthUEOGp8XxEzM1RETWw6CGDE85flJxAB93PxERWQ+DGjI8b566mZpgOIpQRAbATA0RkZUwqCHDSxQKq5OpEVkaAMhl9xMRkWUwqCHDK1a5pkbU07gcNjjt/CdARGQVfEYnw/OqPHxP6XxiloaIyFIY1JDhiUxN71AQsixnfH/sfCIisiYGNWR4IqgJR2UMBMKT3HpynFFDRGRNDGrI8HJcdrgdsYeqGsXCzNQQEVkTgxoyBTWLhUX3EzufiIishUENmYKaxcLKhm5maoiILIVBDZlCcrFwpjhNmIjImhjUkCkU58UzNYNqBDXM1BARWZFmQc2JEyewceNGzJkzBzk5OZg7dy7uv/9+BIOJF6Xt27fj2muvRWVlJfLy8rBs2TI8++yz59zXCy+8gAULFsDj8WDx4sV45ZVXtLpsMiivipu6B4OcU0NEZEWaBTVNTU2IRqP48Y9/jP379+Oxxx7DE088gW984xvKbd566y0sWbIE/+///T/s3bsXn/3sZ/FP//RPeOmll0bc5tOf/jQ2btyI3bt347rrrsN1112Hffv2aXXpZECJVQkqZGpETY2bmRoiIiuRZDWmmaXo4Ycfxo9+9CMcO3Zs3Ntcc801KC8vx9NPPw0AuPHGGzE4ODgi0Ln44ouxbNkyPPHEEyn9uf39/SgqKkJfXx8KCwsz+0uQLn7y+jH8r5cP4JNLq/Dvn16e0X195flG/Oq9Vtx71QJ84bK5Kl0hERGpLd3X76zW1PT19aGkpCSt2+zcuRNr164dcZt169Zh586d495HIBBAf3//iA8yN6+aLd3M1BARWVLWgpojR47g+9//Pr7whS+Me5vnn38ef/3rX/HZz35W+VxHRwfKy8tH3K68vBwdHR3j3s+mTZtQVFSkfNTU1GT+FyBdqbmpmzU1RETWlHZQc88990CSpAk/mpqaRnxPa2sr1q9fjxtuuAG33HLLmPe7bds2fPazn8V//ud/4vzzz5/a3ybu3nvvRV9fn/LR0tKS0f2R/lTN1LD7iYjIktJ+Vr/77ruxYcOGCW9TX1+v/LqtrQ1r1qzBqlWr8OSTT455+x07duATn/gEHnvsMfzTP/3TiK9VVFSgs7NzxOc6OztRUVEx7p/vdrvhdrsn+ZuQmaiaqeHuJyIiS0o7qCktLUVpaWlKt21tbcWaNWuwYsUKPPPMM7DZzk0Mbd++HR//+Mfx3e9+F7feeus5X1+5ciW2bt2KO++8U/ncli1bsHLlynQvnUxMDN8bCIQRDEfhckz95JSZGiIia9LsWb21tRWrV69GXV0dHnnkEXR3dytfE1mWbdu24eMf/zi+/OUv41Of+pRSJ+NyuZRi4S9/+cu47LLL8L3vfQ/XXHMNfvGLX+Ddd98dN+tD1lSY44QkAbIM9A4HUVbgmfJ9caIwEZE1aVYovGXLFhw5cgRbt25FdXU1KisrlQ/hpz/9KYaGhrBp06YRX7/++uuV26xatQo/+9nP8OSTT2Lp0qX45S9/id/85jdYtGiRVpdOBmS3SSjKUecISux+ymOmhojIUrI6p0YvnFNjDWse2Y7jpwfx3K0X46L6GVO6j0hUxtxvxCZS7/qfazEjn7VXRERGZeg5NUSZKFZhU/dwKKL8Oo9zaoiILIVBDZmGGpu6h+KdT5IEuDMoNiYiIuPhszqZhhpLLQeDiXoaSZJUuS4iIjIGBjVkGmostRQzanI5TZiIyHIY1JBpFOdlPlVYzKhhPQ0RkfUwqCHT8KpQKCxm1DBTQ0RkPQxqyDRUKRQOckYNEZFVMagh01AjU6PU1HCaMBGR5TCoIdNgpoaIiCbCoIZMIxHUhDDVQdiDrKkhIrIsBjVkGuL4KRyV4YsfI6VrKMDuJyIiq2JQQ6bhcdqR44xlWHoHp1ZXw0wNEZF1MaghU0nsf5paXQ0zNURE1sWghkwlsSphakENMzVERNbFoIZMpThPrEqY2vETu5+IiKyLQQ2ZSqaZmgHOqSEisiy+XSVTKU5xAF/fUAiHunw41OnD4c4BHOqM/fr0QCwY4vETEZH1MKghUxGzas4OxoKTfn8Ihzt9OBQPXEQA0+ULjHsf51cV4oLa4qxcLxERZQ+DGjIVcfz0yvvt+NOBTrT3+ce97SxvDs4rz8f88gKcVxb777yyfHY+ERFZFJ/dyVRmeXMAAD2DiZqaikKPErzML08ELwUep16XSUREOmBQQ6aydmEZ/u26RXDYJMwvz8e8sgIU5TB4ISIiBjVkMg67Df/fxXV6XwYRERkQW7qJiIjIEhjUEBERkSUwqCEiIiJLYFBDRERElsCghoiIiCyBQQ0RERFZAoMaIiIisgQGNURERGQJDGqIiIjIEhjUEBERkSUwqCEiIiJLYFBDRERElsCghoiIiCxhWmzplmUZANDf36/zlRAREVGqxOu2eB2fzLQIanw+HwCgpqZG5yshIiKidPl8PhQVFU16O0lONfwxsWg0ira2NhQUFECSJNXut7+/HzU1NWhpaUFhYaFq92t1/LlNDX9uU8OfW/r4M5sa/tymZqKfmyzL8Pl8qKqqgs02ecXMtMjU2Gw2VFdXa3b/hYWFfABPAX9uU8Of29Tw55Y+/symhj+3qRnv55ZKhkZgoTARERFZAoMaIiIisgQGNRlwu924//774Xa79b4UU+HPbWr4c5sa/tzSx5/Z1PDnNjVq/tymRaEwERERWR8zNURERGQJDGqIiIjIEhjUEBERkSUwqCEiIiJLYFCTgR/+8IeYPXs2PB4PLrroIvzlL3/R+5IM7YEHHoAkSSM+FixYoPdlGc6f//xnfOITn0BVVRUkScJvfvObEV+XZRn33XcfKisrkZOTg7Vr1+Lw4cP6XKxBTPYz27BhwzmPvfXr1+tzsQayadMmfPjDH0ZBQQHKyspw3XXX4eDBgyNu4/f7cccdd2DGjBnIz8/Hpz71KXR2dup0xfpL5We2evXqcx5vt912m05XbAw/+tGPsGTJEmXA3sqVK/H73/9e+bpajzMGNVP03HPP4Stf+Qruv/9+vPfee1i6dCnWrVuHrq4uvS/N0M4//3y0t7crH2+88Ybel2Q4g4ODWLp0KX74wx+O+fWHHnoI//7v/44nnngC77zzDvLy8rBu3Tr4/f4sX6lxTPYzA4D169ePeOz9/Oc/z+IVGtOOHTtwxx134O2338aWLVsQCoVw5ZVXYnBwULnNXXfdhd/97nd44YUXsGPHDrS1teH666/X8ar1lcrPDABuueWWEY+3hx56SKcrNobq6mp85zvfwa5du/Duu+/i8ssvx7XXXov9+/cDUPFxJtOUXHjhhfIdd9yh/D4SichVVVXypk2bdLwqY7v//vvlpUuX6n0ZpgJA/vWvf638PhqNyhUVFfLDDz+sfK63t1d2u93yz3/+cx2u0HhG/8xkWZY/85nPyNdee60u12MmXV1dMgB5x44dsizHHltOp1N+4YUXlNscOHBABiDv3LlTr8s0lNE/M1mW5csuu0z+8pe/rN9FmURxcbH8k5/8RNXHGTM1UxAMBrFr1y6sXbtW+ZzNZsPatWuxc+dOHa/M+A4fPoyqqirU19fjH/7hH9Dc3Kz3JZnK8ePH0dHRMeKxV1RUhIsuuoiPvUls374dZWVlaGhowO23346enh69L8lw+vr6AAAlJSUAgF27diEUCo14vC1YsAC1tbV8vMWN/pkJzz77LGbOnIlFixbh3nvvxdDQkB6XZ0iRSAS/+MUvMDg4iJUrV6r6OJsWCy3Vdvr0aUQiEZSXl4/4fHl5OZqamnS6KuO76KKLsHnzZjQ0NKC9vR3f+ta3cMkll2Dfvn0oKCjQ+/JMoaOjAwDGfOyJr9G51q9fj+uvvx5z5szB0aNH8Y1vfANXXXUVdu7cCbvdrvflGUI0GsWdd96Jj3zkI1i0aBGA2OPN5XLB6/WOuC0fbzFj/cwA4Oabb0ZdXR2qqqqwd+9efP3rX8fBgwfxq1/9Sser1d/777+PlStXwu/3Iz8/H7/+9a/xoQ99CI2Njao9zhjUUNZcddVVyq+XLFmCiy66CHV1dXj++eexceNGHa+MrO6mm25Sfr148WIsWbIEc+fOxfbt23HFFVfoeGXGcccdd2Dfvn2sc0vDeD+zW2+9Vfn14sWLUVlZiSuuuAJHjx7F3Llzs32ZhtHQ0IDGxkb09fXhl7/8JT7zmc9gx44dqv4ZPH6agpkzZ8Jut59Tmd3Z2YmKigqdrsp8vF4v5s+fjyNHjuh9KaYhHl987GWmvr4eM2fO5GMv7otf/CJeeuklbNu2DdXV1crnKyoqEAwG0dvbO+L2fLyN/zMby0UXXQQA0/7x5nK5MG/ePKxYsQKbNm3C0qVL8fjjj6v6OGNQMwUulwsrVqzA1q1blc9Fo1Fs3boVK1eu1PHKzGVgYABHjx5FZWWl3pdiGnPmzEFFRcWIx15/fz/eeecdPvbScOrUKfT09Ez7x54sy/jiF7+IX//613jttdcwZ86cEV9fsWIFnE7niMfbwYMH0dzcPG0fb5P9zMbS2NgIANP+8TZaNBpFIBBQ93Gmbi3z9PGLX/xCdrvd8ubNm+UPPvhAvvXWW2Wv1yt3dHTofWmGdffdd8vbt2+Xjx8/Lr/55pvy2rVr5ZkzZ8pdXV16X5qh+Hw+effu3fLu3btlAPKjjz4q7969Wz558qQsy7L8ne98R/Z6vfKLL74o7927V7722mvlOXPmyMPDwzpfuX4m+pn5fD75q1/9qrxz5075+PHj8p/+9Cf5ggsukM877zzZ7/frfem6uv322+WioiJ5+/btcnt7u/IxNDSk3Oa2226Ta2tr5ddee01+99135ZUrV8orV67U8ar1NdnP7MiRI/K3v/1t+d1335WPHz8uv/jii3J9fb186aWX6nzl+rrnnnvkHTt2yMePH5f37t0r33PPPbIkSfKrr74qy7J6jzMGNRn4/ve/L9fW1soul0u+8MIL5bffflvvSzK0G2+8Ua6srJRdLpc8a9Ys+cYbb5SPHDmi92UZzrZt22QA53x85jOfkWU51tb9r//6r3J5ebnsdrvlK664Qj548KC+F62ziX5mQ0ND8pVXXimXlpbKTqdTrqurk2+55Ra+AZHlMX9mAORnnnlGuc3w8LD8z//8z3JxcbGcm5sr/+3f/q3c3t6u30XrbLKfWXNzs3zppZfKJSUlstvtlufNmyf/y7/8i9zX16fvhevsc5/7nFxXVye7XC65tLRUvuKKK5SARpbVe5xJsizLU8wcERERERkGa2qIiIjIEhjUEBERkSUwqCEiIiJLYFBDRERElsCghoiIiCyBQQ0RERFZAoMaIiIisgQGNURERGQJDGqIiIjIEhjUEBERkSUwqCEiIiJLYFBDRERElvD/A9miAri8nrIYAAAAAElFTkSuQmCC" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "all_rewards = []\n", + "\n", + "ppo = CrowdPPOptimizer(HomogeneousGroup(actor.agent), config={\n", + " \"gamma\": 0.99,\n", + " \"gae_lambda\": 0.95,\n", + " \"minibatch_size\": 256,\n", + "})\n", + "\n", + "for t in (pbar := trange(30)):\n", + " num_steps = 0\n", + " episodes = []\n", + " while num_steps < 2000:\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id, env_name=\"mcar\")\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + " num_steps += len(data.rewards)\n", + " \n", + " all_data = concatenate(episodes)\n", + " \n", + " last_value = actor.agent.act(Observation(vector=all_data.last_observation), get_value=True)[2][\"value\"]\n", + " values = actor.agent.act(Observation(vector=all_data.observations), get_value=True)[2][\"value\"]\n", + " \n", + " record = convert_trial_data_to_coltra(all_data, agent=actor.agent)\n", + " metrics = ppo.train_on_data({\"crowd\": record}, shape=(1,) + record.reward.shape)\n", + " \n", + " mean_reward = metrics[\"crowd/mean_episode_reward\"]\n", + " all_rewards.append(mean_reward)\n", + " pbar.set_description(f\"mean_reward: {mean_reward:.3}\")\n", + " \n", + "plt.plot(all_rewards)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:32:23.974676Z", + "start_time": "2023-12-13T17:31:11.816425Z" + } + }, + "id": "56b220d45561a042" + }, + { + "cell_type": "code", + "execution_count": null, + "outputs": [], + "source": [ + "await cog.cleanup()" + ], + "metadata": { + "collapsed": false + }, + "id": "1fb0dfc4957749d2" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/gymnasium/interactive.ipynb b/examples/gymnasium/interactive.ipynb new file mode 100644 index 0000000..5a0213c --- /dev/null +++ b/examples/gymnasium/interactive.ipynb @@ -0,0 +1,307 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T17:33:49.536453Z", + "start_time": "2023-12-13T17:33:47.755623Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "from cogment_lab.actors import RandomActor, ConstantActor\n", + "from cogment_lab.envs.gymnasium import GymEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.runners import process_cleanup\n", + "from cogment_lab.utils.trial_utils import format_data_multiagent\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:33:49.592824Z", + "start_time": "2023-12-13T17:33:49.544145Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:33:52.081428Z", + "start_time": "2023-12-13T17:33:52.075851Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T18:33:52.073150\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:33:58.345454Z", + "start_time": "2023-12-13T17:33:56.187819Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch an environment in a subprocess\n", + "\n", + "cenv = GymEnvironment(\n", + " env_id=\"CartPole-v1\",\n", + " render=True\n", + ")\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"cartpole\",\n", + " port=9001, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:34:03.722724Z", + "start_time": "2023-12-13T17:33:59.337837Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch two dummy actors in subprocesses\n", + "\n", + "random_actor = RandomActor(cenv.env.action_space)\n", + "constant_actor = ConstantActor(0)\n", + "\n", + "await cog.run_actor(actor=random_actor, \n", + " actor_name=\"random\", \n", + " port=9021, \n", + " log_file=\"actor-random.log\")\n", + "\n", + "await cog.run_actor(actor=constant_actor,\n", + " actor_name=\"constant\",\n", + " port=9022,\n", + " log_file=\"actor-constant.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "outputs": [ + { + "data": { + "text/plain": "{'cartpole': ,\n 'random': ,\n 'constant': }" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:34:03.723625Z", + "start_time": "2023-12-13T17:34:03.721429Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:34:07.883447Z", + "start_time": "2023-12-13T17:34:05.647704Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "MOUNTAIN_CAR_ACTIONS = [\"no-op\", \"ArrowLeft\", \"ArrowRight\"]\n", + "LUNAR_LANDER_ACTIONS = [\"no-op\", \"ArrowRight\", \"ArrowUp\", \"ArrowLeft\"]\n", + "PONG_ACTIONS = [\"no-op\", \"ArrowUp\", \"ArrowDown\"]\n", + "CARTPOLE_ACTIONS = [\"no-op\", \"ArrowRight\"]\n", + "\n", + "# Change this if you use a different environment. Only discrete actions are supported for now.\n", + "# no-op is the default action when no key is pressed\n", + "\n", + "actions = CARTPOLE_ACTIONS\n", + "\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "outputs": [], + "source": [ + "trial_id = await cog.start_trial(\n", + " env_name=\"cartpole\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"gym\": \"random\",\n", + " },\n", + ")\n", + "\n", + "data = await cog.get_trial_data(trial_id)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:34:30.419894Z", + "start_time": "2023-12-13T17:34:29.936857Z" + } + }, + "id": "8052ff03998b0b52" + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [ + { + "data": { + "text/plain": "(19, 4)" + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data[\"gym\"].observations.shape" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:52:54.113076Z", + "start_time": "2023-12-13T17:52:54.105989Z" + } + }, + "id": "1800cbfeca577ec8" + }, + { + "cell_type": "code", + "execution_count": 11, + "outputs": [], + "source": [ + "await cog.cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:53:01.851686Z", + "start_time": "2023-12-13T17:53:00.771Z" + } + }, + "id": "5d5770465a23d064" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/gymnasium/interactive.py b/examples/gymnasium/interactive.py new file mode 100644 index 0000000..fb43108 --- /dev/null +++ b/examples/gymnasium/interactive.py @@ -0,0 +1,72 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import datetime + +from cogment_lab.actors import RandomActor, ConstantActor +from cogment_lab.envs.gymnasium import GymEnvironment +from cogment_lab.process_manager import Cogment +from cogment_lab.utils.runners import process_cleanup +from cogment_lab.utils.trial_utils import format_data_multiagent + + +async def main(): + logpath = f"logs/logs-{datetime.datetime.now().isoformat()}" + + cog = Cogment(log_dir=logpath) + + print(logpath) + + # Launch an environment in a subprocess + + cenv = GymEnvironment(env_id="CartPole-v1", render=True) + + print("Starting env") + + await cog.run_env(env=cenv, env_name="cartpole", port=9001, log_file="env.log") + + # Launch two dummy actors in subprocesses + + print("Starting actors") + + random_actor = RandomActor(cenv.env.action_space) + constant_actor = ConstantActor(0) + + await cog.run_actor(actor=random_actor, actor_name="random", port=9021, log_file="actor-random.log") + + await cog.run_actor(actor=constant_actor, actor_name="constant", port=9022, log_file="actor-constant.log") + + # Start a trial + + print("Starting trial") + + trial_id = await cog.start_trial( + env_name="cartpole", + session_config={"render": True}, + actor_impls={ + "gym": "random", + }, + ) + + print("Waiting for trial to finish") + + data = await format_data_multiagent(datastore=cog.datastore, trial_id=trial_id, actor_agent_specs=cenv.agent_specs) + + print(data) + return + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/gymnasium/local-training.ipynb b/examples/gymnasium/local-training.ipynb new file mode 100644 index 0000000..6c7c300 --- /dev/null +++ b/examples/gymnasium/local-training.ipynb @@ -0,0 +1,318 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 4, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T17:55:13.220688Z", + "start_time": "2023-12-13T17:55:13.216030Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "from cogment_lab.envs.gymnasium import GymEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.coltra_utils import convert_trial_data_to_coltra\n", + "from cogment_lab.utils.runners import process_cleanup\n", + "from cogment_lab.utils.trial_utils import concatenate\n", + "\n", + "from coltra import HomogeneousGroup\n", + "from coltra.models import MLPModel\n", + "from coltra.policy_optimization import CrowdPPOptimizer\n", + "\n", + "from cogment_lab.actors import ColtraActor\n", + "\n", + "from tqdm import trange\n", + "import matplotlib.pyplot as plt\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:53:33.296582Z", + "start_time": "2023-12-13T17:53:33.235940Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:53:33.624181Z", + "start_time": "2023-12-13T17:53:33.621354Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T18:53:33.619848\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:55:18.921705Z", + "start_time": "2023-12-13T17:55:16.764819Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# We'll train on CartPole-v1\n", + "\n", + "cenv = GymEnvironment(\n", + " env_id=\"CartPole-v1\",\n", + " render=False,\n", + ")\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"cartpole\",\n", + " port=9001, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:55:20.063208Z", + "start_time": "2023-12-13T17:55:20.059040Z" + } + }, + "outputs": [], + "source": [ + "# Create a model using coltra\n", + "\n", + "model = MLPModel(\n", + " config={\n", + " \"hidden_sizes\": [64, 64],\n", + " }, \n", + " observation_space=cenv.env.observation_space, \n", + " action_space=cenv.env.action_space\n", + ")\n", + "\n", + "actor = ColtraActor(model=model)\n", + "\n", + "\n", + "actor_task = cog.run_local_actor(\n", + " actor=actor,\n", + " actor_name=\"coltra\",\n", + " port=9021,\n", + " log_file=\"actor.log\"\n", + " )\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [ + { + "data": { + "text/plain": "{'cartpole': ,\n 'coltra': wait_for=>}" + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes | cog.tasks" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:55:31.732225Z", + "start_time": "2023-12-13T17:55:31.729176Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 10, + "outputs": [], + "source": [ + "ppo = CrowdPPOptimizer(HomogeneousGroup(actor.agent), config={\n", + " \"gae_lambda\": 0.95,\n", + " \"minibatch_size\": 128,\n", + "})" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T17:55:38.366171Z", + "start_time": "2023-12-13T17:55:38.201298Z" + } + }, + "id": "582b6bb1bf0c81df" + }, + { + "cell_type": "code", + "execution_count": 12, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 0%| | 0/10 [00:00]" + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABI1UlEQVR4nO3daXxU9d3+8c9MdrKShCSEbISwL2EV2REVREWpIF3cxaptQJHWVvy39b7vLlTr1ipS952qqAjSCiJLWAz7LhCWsISELBCyk2Qyc/4PAqkoCNnmzGSu9+s1DxgmZy4MMld+53e+x2IYhoGIiIiIk1jNDiAiIiKeReVDREREnErlQ0RERJxK5UNEREScSuVDREREnErlQ0RERJxK5UNEREScSuVDREREnMrb7ADf5XA4yM3NJTg4GIvFYnYcERERuQyGYVBWVkZsbCxW6w+vbbhc+cjNzSU+Pt7sGCIiItII2dnZxMXF/eBrXK58BAcHA3XhQ0JCTE4jIiIil6O0tJT4+Pj6z/Ef4nLl49yplpCQEJUPERERN3M5Wya04VREREScSuVDREREnErlQ0RERJxK5UNEREScSuVDREREnErlQ0RERJxK5UNEREScSuVDREREnErlQ0RERJxK5UNEREScSuVDREREnErlQ0RERJxK5UNEROQS7A6Dd9cfZdORIrOjtAoqHyIiIpfw2posfv/Zbu5/ZzPVtXaz47g9lQ8REZEfsD+/jGe+3A/A6UobK/YWmJzI/al8iIiIXITN7mDmR9upsTvw96n7yPx4y3GTU7k/lQ8REZGLeGHFQXbnlBLWxoc3774CgFX7CykoqzI5mXtT+RAREbmAHdnFzFl5EIA/TezFkE4R9EsIw+4wWLgt1+R07k3lQ0RE5DuqbHZmfrQdu8NgQmosN/aJBWDygDgA5m/JxjAMMyO6NZUPERGR7/jb0kwOFVbQLtiP/7upZ/3zN/aJxc/byv78cnbllJiY0L2pfIiIiHzL+qxTvLHuMABPTepD20Df+t8LDfBhXM8YQBtPm0LlQ0RE5Kzy6lp+PX8HhgE/GRTPVd2ivveac6deFm7P1cyPRlL5EBEROetPi/dw/PQZ4toG8Lsbe1zwNcNSImkf6k/JGRtf7dHMj8ZQ+RAREQFW7Mvng03ZWCzw9K2pBPl5X/B1XlYLt/TvAMDHW7KdGbHVUPkQERGPd7qiht9+sguAe4d15MrkiB98/aT+dade0vcXUlCqmR8NpfIhIiIe7/cLd1NYVk1KVBCPjut6ydcntwtiQGJbHAZ8ui3HCQlbF5UPERHxaJ/vyGXxzhN4WS08OyUVfx+vy/q6cxtPP95yXDM/GkjlQ0REPFZBaRW/X7gbgLSrUugTF3bZX3tDn/b4+1g5WFDOjuOa+dEQKh8iIuKRDMPgt5/spLjSRq8OIUwfk9Kgrw/x9+G6+pkf2njaEA0qH3PnzqVPnz6EhIQQEhLCkCFD+OKLL+p/v6qqirS0NCIiIggKCmLSpEnk5+c3e2gREZGm+mhzNiszC/H1tvLslL74eDX85/HJA+IBWLQ9lyqbZn5crgb9l46Li+Ovf/0rW7ZsYfPmzYwZM4abb76Zb775BoBHHnmEzz//nPnz55Oenk5ubi633HJLiwQXERFprOyiSv7v8z0A/HpsF7pEBzfqOEM6RRAb6k9pVS3L9uiH7ctlMZq4SyY8PJy//e1vTJ48mXbt2jFv3jwmT54MwL59++jevTsZGRlceeWVl3W80tJSQkNDKSkpISQkpCnRREREvsfhMPjpq+vZcLiIQUlt+eD+IXhZLY0+3jNfZvLCioOM6tKOt++9ohmTupeGfH43es+H3W7ngw8+oKKigiFDhrBlyxZsNhvXXHNN/Wu6detGQkICGRkZFz1OdXU1paWl5z1ERERayptfH2HD4SLa+Hrx9K2pTSoe8N+ZH2sOFJJXopkfl6PB5WPXrl0EBQXh5+fHgw8+yIIFC+jRowd5eXn4+voSFhZ23uujo6PJy8u76PFmz55NaGho/SM+Pr7BfwgREZHLcbCgnKeW7APg8eu7kxgR2ORjJkUGMijp3MwP3WzucjS4fHTt2pXt27ezYcMGfvGLX3DXXXexZ8+eRgeYNWsWJSUl9Y/sbO0YFhGR5ldrd/Crj7ZTXetgZJd23DY4odmOfevZjaea+XF5Glw+fH19SUlJYcCAAcyePZvU1FT+/ve/ExMTQ01NDcXFxee9Pj8/n5iYmIsez8/Pr/7qmXMPERGR5vbSqkPsOF5CiL83T03qg8XStNMt33Z9n/YE+HiRVVjBtuziZjtua9XkOR8Oh4Pq6moGDBiAj48Py5cvr/+9zMxMjh07xpAhQ5r6NiIiIo22O6eEfyw/AMD/3dyLmFD/Zj1+kJ8343udm/mhUy+XcuFb9l3ErFmzGD9+PAkJCZSVlTFv3jxWrVrF0qVLCQ0NZerUqcycOZPw8HBCQkKYPn06Q4YMuewrXURERJpblc3OzI+2U+swGN8rhpv7xrbI+0weEMen23L4fEcuf7ixx2WPafdEDSofBQUF3HnnnZw4cYLQ0FD69OnD0qVLufbaawF47rnnsFqtTJo0ierqasaNG8dLL73UIsFFREQux3PL9rM/v5zIIF/+NLFXs55u+bYrkyPoEBZATvEZln6Tx819O7TI+7QGTZ7z0dw050NERJrLpiNFTHk5A8OAV+8cyLU9olv0/Z5dtp9/LD/AiM6RvDt1cIu+l6txypwPERERV1ZRXcuvPtqBYdSdEmnp4gEwqX/dasfagyfJLT7T4u/nrlQ+RESkVfrLf/ZyrKiSDmEB/GFCD6e8Z2JEIFd0DMcwYMG2HKe8pztS+RARkVYnfX8h7284BsBTk/sQ4u/jtPe+dUDdxFPN/Lg4lQ8REWlVSipt/PbjnQDcNSSRYSmRTn3/63u3p42vF4dPVrD12Gmnvre7UPkQEZFW5X8+/4a80io6Rgby2PjuTn//QD9vxvdqD8D8zZr5cSEqHyIi0mos2X2CBdtysFrgmSmpBPiaM2vj1oF1p14W7zzBmRq7KRlcmcqHiIi0CoVl1Ty+YDcAD47qRP+EtqZluSIpnPjwAMqra1n6zcVvruqpVD5ERMTtGYbB4wt2UVRRQ7eYYB6+prOpeaxWC5P6/3fjqZxP5UNERNzeJ1tzWLYnHx8vC8/9uC9+3uaPNj9XPtYdOkmOZn6cR+VDRETcWk7xGf530TcAzLimC93bu8Z07PjwNgxJjsAw4FOtfpxH5UNERNyWw2Hwm493UFZdS7+EMB4YmWx2pPNMPjfzY6tmfnybyoeIiLitd9cfZd3BU/j7WHl2Sl+8vVzrY2187xgCfb04eqqSTUc08+Mc1/ouiYiIXKaswnJmf7EXgFnju9MxMtDkRN/XxtebG/rUzfz4eEu2yWlch8qHiIi4nVq7g1/N30GVzcGwlAjuuDLR7EgXNXlAPAD/3nmCyppak9O4BpUPERFxOy+vzmLbsWKC/bx5anIqVqvF7EgXNSipLQnhbaiosbNkt2Z+gMqHiIi4mT25pTz/1X4A/jChBx3CAkxO9MMsFkv9xlONW6+j8iEiIm6jutbOzI+2Y7MbXNM9uv5D3dVNGhCHxQIZWafILqo0O47pVD5ERMRt/P2rA+zLKyM80JfZt/TGYnHd0y3f1iEsgKGdIgD4dGuOyWnMp/IhIiJuYeux0/wz/RAAf57Yi3bBfiYnapj/zvzIxuHw7JkfKh8iIuLyztTY+fVHO3AYMLFvLON7tzc7UoNd17M9QX7eZBedYeORIrPjmErlQ0REXN6TS/aRdbKCmBB//vemXmbHaZQAXy9urJ/54dkbT1U+RETEpa07eJK3vj4CwJOT+xDaxsfcQE1w7tTLf3adoKLac2d+qHyIiIjLKq2y8ej8HQDcNjiBUV3amZyoaQYktqVjZCCVNXb+s+uE2XFMo/IhIiIu6/8+30NuSRUJ4W14/PruZsdpsm/P/PDkUy8qHyIi4pKW7cnn4y3HsVjgmSmpBPp5mx2pWfyoXwcsFthwuIhjpzxz5ofKh4iIuJxT5dXM+nQnAPePSGZQUrjJiZpPbFgAw1MiAfhkq2eufqh8iIiISzEMg999tpuT5TV0iQ7ikWu7mB2p2X371IsnzvxQ+RAREZeycHsuX+zOw9tq4dkpffH38TI7UrMb1zOGYD9vcorPsP7wKbPjOJ3Kh4iIuIy8kir+sHA3ANPHdKZXh1CTE7UMfx8vbkyNBTxz46nKh4iIuATDMPjNJzspraqlT1wov7yqk9mRWtS5Uy9f7Mqj3MNmfqh8iIiIS3h/wzFW7y/E19vKs1NS8fFq3R9R/RPCSG4XyBmbnf/s9KyZH637OysiIm7h6KkK/vKfvQD8ZlxXUqKCTU7U8jx55ofKh4iImMruMPj1/B1U1tgZ3DGce4d1NDuS09zSLw6rBTYeKeLIyQqz4ziNyoeIiJjq9bVZbDpymkBfL56+NRWr1WJ2JKeJCfVneOe6kfGeNPND5UNEREyzP7+Mp5fuB+D3N/YgPryNyYmc79azp14+8aCZHyofIiJiCpvdwcyPtlNjdzCmWxQ/HhRvdiRTXNsjmmB/b3JLqsjI8oyZHyofIiJiihdWHGR3TilhbXz46y29sVg853TLt/n7eHGTh838UPkQERGn25FdzJyVBwH44829iArxNzmRuepnfuw+QWmVzeQ0LU/lQ0REnKrKZmfmR9uxOwxu7NOeCWd/6vdkfePDSIkKosrm8IiZHyofIiLiVH9bmsmhwgraBfvxx5t7mR3HJXjazA+VDxERcZr1Wad4Y91hAJ6c1Ju2gb4mJ3IdP+rXAasFNh89zeFWPvND5UNERJyivLqWX8/fgWHAjwfGM6ZbtNmRXEp0iD+jutTN/Ph4S7bJaVqWyoeIiDjFnxbv4fjpM3QIC+B3N3Y3O45Lmjyg7nLjT7fmYG/FMz9UPkREpMWt2JfPB5vqfpp/+tZUgv19TE7kmq7uHkVogA8nSqr4+tBJs+O0mAaVj9mzZzNo0CCCg4OJiopi4sSJZGZmnveavLw87rjjDmJiYggMDKR///588sknzRpaRETcx+mKGn77yS4A7h3WkSGdIkxO5Lq+PfNj/ubWu/G0QeUjPT2dtLQ01q9fz7Jly7DZbIwdO5aKiv9ujLnzzjvJzMxk0aJF7Nq1i1tuuYUpU6awbdu2Zg8vIiKu7/cLd1NYVk2ndoH85rquZsdxebcOrLvqZek3eZScaZ0zPxpUPpYsWcLdd99Nz549SU1N5a233uLYsWNs2bKl/jVff/0106dP54orriA5OZnf/e53hIWFnfcaERHxDCszC1i88wReVgvPTumLv4+X2ZFcXu8OoXSJDqK61sG/W+nMjybt+SgpKQEgPDy8/rmhQ4fy4YcfUlRUhMPh4IMPPqCqqorRo0df8BjV1dWUlpae9xAREfdnGAZ//+oAAHcPTSI1PszcQG7i/JkfrfOql0aXD4fDwYwZMxg2bBi9ev13SMxHH32EzWYjIiICPz8/HnjgARYsWEBKSsoFjzN79mxCQ0PrH/HxnnljIRGR1mbNgZNszy7Gz9vKg6M6mR3HrUzs1wEvq4Wtx4o5WFBudpxm1+jykZaWxu7du/nggw/Oe/73v/89xcXFfPXVV2zevJmZM2cyZcoUdu3adcHjzJo1i5KSkvpHdnbrbHkiIp7EMAz+sbxu1eNngxNoF+xnciL3EhXsz+izMz8+2dr6Np56N+aLpk2bxuLFi1m9ejVxcXH1zx86dIgXX3yR3bt307NnTwBSU1NZs2YNc+bM4Z///Of3juXn54efn/5Sioi0JhlZp9h89DS+WvVotMkD4li+r4BPtx7n12O74mVtPXf9bdDKh2EYTJs2jQULFrBixQo6dux43u9XVlbWHdR6/mG9vLxwOBxNjCoiIu7iheV1d6z98cB4oj38jrWNNaZ7FGFtfMgvrWbtwdY186NB5SMtLY333nuPefPmERwcTF5eHnl5eZw5cwaAbt26kZKSwgMPPMDGjRs5dOgQzzzzDMuWLWPixIktkV9ERFzMpiNFZGSdwsfLwoOjterRWH7eXkzs2wGA+Ztb15aEBpWPuXPnUlJSwujRo2nfvn3948MPPwTAx8eH//znP7Rr144JEybQp08f3nnnHd5++22uv/76FvkDiIiIazm312PygDg6hAWYnMa9nbvq5cs9+ZRUtp6ZHw3a82EYl54z37lzZ000FRHxUNuOnWbNgZN4WS38YtSFr3KUy9czNoRuMcHsyyvj85253H5lotmRmoXu7SIiIs3mhRV1ez1+1K8DCRFtTE7j/r4982P+ltZz1YvKh4iINIvdOSWs2FeA1QJpV2nVo7lM7NcBb6uFHdnFHMgvMztOs1D5EBGRZnFur8dNqbF0jAw0OU3rERnkx+iuUQB83Epmfqh8iIhIk+09UcqXe/KxWGDaGK16NLdzp14WbM2h1u7+oytUPkREpMlePLvX4/re7UmJCjY5TeszplsU4YG+FJRVs+aA+8/8UPkQEZEmOVhQxn921919dbpWPVqEr7eVm/vGAvBxK9h4qvIhIiJN8uKKgxgGjOsZTbeYELPjtFrnTr0s25NPcWWNyWmaRuVDREQa7fDJChbtyAVg+pjOJqdp3XrGhtK9fQg1dgefn/1v7q5UPkREpNHmrDyIw6jbk9CrQ6jZcVq9W1vJzA+VDxERaZRjpypZsC0H0F4PZ7m5byzeVgs7j5eQmee+Mz9UPkREpFHmph/E7jAY0TmSfgltzY7jESKC/BjTrW7mxyduPPND5UNERBosp/hM/VUXD1+tvR7OdG7j6adbc7C56cwPlQ8REWmwf646hM1uMCQ5goFJ4WbH8ShXdYsiItCXk+XVrN5faHacRlH5EBGRBskrqeLDTdkAPKRVD6fz8bIysV8HwH1nfqh8iIhIg7y8+hA1dgeDktpyZbJWPcxw7tTLV3vzOV3hfjM/VD5EROSyFZRVMW/DMaBu1cNisZicyDN1bx9Crw4h2OwGC7fnmB2nwVQ+RETksr225jDVtQ76xocxPCXS7DgebXL/utUPd7zTrcqHiIhcllPl1bybcRSAh65O0aqHyW7q2wEfLwu7c0rZe6LU7DgNovIhIiKX5fW1hzljs9OrQwhXdY0yO47HCw/05epu0QB84mYbT1U+RETkkoora3jn7KrH9DHa6+Eqbh1Yd+rls+3uNfND5UNERC7pjXVHKK+upVtMMNd2jzY7jpw1sks7IoP8OFlew6pM95n5ofIhIiI/qLTKxpvrDgN1qx5Wq1Y9XIWPl5Uf9YsF4OMt2SanuXwqHyIi8oPeXneEsqpaOkcFMb5XjNlx5DsmD4gHYPneAk6VV5uc5vKofIiIyEWVV9fy+tlVj2ljUrTq4YK6xgTTJy6UWofBwu25Zse5LCofIiJyUe+tP0pxpY3kyEBu7BNrdhy5iHMTT91l3LrKh4iIXFBlTS2vrs4C4JdXpeClVQ+XdVNqLL5eVvacKOWb3BKz41ySyoeIiFzQvA3HOFVRQ3x4ADf31aqHKwtr48u1PequQnKH1Q+VDxER+Z4qm52Xz656pI1OwcdLHxeu7typl4Xbc6mpde2ZH/rbJCIi3/PhpmwKy6rpEBbALWfvISKubUTnSNoF+1FUUcPKzAKz4/wglQ8RETlPda2duasOAfDg6E74euujwh14e1m5pV8HwPVPvehvlIiInOfjLcfJK60iOsSPWwdo1cOdnDv1snJfASddeOaHyoeIiNSz2R28tPLsqseoTvj7eJmcSBqic3QwqfFh1DoMPtuWY3aci1L5EBGRegu25pBTfIbIID9+ekWC2XGkEb4988MwDJPTXJjKh4iIAFBrdzBn1UEAHhiZrFUPN3VTn1h8va3syyvjm9xSs+NckMqHiIgAsGhHLkdPVRIe6MttV2rVw12FtvFhrIvP/FD5EBER7A6DF1fWrXpMHd6RNr7eJieSpjh36uWz7TlU19pNTvN9Kh8iIsK/d50gq7CC0AAf7hySaHYcaaIRndsRHeJHcaWNlftcb+aHyoeIiIdzOAxeXHEAgHuHdSTY38fkRNJUXlZL/XC4+Ztd79SLyoeIiIdb+k0e+/PLCfbz5u5hSWbHkWYy6Wz5WLW/kIKyKpPTnE/lQ0TEgxmGwQsr6vZ63D0sidAArXq0FilRQfRLCMPuMFi4LdfsOOdR+RAR8WDL9xaw50Qpgb5e3Duso9lxpJm56swPlQ8REQ9lGAb/OLvX444hSbQN9DU5kTS3G/vE4udtJTO/jF05JWbHqafyISLiodL3F7LzeAkBPl7cN0KrHq1RaIAP43rGAK4180PlQ0TEAxmGwT+W16163DY4gcggP5MTSUs5d+pl4fZcl5n5ofIhIuKBvj50iq3HivH1tnL/yGSz40gLGpYSSftQf0rO2Phqj2vM/GhQ+Zg9ezaDBg0iODiYqKgoJk6cSGZm5vdel5GRwZgxYwgMDCQkJISRI0dy5syZZgstIiJN8/ezqx4/uyKBqBB/k9NIS6qb+dEBgI+3ZJucpk6Dykd6ejppaWmsX7+eZcuWYbPZGDt2LBUVFfWvycjI4LrrrmPs2LFs3LiRTZs2MW3aNKxWLbKIiLiC9Vmn2Hi4CF8vKw+M0qqHJzg38yN9fyEFpebP/GjQ8P4lS5ac9+u33nqLqKgotmzZwsiRIwF45JFHeOihh3jsscfqX9e1a9dmiCoiIs3hhbNXuNw6MI72oQEmpxFnSG4XxIDEtmw5epoF23J4YFQnU/M0aTmipKTusp3w8HAACgoK2LBhA1FRUQwdOpTo6GhGjRrF2rVrL3qM6upqSktLz3uIiEjL2HL0NOsOnsLbauEXo839ABLnuvXsxtP5LjDzo9Hlw+FwMGPGDIYNG0avXr0AyMrKAuB//ud/+PnPf86SJUvo378/V199NQcOHLjgcWbPnk1oaGj9Iz4+vrGRRETkEs6tekzqH0dc2zYmpxFnur5Pe/x9rBwsKGfHcXNnfjS6fKSlpbF7924++OCD+uccDgcADzzwAPfccw/9+vXjueeeo2vXrrzxxhsXPM6sWbMoKSmpf2Rnu8ZmGBGR1mZHdjGrMgvxslr45VVa9fA0If4+XFc/88Pcz9pGlY9p06axePFiVq5cSVxcXP3z7du3B6BHjx7nvb579+4cO3bsgsfy8/MjJCTkvIeIiDS/c/dwublvLIkRgSanETNMHlB3dmHR9lyqbObN/GhQ+TAMg2nTprFgwQJWrFhBx47nT8RLSkoiNjb2e5ff7t+/n8TExKanFRGRRvkmt4Sv9uZjsUDaVSlmxxGTDO0UQbeYYG7o056K6lrTcjToape0tDTmzZvHwoULCQ4OJi8vD4DQ0FACAgKwWCw8+uijPPHEE6SmptK3b1/efvtt9u3bx8cff9wifwAREbm0F8+uetzYJ5ZO7YJMTiNmsVotfPHwCCwWi6k5GlQ+5s6dC8Do0aPPe/7NN9/k7rvvBmDGjBlUVVXxyCOPUFRURGpqKsuWLaNTJ51fFBExQ2ZeGV/srvthcfoYrXp4OrOLB4DFMPt6m+8oLS0lNDSUkpIS7f8QEWkG0/+1jc935DK+Vwxzbx9gdhxppRry+a2xoyIirdjBgnIW78wFYJpWPcRFqHyIiLRiL608iGHANd2j6RkbanYcEUDlQ0RaIYfDpc4mm+bIyQoW7qhb9Xjoaq16iOtQ+RCRVuU/u07Q/Q9LmPnhdkoqbWbHMdVLqw5idxiM7tqOPnFhZscRqafyISKthsNh8LelmVTXOvh0Ww7jnl9N+v5Cs2OZIruokk+35gAwfUxnk9OInE/lQ0RajRX7Cjh8soJgf286RgaSV1rFXW9s5PEFu0wdqGSGuemHqHUYDE+JZEBiW7PjiJxH5UNEWo3X1tbd3PJngxP490PDuXtoEgDzNhzjur+vZkPWKRPTOU9u8Rnmb667d4fmeogrUvkQkVZhd04J67OK8LJauGtIEm18vfmfm3oy777BdAgLILvoDD95dT1/WrzH1HtaOMPL6Yew2Q0GdwxncHKE2XFEvkflQ0RahTfWHgbght7tiQ0LqH9+aEokS2aM4McD4zEMeG3tYW74xxp2ZBeblLRlFZRW8a9NdaseD12tvR7imlQ+RMTt5ZdWsejsJaVTh3f83u8H+/vw5OQ+vHH3QNoF+3GosIJb5n7NM19mUlPrcHbcFvXK6ixqah0MSGzL0E5a9RDXpPIhIm7vnYwj1DoMBiW1JTU+7KKvG9Mtmi9njOSm1FjsDoMXVhxk4px17MsrdV7YFnSyvJr3NhwF6vZ6uMI9PEQuROVDRNzamRo77284BsDU4cmXfH3bQF/+8dN+zPlZf9q28WHPiVImvLCWl1YdpNbu3qsgr605TJXNQWpcKKO6tDM7jshFqXyIiFv7ZOtxiittJIS34doe0Zf9dTf0ac/SR0ZyTfcobHaDp5ZkcuvLGWQVlrdg2pZzuqKGdzKOAHVzPbTqIa5M5UNE3JbDYdRvNL1nWBJe1oZ94EYF+/PqnQN5+tZUgv282XasmOv/sYY31x12uxHtb6w7TGWNnR7tQ7i6e5TZcUR+kMqHiLitlZkFZJ2sINjPm1sHxjfqGBaLhckD4lj6yEiGp0RSZXPwv5/v4bbXNnD8dGUzJ24ZJWdsvLXuCFB3DxeteoirU/kQEbf1+tlVj58OTiDIz7tJx4oNC+Cde6/gjzf3JMDHi4ysU1z3/Bo+3HQMw3DtVZC31h2hrLqWrtHBjO0RY3YckUtS+RARt/RNbglfHzpVN1Ts7CTTprJaLdwxJIkvHh7BwMS2lFfX8ttPdnHvW5vIL61qlvdobmVVNl4/O9l12pgUrA089SRiBpUPEXFLb6w9AsD4XjF0+NZQseaQFBnIhw8M4fHru+HrZWVlZiFjn1vNoh25LrcK8k7GUUqrakluF8j1vdubHUfksqh8iIjbKSitYtGOuju23jfi0pfXNoaX1cL9Izux+KHh9OoQQskZGw/9axvT5m2jqKKmRd6zoSqqa+tPPU0fk9LgDbciZlH5EBG3807GUWx2gwGJben7A0PFmkOX6GAW/HIYM67pjLfVwr93nWDsc+ks25Pfou97Od7fcJSiihoSI9owoU+s2XFELpvKh4i4lbqhYnVTPO+7wCj1luDjZWXGNV1Y8MthdIkO4mR5DT9/ZzO/+mgHJWdsTsnwXWdq7Lyyum7VI+2qFLy99M+5uA/9bRURt/LptuOcrrQR1zaAsT2de2VH77hQFk0bzgOjkrFY6gacXff8atYeOOnUHAD/2niMk+XVxLUN4Ef9Ojj9/UWaQuVDRNzG+UPFOpqyx8Hfx4tZ47sz/4EhJEa04URJFbe/voHff7abiupap2Sostl5efUhAH45OgUfrXqIm9HfWBFxG+n7CzlUWDdUbMrAOFOzDEwK54uHR3DnkEQA3l1/lOv/sYZNR4pa/L3nb84mv7Sa9qH+TBqgVQ9xPyofIuI2zl3Z8eNB8QT7+5icBtr4evN/N/fivamDiQ315+ipSqa8nMFf/rOXKpu9Rd6zptbB3FV1qx4PjuqEn7dXi7yPSEtS+RARt7D3RClrD57EaoG7hyWZHec8wztHsuSRkdw6IA7DgFdWZzHhhbXsOl7S7O/16dbj5JZUERXsx48HNW6kvIjZVD5ExC2cW/UY36s9cW3bmJzm+0L8ffjbram8dudAIoP8OFBQzsSX1vHcsv3Y7I5meQ+b3cGcVQcBuH9kMv4+WvUQ96TyISIur6CsikXbcwGYOsI5l9c21jU9oln2yEhu6NMeu8Pg78sPMHHOOjLzypp87IXbc8kuOkNkkC+3DU5shrQi5lD5EBGX917GUWrsDvonhNE/oa3ZcS6pbaAvc37Wnxd+2o+wNj58k1vKhBfW8s/0Q9gdjRvPbncYzFlZt+px34hkAny16iHuS+VDRFxalc3OexuOATB1eMuMUm8pE1Jj+XLGSK7uFkWN3cFfv9jHlJczOHKyosHHWrwzl8MnK2jbxoc7rtSqh7g3lQ8RcWkLtuVQVFFDh7AAxvWMNjtOg0WF+PPaXQN5anIfgvy82XL0NOP/voZ3Mo7guMxVEIfD4IUVdaseU4d3JNDPuyUji7Q4lQ8RcVmGYdRvNL1nWJLbjhC3WCxMGRjPkhkjGNopgjM2O39Y+A13vLGBnOIzl/z6L3bncbCgnBB/b+4cmtTygUVamHv+nywiHiF9fyEHC8oJ8vNmSiu4rDSubRvemzqY/72pJ/4+VtYdPMV1z63mo83ZGMaFV0HqVj0OAHVTXUNcYL6JSFOpfIiIyzq36jFlYHyr+dC1Wi3cNTSJLx4eSf+EMMqqa/nNxzv5+TubKSir+t7rl+3NZ19eGUF+3tw7zLWv9BG5XCofIuKSMvPKWHOgbqjYPS42VKw5dIwMZP6DQ3lsfDd8vax8tbeAsc+tZvHO3PrXGMZ/Vz3uGppIaJvWUcBEVD5ExCW9vjYLgOt6xRAf7npDxZqDl9XCg6M68fn04fSMDaG40sa0eduYNm8rpytqWJlZwO6cUtr4erndlT4iP0RbpkXE5RSWVfPZuaFiw1v/qYauMcEs+OUwXlx5kDkrD7J45wk2HC4i2L/un+g7rkwkPNDX5JQizUcrHyLict5bf5SaWgd9491jqFhz8PW2MvPaLiz45VBSooIoLKsmq7ACfx8r943Qqoe0LiofIuJSqmx23lt/FKhb9bBYLCYncq4+cWEsnj6c+0cm4+tlZfqYzrQL9jM7lkiz0mkXEXEpC7fncOrsULHxvWLMjmMKfx8vHr++O7+9rhteVs8qX+IZtPIhIi7j20PF7hqa6LZDxZqLioe0Vp79f7aIuJQ1B06yP7+cQF8vfjwowew4ItJCVD5ExGW8dm6o2KB4QgM000KktVL5EBGXsD+/jNX7C7FY4J6hrf/yWhFP1qDyMXv2bAYNGkRwcDBRUVFMnDiRzMzMC77WMAzGjx+PxWLhs88+a46sItKKvXF21WNcjxgSIlrnUDERqdOg8pGenk5aWhrr169n2bJl2Gw2xo4dS0VFxfde+/zzz3vcJXIi0jgny6v5dFsOAFNHaNVDpLVr0KW2S5YsOe/Xb731FlFRUWzZsoWRI0fWP799+3aeeeYZNm/eTPv27ZsnqYi0Wu+vP0ZNrYPUuFAGJnrGUDERT9akOR8lJSUAhIeH1z9XWVnJz372M+bMmUNMzKWv0a+urqa6urr+16WlpU2JJCJupspm5931RwCYOiJZK6YiHqDRG04dDgczZsxg2LBh9OrVq/75Rx55hKFDh3LzzTdf1nFmz55NaGho/SM+Pr6xkUTEDS3akcvJ8hrah/p77FAxEU/T6JWPtLQ0du/ezdq1a+ufW7RoEStWrGDbtm2XfZxZs2Yxc+bM+l+XlpaqgIh4CMMweH1N3UbTu4cm4ePhQ8VEPEWj/k+fNm0aixcvZuXKlcTFxdU/v2LFCg4dOkRYWBje3t54e9d1m0mTJjF69OgLHsvPz4+QkJDzHiLiGdYePElmfhltfL34yRUaKibiKRq08mEYBtOnT2fBggWsWrWKjh3P35X+2GOPcd999533XO/evXnuueeYMGFC09OKSKtybpT6lIEaKibiSRpUPtLS0pg3bx4LFy4kODiYvLw8AEJDQwkICCAmJuaCm0wTEhK+V1RExLMdLChjVebZoWLDksyOIyJO1KDTLnPnzqWkpITRo0fTvn37+seHH37YUvlEpJV6fe0RAK7tHk1iRKC5YUTEqRp82qWhGvM1ItK6FVXU8OnW4wDcNyLZ5DQi4mzaWi4iTvf++qNU1zro3SGUQUkaKibiaVQ+RMSpqmvtvJ1xFID7RnTUUDERD6TyISJOtWh7LifLq4kJ8ef63rr9gognUvkQEacxDKP+8tq7NFRMxGPp/3wRcZqvD51iX14ZAT5e/ExDxUQ8lsqHiDjNuVWPWwfGEdpGQ8VEPJXKh4g4xcGCclbsKzg7VExDB0U8mcqHiDjFm+vqVj2u7hZNx0gNFRPxZCofItLiiipq+KR+qJhWPUQ8ncqHiLS4eRuOUmVz0DM2hMEdw82OIyImU/kQkRaloWIi8l0qHyLSohbvOEFhWTXRIX7c0DvW7Dgi4gJUPkSkxXx7qNidQ5Lw9dY/OSKi8iEiLSgj6xR7TpQS4OPFbYM1VExE6qh8iEiLeX1N3arHpAEdCGvja3IaEXEVKh8i0iKyCstZvq8AgHs1VExEvkXlQ0RaxBv1Q8WiSG4XZHIaEXElKh8i0uyKK2v4eEvdULGpGiomIt+h8iEize79Dceosjno0T6EIckRZscRERej8iEizaqm1sE7GUcAmDpcQ8VE5PtUPkSkWf17Vy75pdVEBfsxIVVDxUTk+1Q+RKTZGIbBa2vODRVL1FAxEbkg/csgIs1mw+Eivsktxd/Hys8GJ5odR0RclMqHiDSbc6set/SPIzxQQ8VE5MJUPkSkWRw+WcHyffmAhoqJyA9T+RCRZvHmusMYBozpFkVKlIaKicjFqXyISJOVVNqYv/nsULHhWvUQkR+m8iEiTTZv4zHO2Ox0iwlmaCcNFRORH6byISJNYrM7ePvrI4CGionI5VH5EJEm+c+uE+SVVhEZ5MdNfTVUTEQuTeVDRBrt20PF7hqSiJ+3l8mJRMQdqHyISKNtOnKaXTkl+Hlbue1KDRUTkcuj8iEijfbamixAQ8VEpGFUPkSkUY6crGDZ3rqhYlOHJ5kbRkTcisqHiDTKW18fwTBgdNd2pEQFmx1HRNyIyoeINFjJGRsfbc4GNFRMRBpO5UNEGuyDjceorLHTNTqY4SmRZscRETej8iEiDWKzO3jr3FCxERoqJiINp/IhIg3yxe48TpRUERnky02pGiomIg2n8iHSRHaHQVFFjdkxnKJuqFjd5bV3XJmEv4+GiolIw3mbHUDE3c36dCcfbT7OoKS23DEkiet6xuDr3Tp7/eajp9l5vARfbyu3XZlgdhwRcVMqHyJNsOt4CR+dvZX8piOn2XTkNO2C/fjpFQn87IoEYkL9TU7YvF4/O0r9ln4diAzyMzmNiLir1vnjmYiTPLV0HwDjekYz45rORAX7UVhWzT+WH2DYkytIe38r67NOYRiGyUmb7tipSpbuyQPgXl1eKyJNoJUPkUb6+tBJ1hw4iY+Xhf93fQ8SItqQdlUKS7/J452vj7LxSBH/3nWCf+86QdfoYO4YksiP+nUg0M89/7d78+vDGAaM7NKOLtEaKiYijeee/wqKmMwwDJ5akgnAT69IICGiDQA+XlZu7BPLjX1i2XuilHfXH2XB1hwy88v43We7efKLfUwaEMcdQxLp1C7IzD9Cg5RW2fhoU91Qsfu06iEiTdSg0y6zZ89m0KBBBAcHExUVxcSJE8nMzKz//aKiIqZPn07Xrl0JCAggISGBhx56iJKSkmYPLmKmL/fksz27mAAfL6aNSbnga7q3D+EvP+rN+sev5g839qBjZCBl1bW89fURrn4mndtf28CX3+Rhd7j+KZkPN2ZTUWOnS3QQIzprqJiINE2Dykd6ejppaWmsX7+eZcuWYbPZGDt2LBUVFQDk5uaSm5vL008/ze7du3nrrbdYsmQJU6dObZHwImawOwz+trSudE8d3pGo4B/eVBoa4MO9wzuyfOYo3rn3Cq7pHo3FAmsPnuT+d7cw8qmVzFl5kFPl1c6I32C1dgdvrqvbaDp1uIaKiUjTWYwm7IQrLCwkKiqK9PR0Ro4cecHXzJ8/n9tvv52Kigq8vS99lqe0tJTQ0FBKSkoICQlpbDSRFjN/czaPfryTsDY+rP7NVYT4+zT4GNlFlby/4RgfbjrG6UobAL5eVm7s0547hybRNz6smVM33uc7cpn+r21EBPqy7rExmu0hIhfUkM/vJu35OHc6JTw8/AdfExISctHiUV1dTXX1f3/iKy0tbUokkRZVXWvn+a8OAPDL0Z0aVTwA4sPb8Nj4bsy4pjOLd57g3Ywj7Dhewqfbcvh0Ww594kK548pEJqTGmvphbxgGr62tW/W4/cpEFQ8RaRaNvtTW4XAwY8YMhg0bRq9evS74mpMnT/LHP/6R+++//6LHmT17NqGhofWP+Pj4xkYSaXHvrz9GTvEZYkL8uXNIUpOP5+/jxeQBcSycNpzP0oZxS/8O+Hpb2Xm8hEc/3smVs5cz+4u9ZBdVNj18I2w9dpod2cX4elu5/cpEUzKISOvT6NMuv/jFL/jiiy9Yu3YtcXFx3/v90tJSrr32WsLDw1m0aBE+Phf+CfFCKx/x8fE67SIup7y6lpFPraSooobZt/Tmp1e0zITPoooaPtyUzXvrj5JTfAYAiwXGdI3izqFJjEiJxGp1zr6LX76/hf/symPKwDiempzqlPcUEffU4qddpk2bxuLFi1m9evUFi0dZWRnXXXcdwcHBLFiw4KLFA8DPzw8/P01KFNf32posiipqSI4M5NYB3/9731zCA335xehO3D8ymRX7Cngn4whrDpxk+b4Clu8rICmiDXcMSWLygDhCAxp32udyZBdVsmR33VCxqcOTW+x9RMTzNKh8GIbB9OnTWbBgAatWraJjx+9f719aWsq4cePw8/Nj0aJF+Pu3rvHS4plOlVfz6uq6G6r9amxXvL1afjiwl9XCtT2iubZHNIcKy3lv/VE+3nycI6cq+ePiPTy9NJOJ/WK548okesQ2/yrhm+uO4DBgROdIusZoqJiINJ8GlY+0tDTmzZvHwoULCQ4OJi+v7qei0NBQAgICKC0tZezYsVRWVvLee+9RWlpav4G0Xbt2eHlps5q4pzkrD1FRY6d3h1DG94px+vt3ahfEExN68uuxXflsew7vZhxlX14Z/9qYzb82Zjf7Te1Kq2x8tLluqNhUDRUTkWbWoD0fF7u+/8033+Tuu+9m1apVXHXVVRd8zeHDh0lKSrrke+hSW3E1OcVnuOpvq6ixO3h36hWM6NzO7EgYhsHGw0W8s/4oS3fnUXt2UNm5m9rdNjiB6JDGrzq+tiaLP/17LylRQSx7ZKRme4jIJbXYno9L9ZTRo0e3ihtoiXzb88v2U2N3MCQ5guEprjHd02KxMDg5gsHJEeSXVjFvwzHmbTxWf1O7l1YeZFzPGO4YksjgjuENKg91Q8WOABoqJiItQ/d2EfkBB/LL+GTrcQB+c11Xl/wgjg7x55Fru9Tf1O7djKbd1G7pN/nkFJ8hPNCXH/Xr4IQ/gYh4GpUPkR/w9JeZOAwY1zOafgltzY7zg3y9rUxIjWVCat1N7d7JOMpn2xp+U7vX19ZtrL19cIKGiolIi2jSePWWoD0f4iq2HTvNj176GqsFls4YSWc3vI18yRkbn2w5zrvrj3L4ZEX98yM6R3LHlYlc3T0ar2/NDNly9DST5n6Nr5eVtY9ddcn71oiInOO08eoirZVhGDy1pO7mcZP6x7ll8YD/3tTu7qFJrD14kncyjrB8XwFrDpxkzYGTdAgL4LYrE/jxwHgigvx44+wo9Zv6xqp4iEiLUfkQuYC1B0+SkXUKXy8rM67tYnacJrNaLYzs0o6RXdqdd1O7nOIzPLUkk+eXHeC6XjF8sfsEoMtrRaRltfykJBE343D8d9Xj9isT6RAWYHKi5nXupnYZs67m6VtT6RMXSo3dwaIduTgMGJYSQff2OuUpIi1HKx8i3/HF7jx25ZQQ6OtF2lWdzI7TYs7d1G7ygDi2ZxfzTsYRdh4v4TfjupkdTURaOZUPkW+x2R08/WXdqsfPRyYTEeQZ9x3qGx9G3/i+ZscQEQ+h0y4i3/LxluMcPllBeKAv943QzdRERFqCyofIWVU2O3//6gAAaVelEHQZA7lERKThVD5Eznon4wh5pVV1l58OTjA7johIq6XyIULdMK45Kw8BMOOazprsKSLSglQ+RIBXV2dRcsZG56ggbukfZ3YcEZFWTeVDPF5BWRWvn53s+etxXc8bNy4iIs1P5UM83pwVBzljs9M3PoyxPaLNjiMi0uqpfIhHO3aqknkbjwHwm+u6YrFo1UNEpKWpfIhHe+6r/djsBiM6RzK0U6TZcUREPILKh3isvSdK+Wx7DoBGiouIOJHKh3isp5dmYhhwQ5/29I4LNTuOiIjHUPkQj7T5SBHL9xXgZbXwq2u7mB1HRMSjqHyIxzEMgyeX7ANgysA4ktsFmZxIRMSzqHyIx1mVWcimI6fx87by0NWdzY4jIuJxVD7Eozgc/131uHtoEu1DA0xOJCLieVQ+xKN8vjOXfXllBPt784vRncyOIyLikVQ+xGPU1Dp45sv9ADw4qhNhbXxNTiQi4plUPsRjfLg5m2NFlUQG+XHPsCSz44iIeCyVD/EIlTW1/GP5AQAeujqFNr7eJicSEfFcKh/iEd5cd4TCsmriwwP4yaAEs+OIiHg0lQ9p9Yora/hn+iEAfnVtV3y99ddeRMRM+ldYWr256Ycoq6qlW0wwN6XGmh1HRMTjqXxIq5ZXUsVb644A8JvrumK1WswNJCIiKh/Suv1jxQGqax0MTGzLVV2jzI4jIiKofEgrdvhkBR9uygbgt+O7YbFo1UNExBWofEir9cyXmdgdBmO6RTEoKdzsOCIicpbKh7RKu3NKWLzzBBYLPDquq9lxRETkW1Q+pFV6amkmADenxtK9fYjJaURE5NtUPqTVyTh0itX7C/G2Wnjk2i5mxxERke9Q+ZBWxTAMnlq6D4CfXpFAYkSgyYlEROS7VD6kVVm2J59tx4oJ8PFi+pgUs+OIiMgFqHxIq2F3GPzt7F6Pe4cnERXib3IiERG5EJUPaTUWbMvhQEE5oQE+3D+yk9lxRETkIlQ+pFWorrXz3LL9APxydCdCA3xMTiQiIhej8iGtwrwNx8gpPkN0iB93DU0yO46IiPwAlQ9xe+XVtby44iAAD1/dBX8fL5MTiYjID2lQ+Zg9ezaDBg0iODiYqKgoJk6cSGZm5nmvqaqqIi0tjYiICIKCgpg0aRL5+fnNGlrk215fc5hTFTV0jAzk1oFxZscREZFLaFD5SE9PJy0tjfXr17Ns2TJsNhtjx46loqKi/jWPPPIIn3/+OfPnzyc9PZ3c3FxuueWWZg8uAnCqvJpX12QB8KuxXfDx0mKeiIirsxiGYTT2iwsLC4mKiiI9PZ2RI0dSUlJCu3btmDdvHpMnTwZg3759dO/enYyMDK688spLHrO0tJTQ0FBKSkoICdFYbPlhf1y8h9fXHqZXhxAWpQ3HatWda0VEzNCQz+8m/ZhYUlICQHh43R1Dt2zZgs1m45prrql/Tbdu3UhISCAjI+OCx6iurqa0tPS8h8jlyCk+w7vrjwLw6LhuKh4iIm6i0eXD4XAwY8YMhg0bRq9evQDIy8vD19eXsLCw814bHR1NXl7eBY8ze/ZsQkND6x/x8fGNjSQe5u9f7aem1sGVyeGM7BxpdhwREblMjS4faWlp7N69mw8++KBJAWbNmkVJSUn9Izs7u0nHE89wsKCMj7ccB+A313XDYtGqh4iIu/BuzBdNmzaNxYsXs3r1auLi/nt1QUxMDDU1NRQXF5+3+pGfn09MTMwFj+Xn54efn19jYogHe3rpfhwGjO0RTf+EtmbHERGRBmjQyodhGEybNo0FCxawYsUKOnbseN7vDxgwAB8fH5YvX17/XGZmJseOHWPIkCHNk1g83vbsYpZ8k4fVAr8e19XsOCIi0kANWvlIS0tj3rx5LFy4kODg4Pp9HKGhoQQEBBAaGsrUqVOZOXMm4eHhhISEMH36dIYMGXJZV7qIXI6/Ld0HwC394+gSHWxyGhERaagGlY+5c+cCMHr06POef/PNN7n77rsBeO6557BarUyaNInq6mrGjRvHSy+91CxhRdYeOMm6g6fw9bIy45rOZscREZFGaNKcj5agOR9yMYZhcNOL69iVU8I9w5J4YkJPsyOJiMhZTpvzIeJMX+zOY1dOCYG+XqRdlWJ2HBERaSSVD3ELtXYHTy+tu4/QfSOSiQzSFVIiIu5K5UPcwsdbjpN1soLwQF/uG9Hx0l8gIiIuS+VDXF6Vzc7flx8A4JejOxHs72NyIhERaQqVD3F572Yc5URJFbGh/tx+ZaLZcUREpIlUPsSllVbZmLPqIAAzru2Cv4+XyYlERKSpVD7Epb26OoviShspUUHc0q+D2XFERKQZqHyIyyosq+b1tYcB+PXYrnh76a+riEhr4FH/ms/8aDvPLtvPqfJqs6PIZZiz8iCVNXZS48MY1zPa7DgiItJMGnVXW3d0IL+MT7fmAPBy+iGmDIznvhEdSYwINDmZXEh2USXvbzgKwG/HdcVisZicSEREmovHrHx0jAzkxZ/1o3eHUKprHby7/ihXPb2KtHlb2Xm82Ox48h3PLduPzW4wonMkQ1MizY4jIiLNyOPu7WIYBhlZp3g5PYv0/YX1zw9JjuCBUcmM6tJOP2WbbF9eKeP/vgbDgEXThtEnLszsSCIicgkN+fz2mNMu51gsFoZ2imRop0j2nijlldVZfL4jl4ysU2RknaJbTDAPjErmxj6x+GiDoymeXpqJYcANvdureIiItEIet/JxITnFZ3hj7WH+tfEYlTV2AGJD/bl3eEd+ckUCQX4e19FMs+VoEZPmZuBltfDlIyPp1C7I7EgiInIZGvL5rfLxLSWVNt7bcJQ31x3h5NkrYkL8vbljSCJ3DU0iKtjfqXk8jWEY/Pjl9Ww8UsRPBsXz10l9zI4kIiKXSeWjiapsdhZsy+GV1VkcPlkBgK+3lUn94/j5iI4k66fxFrEys4B73tyEr7eV9EdH0z40wOxIIiJymVQ+mondYbBsTz4vrz7EtmPFAFgsMLZHNA+M6kT/hLam5mtNHA6DG15Yy94Tpdw/MpnHr+9udiQREWkAbThtJl5WC9f1imFcz2g2Hz3Ny+mH+GpvAUu/yWfpN/lckRTO/SOTGdMtCqtVV8g0xec7c9l7opRgP29+MaqT2XFERKQFqXxcBovFwqCkcAYlhXMgv4xXVmfx2fYcNh4pYuORIlKigrh/ZDI3943Fz1s3Pmsom93Bs8v2A/DAqGTaBvqanEhERFqSTrs0Ul5JFW9+fZh5649RVl0LQHSIH/cM68jPBicQ4u9jckL38d76o/zus91EBvmS/uhVBOrqIhERt6M9H05UWmXjXxuO8ca6w+SX1l0hE+TnzW2DE7hnWEdiQnWFzA85U2Nn5N9WUlhWzf/e1JO7hiaZHUlERBpB5cME1bV2Fm7P5dXVWRwoKAfAx8vCxL4duH9kMp2jg01O6JpeWnWQp5ZkEtc2gBW/Go2vtwa7iYi4I5UPEzkcBiszC3g5PYuNR4rqn7+6WxQPjOrEoKS2Gt9+VkmljRFPraC0qpbnfpzKj/rFmR1JREQaSVe7mMhqtXB192iu7h7NlqOneWX1Ib7ck8/yfQUs31dAv4QwHhjZiWt7ROPlYVfI2B0G2UWVHCwo50BBOav3F1JaVUu3mGBuSu1gdjwREXESrXw4QVZhOa+uOcwnW49TU+sAIDkykPtGJHNL/w74+7SuK2RsdgdHT1VwIL+uZBwoKOdgQTmHCsvr//zf9ubdg7iqW5QJSUVEpLnotIuLKiir4u2vj/BuxlFKq+qukIkM8uOeYUncPjiR0DbudYVMlc1OVmEFBwrKOHi2YBwoKOfIyQpqHRf+a+XnbaVTuyA6RweR0i6IoSkRDEgMd3JyERFpbiofLq68upYPN2Xz+posckuqAGjj68VPr0jg3uEd6RDmWmPFK6pr64tFXcko40BBOdlFlVykYxDo60VKdDApZ4tG56ggUqKCiGvbxuNON4mIeAKVDzdhsztYvDOXl9Oz2JdXBoC31cKE1FjuH5lM9/bOv7HewcKy806XHCooJ6f4zEW/JjTAh85RZ1cyooJJiaorGu1D/bWxVkTEg6h8uBnDMFh94CQvpx/i60On6p8f1aUdD4xKZkhyRLN9kBuGwamKGg7k161gnFvROFBQTmFZ9UW/LjLIr371oq5oBNE5KpjIIF+VDBERUflwZzuPF/Py6iy+2HWi/pRG7w6hPDAqmet6xuDtdXlzMAzDIK+06mzJOHfKpO50SXGl7aJfFxvqT6ezxeLbp0vC2mjkuYiIXJzKRytw7FQlr63N4qPN2VTZ6q4QSQhvw89HdGTygHgCfOuukHE4DI6fPnPB0yXlZ8e+f5fFAvFt29QVi7MbPztHB9OpXSDBGgsvIiKNoPLRipwqr+adjKO8k3GE02dXLMIDfRmSHMGRUxUcKiyvLyff5WW1kBTRhs7n9mKcPV3SqV1Qq7u8V0REzKXy0QpV1tQyf/NxXlubRXbR+RtAfb2sJLcLrN+Hca5kJEUEaly5iIg4hcpHK1Zrd/DlnnyOFVWSHBlI5+hg4tsGXPZeEBERkZag8eqtmLeXlet7tzc7hoiISKPpx2URERFxKpUPERERcSqVDxEREXEqlQ8RERFxKpUPERERcSqVDxEREXEqlQ8RERFxKpUPERERcSqVDxEREXEqlQ8RERFxqgaXj9WrVzNhwgRiY2OxWCx89tln5/1+eXk506ZNIy4ujoCAAHr06ME///nP5sorIiIibq7B5aOiooLU1FTmzJlzwd+fOXMmS5Ys4b333mPv3r3MmDGDadOmsWjRoiaHFREREffX4BvLjR8/nvHjx1/097/++mvuuusuRo8eDcD999/Pyy+/zMaNG7npppsaHVRERERah2a/q+3QoUNZtGgR9957L7GxsaxatYr9+/fz3HPPXfD11dXVVFdX1/+6pKQEqLs1r4iIiLiHc5/bhmFc+sVGEwDGggULznuuqqrKuPPOOw3A8Pb2Nnx9fY233377osd44oknDEAPPfTQQw899GgFj+zs7Ev2h2Zf+XjhhRdYv349ixYtIjExkdWrV5OWlkZsbCzXXHPN914/a9YsZs6cWf9rh8NBUVERERERWCyWZs1WWlpKfHw82dnZhISENOuxpeH0/XAt+n64Hn1PXIu+Hz/MMAzKysqIjY295GubtXycOXOGxx9/nAULFnDDDTcA0KdPH7Zv387TTz99wfLh5+eHn5/fec+FhYU1Z6zvCQkJ0V8cF6Lvh2vR98P16HviWvT9uLjQ0NDLel2zzvmw2WzYbDas1vMP6+XlhcPhaM63EhERETfV4JWP8vJyDh48WP/rw4cPs337dsLDw0lISGDUqFE8+uijBAQEkJiYSHp6Ou+88w7PPvtsswYXERER99Tg8rF582auuuqq+l+f269x11138dZbb/HBBx8wa9YsbrvtNoqKikhMTOTPf/4zDz74YPOlbiQ/Pz+eeOKJ753mEXPo++Fa9P1wPfqeuBZ9P5qPxbisa2JEREREmofu7SIiIiJOpfIhIiIiTqXyISIiIk6l8iEiIiJO5VHlY86cOSQlJeHv78/gwYPZuHGj2ZE80uzZsxk0aBDBwcFERUUxceJEMjMzzY4lZ/31r3/FYrEwY8YMs6N4rJycHG6//XYiIiIICAigd+/ebN682exYHslut/P73/+ejh07EhAQQKdOnfjjH/94efcvkYvymPLx4YcfMnPmTJ544gm2bt1Kamoq48aNo6CgwOxoHic9PZ20tDTWr1/PsmXLsNlsjB07loqKCrOjebxNmzbx8ssv06dPH7OjeKzTp08zbNgwfHx8+OKLL9izZw/PPPMMbdu2NTuaR3ryySeZO3cuL774Inv37uXJJ5/kqaee4oUXXjA7mlvzmEttBw8ezKBBg3jxxReBunvIxMfHM336dB577DGT03m2wsJCoqKiSE9PZ+TIkWbH8Vjl5eX079+fl156iT/96U/07duX559/3uxYHuexxx5j3bp1rFmzxuwoAtx4441ER0fz+uuv1z83adIkAgICeO+990xM5t48YuWjpqaGLVu2nHdvGavVyjXXXENGRoaJyQSgpKQEgPDwcJOTeLa0tDRuuOGGC96DSZxn0aJFDBw4kFtvvZWoqCj69evHq6++anYsjzV06FCWL1/O/v37AdixYwdr165l/PjxJidzb81+V1tXdPLkSex2O9HR0ec9Hx0dzb59+0xKJVC3AjVjxgyGDRtGr169zI7jsT744AO2bt3Kpk2bzI7i8bKyspg7dy4zZ87k8ccfZ9OmTTz00EP4+vpy1113mR3P4zz22GOUlpbSrVs3vLy8sNvt/PnPf+a2224zO5pb84jyIa4rLS2N3bt3s3btWrOjeKzs7Gwefvhhli1bhr+/v9lxPJ7D4WDgwIH85S9/AaBfv37s3r2bf/7znyofJvjoo494//33mTdvHj179mT79u3MmDGD2NhYfT+awCPKR2RkJF5eXuTn55/3fH5+PjExMSalkmnTprF48WJWr15NXFyc2XE81pYtWygoKKB///71z9ntdlavXs2LL75IdXU1Xl5eJib0LO3bt6dHjx7nPde9e3c++eQTkxJ5tkcffZTHHnuMn/zkJwD07t2bo0ePMnv2bJWPJvCIPR++vr4MGDCA5cuX1z/ncDhYvnw5Q4YMMTGZZzIMg2nTprFgwQJWrFhBx44dzY7k0a6++mp27drF9u3b6x8DBw7ktttuY/v27SoeTjZs2LDvXXq+f/9+EhMTTUrk2SorK7Faz/+o9PLywuFwmJSodfCIlQ+ou/vuXXfdxcCBA7niiit4/vnnqaio4J577jE7msdJS0tj3rx5LFy4kODgYPLy8gAIDQ0lICDA5HSeJzg4+Hv7bQIDA4mIiNA+HBM88sgjDB06lL/85S9MmTKFjRs38sorr/DKK6+YHc0jTZgwgT//+c8kJCTQs2dPtm3bxrPPPsu9995rdjT3ZniQF154wUhISDB8fX2NK664wli/fr3ZkTwScMHHm2++aXY0OWvUqFHGww8/bHYMj/X5558bvXr1Mvz8/Ixu3boZr7zyitmRPFZpaanx8MMPGwkJCYa/v7+RnJxs/L//9/+M6upqs6O5NY+Z8yEiIiKuwSP2fIiIiIjrUPkQERERp1L5EBEREadS+RARERGnUvkQERERp1L5EBEREadS+RARERGnUvkQERERp1L5EBEREadS+RARERGnUvkQERERp1L5EBEREaf6/2dRUK5ytuRLAAAAAElFTkSuQmCC" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(all_rewards)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:00:12.198836Z", + "start_time": "2023-12-13T18:00:12.058221Z" + } + }, + "id": "457bbd9d81cac391" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/gymnasium/local-training.py b/examples/gymnasium/local-training.py new file mode 100644 index 0000000..2163e2e --- /dev/null +++ b/examples/gymnasium/local-training.py @@ -0,0 +1,114 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import datetime + +from cogment_lab.envs.gymnasium import GymEnvironment +from cogment_lab.process_manager import Cogment +from cogment_lab.utils.coltra_utils import convert_trial_data_to_coltra +from cogment_lab.utils.runners import process_cleanup +from cogment_lab.utils.trial_utils import concatenate + +from coltra import HomogeneousGroup +from coltra.models import MLPModel +from coltra.policy_optimization import CrowdPPOptimizer + +from cogment_lab.actors import ColtraActor + +from tqdm import trange +import matplotlib.pyplot as plt + + +async def main(): + process_cleanup() + + logpath = f"logs/logs-{datetime.datetime.now().isoformat()}" + + cog = Cogment(log_dir=logpath) + + print(logpath) + + # We'll train on CartPole-v1 + + cenv = GymEnvironment( + env_id="CartPole-v1", + render=False, + ) + + await cog.run_env(env=cenv, env_name="cartpole", port=9001, log_file="env.log") + + print("Env started") + + # Create a model using coltra + + model = MLPModel( + config={ + "hidden_sizes": [64, 64], + }, + observation_space=cenv.env.observation_space, + action_space=cenv.env.action_space, + ) + + actor = ColtraActor(model=model) + + actor_task = cog.run_local_actor(actor=actor, actor_name="coltra", port=9021, log_file="actor.log") + + print("Actor started") + + ppo = CrowdPPOptimizer( + HomogeneousGroup(actor.agent), + config={ + "gae_lambda": 0.95, + "minibatch_size": 128, + }, + ) + + all_rewards = [] + + for t in (pbar := trange(100)): + num_steps = 0 + episodes = [] + while num_steps < 1000: # Collect at least 1000 steps per training iteration + trial_id = await cog.start_trial( + env_name="cartpole", + session_config={"render": False}, + actor_impls={ + "gym": "coltra", + }, + ) + multi_data = await cog.get_trial_data(trial_id=trial_id) + data = multi_data["gym"] + episodes.append(data) + num_steps += len(data.rewards) + + all_data = concatenate(episodes) + + # Preprocess data + + record = convert_trial_data_to_coltra(all_data, actor.agent) + + # Run a PPO step + metrics = ppo.train_on_data({"crowd": record}, shape=(1, len(record.reward))) + + mean_reward = metrics["crowd/mean_episode_reward"] + all_rewards.append(mean_reward) + pbar.set_description(f"mean_reward: {mean_reward:.3}") + + plt.plot(all_rewards) + plt.show() + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/gymnasium/observe.ipynb b/examples/gymnasium/observe.ipynb new file mode 100644 index 0000000..d0ff154 --- /dev/null +++ b/examples/gymnasium/observe.ipynb @@ -0,0 +1,293 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T18:23:08.799669Z", + "start_time": "2023-12-13T18:23:07.078180Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "from cogment_lab.actors import RandomActor, ConstantActor\n", + "from cogment_lab.envs.pettingzoo import AECEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.runners import process_cleanup\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:09.292376Z", + "start_time": "2023-12-13T18:23:09.234508Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:09.470752Z", + "start_time": "2023-12-13T18:23:09.465578Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T19:23:09.453551\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:11.879060Z", + "start_time": "2023-12-13T18:23:09.667673Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch an environment in a subprocess\n", + "\n", + "cenv = AECEnvironment(env_path=\"cogment_lab.envs.conversions.observer.GymObserverAECAEC\",\n", + " make_kwargs={\"gym_env_name\": \"LunarLander-v2\"},\n", + " render=True)\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"lunar\",\n", + " port=9011, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:16.261481Z", + "start_time": "2023-12-13T18:23:11.876240Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch two dummy actors in subprocesses\n", + "\n", + "random_actor = RandomActor(cenv.env.action_space(\"gym\"))\n", + "constant_actor = ConstantActor(0)\n", + "\n", + "await cog.run_actor(actor=random_actor, \n", + " actor_name=\"random\", \n", + " port=9021, \n", + " log_file=\"actor-random.log\")\n", + "\n", + "await cog.run_actor(actor=constant_actor,\n", + " actor_name=\"constant\",\n", + " port=9022,\n", + " log_file=\"actor-constant.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "outputs": [ + { + "data": { + "text/plain": "{'lunar': ,\n 'random': ,\n 'constant': }" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:16.267399Z", + "start_time": "2023-12-13T18:23:16.260197Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:23.468397Z", + "start_time": "2023-12-13T18:23:21.316245Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "LUNAR_LANDER_ACTIONS = [\"no-op\", \"ArrowRight\", \"ArrowUp\", \"ArrowLeft\"]\n", + "\n", + "# Change this if you use a different environment. Only discrete actions are supported for now.\n", + "\n", + "actions = LUNAR_LANDER_ACTIONS\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=60)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "outputs": [], + "source": [ + "# Get data from a random + random trial\n", + "# You can change the values in `actor_impls` between `web_ui`, `random`, and `constant` to see the different behaviors\n", + "\n", + "trial_id = await cog.start_trial(\n", + " env_name=\"lunar\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"gym\": \"random\",\n", + " \"observer\": \"web_ui\",\n", + " },\n", + ")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:24.017807Z", + "start_time": "2023-12-13T18:23:24.005380Z" + } + }, + "id": "efef1ac3ff97fe90" + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [], + "source": [ + "data = await cog.get_trial_data(trial_id)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:23:35.523005Z", + "start_time": "2023-12-13T18:23:26.560614Z" + } + }, + "id": "8052ff03998b0b52" + }, + { + "cell_type": "code", + "execution_count": 13, + "outputs": [], + "source": [ + "await cog.cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:22:59.869879Z", + "start_time": "2023-12-13T18:22:58.827007Z" + } + }, + "id": "9e64d0d548ac34ef" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/gymnasium/simple.py b/examples/gymnasium/simple.py new file mode 100644 index 0000000..1b5660f --- /dev/null +++ b/examples/gymnasium/simple.py @@ -0,0 +1,65 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import datetime + +from cogment_lab.actors import ConstantActor +from cogment_lab.envs.gymnasium import GymEnvironment +from cogment_lab.process_manager import Cogment + + +async def main(): + logpath = f"logs/logs-{datetime.datetime.now().isoformat()}" + + cog = Cogment(log_dir=logpath) + + print(logpath) + + cenv = GymEnvironment( + env_id="MountainCar-v0", + render=True, + make_kwargs={"max_episode_steps": 10}, + ) + + await cog.run_env(env=cenv, env_name="mcar", port=9011, log_file="env.log") + + # Create a model using coltra + + constant_actor = ConstantActor(1) + + await cog.run_actor(actor=constant_actor, actor_name="constant", port=9022, log_file="actor-constant.log") + + # Estimate random agent performance + + trial_id = await cog.start_trial( + env_name="mcar", + session_config={"render": False}, + actor_impls={ + "gym": "constant", + }, + ) + multi_data = await cog.get_trial_data(trial_id=trial_id) + data = multi_data["gym"] + + # mean_reward = np.mean([sum(e.rewards) for e in episodes]) + print(f"Reward shape: {data.rewards.shape}") + print(f"Rewards: {data.rewards}") + print(f"Observations: {data.observations}") + print(f"Last observation: {data.last_observation}") + print(f"Actions: {data.actions}") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/gymnasium/training.ipynb b/examples/gymnasium/training.ipynb new file mode 100644 index 0000000..97e9582 --- /dev/null +++ b/examples/gymnasium/training.ipynb @@ -0,0 +1,328 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T18:24:37.254269Z", + "start_time": "2023-12-13T18:24:35.182905Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "from cogment_lab.envs.gymnasium import GymEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.coltra_utils import convert_trial_data_to_coltra\n", + "from cogment_lab.utils.runners import process_cleanup\n", + "from cogment_lab.utils.trial_utils import concatenate\n", + "\n", + "from coltra import HomogeneousGroup\n", + "from coltra.buffers import Observation\n", + "from coltra.models import MLPModel\n", + "from coltra.policy_optimization import CrowdPPOptimizer\n", + "\n", + "from cogment_lab.actors import ColtraActor\n", + "\n", + "from tqdm import trange\n", + "import matplotlib.pyplot as plt\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:24:39.351501Z", + "start_time": "2023-12-13T18:24:39.276395Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:24:40.457060Z", + "start_time": "2023-12-13T18:24:40.454080Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T19:24:40.452142\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:24:52.181519Z", + "start_time": "2023-12-13T18:24:49.916590Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# We'll train on CartPole-v1\n", + "\n", + "cenv = GymEnvironment(\n", + " env_id=\"CartPole-v1\",\n", + " render=False,\n", + ")\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"cartpole\",\n", + " port=9001, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:24:56.781677Z", + "start_time": "2023-12-13T18:24:54.489069Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Create a model using coltra\n", + "\n", + "model = MLPModel(\n", + " config={\n", + " \"hidden_sizes\": [64, 64],\n", + " }, \n", + " observation_space=cenv.env.observation_space, \n", + " action_space=cenv.env.action_space\n", + ")\n", + "\n", + "# Put the model in shared memory so that the actor can access it\n", + "model.share_memory()\n", + "actor = ColtraActor(model=model)\n", + "\n", + "\n", + "await cog.run_actor(\n", + " actor=actor,\n", + " actor_name=\"coltra\",\n", + " port=9021,\n", + " log_file=\"actor.log\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "outputs": [ + { + "data": { + "text/plain": "{'cartpole': ,\n 'coltra': }" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:24:56.786229Z", + "start_time": "2023-12-13T18:24:56.781215Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 7, + "outputs": [], + "source": [ + "ppo = CrowdPPOptimizer(HomogeneousGroup(actor.agent), config={\n", + " \"gae_lambda\": 0.95,\n", + " \"minibatch_size\": 128,\n", + "})" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:24:58.339199Z", + "start_time": "2023-12-13T18:24:58.171380Z" + } + }, + "id": "582b6bb1bf0c81df" + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "mean_reward: 27.6: 100%|██████████| 10/10 [00:15<00:00, 1.54s/it]\n" + ] + } + ], + "source": [ + "all_rewards = []\n", + "\n", + "for t in (pbar := trange(10)):\n", + " num_steps = 0\n", + " episodes = []\n", + " while num_steps < 1000: # Collect at least 1000 steps per training iteration\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"cartpole\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id, env_name=\"cartpole\")\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + " num_steps += len(data.rewards)\n", + " \n", + " all_data = concatenate(episodes)\n", + "\n", + " # Preprocess data\n", + " record = convert_trial_data_to_coltra(all_data, actor.agent)\n", + "\n", + " # Run a PPO step\n", + " metrics = ppo.train_on_data({\"crowd\": record}, shape=(1,) + record.reward.shape)\n", + " \n", + " mean_reward = metrics[\"crowd/mean_episode_reward\"]\n", + " all_rewards.append(mean_reward)\n", + " pbar.set_description(f\"mean_reward: {mean_reward:.3}\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:25:49.058929Z", + "start_time": "2023-12-13T18:25:33.652328Z" + } + }, + "id": "56b220d45561a042" + }, + { + "cell_type": "code", + "execution_count": 10, + "outputs": [ + { + "data": { + "text/plain": "[]" + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGiCAYAAABH4aTnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9sUlEQVR4nO3deXxU5cH28Wsm+w7ZdwiEfQsQdgRUFhUVqtW2rijWpQFFrD5q3762r1Vqq9WnLqhVQasUt0IQF0QRMLIvASL7HgghAZJMSMg2c94/ArFUQLLNmeX3/XzmD04mJxdMkrk4933u22IYhiEAAAAnsZodAAAAeBfKBwAAcCrKBwAAcCrKBwAAcCrKBwAAcCrKBwAAcCrKBwAAcCrKBwAAcCrKBwAAcCrKBwAAcKpGlY8ZM2ZowIABCgsLU2xsrCZOnKgdO3ac9ZzCwkLdeuutio+PV0hIiPr166ePP/64RUMDAAD31ajysWzZMmVlZWnVqlVavHixamtrNXbsWFVUVDQ857bbbtOOHTu0YMECbdmyRdddd51uvPFGbdy4scXDAwAA92NpzsZyxcXFio2N1bJlyzRixAhJUmhoqGbOnKlbb7214XlRUVF65plndNdddzU/MQAAcGu+zfnksrIySVJkZGTDsaFDh+r999/X+PHj1aZNG33wwQeqqqrSqFGjznmO6upqVVdXN/zZ4XDoxIkTioqKksViaU48AADgJIZhqLy8XImJibJaf2JgxWgiu91ujB8/3hg2bNhZx0tKSoyxY8cakgxfX18jPDzcWLRo0XnP88QTTxiSePDgwYMHDx4e8MjPz//JDtHkYZf77rtPn3/+uXJycpScnNxwfOrUqVqzZo2efvppRUdHa/78+Xr++ef17bffqlevXj86z39f+SgrK1Nqaqry8/MVHh7elGgAAMDJbDabUlJSVFpaqoiIiAs+t0nlY8qUKcrOztby5cuVlpbWcHzPnj1KT09XXl6eevTo0XB89OjRSk9P16uvvnpR4SMiIlRWVkb5AADATTTm/btRcz4Mw9DUqVM1b948LV269KziIUmVlZWS9KOxHh8fHzkcjsZ8KQAA4KEaVT6ysrI0Z84cZWdnKywsTIWFhZKkiIgIBQUFqWvXrkpPT9c999yjZ599VlFRUZo/f74WL16shQsXtspfAAAAuJdGDbuc7+6TWbNmadKkSZKkXbt26dFHH1VOTo5Onjyp9PR0/fa3vz3r1tsLYdgFAAD305j372at89EaKB8AALifxrx/s7cLAABwKsoHAABwKsoHAABwKsoHAABwKsoHAABwKsoHAABwKsoHAABwKsoHAABwqkYtrw4AANzXkbJTenvFAUWG+OnuER1Ny0H5AADAw20+VKo3c/bp081HVOcwFBnir9uGtFegn48peSgfAAB4ILvD0OKtR/VWzj6t2X+i4figtEjddUkH+fmYN/OC8gEAgAepqK7Th+vy9dZ3+3XwRKUkyddq0TV9EjV5eJp6JkWYnJDyAQCARygoPaW3V+zXnDUHVV5VJ0mKCPLTzYNSdduQ9oqPCDQ54Q8oHwAAuLFN+aV6I2efPttyRHZH/Ub1adEhunN4mq7vl6Rgf9d7q3e9RAAA4ILq53MU6s2cfVq7v6Th+JAOUbrrkjRd2iVWVqvFxIQXRvkAAMBNnKyu0wdr8zVrxT7lnzglSfLz+WE+R49E8+dzXAzKBwAALu5w6SnN/m6f5q7JV3l1/XyONsE/zOeIC3ed+RwXg/IBAICL2niwRG/m7NPneYUN8zk6NMznSFaQvznrdDQX5QMAABdSZ3foy61H9WbOPq0/8MN8jqEd6+dzjOrs2vM5LgblAwAAF1BeVav31+Zr9or9OlTyw3yOa/skafLwNHVPDDc5YcuhfAAAYKJDJZWa/d1+zV2br5On53O0DfbTLYPb6dbB7RTrZvM5LgblAwAAE2w4WKI3v92nz/OO6PR0DnWMCdHk4R30s75Jbjuf42JQPgAAcJI6u0OLvj+qN3L2auPB0objw9OjNfmSNI3sFOP28zkuBuUDAIBWZquqrV+f47v9OlxaP5/D38eqCRmJunN4mroleM58jotB+QAAoJXkn6jUrO/264N1P8zniAzx1y2D2+mWwamKDfO8+RwXg/IBAEALMgxDGw6W6I1v92nR94UN8znSY0N11/A0TeybpEA/z53PcTEoHwAAtIA6u0Of59Xvt5KbX9pw/JJO0Zo8PE0jO8fIYvH8+RwXg/IBAEAzlJ2q1ftrD+rtFQd+mM/ha9XPMpJ05/A0dYkPMzmh66F8AADQBAePV2rWin36YG2+KmrskqSohvkc7RQTFmByQtdF+QAA4CIZhqH1B+rnc3y59Yf5HJ1iQ3XXJWmakMF8jotB+QAA4CcYhqFPtxzRP5bv1aZDZQ3HR3SO0V3D03RJp2jmczQC5QMAgJ/wl0U7NHPpHkn18zmu61s/n6NzHPM5moLyAQDABfxj+d6G4nHfqI6aPDxN0aHM52gOygcAAOfx0fpDeuqzbZKkR67oot+MSjc5kWewmh0AAABXtHjrUf3Px5slSb++JE33jexociLPQfkAAOC/rN57XFlzNsjuMHR9v2Q9flU3JpS2IMoHAAD/4fuCMt319jrV1Dk0ulucnrm+F8WjhVE+AAA4bf+xCt3+1hqVV9dpYFqkXrqpr3x9eKtsafyLAgAg6aitSre8uVrHTtaoW0K43rg9kwXDWgnlAwDg9coqa3Xbm2t0qOSU2kUF6507Byo80M/sWB6L8gEA8Gqnauya/PZa7ThartiwAL07eRD7srQyygcAwGvV2h36zXvrte5AicIDffXO5IFKiQw2O5bHo3wAALySw2Ho4Q836ZsdxQr0s+qtSQPUNT7c7FhegfIBAPA6hmHo/y3cqvm5BfK1WjTz5v7KbB9pdiyvQfkAAHidl5bs1uwV+yVJz97QR5d2jTU3kJehfAAAvMq7qw7oucU7JUlPXNNdE/smmZzI+1A+AABeY+HmAv0+O0+SdP9l6bpjWJrJibwT5QMA4BW+3VWsB9/PlWFINw9K1YNjOpsdyWtRPgAAHm/jwRLd88/1qrUbGt87Qf9vQk/2azER5QMA4NF2F5XrjtlrVVlj1yWdovX8jRnysVI8zET5AAB4rMOlp3Trm2tUWlmrPilt9Oot/eXvy1uf2XgFAAAe6fjJat365modKatSemyoZk8aoJAAX7NjQZQPAIAHOlldpztmr9Xe4goltQnSPycPVNsQf7Nj4TTKBwDAo1TX2XX3O+u0+VCZIkP89c7kgUqICDI7Fv4D5QMA4DHsDkMP/CtXK/YcV4i/j2bfMUAdY0LNjoX/QvkAAHgEwzD0f+Zv0RffF8rfx6p/3Jap3sltzI6Fc6B8AAA8wl8X7dC/1uTLapH+/qsMDU2PNjsSzoPyAQBwe298u1evLN0jSXrqZ710Rc8EkxPhQigfAAC39vH6Q/rTp9skSY9c0UW/GphqciL8FMoHAMBtfbX1qB75eLMk6a7habpvZEeTE+FiUD4AAG5p9d7jypqzQXaHoev7Jevxq7qxX4uboHwAANzO9wVluuvtdaquc2h0t1g9c30vWdmvxW1QPgAAbuXA8Qrd/tZalVfXaWD7SL10Uz/5+vB25k54tQAAbqPIVqVb3lytYyer1S0hXG9MylSgn4/ZsdBIlA8AgFsoq6zVbW+tUf6JU2oXFay37xyg8EA/s2OhCSgfAACXd6rGrslvr9X2wnLFhAXon3cOUmxYoNmx0ESUDwCAS6u1O/Sb99Zr3YEShQf66p07Byo1KtjsWGgGygcAwGU5HIYe+WizvtlRrEA/q96aNEDdEsLNjoVmonwAAFySYRh68tOtmrfxsHytFs28ub8y20eaHQstgPIBAHBJL3+zW7O+2y9JevaGPrq0a6y5gdBiKB8AAJfz7qoDevbLnZKkJ67prol9k0xOhJbUqPIxY8YMDRgwQGFhYYqNjdXEiRO1Y8eOHz1v5cqVuuyyyxQSEqLw8HCNGDFCp06darHQAADP9enmI/p9dp4kaepl6bpjWJrJidDSGlU+li1bpqysLK1atUqLFy9WbW2txo4dq4qKiobnrFy5UldccYXGjh2rNWvWaO3atZoyZYqsVi6yAAAu7NtdxZr2/kYZhnTzoFRNH9PZ7EhoBRbDMIymfnJxcbFiY2O1bNkyjRgxQpI0ePBgjRkzRk8++WSTzmmz2RQREaGysjKFhzOjGQC8RW5+qW76xypV1tg1vneC/v7LvvJhvxa30Zj372ZdjigrK5MkRUbWzz4uKirS6tWrFRsbq6FDhyouLk4jR45UTk7Oec9RXV0tm8121gMA4F12F5Vr0qw1qqyx65JO0frbjX0oHh6syeXD4XBo2rRpGjZsmHr27ClJ2rt3ryTpD3/4g37961/riy++UL9+/XT55Zdr165d5zzPjBkzFBER0fBISUlpaiQAgBs6XHpKt765RqWVteqT0kav3tJfAb7s1+LJmlw+srKylJeXp7lz5zYcczgckqR77rlHd9xxh/r27avnn39eXbp00VtvvXXO8zz22GMqKytreOTn5zc1EgDAzRw/Wa1b31ytI2VVSo8N1axJAxQS4Gt2LLSyJr3CU6ZM0cKFC7V8+XIlJyc3HE9ISJAkde/e/aznd+vWTQcPHjznuQICAhQQENCUGAAAN3ayuk53zF6rvcUVSowI1Dt3DlRkiL/ZseAEjbryYRiGpkyZonnz5mnJkiVKSzv79qf27dsrMTHxR7ff7ty5U+3atWt+WgCAR6ius+vud9Zp86EyRYb4653Jg5TYJsjsWHCSRl35yMrK0pw5c5Sdna2wsDAVFhZKkiIiIhQUFCSLxaKHH35YTzzxhPr06aOMjAy9/fbb2r59uz766KNW+QsAANyL3WFo2txcrdhzXCH+Ppp9xwClx4aaHQtO1KjyMXPmTEnSqFGjzjo+a9YsTZo0SZI0bdo0VVVV6cEHH9SJEyfUp08fLV68WB07dmyRwAAA92UYhv7P/C36PK9Q/j5WvX5bpnontzE7FpysWet8tAbW+QAAz/XXRdv18jd7ZLVIL9/UT1f2SjA7ElqI09b5AADgYr3x7V69/M0eSdJTP+tF8fBilA8AQKv7eP0h/enTbZKkh8d10a8GppqcCGaifAAAWtVXW4/qkY83S5LuGp6m34xiDqC3o3wAAFrNlkNlypqzQXaHoev6Jenxq7rJYmHZdG9H+QAAtArDMPTkp1tVXefQpV1i9Mz1vWVlvxaI8gEAaCXLdx3Tmn0n5O9r1dPX9ZKfD285qMd3AgCgxRmGob8u2i5Jum1wOyVEsHopfkD5AAC0uC/yCpV32KYQfx/dxwRT/BfKBwCgRdkdhp79sn6Pr8mXdFBUKJuH4myUDwBAi/r3hkPaU1yhNsF+uuuStJ/+BHgdygcAoMVU19n1wle7JEn3jeyo8EA/kxPBFVE+AAAtZu6afB0uPaW48ADdPrS92XHgoigfAIAWUVlTpxeX7JYkTb2skwL9fExOBFdF+QAAtIjZK/br2MlqpUYG68bMFLPjwIVRPgAAzVZWWatXl9bvWPvgmE7y9+XtBefHdwcAoNle/3aPbFV16hwXqmv7JJkdBy6O8gEAaJbi8mq9lbNfkvTbsV3kw/4t+AmUDwBAs7z8zW6dqrWrT0objekeZ3YcuAHKBwCgyQ6VVGrO6oOSpEfGdZHFwlUP/DTKBwCgyf73q12qsTs0tGOUhqVHmx0HboLyAQBokt1FJ/XxhkOSpN+O62JyGrgTygcAoEmeX7xTDkMa0z1O/VLbmh0HboTyAQBotLzDZfp0yxFZLNJDYzubHQduhvIBAGi0vy7aIUma0CdRXePDTU4Dd0P5AAA0yuq9x7VsZ7F8rRZNG81VDzQe5QMAcNEMw9CzX9Zf9bhxQIraR4eYnAjuiPIBALhoS3cWa+3+EgX4WnX/ZZ3MjgM3RfkAAFwUh8PQs6fnetw+tL3iIwJNTgR3RfkAAFyUz/KO6PsCm0IDfHXvyI5mx4Ebo3wAAH5Snd2hv325U5J01yVpigzxNzkR3BnlAwDwk/694bD2HqtQ22A/TR6eZnYcuDnKBwDggqrr7Hrhq/qrHlmXpiss0M/kRHB3lA8AwAXNWX1QBWVVig8P1C2D25kdBx6A8gEAOK+K6jq9tGS3JOn+yzsp0M/H5ETwBJQPAMB5zfpun45X1KhdVLBuyEw2Ow48BOUDAHBOpZU1em35XknS9DGd5efDWwZaBt9JAIBzem35XpVX1alrfJiu6Z1odhx4EMoHAOBHisqrNOu7fZKk347tIqvVYnIieBLKBwDgR15asltVtQ71TW2jy7vFmh0HHobyAQA4S/6JSv1rzUFJ0sPjushi4aoHWhblAwBwlhe+2qVau6Hh6dEa2jHa7DjwQJQPAECDXUfLNW/jIUn1Vz2A1kD5AAA0+NvinXIY0rgeceqT0sbsOPBQlA8AgCRp86FSfZ5XKItFemgsVz3QeigfAABJ0l8X7ZAk/SwjSZ3jwkxOA09G+QAAaOWe4/p21zH5Wi2aNrqz2XHg4SgfAODlDMPQs1/WX/X41cBUpUYFm5wIno7yAQBe7psdRVp/oESBflZNvSzd7DjwApQPAPBiDoehvy7aKUm6fWh7xYYHmpwI3oDyAQBebOGWI9p2xKawAF/dO6Kj2XHgJSgfAOClau0O/e30XI9fj+igtiH+JieCt6B8APAohmHoZHWd2THcwsfrD2n/8UpFhfjrzuFpZseBF6F8APAIDoehz7Yc0dUv5qj3Hxbp5W92yzAMs2O5rKpau/73612SpN9cmq7QAF+TE8Gb8N0GwK3V2h3Kzi3QzKW7tae4ouH4XxftUNmpWj12ZVd2ZT2Hd1cd0JGyKiVEBOrmQalmx4GXoXwAcEtVtXZ9uC5fry7bq8OlpyRJ4YG+mjQsTQG+Vv110Q69vnyvbKdq9dTPesnHSgE542R1nV5ZukeS9MDlnRTo52NyIngbygcAt3Kyuk7vrTqgN3L2qbi8WpIUHRqguy5J082DUhUW6CdJigkN0KP/3qy5a/NVXlWn53+RIX9fRpol6a2cfTpRUaO06BD9vH+y2XHghSgfANxCSUWNZq/Yr9kr9qvsVK0kKalNkO4Z2UE3Zqb86H/vNw5IUWigrx6Yu1Gfbjmik9V1evWW/gry9+7/5ZdU1Ogfy/dKkqaP6SxfHwoZnI/yAcClFdmq9EbOPr276oAqa+ySpA7RIbpvVEdN7Jskvwu8eV7VK0EhAb6695/rtWxnsW59c7XenDRAEUF+zorvcl5dtkfl1XXqlhCu8b0SzI4DL0X5AOCS8k9U6rXle/TBukOqqXNIkronhCvr0nRd0TP+oudwjOwco3fvGqhJs9Zq3YES/er1VXpn8kBFhwa0ZnyXdNRWpdkr9kuSHh7XWVbmwcAklA8ALmV3UbleWbpH2bkFsjvqb5XNbNdWWZela1TnmCbdudK/XaTev3uIbntrtbYesenGV1fqn3cNUlKboJaO79JeXLJL1XUO9W/XVpd2iTU7DrwY5QOAS9hyqEyvLN2tL74v1JnlOS7pFK0pl6ZrYFpks2+X7Z4Yrg/vHapb3litvccqdMPMFfrnXYPUMSa0BdK7voPHKzV3Tb4k6ZFxXbj9GKaifAAw1Zp9J/TSN7u1fGdxw7FxPeKUdWm6eie3adGvlRYdoo/uG6Jb3litPcUVuvHVlXr7zoHqmRTRol/HFb3w1U7VOQyN6ByjQR2izI4DL0f5AOB0hmFo2c5ivfzNbq3dXyJJ8rFadG2fRN03qqM6x4W12tdOiAjSB/cM0e2z1ijvsE2/en2V3pw0QAPTIlvta5ptR2G55uUeliQ9PLaLyWkAygcAJ3I4DC36vlAvL92tvMM2SZK/j1U3ZCbrnhEdlRoV7JQcUaEBmvPrwbrr7XVas++EbntrtWbe0t9j50E89+UOGYZ0Zc949Ur2/Ks8cH2UDwCtrtbu0ILcAr3yH0ugB/v76OZBqbrrkg6KCw90eqbwQD+9c+dA/ea9DVqyvUi/fnudnv9Fhq7pk+j0LK0pN79UX249KqtFemhsZ7PjAJIoHwBaUVWtXR+uP6TXlu3RoZKzl0C/Y2h707dwD/Tz0Wu39tdDH2zSgk0Fun/uRp2srtOvBnrOXifPLtohSbquX7LSY1tvOAtoDMoHgBZ37iXQ/XXXJR3OWgLdFfj5WPX8LzIUFuir91Yf1GP/3qKyU7W6d2RHs6M124rdx5Sz+5j8fCx64PJOZscBGlA+ALSY0soazfru4pdAdxU+Vov+NLGnIoL89MrSPfrz59tVdqrWrW9JNQxDfzl91eOmgalKiXTOfBrgYlA+ADTbmSXQ31t1QBWNXALdVVgsFj1yRVeFB/npz59v18yle2Q7VasnJ/R0y5VAv9pWpNz8UgX6WZV1WbrZcYCzUD4ANFlLLYHuSu4d2VHhgX763fwtem/1QZVX1em5G/u4RYE6w+EwGuZ63DEsTbFhzp/QC1wI5QNAo51rCfT+7dpqyqXpGtWlaUugu5KbBqUqLNBXD76fqwWbCnSyuk6v3NzPZYeN/tsnmwu042i5wgJ9de8I95+7As/TqCo/Y8YMDRgwQGFhYYqNjdXEiRO1Y8eOcz7XMAxdeeWVslgsmj9/fktkBWCyvMNluu/d9Rrz/HL9e8Nh2R2GLukUrbl3D9ZH9w7RpV1j3b54nHFNn0T94/ZMBfpZtWR7kW57a43Kq2rNjvWTau0O/W3xTkn1V3Eigl1nci9wRqPKx7Jly5SVlaVVq1Zp8eLFqq2t1dixY1VRUfGj577wwgse80sI8HZr9p3Q7W+t0dUv5ujzvPq9V8b1iNOCKcP0z8mDNLhDlEf+vF/aJVbv3DlIYQG+WrPvhH71j1U6frLa7FgX9MG6fB04XqnoUH9NGtre7DjAOTVq2OWLL74468+zZ89WbGys1q9frxEjRjQcz83N1XPPPad169YpISGhZZICcKozS6C/8s0erdl/QpLzlkB3JQPTIvWvuwfr9rfql2O/8bWVeveuQUqIcL0dcatq7fr717skSVmXpiskgJF1uKZmfWeWlZVJkiIjf9gTobKyUjfddJNefvllxcfH/+Q5qqurVV39w/8kbDZbcyIBaKbzLYH+88xk3evEJdBdSc+kCH1w7xDdenpDup/PrC8gadEhZkc7yz9XHtBRW7WS2gTppkGes1AaPE+Ty4fD4dC0adM0bNgw9ezZs+H4gw8+qKFDh2rChAkXdZ4ZM2boj3/8Y1NjAGghdXaHsnMLNHPZHu0uOilJCvKrXwL91yPMWQLdlXSMCdWH9w3VrW+s1t5jFbrh1ZV6586B6p4YbnY0SVJ5Va1eWbpbkvTA6E4K8HWPybHwTk0uH1lZWcrLy1NOTk7DsQULFmjJkiXauHHjRZ/nscce0/Tp0xv+bLPZlJKS0tRYAJro/y74XnNWH5R0egn0oe01aViaIk1eAt2VJLUJ0gf3DtFtb67R1iM2/fL1lZp1xwD1b2f+jrhv5uxTSWWtOsSE6Lq+SWbHAS6oSTeuT5kyRQsXLtQ333yj5OTkhuNLlizRnj171KZNG/n6+srXt77bXH/99Ro1atQ5zxUQEKDw8PCzHgCcy1ZVq4/WH5IkPTSms7579DJNH9uF4nEO0aEB+tfdg5XZrq1sVXW65Y01Wr6z2NRMJypq9Ma3+yRJD43pIl83WpME3qlR36GGYWjKlCmaN2+elixZorS0tLM+/uijj2rz5s3Kzc1teEjS888/r1mzZrVYaAAt64sthaqpc6hzXKimXJbuUnuvuKKIID+9M3mgRnSO0alauya/vVafbTliWp6ZS3frZHWdeiSG68qePz3XDjBbo4ZdsrKyNGfOHGVnZyssLEyFhYWSpIiICAUFBSk+Pv6ck0xTU1N/VFQAuI75uYclSRMykjzyltnWEOzvqzduy9SD7+fq0y1HNGXOBv35ut66cYBzh42PlJ3S2ysPSJJ+O66LWy4FD+/TqCsfM2fOVFlZmUaNGqWEhISGx/vvv99a+QC0ssKyKq3ce1ySNCEj0eQ07sXf16q//6qvfpGZIochPfLxZr3x7V6nZnhxyW7V1Dk0sH2kRnWOcerXBpqqUVc+DMNo9BdoyucAcJ5PNhXIMKQB7dsqua333UbbXD5Wi/58fS9FBPvp9eV79adPt8l2qlYPjunc6leR9h+r0Adr8yXVX/XgqhXcBbOSAC+Xval+yOXaDO6QaCqLxaLHruyqh8d1kST9fclu/fGTrXI4Wvc/X89/tVN1DkOjusRoYJr5d9wAF4vyAXix3UUnlXfYJl+rReN7sRpxc1gsFmVdmq4nJ/SQJM1esV+//XCT6uyOVvl6247YtGBTgSTpt2O7tMrXAFoL5QPwYtmnJ5qO7BzDbbUt5NYh7fXCLzLkY7Xo3xsP6773Nqiq1t7iX+e5L3fKMKTxvRLUMymixc8PtCbKB+ClDMNQdm79/5wnsChVi5rYN0mv3dJf/r5WLd56VHfOXquT1XUtdv4NB0v01bajslqk6WM7t9h5AWehfABeasPBUh08UakQfx+N6RZndhyPM7p7nN6+Y6BC/H20Ys9x3fzGapVU1LTIuZ9dtEOS9PP+yeoYE9oi5wScifIBeKkzQy7jesQryJ99QFrDkI5R+tfdg9U22E+b8kv1i9dX6qitqlnnzNl1TCv2HJe/j1X3X96phZICzkX5ALxQrd2hTzfXr8h5LWt7tKreyW30wT1DFBceoJ1HT+rnr67QweOVTTqXYRj666LtkqSbBqVyazTcFuUD8EI5u4/peEWNokP9NTw92uw4Hq9TXJg+uneo2kUFK//EKf381RXaUVje6PN8ufWoNh0qU7C/j7IuTW+FpIBzUD4AL5S9sX7I5ereiWxC5iQpkcH68J4h6hofpqLyat342kptPFhy0Z9vdxh67sv6uR53DktTTFhAa0UFWh2/dQAvU1lTpy+3HpXEcurOFhseqLl3D1bf1DYqO1Wrm99Yre92H7uoz83OPaydR08qPNBXvx7RoZWTAq2L8gF4mcVbj6qyxq52UcHKSGljdhyv0ybYX+9OHqTh6dGqrLHrjllrtej7wgt+Tk2dQ89/tVOSdO+ojooIYtdhuDfKB+Bl5m9kB1uzhQT46s1JmbqiR7xq7A795r0N+nj9ofM+//11+co/cUrRoQGaNLS984ICrYTyAXiR4yertXxX/WV+hlzMFeDro5du6quf90+W3WHooQ83afZ3+370vFM1dr349S5J0tTL0hXs36j9QAGXRPkAvMhnW47I7jDUKymCxalcgK+PVX+5vrfuHJYmSfrDJ1v1v1/tOms38HdW7ldRebWS2wbpVwNTzYoKtCjKB+BF5p9ZTp2rHi7DarXo91d304Oj65dJf/6rnXpy4TY5HIZsVbWauWyPJGna6M7y9+VXNjwD1+8AL5F/olLrD5TIapGu7UP5cCUWi0UPjO6k8CBf/fGTrXrru32yVdUqNixApZW1So8N1c/YfwcehPIBeIkzy6kP7Rit2PBAk9PgXO4YlqbwQD898vFmffQfE1AfGtNZPlYmB8NzcA0P8AKGYTDk4iau75+sV27uJ//Ti7/1SorQFT3jTU4FtCyufABeYOsRm3YXnZS/r1XjeCNzeeN6xOudyQP1z1UHdP9lnbglGh6H8gF4gezTVz1Gd4tVeCALVLmDwR2iNLhDlNkxgFbBsAvg4ewOQwsahlyYtAjAfJQPwMOt3ndchbYqhQf6alSXGLPjAADlA/B02Rvrr3qM752gAF8fk9MAAOUD8GhVtXZ9lndEEkMuAFwH5QPwYEt3FKu8qk4JEYEa2D7S7DgAIInyAXi0MwuLXdsnUVYWqQLgIigfgIeyVdXq6+1FkhhyAeBaKB+Ah/piS6Fq6hzqHBeqbglhZscBgAaUD8BDzT895DIhI4kVMgG4FMoH4IEKy6q0cu9xSexgC8D1UD4AD7Rwc4EMQ8ps11YpkcFmxwGAs1A+AA/UMOTSl4mmAFwP5QPwMLuLTirvsE2+VovG90owOw4A/AjlA/AwZ9b2GNk5RpEh/ianAYAfo3wAHsQwDGWf2cGWIRcALoryAXiQDQdLdfBEpYL9fTSmW5zZcQDgnCgfgAdZcHrIZVyPeAX5s4MtANdE+QA8RK3doYWbz+xgy9oeAFwX5QPwEDm7j+l4RY2iQ/01PD3a7DgAcF6UD8BDZG+sH3K5uneifH340QbguvgNBXiAypo6fbn1qCSGXAC4PsoH4AEWbz2qyhq72kUFKyOljdlxAOCCKB+AB2hY26NPIjvYAnB5lA/AzZ2oqNHyncWSWFgMgHugfABu7tPNBapzGOqVFKGOMaFmxwGAn0T5ANzc/DNDLkw0BeAmKB+AG8s/Uan1B0pksUjX9qF8AHAPlA/AjZ3ZwXZoxyjFhgeanAYALg7lA3BThmH8x5ALE00BuA/KB+Cmth6xaXfRSfn7WnVFz3iz4wDARaN8AG7qzNoeo7vFKjzQz+Q0AHDxKB+AG7I7DC1gyAWAm6J8AG5o9b7jKrRVKTzQV6O6xJgdBwAahfIBuKHsjfVXPa7qlaAAXx+T0wBA41A+ADdTXWfXZ3lHJDHkAsA9UT4AN/PN9mKVV9UpISJQg9IizY4DAI1G+QDczJmFxa7tkyirlR1sAbgfygfgRmxVtfp6e5EkhlwAuC/KB+BGvthSqJo6hzrHhapbQpjZcQCgSSgfgBuZf3rIZUJGkiwWhlwAuCfKB+AmjtqqtHLvcUnsYAvAvVE+ADfxyaYCGYaU2a6tUiKDzY4DAE1G+QDcRMOQS18mmgJwb5QPwA3sLjqpvMM2+VotGt8rwew4ANAslA/ADZxZ22Nk5xhFhvibnAYAmofyAbg4wzCUfXoH22szmGgKwP1RPgAXt+FgqQ6eqFSwv4/GdI8zOw4ANBvlA3BxC04PuYzrEa9gf1+T0wBA81E+ABdWa3do4eYzO9gy5ALAM1A+ABeWs/uYjlfUKDrUX8PTo82OAwAtgvIBuLDsjfVDLlf3TpSvDz+uADxDo36bzZgxQwMGDFBYWJhiY2M1ceJE7dixo+HjJ06c0NSpU9WlSxcFBQUpNTVV999/v8rKylo8OODpKmvq9OXWo5K4ywWAZ2lU+Vi2bJmysrK0atUqLV68WLW1tRo7dqwqKiokSQUFBSooKNCzzz6rvLw8zZ49W1988YUmT57cKuEBT7Z461FV1tjVLipYfVPamB0HAFqMxTAMo6mfXFxcrNjYWC1btkwjRow453M+/PBD3XLLLaqoqJCv70/P1LfZbIqIiFBZWZnCw8ObGg1we3fOXqsl24t0/2Xpmj62i9lxAOCCGvP+3az79s4Mp0RGRl7wOeHh4ectHtXV1aqurm74s81ma04kwCOcqKjR8p3FktjLBYDnafIMNofDoWnTpmnYsGHq2bPnOZ9z7NgxPfnkk7r77rvPe54ZM2YoIiKi4ZGSktLUSIDH+HRzgeochnolRahjTKjZcQCgRTW5fGRlZSkvL09z584958dtNpvGjx+v7t276w9/+MN5z/PYY4+prKys4ZGfn9/USIDHmH96OXXW9gDgiZo07DJlyhQtXLhQy5cvV3Jy8o8+Xl5eriuuuEJhYWGaN2+e/Pz8znuugIAABQQENCUG4JHyT1Rq/YESWSzSNX0oHwA8T6OufBiGoSlTpmjevHlasmSJ0tLSfvQcm82msWPHyt/fXwsWLFBgYGCLhQW8wZkdbId2jFJcOD8/ADxPo658ZGVlac6cOcrOzlZYWJgKCwslSREREQoKCmooHpWVlXr33Xdls9kaJpDGxMTIx8en5f8GgAcxDOM/hlyYaArAMzWqfMycOVOSNGrUqLOOz5o1S5MmTdKGDRu0evVqSVJ6evpZz9m3b5/at2/f9KSAF9h6xKbdRSfl72vVFT3jzY4DAK2iUeXjp5YEGTVq1E8+B8D5ZZ++6jG6W6zCA88/VwoA3BmbRQAuwu4wtIAhFwBegPIBuIjV+46r0Fal8EBfjeoSY3YcAGg1lA/ARWRvrL/qcVWvBAX4MjkbgOeifAAuoLrOrs/yjkhiyAWA56N8AC7gm+3FKq+qU0JEoAalnX+vJADwBJQPwAWcWVjs2j6JslotJqcBgNZF+QBMZquq1dfbiyQx5ALAO1A+AJN9saVQNXUOdYoNVbeEMLPjAECro3wAJpt/eshlYt8kWSwMuQDwfJQPwERHbVVaufe4pPr5HgDgDSgfgIk+2VQgw5Ay27VVSmSw2XEAwCkoH4CJzgy5TOjLRFMA3oPyAZhkd9FJ5R22yddq0fheCWbHAQCnoXwAJjmztseIzjGKDPE3OQ0AOA/lAzCBYRjKbtjBlommALwL5QMwwcb8Uh08Ualgfx+N6R5ndhwAcCrKB2CC7I31Qy7jesQr2N/X5DQA4FyUD8DJau0OLdx8ZgdbhlwAeB/KB+BkObuP6XhFjaJC/DU8PdrsOADgdJQPwMnODLlc3TtBvj78CALwPvzmA5yosqZOX249KomFxQB4L8oH4ESLtx5VZY1d7aKC1TeljdlxAMAUlA/AiRrW9uiTyA62ALwW5QNwkhMVNVq+s1iSdG0GQy4AvBflA3CSTzcXqM5hqGdSuNJjQ82OAwCmoXwATjL/9JDLRK56APBylA/ACfJPVGr9gRJZLNI1fVhYDIB3o3wATrBgU/1Vj6EdoxQXHmhyGgAwF+UDaGWGYWj+6YXFJjDkAgCUD6C1bT1i066ik/L3teqKnvFmxwEA01E+gFZ2Zm2Py7vGKjzQz+Q0AGA+ygfQiuwOQwvOLCzGkAsASKJ8AK1q9b7jKrRVKTzQV5d2jTE7DgC4BMoH0IrOXPW4qleCAnx9TE4DAK6B8gG0kuo6uz7bckQSQy4A8J8oH0Ar+WZ7sWxVdYoPD9SgtEiz4wCAy6B8AK0kO7d+bY9rMxJltbKDLQCcQfkAWoGtqlZfby+SJE3IYDl1APhPlA+gFXyxpVA1dQ51ig1V94Rws+MAgEuhfACtIHtT/ZDLxL5JslgYcgGA/0T5AFrYUVuVVuw5Lkm6lh1sAeBHKB9AC/tkU4EMQ8ps11YpkcFmxwEAl0P5AFrY/NwzO9hy1QMAzoXyAbSg3UUnlXfYJl+rReN7Uz4A4FwoH0ALOrO2x4jOMYoM8Tc5DQC4JsoH0EIMw1B2ww62XPUAgPOhfAAtZGN+qQ6eqFSwv4/GdI8zOw4AuCzKB9BCsjfWD7mM6xGvYH9fk9MAgOuifAAtoNbu0MLN9TvYXsuQCwBcEOUDaAE5u4/peEWNokL8dUl6tNlxAMClUT6AFnBmyOXq3gny9eHHCgAuhN+SQDNV1tTpy61HJUkT+iaZnAYAXB/lA2imxVuPqrLGrnZRweqb0sbsOADg8piSDzTR8ZPV+mzLEb2Rs0+SNKFPIjvYAsBFoHwAjXCyuk6LtxYqO7dA3+46JrvDkCRFBPnphswUk9MBgHugfAA/oabOoWU7i5Wde1hfbTuqqlpHw8d6J0fo2j6JurZPomLDA01MCQDug/IBnIPDYWjN/hPKzi3QZ1uOqOxUbcPH0qJDNCGjvnB0iAk1MSUAuCfKB3CaYRjaesSm7NwCfbKpQEfKqho+FhsWoGv7JGpCRpJ6JoUztwMAmoHyAa934HiFFuQWKHtTgXYXnWw4Hhboq6t6JmhCRqIGdYiSj5XCAQAtgfIBr1RUXqVPNx9Rdm6BcvNLG477+1o1ulusru2TpEu7xijA18e8kADgoSgf8BrlVbVa9P1RZece1ne7j+n0jSqyWqRh6dGakJGkcT3iFBboZ25QAPBwlA94tOo6u77ZXqwFmw7r621Fqq774U6VjJQ2mpCRqKt7JyomLMDElADgXSgf8Dh2h6HVe4/X36mSd0TlVXUNH+sYE6KJGUm6NiNR7aJCTEwJAN6L8gGPYBiGthwua7hTpai8uuFj8eGBuvb0rbE9ErlTBQDMRvmAW9tbfFILNhVoQW6B9h6raDgeEeSnq3rV36kysH2krNypAgAug/IBt1Nkq6ovHJsKtPlQWcPxQD+rRneL04SMJI3oHM2dKgDgoigfcAtlp2q1KK9Q2ZsOa8We4zJO36niY7Xokk7RmpCRqDHd4xUawLc0ALg6flPDZVXV2rVke5Gycw/rm+3FqrH/cKdK/3ZtNSEjUVf1SlB0KHeqAIA7oXzApdTZHVp5+k6VRXmFKq/+4U6VznGhmpCRpGv7JColMtjElACA5qB8wHSGYSg3v1TZuQVauPmIjp384U6VpDZBuqZPoiZkJKprfBh3qgCAB6B8wDQHj1fqo/X5yt5UoAPHKxuOtw320/jeCZqQkaT+qW25UwUAPEyjyseMGTP073//W9u3b1dQUJCGDh2qZ555Rl26dGl4TlVVlR566CHNnTtX1dXVGjdunF555RXFxcW1eHi4pxMVNfr717v07qoDqju9xnmQn4/G9ojThIxEXdIpRn4+VpNTAgBaS6PKx7Jly5SVlaUBAwaorq5Ojz/+uMaOHautW7cqJKR+tcgHH3xQn376qT788ENFRERoypQpuu666/Tdd9+1yl8A7qOq1q5Z3+3XK9/sbpjLMTw9WjdkJmtM9zgF+3MhDgC8gcUwzty02HjFxcWKjY3VsmXLNGLECJWVlSkmJkZz5szRz3/+c0nS9u3b1a1bN61cuVKDBw/+yXPabDZFRESorKxM4eHhTY0GF+JwGFqwqUB/XbRDh0tPSZK6J4Trd+O7aVh6tMnpAAAtoTHv3836r2ZZWf0CT5GRkZKk9evXq7a2VqNHj254TteuXZWamnre8lFdXa3q6h8mGNpstuZEgotZseeYnv5sm/IO17+uCRGB+u3YLvpZ3yTmcgCAl2py+XA4HJo2bZqGDRumnj17SpIKCwvl7++vNm3anPXcuLg4FRYWnvM8M2bM0B//+MemxoCL2l1Urj9/vl1fbSuSJIUG+Oq+UR01eXiaAv1YeRQAvFmTy0dWVpby8vKUk5PTrACPPfaYpk+f3vBnm82mlJSUZp0T5ikur9YLX+3U3LX5sjsM+Vgtumlgqh4Y3YnFwAAAkppYPqZMmaKFCxdq+fLlSk5ObjgeHx+vmpoalZaWnnX14+jRo4qPjz/nuQICAhQQwJuSuztVY9ebOXs1c+keVdTYJUljusfpf67oqvTYUJPTAQBcSaPKh2EYmjp1qubNm6elS5cqLS3trI/3799ffn5++vrrr3X99ddLknbs2KGDBw9qyJAhLZcaLsPuMPTvDYf03Jc7VWirkiT1To7Q41d10+AOUSanAwC4okaVj6ysLM2ZM0fZ2dkKCwtrmMcRERGhoKAgRUREaPLkyZo+fboiIyMVHh6uqVOnasiQIRd1pwvcy7e7ivX0Z9u17Uj9ZNKkNkF65IouuqZ3IpNJAQDn1ahbbc+3tPWsWbM0adIkST8sMvavf/3rrEXGzjfs8t+41db17Sgs19OfbdOyncWSpLBAX025NF23D23PZFIA8FKNef9u1jofrYHy4bqKbFX62+Kd+mBdvhyG5Gu16JbB7XT/5Z0UGeJvdjwAgImcts4HvENFdZ1eX75Xry/fq1O19ZNJr+wZr0eu6Kq06BCT0wEA3A3lA+dldxj6cF2+nlu8U8Xl9QvB9U1to99d1U2Z7SNNTgcAcFeUD/yIYRhaurNYMz7bpp1HT0qSUiOD9T9XdNVVveLZ1h4A0CyUD5zl+4Iyzfhsu3J2H5MkRQT5aepl6bp1SDsF+DKZFADQfF5VPlbtPa4eieEKC/QzO4rLOVJ2Ss8u2ql/bzwkw5D8fay6fWg7Tbm0kyKC+fcCALQcrykfxeXV+uXrq2S1SF3jw5XZvq36t2urzPaRSmoTZHY805RX1eq1ZXv1Rs5eVdU6JEnX9EnUI+O6KCUy2OR0AABP5DXl40jZKSW3DdKhklPaesSmrUdsemflAUn1O61mto9UZrv6QtItIVw+Hr5IVp3doX+tzdcLi3fqeEWNJGlA+7Z6/Kpu6pva1uR0AABP5nXrfBy1VWnd/hKtO3BC6w+U6PsCm+yOs/8JQvx91De1rTLbt1Vmu0hlpLZRaIBn9DTDMPTVtiL9+fNt2lNcIUlKiw7Ro1d21djucUwmBQA0CYuMNUJFdZ025Zdq3YESrTtQoo0HSlReXXfWc6wWqVtCuDJPD9Nktm+rhAj3G6rZfKhUT326Tav3nZAktQ3207TRnXXToFT5+VhNTgcAcGeUj2awOwztKCzX+gMn6gvJ/hIdLj31o+cltQk6PWek/upIl/gwlx2qOVRSqWcX7dD83AJJkr+vVXcOS9NvLu2ocCbfAgBaAOWjhR0pO6V1+0u0/kD9cM3WApv+a6RGYQG+ykhto8x29VdGMlLaKMTkoZqyU7V6Zeluzfpuv2rq6ieT/qxvkh4a21nJbZlMCgBoOZSPVlZRXafc/FKt3V8/b2TjwVKd/K+hGh+rRd0Tws+6OhIfEeiUfDV1Ds1ZfUD/+/UulVTWSpIGd4jU767qrl7JEU7JAADwLpQPJ7M7DG0vtGn9gRKt3V+i9ftPqKCs6kfPS2oTpAHt26r/6TtrOse17FCNYRha9H2h/vz5du0/XilJ6hgTosev6qbLusYymRQA0GooHy6goPSU1h2oLyJr95doe+G5h2r6tmurAe3aqv/poZpg/6YN1Ww8WKKnPt2mdQdKJEnRof6aNrqzfjkgRb5MJgUAtDLKhwsqr6pVbn5pw9yRDQdLVFljP+s5PlaLeiTWD9UMOH11JDb8wkM1B49X6i+Ltmvh5iOSpEA/q359SQfdM7Kjx9weDABwfZQPN1Bnd2h7YbnW7f/hrppC24+HalIig5TZLrKhkHSKDZXValFpZY1eWrJbb6/cr1q7IYtFur5fsh4a29ktbwMGALg3yocbMgxDh0tP1d9Rs79+zZHthTb996sTHuirjNS22pRfqrJT9ZNJh6dH6/Gruql7ovf8ewEAXAvlw0PYqmq18WCp1p++OrLxYKlO1f4wVNMlLkyPXdVVIzvHMJkUAGCqxrx/MynAhYUH+mlk5xiN7BwjqX6oZtuRcm04WKLIEH9d1SvBZRc2AwDgfCgfbsTXx6peyRGs1QEAcGvcgwkAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJyK8gEAAJzK5Xa1NQxDkmSz2UxOAgAALtaZ9+0z7+MX4nLlo7y8XJKUkpJichIAANBY5eXlioiIuOBzLMbFVBQncjgcKigoUFhYmCwWS4ue22azKSUlRfn5+QoPD2/Rc6PxeD1cC6+Ha+H1cD28JhdmGIbKy8uVmJgoq/XCszpc7sqH1WpVcnJyq36N8PBwvnFcCK+Ha+H1cC28Hq6H1+T8fuqKxxlMOAUAAE5F+QAAAE7lVeUjICBATzzxhAICAsyOAvF6uBpeD9fC6+F6eE1ajstNOAUAAJ7Nq658AAAA81E+AACAU1E+AACAU1E+AACAU1E+AACAU3lN+Xj55ZfVvn17BQYGatCgQVqzZo3ZkbzWjBkzNGDAAIWFhSk2NlYTJ07Ujh07zI6F0/785z/LYrFo2rRpZkfxWocPH9Ytt9yiqKgoBQUFqVevXlq3bp3ZsbyS3W7X73//e6WlpSkoKEgdO3bUk08+eVGbp+H8vKJ8vP/++5o+fbqeeOIJbdiwQX369NG4ceNUVFRkdjSvtGzZMmVlZWnVqlVavHixamtrNXbsWFVUVJgdzeutXbtWr732mnr37m12FK9VUlKiYcOGyc/PT59//rm2bt2q5557Tm3btjU7mld65plnNHPmTL300kvatm2bnnnmGf3lL3/Riy++aHY0t+YV63wMGjRIAwYM0EsvvSSpfvO6lJQUTZ06VY8++qjJ6VBcXKzY2FgtW7ZMI0aMMDuO1zp58qT69eunV155RX/605+UkZGhF154wexYXufRRx/Vd999p2+//dbsKJB09dVXKy4uTm+++WbDseuvv15BQUF69913TUzm3jz+ykdNTY3Wr1+v0aNHNxyzWq0aPXq0Vq5caWIynFFWViZJioyMNDmJd8vKytL48ePP+lmB8y1YsECZmZm64YYbFBsbq759++of//iH2bG81tChQ/X1119r586dkqRNmzYpJydHV155pcnJ3JvL7Wrb0o4dOya73a64uLizjsfFxWn79u0mpcIZDodD06ZN07Bhw9SzZ0+z43ituXPnasOGDVq7dq3ZUbze3r17NXPmTE2fPl2PP/641q5dq/vvv1/+/v66/fbbzY7ndR599FHZbDZ17dpVPj4+stvteuqpp3TzzTebHc2teXz5gGvLyspSXl6ecnJyzI7itfLz8/XAAw9o8eLFCgwMNDuO13M4HMrMzNTTTz8tSerbt6/y8vL06quvUj5M8MEHH+i9997TnDlz1KNHD+Xm5mratGlKTEzk9WgGjy8f0dHR8vHx0dGjR886fvToUcXHx5uUCpI0ZcoULVy4UMuXL1dycrLZcbzW+vXrVVRUpH79+jUcs9vtWr58uV566SVVV1fLx8fHxITeJSEhQd27dz/rWLdu3fTxxx+blMi7Pfzww3r00Uf1y1/+UpLUq1cvHThwQDNmzKB8NIPHz/nw9/dX//799fXXXzccczgc+vrrrzVkyBATk3kvwzA0ZcoUzZs3T0uWLFFaWprZkbza5Zdfri1btig3N7fhkZmZqZtvvlm5ubkUDycbNmzYj24937lzp9q1a2dSIu9WWVkpq/Xst0ofHx85HA6TEnkGj7/yIUnTp0/X7bffrszMTA0cOFAvvPCCKioqdMcdd5gdzStlZWVpzpw5ys7OVlhYmAoLCyVJERERCgoKMjmd9wkLC/vRfJuQkBBFRUUxD8cEDz74oIYOHaqnn35aN954o9asWaPXX39dr7/+utnRvNI111yjp556SqmpqerRo4c2btyov/3tb7rzzjvNjubeDC/x4osvGqmpqYa/v78xcOBAY9WqVWZH8lqSzvmYNWuW2dFw2siRI40HHnjA7Bhe65NPPjF69uxpBAQEGF27djVef/11syN5LZvNZjzwwANGamqqERgYaHTo0MH43e9+Z1RXV5sdza15xTofAADAdXj8nA8AAOBaKB8AAMCpKB8AAMCpKB8AAMCpKB8AAMCpKB8AAMCpKB8AAMCpKB8AAMCpKB8AAMCpKB8AAMCpKB8AAMCp/j8M+7BkmV0zXAAAAABJRU5ErkJggg==" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(all_rewards)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T18:26:38.543323Z", + "start_time": "2023-12-13T18:26:38.246992Z" + } + }, + "id": "457bbd9d81cac391" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/lunar-interactive-bc.ipynb b/examples/lunar-interactive-bc.ipynb new file mode 100644 index 0000000..79c63fc --- /dev/null +++ b/examples/lunar-interactive-bc.ipynb @@ -0,0 +1,661 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T19:12:43.977147Z", + "start_time": "2023-12-13T19:12:42.044264Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "\n", + "import numpy as np\n", + "\n", + "import torch\n", + "import torch.nn.functional as F\n", + "\n", + "from cogment_lab.envs.gymnasium import GymEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.coltra_utils import convert_trial_data_to_coltra\n", + "from cogment_lab.utils.runners import process_cleanup\n", + "from cogment_lab.utils.trial_utils import format_data_multiagent, concatenate\n", + "\n", + "from coltra import HomogeneousGroup\n", + "from coltra.buffers import Observation, Action\n", + "from coltra.models import MLPModel\n", + "from coltra.policy_optimization import CrowdPPOptimizer\n", + "\n", + "from cogment_lab.actors import ColtraActor\n", + "\n", + "from tqdm import trange\n", + "import matplotlib.pyplot as plt\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:12:44.045367Z", + "start_time": "2023-12-13T19:12:43.986529Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:12:44.910776Z", + "start_time": "2023-12-13T19:12:44.396144Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T20:12:44.394188\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:12:47.125359Z", + "start_time": "2023-12-13T19:12:44.911292Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# We'll train on \n", + "\n", + "cenv = GymEnvironment(\n", + " env_id=\"LunarLander-v2\",\n", + " render=True,\n", + ")\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"mcar\",\n", + " port=9001, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:12:49.518262Z", + "start_time": "2023-12-13T19:12:47.125042Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Create a model using coltra\n", + "\n", + "model = MLPModel(\n", + " config={\n", + " \"hidden_sizes\": [64, 64],\n", + " }, \n", + " observation_space=cenv.env.observation_space, \n", + " action_space=cenv.env.action_space\n", + ")\n", + "\n", + "# Put the model in shared memory so that the actor can access it\n", + "model.share_memory()\n", + "actor = ColtraActor(model=model)\n", + "\n", + "\n", + "await cog.run_actor(\n", + " actor=actor,\n", + " actor_name=\"coltra\",\n", + " port=9021,\n", + " log_file=\"actor.log\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "outputs": [ + { + "data": { + "text/plain": "{'mcar': ,\n 'coltra': }" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:12:49.519197Z", + "start_time": "2023-12-13T19:12:49.514813Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:12:51.661932Z", + "start_time": "2023-12-13T19:12:49.518157Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "LUNAR_LANDER_ACTIONS = [\"no-op\", \"ArrowRight\", \"ArrowUp\", \"ArrowLeft\"]\n", + "\n", + "actions = LUNAR_LANDER_ACTIONS\n", + "\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=30)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 10/10 [00:01<00:00, 9.98it/s]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean_reward: -52.153119627747216\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "# Estimate random agent performance\n", + "\n", + "episodes = []\n", + "for i in trange(10):\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await format_data_multiagent(datastore=cog.datastore, trial_id=trial_id, actor_agent_specs=cenv.agent_specs)\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + "mean_reward = np.mean([sum(e.rewards) for e in episodes])\n", + "print(f\"mean_reward: {mean_reward}\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:12:54.225530Z", + "start_time": "2023-12-13T19:12:53.217519Z" + } + }, + "id": "b7cde51d7dc0c3a9" + }, + { + "cell_type": "code", + "execution_count": 10, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Reinitialize the agent\n", + "\n", + "cog.stop_service(\"coltra\")\n", + "\n", + "model = MLPModel(\n", + " config={\n", + " \"hidden_sizes\": [64, 64],\n", + " }, \n", + " observation_space=cenv.env.observation_space, \n", + " action_space=cenv.env.action_space\n", + ")\n", + "\n", + "# Put the model in shared memory so that the actor can access it\n", + "model.share_memory()\n", + "actor = ColtraActor(model=model)\n", + "\n", + "\n", + "await cog.run_actor(\n", + " actor=actor,\n", + " actor_name=\"coltra\",\n", + " port=9021,\n", + " log_file=\"actor.log\"\n", + ")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:13:03.878130Z", + "start_time": "2023-12-13T19:13:01.612883Z" + } + }, + "id": "5c1585be28fdae6c" + }, + { + "cell_type": "code", + "execution_count": 11, + "outputs": [], + "source": [ + "# Get some human episodes\n", + "episodes = []\n", + "for i in range(3):\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"gym\": \"web_ui\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id)\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + " \n", + "all_data = concatenate(episodes)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:35:11.374153Z", + "start_time": "2023-12-13T19:34:33.073648Z" + } + }, + "id": "8f1381b80d4c8799" + }, + { + "cell_type": "code", + "execution_count": 12, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean_reward: 77.66748484175616\n", + "rewards: [-90.41456591431051, 277.5436417415343, 45.87337869804469]\n" + ] + } + ], + "source": [ + "mean_reward = np.mean([sum(e.rewards) for e in episodes])\n", + "print(f\"mean_reward: {mean_reward}\")\n", + "print(f\"rewards: {[sum(e.rewards) for e in episodes]}\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:35:19.616407Z", + "start_time": "2023-12-13T19:35:19.613200Z" + } + }, + "id": "73d139b8e5d005d8" + }, + { + "cell_type": "code", + "execution_count": 14, + "outputs": [], + "source": [ + "cog.stop_service(\"web_ui\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:35:31.859929Z", + "start_time": "2023-12-13T19:35:30.845308Z" + } + }, + "id": "73c751b525abf31e" + }, + { + "cell_type": "code", + "execution_count": 15, + "outputs": [], + "source": [ + "all_obs = Observation(vector=all_data.observations).tensor()\n", + "all_actions = torch.tensor(all_data.actions)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:35:34.727391Z", + "start_time": "2023-12-13T19:35:34.724168Z" + } + }, + "id": "cdcef32e17f9afc1" + }, + { + "cell_type": "code", + "execution_count": 20, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "loss: 0.186: 100%|██████████| 500/500 [00:00<00:00, 738.80it/s]\n" + ] + } + ], + "source": [ + "losses = []\n", + "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n", + "for t in (pbar := trange(500)):\n", + " preds = model(all_obs)[0].logits\n", + " loss = F.cross_entropy(preds, all_actions)\n", + " \n", + " optimizer.zero_grad()\n", + " loss.backward()\n", + " optimizer.step()\n", + " \n", + " pbar.set_description(f\"loss: {loss.item():.3}\")\n", + " \n", + " losses.append(loss.item())\n", + " " + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:36:04.511395Z", + "start_time": "2023-12-13T19:36:03.830625Z" + } + }, + "id": "9a1cb51957e9f672" + }, + { + "cell_type": "code", + "execution_count": 21, + "outputs": [ + { + "data": { + "text/plain": "[]" + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGdCAYAAADqsoKGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPdklEQVR4nO3deVxU5f4H8M8MMIDAsAoIjoL7Dgoy4pZdUVxuZmmpWRiZmqamWKm3n0t57wWXyhTTtExTE7XUyoxUFNcRCMR9X1DQAZHYZWBmzu8ParqTqIwCB4bP+/U6r5ee85znfOdUdz73zHOeRyIIggAiIiKiOk4qdgFEREREVYGhhoiIiMwCQw0RERGZBYYaIiIiMgsMNURERGQWGGqIiIjILDDUEBERkVlgqCEiIiKzYCl2ATVFr9fj9u3bcHBwgEQiEbscIiIiqgRBEFBQUAAvLy9IpY9+FlNvQs3t27ehUCjELoOIiIiewK1bt9C4ceNHtqk3ocbBwQFA+U2Ry+UiV0NERESVkZ+fD4VCYfgef5R6E2r+/MlJLpcz1BAREdUxlRk6woHCREREZBYYaoiIiMgsMNQQERGRWWCoISIiIrPAUENERERmgaGGiIiIzAJDDREREZkFhhoiIiIyCww1REREZBYYaoiIiMgsMNQQERGRWWCoISIiIrPAUFMFfi8qxYoDV3An777YpRAREdVbDDVVYN6PZ7H414sY/02y2KUQERHVWww1VWDXqdsAgNMZeSJXQkREVH8x1FQBa0sLw59/PauGXi+IWA0REVH9xFDzlATBOMBM2JCMXafviFQNERFR/cVQ85Ryikpxv0xntO/H1AyRqiEiIqq/GGqe0o17xQ/syy0uE6ESIiKi+o2h5indyC4CADR0sEbnJk4AgPN38jmuhoiIqIYx1DwliQRo5maHgR08sW1CMGSWUhSV6tBr0QE8t/wICjVasUskIiKqFyzFLqCue7FLY7zYpbHh7528HfFb2u/IyL2PjNz7WBV/Fe+GthaxQiIiovqBT2qq2Pwh7eFqJzP8/csj15BTVCpiRURERPUDQ00V6+DtiAPv9cHB9/qgg7ccJWV6rDt2Q+yyiIiIzB5DTTWQ21ihqasdwrv7AgCWxV1Gj6j9WH3o6gPz2hAREVHVYKipRs/5eWGInxcAICP3Pv67+wKW7rssclVERETmiaGmGskspVg2qjNUs/+Bqf9oAQD4LO4y1h29zic2REREVeyJQs2KFSvg4+MDGxsbKJVKJCYmPrTt9u3bERgYCCcnJ9jZ2cHf3x8bNmwwavP6669DIpEYbQMGDDBqk5OTg9GjR0Mul8PJyQljx45FYWHhk5Rf4xo52iKif2tM6N0MADD/p3OYvf00NFrdY84kIiKiyjI51GzZsgURERGYN28eUlJS4Ofnh9DQUGRlZVXY3sXFBR988AFUKhVOnTqF8PBwhIeH49dffzVqN2DAANy5c8ewbd682ej46NGjcfbsWezduxe7du3CoUOHMH78eFPLF9WsgW0wa2AbSCRATNItjFp9HLdyHpyRmIiIiEwnEUz8HUSpVKJr166Ijo4GAOj1eigUCkyZMgWzZs2qVB9dunTB4MGDsWDBAgDlT2pyc3Oxc+fOCtufP38e7dq1Q1JSEgIDAwEAsbGxGDRoENLT0+Hl5fXYa+bn58PR0RF5eXmQy+WVqrO6xF/MwpTNJ1BQokUDmQXmPdcOI7o2EbUmIiKi2siU72+TntSUlpYiOTkZISEhf3UglSIkJAQqleqx5wuCgLi4OFy8eBG9e/c2OhYfHw93d3e0bt0aEydOxL179wzHVCoVnJycDIEGAEJCQiCVSpGQkFDhtTQaDfLz84222qJPa3f8PKUXgnxdUFyqw8zvT2N5HAcQExERPQ2TQk12djZ0Oh08PDyM9nt4eECtVj/0vLy8PNjb20Mmk2Hw4MFYvnw5+vXrZzg+YMAAfPPNN4iLi8PChQtx8OBBDBw4EDpd+ZgTtVoNd3d3oz4tLS3h4uLy0OtGRkbC0dHRsCkUClM+arVr4toAMeO6YWrflgCAj/dewvwfz+J+KcfZEBERPYkaWSbBwcEBqampKCwsRFxcHCIiItCsWTP06dMHADBy5EhD244dO6JTp05o3rw54uPj0bdv3ye65uzZsxEREWH4e35+fq0LNlKpBBH9WsFOZoHIXy5g3bEbiLuQiU9f9kegj4vY5REREdUpJj2pcXNzg4WFBTIzM432Z2ZmwtPT8+EXkUrRokUL+Pv7Y8aMGRg+fDgiIyMf2r5Zs2Zwc3PDlStXAACenp4PDETWarXIycl56HWtra0hl8uNttpqwjPN8WVYILwcbXAr5z5e/SoBR69ki10WERFRnWJSqJHJZAgICEBcXJxhn16vR1xcHIKDgyvdj16vh0ajeejx9PR03Lt3D40aNQIABAcHIzc3F8nJyYY2+/fvh16vh1KpNOUj1Foh7TywN+IZPNu6IUrK9Ahfl4RvVDeg03M+GyIiosow+ZXuiIgIrFmzBuvXr8f58+cxceJEFBUVITw8HAAQFhaG2bNnG9pHRkZi7969uHbtGs6fP4+PP/4YGzZswKuvvgoAKCwsxHvvvYfjx4/jxo0biIuLw/PPP48WLVogNDQUANC2bVsMGDAA48aNQ2JiIo4ePYrJkydj5MiRlXrzqa6ws7bEqtcC0K+dB0q1esz94SyGrjiKO3n3xS6NiIio1jN5TM2IESNw9+5dzJ07F2q1Gv7+/oiNjTUMHr558yak0r+yUlFRESZNmoT09HTY2tqiTZs22LhxI0aMGAEAsLCwwKlTp7B+/Xrk5ubCy8sL/fv3x4IFC2BtbW3oZ9OmTZg8eTL69u0LqVSKYcOGYdmyZU/7+Wsda0sLrHo1AN8mpGHxrxdxOiMPo1YfR8z4YHg62ohdHhERUa1l8jw1dVVtmqemstJ/L8bI1ceR/vt9KFxsET2qC/wUTmKXRUREVGOqbZ4aqlmNnRsgZnw3KFxscSvnPoatPIYVB65wnA0REVEFGGpqucbODbBrci8M7tgIWr2Axb9exNj1SSgp43w2RERE/4uhpg5wbGCF6Fc6Y8lLfrC1skD8xbt47asEZBWUiF0aERFRrcFQU0dIJBIMD2iMta93hYO1JZJu/I7nlh9BctrvYpdGRERUKzDU1DHBzV2xc3IPtHC3R2a+BiNXq7ApIQ31ZLw3ERHRQzHU1EHNG9pj59s9MLCDJ8p0Aj7YcQYzvz/FcTZERFSvMdTUUfbWlvh8dBfMHNAGUgmw9bd0jPhChcx8jrMhIqL6iaGmDpNIJJjYpznWvxEEpwZWOJmeh+ejj+JMRp7YpREREdU4hhoz0KtlQ/zwdvk4G3V+CV5apcKvZ9Vil0VERFSjGGrMRFNXO2yf1B29WrrhfpkOb21Mxsr4qxxATERE9QZDjRmR21jh69e74rVuTSEIwMLYC3j/u1Mo1erFLo2IiKjaMdSYGUsLKRYM7YAPh7SHVAJsS05H2NoE5BaXil0aERFRtWKoMVNjuvvgqzFdYSezwPFrOXjx82O4kV0kdllERETVhqHGjD3bxh3fTewOL0cbXMsuwtDPjyLxeo7YZREREVULhhoz17aRHDsn94BfY0fkFpfh1S8TsONEuthlERERVTmGmnrA3cEGMeODMbCDJ0p1ekzfchKf7LnIN6OIiMisMNTUE7YyC6x4pQsm9mkOAFi2/wreiUnl0gpERGQ2GGrqEalUgpkD2mDRsE6wlErw48nbeGXNcdwr1IhdGhER0VNjqKmHXu6qwDdjgyC3sUTKzVwM/fwoLmcWiF0WERHRU2Goqae6N3fDjrd7oKlrA9zKuY8XVx7DkcvZYpdFRET0xBhq6rHmDe2xY1IPdPVxRkGJFmO+TsS3CTfFLouIiOiJMNTUcy52Mmx8U4kXOntDpxfwrx2nsTD2AvR6vhlFRER1C0MNwdrSAp+87IfpIa0AACvjr2L61lRotHwzioiI6g6GGgIASCQSvBPSEkte8oOlVIIfUm9jzNpE5N0vE7s0IiKiSmGoISPDAxrj6/CusLe2xPFrORi+8hgycu+LXRYREdFjMdTQA3q1bIitE4LhIbfG5axCvLDiKM7ezhO7LCIiokdiqKEKtfOSY8ekHmjt4YCsAg1eXqXCwUt3xS6LiIjooRhq6KG8nGyxbWIwujd3RVGpDm+sS8LWpFtil0VERFQhhhp6JLmNFdaFB+HFP175fv/7U/hk7yUuhklERLUOQw09lsxSio9f9sPkZ1sAAJbFXcZ7351CmU4vcmVERER/YaihSpFIJHg3tDX++0JHWEgl+C45HW+sS0JBCV/5JiKi2oGhhkzyirIJvgwLRAOZBQ5fzsZLq1RQ55WIXRYRERFDDZnu2Tbu2DI+GG721rigLsALnx/FBXW+2GUREVE9x1BDT6RjY0fsmNQdzRva4U5eCV5apULi9RyxyyIionqMoYaemMKlAb6f2B2BTctX+X71qwT8elYtdllERFRPMdTQU3FqUL7Kd0hbD5Rq9Zi4MRkxiTfFLouIiOohhhp6ajZWFlj1aheMCFRALwCztp/G8rjLnMuGiIhq1BOFmhUrVsDHxwc2NjZQKpVITEx8aNvt27cjMDAQTk5OsLOzg7+/PzZs2GA4XlZWhpkzZ6Jjx46ws7ODl5cXwsLCcPv2baN+fHx8IJFIjLaoqKgnKZ+qgaWFFFHDOhrmsvl47yXM+/EsdHoGGyIiqhkmh5otW7YgIiIC8+bNQ0pKCvz8/BAaGoqsrKwK27u4uOCDDz6ASqXCqVOnEB4ejvDwcPz6668AgOLiYqSkpGDOnDlISUnB9u3bcfHiRQwZMuSBvj766CPcuXPHsE2ZMsXU8qka/TmXzfzn2kEiAb5RpWFqzAlotDqxSyMionpAIpj4G4FSqUTXrl0RHR0NANDr9VAoFJgyZQpmzZpVqT66dOmCwYMHY8GCBRUeT0pKQlBQENLS0tCkSRMA5U9qpk2bhmnTpplSrkF+fj4cHR2Rl5cHuVz+RH1Q5f108jYitqaiTCege3NXfPFaABxsrMQui4iI6hhTvr9NelJTWlqK5ORkhISE/NWBVIqQkBCoVKrHni8IAuLi4nDx4kX07t37oe3y8vIgkUjg5ORktD8qKgqurq7o3LkzFi9eDK1Wa0r5VIOe8/PC168HwU5mgWNX72Hk6uO4W6ARuywiIjJjlqY0zs7Ohk6ng4eHh9F+Dw8PXLhw4aHn5eXlwdvbGxqNBhYWFvj888/Rr1+/CtuWlJRg5syZGDVqlFEimzp1Krp06QIXFxccO3YMs2fPxp07d/DJJ59U2I9Go4FG89eXaH4+J4eraT1buiFmfDBe/zoRZ2/nY/iqY/jmjSA0dbUTuzQiIjJDJoWaJ+Xg4IDU1FQUFhYiLi4OERERaNasGfr06WPUrqysDC+//DIEQcDKlSuNjkVERBj+3KlTJ8hkMkyYMAGRkZGwtrZ+4JqRkZH48MMPq+XzUOV1bOyI7yZ2R9jaBKTdK8awlSqsC++KDt6OYpdGRERmxqSfn9zc3GBhYYHMzEyj/ZmZmfD09Hz4RaRStGjRAv7+/pgxYwaGDx+OyMhIozZ/Bpq0tDTs3bv3sb+bKZVKaLVa3Lhxo8Ljs2fPRl5enmG7detW5T4kVTlfNzt8P7E72jaSI7tQg5Grj+PYlWyxyyIiIjNjUqiRyWQICAhAXFycYZ9er0dcXByCg4Mr3Y9erzf6aejPQHP58mXs27cPrq6uj+0jNTUVUqkU7u7uFR63traGXC432kg87g422DKhG7o1c0GhRovXv07Cz6fuiF0WERGZEZN/foqIiMCYMWMQGBiIoKAgLF26FEVFRQgPDwcAhIWFwdvb2/AkJjIyEoGBgWjevDk0Gg12796NDRs2GH5eKisrw/Dhw5GSkoJdu3ZBp9NBrS6fat/FxQUymQwqlQoJCQl49tln4eDgAJVKhenTp+PVV1+Fs7NzVd0LqmZyGyusCw9CxNZU7D6txuTNKcgpao/Xgn3ELo2IiMyAyaFmxIgRuHv3LubOnQu1Wg1/f3/ExsYaBg/fvHkTUulfD4CKioowadIkpKenw9bWFm3atMHGjRsxYsQIAEBGRgZ+/PFHAIC/v7/RtQ4cOIA+ffrA2toaMTExmD9/PjQaDXx9fTF9+nSjcTZUN9hYWWD5qC5wsTuDjcdvYs4PZ3G3QIPp/VpBIpGIXR4REdVhJs9TU1dxnpraRRAELIu7gk/3XQIAjApSYMHzHWBpwZU7iIjoL9U2Tw1RVZFIJHgnpCX+80IHSCXA5sRbmLQpBSVlnH2YiIieDEMNiWq0sik+H90FMksp9pzLRNjaROTdLxO7LCIiqoMYakh0Azo0wjdvBMHB2hKJ13Mw4gsVMvNLxC6LiIjqGIYaqhW6NXPFlgnBaOhgjQvqArz4+TFcu1sodllERFSHMNRQrdHOS47tE7vD180OGbn3MXyVCidv5YpdFhER1REMNVSrKFwa4Lu3gtGpsSNyikoxas1xHLx0V+yyiIioDmCooVrH1d4a347rhl4t3VBcqsPYdUnYeSJD7LKIiKiWY6ihWsne2hJfjemKIX5e0OoFTNuSiq+OXBe7LCIiqsUYaqjWkllKsXSEP8J7+AAAFuw6h6hfLqCezBdJREQmYqihWk0qlWDuP9vh/QGtAQCrDl7Fu9tOoUynF7kyIiKqbRhqqNaTSCSY1KcFFg3vBAupBN+npGPChmTcL+Xsw0RE9BeGGqozXg5UYPVrAbCxkmL/hSyM/vI4cotLxS6LiIhqCYYaqlP6tvXApjeVcLS1QsrNXAxfpcLt3Ptil0VERLUAQw3VOQFNXbDtrWA0crTBlaxCDFt5DJczC8Qui4iIRMZQQ3VSKw8HfD+xO1q42+NOXgmGr1IhOe13scsiIiIRMdRQneXlZIttE4LRuYkT8u6XYfSXxxF3PlPssoiISCQMNVSnOdvJsOlNJZ5t3RAlZXqM35CMbb/dErssIiISAUMN1XkNZJZYHRaIYV0aQ6cX8N53p7Ay/qrYZRERUQ1jqCGzYGUhxZKXOmHCM80AAAtjLyBy93nOPkxEVI8w1JDZkEgkmD2wLT4Y1BYA8MWha5j5/SloOfswEVG9wFBDZmdc72ZYNLwTpBJg62/pmPztCWi0nH2YiMjcMdSQWXo5UIHPRwdAZiFF7Fk13liXhEKNVuyyiIioGjHUkNka0MET68K7wk5mgaNX7mH0muP4vYjLKhARmSuGGjJr3Vu44dtx3eDcwAon0/Pw0hcq3MnjsgpEROaIoYbMnp/CyWhZheErVbh2t1DssoiIqIox1FC90MLdAd9N7I5mbnbIyL2Pl1apcCYjT+yyiIioCjHUUL3h7WSLrW8Fo4O3HPeKSjFq9XEkXLsndllERFRFGGqoXnGzt8bmcd2g9HVBgUaLsLWJXC+KiMhMMNRQveNgY4X1bwQhpK07NNry9aJ2nEgXuywiInpKDDVUL9lYWWDlqwF4sbM3dHoB07ecxLqj18Uui4iIngJDDdVb5etF+SG8hw8AYP5P57B03yWuF0VEVEcx1FC9JpVKMPef7RDRrxUAYOm+y/jwp3PQ6xlsiIjqGoYaqvckEgmm9m2JD4e0BwCsO3YDM7adRBkXwiQiqlMYaoj+MKa7D5aO8IeFVIIdJzIwcWMySsq4ECYRUV3BUEP0P4Z29sbq1wJgbSnFvvNZCFubiPySMrHLIiKiSmCoIfqbvm098M0bQXCwtkTi9RyMWn0c2YUascsiIqLHYKghqoCymSs2j+8GVzsZzt7Ox8urVMjI5UKYRES1GUMN0UN08HbEtreC4e1ki2vZRRi+8hiuZHEhTCKi2uqJQs2KFSvg4+MDGxsbKJVKJCYmPrTt9u3bERgYCCcnJ9jZ2cHf3x8bNmwwaiMIAubOnYtGjRrB1tYWISEhuHz5slGbnJwcjB49GnK5HE5OThg7diwKC/kFQ9WrWUN7fDcxGM0b2uFOXgle/kKFU+m5YpdFREQVMDnUbNmyBREREZg3bx5SUlLg5+eH0NBQZGVlVdjexcUFH3zwAVQqFU6dOoXw8HCEh4fj119/NbRZtGgRli1bhlWrViEhIQF2dnYIDQ1FSUmJoc3o0aNx9uxZ7N27F7t27cKhQ4cwfvz4J/jIRKZp5GiLbW91R6fGjsj5YyHMY1ezxS6LiIj+RiKYOH2qUqlE165dER0dDQDQ6/VQKBSYMmUKZs2aVak+unTpgsGDB2PBggUQBAFeXl6YMWMG3n33XQBAXl4ePDw8sG7dOowcORLnz59Hu3btkJSUhMDAQABAbGwsBg0ahPT0dHh5eT32mvn5+XB0dEReXh7kcrkpH5kIAFCo0WLc+t+gunYPMkspokd1Rv/2nmKXRURk1kz5/jbpSU1paSmSk5MREhLyVwdSKUJCQqBSqR57viAIiIuLw8WLF9G7d28AwPXr16FWq436dHR0hFKpNPSpUqng5ORkCDQAEBISAqlUioSEhAqvpdFokJ+fb7QRPQ17a0t8Hd4V/dp5oFSrx8RNKfgumQthEhHVFiaFmuzsbOh0Onh4eBjt9/DwgFqtfuh5eXl5sLe3h0wmw+DBg7F8+XL069cPAAznPapPtVoNd3d3o+OWlpZwcXF56HUjIyPh6Oho2BQKhSkflahCNlYWWDm6C4YHNIZOL+DdbSfx1REuhElEVBvUyNtPDg4OSE1NRVJSEv7zn/8gIiIC8fHx1XrN2bNnIy8vz7DdunWrWq9H9YelhRSLhnXC2J6+AIAFu87h4z0XuRAmEZHILE1p7ObmBgsLC2RmZhrtz8zMhKfnw8cWSKVStGjRAgDg7++P8+fPIzIyEn369DGcl5mZiUaNGhn16e/vDwDw9PR8YCCyVqtFTk7OQ69rbW0Na2trUz4eUaVJpRL83+C2cG5ghSV7LmH5/ivIu1+G+c+1h1QqEbs8IqJ6yaQnNTKZDAEBAYiLizPs0+v1iIuLQ3BwcKX70ev10GjKZ2j19fWFp6enUZ/5+flISEgw9BkcHIzc3FwkJycb2uzfvx96vR5KpdKUj0BUZSQSCSb/oyUWDO0AiQT4RpWG6VtTuRAmEZFITHpSAwAREREYM2YMAgMDERQUhKVLl6KoqAjh4eEAgLCwMHh7eyMyMhJA+diWwMBANG/eHBqNBrt378aGDRuwcuVKAOVfDNOmTcO///1vtGzZEr6+vpgzZw68vLwwdOhQAEDbtm0xYMAAjBs3DqtWrUJZWRkmT56MkSNHVurNJ6Lq9Fq3ppDbWGLG1pP4IfU2covL8PnoLrCzNvk/LyIiegom/6/uiBEjcPfuXcydOxdqtRr+/v6IjY01DPS9efMmpNK/HgAVFRVh0qRJSE9Ph62tLdq0aYONGzdixIgRhjbvv/8+ioqKMH78eOTm5qJnz56IjY2FjY2Noc2mTZswefJk9O3bF1KpFMOGDcOyZcue5rMTVZnn/b0ht7HCxE3JOHjpLl5ZcxxrX+8KV3v+BEpEVFNMnqemruI8NVQTUm7+jjfWJSG3uAy+bnb45o0gKFwaiF0WEVGdVW3z1BDRo3Vp4ozv3uoObydbXM8uwosrj+FMRp7YZRER1QsMNURVrIW7PbZP6o42ng64W6DByNXHcewKl1UgIqpuDDVE1cBDboMtE4Kh9HVBoUaLMV8n4qeTt8Uui4jIrDHUEFUTR1srrH8jCIM6eqJMJ2DK5hNYy9mHiYiqDUMNUTWysbLA8lFdEBbcFADw0a5ziPrlAmcfJiKqBgw1RNXMQirBh0Pa473Q1gCAVQevYsa2k5ykj4ioijHUENUAiUSCt59tgcXDO8FCKsH2lAyMXf8bijRasUsjIjIbDDVENeilQAXWhAXAxkqKQ39M0nevUCN2WUREZoGhhqiG/aONBzaP6wbnBlY4mZ6HYSuP4ea9YrHLIiKq8xhqiETQuYkzvptYPknfjXvFnKSPiKgKMNQQiaR5w/JJ+to2kiO7sHySviOXOUkfEdGTYqghElH5JH3dENzMFYUaLcLXJeKH1AyxyyIiqpMYaohEJrexwro3umJwp0Yo0wl4JyYVXx6+JnZZRER1DkMNUS1gbWmB5SM74/XuPgCAf/98HpG7z0Ov5yR9RESVxVBDVEtIpRLMe64dZg5oAwD44tA1zNh2EqVaTtJHRFQZDDVEtYhEIsHEPs2x5CU/WEgl2HEiA2PXJ3GSPiKiSmCoIaqFhgc0xpdjAmFrZYHDl7Mxas1xZHOSPiKiR2KoIaqlnm3tjs3ju8HFToZTf0zSl3avSOyyiIhqLYYaolrMX+GE794KRmNnW6TdK8YwTtJHRPRQDDVEtVyzhvbYPrE72jWSI7uwFCO+UOHw5btil0VEVOsw1BDVAe5/TNLXvbkrikp1eGNdEifpIyL6G4YaojrCwcYKX4d3xT85SR8RUYUYaojqEGtLCywb2RnhPXwAlE/S95+fz3GSPiIiMNQQ1TlSqQRz/9kOsweWT9K35vB1RGxN5SR9RFTvMdQQ1UESiQQTnmmOT172g6VUgp2ptzF2fRIKOUkfEdVjDDVEddiLXcon6WsgK5+k7+VVKmTml4hdFhGRKBhqiOq4Pq3dsXlcN7jZy3DuTj5eWHEUF9UFYpdFRFTjGGqIzICfwgk7JvVAs4Z2uJ1XguErj+HYlWyxyyIiqlEMNURmQuHSANsndkdXH2cUaLQY83Uitqeki10WEVGNYaghMiNODWTYMFZpmMsmYutJLI+7DEHgK99EZP4YaojMjI1V+Vw2E55pBgD4eO8lzN5+GmU6vvJNROaNoYbIDEmlEswe2BYLnm8PqQSISbqFset/4yvfRGTWGGqIzNhrwT5Y/VogbK0scOjSXb7yTURmjaGGyMyFtPNAzHi+8k1E5o+hhqge4CvfRFQfMNQQ1RN/vvId5OPCV76JyCwx1BDVI04NZPhmbJDRK9/L+Mo3EZkJhhqieubvr3x/svcSIraehEarE7kyIqKn80ShZsWKFfDx8YGNjQ2USiUSExMf2nbNmjXo1asXnJ2d4ezsjJCQkAfaSySSCrfFixcb2vj4+DxwPCoq6knKJ6r3/nzl+z8vdICFVIIdJzLwypoEZBdqxC6NiOiJmRxqtmzZgoiICMybNw8pKSnw8/NDaGgosrKyKmwfHx+PUaNG4cCBA1CpVFAoFOjfvz8yMjIMbe7cuWO0rV27FhKJBMOGDTPq66OPPjJqN2XKFFPLJ6L/MVrZFOvDg+BgY4nktN8xdMVRXMrkm1FEVDdJBBN/TFcqlejatSuio6MBAHq9HgqFAlOmTMGsWbMee75Op4OzszOio6MRFhZWYZuhQ4eioKAAcXFxhn0+Pj6YNm0apk2bZkq5Bvn5+XB0dEReXh7kcvkT9UFkrq5kFWLs+iSk3SuGvbUlol/pjD6t3cUui4jIpO9vk57UlJaWIjk5GSEhIX91IJUiJCQEKpWqUn0UFxejrKwMLi4uFR7PzMzEzz//jLFjxz5wLCoqCq6urujcuTMWL14Mrfbhs6NqNBrk5+cbbURUsRbu9tg5qQeCfF1QqNHijXVJWH/shthlERGZxKRQk52dDZ1OBw8PD6P9Hh4eUKvVlepj5syZ8PLyMgpG/2v9+vVwcHDAiy++aLR/6tSpiImJwYEDBzBhwgT897//xfvvv//Q60RGRsLR0dGwKRSKStVHVF8528mwcawSLwU0hl4A5v14FnN2noGWa0YRUR1hWZMXi4qKQkxMDOLj42FjY1Nhm7Vr12L06NEPHI+IiDD8uVOnTpDJZJgwYQIiIyNhbW39QD+zZ882Oic/P5/BhugxZJZSLBreCc3d7bEw9gI2HE/DjXtFiH6lCxxtrcQuj4jokUx6UuPm5gYLCwtkZmYa7c/MzISnp+cjz12yZAmioqKwZ88edOrUqcI2hw8fxsWLF/Hmm28+thalUgmtVosbN25UeNza2hpyudxoI6LHk0gkeOuZ5lj1agBsrSxw+HI2hq08hpv3isUujYjokUwKNTKZDAEBAUYDePV6PeLi4hAcHPzQ8xYtWoQFCxYgNjYWgYGBD2331VdfISAgAH5+fo+tJTU1FVKpFO7uHMxIVB1C23ti21vB8JTb4EpWIZ5fcQSJ13PELouI6KFMfqU7IiICa9aswfr163H+/HlMnDgRRUVFCA8PBwCEhYVh9uzZhvYLFy7EnDlzsHbtWvj4+ECtVkOtVqOwsNCo3/z8fGzbtq3CpzQqlQpLly7FyZMnce3aNWzatAnTp0/Hq6++CmdnZ1M/AhFVUgdvR/wwuQc6ejvi9+IyjP7yOLb9dkvssoiIKmTymJoRI0bg7t27mDt3LtRqNfz9/REbG2sYPHzz5k1IpX9lpZUrV6K0tBTDhw836mfevHmYP3++4e8xMTEQBAGjRo164JrW1taIiYnB/PnzodFo4Ovri+nTpxuNmSGi6uEht8HWCcGYsS0Vu0+r8d53p3BRXYBZA9vA0oKTkhNR7WHyPDV1FeepIXo6er2ApfsuYdn+KwCA3q0aYvmozhxATETVqtrmqSGi+ksqlSCif2useKULbKykOHTpLl5YcRRX7xY+/mQiohrAUENEJhncqRG+e6s7vBxtcC27CENXHMWBixUvk0JEVJMYaojIZB28HfHjlJ4IbOqMghItxq5LwppD11BPfs0molqKoYaInoibvTW+HdcNIwIV0AvAf3afx4xtJ1FSphO7NCKqpxhqiOiJySyliBrWEfOfawcLqQTbUzIwcvVxZOWXiF0aEdVDDDVE9FQkEgle7+GL9eFBcLS1QuqtXDwXfQQnb+WKXRoR1TMMNURUJXq2dMMPb/dAS3d7ZOZr8NIXKuw8kSF2WURUjzDUEFGV8XGzw/ZJ3dG3jTtKtXpM25KKqF8uQKfnAGIiqn4MNURUpRxsrLA6LBCT+jQHAKw6eBXjvvkNBSVlIldGROaOoYaIqpyFVIL3B7TBZyP9YW0pxf4LWXjh82O4xon6iKgaMdQQUbV53t/beKXv6KOIO58pdllEZKYYaoioWnVq7IQfp/RAVx9nFGi0GLv+Nyzddwl6jrMhoirGUENE1c7dwQab3uyGsOCmAICl+y5j/IZk5HOcDRFVIYYaIqoRMkspPnq+AxYP7wSZpRT7zmdi6IqjuJJVIHZpRGQmGGqIqEa9FKjAd28Fo5GjDa7dLcLz0Ufx61m12GURkRlgqCGiGtepsRN+mtITSl8XFJXqMGFDMj7ec5HjbIjoqTDUEJEo3OytsfFNJd7o4QsAWL7/CsauT0LefY6zIaInw1BDRKKxspBi7nPt8OkIP1hbSnHg4l08H30EF9UcZ0NEpmOoISLRvdC5Mb6f2B3eTra4ca8YL3x+FLtP3xG7LCKqYxhqiKhW6ODtiJ+m9ESPFq4oLtVh0qYULIzlulFEVHkMNURUa7jYybA+PAjjezcDAKyMv4rXv05EbnGpyJURUV3AUENEtYqlhRT/GtQWy0Z1ho2VFIcvZ+O56CM4fydf7NKIqJZjqCGiWmmInxe2T+wBhYstbuXcx4ufH8OPJ2+LXRYR1WIMNURUa7XzkuOnyT3Rq6Ub7pfpMHXzCXz00zmU6fRil0ZEtRBDDRHVak4NZFgXHoRJfZoDANYevY6Rq4/jTt59kSsjotqGoYaIaj0LqQTvD2iDNWGBcLCxRHLa7/jnsiM4cjlb7NKIqBZhqCGiOqNfOw/smtIT7RrJca+oFK+tTUD0/stcXoGIADDUEFEd09TVDtsndceIQAUEAViy5xLGrk/ia99ExFBDRHWPjZUFFg7vhEXDOxmWVxi87AhOpeeKXRoRiYihhojqrJcDFdg+qTuaujZARu59DF+pwqaENAgCf44iqo8YaoioTmvv5YgfJ/dE/3YeKNXp8cGOM5ix9SSKS7Vil0ZENYyhhojqPEdbK3zxWgBmD2wDC6kE209k4IUVx3D1bqHYpRFRDWKoISKzIJFIMOGZ5vj2TSUaOljjYmYBno/mat9E9QlDDRGZFWUzV/w8pSeCfF1QqNFi0qYULNjFWYiJ6gOGGiIyO+5yG3z7phITnilf7furI+WzEKvzSkSujIiqE0MNEZklSwspZg9si9WvBRhmIR687DCOXuEsxETmiqGGiMxa//ae2DWlJ9r+OQvxV5yFmMhcPVGoWbFiBXx8fGBjYwOlUonExMSHtl2zZg169eoFZ2dnODs7IyQk5IH2r7/+OiQSidE2YMAAozY5OTkYPXo05HI5nJycMHbsWBQW8s0GInq8pq522DGpO14ObAz9H7MQv/nNb5yFmMjMmBxqtmzZgoiICMybNw8pKSnw8/NDaGgosrKyKmwfHx+PUaNG4cCBA1CpVFAoFOjfvz8yMjKM2g0YMAB37twxbJs3bzY6Pnr0aJw9exZ79+7Frl27cOjQIYwfP97U8omonrKxssCi4X5YNKx8FuL9F7Lwz+WchZjInEgEE6feVCqV6Nq1K6KjowEAer0eCoUCU6ZMwaxZsx57vk6ng7OzM6KjoxEWFgag/ElNbm4udu7cWeE558+fR7t27ZCUlITAwEAAQGxsLAYNGoT09HR4eXk99rr5+flwdHREXl4e5HJ5JT8tEZmjs7fzMHFjCm7mFENmIcWc59rhVWUTSCQSsUsjor8x5fvbpCc1paWlSE5ORkhIyF8dSKUICQmBSqWqVB/FxcUoKyuDi4uL0f74+Hi4u7ujdevWmDhxIu7du2c4plKp4OTkZAg0ABASEgKpVIqEhIQKr6PRaJCfn2+0EREB5bMQ/zSlJ/r9MQvxnJ1n8E5MKgo1nIWYqC4zKdRkZ2dDp9PBw8PDaL+HhwfUanWl+pg5cya8vLyMgtGAAQPwzTffIC4uDgsXLsTBgwcxcOBA6HQ6AIBarYa7u7tRP5aWlnBxcXnodSMjI+Ho6GjYFAqFKR+ViMyco60VVr8WgP8b3BaWUgl+PHkbQ5YfwQU1/w8QUV1Vo28/RUVFISYmBjt27ICNjY1h/8iRIzFkyBB07NgRQ4cOxa5du5CUlIT4+Pgnvtbs2bORl5dn2G7dulUFn4CIzIlEIsGbvZphy4RuaORog2vZRRi64ii2/sb/vSCqi0wKNW5ubrCwsEBmZqbR/szMTHh6ej7y3CVLliAqKgp79uxBp06dHtm2WbNmcHNzw5UrVwAAnp6eDwxE1mq1yMnJeeh1ra2tIZfLjTYioooENHXBz1N74ZlWDVFSpsf7353Cu9tO4n6pTuzSiMgEJoUamUyGgIAAxMXFGfbp9XrExcUhODj4oectWrQICxYsQGxsrNG4mIdJT0/HvXv30KhRIwBAcHAwcnNzkZycbGizf/9+6PV6KJVKUz4CEVGFXOxk+Pr1rngvtDWkEuC75HQMXXEUV7I4dQRRXWHyz08RERFYs2YN1q9fj/Pnz2PixIkoKipCeHg4ACAsLAyzZ882tF+4cCHmzJmDtWvXwsfHB2q1Gmq12jDHTGFhId577z0cP34cN27cQFxcHJ5//nm0aNECoaGhAIC2bdtiwIABGDduHBITE3H06FFMnjwZI0eOrNSbT0RElSGVSvD2sy2w6c1uhkUxh0QfwQ+pGY8/mYhEZ3KoGTFiBJYsWYK5c+fC398fqampiI2NNQwevnnzJu7c+WtV3JUrV6K0tBTDhw9Ho0aNDNuSJUsAABYWFjh16hSGDBmCVq1aYezYsQgICMDhw4dhbW1t6GfTpk1o06YN+vbti0GDBqFnz55YvXr1035+IqIHBDd3xc9TeyK4mSuKS3V4JyYVH+w4jZIy/hxFVJuZPE9NXcV5aojIVDq9gM/2XcLyA1cgCEB7Lzk+H90FTV3txC6NqN6otnlqiIjqEwupBBH9W2NdeBBc7GQ4ezsf/1x2BLFn7jz+ZCKqcQw1RESP8Uyrhvh5ak8ENnVGgUaLtzam4MOfzqJUqxe7NCL6Hww1RESV0MjRFpvHd8OE3s0AAF8fvYFX1hxHVkGJyJUR0Z8YaoiIKsnKQorZg9piTVggHGws8Vva7xiy/ChO3soVuzQiAkMNEZHJ+rXzwA9v90DzhnZQ55fgpS9U+D45XeyyiOo9hhoioifQrKE9dr7dAyFt3VGq1WPGtpOYsfUkF8UkEhFDDRHRE3KwscLq1wIxLaQlpBLg+5R0DPrsME7c/F3s0ojqJYYaIqKnIJVKMC2kFWLGB8PbyRY3c4oxfJUKy+MuQ6evF9OAEdUaDDVERFUgyNcFu9/phSF+XtDpBXy89xJGrT6O9N+LxS6NqN5gqCEiqiKOtlb4bKQ/Ph3hB3trSyTeyMHAzw7jx5O3xS6NqF5gqCEiqkISiQQvdG6M3VN7oXMTJxSUaDF18wlEbE1FQUmZ2OURmTWGGiKiatDEtQG2TQjG1L7lg4i3p2Rg8LIjSOEgYqJqw1BDRFRNLC2kiOjXClsn/DWI+KVVKny27zK0Oi6xQFTVGGqIiKpZoI8LfpnWC8/7lw8i/nTfJYxcfRy3cjiImKgqMdQQEdUAuY0VPhvZGUtH+MPeunyJhUGfHcYPqRlil0ZkNhhqiIhq0NDO3vjlnV4I+GPF73diUjF9SyryOYiY6Kkx1BAR1TCFSwNsGd/NMBPxjhMZGPTZYSSn5YhdGlGdxlBDRCQCSwsppoW0wra3gqFwsUX67/fx0ioVPt17iYOIiZ4QQw0RkYgCmrpg99ReeLGzN/QC8FncZbz8hYqDiImeAEMNEZHIHGys8MkIf3w20h8O1pZIuZmLgZ8dxo4T6WKXRlSnMNQQEdUSz/t7Y/c7vdDVxxmFGi2mbzmJd2JOcBAxUSUx1BAR1SIKlwbYPK4bIvq1goVUgh9Sb2Pg0sNIusFBxESPw1BDRFTLWFpIMbVvS2x7KxhNXBogI/c+Rnyhwid7LnIQMdEjMNQQEdVSXZo44+epPTGsS2PoBWDZ/it46QsV0u4ViV0aUa3EUENEVIs52Fjh45f9sHxUZzjYWOLEzVwM+uwwvk9OhyAIYpdHVKsw1BAR1QHP+XkhdlpvBPm4oKhUhxnbTmLK5hPILS4VuzSiWoOhhoiojvB2ssXm8d3wXmhrWEgl2HXqDkKXHsLBS3fFLo2oVmCoISKqQyykErz9bAtsn9gdzRraITNfgzFrE/HBjtMo0mjFLo9IVAw1RER1kJ/CCT9P6YXwHj4AgE0JNzFo2WH8xle/qR5jqCEiqqNsZRaY91x7fPumEl6ONki7V4yXv1Ah6pcL0Gh1YpdHVOMYaoiI6rjuLdwQO7234dXvVQev4vnoozh7O0/s0ohqFEMNEZEZkP/x6veqVwPgaifDBXUBhq44ik/3XkKplhP2Uf3AUENEZEYGdPDEr9N7o387D5TpBHwWdxnPLT+C1Fu5YpdGVO0YaoiIzIybvTW+eC0Ay0d1hqudDBczC/Di50fxn5/P4X4px9qQ+WKoISIyQxKJBM/5eWFvxDMY6u8FvQCsOXwdAz47BNXVe2KXR1QtGGqIiMyYi50MS0d2xtrXA+EpL39DatSa45i9/TTyS8rELo+oSjHUEBHVA/9o44E9Eb3xirIJAGBz4k30/+QQ4s5nilwZUdVhqCEiqifkNlb47wsdsXlcNzR1bQB1fgnGrv8N78ScwL1CjdjlET21Jwo1K1asgI+PD2xsbKBUKpGYmPjQtmvWrEGvXr3g7OwMZ2dnhISEGLUvKyvDzJkz0bFjR9jZ2cHLywthYWG4ffu2UT8+Pj6QSCRGW1RU1JOUT0RUrwU3d0XsO70xvnczSCXAD6m30e/TQ/jx5G2u/E11msmhZsuWLYiIiMC8efOQkpICPz8/hIaGIisrq8L28fHxGDVqFA4cOACVSgWFQoH+/fsjIyMDAFBcXIyUlBTMmTMHKSkp2L59Oy5evIghQ4Y80NdHH32EO3fuGLYpU6aYWj4REaF8NuJ/DWqLHZN6oLWHA3KKSjF18wmM++Y3qPNKxC6P6IlIBBNjuVKpRNeuXREdHQ0A0Ov1UCgUmDJlCmbNmvXY83U6HZydnREdHY2wsLAK2yQlJSEoKAhpaWlo0qT8918fHx9MmzYN06ZNM6Vcg/z8fDg6OiIvLw9yufyJ+iAiMkelWj1Wxl9F9IHLKNMJcLC2xL8Gt8XIrgpIJBKxy6N6zpTvb5Oe1JSWliI5ORkhISF/dSCVIiQkBCqVqlJ9FBcXo6ysDC4uLg9tk5eXB4lEAicnJ6P9UVFRcHV1RefOnbF48WJotQ9fkVaj0SA/P99oIyKiB8kspXgnpCV2TekFP4UTCjRazN5+Gq+sSUDavSKxyyOqNJNCTXZ2NnQ6HTw8PIz2e3h4QK1WV6qPmTNnwsvLyygY/a+SkhLMnDkTo0aNMkpkU6dORUxMDA4cOIAJEybgv//9L95///2HXicyMhKOjo6GTaFQVKo+IqL6qrWnA7ZP7I7/G9wWNlZSqK7dQ+jSQ/jy8DXo9BxrQ7WfST8/3b59G97e3jh27BiCg4MN+99//30cPHgQCQkJjzw/KioKixYtQnx8PDp16vTA8bKyMgwbNgzp6emIj49/5GOmtWvXYsKECSgsLIS1tfUDxzUaDTSav0bz5+fnQ6FQ8OcnIqJKSLtXhFnfn4bqWvlEfX4KJ/zf4Lbo6vPwp+xE1aHafn5yc3ODhYUFMjON5zXIzMyEp6fnI89dsmQJoqKisGfPnocGmpdffhlpaWnYu3fvYwtXKpXQarW4ceNGhcetra0hl8uNNiIiqpymrnb4dpwSkS92hIO1JU7eysVLq1R4c30SLqoLxC6PqEImhRqZTIaAgADExcUZ9un1esTFxRk9ufm7RYsWYcGCBYiNjUVgYOADx/8MNJcvX8a+ffvg6ur62FpSU1MhlUrh7u5uykcgIqJKkkgkGBXUBPtmPINRQU1gIZVg3/ksDPjsEObsPIOSMq4jRbWLpaknREREYMyYMQgMDERQUBCWLl2KoqIihIeHAwDCwsLg7e2NyMhIAMDChQsxd+5cfPvtt/Dx8TGMvbG3t4e9vT3KysowfPhwpKSkYNeuXdDpdIY2Li4ukMlkUKlUSEhIwLPPPgsHBweoVCpMnz4dr776KpydnavqXhARUQU85DaIfLEj3uzli0/2XMLPp+9gw/E0pNz8HStHB6CJawOxSyQC8ASvdANAdHQ0Fi9eDLVaDX9/fyxbtgxKpRIA0KdPH/j4+GDdunUAyl/FTktLe6CPefPmYf78+bhx4wZ8fX0rvM6BAwfQp08fpKSkYNKkSbhw4QI0Gg18fX3x2muvISIiosLxNBXhK91ERFXj8OW7eCcmFTlFpXC0tcL8Ie0wxM8bFlK+/k1Vz5Tv7ycKNXURQw0RUdW5nXsfkzalIPVWLgBA4WKL/xvcDqHtHz2+kshU1TZQmIiICAC8nGyxZUI3vBfaGo62VriVcx8TNiRj4sZk3C3gOlIkDoYaIiJ6ItaWFnj72RY4Prsv3n62OSylEvxyRo3+nx7ErlO3H98BURVjqCEioqdiK7PAe6Ft8MPkHmjbSI7fi8sw+dvydaSu3S0UuzyqRxhqiIioSrT3csQPb/fA1L4tYSGVYO+5TPT/9BDm/nAGvxeVil0e1QMMNUREVGVkllJE9GuFX97phb5t3KHVC/hGlYbQpYdw8NJdscsjM8dQQ0REVa6VhwO+er0rvh2nRPOGdsgq0GDM2kTM2HoS2YUcSEzVg6GGiIiqTffmbtg1pRde7+4DAPg+JR3/WBKPjcfTuEgmVTmGGiIiqla2MgvMH9Ie2yd1R3svOfJLtPi/nWfw4udHcTo9T+zyyIxw8j0iIqoxWp0eG4+n4eM9l1Cg0UIiAV5VNsW7f8x3Q/R3nHyPiIhqJUsLKV7v4Yu4Gc9gqL8XBAHYcDwNfT+Ox/aUdNST/59N1YRPaoiISDTHrmZjzs4zuHq3CACg9HXBv4d2QEsPB5Ero9qCaz9VgKGGiKh2KtXq8eWRa1gWdxklZXpYSiUY090HU/u25E9SxFBTEYYaIqLa7VZOMT786Rz2nc8EALjYyRDRrxVGBTXhCuD1GENNBRhqiIjqhoOX7mLBrnO4klW+xEIbTwfMfa4dujd3E7kyEgNDTQUYaoiI6o4ynR6bjqfh032XkXe/DAAQ2t4D/xrUFk1d7USujmoSQ00FGGqIiOqe34tKsXTfJWxMuAmdXoDMQorwnj6Y/GwLONhwvE19wFBTAYYaIqK661JmARbsOofDl7MBAG72MrzbvzVeClRwvI2ZY6ipAEMNEVHdJggCDlzMwr93nce17PJXwNs1kmPOP9shuLmryNVRdWGoqQBDDRGReSjV6rHheBo+23cJ+SVaAMCA9p7416C2aOLaQOTqqKox1FSAoYaIyLzkFJXi072XsCkhDXoBHG9jphhqKsBQQ0Rkni6qC/Dvn43H20T0a42XAxvD0oKrAdV1DDUVYKghIjJfgiBg/4Us/Ofnv8bbtPZwwAeD26J3q4YiV0dPg6GmAgw1RETmr1Srx6aENCz9n/lt+rRuiA8GteV6UnUUQ00FGGqIiOqP3OJSLN9/BeuP3YBWL8BCKsErQU0wLaQlXO2txS6PTMBQUwGGGiKi+ud6dhEid5/HnnPl60k52Fhiyj9aYEx3H1hbWohcHVUGQ00FGGqIiOov1dV7+PfP53D2dj4AQOFii9kD22JgB09IJJy8rzZjqKkAQw0RUf2m0wvYnpKOxb9eRFaBBgDQ0dsR7/Rtib5t3RluaimGmgow1BAREQAUabT44tA1rDl0DffLdACADt5yTPlHS/Rr6wEpl12oVRhqKsBQQ0RE/+teoQZrDl/HN6obKC4tDzc+rg3wencfDA9UwN7aUuQKCWCoqRBDDRERVSSnqBRfHr6GjcfTDMsuuNlb41+D2uCFzt78WUpkDDUVYKghIqJHKS7V4vuUDHx5+BrS7hUDAHq1dMOSl/zgIbcRubr6i6GmAgw1RERUGRqtDl8duY5lcZdRUqaHcwMrTPlHS4wMUqCBjD9J1TSGmgow1BARkSmuZBVg6uZUnLtT/hq4UwMrjFY2wZhgH7jzyU2NYaipAEMNERGZqlSrx7bkW1h96K+fpGyspJj4TAtMeKYZbKw4gV91Y6ipAEMNERE9KZ1ewN5zmfji0FWcuJkLoHwCvw+HtMc/2niIW5yZY6ipAEMNERE9LUEQ8PPpO/jPz+dxJ68EQPmCme/2b40O3o4iV2eeGGoqwFBDRERVpUijxWdxl/HVkevQ6cu/Rgd28EREv1ZcDbyKmfL9LX2SC6xYsQI+Pj6wsbGBUqlEYmLiQ9uuWbMGvXr1grOzM5ydnRESEvJAe0EQMHfuXDRq1Ai2trYICQnB5cuXjdrk5ORg9OjRkMvlcHJywtixY1FYWPgk5RMRET0VO2tL/GtQW8RFPIOh/l6QSIBfzqjRf+khTN+SirR7RWKXWC+ZHGq2bNmCiIgIzJs3DykpKfDz80NoaCiysrIqbB8fH49Ro0bhwIEDUKlUUCgU6N+/PzIyMgxtFi1ahGXLlmHVqlVISEiAnZ0dQkNDUVJSYmgzevRonD17Fnv37sWuXbtw6NAhjB8//gk+MhERUdXwcbPD0pGdEftObwxo7wlBAHacyEDfjw9i9vbTuJ17X+wS6xWTf35SKpXo2rUroqOjAQB6vR4KhQJTpkzBrFmzHnu+TqeDs7MzoqOjERYWBkEQ4OXlhRkzZuDdd98FAOTl5cHDwwPr1q3DyJEjcf78ebRr1w5JSUkIDAwEAMTGxmLQoEFIT0+Hl5fXY6/Ln5+IiKi6nU7Pw5I9F3Hw0l0AgMxSitHKJpjUpwUaOliLXF3dVG0/P5WWliI5ORkhISF/dSCVIiQkBCqVqlJ9FBcXo6ysDC4uLgCA69evQ61WG/Xp6OgIpVJp6FOlUsHJyckQaAAgJCQEUqkUCQkJpnwEIiKiatOxsSPWvxGEbW8FQ+nrglKtHl8fvYHeiw5gYewF5BaXil2iWTMp1GRnZ0On08HDw/j1NQ8PD6jV6kr1MXPmTHh5eRlCzJ/nPapPtVoNd3d3o+OWlpZwcXF56HU1Gg3y8/ONNiIioprQ1ccFMeO7YcPYIPgpnHC/TIeV8VfRa+EBLIu7jEKNVuwSzdITDRR+UlFRUYiJicGOHTtgY1O9szFGRkbC0dHRsCkUimq9HhER0f+SSCTo1bIhdk7qjjVhgWjj6YACjRaf7L2EXgv3Y/Whq7j/x+rgVDVMCjVubm6wsLBAZmam0f7MzEx4eno+8twlS5YgKioKe/bsQadOnQz7/zzvUX16eno+MBBZq9UiJyfnodedPXs28vLyDNutW7cq9yGJiIiqkEQiQb92Htg9tReWj+qMZm52+L24DP/dfQHPLD6Ab1Q3oNEy3FQFk0KNTCZDQEAA4uLiDPv0ej3i4uIQHBz80PMWLVqEBQsWIDY21mhcDAD4+vrC09PTqM/8/HwkJCQY+gwODkZubi6Sk5MNbfbv3w+9Xg+lUlnhNa2trSGXy402IiIisUilEjzn54U903tj0fBO8HayRVaBBnN/OIt/LDmIrUm3oNXpxS6zTjP57actW7ZgzJgx+OKLLxAUFISlS5di69atuHDhAjw8PBAWFgZvb29ERkYCABYuXIi5c+fi22+/RY8ePQz92Nvbw97e3tAmKioK69evh6+vL+bMmYNTp07h3Llzhp+pBg4ciMzMTKxatQplZWUIDw9HYGAgvv3220rVzbefiIioNinV6rEl6SaW77+CrAINAKCZmx2m9WuFf3ZsBKlUInKFtUO1zygcHR2NxYsXQ61Ww9/fH8uWLTM8MenTpw98fHywbt06AICPjw/S0tIe6GPevHmYP38+gPLJ9+bNm4fVq1cjNzcXPXv2xOeff45WrVoZ2ufk5GDy5Mn46aefIJVKMWzYMCxbtswQjB6HoYaIiGqj+6U6bDyehpUHryKnqPztqDaeDojo1wr92nlAIqnf4YbLJFSAoYaIiGqzQo0WXx+5jtWHr6GgpPztKL/GjpjRvzV6tXSrt+GGoaYCDDVERFQX5BaXYvWha/j66A3cLysfQBzk64L3Qlujq4+LyNXVPIaaCjDUEBFRXZJdqMHnB65iY0IaSrXlA4h7t2qId/u3QqfGTuIWV4MYairAUENERHXRnbz7WL7/SvnbUX+sCB7a3gMR/Vqjtaf5rwjOUFMBhhoiIqrL0u4V4bN9l7EjNQOCAEgkwBA/L0wLaQVfNzuxy6s2DDUVYKghIiJzcDmzAJ/uu4Tdp8uXCbKQSvBiZ2+80dMXbRuZ3/cbQ00FGGqIiMicnMnIwyd7L2H/hb9m3A/yccGrwU3Rv50HbKwsRKyu6jDUVIChhoiIzFHKzd/x1ZHriD2jhu6PMTeOtlZ4o4cvJjzTrM6HG4aaCjDUEBGROVPnlWBz4k1s++0WbueVAAAaO9si6sVO6NnSTeTqnhxDTQUYaoiIqD7Q6QX8fPoOInefx50/ws0/2rhjfO9mUPq61LlJ/BhqKsBQQ0RE9UlxqRaRuy9gY0Ia/vymb+LSAEP9vfBSoAIKlwbiFlhJDDUVYKghIqL66Hp2EVYfuoYfUzNQVFo+Q7GlVIJRQU0wtW9LNHSwFrnCR2OoqQBDDRER1Wf3S3XYez4TW5Ju4uiVewCABjILvNmrGV7v7gMXO5nIFVaMoaYCDDVERETlVFfvYWHsBaTeygUAyCykCO3giVFdFejWzBVSae0Zd8NQUwGGGiIior8IgoBfzqjxefwVnMnIN+xv6toAr3VriteCm8LaUvzXwRlqKsBQQ0REVLEzGXnYnHgTP6TeRqFGC6B8UPE7fVticKdGos51w1BTAYYaIiKiRysu1WLnidtYuu8Ssgo0AACnBlYY3qUxxnT3EeWNKYaaCjDUEBERVU6RRot1x25g0/E0w0R+UgkwsEMjjOvdDP4KpxqrhaGmAgw1REREptHpBcRfzMJ6VRoOXbpr2K/0dcGEZ5rhmVbusKjmQcUMNRVgqCEiInpyF9T5WHPoOn5IzYD2jzWmvJ1s8YqyCV4KbAx3B5tquS5DTQUYaoiIiJ7e7dz7WHvkOrb+dgv5JeWDii2lEvRv74FXuzVF9+ZVu84UQ00FGGqIiIiqTkmZDrtO3cG3CWlIuZkLABjcsRFWjO5Spdcx5fvbskqvTERERPWCjZUFhgc0xvCAxrigzse3CTcxuGMjUWtiqCEiIqKn0sZTjo+e7yB2GZCKXQARERFRVWCoISIiIrPAUENERERmgaGGiIiIzAJDDREREZkFhhoiIiIyCww1REREZBYYaoiIiMgsMNQQERGRWWCoISIiIrPAUENERERmgaGGiIiIzAJDDREREZmFerNKtyAIAID8/HyRKyEiIqLK+vN7+8/v8UepN6GmoKAAAKBQKESuhIiIiExVUFAAR0fHR7aRCJWJPmZAr9fj9u3bcHBwgEQiqdK+8/PzoVAocOvWLcjl8irtm/7C+1wzeJ9rDu91zeB9rhnVdZ8FQUBBQQG8vLwglT561Ey9eVIjlUrRuHHjar2GXC7nfzA1gPe5ZvA+1xze65rB+1wzquM+P+4JzZ84UJiIiIjMAkMNERERmQWGmipgbW2NefPmwdraWuxSzBrvc83gfa45vNc1g/e5ZtSG+1xvBgoTERGReeOTGiIiIjILDDVERERkFhhqiIiIyCww1BAREZFZYKh5SitWrICPjw9sbGygVCqRmJgodkl1yqFDh/Dcc8/By8sLEokEO3fuNDouCALmzp2LRo0awdbWFiEhIbh8+bJRm5ycHIwePRpyuRxOTk4YO3YsCgsLa/BT1H6RkZHo2rUrHBwc4O7ujqFDh+LixYtGbUpKSvD222/D1dUV9vb2GDZsGDIzM43a3Lx5E4MHD0aDBg3g7u6O9957D1qttiY/Sq23cuVKdOrUyTABWXBwMH755RfDcd7n6hEVFQWJRIJp06YZ9vFeP7358+dDIpEYbW3atDEcr3X3WKAnFhMTI8hkMmHt2rXC2bNnhXHjxglOTk5CZmam2KXVGbt37xY++OADYfv27QIAYceOHUbHo6KiBEdHR2Hnzp3CyZMnhSFDhgi+vr7C/fv3DW0GDBgg+Pn5CcePHxcOHz4stGjRQhg1alQNf5LaLTQ0VPj666+FM2fOCKmpqcKgQYOEJk2aCIWFhYY2b731lqBQKIS4uDjht99+E7p16yZ0797dcFyr1QodOnQQQkJChBMnTgi7d+8W3NzchNmzZ4vxkWqtH3/8Ufj555+FS5cuCRcvXhT+9a9/CVZWVsKZM2cEQeB9rg6JiYmCj4+P0KlTJ+Gdd94x7Oe9fnrz5s0T2rdvL9y5c8ew3b1713C8tt1jhpqnEBQUJLz99tuGv+t0OsHLy0uIjIwUsaq66++hRq/XC56ensLixYsN+3JzcwVra2th8+bNgiAIwrlz5wQAQlJSkqHNL7/8IkgkEiEjI6PGaq9rsrKyBADCwYMHBUEov69WVlbCtm3bDG3Onz8vABBUKpUgCOUBVCqVCmq12tBm5cqVglwuFzQaTc1+gDrG2dlZ+PLLL3mfq0FBQYHQsmVLYe/evcIzzzxjCDW811Vj3rx5gp+fX4XHauM95s9PT6i0tBTJyckICQkx7JNKpQgJCYFKpRKxMvNx/fp1qNVqo3vs6OgIpVJpuMcqlQpOTk4IDAw0tAkJCYFUKkVCQkKN11xX5OXlAQBcXFwAAMnJySgrKzO6123atEGTJk2M7nXHjh3h4eFhaBMaGor8/HycPXu2BquvO3Q6HWJiYlBUVITg4GDe52rw9ttvY/DgwUb3FOC/01Xp8uXL8PLyQrNmzTB69GjcvHkTQO28x/VmQcuqlp2dDZ1OZ/QPCgA8PDxw4cIFkaoyL2q1GgAqvMd/HlOr1XB3dzc6bmlpCRcXF0MbMqbX6zFt2jT06NEDHTp0AFB+H2UyGZycnIza/v1eV/TP4s9j9JfTp08jODgYJSUlsLe3x44dO9CuXTukpqbyPlehmJgYpKSkICkp6YFj/He6aiiVSqxbtw6tW7fGnTt38OGHH6JXr144c+ZMrbzHDDVE9czbb7+NM2fO4MiRI2KXYrZat26N1NRU5OXl4bvvvsOYMWNw8OBBscsyK7du3cI777yDvXv3wsbGRuxyzNbAgQMNf+7UqROUSiWaNm2KrVu3wtbWVsTKKsafn56Qm5sbLCwsHhjlnZmZCU9PT5GqMi9/3sdH3WNPT09kZWUZHddqtcjJyeE/hwpMnjwZu3btwoEDB9C4cWPDfk9PT5SWliI3N9eo/d/vdUX/LP48Rn+RyWRo0aIFAgICEBkZCT8/P3z22We8z1UoOTkZWVlZ6NKlCywtLWFpaYmDBw9i2bJlsLS0hIeHB+91NXByckKrVq1w5cqVWvnvM0PNE5LJZAgICEBcXJxhn16vR1xcHIKDg0WszHz4+vrC09PT6B7n5+cjISHBcI+Dg4ORm5uL5ORkQ5v9+/dDr9dDqVTWeM21lSAImDx5Mnbs2IH9+/fD19fX6HhAQACsrKyM7vXFixdx8+ZNo3t9+vRpoxC5d+9eyOVytGvXrmY+SB2l1+uh0Wh4n6tQ3759cfr0aaSmphq2wMBAjB492vBn3uuqV1hYiKtXr6JRo0a189/nKh96XI/ExMQI1tbWwrp164Rz584J48ePF5ycnIxGedOjFRQUCCdOnBBOnDghABA++eQT4cSJE0JaWpogCOWvdDs5OQk//PCDcOrUKeH555+v8JXuzp07CwkJCcKRI0eEli1b8pXuv5k4caLg6OgoxMfHG72aWVxcbGjz1ltvCU2aNBH2798v/Pbbb0JwcLAQHBxsOP7nq5n9+/cXUlNThdjYWKFhw4Z8/fVvZs2aJRw8eFC4fv26cOrUKWHWrFmCRCIR9uzZIwgC73N1+t+3nwSB97oqzJgxQ4iPjxeuX78uHD16VAgJCRHc3NyErKwsQRBq3z1mqHlKy5cvF5o0aSLIZDIhKChIOH78uNgl1SkHDhwQADywjRkzRhCE8te658yZI3h4eAjW1tZC3759hYsXLxr1ce/ePWHUqFGCvb29IJfLhfDwcKGgoECET1N7VXSPAQhff/21oc39+/eFSZMmCc7OzkKDBg2EF154Qbhz545RPzdu3BAGDhwo2NraCm5ubsKMGTOEsrKyGv40tdsbb7whNG3aVJDJZELDhg2Fvn37GgKNIPA+V6e/hxre66c3YsQIoVGjRoJMJhO8vb2FESNGCFeuXDEcr233WCIIglD1z3+IiIiIahbH1BAREZFZYKghIiIis8BQQ0RERGaBoYaIiIjMAkMNERERmQWGGiIiIjILDDVERERkFhhqiIiIyCww1BAREZFZYKghIiIis8BQQ0RERGaBoYaIiIjMwv8D63mNXzluG24AAAAASUVORK5CYII=" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(losses)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:36:05.177923Z", + "start_time": "2023-12-13T19:36:05.113102Z" + } + }, + "id": "78ac13e102efe3a0" + }, + { + "cell_type": "code", + "execution_count": 22, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 10/10 [00:02<00:00, 4.85it/s]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean_reward: -93.20616755490191\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "# Estimate agent performance\n", + "\n", + "episodes = []\n", + "for i in trange(10):\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await format_data_multiagent(datastore=cog.datastore, trial_id=trial_id, actor_agent_specs=cenv.agent_specs)\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + "mean_reward = np.mean([sum(e.rewards) for e in episodes])\n", + "print(f\"mean_reward: {mean_reward}\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:36:08.699440Z", + "start_time": "2023-12-13T19:36:06.634848Z" + } + }, + "id": "d221ebfe535f9317" + }, + { + "cell_type": "code", + "execution_count": 24, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "mean_reward: 87.1: 100%|██████████| 100/100 [04:30<00:00, 2.70s/it]\n" + ] + }, + { + "data": { + "text/plain": "[]" + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGdCAYAAAA8F1jjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB1hElEQVR4nO3dd5hcdb0/8PeZun22t+xusikkpJJGSEKokYAgIKhXjQjIpWhQij8RBMEGQeSiwkUQriJqEEQBAQGNoQaSAAnpnZTdZPtudmfr1PP745zvmTO9b32/nmefh+zMzp5MQs5nP+0rybIsg4iIiGgYMgz1BRARERGFw0CFiIiIhi0GKkRERDRsMVAhIiKiYYuBChEREQ1bDFSIiIho2GKgQkRERMMWAxUiIiIatkxDfQHJ8nq9aGhoQG5uLiRJGurLISIiohjIsozu7m5UVlbCYAifNxnxgUpDQwOqq6uH+jKIiIgoAfX19aiqqgr7+IgPVHJzcwEov9G8vLwhvhoiIiKKhd1uR3V1tXYfD2fEByqi3JOXl8dAhYiIaISJ1rbBZloiIiIathioEBER0bDFQIWIiIiGLQYqRERENGwxUCEiIqJhi4EKERERDVsMVIiIiGjYYqBCREREwxYDFSIiIhq2GKgQERHRsMVAhYiIiIYtBipEREQ0bDFQISIiopD+s7sZd7ywHVvqTgzZNYz405OJiIgoPZ77uB5rdzejIMuCeTUFQ3INzKgQERFRkK4+F97e1wIAuOSUcUN2HQxUiIiIKMgbuxrh8siYWpaLqeW5Q3YdDFSIiIgoyMvbGgAAF59SOaTXwUCFiIiI/LTYB/DBp+0AgIvnMFAhIiKiYeTV7Y2QZWBuTT6qC7OG9FoYqBAREZGff6hln0uGOJsCMFAhIiIinaPtvdhW3wmDBFw4m4EKERERDSOvqNmUJZOKUZJrHeKrYaBCREREKlmW8Y+tw2PaR2CgQkRERACAvU3dONDSA4vRgBUzyof6cgAwUCEiIiKVyKacPa0EtkzzEF+NgoEKERERQZZlrT/l4jlDtzI/UFoDFY/Hgx/+8Ieora1FZmYmJk2ahJ/+9KeQZVl7jizLuPvuu1FRUYHMzEwsX74cBw4cSOdlERERUYDtx7pwvLMf2RYjzj25dKgvR5PWQOXnP/85HnvsMfzv//4v9uzZg5///Od44IEH8Mgjj2jPeeCBB/Dwww/j8ccfx6ZNm5CdnY0VK1ZgYGAgnZdGREREOm/vawUALJtSggyzcYivxietgcoHH3yASy65BBdeeCEmTJiAL3zhCzjvvPPw4YcfAlCyKb/61a9w11134ZJLLsHs2bPxxz/+EQ0NDXjppZfSeWlERERjhscr4+kPjmDn8a6wz3lnv3JS8plTSwbrsmKS1kBlyZIlWLduHfbv3w8A2LZtG9avX48LLrgAAHD48GE0NTVh+fLl2tfYbDYsWrQIGzZsCPmaDocDdrvd74OIiIjC+9vmetzz8i58a80Wv/YLobPPia31nQCAM08aXoGKKZ0vfvvtt8Nut2PatGkwGo3weDy49957sXLlSgBAU1MTAKCsrMzv68rKyrTHAq1evRo//vGP03nZREREo4Ysy/jd+sMAgLqOPnxS34l5NQV+z3n3QBu8MnBSWQ4q8zOH4jLDSmtG5a9//SvWrFmDZ555Blu2bMHTTz+NBx98EE8//XTCr3nHHXegq6tL+6ivr0/hFRMREY0u6w+2YX9zj/brl9URZL131P6Us6YOnyZaIa2Byve+9z3cfvvt+PKXv4xZs2bhiiuuwC233ILVq1cDAMrLlWUyzc3Nfl/X3NysPRbIarUiLy/P74OIiIhC+72aTZlalgsA+OeORni8vvKP1yvjnf1KoDLcyj5AmgOVvr4+GAz+38JoNMLr9QIAamtrUV5ejnXr1mmP2+12bNq0CYsXL07npREREY16B1t68Na+VkgS8OjKucjPMqO124GNh9q15+xutKOtx4EsixELJhREeLWhkdZA5XOf+xzuvfde/POf/8SRI0fw4osv4qGHHsLnP/95AIAkSbj55pvxs5/9DC+//DJ27NiBr3/966isrMSll16azksjIiIa9f7wgZJNOXdaGSaX5uKCmUq1Ql/+EdmUJZOKYDUNn7FkIa3NtI888gh++MMf4lvf+hZaWlpQWVmJ66+/Hnfffbf2nNtuuw29vb247rrr0NnZidNPPx1vvPEGMjIy0nlpREREo1pnnxN/33wcAPCN0ycAAD43pxJ/+bAer+9sxE8vnQmLyaD1p5w5DPtTAECSQ80pjSB2ux02mw1dXV3sVyEiIlI99van+Pkbe3FyRR5e+87pkCQJHq+MxavXoaXbgf/7+gIsrC3EvJ+uhccr473bzkZ1YdagXV+s92+e9UNERDTKuDxePP3BEQDAN5ZOgCRJAACjQcJFsysBAC9va8D7B9vg8cqYWJI9qEFKPBioEBERjTKv72xCk30AxTkWXHxKpd9j4tdrdzfj9Z3KzrKzThqeZR+AgQoREdGo8+cNRwEAXzttfFCD7JwqG8YXZaHf5dFOSx5ua/P1GKgQERGNIj0ONzbXnQAAfGF+VdDjkiThc7N9WZYMswGLagsH7frixUCFiIhoFPn4SAc8XhnVhZmoKgjdd/K5Ob5A5bSJRcPqtORADFSIiIhGkY2HOgAAp9UWhX3O1PJcTCtXNtWeNQy30eqldY8KERERDS6xdfa0ieEDFQB48Itz8MbOJnz51JrBuKyEMVAhIiIaJXocbuw43gUAOG1S5EBl5jgbZo6zDcZlJYWlHyIiolFC9KfUFGZhXH7mUF9OSjBQISIiGiW0/pSJw3eKJ14MVIiIiEaJDTH2p4wkDFSIiIhGge4BF3aq/SmLGKgQERGNTZsOteM3bx+E2+Mdku//5LuHcOeLO+B0+3//j4+eGHX9KQCnfoiIiOJy10s7caClB8U5VnxpQfWgfu9m+wDue30PZBmoLc7Gfy+bqD3mG0sePf0pADMqREREMXN7vDjc1gsAWLOpbtC//6vbGyHLyn//+j8H0Nrt0B7zNdKOnrIPwECFiIgoZsdO9MPtVSKFbfWdWk9IoBO9TvxrVxNkEVWkyMvqIYIWowHdDjd+8a+9AEZvfwrAQIWIiMYw+4ArrueLbIqwZtPRoOd4vDKu+P0mXP+nzVi7uzmp69M72t6LbfWdMEjAw1+ZCwD468fHsK2+c9T2pwAMVIiIaIx6fUcjZv/o33jq/cMxf40IVCptGQCAlz5pCAp2/r75GHYetwMAdjXYU3S1StkHAJZMKsb5M8tx2dxxAIAfvbILGz4dnf0pAAMVIiIao9YfbAMAvLm3JeavOdKuBCqXzB2HyaU56Hd58NInx7XHexxuPPCvfdqv6zv6UnS1wMtblbLPxerJx9+/YBqyLUZ8UteJP244AmD09acADFSIiGiMEkHHzuNdMfeSiIxKbVE2Vi5SDvNbs7FO+/pH3zqIth4HDJLy/KNxBCqyLOPHr+zCD1/aCY/X/3r2NXVjX3M3LEYDVswsBwCU5WXg2+dOAQAMuJRR5dHWnwIwUCEiojHqcKsSdJzoc6GxayC2rxGBSkk2LptXhQyzAfuau/Hx0ROo7+jD795Tykg3nqMEEEfbYw9UDrf14qn3j+BPG4/i0bcO+j328jYla3Pm1BLYMs3a569eOgETirIAYFT2pwAMVIiIaAwacHnQoAtOYuklcbg9aOjsBwBMKMqGLdOslWHWbDyK+17bA6fHi9MnF+OapbUAgLYeB3od7piu6aMjHdp//+o/+7FJ3YsiyzJe2ab0p3xO/X6C1WTEvZ+fhQyzAV+YXxXT9xlpGKgQEdGYE5jpCDdmrFff0QevDORYTSjOsQAAvnbaeADAK9sb8frOJhgk4K6LToYty6xlPupiLP98dOQEACDXaoJXBm56dis6ep3YdqwLdR19yDQbsfzk0qCvWzq5GDt/tALfUctAow0DFSIiGnMOt/X4/TqWjMohtVQ0oTgLkqQ0ocyuysescTatp+Sri2owrTwPADBeLcnEHqgoGZX7L5+NiSXZaLIP4HvPb8M/tipln89ML0OWJfRCeZNx9N7OR+/vjIiIKIzDbUrwIHo6djdEz6iI5tva4hy/z4um2twME25ZfpL2+ZpCNVCJoU+lxT6Ao+19kCTg9CnFePSr82AxGbBubwv+uEHZ1XJxQNlnrGCgQkREI86Ay5PU1x9Rm2IvUCdoGroG0NHrjPg1IripVTMlwhfmV+H/nXcSnrhiAYpyrNrnRUblaIf/krhQRNlnWnkebJlmnFyRhx9eNB2AskAuL8OEM04qieW3NurwUEIiIhpRHvr3Pjz85kHUFGZh/vgCzBtfgAXjCzC1LBcGMRccxWE1OzKryoYJRVk40t6HXQ1dWDYlfDAgykUTirP9Pm8yGrQpHz2RUYll8keUfRZOKNA+97VFNfjgYBte39mEi+ZUwmIam7kFBipERDSivKQuPqvr6ENdRx9eVBeunTOtFL+7coHWPxKJGDOeUJSNGeNsaqBijxioHFEzKoGBSjg1hdnadUbjC1R8m2UlScIv/+sUXDCrGWdPHZvZFIClHyIiGkHaehzajf+3V8zHTedOwbIpxTAbJby5twXvHmiL+ho9Drd26vCE4mzMqFSaXyM11PY7PWiyK+PME2MMVETp5/iJfrg93rDP6x5wYU+j8r31gQoAZJiNuHhOJXIzzKG+dExgoEJERCPGlqNKL8eU0hysmFGOWz5zEv50zSJ8ffEEAMDjb38a9TVEf0pRtgW2TDNmVNoAALsijCiLRtr8LDPysywxXWt5XgYsJgPcXjniQrktdZ3wykB1YSbK1TOEyIeBChERjRhb6joBAPNqCvw+f83ptTAZJGw41I5t9Z0RX0Mr+6iZEZFROdzeG3Y5m75UFCuDQUJ1gTJVFKlP5eMQZR/yYaBCREQjxpY6JaMyb3y+3+cr8zNx8SnK+O5v342cVREZlVo1UCnOsaI8LwOyDK0EE+hwwNfEarwa2ESa/PnwMAOVSBioEBHRiODyeLH9WCeA4IwKAFx/xiQAwOs7m7RgJJTD7cFBh8iqhNtQGxjcxCraLhWH24OtagaIgUpoDFSIiGhE2NvYjQGXF3kZJkwqyQl6fGp5Ls6ZVgpZBp5871DY1wlVxonWUCt6VGKd+BGijSjvPG6Hw+1FYbYFk0rie+2xgoEKERGNCKLsc0pNQdh9KdefMREA8PzmY9pkT6Ajbb5V+MJ00VAbJlDRSj9x9KgA0dfoi7HkBeMLYhqrHosYqBAR0Yig9afU5Id9zqm1hZhbkw+n24unPzgS9HhnnxMn+lwA/DMqM8cpGZUDLd1wuP233nYPuNDWo2yt1Qc3sdAHKrIsBz3ORtroGKgQEdGIIAKV+eOD+1MESZK0XpU/bjgSNMUjMiNleVZkW307T8flZ8KWaYbLI+NAs/+BhWLRW3GONe59JlUFSqDS43AHrej3emVtdf7CWgYq4aQ9UDl+/Di+9rWvoaioCJmZmZg1axY+/vhj7XFZlnH33XejoqICmZmZWL58OQ4cOJDuyyIiohGktduB+o5+SBJwSnV+xOeeN70ME4uzYR9w47mP6v0e03pNAko4kiTp+lT8G2oPqavza+PMpgDKwrbyPGU3ytGA8s/B1h509buQaTZq35uCpTVQOXHiBJYuXQqz2YzXX38du3fvxv/8z/+goMAXDT/wwAN4+OGH8fjjj2PTpk3Izs7GihUrMDAQfjkOERGNLSKbclJpbtSshsEg4Run1wIAnvmwzq/kIg4WnBiicXXmuNB9Ktrq/Dj7U4SaotCTP2IseW5NPsxGFjjCSes78/Of/xzV1dV46qmncOqpp6K2thbnnXceJk1S0nKyLONXv/oV7rrrLlxyySWYPXs2/vjHP6KhoQEvvfRSOi+NiIhGELGRNnB/SjiXnFKJLIsRB1t6tPIKoGukDRF0hBtRTnTiRxhfGLqh9j97mgGwPyWatAYqL7/8MhYsWIAvfvGLKC0txdy5c/Hkk09qjx8+fBhNTU1Yvny59jmbzYZFixZhw4YNIV/T4XDAbrf7fRAR0egmMipzQ+xPCSU3w4yL5ygL4J7ZdFT7fOBWWj0RqOxp7IbHKwd9Taxn/AQSDbX6EeXDbb14e18rJAn4/NxxCb3uWJHWQOXQoUN47LHHMGXKFPzrX//CN7/5TXznO9/B008/DQBoamoCAJSVlfl9XVlZmfZYoNWrV8Nms2kf1dXV6fwtEBFRinT0OlEfw0nCgZxuL7YfU7IcoRa9hfPVRTUAgNd2NuFErxOyLGsZlVBBR21xDnKtJvS7PLjid5twvLMfQOTgJhY1ReIUZd8SOjGRdPbU0oRfd6xIa6Di9Xoxb9483HfffZg7dy6uu+46XHvttXj88ccTfs077rgDXV1d2kd9fX30LyIioiF3xe824TO/fEe78cdqT6OyFM2WaY4rqzG7Kh8zx+XB6fbi71uOob3XiW6HG5IEVBcGN8YaDRLuvWwWMs1GfPBpO87/1bv4w/uH0dUfPM4cj/EBS996HG78bfMxAMBVSyYk9JpjSVoDlYqKCkyfPt3vcyeffDLq6uoAAOXl5QCA5uZmv+c0NzdrjwWyWq3Iy8vz+yAiouHN45Wxp9GOAZcXf3j/cFxf6yv75Idd9BbOV05VsirPfFiHQ61KgFRpy0SG2Rjy+RfPqcRrNy3DKdX56B5w40ev7AagnIScaQn9NdGI7bQt3Q70Oz34++Zj6HG4MakkG8umFCf0mmNJWgOVpUuXYt++fX6f279/P8aPHw8AqK2tRXl5OdatW6c9brfbsWnTJixevDidl0ZERIPoRJ8Tou3jb5uPwT7givlrw52YHItLThmHbIsRh1p78dePlQx8tPN6aouz8bcbFuO7nzkJJjUwiveMH738LDNyM5SdLUc7erWyz1VLJnAbbQzSGqjccsst2LhxI+677z4cPHgQzzzzDJ544gmsWrUKgDK3fvPNN+NnP/sZXn75ZezYsQNf//rXUVlZiUsvvTSdl0ZERIOorce3zr7X6cHzHx+L+Wu1iZ8EApUcqwkXn6I0q/59i/I9Ywk6TEYDvn3uFLz4raW4cFYFVp09Oe7vLUiSpDXU/nnjURxq60Wu1YTL5lUl/JpjSVoDlYULF+LFF1/EX/7yF8ycORM//elP8atf/QorV67UnnPbbbfh29/+Nq677josXLgQPT09eOONN5CRkZHOSyMiokHU1u2/lfXpD474TdaE02IfwPHOfhgkYE61LaHv/VW1/CPWqcTTvDqryoZHV87D6UmWaMYXKt/zmU1K68OXFlb7bcal8NL+Ll100UW46KKLwj4uSRJ+8pOf4Cc/+Um6L4WIiIZIe6+SUTmlOh+H23pR19GHt/a2YPn0sohf987+VgDAtPK8uNfXC7OqbJg1zoYd6n6URDbMJkssffPKgCQBX188ftCvYaTiKjwiIkpK94ALP3hxB3726u6wzxEnGVcVZOLLpyprJZ76IHpT7T+2NgAALpgZesAiVmJUGVDGkAdbjW7K6JyppRif4ATRWMRAhYiIEvZpaw8uffR9PLOpDv+3/nDQwXtCu/r54hwrvr54AgwS8P7Bduxr6g772i32AXzwaRsApSk2GRfPqUSlLQPVhZmoKshM6rUSMV4XqFy1dMKgf/+RjIEKEREl5D+7m3Hp/76PT1t9e1Ga7aHPaWtTMyoluVaMy8/EihlKhuQP6gRMKK9sb4RXBubV5Gulk0RlW01445Yz8MZNZwzJuTozxtlQlG3BqRMKcfpkjiTHg4EKERHFxeuV8ev/HMB///FjdDvcOHVCoVbaCBeoiIxKUbYFAHD1UuXQwBc/OYbOvtBZmJe3HgeQfDZFyMswD1kDqy3TjI0/OBd//u9FHEmOEwMVIiKKy5pNR/HL/+wHoDSF/vm/F2kjvy12R8ivEePJxTlWAMDCCQWYXpGHAZcXz3xYF/T8w2292HasC0aDhAtnV6TjtzHozEYDLCbeduPFd4yIiOLySX0nAODqpRPwk0tmwmIyoDRXCUBauiOXfopylIyKJEm45nQlq/Kbtz5Fg3qujvDSJ0o25fTJxVpwQ2MTAxUiIopLr8MNAJhY4pueKctTdl81h8ioyLKMNl0zrXDp3HGYV5OPHocbd764A7K66ESWZby8rUF9TmV6fhM0YjBQISKiuPSogUqurt+jLE8JQEL1qHQ73HC6vQD8AxWjQcIDX5gDi8mAt/a14kU1i7L9WBcOt/Uiw2zAedOTG0umkY+BChERxaXH4QEAv8bUUpFR6Q7OqLT3KNmUbIsx6GC/yaU5uHn5FADAj1/ZjZbuAW13ymeml3N7KzFQISKi+PSoBwrm+GVUlEClJURGRWukzQ3da3LdsomYNc6Grn4X7nxxJ17ZrpZ9TmHZhxioEBFRnHrVjEpOiNJPa7cD3oAzfNrVQEWMJgcyGQ144AuzYTJIWLu7Ga3dDhRkmXHGSSXpuHwaYRioEBFRXESPSk6GL1ApzrFCkgC3V0ZHwF6U1p7gRtpAJ1fk4Vu6E4o/O6tiSBaz0fDDvwVERBQzr1dGr1MJVLKtvn4Ts9GgZUwCG2q1jEqUMeMbz56M6RV5MEjAlxZUp/KyaQRjlxIREcWsz+WBOkWMXKv/acaluRlo63Gixe7ADF17iehRKckJXfoRLCYD/nrDYjTbBzCpZPAPDqThiRkVIiKKmdihYpCADLP/LSTciHJbt1r6CdNMq5djNTFIIT8MVIiIKGbdA2p/itUUdGaNNvkTMKLc3iuaablhluLHQIWIiGImMio5IfabiDX6QRkVrZk2cumHKBQGKkREFLNQEz9CaZg1+m0xNtMShcJAhYiIYiYClVAbY32lH19GZcDl0cpFJQxUKAEMVIiIKGY9A+FLP6GaaTvUwwjNRgl5mRw0pfgxUCEiGkPcHm9SXy92qIQOVJSMSluPEx51O61W9sm2BjXfEsWCgQoR0Rhx9z92Yt5P16Kxqz/h1+iOkFEpyrZAkgCPV9YmfcSBhMW5bKSlxDBQISIaI97Z3wr7gBtbjnYm/Bq9EXpUTEaDtia/RW2obe3haDIlh4EKEdEYIbIb+mbXeIlm2twQUz9AcJ+KdnIyG2kpQQxUiIjGgAGXRwsyAheyxSPS1A8AlOX6jyi3c4cKJYmBChHRGCCmbwBfWSYRkaZ+AP0uFWZUKDUYqBARjQEiswEkV/qJNPUD+Eo/Imsjvm8RMyqUIAYqRERjgJjCAdKcUVFLPy3MqFCKMFAhIhoDUpVRidqjIpppu0WgInpUGKhQYhioEBGNAfoelRN9LjjdiS1+iz7142um9XhldPSKjApLP5QYBipERGNAW69/uUfsNwm041gX7v3nbnQPuEI+3uvwAAifUSlVMyptPQ609zigLqhFYTYDFUoMAxUiojGgQ1f6AfzP49H7n7X78OR7h/H6zqagx7xe2Xd6cphApSjbCqNBgiwDe5u6AQAFWWaYjLzdUGL4N4eIaAxo7/UPVMI11Na19wEAWkPsWulzebT/DheoGA2SVubZ1WAHwP4USg4DFSKiMaBdLfVY1MxGa4iGWlmWcbxTOQeos88Z9LiY+DEaJGSYw98+RJ/K7kYlUOFoMiWDgQoR0RggMipTynIAhN5O29rjgENtsu3oDe5R0SZ+LMaIJyGLEeVdDV0AmFGh5DBQISIaA8R48skVeQBCl36OnfCdqhwyo6JN/Jgjfi8xony4rRcAAxVKDgMVIqJRrs/pRr/aXyICleYQpZ/jukClI0Sg4js52Rjx+4nSj6xO/HA0mZIxaIHK/fffD0mScPPNN2ufGxgYwKpVq1BUVIScnBxcfvnlaG5uHqxLIiIaE0Q2xWoyYGJxNoBYMirBpZ/uKFtphdJc/wwKMyqUjEEJVD766CP89re/xezZs/0+f8stt+CVV17B888/j3feeQcNDQ247LLLBuOSiIjGDNGfUpRtQUmu/1k8esc7+7T/7uiNlFGJHKiIjIpQxECFkpD2QKWnpwcrV67Ek08+iYKCAu3zXV1d+N3vfoeHHnoI55xzDubPn4+nnnoKH3zwATZu3JjuyyIiGjPEdtiiHKu2kK291wG3x387rT6jYh9wBT0ebSutIL6HwNIPJSPtgcqqVatw4YUXYvny5X6f37x5M1wul9/np02bhpqaGmzYsCHs6zkcDtjtdr8PIiIKT5y3U5htQVG2FQZJ6R8J3K2iD1RkGejq9y//+KZ+4suosPRDyUhroPLss89iy5YtWL16ddBjTU1NsFgsyM/P9/t8WVkZmpqCNyIKq1evhs1m0z6qq6tTfdlERKOKKOMU5VhgNEha+Ue/nVaWZb9mWkA5E0hP20obJaNSmGWByeAbX2agQslIW6BSX1+Pm266CWvWrEFGRkb0L4jRHXfcga6uLu2jvr4+Za9NRDQaiWVvImAQe070DbUdvU5tMqjCpjweOKLcG2V9vmAwSFpDbbbFiExL5CkhokjSFqhs3rwZLS0tmDdvHkwmE0wmE9555x08/PDDMJlMKCsrg9PpRGdnp9/XNTc3o7y8POzrWq1W5OXl+X0QEY11sizD5Ql9InK7rvQD+KZy9A21ouxTlmfVSjeBDbU9MU79AECJ+hpspKVkpS1QOffcc7Fjxw5s3bpV+1iwYAFWrlyp/bfZbMa6deu0r9m3bx/q6uqwePHidF0WEdGodM3TH2Pp/W/CHuLUY/3UD+Brdm3R7VIRq/PH5WeiIEtZ6BY4otwT49QPAJSpwRAbaSlZ0f+2JSg3NxczZ870+1x2djaKioq0z19zzTW49dZbUVhYiLy8PHz729/G4sWLcdppp6XrsoiIRh2n24u397XAKwM7j3VhyeRiv8fbtakfJWgoEaUfv4yKMppcVZAFk1HpLwlc+hbr1A/ga6hlRoWSlbZAJRa//OUvYTAYcPnll8PhcGDFihX4zW9+M5SXNOIdbe9Fr8OD6ZUsiRGNFcc7++FVt8Aebu8NClQ6ekRGRfSoqBkVXTOtaKQdV5AJp3rez4kwPSrRpn4AYIK6WG58YVZcvxeiQIMaqLz99tt+v87IyMCjjz6KRx99dDAvY1T70m83oLPPhY/vWh71PA4iGh2OtPdq/324tdfvMVmW0aab+gF82Y5QPSpVBZlayedEQI9Kd4xTPwDwlVOrUZBlxtlTS+P6vRAFGtKMCqWWw+1Bs9rF397jZKBCNEYcbdMFKm3+gUqPw61lSIIzKqEClSwAShkocDw51qkfAMiymHDZvKp4fhtEITFQGUV6HR7tv/ucngjPJKLR5Ei7b/X94Xb/QEVM7mTpxoRFM21bjwNerwxJ8m+m7VMDksCMSjxTP0Spwr9to4j4aQeAtg+BiEa/ug5foFLX3ge3xwuTURnqbAsYTQaUfSqSBLi9Mjr6nDAZJK1RtqogE61qSUjfo+L1yuhVfwCKZeqHKFUG7fRkSj9xsikA9DOjQjRm6HtU3F5Zy44A+q20vukbs9GAwiwlcGm2D2hln+IcKzLMRhRkK2Vjfemn1+n79yWWqR+iVGGgMoro/yHp0/03EY1eHq+MejWjYstUAoxDuj4VbStttv8+k1JdQ+0x3cQPAC2I6exzwquOE4nSstEgwWrirYMGD/+2jSI9LP0QjTkNnf1weWRYTAYsqi0E4D/5I5a9FQYGKmpDbavdoduhogQq+Wqg4pV9mdoeh5JdybGaIEkSiAYLA5VRpIelH6Ix56jaSFtdkImJJTkA/EtBYn1+4OI13xr9Aa1UJAIVi8mgNcyKpW89akaFjbQ02BiojCL6ZlpO/RCNDSIomVCUjYnqkjX9iLK2lTao9OM770cbTc7P1B7PzxJ9KmqgwokfGiIMVEYRln6IRr4ehxt/23wMAzH+PywmfsYXZaO2JDhQ6QhY9iaIE5Sb7QPaVlplh4pClIrEiLLvnB+ehEyDi6HxKOIXqDCjQjQirX5tD9ZsqkNbjwM3nDkp6vOPqEHJhOIsTChSApXjnf0YcHmQYTZq48nhSz/BPSqAr09FTP70aFtpuUiSBhczKqMISz9EI5ssy1i7uxkAsONYV0xfI3pUxhdlozjHglyrCbLsy7SIqZ/g0o+SUTnU2gu7WtYZpwtUCkXpR82o+LbSMqNCg4uByijSo9tMy9IPUfrYB1xY9cwWvLW3JaWvu7vRrp2/c6ClO+rzvV4ZRzuUjMr4wixIkqQdBni4rReyLEco/SgZla5+JWNSmG1Blu6wQV9Gxb/0wx4VGmwMVEYR/9IP96gQpcs/tzfin9sbceeLO+ARxxanwNv7WrX/PtzWC7fHG/H5Ld0ODLi8MBokLRtSqwtU7P1uuNXrCxxPLsn1LwWN0zXS6p8fGKhwKy0NNgYqowhLP0SDQ5RbGroGsOHT9pS97tv7fBkal0f2W40fipj4qSrIhFldmS8ClSNtvdrET67VBKvJv2STYTZqC+LEa+gVaKUftUdFLQ/lMlChQcZAZRTh1A/R4KjXBRDPb65PyWt29bmwpa4TgC/bcaClJ+LX1On6UwQRqBxq69WWvQWWfYRSXVYlKFBRMypij0ovMyo0RBiojCJc+EYU2YDL45d5TJQ+0/HGziatzyMZ7x1shccrY0ppDpZOKgIAHIwSqPh2qPjGivWlH9FIG1j2EcrUhloguPRToFujDwDd2tQPAxUaXPwbN4r4n/XDQIXGNlmW8YMXd2Dz0RPo7HOhq98Fh9sLgwT8ZuU8nD+zIuHXrlfHeXOtJnQ73Hh1ewNWLhqf1PW+tVfpTzlraolWkvk0SqAiSlA1hb5ARTTTtnY7tIAqcDRZ8M+oZPk9VhAwntzLZloaIsyojCL6nxRjXRZFNFrtb+7BXz6sx/7mHrR0O+BwK42pXhn4546mhF+3q9+FTvXmfc2yWgDA3zYfS+pavV4Z7+xXApWzp5ZicqmyCj9a6Ue/lVawZZq1UeTNR08AAIrDlH5K8nSBSmFg6cc3nizLMqd+aMgwUBlFugeYUSESDrcpN/mpZbl49dun473bzsYTV8wHAGytP5Hw64r+lOIcC766qAZGg4RP6jpxMIZx4nB2NdjR1uNAtsWIBRMKtUDl09Ye7fTiQLIsaxmVCcX+2RBR/hGBSrjSj9hOC4Qv/bi9SpDCqR8aKgxURgmXx6v9xAgAfRxPBgBsP9aJ2/62TVtaFUp9Rx/ufHGHX4MkjXyH25Q/z5MrcjFznA3VhVk4bVIRJAmo7+hHq7qvJF6+La5ZKM3NwNlTSwAAzyeRVRHTPksnF8NiMmB8UTZMBgl9Tg8a7QMhv6a914kehxuSFFy2EYGKtpU2O3Lpx5ZpRm7AxtkMsxGZZmVS6ESvi2f90JBhoDLM9DndeGtfC3766m5c+8eP8eyHdTEFHYENggOuyPsXxorfvPUp/vrxMfxu/eGwz/n1ugNYs6kOP3hxxyBeGaWbb7W8ryySl2HGFDVbsbW+M6HXFX0foi/kC/OrAQAvbjkede9JOG+pgcrZ00oBAGajQbvuA82hMzUim1Jpy0SG2X/0WP97BsJP/UwrzwUAzByXF/LxAt3BhOxRoaHCv3HDgCzL+NPGo/jn9kZsqTsBl8eX6l27uxn3/nMPPj9vHL66qAbTykP/g9ITEKg4PV64PV6YjGM7FhU/ja4/2Ib/t2Jq0OOyLGP9gTYAwHsH2rC1vhOnVOcP5iVSmhxW+zdqA27ac6sLsL+5B5/UncBnppfF/bqBgco500pRmG1BS7cD7x1o04KNWJ3odeITNWg6S83OAMDkkhwcbOnBwZYenDU1+DWPqr8/fSOtMDEwUAmTUZlSlovXb1qGSltmyMcLsi1o6BpAe68DvWo5mVM/NNjG9l1smNjX3I27/7ELmw53wOWRMS4/E19eWI1bP3MSJhRlodvhxh83HMX5v3oP3/nLJyFfo1ddn6//aaePDbVoUQOV7cc6Q46QftragyZdav2RdQcG7doovURGRb9jBADm1uQDAD5Rd5bEq65DOWm4Wm0+tZgMuOSUSgCJ7VR590ArZFnJblToAoYpZb4+lVCOhOlPUT4XW0YFAE6uyIMtK/RBg6JPRZyuDDCjQoOPgcow0NCp/CNQW5yNt/7fWVj//bNx/+Wz8Z1zp+DN756FNf+9COfPKAcAvLytAQ53cADS41BuwgXZZhgk5XMDY7yh1uuVtT4ErwxsPBS8QVRkUyaVZMMgAev2tmDn8dgOg6Phq9fh1s7MqQ0KVAoAKMFrIuvvj6kZlWpdJuOLavnnP7tbIvZD7W/uxv2v78VfP6rH3iY7PF5ZW5sfmDXRJn+aQwcqIqMSGIgB/lNAQPCBhLESS9/q1UDFZJBgNfG2QYOLf+OGAXEznVCUhdribEiSpD1mMEhYOrkYv1k5D+LToTIDPVpGxawdLDZSJn+6+l340cu7sLfJntLX7ehzauecAL6gRG/9QeVzX5hfjc/NUX4q/t83D6b0OmjwibHdgixzULZgcmkOsi1G9Do9IQ/+O9jSg688sREffBr898XjlXFMvWnrSy7TK/MwvSIPTo8Xb+wKP/r8gxd24PF3PsVtf9+O83/1Hmbe8y/8c3sjAP+yDwBMKlEClYOtPZDl4IBKy6gUBWdUMi1GVNp8Ez0FiQYq6nsnGs2zrSa/f5+IBgMDlWFAdOYXh1nKBCgBi1gC1dUXHKjoj2AXjXUjJVD53frD+MMHR/Dgv/an9HVb7P5THe8f9L/xuDxebDzUAQBYNqUYN549GZIEvLGrCfuaEh81paF3pE2URYKzDUaDhDlqH1Ko8s8jbx7AhkPtePLdQ0GPNdsH4PR4YTJIfmUaALhojrJA7rUdjSGvqbGrHx+r48Kn1hYi22JEv8sDp8eLwmwL5o8v8Hv+pJIcSBLQ2efSVuHr+XpUgn+PgO/3bss0a+cAxUuUfsSCO5Z9aCgwUBkGREalODd8oALoVlqHyqgM+HYcZFmUQGWknPcjAohdDaktuTR3K70nVQWZMEjK2SfHO3219m31nehxuFGQZcb0ijxMKcvFBTOVEtv/vsWsykgmMiqBZR/B16fiv0+lx+HGv9SMyLZjXUGZDNFIO64gE0aDf2bhAnXT7Qeftocs/7yuLplbML4Af71+Mbb/aAXW3nIGfvlfc/DMtYuCgolMi1E7fydwlX5Xn2/p3PgQGRXA10QcqT8lGl9GRfn/hoEKDQUGKsNAq3oeR0mEjAoALaMS6h9B/dZILVBJMKPSYh/Aj1/Zha/93ybtrJB06XG4sU2deGjsGkBHhPp+vFrVjMrk0hztJ2h9VkWUfZZMLoZBvencePYUAMCr2xuinrMSqKGzH6f//E385m0GOUPtcIjRZL251Ur2IjCj8u9dTdpof0evUyvzCPUBEz96tcXZOLkiDx6vjLW7m4MeF5mWz85SAhqjQcKUslx8fm5V2Gm+ySWhN9Qe7VB+fyW51rAL2ESgUhxm4icWomQkys2c+KGhwEBlGGiLMaOSr/50Eyqjot9x4Cv9xLf0rb3HgXv/uRvLHngLT71/BOsPtmk383T56HCHXx/J7obU9ak0q9M8pblWnD65GIB/n4r4b/EYoPQaLD+5DLIM/CbOrMo7+1tx7EQ/fvfe4bDbRGlwhNqhoneKmlE50NLj1/P14ifH/Z637Vin36/rQzTS6l04S8nIvbbTv/zT1DWglX0uUJ8Tiyllyp6TwDN/IvWnCGdPK0VZnhXnz4z9+wUSWVyBW2lpKDBQGQba1KxFuPM4hPwIPSr69dbxln7cHi8e+vc+LHvgLTz53mE43F6YjUqGIZUZjlACGxZTWf4RUx9leRlYqgYj7x9sg9cro3vApe2u0AcqAPCdcycDAP6xrUE7OTYWjWpZqb3Xid2NqW0MpvhEK/0U51i18eLtajDSYh/QMm5nnqQ0tm4LWAonSj/VBaEDhAvUbMn7B9v8/j99XQ1c5o8vCOptiURkVAKze++q5wIF7ojRm1SSg00/WI5vnF4b8/cLFLh6P5eBCg0BBirDgOhRKY2aURE9Kqkt/bx3sA0Pv3kQfU4PZo2z4amrF+K/FirjlpFGLVPh/YPKyLDYkJnKG7yWUcnLwLyaAmSajWjvdWJfczc2HeqAxytjfFFW0E/Hs6vyMb4oCx6vjJ3HY7+ehi7fPpZ0Z6IovO4Bl9agHmrHiBBY/nl5WwO8shJMiAmwbfX+gXN9iIkfvUklOZhalguXR8baPb7yT2DZJ1aTtMMJfc3de5vs+PsWZV3/l0+tiev14pUfMDGVbTWGeSZR+jBQGWIOtwd2tRE20tQP4PtH40TEqR9T3FM/ovS0qLYQL9+4FGdPLUWhWtfuiCOjEMqAy4P3D7aF3FfRocs8iJ/6dqWw9NOiCwAtJgMWTSwEoJR8RCARmE0RZlba1OuJPcPT2OXrZwg1Ck2DQ0z8FOdYgs6v0RMNtWKVvij7XDp3HOZUKX/+O453+a3FD9xKG4oo7byuBifNdl/Z57NxlH0A3y6VZrsD9gHl//vVr+2FLAMXzqrAvJqCSF+etMCMSo41/PtJlC4MVIZYu/qTn9noGz8OJ5bST05G/KUfcepyca5V25FQKIKi3uDvFY8/bzyKlf+3Cfe/vifoMbGAbWpZLs5SU+2HWnsSbgIOJLbSluUp+yS0PpWD0QOV6ZVKc+POOAKnxk5fRuXDIx0YGCFTV6ONWJ0fuPQskFj89kndCexv7sauBjtMBgkXzarAxJIc5FhN6Hd5tEbWfqdHy35GClRE1uS9A22wD7jw+o5GyDIwryY/rrIPoDTQi0zrpy09WH+gDe/sb4XZKOG284OPhEi1TLMRFt2CtxxmVGgIMFAZYtpoco416iKlWEo/So+KUkeO9YYvvjZP19FfqGZ32nuTm/oRB6et2VQXtKjufW3qpggluVYU51jglZGSxW9er+yXUQGA06coQcmGT9txsKUHkgQsmRQmozIuvoyKLMva6LPVZIDT7cWHhzuS+j1QYqI10grTK/JgMRlwos+FX/9HOTrhrKmlKMi2wGiQMEv9OyD6VMQukdwMU9iV8wBwUlkuJpfmwOnx4s09LXhNHUuOt+wjiKzK/uZu3PeaEvB/7bTxITfSppokSSjUNdRy6oeGAgOVIeZrpI0+QqhN/YQs/YjNtPEvfOtWU8r6NLn4xynZjEqvOnnU5/TgLx/W+T224VMlo7JkUjEkScJ0tdySij6VE7qttCVqoDK1LBfFORY41VT+7HG2sDecGWpG5XBbb9DJ1KG/nwsOt/K6K9TjDtinMjREoBKp0RRQzugRf87/VMs0n587TntcjLRvO6YEq5FGkwN9Vp20eXrDEXx0VAlYkw1UHnv7U+xutCPXasK3z5mS0GslQt+nwqkfGgoMVIaYyKiURGmkBXQZlUilH6s54dKPvqO/IFv5xynZHhV9VucP7x+BU72ZN3b141BbLwwStN6R6RXKTSMVfSrN6g6VomyLtkhLkiRt+gfwZVhCKc6xojwvA7IM7IkhcBLnNRVlW3DuycqZLe+xTyUm9/xjJy599P2gE8ATFWvpB/A11ALK33/xZwcAp1T7Z1Ri6U8RxPTPJ3WdkGWlH6YyP76yjzBFDVTESPK3zp4c1DuSTvrvxYVvNBTSGqisXr0aCxcuRG5uLkpLS3HppZdi3759fs8ZGBjAqlWrUFRUhJycHFx++eVobg5eljRaxTqaDPh6VEKNzPpKP0bd1E9s//CLQEWf1hX/OJ3odYY8ZyRWvbpApck+oE0/fKBO+8yuykeemskRP92mIlBp6fZN/Ojpe1KWhulPEcT1xHJIYaM68VOZn6l9jz2Ndi0QpdCcbi/WbKrD1vpObeQ2lGb7QMizd0I5GuFU4UCioRZQmmBFNhLwZVT2NXej3+nxjSbHEKhMK8/1y+hcmGA2BfBN/gBApS0DVy+dkPBrJUK/S4WBCg2FtAYq77zzDlatWoWNGzdi7dq1cLlcOO+889Db26s955ZbbsErr7yC559/Hu+88w4aGhpw2WWXpfOy0urZD+tw87OfaJmDaGI550cQKdhepyfo9cUK/USmfrrVIEdf+hH/OLm9sjaVlIg+9bVnjlNu+k++dwiyLON99aazZFKR9lwRGOxttPtNWiRCnPMTOPK9bEoJLEYD8rPMQWerBJqh9alED5zExE+FLQNFOVbt9xJ4vhD5O9TWo5XoIr1Xq9ZswVef3IS39rZEfL2ufpe2+yemjIouULlUV/YBgPK8DJTmWuHxytjV0KWtkY8lUJEkSTuOAfBlWBIxWReo/L8VU/2CqcGgL/0wUKGhkNZA5Y033sBVV12FGTNmYM6cOfjDH/6Auro6bN68GQDQ1dWF3/3ud3jooYdwzjnnYP78+XjqqafwwQcfYOPGjem8tLR57J1P8dLWhphvUPGUfvIyzNoJyvqGWo9X1so8OQksfPP1qPj+EcowG5Gtvk4yu1RERuXaZRORYTZgV4MdGw61axkVfVZjQlE2sixGONxebQV6okRGpSzP/30tt2XgrzcsxnPXLYbVFPkf/BlxTP40dPoyKoCvrMTyT2R7G337QUTPUqDWboc23vvcR/URX0/0p0RaLa83Lj8TV5w2HpfPq8JptUV+j0mS7/DCrfWdcfWoAMBl88Yhw2zA2VNLMC7Bsg8AlOZm4PozJuJrp9Xg0lPGRf+CFNOXftijQkNhUHtUurqUFHphodKTsHnzZrhcLixfvlx7zrRp01BTU4MNGzYM5qWljBhJ3RFDuQDwnfMTS0Yl3AnKvboST3YCC99C9agAvnM+kulTEeWnqoJMXD6vCgDwk1d2o8k+AIvJ4JfVMBgknKz2qSTbUNusZVQygh47pTofU9UFc5GIQOVAczcc7sjvpT6jAgBnTFHGrdcfbE2qdDba7dWdUn2orddvF42gLwmt29sccVtwtI20gSRJwk8vnYn/+dIc7bwnvVN0gUo8PSoAMLk0Fx/cfi4e+9r8mJ4fyR2fPRk/u3RWyGtMt3xd6SeXUz80BAYtUPF6vbj55puxdOlSzJw5EwDQ1NQEi8WC/Px8v+eWlZWhqakp5Os4HA7Y7Xa/j+FElGRi6WsAfD0qsWRUAF2fim7UV5R9TAYJVpMh7tKP+PrA5VjiJ6mOnuQzKplmE645vRaS5Ls5za8pCEpjp6qhNlxGJR7j8jORn2WG2ytjf1PkAwrFDpUK9Sfn+eMLYDUZ0Gx3+B0o5/XK+NeuJnzaGt+Bh0JXvwvbj3XilW0NePStg3hjZ+j/T0aKfQGj6KGyKm/rAhWXR8Yr2xuDniP4DiOMLZiIZk5VPgAlM9bv8kCSgMr84OA3nMJsy6CXalKtMJtTPzS0Bi1QWbVqFXbu3Ilnn302qddZvXo1bDab9lFdXZ2iK0wNl0f56TnWQEW/RyUWthCTP726ZW+SJGl7VGJdOBaq9APoApUkMip9uibfiSU5OHdamfbY0slFQc8XWYxkDycUGZWSEBmVWEmSpGvwjfzn2aBmAirVjEqG2YhTa5XMoSj/NHUN4Irfb8L1f9qMbz/zSVzX8v7BNiy89z+Y8+N/4+L/fR/f/ssn+MW/9uFbazajSbe6f6QRQevCCUpmTRypIHi8Mt47oAQq56tj3y+o6+NDiXWHSqxmqRtqxQ6giryMqCXD0SafzbQ0xAYlULnxxhvx6quv4q233kJVVZX2+fLycjidTnR2dvo9v7m5GeXloVdN33HHHejq6tI+6usj16wHm8ioNHQNoL0n8sTHgMujlV1KYgxUCrQ1+r7gQZv4UQMUUfqJJaPi8cpa1iMoUMnyTf4kwuuV0acGSyJ4+u9lvgPSFodYtjZDt7o+mZKJbytt4hkVQL9KP3zg5PXK2rlCFbpeBK38c6AVb+xswvm/fle7ER9Tl4fF6q8f1/sFtQvGF2gL8jYdDt3bEUqLfQCXP/YB/hql12MwdPW5tGmpq5Yofy82fNrm9+e+7VgnOvtcyMsw4UcXz4DRIOGTuk4cCpOROqxO/MRa+onGlmnGRF3QE0sj7Wgj/h0QGVuiwZbWv3WyLOPGG2/Eiy++iDfffBO1tf6neM6fPx9msxnr1q3TPrdv3z7U1dVh8eLFIV/TarUiLy/P72O4kGVZWyYGRG/CbFcDAIvRgLzM2H5SCbVGX38gIQBd6Sf6tE6PbqIncOtksj0qA24PxD1HHGa2qLYQX11Ug8/NqdTq/3pTynJgNEg4obuJxcvrlbXen8Dx5Hj5VumHz6i09Tjg8sgwSECZroQnGmrf2d+KG/68GZ19Lm0nhn3ADVcck01il8sTV8zHx3ctx9++uQSXqI2V8WzAff/TNmw+egKPv/tpzF+TjBe2HAta9Cfsa1ayKePyM3HOtFKYjRIauga08WIAeHufkk1ZNqUE5bYMnKG+p+JcnkCpzqgAvjFlYGwGKjWFWbCYDJhQnB11ezZROqQ1UFm1ahX+/Oc/45lnnkFubi6amprQ1NSE/n4lTW6z2XDNNdfg1ltvxVtvvYXNmzfj6quvxuLFi3Haaael89LSQpR9hGjlH99PyJaY/wEItUZfX/oBfBmVAVf0G2G3Qwl4LCZDUEo72R4VfUYnQ31tSZJw3+dn4ZGvzIUxRGNghtmo3cwTLf+c6HNqfxaxZqrCEav09zTaQx6sCPhOTS7NzYDJ6Ptfalp5LopzrPDKgCQBN5w5Ca98+3Tf5FaIxX2hDLg8+LS11+96AGilpXgCFRGYHmrtjStTdqi1B7/+z4G4zi9yur247W/bcccLO7SJGT1xVMLU8lxkWoza2Tsf6PpU3tmnjCOfOVXJTl2mNmS/sOU4vAF/Hid6nVqJJpbR5FiJAwqB2BtpR5OCbAveuGkZ/nLtyPs3mUaHtAYqjz32GLq6unDWWWehoqJC+3juuee05/zyl7/ERRddhMsvvxxnnHEGysvL8cILL6TzstIm8CfkaIGKOLW4OMZGWgDa1E+nX0ZFuXmIRrdMNaPi9Hij7iMRpae8EN382tK3BDMqfQ5R9jHGNa2QbEOtOOOnMNvid6BaImrVkekBlzdsuaFR3Uob2GQpSRK+fc5kzKmyYc1/L8LtF0xDhtmoZcVifV8PtvTA45VhyzRrU0UAsHCCEqgcaOmJWmYU9DtxPqk/EdPXAMA9L+/CL/+zH3/acDTmr+nodUbckSL6U8QE1lK1FCh27LT3OLBd/X9IHFr5mellyM0w4XhnPzYFBGhiI215XgYyLanrI9FnVMZioAIAE0tyYm74J0q1tJd+Qn1cddVV2nMyMjLw6KOPoqOjA729vXjhhRfC9qcMd4FL2CKVCwDfaHI8P/UXhDjvp0dthhUnm+r/ke6L8hNwd5iJH+V7qRmVBHtUxNi06E+JlSi37G6MrSE5kOgXCVz2lgiDQYoaOInDCCtC7Mq4cskE/OPG0/0OP9RKajG+r6Lsc3JFrl/mrTDbgpPKlOzTR0diCzr0a+o3H43ta/qdHi0o2HAo9n4Y/e/v/RDTPPvUQGWaGqgsUZurN37aDq9XxnsH2iDLwMkVeVoJL8NsxEWzleVpgU21R1I88SOcXJEHs1F536sLE9+HQkSJYWdUCgVmVOo7+iPufGiLc+IHCFP6cfqWvQHK6b0igTEQpaG2WwtywmdUEg1URI9MdpxHw09PcpW+dmpykv0pQrRV+tr6fFts3y/eJuU96lI0sWNGT5R/PjoSW/lH35O05WhnTF/z4ZEOLQj/6HBH2BJYIP3fm8AmWVmWdYGK8vuaU5WPTLMR7b1O7G/pxttq2ecstewjiPLPazsa/XYFxXoYYbwyzEZ888xJOHtqCWaNy0/paxNRdJw1SyFxem6G2YDS3AzUdfRhV4M97Jky2jk/ubEfMGYLlVHRRoCVP05JkpBpNqLX6Yk6+dOjrc9PR6AidqjEF6jMqFB6Ao6d6EdXv0srd8VKm/hJUap6RpTJH9+yt9h+2o63SdmXUQkOVBZOKMSfN9bF3Keiz6hsre+E2+P166sJ5T3dHpNuhxt7Gu1+vTLh6H9/bT1O7Gvu1oKSYyf60eNww2yUMLFECSwsJgMW1hbi3f2tWH+gDe+qY92i7CMsGF+AmsIs1HX04TdvH4QE5aRqccpxKvtThFvPm5ry1ySi2DCjkkJi4sdiNGhn20TaUJtI6Sc/VI9KiM2ymWq5JdoafftA9EAl3gkVoTegdyZWtixfL0a4vpBIREalLFUZlXG+XSqhRqZ96/NTn1GRZRl71KbT6REyKrsaurTsWCT65/S7PH6bYcMRe2BEwLkxxvJPR0DfzHrdcQIimzKpJEc73Rrwnf301PtH0NHrRK7VhHkBZzJJkoTL5ikTT4+8eRAPv3kQW+o64fHKmFyag/NnjszSMRGFxkAlhcTN3GIyaj9xRmqobetWDySM4yf/Am3hW/DUjz4giHWXiq/0E5y1sGWa455Q0evTelTib2wUJa5EDkTUelSS3KEiTCnNhdkowT7gxrETwSveE86o9EZ/T5vtDnT2uWA0SH6H0wkVtkzUFGbBK8fWcyJ6kkRpcEtd5K9psQ9gX3M3JAn4+pLxABDUxBpOh/p3RvR36Kd5xMTPtICjDERDrej7WTq52C+QEf5rYTWKc6wozrHg4jmVeODy2Vj//bPxn1vPxPg0ZFSIaOgwUEkhUce3GCVtUVjEQCWRjEqIE5QDSz+A76ffaOf99ETIqBgNkpbBSaT8I3pnsuNspgV8U0ixZAkCRTrnJxEWk0GbTAn883R5vFoGpyLWjEp27FM/ouwzsTg77Cr2eMaUxd8VscNmS5TgRmRTZo2zaZthPzrSETQaHEpHr/K+iMV3mw61a8G8b+LHP0s0vTLPbwItsD9FqLBl4qM7z8VHdy7Hw1+Ziy8trEZVwdicyCEa7RiopJAvo2LQMipH2vtgD3OzbU1gPDlXd4Ky2BkRqs8kM8YTlCONJwPJ9an0J5FRyVMDJHt//BmVVq2ZNnXjlKJvJrBPpdk+AFlWsgbF2bFuF479Pd0doT9FiKehVvxdOUPt+9gcJaMi1tcvm1KMmeNsyLIY0dnn0pa1RXJCzRgtnVyM/Cwzep0ebD/WCSB44kcwGiScNtF3tMKZYQIVQCkBcQEZ0ejHQCWFRDOt2WhAYbZFO9p91/HgJswBlwfd6k0jnqkfo0FCnjpK3KVO/vQGrNAH9KWfyDd63zk/oRtWk9mlInpUsuKc+gGg/R7DBXnhyLKsO5AwNRkVAJhdrQQqgf0ZYuKn3JYR866YeN7TSI20wiI1UNlW3xV1IZsITJdNKYYkKZNp4v0K5PXKWK/uP1k2pQRmo++0600x9Km0qxmVohyL1nvy/sF2ONweHFIndKZVBJ9iLZrPp5XnxlxOI6LRi4FKColtqGLJ2ExdE2YgUfaxmAxhsxnh+M778c+oJFL6ETeuwPX5vu+VeEZFG09OpPSjHilg748vUDnR50rZVlq9s6eWAlAyEG26JtEGsewtjhtqPHtU9DtUwqkpzEJZnhVOjxdb6zsjvp4o9VXYMjG1THnNcGPKe5rsaOtxIstixDx1a6zIdmw8FD17IzIqRdlWbY/M+wfbtAV2eRkmlIcIJr+0oBpXLZmAn146M+r3IKLRj4FKCjl1GRXAd6BdqMkfUZ4oybHGnb4OPEE5qdJPhPFkILnSj+hRiXfhG5B4RkU00qZiK61eZX4mZo7LgywDb+5t0T6v7VAJsewtnFinfgZcHhxWMw+hJn4ESZK0LbWR+lQcbo82mZaT4Zum+SRM+Uf0p5w2sUh7L0X25sMjHVEPjRRnWRVkm7UsySd1nVowNa08L+Tf/UyLET+6eIb2eyKisY2BSgrpe1QAYGZV+IbaNvX8nOKc2HeoCL4RZVH6CR4Djn3qJ/xmWiDZHhXfCv14icAp3h4VbdlbGtZ9Lz+5DACwdnez9jmxPr8ixmVvgC+j0uv0RCzV7G/uhlcGirItUdeXL4qhoVa/7C3bYtKyJOGmhcQ48bIpvj1As6vykWE2oKPXiQMt4UfHZVnWSltF2VZMKMpCpS0DTo8Xf96oHFI4tTx8loiISGCgkkK+qR//jMqhtl6tj0TQMioJ3FDF5E9Xvwter6wr/fgCgthLP6JHJXJGJbEeFbWZNpEelczkMiqp2kqr95npSqDy3oFWLcAQBxKGWp8fTl6GSTuQMdLYt74/JVrW7dRapSSzpe5E2J032t8TixFGg6T1m2w/3hV0/EO/04MP1ebcZVN8Da0Wk0ELcCL1qdj73doG24JsMyRJwhI1qyJ+X6H6U4iIAjFQSSFnQEalJNeK8rwMyLJvekPQttIm0Ech+kZO9Dn9zvLJ1e1CiXXhm1Y2CrOULbkelWTGk5XfS3ece1Ra05hRmV6Rh3H5mRhwebVsg69HJfbASJKkmN5XsTo/cDImlCmlOcjPMqPP6Qm7QTcwezahKAuF2RY43d6gPiqxNr/SloFJJf57SRapQdHGCNkbsZU2x2rSTuU+PWBDcyy/LyIiBiop5OtR8f30KxpqA8s/2g6VBG6o+hOURdbCICmr+4VYSj+yLEcv/eQk06OSzHhyYs20IqNSlsLRZEGSJCw/WWmq/c8epfwjelTinU6JZZdKLKPJgsGg71MJnekIbJyWJAnzavIBBJd/xNr8ZVNKgrI5iyYq32fTofB9KmKHSkG27++VmPwRTipjoEJE0TFQSSH9ZlpB7FMJbKhtTeBAQkGUfjr7XdrNJ9tq8ruh+Eo/4TMS/S6Plp4PW/qJ8wA9v9cfgmbalhQvewu0XC3//GdPC/qcbi2Ai3V9vhAtoyLLckyjyXqiJBMuoyKyZ/oDKH0NtZ1+zxWNtMtOCj6n6pTqfFhMBrT1OLQx40Bi626hbrdMaV4GpqjbdasKMsMGx0REegxUUihURmWOugH03f1tfn0AyZR+tB4VXUYlsHQTy9SPfp16uKxHYZwH6OlpGZUEelRsCS58a+5OX0YFUMoeuVYT2noceGNnEwAlKIz34MRovT8NXQPoHlAO7Qu1Oj8UkZ0L1/fS4wjuR9I31PY7PXhrbwvu+cdObW2+WGmvl2E2apttN4UZUxYZlcIs//dFvyOFiCgWDFRSSAQiVt1Y7OmTi1GSa0VbjwP/3t2kfV5M/STWTOu7yYU65weIrfSjlQICsjF6YkJlwOWNujwuUJ8j+R6VfpcnqNEzEpFRKUlTRsViMmjbUv+44SgAZXV+vCPm0Xap7FGzIpNKcmIes9amwcKUy/R/3sKcqnwYDRKa7AOY8+N/4+o/fISn1d/XkklF2nUGEvtUNoUpM4XKqADANafX4qypJbjujEkx/Z6IiBiopJDL4z/1Ayg7Vb6ysBoA8OeNR7XP+0o/yYwnu7Q9KIGBSixTP9G20gLKhIi4Ucbbp5JMj4p+AV2s5/34b6VNT0YF8E3/iH0g4+KY+BGildRE2SfS/pRAvkxb6NfsDnGuU6bFiLlqdsTp8WJcfia+uqgGv71iPn535cKw3+u0KOPQWkYl2//vVnVhFv5w9ana2n8iomji/1GXwnJ4/Be+CV9ZVINH3/4UGw914EBzN6oLs7R+gXjO+RFERqWrX1f6yUi89BOuPwVQGi4Lsyxosg/gRK8LVQWxXaPHK2PApbwfiQQqRoOEHKsJPQ437ANuFMVQIvPbSpuGqR/hrKmlMBkkuNX+nnh2qAhaRiVMmWZPU3z9KYB/71Iovh4V/+DhoS+dgo2H2jFvfAEmlWTHlB2aofZeNXYNoM/pDupDCpdRISKKFzMqKeRy+6/QFypsmTh3mjIt8ueNR7VsitVkCDsWHInIqPQ43Noa/cDyirhxRM6oRA9UAP1NNfaMij5ACsz2xCreE5TFqHBBllkbiU0HW6ZZm3wB4p/4AXRTP2EzKupochy7RmyZvgA21OnGPWGOS6gpysKXFlZjcmlOzCUsW6ZZ+3tz/ER/0OPhMipERPFioJJCTo9ycw7MqADAFYvHAwBe2HIcdR19AJRG2kROf83L9J2gLG4SgTcfUfqJ1KPia66MfDMRNxtx84lFn25s2prgKvt4T1B+7qN6AMr21HQTW2qB+Cd+gMhTP31ON460K9M08WRUREOvLIfePxNtZ068RMnrWGeIQKWPGRUiSg0GKikULqMCKNMTE4qy0O1w43frDwNIvDyhP0H5eKcS9OQkMfUTLaMibjYinR+LXt2yt0SCMSC+EeXWbgf++rESqFx/xsSEvl889IFKYhmV8FM/h9t6IcvKc+KZCrOYDMhW/9w7+4NfN9oW4nhVFSi/b2ZUiCidGKikkDNEM61gMEj42mlKVkUcapfIaLIg+hGOqz/NZgeMAIu+kEilH3usgUpW5DJFKGJCKDOB/hQhnqVvf/jgMBxuL+ZU52NxwGKxdKguzMLZU0tgyzRjltqvEQ99RiVwadrRdiX4HF+UFffr5gccWKkX7aTseGkZlRCBygn2qBBRijBQSaHAFfqBvjC/yq8MUpIb/8SPIPpUxE0isEFSlH6cHi/c4c5+GQjdXBlI9Ki0xxWoBB+UGK/cGDMq3QMubVT4m2dOSjiDE68nvr4AH9+1POwIbyQio+Jwe4OyXiJQmVCUHfR10dgijCiHWviWjHEioxJQ+nG4Pdr3EtNNRESJYqCSQr6Fb6Hf1vwsCz43p1L7dUlSGRX/n5xzAjIq+kxGuPJPrKUArUwRR6CiHUiYTEYlxhOU12yqQ/eAG5NKsnHe9LKIz00ls9EQ9s86mqwIY99H1f6UmsJEMir+J2vr9YSZEEtUVYFyfcdP9Pl9XmRTjAZJy4oRESWKgUoKuaJkVABo5R8gsdFkIT9g42dg5sJqMkA9oDds+UeUAvJiDFTimfpJ5kBCIZYTlAdcHq3n54YzJ8FgGJxsSrLE2Dfgu7ELWkalOPFApStERiXauU7xEqWfwIxKuzjnJ8syaNktIhq9GKikUKgV+oHmVNm0tfoTi2NbjR5KfsDK9sB0viRJUSd/utWpn2g9C4mc9yO+Z1I9KjGcoPzCluNo7XagwpaBS04Zl/D3Ggrhxr59GZVESj/he1R6QmymTYYo/bR0O+Bw+/6OicCrKIGSGBFRIOZlUyjUCv1AkiThya/Px7b6LiydnHjTpy2g9h/q5pNpMaHX6Qlb+hE3rtwYe1Ti2UwrmmkDm3zjEa2Z1uOV8dt3PwUAXLtsYsyr5oeLULtUHG4PGtUToCck1Ezr21qs53B7tB6qVDXTFmVbkGE2YMDlRWPnACYUK4FVe4iTk4mIEjWy/mUf5mIp/QDKyb6fmV6WVFq8IErpB4h+3k+s48lFulHaUIvEQul1JH5yshBtPPm1HY042t6HgiwzvnxqdcLfZ6iE2qVS39EPWVYCz8IEMhK+8378g8oeXVYqmXKcniRJqAxR/hGBVxEnfogoBRiopJAjSjNtKgX2qIT6KTnaeT/2GHsWROOuV45tpwmgy6gkNZ4ceeGbWPB25ZIJSQVEQyXULhVR9hlflJVQIKs/WVtPBKXZFiOMKezj8TXU+gIVEXgxo0JEqcBAJYVCHUqYLvmZsZR+Ii99822mjXyTt+hW/cda/hEZlcwkAghxXeGCo2PqtMmSScUJf4+hFCqjkswOFUDXoxJQLvNN/KQ2eAi1nVb03HCHChGlAgOVFBI9AOZB6JWwxVD68TXTBmckXB6vdmhgLOOq8fap9LtSkFERpZ8wPSot6plJpWk8gDCdImdU4m+kBcKPJ6d62ZsQajut+DtSmMWMChElj4FKCmkr9Acho1Kga6aVJCDLHBwQRNpOq5+kiWUKpDDOQEXrUUliwkSUfnqdnqCldT0Ot9Z7k86TktMpVPB3VD0HanwCO1SA8OPJqV72Jvi20/p2qWiBShJ7goiIBAYqKRRtM20q6ceTsy2mkPtDIpV+xLK3TLMRphgCq0hn04SSih4VfaZH3GgFcQJ1tsWY1PbboRRqj4oo/dQkWPrJ140n61fzx1rmi1eo7bS+jArHk4koeQxUUsjlHrwelTx9oBJmBDjSHpVYJ34EkcGJdY2+r0cl8UDFbDRoWaHAhtoWdYS3NC/+k4uHC9FsKno63B6vlplIZH0+4MuouL2ydjAkEP+fd6xE6aepawAedSKsQzvnh4EKESWPgUoKOQaxR0U5QVm56YRL58dS+on1xhVq50ckfa7kN9MC4UeURX9KMscQDDX90QSyLKOxawAujwyLyYDyBAOwDLNR2+Oj71PpTvGyN6E0NwMmgwS3V0azfQBer6xl3RioEFEqMFBJEVmWB3XqB/D1OIS7+YiJm0iln1inQMQER0dvjOPJ4qyfJBa+AbrJn4CeC1H6KckbuYGKyFK5vTK6HW5f2acwK6mjAEItffP1qKS2wdVokFCRrwRVxzv70T3g1jIrHE8molRgoJIibq8M0RIwWIGK6FMJN8mRytKPllGJuUclRRmVMOf9jPSJH0DJfois14leJ46IiZ8EG2mF/BBr9HvSNPUD+DfUiq20OVYTrKbkglQiImCYBCqPPvooJkyYgIyMDCxatAgffvjhUF9S3Fy6qZTBWuUu1uiHCwZ8pZ/g8eR4T9KNu0fFmfzpyUD4E5RbupUelZE68SPod6nUiYmfBPtTBDG6rt9OKzJo0Q6gTMS4fN/SN5Z9iCjVhjxQee6553DrrbfinnvuwZYtWzBnzhysWLECLS0tQ31pcRHn/ACRDyVMJS2jErb0E33qJ9o5P4JvPNkR0/NFRiWZ8WQgfEalVcuojNxmWsB/mupIm28rbTK0NfohSz+pD1SqdJM/7T1iKy0DFSJKjSEPVB566CFce+21uPrqqzF9+nQ8/vjjyMrKwu9///uhvrS4iEDFICGmcd9UEOf9DEbpp0xt7mzuckQ978fl8WrvRzLjyYC+mTb0ePJILv0A+l0qLl1GJclAJcQulXQtfAN8I8rHdBkVnpxMRKkypIGK0+nE5s2bsXz5cu1zBoMBy5cvx4YNG0J+jcPhgN1u9/sYDgZzh4pw+pQSZFuMWDIp9CnMEad+HPHduMptGTBIyu+zLUpWRR8YJXsGT7gTlLWpnxEeqIjtrR29Dt36/ORKP+JsJv3UT1ozKrqDCUVpsIA7VIgoRYY0UGlra4PH40FZWZnf58vKytDU1BTya1avXg2bzaZ9VFcPj1NznYN4IKHwmell2PGjFTh/ZkXIxyOXfuI7+8VsNGhZlYbOgYjPFcveTAYp6cAt1Hiyy+PVloqNlozKvqYe9Ls8MBokrTk1UbYQpZ907VEBdEvfTvSjQy39FOUwUCGi1Bjy0k+87rjjDnR1dWkf9fX1Q31JAACXRymHWAcxowIg4hhrpNOTfePJsd+4KtUbaINuC2koWn9KkmUfwBdI6Ztp23qUbIrJII34n9zF9tZP6k8AACrzM5IO7rTx5P7gHpVUH0oIABW2TEiScnr4gZYeAMyoEFHqDOnu8eLiYhiNRjQ3N/t9vrm5GeXl5SG/xmq1wmodfj9FD0VGJRpRdonYoxJHKaAyPxObj56IHqioW2lTsdpeK/3oMiotdiVQKc6xJrVvZDgQGZVDrUojbaIbafXEeHJXqPHkNJR+LCYDynIz0GQfwI7jXQDYo0JEqTOkd1WLxYL58+dj3bp12ue8Xi/WrVuHxYsXD+GVxW8oelSiiVT66Ymz9AMoP+0D/ue6hJKq0WQg9AnKWiPtCF72JgSO8dYkuUMF0GdUlDKMw+3R/n6mo5kW8JV/REmOUz9ElCpDfle99dZb8eSTT+Lpp5/Gnj178M1vfhO9vb24+uqrh/rS4jIcMyqZEVfox1/6Eb0Tx09EK/2IQCUVGRXlpqs/7Xk0rM8XAkskyU78AME9Kj269y7ZBXzhBPbVcI8KEaXKkB87+1//9V9obW3F3XffjaamJpxyyil44403ghpsh7vBXp8fiyy1R8Xp8cLt8fqNTSfSXFlpU3tUugavR0Vb+KYv/XSLAwlHfqASeENPduIH8O9RkWXZ75wfY5pKZSKjIjBQIaJUGfJABQBuvPFG3HjjjUN9GUnRMirDsPQDKOWfXDVQ8Xpl9Djj36shbkZRp35S2qOi3HR7HG54vTIMBsl3zs8IX/YGBJ+Hk4qMihhPdrq9GHB50zqaLFQxUCGiNBk+d9URTmRUrMMoo2I1GSCpP0Dryz+9Trd2LlFeXD0qvj6EUOUk/esDqZr6UW6usuzb/TJadqgAwaWfVPSoZFuMMKmZk85+Z1qXvQn60o9Jd7I3EVGyhs9ddYQTzYpm0/CZQpEkSSv/6Cd/xI3LbJTiGqfOyzBpP5VHKv+ksvRjNRm1axQNtaNlKy2g9DSJYKw015qSvh5JkvxOUB7sjEpBtgWSNHz+PyCikY2BSoo43MOvRwUAMtUbn37yR7/sLZ4biiRJ2uRPpBHlVDbTAsHn/YymQAXwlUlSMZos6BtqE2mcjlelLqNSyB0qRJRCw+uuOoKJ0s9wmvoBgEyLcj36jEqPQ7lxJfITdmUMkz+9Wo9K8hkVwNdQ2z3ghizLuh6V0RGoiPJPTQr6UwTRp9LV74z7pOxEZFlMWsDF/hQiSqXhdVcdwUQz7XDaowIAWWY1o6ILVOxJrFOPZTtt2jIq/S509bu0MttoCVR8GZUUBip+GZX0l34AX/mHgQoRpdLwuquOYK5huPANCL30LZlzX7RdKhEmf3pT2KMC+J+gLBpp87PMsJpS8/pD7XNzKlBbnI3PTA+9jTkRNt2Isq9HJfXr8/XE3w0GKkSUSmzNTxHncO1R0ZppfUu/RM9CIjeucTFkVET2JlXLxfQZFbE+fzQsexM+P7cKn59bldLXFGv0lWZa9c87zZM4M8fZ8PrOJkwqSV2vDRERA5UUcaqHEg63jEpWiO20YlNpIiOkWuknwtRPr/oTfFaKelRydUvfWntGz7K3dBJTP139vlHydI8MX7tsIk6bWIg5Vflp/T5ENLYwUEmR4bhCH0h96UdM/TR2DmgL2AL1pTqjojtBWWRUSkfBsrd00o8ni7JkuntULCYD5o8vTOv3IKKxh4FKigzbHpWQe1TEuGr8pZ+yvAwYJGVvTFuPA6V5wQGDWPiWmaoeFd0JymKaerQ00qaLfjzZq273S3fph4goHYbXXXUEG64ZlVClH7HhNZEbl9loQFle5FOUU96jkiEOJnSNuh0q6SLGk/2baRmoENHIM7zuqiOYtkJ/mGVUMlJc+gH0DbWhJ39S3aPia6Z1awcSMqMSmRhP7upz+i34IyIaaYbXXXUE82VUhtfqcLFHJVWlHyDyLhVZltPQo+Ir/Yymc37SKT/EeHI6F74REaUL/+VKEYdneI4ni9LPq9sbUN/RhwnFWTjc1gsg8RuXtp02RKDi9Hjh9io9EanrUfGt0O/sU4IsNtNGJsaT+5we7XgHln6IaCTiv1wp4hIZlWFW+plTnQ+zUUL3gBvrD7Zh/UHfY6LhMl7jIpz3o++FSd3CN+WvaWu3AwOu0bWVNl1yM0yQJOXUaY+XzbRENHLxX64UcQ7TjMqptYX48AfLcbC1B0faenGkvRdH2vtQkGVOeN9FpF0qYiutxWRIWWOxaKYVQYrVZEj7TpCRzmCQYMs0axkoAMhJUSmOiGgw8V+uFBmu48kAUJBtwcLsQiyckJodF5EOJuxT+yGyU5RNAXylH6E0zxrXqc9jVb4uUMmxmkLuvCEiGu6G3111hBquK/TTQQQqJ/pcfqv5Af05P6mLga0mg9/7OprW56eTLct35g77U4hopBr9d9VBMlxX6KeDLdOMXPXGFziiLDIqqepPAQBJkrSlbwAbaWOVr8tEsT+FiEaq0X9XHSTDdeFbuoQbURajyVkp/gk+TzdKzXN+YiNGlAGOJhPRyDU27qqDYDj3qKRDZZjJH7E+P5U9KoD/jZaln9j4ZVRY+iGiEWps3FUHATMqCi2jkuJARd9Qy4xKbPQ9KsyoENFINTbuqoNguK7QTxcRqBwLzKhoPSppLP2wRyUmzKgQ0WgwNu6qg2CsZVTGhcmoaAcSpuicH0HfTMtlb7HR96jkWHnODxGNTGPjrjoInGOsR2VcQeiDCdMxngwEZlQYqMSCzbRENBqMjbvqIBiuhxKmiyj9NHb1w6uuaAeg7VVJV4+KJAGF2ZYozyYAsGWyR4WIRj4GKikgy/KYy6iU5VphkACXR0Zbj0P7fK8jXRkV5fWKsq0wjZHyWrL8Sz8MVIhoZOK/+Cng8cqQ1aTCWNhMCwAmowHleUpTq/4U5X6XOp6c4h6VXLX0w7JP7LjwjYhGg7FxV00Rp9sbtDIe8PWnAGMnowLozvzRBSrpyqicWluIqoJMfG5OZUpfdzTTn46dm8FmWiIamcbOXTUFPv+b93HGA29jwOXx+7zL7evRGCsZFQCoLc4GAPxja4P2uXT1qFTmZ2L998/BN8+alNLXHc1MRoN21AFLP0Q0Uo2du2qSehxu7Gqwo63HgcYu/0kXh0cJXCQJMI6hE2qvPWMiTAYJa3c3Y92eZgD6jEpqAxVKzNzxBci1mjBRDSqJiEYaBioxatIFJ2KpmeASBxIaDZCksROonFSWi2uW1QIA7nl5F/qdHvS7xB4V/gQ/HDx11UJs+MG5KOCkFBGNUAxUYtRs9wUqPQGBihhNHktlH+E750xBpS0Dx07049G3Duo20zKjMhwYDRLLPkQ0oo29O2uCGiNmVMbWaLJettWEuz83AwDw23c/xYk+J4DUN9MSEdHYNPburAnSZ1TE9lVhrK3PD7RiRhnOnloCl0fWymCpPj2ZiIjGprF5Z01AY5d+BDeg9DOGMyoAIEkSfnzxTL8DGbNYbiAiohQYm3fWBERqph1r6/NDqSnKwqqzJ2u/zjQzo0JERMlLW6By5MgRXHPNNaitrUVmZiYmTZqEe+65B06n0+9527dvx7Jly5CRkYHq6mo88MAD6bqkpDTpSz+O0KUfi2ls35yvP3MiLpxVgf8+vXZMjWkTEVH6pC0/v3fvXni9Xvz2t7/F5MmTsXPnTlx77bXo7e3Fgw8+CACw2+0477zzsHz5cjz++OPYsWMHvvGNbyA/Px/XXXddui4tIX4ZFWeYZtoxnFEBAKvJiEdXzhvqyyAiolEkbYHK+eefj/PPP1/79cSJE7Fv3z489thjWqCyZs0aOJ1O/P73v4fFYsGMGTOwdetWPPTQQ8MqUHG4PWjr8WWCwo4nj9EeFSIionQZ1DtrV1cXCgsLtV9v2LABZ5xxBiwW3zKqFStWYN++fThx4kTI13A4HLDb7X4f6dZid/j9Olwz7Vid+iEiIkqXQbuzHjx4EI888giuv/567XNNTU0oKyvze574dVNTU8jXWb16NWw2m/ZRXV2dvosW12n3X5kfvkeFgQoREVEqxX1nvf322yFJUsSPvXv3+n3N8ePHcf755+OLX/wirr322qQu+I477kBXV5f2UV9fn9TrxSLwbJ9IK/SJiIgodeLuUfnud7+Lq666KuJzJk6cqP13Q0MDzj77bCxZsgRPPPGE3/PKy8vR3Nzs9znx6/Ly8pCvbbVaYbVa473spDSrgUqu1YRuhzuomdbpVjIsZmZUiIiIUiruQKWkpAQlJSUxPff48eM4++yzMX/+fDz11FMwGPxv5IsXL8add94Jl8sFs9kMAFi7di2mTp2KgoKCeC8tbURGZWJpDrbVdwY104qMipUZFSIiopRK2531+PHjOOuss1BTU4MHH3wQra2taGpq8us9+epXvwqLxYJrrrkGu3btwnPPPYdf//rXuPXWW9N1WQlpsitbaSeVZAMA+gJ7VNhMS0RElBZpG09eu3YtDh48iIMHD6KqqsrvMVlWMhA2mw3//ve/sWrVKsyfPx/FxcW4++67h9VoMuDboTK5NAdA+M20bKYlIiJKrbQFKldddVXUXhYAmD17Nt577710XUZKiEBlUokaqDjdkGUZkqQseGNGhYiIKD14Z43C45XR3K3sURGBilcGBlxe7TnMqBAREaUH76xRtPc44PHKMEjA+KIs7fP6hlqu0CciIkoPBipRiImf0twMmI0GZFuUgwf1fSrMqBAREaUH76xRiECl3JYBAMi2Km09+owKe1SIiIjSg3fWKJrV9fnlef6BSp/TN6LMjAoREVF68M4aRXBGJbj042JGhYiIKC14Z41CZFQqRKBiCVH6YUaFiIgoLXhnjaKxS9lKKzIqOWrpxz+joq7QZ6BCRESUUryzRiGWvYkelSwRqIToUWHph4iIKLV4Z41AlmU0aaWfTABATogeFae2R4VvJxERUSrxzhpBV79L20BbmmcF4OtRCbVHxczSDxERUUrxzhqBmPgpzLYgw6xkUrJC7FFxMaNCRESUFryzRtAUsEMF8JV+/PaoiEDFxBX6REREqcRAJYKmgB0qQJjNtGI82WgcxKsjIiIa/RioRBC47A0IN54selSYUSEiIkolBioRNKuBSoWu9JNlCR5PdrjZo0JERJQOvLNG0Kj2qJT5lX64Qp+IiGiw8M4aQZO6lbYiSulH9KhwMy0REVFq8c4agWimrYjQTOvxyvAqG/SZUSEiIkox3lnD6HW4YR9QgpEyXY+KWPjW5/RAlmUtmwLwUEIiIqJU4501DLFDJcdqQm6GWfu86FHxeGU43F5thwrAQIWIiCjVTEN9AcNVc4jRZMCXUQGU8o8s+x4zGTieTERElEpMAYTR2BW8lRYADAYJWRbf5I+2Pt9kgCQxUCEiIkolBiphaOvzAzIqgG6XisOj20rLt5KIiCjVeHcNo8UePPEjiPN+ep1u3Tk/fCuJiIhSjT0qYfzo4hm4eflJCFXN0Y8oZ6qnKpuNLPsQERGlGgOVMCRJQkG2JeRj2oiywwNnJjMqRERE6cK7awL0a/Rdbq7PJyIiShfeXROgL/1oPSoMVIiIiFKOd9cE6M/7cbGZloiIKG14d02ANp7s5HgyERFROvHumoAcXY+K06OspmVGhYiIKPV4d01Atq7042QzLRERUdrw7poAfTMte1SIiIjSh3fXBIjx5D72qBAREaUV764JEAvfenSlH2ZUiIiIUo931wTox5PFHhWu0CciIkq9QQlUHA4HTjnlFEiShK1bt/o9tn37dixbtgwZGRmorq7GAw88MBiXlJQsNVDxK/0wo0JERJRyg3J3ve2221BZWRn0ebvdjvPOOw/jx4/H5s2b8Ytf/AI/+tGP8MQTTwzGZSVMjCfrm2k59UNERJR6aT+U8PXXX8e///1v/P3vf8frr7/u99iaNWvgdDrx+9//HhaLBTNmzMDWrVvx0EMP4brrrkv3pSUs1HgyMypERESpl9a7a3NzM6699lr86U9/QlZWVtDjGzZswBlnnAGLxXdK8YoVK7Bv3z6cOHEi5Gs6HA7Y7Xa/j8EmAhW3V0aPww2AUz9ERETpkLa7qyzLuOqqq3DDDTdgwYIFIZ/T1NSEsrIyv8+JXzc1NYX8mtWrV8Nms2kf1dXVqb3wGGSZjdp/n+hzAmCgQkRElA5x311vv/12SJIU8WPv3r145JFH0N3djTvuuCOlF3zHHXegq6tL+6ivr0/p68fCZDQgw6y8dSf6XAAAM0s/REREKRd3j8p3v/tdXHXVVRGfM3HiRLz55pvYsGEDrFar32MLFizAypUr8fTTT6O8vBzNzc1+j4tfl5eXh3xtq9Ua9JpDIcdqwoDLiU5mVIiIiNIm7kClpKQEJSUlUZ/38MMP42c/+5n264aGBqxYsQLPPfccFi1aBABYvHgx7rzzTrhcLpjNZgDA2rVrMXXqVBQUFMR7aYMq22pCW48THb1KRoXNtERERKmXtqmfmpoav1/n5OQAACZNmoSqqioAwFe/+lX8+Mc/xjXXXIPvf//72LlzJ37961/jl7/8ZbouK2Wy1O20zKgQERGlT9rHkyOx2Wz497//jVWrVmH+/PkoLi7G3XffPaxHkwWxS8XtlQEwo0JERJQOgxaoTJgwAbIsB31+9uzZeO+99wbrMlJGjCgLXPhGRESUery7JkgcTCgwo0JERJR6vLsmKNtq9Ps1DyUkIiJKPQYqCQos/TCjQkRElHq8uyYoJzBQYY8KERFRyvHumqAs9qgQERGlHe+uCcoJ6lHhW0lERJRqvLsmiD0qRERE6ce7a4KCAhVmVIiIiFKOd9cEcY8KERFR+vHumqDAPSrMqBAREaUe764JChxPNjOjQkRElHK8uyaIPSpERETpx7trggJ7VLhCn4iIKPUYqCRI36NiMRogSQxUiIiIUo2BSoJMRgOsal8KsylERETpwUAlCaJPhaPJRERE6cE7bBJE+Yfr84mIiNKDd9gkiIZaZlSIiIjSg3fYJIhdKhxNJiIiSg/eYZOQxR4VIiKitOIdNgk5ao8KAxUiIqL04B02CaJHhc20RERE6cE7bBKy2aNCRESUVrzDJkEbT2bph4iIKC14h00CMypERETpxTtsEkpzMwAA+VnmIb4SIiKi0ckU/SkUzoWzKtDvdOOck8uG+lKIiIhGJQYqSci0GHHF4glDfRlERESjFks/RERENGwxUCEiIqJhi4EKERERDVsMVIiIiGjYYqBCREREwxYDFSIiIhq2GKgQERHRsMVAhYiIiIattAYq//znP7Fo0SJkZmaioKAAl156qd/jdXV1uPDCC5GVlYXS0lJ873vfg9vtTuclERER0QiSts20f//733HttdfivvvuwznnnAO3242dO3dqj3s8Hlx44YUoLy/HBx98gMbGRnz961+H2WzGfffdl67LIiIiohFEkmVZTvWLut1uTJgwAT/+8Y9xzTXXhHzO66+/josuuggNDQ0oK1POynn88cfx/e9/H62trbBYLDF9L7vdDpvNhq6uLuTl5aXs90BERETpE+v9Oy2lny1btuD48eMwGAyYO3cuKioqcMEFF/hlVDZs2IBZs2ZpQQoArFixAna7Hbt27Qr72g6HA3a73e+DiIiIRqe0BCqHDh0CAPzoRz/CXXfdhVdffRUFBQU466yz0NHRAQBoamryC1IAaL9uamoK+9qrV6+GzWbTPqqrq9PxWyAiIqJhIK4eldtvvx0///nPIz5nz5498Hq9AIA777wTl19+OQDgqaeeQlVVFZ5//nlcf/31CV4ucMcdd+DWW2/Vft3V1YWamhpmVoiIiEYQcd+O1oESV6Dy3e9+F1dddVXE50ycOBGNjY0AgOnTp2uft1qtmDhxIurq6gAA5eXl+PDDD/2+trm5WXssHKvVCqvVqv1a/EaZWSEiIhp5uru7YbPZwj4eV6BSUlKCkpKSqM+bP38+rFYr9u3bh9NPPx0A4HK5cOTIEYwfPx4AsHjxYtx7771oaWlBaWkpAGDt2rXIy8vzC3CiqaysRH19PXJzcyFJUjy/najsdjuqq6tRX1/PRt0043s9ePheDx6+14OH7/XgSdV7Lcsyuru7UVlZGfF5aRlPzsvLww033IB77rkH1dXVGD9+PH7xi18AAL74xS8CAM477zxMnz4dV1xxBR544AE0NTXhrrvuwqpVq/wyJtEYDAZUVVWl47ehycvL41/8QcL3evDwvR48fK8HD9/rwZOK9zpSJkVI2x6VX/ziFzCZTLjiiivQ39+PRYsW4c0330RBQQEAwGg04tVXX8U3v/lNLF68GNnZ2bjyyivxk5/8JF2XRERERCNM2gIVs9mMBx98EA8++GDY54wfPx6vvfZaui6BiIiIRjie9ROB1WrFPffcE1cpihLD93rw8L0ePHyvBw/f68Ez2O91WjbTEhEREaUCMypEREQ0bDFQISIiomGLgQoRERENWwxUiIiIaNhioBLGo48+igkTJiAjIwOLFi0KWvdP8Vu9ejUWLlyI3NxclJaW4tJLL8W+ffv8njMwMIBVq1ahqKgIOTk5uPzyy7WjFShx999/PyRJws0336x9ju916hw/fhxf+9rXUFRUhMzMTMyaNQsff/yx9rgsy7j77rtRUVGBzMxMLF++HAcOHBjCKx6ZPB4PfvjDH6K2thaZmZmYNGkSfvrTn/qdFcP3OjHvvvsuPve5z6GyshKSJOGll17yezyW97WjowMrV65EXl4e8vPzcc0116Cnpyf5i5MpyLPPPitbLBb597//vbxr1y752muvlfPz8+Xm5uahvrQRbcWKFfJTTz0l79y5U966dav82c9+Vq6pqZF7enq059xwww1ydXW1vG7dOvnjjz+WTzvtNHnJkiVDeNUj34cffihPmDBBnj17tnzTTTdpn+d7nRodHR3y+PHj5auuukretGmTfOjQIflf//qXfPDgQe05999/v2yz2eSXXnpJ3rZtm3zxxRfLtbW1cn9//xBe+chz7733ykVFRfKrr74qHz58WH7++eflnJwc+de//rX2HL7XiXnttdfkO++8U37hhRdkAPKLL77o93gs7+v5558vz5kzR964caP83nvvyZMnT5a/8pWvJH1tDFRCOPXUU+VVq1Zpv/Z4PHJlZaW8evXqIbyq0aelpUUGIL/zzjuyLMtyZ2enbDab5eeff157zp49e2QA8oYNG4bqMke07u5uecqUKfLatWvlM888UwtU+F6nzve//3359NNPD/u41+uVy8vL5V/84hfa5zo7O2Wr1Sr/5S9/GYxLHDUuvPBC+Rvf+Ibf5y677DJ55cqVsizzvU6VwEAllvd19+7dMgD5o48+0p7z+uuvy5IkycePH0/qelj6CeB0OrF582YsX75c+5zBYMDy5cuxYcOGIbyy0aerqwsAUFhYCADYvHkzXC6X33s/bdo01NTU8L1P0KpVq3DhhRf6vacA3+tUevnll7FgwQJ88YtfRGlpKebOnYsnn3xSe/zw4cNoamrye69tNhsWLVrE9zpOS5Yswbp167B//34AwLZt27B+/XpccMEFAPhep0ss7+uGDRuQn5+PBQsWaM9Zvnw5DAYDNm3alNT3T9sK/ZGqra0NHo8HZWVlfp8vKyvD3r17h+iqRh+v14ubb74ZS5cuxcyZMwEATU1NsFgsyM/P93tuWVkZmpqahuAqR7Znn30WW7ZswUcffRT0GN/r1Dl06BAee+wx3HrrrfjBD36Ajz76CN/5zndgsVhw5ZVXau9nqH9T+F7H5/bbb4fdbse0adNgNBrh8Xhw7733YuXKlQDA9zpNYnlfm5qaUFpa6ve4yWRCYWFh0u89AxUaEqtWrcLOnTuxfv36ob6UUam+vh433XQT1q5di4yMjKG+nFHN6/ViwYIFuO+++wAAc+fOxc6dO/H444/jyiuvHOKrG13++te/Ys2aNXjmmWcwY8YMbN26FTfffDMqKyv5Xo9iLP0EKC4uhtFoDJp+aG5uRnl5+RBd1ehy44034tVXX8Vbb72Fqqoq7fPl5eVwOp3o7Oz0ez7f+/ht3rwZLS0tmDdvHkwmE0wmE9555x08/PDDMJlMKCsr43udIhUVFZg+fbrf504++WTU1dUBgPZ+8t+U5H3ve9/D7bffji9/+cuYNWsWrrjiCtxyyy1YvXo1AL7X6RLL+1peXo6Wlha/x91uNzo6OpJ+7xmoBLBYLJg/fz7WrVunfc7r9WLdunVYvHjxEF7ZyCfLMm688Ua8+OKLePPNN1FbW+v3+Pz582E2m/3e+3379qGuro7vfZzOPfdc7NixA1u3btU+FixYgJUrV2r/zfc6NZYuXRo0Zr9//36MHz8eAFBbW4vy8nK/99put2PTpk18r+PU19cHg8H/tmU0GuH1egHwvU6XWN7XxYsXo7OzE5s3b9ae8+abb8Lr9WLRokXJXUBSrbij1LPPPitbrVb5D3/4g7x79275uuuuk/Pz8+WmpqahvrQR7Zvf/KZss9nkt99+W25sbNQ++vr6tOfccMMNck1Njfzmm2/KH3/8sbx48WJ58eLFQ3jVo4d+6keW+V6nyocffiibTCb53nvvlQ8cOCCvWbNGzsrKkv/85z9rz7n//vvl/Px8+R//+Ie8fft2+ZJLLuHIbAKuvPJKedy4cdp48gsvvCAXFxfLt912m/YcvteJ6e7ulj/55BP5k08+kQHIDz30kPzJJ5/IR48elWU5tvf1/PPPl+fOnStv2rRJXr9+vTxlyhSOJ6fTI488ItfU1MgWi0U+9dRT5Y0bNw71JY14AEJ+PPXUU9pz+vv75W9961tyQUGBnJWVJX/+85+XGxsbh+6iR5HAQIXvdeq88sor8syZM2Wr1SpPmzZNfuKJJ/we93q98g9/+EO5rKxMtlqt8rnnnivv27dviK525LLb7fJNN90k19TUyBkZGfLEiRPlO++8U3Y4HNpz+F4n5q233gr57/OVV14py3Js72t7e7v8la98Rc7JyZHz8vLkq6++Wu7u7k762iRZ1q30IyIiIhpG2KNCREREwxYDFSIiIhq2GKgQERHRsMVAhYiIiIYtBipEREQ0bDFQISIiomGLgQoRERENWwxUiIiIaNhioEJERETDFgMVIiIiGrYYqBAREdGwxUCFiIiIhq3/D7EypAJ+oC8dAAAAAElFTkSuQmCC" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "all_rewards = []\n", + "\n", + "ppo = CrowdPPOptimizer(HomogeneousGroup(actor.agent), config={\n", + " \"gamma\": 0.99,\n", + " \"gae_lambda\": 0.95,\n", + " \"minibatch_size\": 256,\n", + "})\n", + "\n", + "for t in (pbar := trange(100)):\n", + " num_steps = 0\n", + " episodes = []\n", + " while num_steps < 2000:\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"mcar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await format_data_multiagent(datastore=cog.datastore, trial_id=trial_id, actor_agent_specs=cenv.agent_specs)\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + " num_steps += len(data.rewards)\n", + " \n", + " all_data = concatenate(episodes)\n", + " \n", + " record = convert_trial_data_to_coltra(all_data, actor.agent)\n", + " metrics = ppo.train_on_data({\"crowd\": record}, shape=(1,) + record.reward.shape)\n", + " \n", + " mean_reward = metrics[\"crowd/mean_episode_reward\"]\n", + " all_rewards.append(mean_reward)\n", + " pbar.set_description(f\"mean_reward: {mean_reward:.3}\")\n", + " \n", + "plt.plot(all_rewards)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:41:16.217109Z", + "start_time": "2023-12-13T19:36:45.952578Z" + } + }, + "id": "56b220d45561a042" + }, + { + "cell_type": "code", + "execution_count": 25, + "outputs": [], + "source": [ + "await cog.cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T19:41:48.682828Z", + "start_time": "2023-12-13T19:41:48.645559Z" + } + }, + "id": "e23f3f5ecd4ffc2a" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/lunarlander-teach.ipynb b/examples/lunarlander-teach.ipynb new file mode 100644 index 0000000..23bb0a9 --- /dev/null +++ b/examples/lunarlander-teach.ipynb @@ -0,0 +1,461 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2024-01-15T15:16:54.108165Z", + "start_time": "2024-01-15T15:16:52.130847Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "import os\n", + "\n", + "from cogment_lab.envs.gymnasium import GymEnvironment\n", + "from cogment_lab.envs.pettingzoo import AECEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.coltra_utils import convert_trial_data_to_coltra\n", + "from cogment_lab.utils.runners import process_cleanup\n", + "from cogment_lab.utils.trial_utils import format_data_multiagent, concatenate\n", + "\n", + "from coltra import HomogeneousGroup, DAgent\n", + "from coltra.buffers import Observation\n", + "from coltra.models import MLPModel\n", + "from coltra.policy_optimization import CrowdPPOptimizer\n", + "\n", + "from cogment_lab.actors import ColtraActor, RandomActor, ConstantActor\n", + "\n", + "from tqdm import trange\n", + "import matplotlib.pyplot as plt\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:16:54.168837Z", + "start_time": "2024-01-15T15:16:54.116233Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:16:55.754069Z", + "start_time": "2024-01-15T15:16:55.749952Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2024-01-15T16:16:55.746880\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:17:00.641327Z", + "start_time": "2024-01-15T15:16:58.281030Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "cenv = GymEnvironment(\n", + " env_id=\"LunarLander-v2\",\n", + " render=False,\n", + ")\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"lunar\",\n", + " port=9011, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:17:27.461202Z", + "start_time": "2024-01-15T15:17:20.645759Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Create a model using coltra\n", + "\n", + "if os.path.exists(\"models/lunar\"):\n", + " agent = DAgent.load(\"models/lunar\")\n", + " model = agent.model\n", + "else:\n", + " model = MLPModel(\n", + " config={\n", + " \"hidden_sizes\": [64, 64],\n", + " }, \n", + " observation_space=cenv.env.observation_space, \n", + " action_space=cenv.env.action_space\n", + " )\n", + "\n", + "# Put the model in shared memory so that the actor can access it\n", + "model.share_memory()\n", + "actor = ColtraActor(model=model)\n", + "\n", + "\n", + "await cog.run_actor(\n", + " actor=actor,\n", + " actor_name=\"coltra\",\n", + " port=9021,\n", + " log_file=\"actor.log\"\n", + ")\n", + "\n", + "random_actor = RandomActor(cenv.env.action_space)\n", + "constant_actor = ConstantActor(0)\n", + "\n", + "await cog.run_actor(actor=random_actor, \n", + " actor_name=\"random\", \n", + " port=9022, \n", + " log_file=\"actor-random.log\")\n", + "\n", + "await cog.run_actor(actor=constant_actor,\n", + " actor_name=\"constant\",\n", + " port=9023,\n", + " log_file=\"actor-constant.log\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "outputs": [], + "source": [ + "ppo = CrowdPPOptimizer(HomogeneousGroup(actor.agent), config={\n", + " \"gae_lambda\": 0.95,\n", + " \"minibatch_size\": 128,\n", + "})" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:17:36.574395Z", + "start_time": "2024-01-15T15:17:36.402215Z" + } + }, + "id": "582b6bb1bf0c81df" + }, + { + "cell_type": "code", + "execution_count": 7, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "mean_reward: -96.0: 100%|██████████| 100/100 [02:15<00:00, 1.35s/it] \n" + ] + } + ], + "source": [ + "all_rewards = []\n", + "\n", + "for t in (pbar := trange(100)):\n", + " num_steps = 0\n", + " episodes = []\n", + " while num_steps < 1000: # Collect at least 1000 steps per training iteration\n", + " trial_id = await cog.start_trial(\n", + " env_name=\"lunar\",\n", + " session_config={\"render\": False},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " },\n", + " )\n", + " multi_data = await cog.get_trial_data(trial_id=trial_id, env_name=\"lunar\")\n", + " data = multi_data[\"gym\"]\n", + " episodes.append(data)\n", + " num_steps += len(data.rewards)\n", + " \n", + " all_data = concatenate(episodes)\n", + "\n", + " # Preprocess data\n", + " record = convert_trial_data_to_coltra(all_data, actor.agent)\n", + "\n", + " # Run a PPO step\n", + " metrics = ppo.train_on_data({\"crowd\": record}, shape=(1,) + record.reward.shape)\n", + " \n", + " mean_reward = metrics[\"crowd/mean_episode_reward\"]\n", + " all_rewards.append(mean_reward)\n", + " pbar.set_description(f\"mean_reward: {mean_reward:.3}\")\n", + " \n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:20:01.409375Z", + "start_time": "2024-01-15T15:17:45.954719Z" + } + }, + "id": "56b220d45561a042" + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [ + { + "data": { + "text/plain": "[]" + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "text/plain": "
", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAGdCAYAAADnrPLBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAACJGklEQVR4nO2dd5gc5ZXu3+o8eUYzoxkJBZSQBBIgEGEENmC0SDZrLNtobZzAizPYpMUE2ziwGO9i7MWR5XIJexeWsMY4YYMQxiaIICEBkpBAOc5oRpNTx7p/dH9fVVdXdVfnMO/veeYBdVd3f1PTVXXqPe85R1FVVQUhhBBCSJniKPYCCCGEEEKygcEMIYQQQsoaBjOEEEIIKWsYzBBCCCGkrGEwQwghhJCyhsEMIYQQQsoaBjOEEEIIKWsYzBBCCCGkrHEVewGFIBKJ4NChQ6irq4OiKMVeDiGEEEJsoKoqhoaGMHXqVDgc1vrLhAhmDh06hOnTpxd7GYQQQgjJgP3792PatGmWz0+IYKaurg5AdGfU19cXeTWEEEIIscPg4CCmT58ur+NWTIhgRqSW6uvrGcwQQgghZUYqiwgNwIQQQggpaxjMEEIIIaSsYTBDCCGEkLKGwQwhhBBCyhoGM4QQQggpaxjMEEIIIaSsYTBDCCGEkLKGwQwhhBBCyhoGM4QQQggpaxjMEEIIIaSsYTBDCCGEkLKGwQwhhBBCyhoGM2XC4HgQv35+J/b3jhZ7KYQQQkhJwWCmTPjtGwfxb3/Zhp+tfa/YSyGEEEJKCgYzZULX4DgAYO9RKjPp8Ny2Lvxlc2exl0EIISSPMJgpMKqq4ob/fQv/9pdtab1uYCwIADg8OJaPZVUk7xwexOUPrsdX/nsDNh8cKPZyCCGE5AkGMwVmV88IHl2/H79+fif2paGy9MeCmc6BcUQiar6WV1Hc/udtUGO76j+eZXqOEEIqFQYzBeZQv6as/GXLYduvG4wFM8Gwip4Rf87XVWn8/d1u/P3dbridChwK8Ow7XXj7ANUZQgipRBjMFBh9MPPnNLwcIs0EAIf7x3O6pkojHFHxw6feAQB89sxj8ZGTjwEA/Mez7xZzWYQQQvIEg5kCc1AXiGzc14/DA/Y8MP2jumDG5msmKk+8cQDbOodQ73Ph6x+Yi69/YC4cCrB22xG8daC/2MsjhBCSYxjMFBi9MgPAdqVNnDIzQGXGirFAGD9+ZjsA4MoPzEVTjQezW2uxSqoz9M4QQkilwWCmwAhVZfExDQDspZoiERWD4wxm7PB/X9yFrkE/jmmswuc6jpWPf/38eXA6FDy37Qg27e8v2voIIYTkHgYzBeZQLM30z2cfCwB4fU8vjgwlD06GxkOyKif6HuWTZgqEIhjQpciMqKqKN/f346/bj2T9Wd1Dfvz6+Z0AgG+unA+f2ymfm9VSo1Nn6J0hhJBKgsFMAVFVFQdjgcjSmZNw0rQGqCrwzJaupK/Tp5iA8lJm/vmB13HSD57BP929Dg+9uhf9owEA0XTQo6/vw4d/8SI+8suX8Pn7X8ff3+3O6rP+a90ejATCOHFaAz584tSE579x/lw4HQqe396Njfv6svosQgghpQODmQJydCSAQCgCRQHaG3z44OIpAFL7ZvrHAnH/7kwjmNndM4LfbjwAVS18b5oRfwgv7ewBALy2pxff+u1mnHbbs/j0va/g9B8+ixt+8zY2HxyU2z+9JbtOvSJ9dMnpM+BwKAnPz2yuwceWRNWZW363BYFQJKvPI4QQUhowmCkgoqR6cp0XbqcDH1zUDgBYt+so+kYClq8TykxjtRsA0Dk4jrCNxnm7e0bw0V+9hGsefRPPZ6l6ZMK2ziGoKtBS68HNH1qAhVPqEQyreGnHUQyNhzBjUjVu+uAC3Ln6JADA89u7swq6tnUOAQAWTqm33Oa6C+ajocqNtw8OMN1ESIb8ZXMn7n9pd7GXQYiEwUwBESmmqY1VAKJKwcIp9QhHVKzZap1qEsHMvMm1cDkUhCMquoeSN87rHQng8/e/Jku6X9/dm4tfIS3eORxVXU6Y2oAvvX8O/nzV+/DMNe/Hty9ciAc+fxqe/5dz8eVz5uBDi6fA63LgYP8Y3jsynNFn9Qz70T3kh6IAx7XVWm7X3uDD7R9bDAD49d924tVdRzP6PEImKuGIimsf24Tv/2Er9vdyVhwpDRjMFJBDhmAGAD4UU2f+vNm6G7AISBqrPWir90XfK0mvmfFgGF/6r/XYc3QUIttSjAoeEczolZLj2urwhffNxrnzJ8tUUJXHiY45zQCAv27LzAi8PabKHNtcg2qPK+m2H1o8BRefOg2qClz72JsJniQ7jAfDuP7xN/HEGwcyWi8h5cqBvlGMBsIAgO5h65uq0UCIwQ4pGAxmCogMZhp88rEPLo4GMy/u6Ikrv9Yj00xVbkyJvdaqC3AkouKb//sW1u/tQ53Phf/45BIAwFsHBixnOnUP+fG7TQfhD4Uz+K2s2SqDmbqU235gwWQAyLiqSQROC9pTfxYAfO+iEzBjUjUO9o/hlt9tlo8fGRrHL/+6Ax//9ct4cuNBy9f/ftMhPL7hAL7/h622Un6EVAo7uzX1VBj6zfjy/9uA99/xV+w9OlKIZZEJDoOZAiKqkPTKzNzJdZg3uRbBsIq175inmsRcpoYqN9pFMGOhzPz02Xfx+zcPweVQcPdnTsWHFrWjyu3EsD8UdxLS870/bMFVj2zCNY9uytkQy0hElWrJ8Uk8LIJzj4sGM+v39FkGdckQfpkF7ak/CwBqvS789BMnw+lQ8LtNh3DnM9vxlf+3Actufw53PL0dG/b24YdPvWMZqPzp7aiSNjAWxJZDnPlEJg47juiDGetjdXvMM/fO4aFCLItMcBjMFBCjZ0ZwznGtAKLqiRnihNFQ5ZavNSvP3rC3Fz9/bgcA4IcfW4yz5rbA5XRg8bRog76NJqmmUDgiS6KfertTzjTKlr29USna63JgVktNyu1nNFdjTmsNQhEVL77Xk/bnbeuMKTM2VCDBqTObcOV5cwEAP39uB/6ypROhiIpTZjSi3ufCkSE/XjHx1PSPBvDSDm2NL+2g74ZMHHYe0ZSWPotgRlVV9MVUm+4UfbQIyQUMZgqISDMdYwhmhNrSM2wu2eqrmaYkUWbERfWDi9rxT0uny8eXTG8EALxpEsxsPjSIofEQPM7oV+HeF3fnpEpBpH3mt9fB5bT3NTtvfizVlKZvJhSO4N2u6N2i3TST4OsfmIv3H9eKpmo3Llt2LJ6++v144mtn4R9PivapMUs1PbO1CyGdYqMPbAipdHbYSDONBMIIhqPHSKpiBUJyAYOZAhEIRaRZborOMwMArXVeAECPxUEv+szUV7kxpSEaCB0y8cxsPhhVdpYeOynu8ZNiwYyZCVhciM9b0IobVi4AAPzgj1vxlySGZDtI86/NtE90DdFg5vl3u9NKd+05OoJAKIJqjxPTm6rTWqfL6cB//fPp2HjLBfjeRSdgfiwYEt2C/7K5E+PBeC/RU7EU00WxgOf1Pb0J2xBSiaiqGpdm6rMIZvStJo4wmCEFgMFMgegaHIeqAl6XA5NqPHHPtdTGghmLyoCBsRCAaDWTCITMGudtORQNIBZNjQ8gTo4FM9s6hzAWiL/oimDmrLkt+Mo5s/GZM2dAVYGrHtmEDXuTl3PvODIk0ztGth6yb/4VLD22CTUeJ7qH/NI8bAeRk5/fXmfaLC8Tls5swjGNVRjyh/CcTikaGA3KffaN8+dhcp0X/lAEb+xlR2FS+fSOBOKq/6zSTL0MZkiBYTBTIA7qUkyKEn/BTRXM6A3AUxqjwcyRoXGEwloH296RgPyM4w3BzJQGHybXeRGOqHFm1fFgGOtjF+Gz5rZAURR878MnYPnCyfCHIvj8/a9jg8VF+i+bO/HBu17Aql++ZLpuocwcP7XB9PVmeF1OnDW3BUB6qSbpl0lDBUqFw6HgopMTU03PbO1EMKxiflsd5k6ulesVnY4JqWR2GPpAWaWZ9IoN00ykEDCYKRBmPWYELbVRpaZvNIhgOLHFvjhhNFS50VLjhdupIKICXbqThEgxzW6pQZ3PHfd6RVGkOqNPNa3f04dAKIL2eh9mx0y6LqcDP7tkCZbObMLgeAifuffVBEPu7zYdxBUPv4FgWMV4MJIwW6p/NIBDMeUoHUMuoKWa0inR3nZYdP5N77NSIVJNz2/vlsMyRYrpQ7FRFCKYeZEmYDIB2NkdNf8Kj51VNZM+mEk1SJcUhi2HBnDzb9+u2L8Hg5kCIYIZo18GAJqqPXDG0iO9hrEGwXAEI7HUUGOVGw6HopVn66Znvx0LZk44xlwJEb4ZfUXTi7oUk14tqva48F+Xn473zWvBWDCMf37gdTk/6pHX9uHqRzchHFGlkdnY8E+kfaY1VaHeEFil4tz5rXKdxn1hRbpl2XaZ316HBe11CIQjeGrzYQyMBeU+u/DEaH+gs+ZGm/29faA/o+Z7hJQTQpk5MVYhaRnMjGiP9wwH2IupBPi/L+zGw6/uw+83HSr2UvICg5kCccikx4zA4VDQHPPRGCXZQd0Fsr4qGhhMqa+Ke08AMn20+BjzC7pZRdPLO0Uw05ywfbXHhXsvXYoPLmpHIBzB1x7agGse3YQbn3gbqgp8+owZ+H+Xnx57n/jZUltNOv/aZUpDFRa010FVgRfeSz1PamAsKNNr89OsZLLDR2LqzJMbD2LN1i4EwyqOa6vF3Ml1cr2zW2sQUWFaxk1IJSF6VZ16bBOAJAZg3ePhiGq5HSkc4mbL7k1iucFgpkBYlWULhG/G2B68P/YFrPO6pHojfDOdA4nKzCILj8riaQ1QFOBA3xh6hv3oHw3I14hUiRGvy4mfX7IE/7R0GiIq8NuYd+SL75uFf121CLNba01nS0m/TAbBDKClmp6z4ZsRjfmOaaxCQ1V6KpAdhG/m1d29ePDlPQCADy6aErfNWXOi++9llmiTCkcoM0tnRismRwNh087hxgvmkUH6ZorNsD9aSFKpCjKDmQKRzDMDAC0W5dnii9dQrV2ojeXZA6NB7O+Nvv8JFsFMnc+Nua3RAYyb9vXjlV1HoarA3Mm1ct6TGS6nA//28RPx5ffPhtflwNXL5+HmDy2UaSkxW+opXarJbCZTOpwbayL40o6jKadoa+bf3KsyQDRIOn1W9MQtgr8LTzQEM9I3w2CGVC5jgbBUQZfMaJRz38xSTcbHKtWnUU6MBKLBzOB4KOfv/S+Pv4nLH3hd3lwWAwYzBUIEHkJVMSJMwMbGeQO67r+CqY3xjfNEimnGpOq4oMeIMAG/eaBfXnjPtlBl9CiKgps+tBCbv78CVy8/Ls5f88GYEfalHT0YiBmY34s1sMtUmTlpeiPcTgU9w34c6LMeqAlo/px0jcbpIIzAQDT4O64t/rM6ZjfDoUTNkWYl84RUArt6osd1U7UbLbVeNFaLwoXEtIVRmWFFU/EZ8UcVtHwoM+t2HsXabUcwVsR+W3kLZm677TYsW7YM1dXVaGxsNN1m3759uPDCC1FdXY3Jkyfj+uuvRygUHzU+//zzOOWUU+D1ejF37lw88MAD+Vpy3hgcD0qJb2qDuTLTalGere/+K2ivF8FM9MIpU0wWfhnByTMaAUQrml6OVd8sm5Pol7HCbdLJN3pxj86WevadLuzqHkEgHEGt14VpTea/ayp8bqcMhN7Yl7x/Sz7Kso18aHE73M6YErV4SsLzDdVuLIoZr9kNmFQqIsU0J6bwNsZusMyUGRHgtNVHz2vsNVN8RvKYZhKqT43HmfP3tkvegplAIIDVq1fjq1/9qunz4XAYF154IQKBAF5++WU8+OCDeOCBB3DLLbfIbXbv3o0LL7wQ5513HjZt2oSrr74aX/jCF/D000/na9l5QaSYJtV4UGXxx7bqNTMwZqbMxM9n2hxrUGeVYhKcNK0RAPDa7l7s6hmBQwHOTCOYsUJ4SP68+TC2Ho4GVguybGC3ZEbUYLhxX7/lNvphlrkuy9bTWO3BpR3HYkqDD/+0dJrpNuw3QyodUZY9d3IsmKkWwUyiMiOCGaFiUpkpPiKYGcxHMBN77xqvK+fvbZe8BTPf//73cc0112Dx4sWmzz/zzDPYunUr/vu//xsnn3wyPvjBD+LWW2/FL3/5SwQC0QPh7rvvxqxZs3DnnXdi4cKFuPLKK3HxxRfjpz/9ab6WnRc0v4y1N6Wlzryaqd8kzTRFznLyIxCKyB4ziy3KsgUL2uvgczvgD0V72Zw4rTHt0mkzhFrx93d78NruqJJibNyXLqfMjAYzyZSZ/X3RYZYelwPHNqceZpkN3/7H47HupvMxzWJcgjABv7SjJ6XPh5ByZKdBmWmq1vpj6VFVVZZmCy8bg5niEomossVHrpUZf0ibw1WRwUwq1q1bh8WLF6OtrU0+tmLFCgwODmLLli1ym+XLl8e9bsWKFVi3bl3S9/b7/RgcHIz7KSYHhV/GIsUEAK21WoCiR1NmtBEIk2o88LgcUNWo9Lu7J3rHdEKKAMLldMQFPHb8MnY4rq0Wc1prEAhH8JsNBwBkbv4VnBJLiW09NGg590j4ZY5rq7U9zDJfLD22CR6XA12DfnkHS0glIcqyNWXG3DMzGggjEGv+KZQZGoCLy6juHDowFszpDdeoX3vvikwzpaKzszMukAEg/93Z2Zl0m8HBQYyNWRtDb7/9djQ0NMif6dOnW25bCA6nKMsGNGUmwQBskmZSFEWqM2vfiZZET23woTmWqkqGMAEDwDKT/jKZoCiKVGfESSzbYOaYxiq01nkRiqh468CA6TaF8MvYxed24rRY743/eW1fkVdDSG4JR1Tsit00acqMuWdGmH+9LgdmxhRTKjPFRaSBgOjfciSQO6Ou8IN6XY6i3lSm9ck33ngjFEVJ+rNt27Z8rdU2N910EwYGBuTP/v37i7oeW2mmWCDSNxqIm7k0MKaNMtAjgpk1sWBmUYoUk0B0Ava6HDgl5kvJBfreKw4FmN+WnYdFURSpzmy0SDWJMQb5KstOly++bzYA4IGX91gO4CSkHDnQN4pAKAKvy4FjYsb+plijzz5D5ZIIbpqqPZhcRwNwKaAPZoDcppqE+be2iCkmAEjr06+77jpcdtllSbeZPXu2rfdqb2/Ha6+9FvdYV1eXfE78Vzym36a+vh5VVdYqh9frhdebWqUoFKIs26rHDBA98B0KEFGjdzaTYxVLZtVMgFYVJVQLu8HMufMno2N2M86c3QyfO3eS4MIpdTi2uRp7jo5iVkuNpdE5HU6Z0YSnt3RZ+mZEwJCtCpQrzp0/GStPaMdftnTilie34NEvn5kwVJSQckSkmGa11MjmneIGq99wYeyNpZ2aajxojQUzo4Ewhv2hol/wJioj/nglZmA0mDRTkMl7V3uLl2IC0gxmWltb0drampMP7ujowG233YYjR45g8uRox9c1a9agvr4exx9/vNzmqaeeinvdmjVr0NHRkZM1FIqDci6T9ZfH6VAwqcaLnmE/uof9MpgxMwADif1qUpVlC2q9LvzPl860vXa7iFTTr57fmdKIbBfNBNwPVVXjAoMRfwh7e0cBlI4yAwC3fPh4/O3dbry2pxdPvHEQHz/VvPqJkHJClmXH/DKAZgA2VjMJpaap2o0arws1HidGAmF0D/kZzBSJ4XwqM6KSyVPcv23eElz79u3Dpk2bsG/fPoTDYWzatAmbNm3C8HD0oLjgggtw/PHH47Of/SzefPNNPP300/j2t7+NK664QqoqX/nKV7Br1y5885vfxLZt2/CrX/0Kjz32GK655pp8LTvnhCMqugajykyqSFg0ztPnl808M0BiYGRXmcknV35gLq5fMR//smJ+Tt5v8TENcDkUdA8lNs/701uHoarRRoF2vEKFYmpjFb5x/jwAwO1/fqdiW4eTicXOI7Gy7FZ9MBM9Jxmrmfp0ygwAeWN2ZJAm4GKRzzTTaImkmfIWzNxyyy1YsmQJvvvd72J4eBhLlizBkiVLsH79egCA0+nEH//4RzidTnR0dOAzn/kMPve5z+EHP/iBfI9Zs2bhT3/6E9asWYOTTjoJd955J+69916sWLEiX8vOOd1DfoQiKlwORUquVojn9SZg62BGU2Ym13kxuc7aj1Moqj0uXHHeXMvy5XTxuZ2yxFs/7VtVVTwQm5P0qTNm5OSzcsnlZ8/CnNYa9AwHcOcz24u9HEKyZkd3ojLTmEKZmRR7XjQEpW+meAhfiyCXvWaGZZqpjDwz6fDAAw+k7NY7c+bMhDSSkXPPPRcbN27M4coKi0gxtTf4ZK7ZCmMX4PFgWPaEMY4p0CszuUrrlCKnzGjCWwcG8MbePlx0UnTo4xv7+rD18CC8Lgc+sbS4lWpmeFwO3PqRRfjUva/iv1/Zi39aOr0klDNCMkFVVZlmilNmarRqJn0auE8agKPPt8a6ALOiqXgYPTOD47lPM9UW2TPD2Ux5RlYyJfHLCIzDJoUq43QoqDNEvfrKqBMq+EK5xKSi6cGX9wIAPnLyVClllxrL5rbgwydNRUQFfvDHrcVeDikAQ+NBBEKR1BuWGUdHAhgYC0JRgNmtWnNK4ZkJRVQM6dIYvcY0Eyuaik4hqpmqK9UzQ6KIYZDJyrIF2rDJ+GCm3udKqIppqHKjKlaNtCjLbruljCgf3xJrnndkcBxPvR2d0P25jmOLuLLUXBXzzmzSpchIZTIwGsSyHz2Hz9z7arGXknNE599pTVVxFZA+txM+d/QSMqDzzYi0kwh2WmUwQ89MsSiEAbhiPTMkip2ybIE2nyl6MhCVTCI3rUdRFHxwUTuOaazCGbNz0/yuFJnWVIWW2mjzvM0HB/A/r+1HKKLi1JlNJZ+6EWnDQCiCYLjy7tiJxtbDgxgaD2Hr4crrLyQ6Ws/RpZgEjVWJXYB7Y6MMNGUmeiOXyzRTX0wtIvYQAYcYmJvbYCaawqphmqmyET1SZkxKbYoVwUy3Ic1UX2U+P+knnzgZL95wXoI5uJLQN897dXcvHno1mmL6XMfMIq7KHvq+C0aZt5hYjYcgmbO/L9omoBL37d6j0WDGbP5Zo0lFk740G9DSTLkKZvyhMJb/5G/40F0vIBzhHDQ7iFSQ8FrmQ5lhmqmC2XpoEG8dGIDbqeAfjm9Lub1xcrZVJZOeidCUTfSbuefvu3BkyI/WOm9cx+FSxe10wOOKHmJGmbdY/Mez7+LE7z+D57cfKfZSKgrROiAUUStOhescFOpyYqrc2GtGVVWtNDshzZSbYObIoB9HRwI42D9GU7FNhHoi/oaV2AGYwUweeWx9dIzCBce32+qFIg763thIA3GCaKxg5cUOwjcjDsBPnT5DBgmljjjAR3M4CyUb1u08ikAoglt+twX+UGmsqRI4EGvgCFSeOtM5EA1m2upNgplYRZNQY8Z0FZiTDAbg3pFATgI9/TDeA32jSbYkAqGeTM2DMjMs00wMZiqS8WAYv914EADwT6fZKx+eVBMdaaCq0YBm0IYyMxEQzfMAwOVQSrK3jBUij1wqyozwYe3rHcV9L+4p7mIqCH1Tx/FgZSkzQlExC2a0ydnBuP96nA5Ux0aaNFV75PGrD0Qy5aiuD5dofUGSI84/onN8LvvMjMoOwPTMVCRPb+nEwFh0/sXZc1tsvSY60iBW0TQUkDNPJnowU+VxyvlLKxe1m55USxXR4rtUPDO9OqPmL557j11Zc4ReIagkZUZVVanMtJspMzFfjLjTl36ZGrdMgTscikyhHxnMPpjp1Q22NHYGJ+aIVJAoRBkYi/YGygUiUKIyU6E8+no0xXTxqdNSNsvTo/fNWA2ZnIhcfvYsLGivk+XO5YI4wEshmFFVVV5sZjZXYyQQxr/9hR2KsyUQiuCwLiispGBmcDyEsdjv095g7ZkRPpnekXi/jGByDhvn9Yzo00wMZuwwKj0z0WAmGFbl3zXr9w4wzVSx7Ds6ipd3HoWiAKuXpjdoUF/RlKqaaSKxaskx+MvV78e8ttIZKmkHLZgp/gVu2B9CKFb98aOPnQgA+M0bB9gHJ0sOD4xBf5NbSWkmMVeuocod12NGIFRjLc1kEczk0ATMNFP6CPWktdYrb65z5ZuRgyZZml15COPv2XNb0p5TpG+cJ/vMMJgpW0SLb+NslGLQF+v/UeV2omNOMz5+SjTQ/t7vtyDCEteM2d8bf0EdryBjdbIUE5BYzSTnMhk6c+eycd5Rne/mYBID8A/+sBVfePB1frcR39hOBKC5CmaGK31q9kRgw94+PLu1Ky73GApH8PiGaDDzydPSN6pqwyb9NABXAKL3QikYgGWb+Vja8oaV81HjcWLT/n48uelgMZdW1hgraiopzSTKskWayIisZhLBzKh5arw1h43zjo7EKzNm3o/xYBj3v7wbz75zBPt6K7/iKRiOxHVh1hOJqBjRpYJkMGOxfTqEwhFZvcY0U5miqir+9U9b8YX/Wo9Vv3oZf3u3G6qq4u/vdaNr0I+majeWHz857ffVdwGWfWbomSlbakvIM9NnnJlT78MVH5gLAPjZ2veKtq5yx+jbGCuRMvxcIAziVsqMnJw9Ep9mslZmcptmGg9G4oIbwe6eEZn6KwVVNN/c8rstWHLrM9jeOZTw3KguuK71uqRtYXA8+/0yovuuM81UpgTDKs6c3YwqtxNv7u/Hpfe9htV3r8MvntsBAPjYKdPgdaX/x9UbgEU1k2gZTsoPcYCXgmfGLAXwsSXRVNP+PvM7XJKa/UZlpoKGTQplxsz8C2hppiF/CMFwRBqAjSNYcuqZGYl/j4MmJuBdsREMQGUFl2aMBcL47cYDiKjAmyb+N1E67VAAn9uR0zSTfkxCJte7XMJgJkM8LgduWLkAL9xwHr5w9ix4XQ6s39uHN/b1AwA+YbO3jBExOXvP0RHZqptppvKllKqZ+kxmfdVXRdcXjqgl09iv3DAqMxWVZhqw7jEDRM9Nogn5wFhQ+vwm1cSfs0Qw05NlMKOqqgyYpsYCLLOKpl3dw/L/Ryr8e/3397ql6Vw/I0ugL51WFCWnwcxoiUzMBhjMZE1LrRff/sfj8fdvnofPdcyUowuOy7DqRhiAxQHqcTrkZFpSfsg0UwlI3VKZ0aUtq9xO2dBsKAey80Rkf8yTcUys7NVfQcFMV4o0k9OhoN4X/T71jwYslZlW3XymbBTAwfEQguHo6xdPiw6aPdif6InZ1aMpM6MlcCORT57Z0iX/v8/EByMHQcYCjobYDUwughnR/bfYowwAoPgrqBDa6n34wUcW4Tv/eDyymZYkJi2L472h2j0h5i9VKpoBuPgXOGEA1l9oFEVBfZUbvSMBDI4HLdMJxJzxYFimTua11eJg/1jO+neUAqnSTEDU7DswFkTfaFBWNU2yCGYC4Qj6R4PSt5UuopKp1uuKTfHuskgzacpMJSuOoXAEz23TgpmBsWTKTDQNJJSZXHQBLpWybIDKTM5xOx1wOTPfrZNqPNDHLkwxlTeyNLsE7g77LcyZdb5owJXLFucTBdHnpNrjlBOJK6XPTCgckeMHrKqZAN1Ig5GADJiN3zGvyykrnLqzGGkglJ/mWg+OaYrub2OaSVXVOM/MaAmoovli/d4+w8RyM2UmfhBkPjwzTDORBFxOR9xdDXvMlDel5JmR3VkNFxqRJmCaKX3EhXR6U7VMB1eKZ6Z72A9Vjc5Da6mxDmZEqX/n4LgM5My6lrfmYKRBz7AWLIm0nrFxXvewH0O6462SlRmRYqqP3ZD0mygzIsUtzkXieM9JMFMiE7MBBjMlSYtuwjaVmfKmpqQ8M9GTV5PhQiNMwIPjVGbSRfSYmdZUJTvkVooyIxrmTa7zwpFkJIuoaBJqiNupmF7chLqTTeM8UcnUXOOVDUkPGirx9KoMULkGYFVV8czWTgDAR04+BoA2SFaP8MxUe/KhzIj+NUwzERNa6rQ7ZwYz5Y02aLL4J1SrVvN13tzl0Ccaovvv9EnV8MVKUyulA7Aw/7al8FEJFUaYbhurPaY+v8k5aJzXG1NmWmo1ZWbIH8LgmHazYAxmKtUAvK1zCAf6xuBzO/Dhk6YCMK9m0tJM8Z6ZXKaZit39F2AwU5LEKTNsmFfWiDuWYncAVlU1oWmeQFNmKvOkn0/0ykyVJ5ZmqhAlINUoA4EIjnf3RE23RvOvIBeN847qPDNVHieaY9/lA7qKJr35F4hvGpcJ+3tH8cQbB2SrjFJBpJjeN68VU2IBp5kyY5xqXZ+PYIZpJmIG00yVg5DbA6EIguHipR9GAmFZ0mq82IgcOtNM6bM/5pmJSzNViDLTOZi8x4xAKDPCP2TmlwFy0zhPGJInxTw802ImYH1Fk1CIjm2OpqGyVWa+/4etuPaxN/G3d49k9T65RqSY/uH4NrnP/aFIQpPAvBqAY59VzTQTMYPBTOWgd/mPFjHVJHrM+NwOVHniTzx1IpgZozKTLgelMqNLM1WIZ0aMMkgdzESDY2FbMVYyCaQyM5iFZ0aXZgIgK5r0JmChzCw6JtqHJlsDsHjvQ/3ZD8nMFQf6RrHl0CAcCnD+gsmo9bpkvyijCVg/lwnQ1P5AKJK1WV0GSkwzETPEgQpY3+WQ8sDjcsATK9UfLqIJWFYymaQARJppiMpMWowFwrK6ZnpTNbwVVs2k9ZixrmQCEg3lVj1kZrXUAAC2Hh7MOGUjS7NjyozwzQhVKBCKSLVscY6CGdHSIFdTpnPBs1ujKaalx05Cc60XiqLoSuTj16mVT0eD7VqPCw5d1+ZsMKawigmDmRJE3MEAVGYqgZoS6DVjZf4F9GkmKjPpIPwydT4XGqrdqIqlmSqlaV6nTWXG+J0yBjeCE6Y2oM7nwtB4CG8fHMhoTaKaSag/+oomANjXGx0DU+NxYmZzNHjKts+MOHZKKQ37TCyYueD4NvmYuPHtN5iAjWkmh0PRhk1mGcyMBljNRJIQn2bikMlyR9y1FNMEbDXNGGDTvEw5IP0y0QtqpZVmd9k0ABvVY7OAGYiOPuiY3QwAeGlHT9rriUS0uUwyzWToNbPjSNQvM7u1Vl5gs1FmxoNh+fcslTRs/2gAr+7uBRD1ywhEENlvOI7N1JNc+WaozJCkUJmpLMQdUTE9M70jYshk4vdJ3KUxzZQeYlr29JhvQwQzlTCbaWg8KL0W6Ssz1jdgZ81tAQC8vDP9YKZ/LAiRnRKpLK0LcPRvsStWUTW7tUb61bLp8aSvDiqVYP/V3b0IR1TMm1wr1SdAu/E1lmeLYK42D8EMq5lIUibVeOB1OeBQ4v0zpDwpBWXGapQBoFNmmGZKi0RlpnI8M12xSqY6ryvlhara45S+MMDaAAwAZ82NKjPr9/SlvZ/EXKaGKjfcsc8TwUzfaBCjgZDsMTO7pVZ6RIzVPemgDwxKJc20J1attWBKfdzjUpkZNffM5EOZkWmmEjAAF38FJAG304GfXbIEo4FQwvRZUn6Ik2oxPTNW04wBnWemRO48ywUxLVuUB1eSZ8ZuwzwgOqy0odotm+ElGyI5p7UWk+u8ODLkxxt7+7AsptTYQZitm3U3ePU+N+p9LgyOh3Cwb0xWMs1urclJw0p9MFMqBuB9se/dzEnVcY+L/W70zAwbDMBA7nrNGIdYFhMqMyXKihPa8dEl04q9DJIDaktgpIH0zCRJM/lDEfgrpEdKIZBzmSZVnmfGbsM8gd70a2UABqKBj0g1vZRmqkn6ZQxzoo6JKWMH+sZkj5nZrTWy98lYMIxIhtVTpZhmEsHMjOb4YEaoLX0Wyow+zZSr+UyjJu9dLBjMEJJnSiHNJOcymdw1609EHDZpH333XwBaaXYoHDcrqByxW8kk0Ct+yZQZAFg2J5pqenHH0bTWJOcyGVLvYv9vPjggg49ZLTVxSkSmalkpKjN7j1ooM9VCmdHWGYmoCX1mgNykmfTvzanZhEwASsEAnKw02+lQUOdlRVM6DPtD8g54msEArKpAoIjdnnNBl80eMwKhxrh03yUrhDLz9oH+tC6m+onZekRF09/f6wYATG3wodrjgs/lhBgRlakqGqfMjIeKHqQGwxFZuWVUZsxKs/VBXK4NwKMW710sGMwQkmfEHWKplmYD+oomKjN2EKpMY7VbdlAWnhkAGA8UP5jZcmgAP1nzbkYG2PTTTNHvldWQST1TG6swq6UGERV4dZd9daZXKjPxAZYIJt/Y1w8gWpYNRPupSB9ThiZg0TkbAMIRNesGfNlyqH8M4YgKj8uBtrr4v40IZvRqkkgxORTNoA5owUw2Ny+jFu9dLIq/AkIqHCHvFssArKpq0jQToK9oojJjBzktu0m7O3Y7HXDGWquWwnymO595Fz9b+x6efacr7dd2xcy8k9NMMyXzy+gRqaaXd9oPZoyjDAQimBFdhWe3auXK1VmagI09W4qdahIpphmTquFwxAeNIqDUr3FYN9VaH2TmQpmxeu9iwWCGkDxTbAPwSCAs0x5WFxthCKQyYw+jX0bgc5VOebaoLuodCaTYMhG7DfME4nuVyi8jOFuYgNNonnfUMs0Un26Z3aIFMzXSBJxpmil+3xU72N9rUckE6NNMQZkOE0Gcsbw+F8GM1XsXCwYzhOQZTZkpzgVOSOVelyMuFaJHzGeiZ8YexkomQSlVNIkLVbrNEMMRFd2xni7tNkqzAW3u0hydKpKMjjnNUBTgvSPD0p+TCmkATqhmig8oRZoJ0FJ/mR57xsqggdHiHh/7LSqZAE2ZCUVUqZpYlU7nJJiJ3ZyVwsRsgMEMIXmnpsh9ZvTmXys5WE7OZprJFkKZEeZTga+Ees1owUx637ueYT/CERVOhxI3WiUZyxe24bEvd+BbFx5va/vGag9OmBpt+ma3G/DREfM0U1O1O65ySZ9mEjcSmXpdjN10i91Ycu/RaOm5mTLjczvhjSmDwrgs5lIZDbq5UWZKpywbYDBDSN4pdmm2nJidJAVQ7xOTs5lmsoOY02Pcp6XSBTgSUaUiM5Tm906Yf1trvdIDlAqHQ8HpsyaldWE7a45INaX2zQTDEXmBNqaZFEWRQaXP7cDUBi3AFEFOpsMmxWeKdFupeGb0Ywz0CHVGBGFawzzzYGY8mHlvqZES6v4LMJghJO/UFNkzo10ErM2ZuZqiO1EQyosxbaelmYobzAwHQnKOUboBamca3X+zQXT/fXlHT8qSZ5EqdSjmXaxFqunY5po4Y6zsvp2BMhOJqNIzI9I6xTw+VFW1bJgnaDSMNLDytdT5XLJsPdMBmiMl1P0XYDBDSN6pLbJnJtkoAwHnM6XHeMpgprieGb23I13PzBERzNTZSzFlymnHNsHtVHBoYBx7YoqDFSLFNKnGY6oWCSP2HJ1fBtAUibEMbiSGxrWAcEYsrVPMNGzPcACjgTAUJdF4LjCWZ2upoPjvqUPXDyhTtamUhkwCDGYIyTvizqVYnhk5ZDJJMKNVM1GZsYNUZjzxp9BSSTPFledmqMzYNf9mSrXHhZOmNQIA3tzfn3Rbq0omwTnHTYbP7cA/HN9m+IzMDcD9YwH5HsI7VMw0077eqF9mSr0PXpe5GmIszx5OEnA0VGfnmxH7tBS6/wIcNElI3hE5ZX8oglA4ApezsPcQvdIAbCfNRGXGDqIJm8+gzFSVSJpJryCknWYaiFYN2R1lkA0ilZWqfNyqkknwD8e3YfP3ViQcW+IinokhW1QyNVV7dE3mind8pEoxATplZiTeAGwWzERvYMYyTp2NBMxVn2JBZYaQPKM/kZjdIf587Xv41z9uRTBJC/xgOJKxgThVwzyATfPSxcoz4y2VYGYs8zSTHGVQgGCmyaQFvxlHTSZmGzG7SdBKs9M/dkSqprHaLVsXFFOZ0WYyWZe/NyYYgK1NutlWNCVTfYpBaayCkArG43LA43QgEI5gOBCS8i4QPSHcueZdANFA4t8+fmJC+fTB/jF85t5X0TPsx4s3fECehOySapQBwKZ56SI9Mx6DZyYm/4+HiuyZ0Qcz6VYzFSjNBOirb5JfUIUyY7dUXCBSvJmUZvfrWhpIZaaIwf6+o6mVmSZD6iiZSTfbYGZU1wG4FKAyQ0gBkCdVw4XliK5h2GPrD+CnscBGsL93FJ/4z3XY3TOCofEQdnYPp/3ZaRmAs7jzfLdrCOvSaE9frgTDEQTDUWdoogE4ekrNdBZQrjC2tI9E7A9I7EpzYnY2iAuqsZ+LkVSeGSuqPKLPTAbKTEzRbKx2y2C/mNVMsvtvsjRTVbwyk6wXTPbKDDsAEzLhECY5Y6pItJz3xCTynz23Aw+9uhdA9E7sk/e8IrvNAvGD7+zSZ8cALAZN+kNyxk26fOm/1uPT976Cd7uGMnp9uaBPIVl6ZvI8m+mXf92Bj/ziRUulQH+BUlX7bQFGAyGpzrXV57eaCTCfJ2SGqGZKlmYyo8aTY2Wm5NNMIjiMKTNJPDPZBjMTpjT7tttuw7Jly1BdXY3GxkbTbRRFSfh55JFH4rZ5/vnnccopp8Dr9WLu3Ll44IEH8rVkQvKGVXm2aBt/8oxGXHX+PADAd57cjPtf2o1P3rMOB/vHMLulBouPaQCgndTtoqqqZmRM0mdGKDNAZs39IpFoD4yIiowGG5YTwi+jKJAdVwUiuPHnuTT7NxsO4M0DA9iwp8/0eaNR1W76UCggHpejIJ1dxXcytTKT3ABsRbVUZjI3AEc9M9l3zM2GEX8IPbF9MMOk+69A+OIGpDIj1JPEgCPb30maiys9zRQIBLB69Wp89atfTbrd/fffj8OHD8ufVatWyed2796NCy+8EOeddx42bdqEq6++Gl/4whfw9NNP52vZhOQFcTKxUmZa67y4evk8fPK06YiowPf/sBWHBsYxp7UGj3zpTMybHO2fka4yMxoIIxASQyat72q9Lq0VeiZ3n0N+rSfHX7cdSfv15cR4ILo/q9zOBH9ToUqzxftbVQEZL1B2A1TZYDHJ6ItcIg2rI/lRZqqzGCUiJmY3Vntkh+yRQBihJEb9fCEqmRqq3HGeOyONVQZlJomvhQZgm3z/+98HgJRKSmNjI9rb202fu/vuuzFr1izceeedAICFCxfixRdfxE9/+lOsWLEip+slJJ9owyYNnhkRzNR6oSgK/nXVInQP+bF22xHMm1yLh794JlrrvPKOK90JyGJ7j8sRN7/GjPoqN7qH/BmZHPUB0Ia9fRgYDSY96ZYzVpVMQOFmM/ljAaqVomG8QNmtaOrVVfAUAnHxTVXN1CuqmdL0zGhTs7NJM2nKDBBVuexOB88V+2z4ZQAtOBwcDyKsGziZjzSTULs4mynGFVdcgZaWFpx++um477774tpar1u3DsuXL4/bfsWKFVi3bl3S9/T7/RgcHIz7IaSY1HrNjYh6ZQaIlpf+6jOn4N7PLcVvvrZMPj4pw2CmX/bKcKe8085mPlO/rhologJ/e6877fcoF8SF0eiX0T+Wb2VGBDNWaUfjBcpuZ+d+G5VvuUSohSM6BdHIeDAsK7Ka06xmqnJn3n1bP6DV7dRuBoqRapKVTElSTIAWoKhq9AbDjgE4Ux+QnPtU6Z4ZO/zgBz/AY489hjVr1uDjH/84vva1r+HnP/+5fL6zsxNtbfEdHdva2jA4OIixsTHj20luv/12NDQ0yJ/p06fn7XcgxA6aAdjgmYkFM5N1reO9LieWH98mKygA7eKSyltgpFd3Qk5FXRYVG8YTfCWnmkSlkrEsGyjcOAMRLFmlHYW65oq1/rfbBdjOUNJcUl/lljOCRMddqzW5nYoMuO2ilWZnV80EoKjl2Xtj3X9TKTN6r1PvaACjse+JWcCRTTCjqmp5T82+8cYbTU27+p9t27bZfr/vfOc7OOuss7BkyRLccMMN+OY3v4k77rgj7V/CyE033YSBgQH5s3///qzfk5BsqLUYaWBUZqwQwUy6BuB07rRlF+BMlJnYhUikXv72bnfGVVGljtVcJqAwnplQOIJQbN9afR/EBWpKY7S82q7a1qdT8gqB06HIi2q/Ra+ZXt1cpnR9PCLgHAuG0ypPj64n/kZA3FzkUpl5fU8v1mxNbZi3U8kkEMHX4f5xiERHrkuz/aGI9MilSl8XirRCquuuuw6XXXZZ0m1mz56d8WLOOOMM3HrrrfD7/fB6vWhvb0dXV/wfuqurC/X19aiqMh+0BQBerxdeb/7LCgmxi8hZG42YR9IMZtI1AMs7bRvKjJZmSv/kJi5EHXOa8fqeXvSOBPDmgX6cMqMp7fcqdZJ6Zlz5TzMFdAZUs++DqqryAnVMYxX2947Z/puK90tWxp9rGqvc6B8NWgYzPRlWMgGa8VVVo+XyducIBUIROWk7QZnJ0UgDVVXxhQfXY2g8iFduPh+T66z7+tgZZSBorHbjQN8YDvZHX+NQzL+rzbUeKEo0xXdkaDzp5xvRn8dKpZoprVW0traitbU1X2vBpk2b0NTUJAORjo4OPPXUU3HbrFmzBh0dHXlbAyH5oMbEMxOOqOgdsRfMiGAkXc9Mn0wbpL7T1tJM6Z+sxcWzpdaD989rxZ/ePozntx2pzGBGzGUyuSMVSkA+00z69zb7PowFw7Kp3zGN1QB6bVczaS38CxjMVHuAo6OWKVQ7owys0F/ERwP2gxmhyjgUTZHJ9UiDwbGQfK+9R0ctg4lQOIKDsV5TqdJMgHauEP2pajwuU0WrzufGwvZ6bD08iFd29eKik6baXrtQmKs9TjhMppgXg7x5Zvbt24dNmzZh3759CIfD2LRpEzZt2oTh4WgH0z/84Q+49957sXnzZuzYsQO//vWv8cMf/hBf//rX5Xt85Stfwa5du/DNb34T27Ztw69+9Ss89thjuOaaa/K1bELyQo3J9N6jI35E1OgJM9Vdp6jiGBwPJZ3hZKRPV2qbCnGyzsQTIE7KDVVunLdgMgDgue2V6ZvRlJnE06dMM+WxaZ5f9969JgGACEadDgXtDdHvlf00U2ENwEDq+UzakMn01+RwKDKgGU3DBNyv+z6Li3V9jj0zXUNa9+9D/dYe0MMD4whFVHhcDrTZUE+EgiQCoGSl0x1zmgEA63b22FqzoNQmZgN5LM2+5ZZb8OCDD8p/L1myBADw17/+Feeeey7cbjd++ctf4pprroGqqpg7dy5+8pOf4Itf/KJ8zaxZs/CnP/0J11xzDe666y5MmzYN9957L8uySdlhlmYSfpnmWi+cKe5u6qvccCjRSqG+0YBtSbg3jTttbT5TBsGMbDDmwTnHRdXbzQcHcWRwHJML0Ba/kCTzzHgLkGbSKzP9o8GESez6wFKqbbbTTPGm10KQaj6T1mMmM+tAjdeJsWDYdhdkQKdo6o6bXI80ODLol/9/qH/ccjvhl5neVGVLBTEqM8mqjZbNacb/fXF32mNISm1iNpDHYOaBBx5I2mNm5cqVWLlyZcr3Offcc7Fx48YcroyQwlNr0memW9djJhVOh4LGag96RwLoHbEfzPSNpGEAlvOZMjcA11e50VrnxUnTGvDmgQE8v70b/3RaZVUT2qlmyudsJr9B9ekfC8YNYNQHM+J7Z7eaqRjKTEN1cgNwpnOZBFUZjDTQd/8V5LoL8BGbyoxWyZTa/AtoStfB2HsmqzY6fdYkOB0K9hwdxaH+MUxttPai6im1hnlACfSZIWQiYKbM2DX/CjLpNWN2UrYiGxldXIhEE7Rz58dSTRVYop28z4xIM+XPM2MclWD8PoiLbX2VW46pSDfNZMcwnivEZ1mlmY6YtC9Ih5oMhk0aK5kAfWl2bgzA4vcCgMMD1sGM3R4zgobYmsV7JjPo1vncWBQblZKOOiNSdqVi/gUYzBBSELR+F9rdod2ybMEkm63f9aSnzIg0U+YGYBE0fSDmm3lxR49lM7RyJVk1k3gsEIqkXQpsF2MKyyqYaajSpj0P+VN/Z8YCYZnCKmSH2yY5HNEimIlN8c40XVmdhTKj72ItlMucKTO6NNNBG2kmO+ZfQNuf4uuXSj1ZFvPNvJxGMFNqQyYBBjOEFASzcQZmDfOSoSkz/hRbRokOmUynaV5uDMAAsPiYBrTUejDsD2H9nt6036+UERf8ZOMMAK1Lb64xvq8xmBGejnqfC7U++2km8V1xOxVpWC8EjSk8M0LByHSKd3WulZmCp5nSC2aMKmyqgKNjtmYC1nfgT0ayadzFgsEMIQWgxmNiAB5OT5nR5jPZO5mOBcPywmfnTrs+i5O1VGaqop/jcCg457jKTDVJA3ASzwyQv/lMqYKZeAOw/TSTvidRIYZMCsTFd8AkmAmEInJd6fRB0ZNMmVFV1VQ57DdpHpjN8WGGPs00oBs9YFzf3qNRz8wMGw3zgESzf6qAY+mxTXA7FRwaGJf9bFKRbIBlsWAwQ0gBECY8fygip+52D6brmUkuxxuRQyadDlt32voLn907NCBqSBUXigbdQL73zWsBALyxr8/2e5UDss+MiTLjdChwO6OBQL4qmtJJM9WlkTrULuCFHaKoVTMlfq9FwO92Khl3JZbBjElp9rWPvYnTbnsWXYPxaR6zfju5HmfQPRSvsJr5ZrqH/RgNhOFQ7HtmjH+/VOMGqj0uLJke7QdlN9UkxrJQmSFkgqE/6EVnUanM2Cw5nRTrRWN3pEG/zvxr505b+CtCETUtVUFcPBVFC4gAYF5bLQBgV8+I7fcqB5J5ZoD8D5u0m2bSVzMFwpGEKigjco6XjQaLuaRRV81kDKKlX6bOl7FaVC1SvCZpphfe68HAWBAvvBffZ8UssKvXdQBOJ9i3Qvxu4ntk5pvZ0xNVSo5pqoLHZe9y3VhlSDPZUE/OlP1m7AUzowF6ZgiZkHhcDnnHLiTatA3AQpmxGcwcTcP8C0TvYEW/m3TKswdNGowBwKyWqCzePxpMu3NxKTOWJM0E5H/YZCplRigH9bpgBkitzpj1VikE4vMC4UhCKijdij8zqi3K5fUduN/c3x/3nOY10wIDocwEwpGs/7bD/pC8qVk8LVpNdNjEN7MnlmI61mZZNhA/vBOwF3DoTcB2AjWWZhMygdGbgEcDIXlCsFulIU76dpWZdIMlRVF0qSb7Urq4i20w3BFWe1w4Jta3Ymf3sO33K3WSNc0DtPLsfHtmxAXLmJ7Rp5mcDkUGNCmDGanMFDaYqfY44Yk1/es3+FE0ZSaLYMZCmREduAFgU0IwozWBFNR4nBCxerapJvF71XicmDs5qmCamYD39KQfzDgdilRZAXsBx5IZjfC6HOgZ9mPHkdTH6giDGUImLnoTsAg0qtxO25UjYuSBXWUm3WAG0HU5TeNkrZl/E9MTs1ujJ+FdFRTMJPPMANqwSX/egpno+4pGeaKpnMBYWWa3cZ6mzBQ2zaQoiiyBNn63tUqmzLtI11gYgHuGtM965/CgDFJVVZXVTPrKIEVRctY4T/bOqffJgN8szZRuWbZA/ze0E3B4XU4sPTbqm1m3K3WqSezLQla9pYLBDCEFQusCHI4LNOx6AYSXoXc0YEsKziiYkfOZ0ilj1dIaRua0xnwz3dn5ZroGxzMas5APUqWZ5LDJPM1nEimOqQ3RC3wyZQaAbbWtr0gG4OhnmncBFr1YslJmLAzAwrMGRH1iWw8PAojebIRiko1xX+SqPFufPpvaGP07mhmAd8eUGZGytYteUbI7cmDZnKhh/+UdqYMZppkImcCIGSkjgVBGgYZQZgKhiMy3JyNdgzEA1HnTP1n3jyVK8gKhzGSTZhoYDeKcO/6KT97zSsbvkUtSpplc+fXMCGVmSkP0jv7oSHxwKzsA++KDmVQBajG6/wrEd0eMxRCIYYyTM+wxA2h9ZoxpJmM1kfDNiIDK63IkBKxin2atzOjSZ+LvaEwz6cuy7Y4yEOgVJbvl02fG+s28svtoyoaPIs2UqlKqkDCYIaRA6OcziUAjnTvOKo9T+jHspJq6YxeCfCszSdNMLdkrM/t6RzEejNjK5RcCOZvJIpjxCs9MnuYziXEGU2J39IGQZpz1h7QuvjLNFLsAD5v0MdFTjLlMAq0LsIUyk0WaSSgzxr+HMZjZZAhmzIK6XJVnd+vSZyLNdGhgPC4o7R72YyRWlj19kr2ZSQL92u2qJydOa0CNx4n+0SDe6RxMuq02NZtpJkImHOIOacSfmTIDaCMN7JiAs/HMpJPSGYhdBI0GYACYMzl6R7m3dzTjsQbiwqHv0VMsVFXVzWYyP33KaqY8pZmEMtNU7YE3Vq4rKppEFZq+TN52mqkIE7MFotliv4VnJjcGYPNg5vgp9QA0ZabPxC8jkMF+BsNY9eh/r7Z6HxQlGpTqj2vhl5naWCWnsdtFfyzaDWbcTgdOmzUJQOoSbW1qNpUZQiYc2rDJsLzjTCcFBACTasV8ptTBTCYXAtFkLZ2TtXEuk572eh+qPU6EI6rt7qJG9BdhO+m1fBIIR2QFjM/KM5Pn0myhzPjcDjQbho+Kv0Wd1yXL5OttdgEupjLTWJOozITCERwdEd/hXCgzhjRTTB09f2G0U/Weo6PoHw0kTbc15MwArKXPPC6HPA/oU02Z+mUAozJjPxA6YWo0sNuf4lgdZdM8QiYu2rDJUNqjDATiJJWqb8t4MCwvXq219i8EWpopfc+MmQFYUZSsK5r0gVU683XywXhAC1BSlWbnrQNwTJnxupy6ERfxwYz+b1FrMrE94T2DWhdnM+9Tvmky8cz0DAegqtFS4+YsAiwRzCQqM9GAYu7kWhkwvHlgIK7ZpBFZ7Ze1ZyY+SJvamOib0fwy6VUyAfGND9NRT4S/KFlbgUAogkBMIeU4A0ImIDW6i0qmaSbjnbgV4v09TocMUOyQyeRsefI3CWYAzTezM0PfjD6wGjFpSV9IxEne5VDgdqZIM+WrNDumzHhdDt3w0ViaaTyx50+djdSh+BtGe5QU/gJlVs0k1IvWWm9cM8Z0ERdco2emJ1bS3lrnxUmxxnVv7u83HWUgyHlpduz4l74ZXXm26P6bTo8Zgfj7K4p10G2G2DbZhHH9DKlqdgAmZOIRZwAeykw+l3fiKeYz6ZWfdNrAy8qXNE7Wg0mqmYDse83o12I2jK+QpBplABRunIHP7UwMZsbMgpnUpu5eXY+ZQg6ZFDRUJc5n6pLm38z9MoBemYkfQ6CfWn/y9EYAUROw2ZBJQX0ODMDjwbAMhsTxPyVWZq9XZjLp/isQSleNx5XW31Psq2TfXeGXiXY1L50QonRWQkiFIxpMDY2H0JNhmkkYgFN5ZsSJuiXN98/kZN1vcgHVI3rNZFqerb8Im83XKSSyYV6SKo5CjTOIU2ZG49NMDWZppiTBTH8Ry7Kjn2utzGTjlwE0A7Cqan8Tf0gLKFprfTgpFszolRmzfSFUq2yUGamaujTVVKaZYr1mVFXVuv9m4JlpjwVHzbXp/T1FKXqyNJNQR0vJ/AsApbUaQioYkWY60Dcmm3Kle7IRBuBU1UzdGVaB1Nk0iwpUVU1qAAZ0ykyGAyf1gZXZ5ONCYk+Zya9nJk6ZER6qWMpkQDQw9KWXZuot0igDgfjcfp0ycyRHyoz+bzUaCKHK45QpJpGGXTilHm6ngqMjAWw+OADA/PvcoBs2qad3JIAX3uvGihPaLTtDC7QgTVNNpxrSTD3DgYzLsgHguLY6/PCji+WoBLv4bKSZtIZ5pZNiAqjMEFIwRDAj7rgm1XjSlmnTVWbSVX7SNTgO+0MIxwIzK2VGeGYyHTipv3AUW5lJ1TAP0Jrm5W82k06ZqbVQZnQXYjvVTH1JUiuFQAQOA2NB2bBNf9HPBqdDkQGmuEhL5bLWA0VR4HM7sTBWoi28XabKjIVyeduf3sFVj2zClQ+/IY8HK8y6GosuwCLNJFJMmZRlCz51xgycHiu1totVTx49cmJ2CZl/AQYzhBQMEcwMiQGTGZykjdUrVmTS/RdIvymYvluq1R1plceZ1cDJkjIAl0SaKWYAdjs0ZSaJAbjWl7qaqS/NCeu5RvSZiaja7yAu+tnMZRKIC68xmNEH+8I3I9BXBAnMSrPDERVrt3UBAJ595wj+9U9bk67liIlfTigz3cN+BEKRjAZM5gIRpCdPM5XeKAOAwQwhBcM4IyVd1QTQVTOlMgBnqMyINNN4MGKryV2qFJMgGxOwXiUqdmm2lmayPnUKFcCf56Z5XpdmAO5LUppdZ6NCLVkFTyHwuBzSUyZUoq4cKTOA5gURyp6ZZ+2kaY1xrxGmZD31um7KQkF680DUNOyJNTC8/6U9eOCl3ZZrOWIyoqG5xgOPywFVjc4hk+bflvTLsrOhyoYyM1KCPWYABjOEFAzjwZ+uagJoyszAWDBpN9xMgxm9qc9OF2Azw6kZmgk4fd+M/iKcqiV/vrHjmanKczXTuK5pnghmjo5YG4DrdMqMVQpEKjNFCmYA3XymWGBl7MWSDVKZ8VsrMycZlZkkHYBVVftePr/tCADgHxa24YaVCwAAP/jjVqx9p8t0LWZpJkVR5ODQg/1j2HM087LsbJDKTLJgRqaZ6JkhZEJizDFnosw0VrmhKNGTaX8SX0umwYzL6ZABjZ35TNpcpuQXwayUGb0BuMgdgMdTTMwGtDRT/j0zmjIjgtuBmL9I3ytGH6BaeY76kjSKKxQirdM/GkQ4okr1JFsDMKD9vYSyJ48P3Q3F7JYaGfgB5gG616XNRxPfy+ff7QYAnDO/FV85ZzY+sXQ6Iirw9f/ZKM3EemSayZA+0zfOK1aayU7TPG0uE5UZQiYkCcpMBsGMy+mQJ1krE7Cqqhl7ZgD7s3wAzTNj1v1Xj1Bm0h04GYmocWpM0fvMCM9MEmXGK6uZcu+ZUVVVVjN53Q40Vnsg2oj0jQZN+8z43E54YkZzq1RTMUcZCIThtm80gKPDfkRUwKEgq+6/Aq37trUy43AoMtVU73PBZWHO10/O7h7y460D0YDl3ONaoSgK/vWji3DW3GaMBsL4yn9vSFDDrMaM6IMZMZcpk7LsbBDKTCiiImih/IqxEKU0ZBJgMENIwTCWMmYSzABIMH0aGRwPSb9LJp9Rn8Z8JtF+3q5nJt2Bk0P+EHR9zoofzBS5aV4gHJH7w+tywulQZOflvtGAaTADpA5QxXepWJ4ZQFtz32hQXvCba72WQUU6VLkNBuBhUc0Uf3ycND3aCThZibpWnh3E32OqzAlT66XS4nY68KtPn4o6rwsH+saw5VC8OtNt0T9HpJneOjCAYX8o47LsbNArjlYqqHi8lLr/AgxmCCkYXpcTbqfWjTPjYCZFRZO466zzuVL2vDAjnflMWpopeTCT6cBJ48W32IMm0/PM5F6Z8esCQZHuEBfe7iG/rJQzBjO1KcqzhcJWCsrMwGggZ2XZAv1cNMA6Dbv02Ggps+jIa4Z+pIFIMZ07vzVum4YqN86c0wwAeHFHj3w8Ojwzetwa02dCmXll11H570zLsjPF7VTgjI2OsArGxTFY7WaaiZAJiz7PnOmJOtVIAznTJsP3t9NkTSCatKUyAGc6cNKoDhW7mmk8YN8z48+DMuPXBUgidSTSMKICBkhM+0kTsEkwEwhFZCqvWH1m9J/dNxqUowxyUZYNaMedUBWsOnCfe1wr7rj4RNz20cWW76VXkF54LxrMnDd/csJ2Z89tAQC8pAtmxPBMl0NJMFuLYEZ41QrtlwGix2mq+UwizcSmeYRMYPRmzHSmWesxdn01YmZuTId6OZ8pDQOwjYtgJgMnjerQcLH7zARTe2aEYpIPA7B+lIHoHisUDWEarfY4E5ox1nmt+weJ6iGHEt85uNA06jwzZhU/2aCfzzTiD8kLtTHNpCgKVi+dLj1eZojj44X3utE/GkS9z5XQowYAzooFM6/v6ZN/t67Bcfm5xuGZonGeoNBl2YJU5dkjNgL6YsBghpACIu5m9HNZ0sXY9dVIppVMAnFRSdXLBrBvAAYyq2gydiIeLbpnJqqM2OkAHIqoScvnM0E/ykAgRmLsjk1ZNlPJkqWZenU9ZrKZTp0t+i7AOU8ziWomf1geHzUeZ0a9UsT+XRsryX7fca2mvp45rTWY0uBDIBTB63t6AegrmRJ/rykN8f6YYigzgL5xnvmxJoIcGoAJmcCIk2drbXrTrPWkGmnQneEQS4E40Yq7yGT0p5iYrSeTgZNCchcXumKXZo/ZuCvVPzeewuw8MBrET9a8iyM29jUQr8wIpDITSzOZBTN1SboA940Ud5SBQF/NpE3Mzk2aqUqXZsr2+BCBuzCyn3tcq+l2iqJIdUb4ZpIFaTVeV5zCWaxgRhtpYP7dHZHVTPTMEDJhEWmmTE+kABIapRnpNmmXng5tsdfZCWYGbRqAgcwGTgrfTnvsolbspnl2ZjPpA41UFU3/75U9+Nna9/CjP2+z9fn6smyB+D7si5XzmqWK6pP4oJJNiS4k4kLeNxLUVfzk3gCszWXKNA0bv3/PmW8ezACJvhmRPmu1ODb16kyx0kzasEkqM4QQC8QJIBfBTF+e0kztDSKY8afcVvgtUhmAgcwGTgrfjjBHFtsAbMczoyiKDGiSdVIFot1eAeClnT1Q1eQDCgGtYZ5PV+Uivg+BWErLLOUngmizNFNfkSdmC5p0HYCtGstlit4AnO3xof+uLzqmPulNw7K50YqmLYcG0TsSsOwxIzgm5ptRFGD6pOIEM1KZsQjEZWk2lRlCJi41OVBmZDVTKgNwhp8hKki6BpIrM8FwRJoB7RiA9QMn7fpmhGFVBFjBsJpWn5pcYyfNBOgqmlLMZ+oeiv4Nuwb92G1DsfIHrZUZQdI0k1kwMyKUmeKmmcR3aCQQlhf9thx0/wW0C7Remck8zaRdxM89LrGKSc/kOh/mt9VBVYGXd/ZoipPF7yWC9qkNhS/LFqQaxzHKpnmEkKkxGXl2Fp09Uw2b7Mmi+y+gBQ5D/lDSJnX6ycF1NqtgWmq19vt2EGmsKbo79GI2zrOTZtI/n6rXjPhbAcDLO4+m/PxkyozAPJgR1UxmykzMM1NkZabe54bwH4cjKhQl81SQEVnN5A9nfXzola/zFlinmARnz9NSTWYTs/WINNOsAnf+1aONfkilzDCYIWTC8qVzZuOXnzoFnz5jZsbvIS4648FIQhpD35Qr0zvPWq9LVn90JvHNyEomn0s22kpFqhOlEZEWaazxyNSN1XyhQmCnaR6glWen8szog5l1u+wEM6mVGbMqudokHYA1Zaa4wYzDocQFYpOqPQkl5pminzmUrTIjAo5JNZ6ESdtmnK0zAR8ZTK44rTihDYuPacAlp8/IaG25oCrJbLFIRC3ZNFNprYaQCqfe58aFJ07J6j1qPNFZO4FwBEdH/Jjm0XLrvSPRplwOJbturm0NPuzqHkHX4Lhlz42BNCqZBPKiYjOYEWmmep8LNV4X/KFAUSuaZDDjSX6RtTNsUlXVuGDmlZ1Hoapq0io3rZopXWUmSTWTmMtU5GAGiAZUQinKlV8G0CszoayrmWa11OAXn1qCaU3VtkYtnD5rElwOBft7x+RjVsrM7NZa/OHrZ2e0rlyRrM/MuC5tSmWGEJIViqJoJuCR+Dtt/Uwbu2qJGaJ6KFlF08CYffOvoErXvMwOMpipcsuTZzErmuwMmgQAr40000ggLJ/3OB04OhLAu13JvURanxnt1F3tccX92+zvUZ+0z0zxJ2YLGnRryFUlE6AvN85emQGAfzxxqmmjPDNqvC6cMqNJ/juaPit+4GhFsmBGfyORSp0sNAxmCClDrEYaZDMtW48wAXcOWFc0pdP9V1CdolW6EVHNVO9zy4qc0SJ1AY5EtInVKdNMrtRppqOxv1WV24kzZkdnAq3b2WO5vf79jOZQvaqSzDNjlmbqL4GJ2QJ9qiuXwYww3o8EQlINy5Ufxw7CNwNEPW+5GJ6ZL+Q4A5Pvrjj2qtzOojZYNKN09yghxBJpAh6JDzZycdcJ6Cqa7Hhm0lBmqlO0SjeiTzNVp6nq5Br9kMdU1Uzi+WTBjLyo1nlw5uxoCW8q34ysZnLFn7on1SYPZkQgOOwPJZSAl8LEbIE+MM7VXCZA+95F1GhFHKB1Ti4EonkeYN1jplQQ+2rcTJkJluZcJoDBDCFliVRmDGmmXAUz7Ta6AItgxk7DPEG1V+v3kQpVVWVapL7Krd1dFynNpPe/+FKUzYrnkwUzoiy7pdaLZbEJy6/s6kUkYt1vxmycARCvaJgFl8IzEwyrcUFZMByR+7jklJkclWUDiWbVxmp3QUufT5rWIAPKXCpO+SCZAXhEKDMl5pcBGMwQUpZMit3B5kuZEeXZyaqZskkzWc190TMaCCMcu7DX+9yo8YhUQXHSTGO6UQKpJHatmsnaM6NPdyw+JnqxGxgLYuvhQcvXmI0zADSlDjBXZmo8LghfsX7YpAhIFSU971O+0AfGubzoOx1K3D7LNg2bLi6nQ6pvJR/MeKxvOGT3X3fp1Q4xmCGkDJlUEz0hWiozWZ6sRSXJkSRdgEUwk4kB2I4yIy66bqcCn9uBatGSvljKTBrTgn0pGo8B8cGMy+nAacdGTaKvJEk1aaXZBmUmRTDjcCio9SQ2ztN3cM7GMJ4rGmv0ykxu0zH66ptsg/1MuHTZTDRVu7HihPaCf3Y6JFVmRMM8ppkIIblgUk30gnV0OE/KjM4zY5X2kMpMVfql2baCGZ35V1EUTZkpUjBjt2EeoAtmknQA1pq3RfffsjlRX0Wy5nmiaZ6VMuNxOSwrrepMKpqEX6YUyrKB+C7EuVYw9KmmYgQz75vXio23XIDlx7cV/LPTIZmvrVTnMgEMZggpS+a11QGIXvj0pcqiminbC0FrnReKAoQiquVAS3FXn4kB2M6MJaHMiIuwVpFS3DRTOsGM1eRhAOgRnpnY36oj5pt5bXcvQmHz141bGICFMmM2ZFKgVTRp+76vhMqygXjPTK4DDr1ptZCVTOVGsh5JpdowD2AwQ0hZcsasSZjTWoNhfwi/2XBAPp4rZcbtdKC5JrkJuD8Dz0w6aaah8fhqqZo0AqF8YLfHTHSbmGfGhjIjLqwLp9Sj3ufCsD+Etw8OmL5GjjMwrEEoMw0m3X8FtbJxnpaa7CuhsmxA+9621HpybtCtKrIyUy4k7zNTmnOZAAYzhJQliqLgsmXHAgAefHlPrM14SKo0uThZtzckD2YGMzEAp1GarU8zAZoyM1ykPjNa99/ce2aAqEk1VYm2lTJzyswmTGnwJfVjCIVr0CTNVApl2QAwb3Itrv2H4/Cvqxbl/L1r9J4ZKjOWJJuaXapzmYA8BjN79uzB5ZdfjlmzZqGqqgpz5szBd7/7XQQC8ZL1W2+9hfe9733w+XyYPn06/v3f/z3hvR5//HEsWLAAPp8PixcvxlNPPZWvZRNSNnzslGmo87qwq2cEL+zokWkLn9shy0CzQfhmzCqaVFWVlTDpGIDT8sxIZUakmYprAE7HMyO28SetZhKl2VogIVJN6yx8M1bKzOQ6H16+8QP45soFlp9nlmYqpYZ5QDRI/8b587ByUXYjP8wotgG4XJBN80wUUGkAnkhppm3btiESieA///M/sWXLFvz0pz/F3XffjZtvvlluMzg4iAsuuAAzZ87Ehg0bcMcdd+B73/se7rnnHrnNyy+/jEsuuQSXX345Nm7ciFWrVmHVqlXYvHlzvpZOSFlQ43Xh4qXTAETVme7haNAR9btkX5kiG+cNJAYzo4EwQjFjcHoG4DSqmWLKT53XHXut1sW1GGSSZrKazTQeDEsVrUV3YRUm4PV7+hAIJQZCsprJlXjqTvU3l43zYsFMMBzB39+Ndhwu9XLhXFBsA3C5oDV8jCSY/yekAXjlypW4//77ccEFF2D27Nm46KKL8C//8i944okn5DYPPfQQAoEA7rvvPpxwwgn45Cc/iW984xv4yU9+Ire56667sHLlSlx//fVYuHAhbr31Vpxyyin4xS9+ka+lE1I2fK7jWADAX7cfwet7+gDkTkLXugAnlmcLv4zH6YibC5QKLc2UOiDRGuZFL0K1smlecdNMdn7fVGkm4W3yuByo06lox7XVotbrwlgwjH29owmvk2mmDObi1BsmZ9//0m5s7xrCpBoPPn7KtLTfr9ygMmMPvfLoNwTU4tibUMqMGQMDA5g0aZL897p16/D+978fHo92Z7dixQps374dfX19cpvly5fHvc+KFSuwbt06y8/x+/0YHByM+yGkEpnVUoNz57dCVYH//NtOALk7USdLM8n+JNXutFQgaQAOhhPa6hvRRhkIZaa44wzSqWbypugA3KOboaXff4qiyLSd2RwlmWYyUWZSoS/NPtQ/hv949j0AwI0fXBDXp6ZSERdgp0OJq5oi8ei/38ZUk2h2OaGUGSM7duzAz3/+c3z5y1+Wj3V2dqKtLb7mXvy7s7Mz6TbieTNuv/12NDQ0yJ/p06fn6tcgpOQQRmBRZpurYKatwXo+k9ZjJr2SXnFBUdXk3XEBnQG4Kt4AXKxBk+NpNM3TS/VmmPllBGb9YAT+LJQZ/Xym7/9hC0YDYSyd2YSLJ4AqA2ieq0k1npJoEFiqOHTdko1p0ooyAN94441QFCXpz7Zt2+Jec/DgQaxcuRKrV6/GF7/4xZwt3oqbbroJAwMD8mf//v15/0xCisX757ViVkuN/HdrbW46p7Ylmc80kIH5F0h+12fEss9MkWcz2eozk2JqdrLJzXU+LegwYtU0zw7CAPzSzh48vaULToeCf/3oopKbfpwvRIDJSqbUWFUdjpZwmintFV133XW47LLLkm4ze/Zs+f+HDh3Ceeedh2XLlsUZewGgvb0dXV1dcY+Jf7e3tyfdRjxvhtfrhdfLLyyZGDgcCj7XMRPf/8NWALlPM/WNBjEeDMcZXzOZywRoM3L8oQhGA2E0J9lWGIBlabYuzaSqak5MzumgeWayL83uGUoWzCRJMwXNB03aQQRJogrt8rNnYUF7fdrvU66IDtL0y6Smyu1EH4KJykwJp5nSDmZaW1vR2tpqa9uDBw/ivPPOw6mnnor7778fDkf83URHRwe+9a1vIRgMwu2OHsBr1qzB/Pnz0dTUJLdZu3Ytrr76avm6NWvWoKOjI92lE1KxXHzqNPz46e0YCYRzVpnSUOWWgceRQT9mNFfL50RX4HS6/wpqvC74QwHLSh+BfmI2oE3cjqhRY2ImF/RsEN180+ozY1KRBOiUmbr00kzjWSgzomkeAExp8OGq8+el/R7lzHnzJ+N3Mw7iktNpO0iFz6LqUFNmSi+YyZtn5uDBgzj33HMxY8YM/PjHP0Z3dzc6OzvjvC6f+tSn4PF4cPnll2PLli149NFHcdddd+Haa6+V21x11VX4y1/+gjvvvBPbtm3D9773Paxfvx5XXnllvpZOSNlR53Pj9o+fiI+dcgzed1xLTt5TURTL6dlv7u8HAMydXJv2+2p9LJIHM8Y+M9W64MUsBZNvMukzY51mEp4Z6zTToCGYCUdUBMNR03QmgZx+1MF3P3y8TNtNFGY0V+OJr52Vlx42lYZV47xSHmeQtxWtWbMGO3bswI4dOzBtWrzBTFQxNDQ04JlnnsEVV1yBU089FS0tLbjlllvwpS99SW67bNkyPPzww/j2t7+Nm2++GfPmzcOTTz6JRYty3yGSkHLmopOm4qKTpub0PdvqfNh7dDTONxOJqHh1dy8AoGN2skSROXbmM6mqmtAB2OFQUO1xYjQQjt4hph9HZUV6s5k0A6VZSqw7iWemNtZXZ9gQzOj7zmSizMxvr8O586P+qlKf3EyKiwzGjcpMCU/Nzlswc9lll6X01gDAiSeeiBdeeCHpNqtXr8bq1atztDJCiF3MKpre6RzEwFgQNR4nFh/TkPZ72hlp4A9FEIgNW6zTpUeqPS6MBsJFKc+WTfNsSOyi2khVgUA4kjBnSKSZmpNWM8V7ZvQqTybBjNvpwAOfPz3t15GJR5VFp+6KqmYihEwc2mMVTZ26LsCv7IqqMqfNmgSXM/1TSJU08loHMyLF5FA04yYA1MbuCItR0ZSJMgOYl2cLA7BZZU29hWdGNDBzOZSM9jshdqky6WAdCEVk1+9qd+mlmXhEEEIskV2Ah7QuwGJu0JkZpJgALd+erAuwSDHV+dxxpcPaSIPC95pJxzPjcTogMkt+g+8gEIpIP0zSaiZ/vDKTTVk2IemgHaPad1efFrZjgi80PCoIIZYY5zOFIype2x0NZjLxywC6LsA2lBlh/hUUc9ikNjU79WlTURQZ9BhNlEdHooGhy6GY9ukxzlASZDPKgJB08Jl8d8Xx6nYq8JRgQF16KyKElAzGaqZ3Dg9icDyEWq8LJ0zNrEdJjZ1gxjBkUr7Wa91QLt+kM2hSv50xzSSmmzfXekwb1lmVZmczyoCQdDCrOBwt4YnZAIMZQkgS9POZVFXFK7uiqszpGfplAHMJ28igYcikoMbCmFgI0vHMANZdgJN1/wW0NJOxNJvKDCkU1Z7E1gKlbP4FGMwQQpIguqUGQhEMjAVlMHPm7EnJXpYUO2mmIcOQSUExh01Kz4zNk7lVF+BkZdmAdTUTPTOkUFSZtE9gMEMIKVt8bieaYiMLDvaPyf4ymZp/Aa35nZjAa4ZxyKSgWPOZguGIbFhnW5mx8MykVmaiv6M/FInrLZPNkElC0kHze2nfP6aZCCFljTAB/3XbEQyNh1DndeGEqen3lxGkYwDW95gBNAPwSIEnZ+vVFfueGZFmMvfMmI0yADQDMBDvDcpmlAEh6VBl0gtqNI2p8cWARwUhJCkimHly0yEAUb+MM4tJy7K8OklAYhwyaXxtqonbuUaoK4piP5gQQY9IDwmEMmM1vdnldEgpX59qymbIJCHpoI0z0KWZYsdrDYMZQkg5IkzAO44MA8guxQRo6kqyNJNxyKSg1ps6EMoH42LIpNtpe1q3lWcmVZoJMK9oEk3zqMyQfCNTpKxmIoRUCmKkgaBjTnbBjJ1Bk7LPjCHNVCwDcLqVTPptjVVbdoIZEbTpgxkRFDGYIfmm2iQVPBqkAZgQUsYIZQaIBhcLp2TWX0ZgqzRbpJksDMCjBVZmRDCTTorHG/PMjCYoM8k9M4CuC7A+zRRimokUBrOp7+KYYzBDCClL2uo1BeH0Wc1Z+WUAuwZgMc7AaAAuTtO8sQzMj8c21wDQxj8AQCgcQd9oLJhJM81EZYYUCp9p0zxxDDDNRAgpQ9p0ykw2/WUEZhK2ESsDcI1J/4tCkM5cJsFHTp4KAHhxRw8OD4wBAHpHAlDV6ADNpmprZUb83vqgjcoMKRSaATjRM0MDMCGkLGlv0Acz2fllAN2JMklAIhQJ4+yiYg2azMQzM7O5BqfPmgRVBX678SAArWHepBpvUoVL88zo00xUZkhhqErSAZil2YSQsqS5xoMLT5yCC45vw/FZ+mUAXZopGIaqqgnPB0IRGTwYlZnaIjXNk3OZ0jyRX3zKNADA/244AFVVNb9MrbUqA1ilmUQ1U2leTEjlUO2Ofv+CYRXBcPR7J5UZL9NMhJAyRFEU/PJTp+Cezy01HYyYLmK+kqomNpQD4tWIWmM1k1dLUUUiiYFQvtCUmfROmR86cQqq3E7s6h7Bxv396BmK9Zips/bLAObzmeSgyTTXQEi6+HST4cV3n+MMCCFEhz5VY+Z9ERfwWq8rIRVTozMfGscE5JNMPDNA9Hf44KJ2AFF1xk5ZNqApM2aeGaaZSL7xOB3y2BOqpBbMUJkhhBA4HIpUF8xMwJr5N/Gk6XM7IOKbQqaaMqlmEnz81Giq6Q9vHsLB/qgROFWaqdZk2KQ/g/JwQjJBUZSEPkla07zS/P4xmCGEFBzZa8ZEXbHq/gtET7I1RTABj4cyDyQ6ZjdjaoMPQ+Mh/C42EiKVMlOfrAMw00ykABhbKNAATAghBpJ1Ada6/yYGM0BxJmeP6cYZpIvDoUh1ZiCmOqVOMyU2zdP6zJTmxYRUFlWGqe/iWK1hmokQQqJUJ+kXI9JMxoZ58rVycnYBg5kMPTOCj8eqmgQtKQ3AMc+MiTJDAzApBEwzEUJICmQwYzKWQCozJmkmQLszTNZ0L9dIA3CGJ/JjW2qwdGaT/HdzTQrPjMlsJj9Ls0kBqdI1zgtHVFl5yGCGEEJiCM+McW4RoPPMWCgzYuq2cdjkyzt78PLOnlwuUyL7zGRhvr34VE2dsVuaPRwIyRL0cTbNIwVEn2bSe9tYzUQIITGSdQG2GjIpkAZgXZppYDSIy+5/HZ+///W8zG3KNs0EABeeOAWtdV4c01iVUpkRaSZVjQY0gKbMsJqJFAL9MSpSTIpSumnO0gyxCCEVTbJhk1ZDJgXV0gCsvXbzoQEEYp6S3d0jWDytIafrHcsyzQRE1ZY/X/U+OBUFLmfyC4LP7YTH6UAgHMHweAj1PjfHGZCC4vNonhk5MdvthKJk3zgzH/CoIIQUnGTDJjUDsLkyU+tNNA+/fXBA/v/O7uGcrVOQadM8Iy21XjSlUGUEtYby7HEqM6SAVIuKw2C45CdmAwxmCCFFQPaZMQlmjo5E5xdNsrjoi9cO65QZfTCzKw/BTC48M+lSp2ucp6oqlRlSUOSwyUBYN5epdANpHhWEkIKTLM10dES0/DcPZmpMyrrfPqBTZnpGcrZOwZjsvlu4U6YMZvwhBMMqxCgqVjORQqA/RqUyU8KqIIMZQkjBkRK2iQG4NzZZurnGvOKnxuCZGRgNYl/vqHx+55E8ppkKWJZa5xWN80JSlQHYAZgUBn01k2yYV6ITswEGM4SQIiBMvEZlZiwQlmMKJlkoM9WGDsCbD0VVGaGa7O4ZyflE7bEi3Jnq5zOJhnkA00ykMMQHM6XdMA9gMEMIKQJWBmCRYvI4HaizuAsUaSbRZ0b4Zc45rhUepwP+UEQOdMwFqqrmpDQ7Xep0BmChDHlcjpKtJiGVRbW+molpJkIISUSeKIPxaabemPm3udZjedGuMag6Ipg5aXojZjZXAwB25dA3EwhHpF/FV8A7UzGbang8pI0yoCpDCoTPRJlhmokQQnRYDZo8Opy8kglIbJq3ORbMLD6mAXNaawHktqJpPKCleIqjzAS1UQYlfGdMKotq3diQUp+YDTCYIYQUAavS7J7haJqpOclU6WrdOIOBsSD2Ho2afxdNbcDs1hoAue01I1JMLocCd4pmd7lEP59JjDIo1e6rpPKo8kS/a+N6AzCDGUII0bAqzRZpppYkyoy4yI/6w9gSU2WmNVWhqcaD2VKZyV2aqRh+GUBrGjg4HuKQSVJwqtx6ZSaqgrJpHiGE6Kgx6eILpG6YB2h+m2F/SPplFh8THV8wJ6bM5DSYEQ3zCnxXKtJMw/4gh0ySglNlYgBmNRMhhOiodpuXZttJMwnPjD8Uwab9/QAgZzEJZaZzcDxnAyeLp8xoaSYOmSSFRgQu40FtNhPTTIQQokPe9QXDUFWtJ4wwADdb9JgB4isqXtvdC0BTZhqq3LJz8O4cqTPjRej+CxiCGSozpMDoTfqjsmkk00yEECIRd32qqg1QBHSl2UnSTB6XA25ntGxbpKUWTdWmZEvfTE9uTMC5GjKZLsIzo69mojJDCoW+NFtUDlKZIYQQHfrAQO+bOWojzQRo1VCAZv4VCN9MrsYaiB4vhTbfap6ZED0zpODo/TF9sZsGlmYTQogOh0ORaRvhm1FVFT02lBlAq2gCtBSTYHZLVJnJ1cBJmeIpeJopqswEwyoGx4LRNTCYIQVCrwIKLxub5hFCiAHZayaWxhkJhBGIqSDJPDPR12on2kWGYGbO5BwrM7IsurCny2q3E6IJck/MS8Q0EykUTociv/OD47HS7BL+/jGYIYQUBRGQiHy8SDFVuZ1xaSTT1+ruEE+cZq7M7Dmam4GTMs1U4BO5w6FIBap7KLpvqMyQQmJMK7E0mxBCDOgH2QGa+pBKlQGAWq9OmZkaH8xMa6qC26lgPBjBoYHsB04KA3AxAgkxn6k7FuhxnAEpJNWG7xvTTIQQYqDKE99rxk4lk0AoN0bzLwC4nA4c2yzGGmTvmymWARjQTMDCs8BBk6SQGBtF0gBMCCEGxF2f6GFht5IJ0EpEjeZfwWzZCTi1b6Zn2B/X68ZIMXu8iDRTzxCVGVJ4jB4Zo1JTSuTt6NyzZw8uv/xyzJo1C1VVVZgzZw6++93vIhAIxG2jKErCzyuvvBL3Xo8//jgWLFgAn8+HxYsX46mnnsrXsgkhBUJLM8U8M2koM9OaqgEAZ8yaZPq83RlNG/b2Yum/PosfPvWO5TbaxOrCBzNCmREGTHpmSCHRe2Q8LgdcBRy0mi55S4Bt27YNkUgE//mf/4m5c+di8+bN+OIXv4iRkRH8+Mc/jtv22WefxQknnCD/3dzcLP//5ZdfxiWXXILbb78d//iP/4iHH34Yq1atwhtvvIFFixbla/mEkDxjHDapdf9Nrcxccd5cnDZrEjpmN5s+PycWzKSanr310CAAYFvnkOU2xU0zueP+TWWGFBJ99Vwpm3+BPAYzK1euxMqVK+W/Z8+eje3bt+PXv/51QjDT3NyM9vZ20/e56667sHLlSlx//fUAgFtvvRVr1qzBL37xC9x99935Wj4hJM9UG4OZkViayYYyU+Vx4pzjWi2fn21z4ORwbOaMMPmaIdJMhR5nAGjKjIDKDCkk+gCmpoRHGQAF9swMDAxg0qREWfiiiy7C5MmTcfbZZ+P3v/993HPr1q3D8uXL4x5bsWIF1q1bZ/k5fr8fg4ODcT+EkNKiWhqARWm2/WqmVMxpsTdwUpSFC/XFjPFg8ZSZWkMwwz4zpJDoPTOlbP4FChjM7NixAz//+c/x5S9/WT5WW1uLO++8E48//jj+9Kc/4eyzz8aqVaviAprOzk60tbXFvVdbWxs6OzstP+v2229HQ0OD/Jk+fXrufyFCSFYYlRk7E7Pt0lBtb+CkCHTsKDPFLM0WUJkhhUQ/WLKU5zIBGQQzN954o6lpV/+zbdu2uNccPHgQK1euxOrVq/HFL35RPt7S0oJrr70WZ5xxBk477TT86Ec/wmc+8xnccccdWf1SN910EwYGBuTP/v37s3o/QkjuMfaZSac02w6ieV6ygZMjMpixVmY0zwzTTGRiUU7KTNpJsOuuuw6XXXZZ0m1mz54t///QoUM477zzsGzZMtxzzz0p3/+MM87AmjVr5L/b29vR1dUVt01XV5elxwYAvF4vvN7s7+4IIflD32cmElG1YCYHaSYg6pt5bU9v0rEGIwGRZkqizASL0wEYSAxmmGYihaScPDNpr661tRWtrdbGOz0HDx7Eeeedh1NPPRX3338/HI7UdxWbNm3ClClT5L87Ojqwdu1aXH311fKxNWvWoKOjI92lE0JKCH2aaXA8iFBs9MCkHCkz0ydFy7cPD4xbbqMZgJMpM8XsM8M0EykeejWm4pQZuxw8eBDnnnsuZs6ciR//+Mfo7u6WzwlV5cEHH4TH48GSJUsAAE888QTuu+8+3HvvvXLbq666Cueccw7uvPNOXHjhhXjkkUewfv16WyoPIaR0kWmmYEj2mKnzunJmtBWqhlBfzBix5ZmJBjrFUEWozJBiwtJsRNWTHTt2YMeOHZg2bVrcc/pum7feeiv27t0Ll8uFBQsW4NFHH8XFF18sn1+2bBkefvhhfPvb38bNN9+MefPm4cknn2SPGULKHJGPHw2Ec1rJJBDdc4fG7VUzqaoKRYyp1lHM2Uz0zJBiog9gUg1/LTZ5W91ll12W0ltz6aWX4tJLL035XqtXr8bq1atztDJCSCkghtaN+sNpjTKwiwhmkpZm61QbfyhiqnwU0wCcUM1EZYYUkKoyUmYY5hNCioLsAKxLM+XKLwPogpmkyoyWXvJb+GZKYdCkgIMmSSHR+2RKeWI2wGCGEFIk9KXZIs3Ukss0ky+1MqN/btyioskv0kxF6ABsvIBQmSGFJK40u8S/ewxmCCFFodqtlWZrowzykGayUGaC4QgCus6/VibgYhqA3U5H3EWEnhlSSOI9MwxmCCEkgSpZzRSW3X9zmmYSykwgFFd0IBgxKDZmIw1UVS2qZwbQUk1OhwJ3CU8tJpVHXDUT00yEEJKIuNNTVeBgf7QXTC6rmepiPVpUVRuZoMeYfjJTZvQBTrGDGaoypNDEKTNMMxFCSCL69MmB3lEAQEsOq5l8bgecjmiptZlvRm/+Bcwb58UHM8U5mdfGKprYY4YUGr0BuNpb2t8/BjOEkKLgcCgyoMlHNZOiKEl7zRgDHLORBuIxRQHczsQeNIWgnsoMKRLxpdlMMxFCiClGU2Eu00xA8l4zRs+MqTIT1PwyZg31CgHTTKRYxJVm0wBMCCHmGOe9TKrObTAjAgGziqbEYMbaM1PMFI/w/jDNRAqNx+mQimSp95kp7dURQioavTLTWO2GK8fVOpoyE0x4zp4BuHijDAS1VGZIkVAUBdddMB9dg+OY0uAr9nKSwmCGEFI0qnR5+OYc+mUEIhAw88zYKc0eDxav+69AppmozJAi8JVz5hR7CbZgqE8IKRr6cs9czmUS1CTzzASM1UylqczUxaqZqMwQYg2PDkJI0ajRlXvmcpSBoC5JF+DEaibr0uxijDIQTG+qAoCSl/kJKSZMMxFCioY+zZTLsmxBetVMJspMCaSZzl/Yhgf/+XScPK2xaGsgpNRhMEMIKRpxaaYczmUSJBs2KR5TlGiXYHNlJhrg+IqozDgdCs45rrVon09IOcA0EyGkaOhLs/ORZrKjzIhy8GSl2cVUZgghqWEwQwgpGvrS7El5UGaS9ZkR85pEess8zVR8AzAhJDU8QgkhRUMfzOS6+y8A1MYazg0lSTOJz01qAGYwQ0hJwyOUEFI09AbgvKSZbHQAFiXhTDMRUr4wmCGEFI2aPKeZkntmosFLs0wzmc1miqWZimgAJoSkhkcoIaRoCAOwQwEaq9w5f/86G9VMooqqVGczEUJSw2CGEFI0qmNppkk1XjgcuZ9KXWvRNE9VVa2aiZ4ZQsoeHqGEkKLRWhdVRabFutzmGjHOIBCOyJ4xQDRICUVUAEBLkmqmcVYzEVIWsGkeIaRonDStAT9efRJOnNaQl/cXygwQVWe8tdF0kb77b1ONHWWGaSZCShkGM4SQoqEoCi4+dVre3t/pUFDtcWI0EMaIP4zm2ujjwvxb5XbK8vCkgyZpACakpOERSgipaIQ6M+QPyseE+bfG65LmXlNlJkjPDCHlAI9QQkhFY9ZrZiQQ/f9arxM+VzJlhtVMhJQDDGYIIRVNnUmvmXhlJnoaHA+Goapq3GtlmonKDCElDY9QQkhFYzY5e0QXzAhzb0QFguH4YGY8SAMwIeUAgxlCSEUjPTPjicFMrdcVZ+7Vl2/r/01lhpDShkcoIaSiEcMm49NM0SAlqsxop0HjSANZms1qJkJKGh6hhJCKps7MAOzXDMCKosiAxmgC9jPNREhZwGCGEFLRmA2blJ6Z2DgFq/JskWbyUZkhpKThEUoIqWhqzDwzgVDcc/qKJj3sAExIecBghhBS0YhqppE4ZSYatNR6jcqMFsyoqsrZTISUCTxCCSEVTao+MwB0nhktzRSKqIjNoqQyQ0iJw2CGEFLRaOMMzPrMRIMUM2VG759hNRMhpQ2PUEJIRaONM9BmMyUYgOVIAy2A8ev8M0wzEVLa8AglhFQ0ZtVMCWkmEwOwUGY8LgcURSnIWgkhmcFghhBS0Zj3mYk3AHtNlBmafwkpH3iUEkIqGhGwjATCCMccvYmemeip0MwzQ/MvIaUPgxlCSEUjPDNAtL+Mqqqyz4yxNDvOMyODGZ4mCSl1eJQSQioar8sJjzN6qhseD2EsGJYl14ml2TplRqSZWMlESMnDo5QQUvHIiiZ/SJp/FQWo9hhLsxOVGR/TTISUPAxmCCEVj76iSZh/azwuWaVkNs6AE7MJKR9cqTchhJDyRqSThsdDMuUkzL+Apr7oDcCsZiKkfGAwQwipePQjDTwuEcxopz+tz4yZAZhpJkJKnbzeclx00UWYMWMGfD4fpkyZgs9+9rM4dOhQ3DZvvfUW3ve+98Hn82H69On493//94T3efzxx7FgwQL4fD4sXrwYTz31VD6XTQipMGp1vWZEWXatLpgxH2dAZYaQciGvR+l5552Hxx57DNu3b8dvfvMb7Ny5ExdffLF8fnBwEBdccAFmzpyJDRs24I477sD3vvc93HPPPXKbl19+GZdccgkuv/xybNy4EatWrcKqVauwefPmfC6dEFJB6OczDRtGGQBW4wyEZ4bKDCGlTl7TTNdcc438/5kzZ+LGG2/EqlWrEAwG4Xa78dBDDyEQCOC+++6Dx+PBCSecgE2bNuEnP/kJvvSlLwEA7rrrLqxcuRLXX389AODWW2/FmjVr8Itf/AJ33313PpdPCKkQ9MpMVSw4MU8zJRqAfVRmCCl5CnaU9vb24qGHHsKyZcvgdrsBAOvWrcP73/9+eDweud2KFSuwfft29PX1yW2WL18e914rVqzAunXrLD/L7/djcHAw7ocQMnHRPDNBjMqGeZrioo0zMEkzsZqJkJIn70fpDTfcgJqaGjQ3N2Pfvn343e9+J5/r7OxEW1tb3Pbi352dnUm3Ec+bcfvtt6OhoUH+TJ8+PVe/DiGkDNGXZos0U3WcZ0aMM9DPZqIBmJByIe1g5sYbb4SiKEl/tm3bJre//vrrsXHjRjzzzDNwOp343Oc+B1VVc/pLGLnpppswMDAgf/bv35/XzyOElDYizTSUwgBsqswwzURIyZO2Z+a6667DZZddlnSb2bNny/9vaWlBS0sLjjvuOCxcuBDTp0/HK6+8go6ODrS3t6OrqyvuteLf7e3t8r9m24jnzfB6vfB6ven8WoSQCiZemdGa5gm0cQYszSakHEk7mGltbUVra2tGHxaJRE8Ofr8fANDR0YFvfetb0hAMAGvWrMH8+fPR1NQkt1m7di2uvvpq+T5r1qxBR0dHRmsghEw86mLKzIg/lDAxG7AYZxBkB2BCyoW8HaWvvvoqfvGLX2DTpk3Yu3cvnnvuOVxyySWYM2eODEQ+9alPwePx4PLLL8eWLVvw6KOP4q677sK1114r3+eqq67CX/7yF9x5553Ytm0bvve972H9+vW48sor87V0QkiFUeuN3iylSjP5TdJMrGYipPTJ21FaXV2NJ554Aueffz7mz5+Pyy+/HCeeeCL+9re/yRRQQ0MDnnnmGezevRunnnoqrrvuOtxyyy2yLBsAli1bhocffhj33HMPTjrpJPzv//4vnnzySSxatChfSyeEVBhChdEbgGtMDMDjIbPZTEwzEVLq5K3PzOLFi/Hcc8+l3O7EE0/ECy+8kHSb1atXY/Xq1blaGiFkglGnm5o9EkhUZoQvJhhWEY6ocDoUzmYipIzgUUoIqXhEmik6ziBmADZRZgAtvUQDMCHlA4MZQkjFI0qzQxEVPcPRAoQak6Z5gFbRpAUzPE0SUurwKCWEVDzVbicUJfr/Q+OJaSanQ4HbGd1ApJeEGZjVTISUPjxKCSEVj8OhoNYTbxHUp5kA/bDJaBATELOZaAAmpORhMEMImRCIVJP8tyGY8Rp6zTDNREj5wKOUEDIhMKaVjEGKzzA5W6tmojJDSKnDYIYQMiHQKzM1HicUYaKJYRxpQGWGkPKBRykhZEKgV2aMKSZAP9IgHPdfGoAJKX14lBJCJgR1OmWmOkkwMx6MIBxREQyr0ceZZiKk5GEwQwiZEOinZBsrmQAtneQPhWUlE0BlhpBygEcpIWRCoPfM1HoT1RZNmQnLFBMAeJw8TRJS6vAoJYRMCOq8egOwWZpJKDMRaQJ2ORS4GMwQUvLwKCWETAjilRmTYMaVqMywkomQ8oBHKiFkQiCGTQIWnhm3Vpoty7LZ/ZeQsoDBDCFkQhDXZ8bUAKxTZmJpJh+VGULKAh6phJAJQZ3XngHYH4roesxQmSGkHGAwQwiZEKRSZvTjDNj9l5DygkcqIWRCoDf9Jk8zRXRzmXiKJKQc4JFKCJkQpB5noDXN05QZppkIKQcYzBBCJgR1KdNMmjLDuUyElBc8UgkhE4KaFAZg/TgDUc1EZYaQ8oDBDCFkQuB2OmTAklyZ0aWZqMwQUhbwSCWETBjef1wrpjVVYeakmoTn9OMM2AGYkPIi8faEEEIqlHs+eyoiKuB0KAnP6ccZjDPNREhZwWCGEDJhUBQFzsQ4BoBxnAGVGULKCR6phBAC83EG9MwQUh7wSCWEEBjHGYjZTEwzEVIOMJghhBAYxxmwzwwh5QSPVEIIgZZm8ociNAATUmYwmCGEEGjKDAAMjgcB0ABMSLnAI5UQQqB5ZgBgYIzBDCHlBI9UQggB4HIoEO1nZDDjZpqJkHKAwQwhhCDag0aoM4OxYMZHZYaQsoBHKiGExBDBDJUZQsoLBjOEEBJDKDHBsAqAnhlCygUeqYQQEsOoxDCYIaQ84JFKCCExjMEL+8wQUh4wmCGEkBg+gzLjYwdgQsoCHqmEEBIjQZmhAZiQsoDBDCGExDAqM/TMEFIe8EglhJAYxrQSgxlCygMeqYQQEiNRmWGaiZBygMEMIYTE0CsxigK4nUoRV0MIsQuDGUIIiaFXZnwuJxSFwQwh5QCDGUIIiaEPZrwsyyakbODRSgghMfSDJWn+JaR84NFKCCEx9H1laP4lpHzIazBz0UUXYcaMGfD5fJgyZQo++9nP4tChQ/L5PXv2QFGUhJ9XXnkl7n0ef/xxLFiwAD6fD4sXL8ZTTz2Vz2UTQiYoXiozhJQleT1azzvvPDz22GPYvn07fvOb32Dnzp24+OKLE7Z79tlncfjwYflz6qmnyudefvllXHLJJbj88suxceNGrFq1CqtWrcLmzZvzuXRCyASEnhlCyhNFVVW1UB/2+9//HqtWrYLf74fb7caePXswa9YsbNy4ESeffLLpaz7xiU9gZGQEf/zjH+VjZ555Jk4++WTcfffdtj53cHAQDQ0NGBgYQH19fS5+FUJIBfL4+v24/n/fAgAsndmE//3qsiKviJCJjd3rd8FuPXp7e/HQQw9h2bJlcLvdcc9ddNFFmDx5Ms4++2z8/ve/j3tu3bp1WL58edxjK1aswLp16yw/y+/3Y3BwMO6HEEJSQWWGkPIk70frDTfcgJqaGjQ3N2Pfvn343e9+J5+rra3FnXfeiccffxx/+tOfcPbZZ2PVqlVxAU1nZyfa2tri3rOtrQ2dnZ2Wn3n77bejoaFB/kyfPj33vxghpOLw0QBMSFmSdjBz4403mpp29T/btm2T219//fXYuHEjnnnmGTidTnzuc5+DyGy1tLTg2muvxRlnnIHTTjsNP/rRj/CZz3wGd9xxR1a/1E033YSBgQH5s3///qzejxAyMdDPZqIBmJDywZXuC6677jpcdtllSbeZPXu2/P+Wlha0tLTguOOOw8KFCzF9+nS88sor6OjoMH3tGWecgTVr1sh/t7e3o6urK26brq4utLe3W36+1+uF1+u18dsQQoiGXo1hMENI+ZB2MNPa2orW1taMPiwSiQCIelqs2LRpE6ZMmSL/3dHRgbVr1+Lqq6+Wj61Zs8YyGCKEkEyJV2aYZiKkXEg7mLHLq6++itdffx1nn302mpqasHPnTnznO9/BnDlzZCDy4IMPwuPxYMmSJQCAJ554Avfddx/uvfde+T5XXXUVzjnnHNx555248MIL8cgjj2D9+vW455578rV0QsgEJW42Ew3AhJQNeQtmqqur8cQTT+C73/0uRkZGMGXKFKxcuRLf/va341JAt956K/bu3QuXy4UFCxbg0UcfjetFs2zZMjz88MP49re/jZtvvhnz5s3Dk08+iUWLFuVr6YSQCYpPn2ZyU5khpFwoaJ+ZYsE+M4QQO3QNjuOMH64FAHz9A3Nx3QXzi7wiQiY2JddnhhBCSh0fDcCElCU8WgkhJIaXBmBCyhIGM4QQEiNu0CQNwISUDTxaCSEkhqIoMqDxUZkhpGxgMEMIITpEeTaVGULKBx6thBCiQ/SXoQGYkPKBRyshhOgQxl8agAkpHxjMEEKIjuZaDwCgqcZT5JUQQuyStw7AhBBSjvzwo4vx9oEBnDStodhLIYTYhMEMIYToWDilHgunsFM4IeUE00yEEEIIKWsYzBBCCCGkrGEwQwghhJCyhsEMIYQQQsoaBjOEEEIIKWsYzBBCCCGkrGEwQwghhJCyhsEMIYQQQsoaBjOEEEIIKWsYzBBCCCGkrGEwQwghhJCyhsEMIYQQQsoaBjOEEEIIKWsmxNRsVVUBAIODg0VeCSGEEELsIq7b4jpuxYQIZoaGhgAA06dPL/JKCCGEEJIuQ0NDaGhosHxeUVOFOxVAJBLBoUOHUFdXB0VRcva+g4ODmD59Ovbv34/6+vqcvS9JhPu6cHBfFxbu78LBfV04crWvVVXF0NAQpk6dCofD2hkzIZQZh8OBadOm5e396+vreWAUCO7rwsF9XVi4vwsH93XhyMW+TqbICGgAJoQQQkhZw2CGEEIIIWUNg5ks8Hq9+O53vwuv11vspVQ83NeFg/u6sHB/Fw7u68JR6H09IQzAhBBCCKlcqMwQQgghpKxhMEMIIYSQsobBDCGEEELKGgYzhBBCCClrGMxkwS9/+Usce+yx8Pl8OOOMM/Daa68Ve0llz+23347TTjsNdXV1mDx5MlatWoXt27fHbTM+Po4rrrgCzc3NqK2txcc//nF0dXUVacWVwY9+9CMoioKrr75aPsb9nFsOHjyIz3zmM2hubkZVVRUWL16M9evXy+dVVcUtt9yCKVOmoKqqCsuXL8d7771XxBWXJ+FwGN/5zncwa9YsVFVVYc6cObj11lvjZvtwX2fG3//+d3z4wx/G1KlToSgKnnzyybjn7ezX3t5efPrTn0Z9fT0aGxtx+eWXY3h4OPvFqSQjHnnkEdXj8aj33XefumXLFvWLX/yi2tjYqHZ1dRV7aWXNihUr1Pvvv1/dvHmzumnTJvVDH/qQOmPGDHV4eFhu85WvfEWdPn26unbtWnX9+vXqmWeeqS5btqyIqy5vXnvtNfXYY49VTzzxRPWqq66Sj3M/547e3l515syZ6mWXXaa++uqr6q5du9Snn35a3bFjh9zmRz/6kdrQ0KA++eST6ptvvqledNFF6qxZs9SxsbEirrz8uO2229Tm5mb1j3/8o7p792718ccfV2tra9W77rpLbsN9nRlPPfWU+q1vfUt94oknVADqb3/727jn7ezXlStXqieddJL6yiuvqC+88II6d+5c9ZJLLsl6bQxmMuT0009Xr7jiCvnvcDisTp06Vb399tuLuKrK48iRIyoA9W9/+5uqqqra39+vut1u9fHHH5fbvPPOOyoAdd26dcVaZtkyNDSkzps3T12zZo16zjnnyGCG+zm33HDDDerZZ59t+XwkElHb29vVO+64Qz7W39+ver1e9X/+538KscSK4cILL1T/+Z//Oe6xj33sY+qnP/1pVVW5r3OFMZixs1+3bt2qAlBff/11uc2f//xnVVEU9eDBg1mth2mmDAgEAtiwYQOWL18uH3M4HFi+fDnWrVtXxJVVHgMDAwCASZMmAQA2bNiAYDAYt+8XLFiAGTNmcN9nwBVXXIELL7wwbn8C3M+55ve//z2WLl2K1atXY/LkyViyZAn+z//5P/L53bt3o7OzM25/NzQ04IwzzuD+TpNly5Zh7dq1ePfddwEAb775Jl588UV88IMfBMB9nS/s7Nd169ahsbERS5culdssX74cDocDr776alafPyEGTeaanp4ehMNhtLW1xT3e1taGbdu2FWlVlUckEsHVV1+Ns846C4sWLQIAdHZ2wuPxoLGxMW7btrY2dHZ2FmGV5csjjzyCN954A6+//nrCc9zPuWXXrl349a9/jWuvvRY333wzXn/9dXzjG9+Ax+PBpZdeKvep2TmF+zs9brzxRgwODmLBggVwOp0Ih8O47bbb8OlPfxoAuK/zhJ392tnZicmTJ8c973K5MGnSpKz3PYMZUrJcccUV2Lx5M1588cViL6Xi2L9/P6666iqsWbMGPp+v2MupeCKRCJYuXYof/vCHAIAlS5Zg8+bNuPvuu3HppZcWeXWVxWOPPYaHHnoIDz/8ME444QRs2rQJV199NaZOncp9XcEwzZQBLS0tcDqdCZUdXV1daG9vL9KqKosrr7wSf/zjH/HXv/4V06ZNk4+3t7cjEAigv78/bnvu+/TYsGEDjhw5glNOOQUulwsulwt/+9vf8LOf/QwulwttbW3czzlkypQpOP744+MeW7hwIfbt2wcAcp/ynJI9119/PW688UZ88pOfxOLFi/HZz34W11xzDW6//XYA3Nf5ws5+bW9vx5EjR+KeD4VC6O3tzXrfM5jJAI/Hg1NPPRVr166Vj0UiEaxduxYdHR1FXFn5o6oqrrzySvz2t7/Fc889h1mzZsU9f+qpp8Ltdsft++3bt2Pfvn3c92lw/vnn4+2338amTZvkz9KlS/HpT39a/j/3c+4466yzEloMvPvuu5g5cyYAYNasWWhvb4/b34ODg3j11Ve5v9NkdHQUDkf8pc3pdCISiQDgvs4XdvZrR0cH+vv7sWHDBrnNc889h0gkgjPOOCO7BWRlH57APPLII6rX61UfeOABdevWreqXvvQltbGxUe3s7Cz20sqar371q2pDQ4P6/PPPq4cPH5Y/o6OjcpuvfOUr6owZM9TnnntOXb9+vdrR0aF2dHQUcdWVgb6aSVW5n3PJa6+9prpcLvW2225T33vvPfWhhx5Sq6ur1f/+7/+W2/zoRz9SGxsb1d/97nfqW2+9pX7kIx9huXAGXHrppeoxxxwjS7OfeOIJtaWlRf3mN78pt+G+zoyhoSF148aN6saNG1UA6k9+8hN148aN6t69e1VVtbdfV65cqS5ZskR99dVX1RdffFGdN28eS7OLzc9//nN1xowZqsfjUU8//XT1lVdeKfaSyh4Apj/333+/3GZsbEz92te+pjY1NanV1dXqRz/6UfXw4cPFW3SFYAxmuJ9zyx/+8Ad10aJFqtfrVRcsWKDec889cc9HIhH1O9/5jtrW1qZ6vV71/PPPV7dv316k1ZYvg4OD6lVXXaXOmDFD9fl86uzZs9Vvfetbqt/vl9twX2fGX//6V9Pz86WXXqqqqr39evToUfWSSy5Ra2tr1fr6evXzn/+8OjQ0lPXaFFXVtUUkhBBCCCkz6JkhhBBCSFnDYIYQQgghZQ2DGUIIIYSUNQxmCCGEEFLWMJghhBBCSFnDYIYQQgghZQ2DGUIIIYSUNQxmCCGEEFLWMJghhBBCSFnDYIYQQgghZQ2DGUIIIYSUNQxmCCGEEFLW/H9pqPjHR47BBgAAAABJRU5ErkJggg==" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(all_rewards)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:22:36.059537Z", + "start_time": "2024-01-15T15:22:35.985945Z" + } + }, + "id": "457bbd9d81cac391" + }, + { + "cell_type": "code", + "execution_count": 10, + "outputs": [], + "source": [ + "cog.stop_service(\"lunar\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:22:39.592481Z", + "start_time": "2024-01-15T15:22:39.571297Z" + } + }, + "id": "4099d057af5affcf" + }, + { + "cell_type": "code", + "execution_count": 11, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "cenv = AECEnvironment(env_path=\"cogment_lab.envs.conversions.teacher.GymTeacherAEC\",\n", + " make_kwargs={\"gym_env_name\": \"LunarLander-v2\", \n", + " \"gym_make_kwargs\": {}, \n", + " \"render_mode\": \"rgb_array\"},\n", + " render=True)\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"lunar-teach\",\n", + " port=9011, \n", + " log_file=\"env.log\")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:22:53.079736Z", + "start_time": "2024-01-15T15:22:50.933845Z" + } + }, + "id": "8dc36a4b38990059" + }, + { + "cell_type": "code", + "execution_count": 12, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "actions = {\n", + " \"no-op\": {\"active\": 0, \"action\": 0},\n", + " \"ArrowDown\": {\"active\": 1, \"action\": 0},\n", + " \"ArrowRight\": {\"active\": 1, \"action\": 1},\n", + " \"ArrowUp\": {\"active\": 1, \"action\": 2},\n", + " \"ArrowLeft\": {\"active\": 1, \"action\": 3},\n", + "}\n", + "\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=60)" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:22:57.152462Z", + "start_time": "2024-01-15T15:22:54.906515Z" + } + }, + "id": "d4a8788421ffd8f4" + }, + { + "cell_type": "code", + "execution_count": 13, + "outputs": [], + "source": [ + "\n", + "trial_id = await cog.start_trial(\n", + " env_name=\"lunar-teach\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"gym\": \"coltra\",\n", + " \"teacher\": \"web_ui\",\n", + " },\n", + ")\n", + "\n", + "data = await cog.get_trial_data(trial_id=trial_id)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:23:24.879012Z", + "start_time": "2024-01-15T15:22:59.156914Z" + } + }, + "id": "94ef1fc6ba14903f" + }, + { + "cell_type": "code", + "execution_count": 14, + "outputs": [ + { + "data": { + "text/plain": "-438.36224" + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data[\"gym\"].rewards.sum()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:23:28.514466Z", + "start_time": "2024-01-15T15:23:28.507954Z" + } + }, + "id": "cad1754096b0b41b" + }, + { + "cell_type": "code", + "execution_count": 15, + "outputs": [], + "source": [ + "await cog.cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T15:23:33.250140Z", + "start_time": "2024-01-15T15:23:32.158203Z" + } + }, + "id": "d8c9467e9c724d7b" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/models/lunar/agent.pt b/examples/models/lunar/agent.pt new file mode 100644 index 0000000..3627621 Binary files /dev/null and b/examples/models/lunar/agent.pt differ diff --git a/examples/models/lunar/model.pt b/examples/models/lunar/model.pt new file mode 100644 index 0000000..4877901 Binary files /dev/null and b/examples/models/lunar/model.pt differ diff --git a/examples/pettingzoo/pz-atari-interactive.ipynb b/examples/pettingzoo/pz-atari-interactive.ipynb new file mode 100644 index 0000000..03f1732 --- /dev/null +++ b/examples/pettingzoo/pz-atari-interactive.ipynb @@ -0,0 +1,285 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 2, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2024-01-09T16:47:13.176123Z", + "start_time": "2024-01-09T16:47:11.657179Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "\n", + "from cogment_lab.actors import RandomActor, ConstantActor\n", + "from cogment_lab.envs.pettingzoo import AECEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.runners import process_cleanup\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:47:13.230710Z", + "start_time": "2024-01-09T16:47:13.184354Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:47:13.750134Z", + "start_time": "2024-01-09T16:47:13.746042Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2024-01-09T17:47:13.230277\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:47:16.131854Z", + "start_time": "2024-01-09T16:47:13.771295Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch an environment in a subprocess\n", + "# \n", + "# cenv = AECEnvironment(env_path=\"pettingzoo.butterfly.cooperative_pong_v5.env\",\n", + "# render=True)\n", + "\n", + "\n", + "cenv = AECEnvironment(env_path=\"pettingzoo.atari.pong_v3.env\",\n", + " render=True,\n", + " make_kwargs={\"max_cycles\": 1000})\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"pong\",\n", + " port=9011, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:47:21.823661Z", + "start_time": "2024-01-09T16:47:17.528096Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch two dummy actors in subprocesses\n", + "\n", + "random_actor = RandomActor(cenv.env.action_space(\"first_0\"))\n", + "constant_actor = ConstantActor(1)\n", + "\n", + "await cog.run_actor(actor=random_actor, \n", + " actor_name=\"random\", \n", + " port=9021, \n", + " log_file=\"actor-random.log\")\n", + "\n", + "await cog.run_actor(actor=constant_actor,\n", + " actor_name=\"constant\",\n", + " port=9022,\n", + " log_file=\"actor-constant.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "outputs": [ + { + "data": { + "text/plain": "{'pong': ,\n 'random': ,\n 'constant': }" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:47:21.829448Z", + "start_time": "2024-01-09T16:47:21.822231Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:47:23.978317Z", + "start_time": "2024-01-09T16:47:21.827487Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# PONG_ACTIONS = [\"no-op\", \"ArrowUp\", \"ArrowDown\"]\n", + "\n", + "PONG_ACTIONS = [\"no-op\", \"F\", \"ArrowRight\", \"ArrowLeft\", \"ArrowUp\", \"ArrowDown\"]\n", + "\n", + "actions = PONG_ACTIONS\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=60)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [], + "source": [ + "# Get data from a random + random trial\n", + "# You can change the values in `actor_impls` between `web_ui`, `random`, and `constant` to see the different behaviors\n", + "\n", + "trial_id = await cog.start_trial(\n", + " env_name=\"pong\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"first_0\": \"random\",\n", + " \"second_0\": \"web_ui\",\n", + " },\n", + ")\n", + "\n", + "data = await cog.get_trial_data(trial_id)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:48:10.877960Z", + "start_time": "2024-01-09T16:47:25.907512Z" + } + }, + "id": "8052ff03998b0b52" + }, + { + "cell_type": "code", + "execution_count": 11, + "outputs": [ + { + "data": { + "text/plain": "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,\n 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 4, 4,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5,\n 5, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 4, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,\n 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 4, 4, 4, 4, 4, 0, 5, 5, 5, 5, 5, 0, 0, 4, 4, 4, 4,\n 4, 4, 0, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 5, 5, 5, 0, 0, 0, 0, 0, 0, 4, 4, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0,\n 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 5, 5, 5, 5, 5, 5, 5, 5, 5,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 0, 4, 4,\n 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5,\n 5, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 4,\n 4, 4, 4, 4, 4, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,\n 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0,\n 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 4, 4, 4,\n 4, 4, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 0, 0, 0,\n 4, 4, 4, 4, 5, 5, 5, 5, 5, 0, 0, 0, 4, 4, 4, 4, 5, 5, 5, 5, 5, 0,\n 0, 4, 4, 4, 0, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 0, 0, 4, 4, 4,\n 0, 0, 0, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 4, 4, 0, 0,\n 0, 0, 0, 0, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0])" + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data[\"second_0\"].actions" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-09T16:48:20.388506Z", + "start_time": "2024-01-09T16:48:20.385703Z" + } + }, + "id": "1800cbfeca577ec8" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/pettingzoo/pz-interactive.ipynb b/examples/pettingzoo/pz-interactive.ipynb new file mode 100644 index 0000000..b812132 --- /dev/null +++ b/examples/pettingzoo/pz-interactive.ipynb @@ -0,0 +1,286 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 2, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T21:10:04.931015Z", + "start_time": "2023-12-13T21:10:02.952918Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "\n", + "from cogment_lab.actors import RandomActor, ConstantActor\n", + "from cogment_lab.envs.pettingzoo import AECEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.runners import process_cleanup\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:10:04.993162Z", + "start_time": "2023-12-13T21:10:04.931405Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:10:04.998015Z", + "start_time": "2023-12-13T21:10:04.992371Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T22:10:04.992766\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:12:23.631838Z", + "start_time": "2023-12-13T21:12:20.897201Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch an environment in a subprocess\n", + "\n", + "cenv = AECEnvironment(env_path=\"pettingzoo.butterfly.cooperative_pong_v5.env\",\n", + " render=True)\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"pong\",\n", + " port=9011, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:12:28.040971Z", + "start_time": "2023-12-13T21:12:23.630748Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch two dummy actors in subprocesses\n", + "\n", + "random_actor = RandomActor(cenv.env.action_space(\"paddle_0\"))\n", + "constant_actor = ConstantActor(1)\n", + "\n", + "await cog.run_actor(actor=random_actor, \n", + " actor_name=\"random\", \n", + " port=9021, \n", + " log_file=\"actor-random.log\")\n", + "\n", + "await cog.run_actor(actor=constant_actor,\n", + " actor_name=\"constant\",\n", + " port=9022,\n", + " log_file=\"actor-constant.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "outputs": [ + { + "data": { + "text/plain": "{'pong': ,\n 'random': ,\n 'constant': }" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:12:41.346850Z", + "start_time": "2023-12-13T21:12:41.339815Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:12:47.732796Z", + "start_time": "2023-12-13T21:12:45.491361Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "PONG_ACTIONS = [\"no-op\", \"ArrowUp\", \"ArrowDown\"]\n", + "\n", + "actions = PONG_ACTIONS\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=60)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "outputs": [], + "source": [ + "# Get data from a random + random trial\n", + "# You can change the values in `actor_impls` between `web_ui`, `random`, and `constant` to see the different behaviors\n", + "\n", + "trial_id = await cog.start_trial(\n", + " env_name=\"pong\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"paddle_0\": \"constant\",\n", + " \"paddle_1\": \"web_ui\",\n", + " },\n", + ")\n", + "\n", + "data = await cog.get_trial_data(trial_id)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:13:35.139518Z", + "start_time": "2023-12-13T21:13:25.521318Z" + } + }, + "id": "8052ff03998b0b52" + }, + { + "cell_type": "code", + "execution_count": 17, + "outputs": [ + { + "data": { + "text/plain": "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 0, 1, 1, 1, 0, 2, 2,\n 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0])" + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data[\"paddle_1\"].actions" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:17:41.560517Z", + "start_time": "2023-12-13T21:17:41.555599Z" + } + }, + "id": "1800cbfeca577ec8" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/pettingzoo/pz-kaz.ipynb b/examples/pettingzoo/pz-kaz.ipynb new file mode 100644 index 0000000..bea18da --- /dev/null +++ b/examples/pettingzoo/pz-kaz.ipynb @@ -0,0 +1,302 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2023-12-13T21:21:42.488660Z", + "start_time": "2023-12-13T21:21:40.354720Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "\n", + "from cogment_lab.actors import RandomActor, ConstantActor\n", + "from cogment_lab.envs.pettingzoo import AECEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.runners import process_cleanup\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:21:46.747801Z", + "start_time": "2023-12-13T21:21:46.684053Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:21:48.556919Z", + "start_time": "2023-12-13T21:21:48.549677Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2023-12-13T22:21:48.548102\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ariel/PycharmProjects/cogment_lab/venv/lib/python3.10/site-packages/cogment/context.py:213: UserWarning: No logging handler defined (e.g. logging.basicConfig)\n", + " warnings.warn(\"No logging handler defined (e.g. logging.basicConfig)\")\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:21:54.881072Z", + "start_time": "2023-12-13T21:21:51.714545Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch an environment in a subprocess\n", + "\n", + "cenv = AECEnvironment(env_path=\"pettingzoo.butterfly.knights_archers_zombies_v10.env\",\n", + " render=True)\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"kaz\",\n", + " port=9001, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:22:06.145915Z", + "start_time": "2023-12-13T21:22:01.760364Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch two dummy actors in subprocesses\n", + "\n", + "random_actor = RandomActor(cenv.env.action_space(\"knight_0\"))\n", + "constant_actor = ConstantActor(5)\n", + "\n", + "await cog.run_actor(actor=random_actor, \n", + " actor_name=\"random\", \n", + " port=9021, \n", + " log_file=\"actor-random.log\")\n", + "\n", + "await cog.run_actor(actor=constant_actor,\n", + " actor_name=\"constant\",\n", + " port=9022,\n", + " log_file=\"actor-constant.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "outputs": [ + { + "data": { + "text/plain": "{'kaz': ,\n 'random': ,\n 'constant': }" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:22:23.177880Z", + "start_time": "2023-12-13T21:22:23.174452Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:22:39.092581Z", + "start_time": "2023-12-13T21:22:36.945998Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "PONG_ACTIONS = [\"ArrowUp\", \"ArrowDown\", \"ArrowLeft\", \"ArrowRight\", \"f\", \"no-op\"]\n", + "\n", + "actions = PONG_ACTIONS\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=60*4)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [], + "source": [ + "# Get data from a random + random trial\n", + "# You can change the values in `actor_impls` between `web_ui`, `random`, and `constant` to see the different behaviors\n", + "\n", + "trial_id = await cog.start_trial(\n", + " env_name=\"kaz\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"knight_0\": \"random\",\n", + " \"knight_1\": \"random\",\n", + " \"archer_0\": \"web_ui\",\n", + " \"archer_1\": \"random\",\n", + " },\n", + ")" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:22:48.142771Z", + "start_time": "2023-12-13T21:22:48.129780Z" + } + }, + "id": "e41e4ba2ff066f5c" + }, + { + "cell_type": "code", + "execution_count": 10, + "outputs": [], + "source": [ + "\n", + "data = await cog.get_trial_data(trial_id)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-13T21:23:46.501926Z", + "start_time": "2023-12-13T21:22:53.704091Z" + } + }, + "id": "8052ff03998b0b52" + }, + { + "cell_type": "code", + "execution_count": 15, + "outputs": [ + { + "data": { + "text/plain": "array([4, 0, 2, 4, 3, 2, 3, 2, 5, 2, 0, 5, 0, 0, 1, 2, 4, 3, 2, 5, 4, 5,\n 3, 1, 5, 0, 2, 4, 5, 0, 4, 1, 2, 5, 5, 0, 1, 5, 0, 2, 5, 4, 3, 4,\n 2, 2, 2, 1, 2, 2, 4, 0, 2, 3, 2, 4, 2, 1, 3, 5, 1, 4, 0, 4, 5, 1,\n 5, 4, 3, 4, 1, 5, 0, 4, 4, 2, 0, 5, 2, 5, 2, 2, 4, 3, 4, 3, 2, 0,\n 3, 0, 4, 3, 3, 3, 1, 0, 1, 0, 0, 1, 3, 3, 2, 1, 1, 3, 2, 3, 1, 5,\n 5, 3, 4, 5, 5, 3, 0, 0, 5, 5, 4, 2, 3, 2, 2, 5, 0, 5, 0, 0, 3, 3,\n 3, 5, 3, 4, 3, 2, 0, 0, 0, 4, 1, 1, 1, 0, 1, 0, 2, 5, 0, 4, 5, 0,\n 3, 0, 4, 1, 1, 2, 2, 0, 5, 0, 0, 0, 1, 5, 4, 0, 0, 5, 2, 0, 1, 5,\n 3, 2, 3, 5, 4, 0])" + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data[\"knight_0\"].actions" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2023-12-08T16:45:25.290174Z", + "start_time": "2023-12-08T16:45:25.287374Z" + } + }, + "id": "1800cbfeca577ec8" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/pettingzoo/pz-parallel-interactive.ipynb b/examples/pettingzoo/pz-parallel-interactive.ipynb new file mode 100644 index 0000000..ca96f2a --- /dev/null +++ b/examples/pettingzoo/pz-parallel-interactive.ipynb @@ -0,0 +1,278 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 3, + "id": "initial_id", + "metadata": { + "ExecuteTime": { + "end_time": "2024-01-15T20:53:56.772459Z", + "start_time": "2024-01-15T20:53:56.368460Z" + } + }, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "\n", + "from cogment_lab.actors import RandomActor, ConstantActor\n", + "from cogment_lab.envs.pettingzoo import ParallelEnvironment\n", + "from cogment_lab.process_manager import Cogment\n", + "from cogment_lab.utils.runners import process_cleanup\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Processes terminated successfully.\n" + ] + } + ], + "source": [ + "# Cleans up potentially hanging background processes from previous runs\n", + "process_cleanup()" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:53:56.840328Z", + "start_time": "2024-01-15T20:53:56.773095Z" + } + }, + "id": "d431ab6f9d8d29cb" + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "2658232039e652c3", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:53:56.844588Z", + "start_time": "2024-01-15T20:53:56.841026Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logs/logs-2024-01-15T21:53:56.839732\n" + ] + } + ], + "source": [ + "logpath = f\"logs/logs-{datetime.datetime.now().isoformat()}\"\n", + "\n", + "cog = Cogment(log_dir=logpath)\n", + "\n", + "print(logpath)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "a074d1b3-b399-4e34-a68b-e86adb20caee", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:53:57.824755Z", + "start_time": "2024-01-15T20:53:57.007830Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch an environment in a subprocess\n", + "\n", + "cenv = ParallelEnvironment(env_path=\"pettingzoo.butterfly.cooperative_pong_v5.parallel_env\",\n", + " render=True)\n", + "\n", + "await cog.run_env(env=cenv, \n", + " env_name=\"pong\",\n", + " port=9011, \n", + " log_file=\"env.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "3374d134b845beb2", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:53:58.771737Z", + "start_time": "2024-01-15T20:53:57.822248Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Launch two dummy actors in subprocesses\n", + "\n", + "random_actor = RandomActor(cenv.env.action_space(\"paddle_0\"))\n", + "constant_actor = ConstantActor(1)\n", + "\n", + "await cog.run_actor(actor=random_actor, \n", + " actor_name=\"random\", \n", + " port=9021, \n", + " log_file=\"actor-random.log\")\n", + "\n", + "await cog.run_actor(actor=constant_actor,\n", + " actor_name=\"constant\",\n", + " port=9022,\n", + " log_file=\"actor-constant.log\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "outputs": [ + { + "data": { + "text/plain": "{'pong': ,\n 'random': ,\n 'constant': }" + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Check what's running\n", + "\n", + "cog.processes" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:53:58.773901Z", + "start_time": "2024-01-15T20:53:58.771546Z" + } + }, + "id": "896164c911313b40" + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "835c4d6ecb2afb23", + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:48:09.979970Z", + "start_time": "2024-01-15T20:48:09.351179Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": "True" + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "PONG_ACTIONS = [\"no-op\", \"ArrowUp\", \"ArrowDown\"]\n", + "\n", + "actions = PONG_ACTIONS\n", + "await cog.run_web_ui(actions=actions, log_file=\"human.log\", fps=60)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "outputs": [], + "source": [ + "# Get data from a random + random trial\n", + "# You can change the values in `actor_impls` between `web_ui`, `random`, and `constant` to see the different behaviors\n", + "\n", + "trial_id = await cog.start_trial(\n", + " env_name=\"pong\",\n", + " session_config={\"render\": True},\n", + " actor_impls={\n", + " \"paddle_0\": \"random\",\n", + " \"paddle_1\": \"constant\",\n", + " },\n", + ")\n", + "\n", + "data = await cog.get_trial_data(trial_id)\n" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:54:09.441520Z", + "start_time": "2024-01-15T20:54:07.117269Z" + } + }, + "id": "b4854b1295d75e9d" + }, + { + "cell_type": "code", + "execution_count": 10, + "outputs": [ + { + "data": { + "text/plain": "array([2, 2, 2, 2, 1, 0, 2, 1, 0, 1, 0, 2, 1, 1, 0, 1, 1, 2, 1, 1, 1, 0,\n 1, 2])" + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data[\"paddle_0\"].actions" + ], + "metadata": { + "collapsed": false, + "ExecuteTime": { + "end_time": "2024-01-15T20:54:09.652354Z", + "start_time": "2024-01-15T20:54:09.649361Z" + } + }, + "id": "1800cbfeca577ec8" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/pettingzoo/pz-simple.py b/examples/pettingzoo/pz-simple.py new file mode 100644 index 0000000..551d872 --- /dev/null +++ b/examples/pettingzoo/pz-simple.py @@ -0,0 +1,83 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import datetime +import asyncio + +import matplotlib.pyplot as plt +import numpy as np +import torch +import torch.nn.functional as F +from coltra import HomogeneousGroup +from coltra.buffers import Observation +from coltra.models import MLPModel +from coltra.policy_optimization import CrowdPPOptimizer +from tqdm import trange + +from cogment_lab.actors import ColtraActor, ConstantActor +from cogment_lab.envs import AECEnvironment +from cogment_lab.envs.gymnasium import GymEnvironment +from cogment_lab.process_manager import Cogment +from cogment_lab.utils.coltra_utils import convert_trial_data_to_coltra +from cogment_lab.utils.runners import process_cleanup +from cogment_lab.utils.trial_utils import concatenate + + +async def main(): + logpath = f"logs/logs-{datetime.datetime.now().isoformat()}" + + cog = Cogment(log_dir=logpath) + + print(logpath) + + cenv = AECEnvironment( + env_path="pettingzoo.butterfly.cooperative_pong_v5.env", render=False, make_kwargs={"max_cycles": 20} + ) + + await cog.run_env(env=cenv, env_name="pong", port=9011, log_file="env.log") + + # Create a model using coltra + + constant_actor = ConstantActor(1) + + await cog.run_actor(actor=constant_actor, actor_name="constant", port=9022, log_file="actor-constant.log") + + # Estimate random agent performance + + trial_id = await cog.start_trial( + env_name="pong", + session_config={"render": True}, + actor_impls={ + "paddle_0": "constant", + "paddle_1": "constant", + }, + ) + + data = await cog.get_trial_data(trial_id=trial_id) + + # mean_reward = np.mean([sum(e.rewards) for e in episodes]) + print(f"Reward shape: {data['paddle_0'].rewards.shape}") + print(f"Rewards: {data['paddle_0'].rewards}") + print(f"Action shape: {data['paddle_0'].actions.shape}") + print(f"Actions: {data['paddle_0'].actions}") + + # Other agent + print(f"Reward shape: {data['paddle_1'].rewards.shape}") + print(f"Rewards: {data['paddle_1'].rewards}") + print(f"Action shape: {data['paddle_1'].actions.shape}") + print(f"Actions: {data['paddle_1'].actions}") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..99fab85 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,192 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "cogment_lab" +dynamic = ["version"] +readme = "README.md" +license = "" + +dependencies = [ + "cogment[generate]>=2.0.0,<3.0.0", + "grpcio>=1.44", + "PyYAML~=6.0.1", + "starlette>=0.21.0", + "uvicorn>=0.17.6", + "Gymnasium[classic_control]~=0.29", + "PettingZoo~=1.23.1", + "numpy", + "opencv-python>=4.8", + "fastapi>=0.105", + "pillow>=9.0" +] + +[project.scripts] +cogmentlab = "cogment_lab.cli.cli:main" + +[tool.hatch.version] +path = "cogment_lab/__init__.py" + +[tool.hatch.build.targets.sdist] +include = [ + "/cogment_lab", +] + +# Package ###################################################################### + + +[project.optional-dependencies] +# Update dependencies in `all` if any are added or removed +all = [ + "pytest >=7.1.3", + "pytest-asyncio", + "coltra>=0.2.1", + "torch", + "Grid2Op==1.9.6", + "lightsim2grid", + "numba==0.56.4", + "matplotlib", + "wandb>=0.13.9", +] +dev = [ + "pytest >=7.1.3", + "pytest-asyncio", + "ruff>=0.1.7", + "jupyter>=1.0.0", + "jupyterlab>=3.5.3", +] +coltra = [ + "coltra>=0.2.1", + "torch", + "wandb>=0.13.9", +] +grid2op = [ + "Grid2Op==1.9.6", + "lightsim2grid", + "numba==0.56.4", +] + +[project.urls] +Homepage = "https://cogment.ai/lab" +Repository = "https://github.com/cogment/cogment_lab" +Documentation = "https://cogment.ai/lab" +"Bug Report" = "https://github.com/cogment/cogment_lab/issues" + +[tool.setuptools] +include-package-data = true + +[tool.setuptools.packages.find] +include = ["cogment_lab", "cogment_lab.*"] + +[tool.setuptools.package-data] +cogment_lab = [ + "py.typed", +] + +# Linters and Test tools ####################################################### + +[tool.black] +safe = true + +[tool.isort] +atomic = true +profile = "black" +src_paths = ["cogment_lab", "tests", "docs/scripts"] +extra_standard_library = ["typing_extensions"] +indent = 4 +lines_after_imports = 2 +multi_line_output = 3 + +[tool.pyright] +include = ["cogment_lab/**", "tests/**"] +exclude = ["**/node_modules", "**/__pycache__"] +strict = [] + +typeCheckingMode = "basic" +pythonVersion = "3.7" +pythonPlatform = "All" +typeshedPath = "typeshed" +enableTypeIgnoreComments = true + +# This is required as the CI pre-commit does not download the module (i.e. numpy, pygame, box2d) +# Therefore, we have to ignore missing imports +reportMissingImports = "none" +# Some modules are missing type stubs, which is an issue when running pyright locally +reportMissingTypeStubs = false +# For warning and error, will raise an error when +reportInvalidTypeVarUse = "none" + +# reportUnknownMemberType = "warning" # -> raises 6035 warnings +# reportUnknownParameterType = "warning" # -> raises 1327 warnings +# reportUnknownVariableType = "warning" # -> raises 2585 warnings +# reportUnknownArgumentType = "warning" # -> raises 2104 warnings +reportGeneralTypeIssues = "none" # -> commented out raises 489 errors +reportUntypedFunctionDecorator = "none" # -> pytest.mark.parameterize issues + +reportPrivateUsage = "warning" +reportUnboundVariable = "warning" + +[tool.pytest.ini_options] +filterwarnings = [] + + +[tool.ruff] +# Exclude a variety of commonly ignored directories. +exclude = [ + ".bzr", + ".direnv", + ".eggs", + ".git", + ".git-rewrite", + ".hg", + ".mypy_cache", + ".nox", + ".pants.d", + ".pytype", + ".ruff_cache", + ".svn", + ".tox", + ".venv", + "__pypackages__", + "_build", + "buck-out", + "build", + "dist", + "node_modules", + "venv", + "scripts", + "cogment_lab/generated" +] + +# Same as Black. +line-length = 120 +indent-width = 4 + +# Assume Python 3.8 +target-version = "py38" + +[tool.ruff.lint] +# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`) codes by default. +select = ["E4", "E7", "E9", "F"] +ignore = [] + +# Allow fix for all enabled rules (when `--fix`) is provided. +fixable = ["ALL"] +unfixable = [] + +# Allow unused variables when underscore-prefixed. +dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$" + +[tool.ruff.format] +# Like Black, use double quotes for strings. +quote-style = "double" + +# Like Black, indent with spaces, rather than tabs. +indent-style = "space" + +# Like Black, respect magic trailing commas. +skip-magic-trailing-comma = false + +# Like Black, automatically detect the appropriate line ending. +line-ending = "auto" \ No newline at end of file diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..c220c0e --- /dev/null +++ b/requirements.txt @@ -0,0 +1,30 @@ +cogment[generate]>=2.10.1,<3.0.0 +PyYAML~=6.0.1 +starlette>=0.21.0 +uvicorn[standard]==0.17.6 +fastapi>=0.103.2 +pillow>=9.0 + +# environments +Gymnasium~=0.29 +PettingZoo~=1.23.1 + +# actors +torch +jax +jaxlib +numpy + +# For testing +black~=22.3.0 +pylint~=2.14 +pytest>=7.0,<8.0 +pytest-benchmark~=4.0.0 +pytest-timeout~=2.1.0 + +Grid2Op==1.9.6 +lightsim2grid +numba==0.56.4 + +# RL +coltra>=0.2.1 \ No newline at end of file diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..d50072a --- /dev/null +++ b/setup.py @@ -0,0 +1,56 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Setups the project.""" + +import pathlib + +from setuptools import setup + + +CWD = pathlib.Path(__file__).absolute().parent + + +def get_version(): + """Gets the cogment_lab version.""" + path = CWD / "cogment_lab" / "__init__.py" + content = path.read_text() + + for line in content.splitlines(): + if line.startswith("__version__"): + return line.strip().split()[-1].strip().strip('"') + raise RuntimeError("bad version data in __init__.py") + + +def get_description(): + """Gets the description from the readme.""" + with open("README.md") as fh: + long_description = "" + header_count = 0 + for line in fh: + if line.startswith("##"): + header_count += 1 + if header_count < 2: + long_description += line + else: + break + return long_description + + +setup( + name="cogment_lab", + version=get_version(), + long_description=get_description(), + entry_points={"console_scripts": ["cogment_lab=cogment_lab.cli.cli:main"]}, +) diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..7392fcb --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1,15 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Testing for CogmentLab.""" diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..46545dc --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,29 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import subprocess +import time + +import pytest + + +@pytest.fixture(scope="session", autouse=True) +def cogment_lab_subprocess(): + # pass + process = subprocess.Popen(["cogmentlab", "launch", "base"]) + + yield process + + process.terminate() + process.wait() diff --git a/tests/test_core.py b/tests/test_core.py new file mode 100644 index 0000000..a3e6480 --- /dev/null +++ b/tests/test_core.py @@ -0,0 +1,15 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests of the core functionalities.""" diff --git a/tests/test_gymnasium.py b/tests/test_gymnasium.py new file mode 100644 index 0000000..aa6ed94 --- /dev/null +++ b/tests/test_gymnasium.py @@ -0,0 +1,54 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +import numpy as np +import pytest + +from cogment_lab.actors import ConstantActor +from cogment_lab.envs.gymnasium import GymEnvironment +from cogment_lab.process_manager import Cogment + + +@pytest.mark.asyncio +async def test_cartpole(): + """Test the cartpole environment.""" + + cog = Cogment(log_dir="logs") + + cenv = GymEnvironment(env_id="CartPole-v1", render=True) + + await cog.run_env(env=cenv, env_name="cartpole", port=9011, log_file="env.log") + + constant_actor = ConstantActor(0) + + await cog.run_actor(actor=constant_actor, actor_name="constant", port=9021, log_file="actor-constant.log") + + trial_id = await cog.start_trial( + env_name="cartpole", + session_config={"render": True}, + actor_impls={ + "gym": "constant", + }, + ) + + data = await cog.get_trial_data(trial_id=trial_id, env_name="cartpole") + + assert isinstance(data, dict) + assert isinstance(data["gym"].observations, np.ndarray) + assert isinstance(data["gym"].actions, np.ndarray) + assert isinstance(data["gym"].rewards, np.ndarray) + + await cog.cleanup() diff --git a/tests/test_pettingzoo.py b/tests/test_pettingzoo.py new file mode 100644 index 0000000..91d9a3e --- /dev/null +++ b/tests/test_pettingzoo.py @@ -0,0 +1,56 @@ +# Copyright 2024 AI Redefined Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +import numpy as np +import pytest + +from cogment_lab.actors import ConstantActor +from cogment_lab.envs.pettingzoo import AECEnvironment +from cogment_lab.process_manager import Cogment + + +@pytest.mark.asyncio +async def test_pong(): + """Test the cartpole environment.""" + + cog = Cogment(log_dir="logs") + + cenv = AECEnvironment(env_path="pettingzoo.butterfly.cooperative_pong_v5.env", render=False) + + await cog.run_env(env=cenv, env_name="pong", port=9012, log_file="env.log") + + constant_actor = ConstantActor(1) + + await cog.run_actor(actor=constant_actor, actor_name="constant", port=9022, log_file="actor-constant.log") + + trial_id = await cog.start_trial( + env_name="pong", + session_config={"render": False}, + actor_impls={ + "paddle_0": "constant", + "paddle_1": "constant", + }, + ) + + data = await cog.get_trial_data(trial_id=trial_id, env_name="pong") + + for agent_name in ["paddle_0", "paddle_1"]: + assert isinstance(data, dict) + assert isinstance(data[agent_name].observations, np.ndarray) + assert isinstance(data[agent_name].actions, np.ndarray) + assert isinstance(data[agent_name].rewards, np.ndarray) + + await cog.cleanup()