From cf0e015a6a30ca1971ae1efd7b600084cf451612 Mon Sep 17 00:00:00 2001 From: Aleksey Veresov Date: Mon, 15 Jul 2024 09:44:36 +0200 Subject: [PATCH] Merge CONTRIBUTING --- CONTRIBUTING.md | 116 +++++++++++------------ hsfs/CONTRIBUTING.md | 220 ------------------------------------------- hsml/CONTRIBUTING.md | 215 ------------------------------------------ 3 files changed, 57 insertions(+), 494 deletions(-) delete mode 100644 hsfs/CONTRIBUTING.md delete mode 100644 hsml/CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 564734d53..e2801b11b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,16 +1,17 @@ ## Python development setup + --- - Fork and clone the repository -- Create a new Python environment with your favourite environment manager, e.g. virtualenv or conda +- Create a new Python environment with your favourite environment manager (e.g. virtualenv or conda) and Python 3.9 (newer versions will return a library conflict in `auto_doc.py`) - Install repository in editable mode with development dependencies: - ```bash - cd python - pip install -e ".[dev]" - ``` + ```bash + cd python + pip install -e ".[dev]" + ``` - Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The library uses pre-commit to ensure code-style and code formatting through [ruff](https://docs.astral.sh/ruff/). Run the following commands from the `python` directory: @@ -37,75 +38,67 @@ We follow a few best practices for writing the Python documentation: 1. Use the google docstring style: - ```python - """[One Line Summary] + ```python + """[One Line Summary] - [Extended Summary] + [Extended Summary] - [!!! example - import xyz - ] + [!!! example + import xyz + ] - # Arguments - arg1: Type[, optional]. Description[, defaults to `default`] - arg2: Type[, optional]. Description[, defaults to `default`] + # Arguments + arg1: Type[, optional]. Description[, defaults to `default`] + arg2: Type[, optional]. Description[, defaults to `default`] - # Returns - Type. Description. + # Returns + Type. Description. - # Raises - Exception. Description. - """ - ``` - - If Python 3 type annotations are used, they are inserted automatically. + # Raises + Exception. Description. + """ + ``` + If Python 3 type annotations are used, they are inserted automatically. 2. Hopsworks entity engine methods (e.g. ExecutionEngine etc.) only require a single line docstring. -3. Private REST Api implementations (e.g. GitRemoteApi etc.) should be fully documented with docstrings without defaults. -4. Public Api such as metadata objects and public REST Api implementations should be fully documented with defaults. +3. Private REST API implementations (e.g. FeatureGroupApi etc.) should be fully documented with docstrings without defaults. +4. Public API such as metadata objects and public REST API implementations should be fully documented with defaults. #### Setup and Build Documentation We use `mkdocs` together with `mike` ([for versioning](https://github.com/jimporter/mike/)) to build the documentation and a plugin called `keras-autodoc` to auto generate Python API documentation from docstrings. **Background about `mike`:** - `mike` builds the documentation and commits it as a new directory to the gh-pages branch. Each directory corresponds to one version of the documentation. Additionally, `mike` maintains a json in the root of gh-pages with the mappings of versions/aliases for each of the directories available. With aliases you can define extra names like `dev` or `latest`, to indicate stable and unstable releases. +`mike` builds the documentation and commits it as a new directory to the gh-pages branch. Each directory corresponds to one version of the documentation. Additionally, `mike` maintains a json in the root of gh-pages with the mappings of versions/aliases for each of the directories available. With aliases you can define extra names like `dev` or `latest`, to indicate stable and unstable releases. -1. Currently we are using our own version of `keras-autodoc` +1. Install Hopsworks with `dev-docs` extras: - ```bash - pip install git+https://github.com/logicalclocks/keras-autodoc - ``` + ```bash + pip install -e ".[dev-docs]" + ``` -2. Install HOPSWORKS with `docs` extras: - - ```bash - pip install -e .[dev,docs] - ``` +2. To build the docs, first run the auto doc script: -3. To build the docs, first run the auto doc script: - - ```bash - cd .. - python auto_doc.py - ``` + ```bash + python auto_doc.py + ``` ##### Option 1: Build only current version of docs -4. Either build the docs, or serve them dynamically: +3. Either build the docs, or serve them dynamically: - Note: Links and pictures might not resolve properly later on when checking with this build. - The reason for that is that the docs are deployed with versioning on docs.hopsworks.ai and - therefore another level is added to all paths, e.g. `docs.hopsworks.ai/[version-or-alias]`. - Using relative links should not be affected by this, however, building the docs with version - (Option 2) is recommended. + Note: Links and pictures might not resolve properly later on when checking with this build. + The reason for that is that the docs are deployed with versioning on docs.hopsworks.ai and + therefore another level is added to all paths, e.g. `docs.hopsworks.ai/[version-or-alias]`. + Using relative links should not be affected by this, however, building the docs with version + (Option 2) is recommended. - ```bash - mkdocs build - # or - mkdocs serve - ``` + ```bash + mkdocs build + # or + mkdocs serve + ``` ##### Option 2 (Preferred): Build multi-version doc with `mike` @@ -113,34 +106,36 @@ We use `mkdocs` together with `mike` ([for versioning](https://github.com/jimpor On docs.hopsworks.ai we implement the following versioning scheme: -- current master branches (e.g. of hopsworks corresponding to master of Hopsworks): rendered as current Hopsworks snapshot version, e.g. **3.1.0-SNAPSHOT [dev]**, where `dev` is an alias to indicate that this is an unstable version. -- the latest release: rendered with full current version, e.g. **3.0.1 [latest]** with `latest` alias to indicate that this is the latest stable release. -- previous stable releases: rendered without alias, e.g. **3.0.0**. +- current master branches (e.g. of hopsworks corresponding to master of Hopsworks): rendered as current Hopsworks snapshot version, e.g. **4.0.0-SNAPSHOT [dev]**, where `dev` is an alias to indicate that this is an unstable version. +- the latest release: rendered with full current version, e.g. **3.8.0 [latest]** with `latest` alias to indicate that this is the latest stable release. +- previous stable releases: rendered without alias, e.g. **3.4.4**. ###### Build Instructions -4. For this you can either checkout and make a local copy of the `upstream/gh-pages` branch, where -`mike` maintains the current state of docs.hopsworks.ai, or just build documentation for the branch you are updating: +4. For this you can either checkout and make a local copy of the `upstream/gh-pages` branch, where `mike` maintains the current state of docs.hopsworks.ai, or just build documentation for the branch you are updating: Building *one* branch: Checkout your dev branch with modified docs: + ```bash git checkout [dev-branch] ``` Generate API docs if necessary: + ```bash python auto_doc.py ``` Build docs with a version and alias + ```bash mike deploy [version] [alias] --update-alias # for example, if you are updating documentation to be merged to master, # which will become the new SNAPSHOT version: - mike deploy 3.1.0-SNAPSHOT dev --update-alias + mike deploy 4.0.0-SNAPSHOT dev --update-alias # if you are updating docs of the latest stable release branch mike deploy [version] latest --update-alias @@ -158,17 +153,20 @@ On docs.hopsworks.ai we implement the following versioning scheme: ``` You can now checkout the gh-pages branch and serve: + ```bash git checkout gh-pages mike serve ``` You can also list all available versions/aliases: + ```bash mike list ``` Delete and reset your local gh-pages branch: + ```bash mike delete --all @@ -194,7 +192,7 @@ PAGES = { Now you can add a template markdown file to the `docs/templates` directory with the name you specified in the auto-doc script. The `new_template.md` file should contain a tag to identify the place at which the API documentation should be inserted: -``` +```` ## The XYZ package {{module}} @@ -207,7 +205,7 @@ Some extra content here. ``` {{xyz.asd}} -``` +```` Finally, run the `auto_doc.py` script, as decribed above, to update the documentation. diff --git a/hsfs/CONTRIBUTING.md b/hsfs/CONTRIBUTING.md deleted file mode 100644 index 0df3de08e..000000000 --- a/hsfs/CONTRIBUTING.md +++ /dev/null @@ -1,220 +0,0 @@ -## Python development setup - ---- - -- Fork and clone the repository - -- Create a new Python environment with your favourite environment manager (e.g. virtualenv or conda) and Python 3.9 (newer versions will return a library conflict in `auto_doc.py`) - -- Install repository in editable mode with development dependencies: - - ```bash - cd python - pip install -e ".[python,dev]" - ``` - -- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The Feature Store uses pre-commit to ensure code-style and code formatting through [ruff](https://docs.astral.sh/ruff/). Run the following commands from the `python` directory: - - ```bash - cd python - pip install --user pre-commit - pre-commit install - ``` - - Afterwards, pre-commit will run whenever you commit. - -- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use `ruff`, or run it via the command line: - - ```bash - # linting - ruff check python --fix - # formatting - ruff format python - ``` - -### Python documentation - -We follow a few best practices for writing the Python documentation: - -1. Use the google docstring style: - - ```python - """[One Line Summary] - - [Extended Summary] - - [!!! example - import xyz - ] - - # Arguments - arg1: Type[, optional]. Description[, defaults to `default`] - arg2: Type[, optional]. Description[, defaults to `default`] - - # Returns - Type. Description. - - # Raises - Exception. Description. - """ - ``` - - If Python 3 type annotations are used, they are inserted automatically. - -2. Feature store entity engine methods (e.g. FeatureGroupEngine etc.) only require a single line docstring. -3. REST Api implementations (e.g. FeatureGroupApi etc.) should be fully documented with docstrings without defaults. -4. Public Api such as metadata objects should be fully documented with defaults. - -#### Setup and Build Documentation - -We use `mkdocs` together with `mike` ([for versioning](https://github.com/jimporter/mike/)) to build the documentation and a plugin called `keras-autodoc` to auto generate Python API documentation from docstrings. - -**Background about `mike`:** -`mike` builds the documentation and commits it as a new directory to the gh-pages branch. Each directory corresponds to one version of the documentation. Additionally, `mike` maintains a json in the root of gh-pages with the mappings of versions/aliases for each of the directories available. With aliases you can define extra names like `dev` or `latest`, to indicate stable and unstable releases. - -1. Currently we are using our own version of `keras-autodoc` - - ```bash - pip install git+https://github.com/logicalclocks/keras-autodoc - ``` - -2. Install HSFS with `docs` extras: - - ```bash - pip install -e ".[python,dev]" && pip install -r ../requirements-docs.txt - ``` - -3. To build the docs, first run the auto doc script: - - ```bash - cd .. - python auto_doc.py - ``` - -##### Option 1: Build only current version of docs - -4. Either build the docs, or serve them dynamically: - - Note: Links and pictures might not resolve properly later on when checking with this build. - The reason for that is that the docs are deployed with versioning on docs.hopsworks.ai and - therefore another level is added to all paths, e.g. `docs.hopsworks.ai/[version-or-alias]`. - Using relative links should not be affected by this, however, building the docs with version - (Option 2) is recommended. - - ```bash - mkdocs build - # or - mkdocs serve - ``` - -##### Option 2 (Preferred): Build multi-version doc with `mike` - -###### Versioning on docs.hopsworks.ai - -On docs.hopsworks.ai we implement the following versioning scheme: - -- current master branches (e.g. of hsfs corresponding to master of Hopsworks): rendered as current Hopsworks snapshot version, e.g. **2.2.0-SNAPSHOT [dev]**, where `dev` is an alias to indicate that this is an unstable version. -- the latest release: rendered with full current version, e.g. **2.1.5 [latest]** with `latest` alias to indicate that this is the latest stable release. -- previous stable releases: rendered without alias, e.g. **2.1.4**. - -###### Build Instructions - -4. For this you can either checkout and make a local copy of the `upstream/gh-pages` branch, where `mike` maintains the current state of docs.hopsworks.ai, or just build documentation for the branch you are updating: - - Building _one_ branch: - - Checkout your dev branch with modified docs: - - ```bash - git checkout [dev-branch] - ``` - - Generate API docs if necessary: - - ```bash - python auto_doc.py - ``` - - Build docs with a version and alias - - ```bash - mike deploy [version] [alias] --update-alias - - # for example, if you are updating documentation to be merged to master, - # which will become the new SNAPSHOT version: - mike deploy 2.2.0-SNAPSHOT dev --update-alias - - # if you are updating docs of the latest stable release branch - mike deploy [version] latest --update-alias - - # if you are updating docs of a previous stable release branch - mike deploy [version] - ``` - - If no gh-pages branch existed in your local repository, this will have created it. - - **Important**: If no previous docs were built, you will have to choose a version as default to be loaded as index, as follows - - ```bash - mike set-default [version-or-alias] - ``` - - You can now checkout the gh-pages branch and serve: - - ```bash - git checkout gh-pages - mike serve - ``` - - You can also list all available versions/aliases: - - ```bash - mike list - ``` - - Delete and reset your local gh-pages branch: - - ```bash - mike delete --all - - # or delete single version - mike delete [version-or-alias] - ``` - -#### Adding new API documentation - -To add new documentation for APIs, you need to add information about the method/class to document to the `auto_doc.py` script: - -```python -PAGES = { - "connection.md": [ - "hsfs.connection.Connection.connection" - ] - "new_template.md": [ - "module", - "xyz.asd" - ] -} -``` - -Now you can add a template markdown file to the `docs/templates` directory with the name you specified in the auto-doc script. The `new_template.md` file should contain a tag to identify the place at which the API documentation should be inserted: - -```` -## The XYZ package - -{{module}} - -Some extra content here. - -!!! example - ```python - import xyz - ``` - -{{xyz.asd}} -```` - -Finally, run the `auto_doc.py` script, as decribed above, to update the documentation. - -For information about Markdown syntax and possible Admonitions/Highlighting etc. see -the [Material for Mkdocs themes reference documentation](https://squidfunk.github.io/mkdocs-material/reference/abbreviations/). diff --git a/hsml/CONTRIBUTING.md b/hsml/CONTRIBUTING.md deleted file mode 100644 index b287467c6..000000000 --- a/hsml/CONTRIBUTING.md +++ /dev/null @@ -1,215 +0,0 @@ -## Python development setup ---- - -- Fork and clone the repository - -- Create a new Python environment with your favourite environment manager, e.g. virtualenv or conda - -- Install repository in editable mode with development dependencies: - - ```bash - cd python - pip install -e ".[dev]" - ``` - -- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The Model Registry uses pre-commit to ensure code-style and code formatting through [ruff](https://docs.astral.sh/ruff/). Run the following commands from the `python` directory: - - ```bash - cd python - pip install --user pre-commit - pre-commit install - ``` - - Afterwards, pre-commit will run whenever you commit. - -- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use [ruff](https://docs.astral.sh/ruff/tutorial/#getting-started): - - ```bash - cd python - ruff check --fix - ruff format - ``` - -### Python documentation - -We follow a few best practices for writing the Python documentation: - -1. Use the google docstring style: - - ```python - """[One Line Summary] - - [Extended Summary] - - [!!! example - import xyz - ] - - # Arguments - arg1: Type[, optional]. Description[, defaults to `default`] - arg2: Type[, optional]. Description[, defaults to `default`] - - # Returns - Type. Description. - - # Raises - Exception. Description. - """ - ``` - - If Python 3 type annotations are used, they are inserted automatically. - - -2. Model registry entity engine methods (e.g. ModelEngine etc.) only require a single line docstring. -3. REST Api implementations (e.g. ModelApi etc.) should be fully documented with docstrings without defaults. -4. Public Api such as metadata objects should be fully documented with defaults. - -#### Setup and Build Documentation - -We use `mkdocs` together with `mike` ([for versioning](https://github.com/jimporter/mike/)) to build the documentation and a plugin called `keras-autodoc` to auto generate Python API documentation from docstrings. - -**Background about `mike`:** - `mike` builds the documentation and commits it as a new directory to the gh-pages branch. Each directory corresponds to one version of the documentation. Additionally, `mike` maintains a json in the root of gh-pages with the mappings of versions/aliases for each of the directories available. With aliases you can define extra names like `dev` or `latest`, to indicate stable and unstable releases. - -1. Currently we are using our own version of `keras-autodoc` - - ```bash - pip install git+https://github.com/logicalclocks/keras-autodoc - ``` - -2. Install HSML with `docs` extras: - - ```bash - pip install -e .[dev,docs] - ``` - -3. To build the docs, first run the auto doc script: - - ```bash - cd .. - python auto_doc.py - ``` - -##### Option 1: Build only current version of docs - -4. Either build the docs, or serve them dynamically: - - Note: Links and pictures might not resolve properly later on when checking with this build. - The reason for that is that the docs are deployed with versioning on docs.hopsworks.ai and - therefore another level is added to all paths, e.g. `docs.hopsworks.ai/[version-or-alias]`. - Using relative links should not be affected by this, however, building the docs with version - (Option 2) is recommended. - - ```bash - mkdocs build - # or - mkdocs serve - ``` - -##### Option 2 (Preferred): Build multi-version doc with `mike` - -###### Versioning on docs.hopsworks.ai - -On docs.hopsworks.ai we implement the following versioning scheme: - -- current master branches (e.g. of hsml corresponding to master of Hopsworks): rendered as current Hopsworks snapshot version, e.g. **2.2.0-SNAPSHOT [dev]**, where `dev` is an alias to indicate that this is an unstable version. -- the latest release: rendered with full current version, e.g. **2.1.5 [latest]** with `latest` alias to indicate that this is the latest stable release. -- previous stable releases: rendered without alias, e.g. **2.1.4**. - -###### Build Instructions - -4. For this you can either checkout and make a local copy of the `upstream/gh-pages` branch, where -`mike` maintains the current state of docs.hopsworks.ai, or just build documentation for the branch you are updating: - - Building *one* branch: - - Checkout your dev branch with modified docs: - ```bash - git checkout [dev-branch] - ``` - - Generate API docs if necessary: - ```bash - python auto_doc.py - ``` - - Build docs with a version and alias - ```bash - mike deploy [version] [alias] --update-alias - - # for example, if you are updating documentation to be merged to master, - # which will become the new SNAPSHOT version: - mike deploy 2.2.0-SNAPSHOT dev --update-alias - - # if you are updating docs of the latest stable release branch - mike deploy [version] latest --update-alias - - # if you are updating docs of a previous stable release branch - mike deploy [version] - ``` - - If no gh-pages branch existed in your local repository, this will have created it. - - **Important**: If no previous docs were built, you will have to choose a version as default to be loaded as index, as follows - - ```bash - mike set-default [version-or-alias] - ``` - - You can now checkout the gh-pages branch and serve: - ```bash - git checkout gh-pages - mike serve - ``` - - You can also list all available versions/aliases: - ```bash - mike list - ``` - - Delete and reset your local gh-pages branch: - ```bash - mike delete --all - - # or delete single version - mike delete [version-or-alias] - ``` - -#### Adding new API documentation - -To add new documentation for APIs, you need to add information about the method/class to document to the `auto_doc.py` script: - -```python -PAGES = { - "connection.md": [ - "hsml.connection.Connection.connection", - "hsml.connection.Connection.setup_databricks", - ] - "new_template.md": [ - "module", - "xyz.asd" - ] -} -``` - -Now you can add a template markdown file to the `docs/templates` directory with the name you specified in the auto-doc script. The `new_template.md` file should contain a tag to identify the place at which the API documentation should be inserted: - -``` -## The XYZ package - -{{module}} - -Some extra content here. - -!!! example - ```python - import xyz - ``` - -{{xyz.asd}} -``` - -Finally, run the `auto_doc.py` script, as decribed above, to update the documentation. - -For information about Markdown syntax and possible Admonitions/Highlighting etc. see -the [Material for Mkdocs themes reference documentation](https://squidfunk.github.io/mkdocs-material/reference/abbreviations/).