[FSTORE-1371][FSTORE-1346][FSTORE-1212] Migrate to ruff, pyproject.to…

…ml, upgrade doc dependencies (logicalclocks#227) * Upgrade dependencies, contributing.md, workflows * Fix contributing.md * Add minor change to contributing.md * Adapt to new material theme * Add minifier * Migrate to pyproject.toml * Change contributing * Run ruff check and ruff format * Minor mkdocs update * Update ruff and readme minor fix * Fix version typo * Add python version badges + code style changes * Fix import order messed up by merge * Add actual package to what should be installed * Fix color scheme * Add upper limit to the python version * Fix import order
aversey · May 6, 2024 · f7b78f4 · f7b78f4
1 parent f51af57
commit f7b78f4
Show file tree

Hide file tree

Showing 80 changed files with 653 additions and 409 deletions.
diff --git a/.github/workflows/mkdocs-main.yml b/.github/workflows/mkdocs-main.yml
@@ -7,21 +7,21 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
       - name: set dev version
         working-directory: ./java
         run: echo "DEV_VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -Ev 'Download|INFO|WARNING')" >> $GITHUB_ENV
 
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v5
         with:
-          python-version: '3.8'
+          python-version: "3.10"
 
       - name: install deps
         working-directory: ./python
-        run: cp ../README.md . && pip3 install pip==22.0.3 && pip3 install -e .[dev,docs]
+        run: cp ../README.md . && pip3 install -e .[dev,docs]
 
       - name: generate autodoc
         run: python3 auto_doc.py
@@ -33,4 +33,3 @@ jobs:
 
       - name: mike deploy docs
         run: mike deploy ${{ env.DEV_VERSION }} dev -u
-
diff --git a/.github/workflows/mkdocs-release.yml b/.github/workflows/mkdocs-release.yml
@@ -2,14 +2,14 @@ name: mkdocs-release
 
 on:
   push:
-    branches: [ branch-* ]
+    branches: [branch-*]
 
 jobs:
   publish-release:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
@@ -20,16 +20,13 @@ jobs:
       - name: set major/minor release version
         run: echo "MAJOR_VERSION=$(echo $RELEASE_VERSION | sed 's/^\([0-9]*\.[0-9]*\).*$/\1/')" >> $GITHUB_ENV
 
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v5
         with:
-          python-version: '3.8'
+          python-version: "3.10"
 
       - name: install deps
         working-directory: ./python
-        run: cp ../README.md . && pip3 install pip==22.0.3 && pip3 install -e .[dev,docs]
-
-      - name: use dev mike
-        run: pip3 uninstall -y mike && pip3 install git+'https://github.com/jimporter/mike.git'
+        run: cp ../README.md . && pip3 install -e .[dev,docs]
 
       - name: generate autodoc
         run: python3 auto_doc.py
@@ -43,4 +40,3 @@ jobs:
         run: |
           mike deploy ${{ env.RELEASE_VERSION }} ${{ env.MAJOR_VERSION }} -u --push
           mike alias ${{ env.RELEASE_VERSION }} latest -u --push
-
diff --git a/.github/workflows/python-lint.yml b/.github/workflows/python-lint.yml
@@ -1,27 +1,46 @@
-name: pre-commit
+name: python
 
 on: pull_request
 
 jobs:
   stylecheck:
+    name: Lint and Stylecheck
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
         with:
-          python-version: '3.8'
-      - name: install deps
-        run: pip install flake8==3.9.0 black==22.3.0 pre-commit-hooks==2.4.0
+          python-version: "3.11"
 
-      - name: black
-        run: black --check python/hsml
+      - name: Get all changed files
+        id: get-changed-files
+        uses: tj-actions/changed-files@v44
+        with:
+          files_yaml: |
+            src:
+              - 'python/**/*.py'
+              - '!python/tests/**/*.py'
+            test:
+              - 'python/tests/**/*.py'
+
+      - name: install deps
+        run: pip install ruff==0.4.1
 
-      - name: flake8
-        run: flake8 --config python/.flake8 python/hsml
+      - name: ruff on python files
+        if: steps.get-changed-files.outputs.src_any_changed == 'true'
+        env:
+          SRC_ALL_CHANGED_FILES: ${{ steps.get-changed-files.outputs.src_all_changed_files }}
+        run: ruff check --output-format=github $SRC_ALL_CHANGED_FILES
 
-      - name: trailing-whitespace-fixer
-        run: trailing-whitespace-fixer $(find python/hsml -type f) || exit 1
+      - name: ruff on test files
+        if: steps.get-changed-files.outputs.test_any_changed == 'true'
+        env:
+          TEST_ALL_CHANGED_FILES: ${{ steps.get-changed-files.outputs.test_all_changed_files }}
+        run: ruff check --output-format=github $TEST_ALL_CHANGED_FILES
 
-      - name: end-of-file-fixer
-        run: end-of-file-fixer $(find python/hsml -type f) || exit 1
+      - name: ruff format --check $ALL_CHANGED_FILES
+        env:
+          ALL_CHANGED_FILES: ${{ steps.get-changed-files.outputs.all_changed_files }}
+        run: ruff format $ALL_CHANGED_FILES
diff --git a/.gitignore b/.gitignore
@@ -50,6 +50,7 @@ coverage.xml
 *.cover
 .hypothesis/
 .pytest_cache/
+.ruff_cache/
 
 # Translations
 *.mo
@@ -127,4 +128,3 @@ target/
 
 # mkdocs intemediate files
 docs/generated
-
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -12,7 +12,7 @@
     pip install -e ".[dev]"
     ```
 
-- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The Model Registry uses pre-commit to ensure code-style and code formatting through [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8). Run the following commands from the `python` directory:
+- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The Model Registry uses pre-commit to ensure code-style and code formatting through [ruff](https://docs.astral.sh/ruff/). Run the following commands from the `python` directory:
 
     ```bash
     cd python
@@ -22,12 +22,12 @@
 
   Afterwards, pre-commit will run whenever you commit.
 
-- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use black and flake8, or run them via the command line:
+- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use [ruff](https://docs.astral.sh/ruff/tutorial/#getting-started):
 
     ```bash
     cd python
-    flake8 hsml
-    black hsml
+    ruff check --fix
+    ruff format
     ```
 
 ### Python documentation
@@ -74,7 +74,7 @@ We use `mkdocs` together with `mike` ([for versioning](https://github.com/jimpor
 1. Currently we are using our own version of `keras-autodoc`
 
     ```bash
-    pip install git+https://github.com/logicalclocks/keras-autodoc@split-tags-properties
+    pip install git+https://github.com/logicalclocks/keras-autodoc
     ```
 
 2. Install HSML with `docs` extras:

diff --git a/README.md b/README.md
@@ -9,6 +9,10 @@
     src="https://img.shields.io/badge/docs-HSML-orange"
     alt="Hopsworks Model Management Documentation"
   /></a>
+  <a><img
+    src="https://img.shields.io/badge/python-3.8+-blue"
+    alt="python"
+  /></a>
   <a href="https://pypi.org/project/hsml/"><img
     src="https://img.shields.io/pypi/v/hsml?color=blue"
     alt="PyPiStatus"
@@ -21,9 +25,9 @@
     src="https://pepy.tech/badge/hsml/month"
     alt="Downloads"
   /></a>
-  <a href="https://github.com/psf/black"><img
-    src="https://img.shields.io/badge/code%20style-black-000000.svg"
-    alt="CodeStyle"
+  <a href=https://github.com/astral-sh/ruff><img
+    src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json"
+    alt="Ruff"
   /></a>
   <a><img
     src="https://img.shields.io/pypi/l/hsml?color=green"
@@ -33,22 +37,44 @@
 
 HSML is the library to interact with the Hopsworks Model Registry and Model Serving. The library makes it easy to export, manage and deploy models.
 
-The library automatically configures itself based on the environment it is run.
-However, to connect from an external Python environment additional connection information, such as host and port, is required. For more information about the setup from external environments, see the setup section.
+However, to connect from an external Python environment additional connection information, such as host and port, is required.
 
 ## Getting Started On Hopsworks
 
-Instantiate a connection and get the project model registry and serving handles
+Get started easily by registering an account on [Hopsworks Serverless](https://app.hopsworks.ai/). Create your project and a [new Api key](https://docs.hopsworks.ai/latest/user_guides/projects/api_key/create_api_key/). In a new python environment with Python 3.8 or higher, install the [client library](https://docs.hopsworks.ai/latest/user_guides/client_installation/) using pip:
+
+```bash
+# Get all Hopsworks SDKs: Feature Store, Model Serving and Platform SDK
+pip install hopsworks
+# or just the Model Registry and Model Serving SDK
+pip install hsml
+```
+
+You can start a notebook and instantiate a connection and get the project feature store handler.
+
+```python
+import hopsworks
+
+project = hopsworks.login() # you will be prompted for your api key
+
+mr = project.get_model_registry()
+# or
+ms = project.get_model_serving()
+```
+
+or using `hsml` directly:
+
 ```python
 import hsml
 
-# Create a connection
-connection = hsml.connection()
+connection = hsml.connection(
+    host="c.app.hopsworks.ai", #
+    project="your-project",
+    api_key_value="your-api-key",
+)
 
-# Get the model registry handle for the project's model registry
 mr = connection.get_model_registry()
-
-# Get the model serving handle for the current model registry
+# or
 ms = connection.get_model_serving()
 ```
 
@@ -96,7 +122,9 @@ data = { "instances": [ model.input_example ] }
 predictions = deployment.predict(data)
 ```
 
-You can find more examples on how to use the library in [examples.hopsworks.ai](https://examples.hopsworks.ai).
+# Tutorials
+
+You can find more examples on how to use the library in our [tutorials](https://github.com/logicalclocks/hopsworks-tutorials).
 
 ## Documentation
 

diff --git a/auto_doc.py b/auto_doc.py
@@ -16,7 +16,7 @@
 
 import pathlib
 import shutil
-
+import os
 import keras_autodoc
 
 JSON_METHODS = [
@@ -183,12 +183,18 @@
 }
 
 hsml_dir = pathlib.Path(__file__).resolve().parents[0]
+if "GITHUB_SHA" in os.environ:
+    commit_sha = os.environ["GITHUB_SHA"]
+    project_url = f"https://github.com/logicalclocks/machine-learning-api/tree/{commit_sha}/python"
+else:
+    branch_name = os.environ.get("GITHUB_BASE_REF", "master")
+    project_url = f"https://github.com/logicalclocks/machine-learning-api/blob/{branch_name}/python"
 
 
 def generate(dest_dir):
     doc_generator = keras_autodoc.DocumentationGenerator(
         PAGES,
-        project_url="https://github.com/logicalclocks/machine-learning-api/blob/master/python",
+        project_url=project_url,
         template_dir="./docs/templates",
         titles_size="###",
         extra_aliases={},

diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md
@@ -12,7 +12,7 @@
     pip install -e ".[dev]"
     ```
 
-- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The Model Registry uses pre-commit to ensure code-style and code formatting through [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8). Run the following commands from the `python` directory:
+- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The Model Registry uses pre-commit to ensure code-style and code formatting through [ruff](https://docs.astral.sh/ruff/). Run the following commands from the `python` directory:
 
     ```bash
     cd python
@@ -22,12 +22,12 @@
 
   Afterwards, pre-commit will run whenever you commit.
 
-- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use black and flake8, or run them via the command line:
+- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use [ruff](https://docs.astral.sh/ruff/tutorial/#getting-started):
 
     ```bash
     cd python
-    flake8 hsml
-    black hsml
+    ruff check --fix
+    ruff format
     ```
 
 ### Python documentation
@@ -74,7 +74,7 @@ We use `mkdocs` together with `mike` ([for versioning](https://github.com/jimpor
 1. Currently we are using our own version of `keras-autodoc`
 
     ```bash
-    pip install git+https://github.com/logicalclocks/keras-autodoc@split-tags-properties
+    pip install git+https://github.com/logicalclocks/keras-autodoc
     ```
 
 2. Install HSML with `docs` extras:

diff --git a/docs/css/custom.css b/docs/css/custom.css
@@ -1,4 +1,4 @@
-:root {
+[data-md-color-scheme="hopsworks"] {
   --md-primary-fg-color: #1EB382;
   --md-secondary-fg-color: #188a64;
   --md-tertiary-fg-color: #0d493550;
@@ -24,6 +24,10 @@
   box-shadow: 0 0 0 0;
 }
 
+.md-tabs__item {
+  min-width: 2.25rem;
+}
+
 .md-tabs__item:hover {
   background-color: var(--md-tertiary-fg-color);
   transition: background-color 450ms;