Skip to content

Commit

Permalink
Address comments
Browse files Browse the repository at this point in the history
  • Loading branch information
javierdlrm committed Mar 19, 2024
1 parent 6cf8170 commit d31dd4d
Show file tree
Hide file tree
Showing 11 changed files with 28 additions and 29 deletions.
11 changes: 5 additions & 6 deletions docs/concepts/mlops/kserve.md → docs/concepts/mlops/serving.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
In Hopsworks, you can easily deploy models from the model registry in KServe or in Docker containers (for Hopsworks Community). KServe is an open-source framework for model serving on Kubernetes. You can deploy models in either programs, using the HSML library, or in the UI. A KServe model deployment can include the following components:
In Hopsworks, you can easily deploy models from the model registry in KServe or in Docker containers (for Hopsworks Community). KServe is the defacto open-source framework for model serving on Kubernetes. You can deploy models in either programs, using the HSML library, or in the UI. A KServe model deployment can include the following components:

**`Transformer`**

Expand All @@ -15,16 +15,15 @@ In Hopsworks, you can easily deploy models from the model registry in KServe or

**`Inference Batcher`**

: Inference requests can be batched in different ways to adjust the trade-off between throughput and latencies of the model predictions.
: Inference requests can be batched to improve throughput (at the cost of slightly higher latency).

**`Istio Model Endpoint`**

: You can publish a model via a ^^REST or gRPC Endpoint^^ using Istio and access it over HTTP using a Hopsworks API key (with serving scope). Secure and authorized access is guaranteed by Hopsworks.
: You can publish a model over ^^REST(HTTP)^^ or ^^gRPC^^ using a Hopsworks API key. API keys have scopes to ensure the principle of least privilege access control to resources managed by Hopsworks.


Models deployed on KServe in Hopsworks can be easily integrated with the Hopsworks Feature Store using a Transformer Python script, that builds the predictor's input feature vector using the application input and pre-computed features from the Feature Store.
Models deployed on KServe in Hopsworks can be easily integrated with the Hopsworks Feature Store using either a Transformer or Predictor Python script, that builds the predictor's input feature vector using the application input and pre-computed features from the Feature Store.

<img src="../../../assets/images/concepts/mlops/kserve.svg">

!!! info "Model Serving Guide"
More information can be found in the [Model Serving guide](../../../user_guides/mlops/serving/index.md).
More information can be found in the [Model Serving guide](../../user_guides/mlops/serving/index.md).
2 changes: 1 addition & 1 deletion docs/concepts/mlops/training.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ If you train models with Hopsworks, you can setup CI/CD pipelines as shown below

<img src="../../../assets/images/concepts/mlops/training-pipeline.svg">

Hopsworks [Model Registry](registry.md) and [Model Serving](kserve.md) capabilities can then be used to build a batch or online prediction service using the model.
Hopsworks [Model Registry](registry.md) and [Model Serving](serving.md) capabilities can then be used to build a batch or online prediction service using the model.
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ pointer-events: initial;
</div>
<div class="frame_with_arrow">
<div class="rec_frame">
<div class="name_item"><a href="./concepts/mlops/kserve/">Model Serving</a></div>
<div class="name_item"><a href="./concepts/mlops/serving/">Model Serving</a></div>
</div>
</div>
</div>
Expand Down
12 changes: 6 additions & 6 deletions docs/user_guides/mlops/serving/api-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
Hopsworks supports both REST and gRPC as API protocols for sending inference requests to model deployments. While REST API protocol is supported in all types of model deployments, support for gRPC is only available for models served with [KServe](predictor.md#serving-tool).

!!! warning
At the moment, gRPC API protocol is only supported for **Python model deployments** (e.g., scikit-learn, xgboost, ...). Support for Tensorflow model deployments is coming soon.
At the moment, the gRPC API protocol is only supported for **Python model deployments** (e.g., scikit-learn, xgboost). Support for Tensorflow model deployments is coming soon.

## GUI

### Step 1: Create new deployment
### Step 1: Create a new deployment

If you have at least one model already trained and saved in the Model Registry, navigate to the deployments page by clicking on the `Deployments` tab on the navigation menu on the left.

Expand All @@ -20,11 +20,11 @@ If you have at least one model already trained and saved in the Model Registry,
</figure>
</p>

Once in the deployments page, click on `New deployment` if there are not existing deployments or on `Create new deployment` at the top-right corner to open the deployment creation form.
Once in the deployments page, you can create a new deployment by either clicking on `New deployment` (if there are no existing deployments) or on `Create new deployment` it the top-right corner. Both options will open the deployment creation form.

### Step 2: Go to advanced options

A simplified creation form will appear including the most common deployment fields among all the configuration possible. Resource allocation is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.
A simplified creation form will appear including the most common deployment fields from all available configurations. Resource allocation is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.

<p align="center">
<figure>
Expand All @@ -46,8 +46,8 @@ Enabling gRPC as the API protocol for a model deployment requires KServe as the

Then, you can select the API protocol to be enabled in your model deployment.

!!! note "Only one API protocol can be enabled simultaneously"
Currently, KServe model deployments are limited to one API protocol at a time. Therefore, only one of REST or gRPC API protocols can be enabled at the same time on the same model deployment.
!!! info "Only one API protocol can be enabled for a model (they cannot support both gRPC and REST)"
Currently, KServe model deployments are limited to one API protocol at a time. Therefore, only one of REST or gRPC API protocols can be enabled at the same time on the same model deployment. You can change the API protocol of existing deployments.

<p align="center">
<figure>
Expand Down
6 changes: 3 additions & 3 deletions docs/user_guides/mlops/serving/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ If you have at least one model already trained and saved in the Model Registry,
</figure>
</p>

Once in the deployments page, click on `New deployment` if there are not existing deployments or on `Create new deployment` at the top-right corner to open the deployment creation form.
Once in the deployments page, you can create a new deployment by either clicking on `New deployment` (if there are no existing deployments) or on `Create new deployment` it the top-right corner. Both options will open the deployment creation form.

### Step 2: Basic deployment configuration

A simplified creation form will appear including the most common deployment fields among all the configuration possible. We provide default values for the rest of the fields, adjusted to the type of deployment you want to create.
A simplified creation form will appear including the most common deployment fields from all available configurations. We provide default values for the rest of the fields, adjusted to the type of deployment you want to create.

In the simplified form, select the model framework used to train your model (i.e., _TensorFlow Serving_ or _Python_). Then, select the model you want to deploy from the list of available models under `pick a model`.

Expand Down Expand Up @@ -195,7 +195,7 @@ Resources include the number of replicas for the deployment as well as the resou

## API protocol

Hopsworks supports both REST and gRPC as the API protocols to send inference requests to model deployments. To learn about REST and gRPC API protocols for model deployments, see the [API Protocol Guide](api-protocol.md).
Hopsworks supports both REST and gRPC as the API protocols to send inference requests to model deployments. In general, you use gRPC when you need lower latency inference requests. To learn more about the REST and gRPC API protocols for model deployments, see the [API Protocol Guide](api-protocol.md).

## Conclusion

Expand Down
4 changes: 2 additions & 2 deletions docs/user_guides/mlops/serving/inference-batcher.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ If you have at least one model already trained and saved in the Model Registry,
</figure>
</p>

Once in the deployments page, click on `New deployment` if there are not existing deployments or on `Create new deployment` at the top-right corner to open the deployment creation form.
Once in the deployments page, you can create a new deployment by either clicking on `New deployment` (if there are no existing deployments) or on `Create new deployment` it the top-right corner. Both options will open the deployment creation form.

### Step 2: Go to advanced options

A simplified creation form will appear including the most common deployment fields among all the configuration possible. Inference batching is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.
A simplified creation form will appear including the most common deployment fields from all available configurations. Inference batching is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.

<p align="center">
<figure>
Expand Down
4 changes: 2 additions & 2 deletions docs/user_guides/mlops/serving/inference-logger.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ If you have at least one model already trained and saved in the Model Registry,
</figure>
</p>

Once in the deployments page, click on `New deployment` if there are not existing deployments or on `Create new deployment` at the top-right corner to open the deployment creation form.
Once in the deployments page, you can create a new deployment by either clicking on `New deployment` (if there are no existing deployments) or on `Create new deployment` it the top-right corner. Both options will open the deployment creation form.

### Step 2: Go to advanced options

A simplified creation form will appear including the most common deployment fields among all the configuration possible. Inference logging is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.
A simplified creation form will appear including the most common deployment fields from all available configurations. Inference logging is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.

<p align="center">
<figure>
Expand Down
6 changes: 3 additions & 3 deletions docs/user_guides/mlops/serving/predictor.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ If you have at least one model already trained and saved in the Model Registry,
</figure>
</p>

Once in the deployments page, click on `New deployment` if there are not existing deployments or on `Create new deployment` at the top-right corner to open the deployment creation form.
Once in the deployments page, you can create a new deployment by either clicking on `New deployment` (if there are no existing deployments) or on `Create new deployment` it the top-right corner. Both options will open the deployment creation form.

### Step 2: Choose a model server

A simplified creation form will appear, including the most common deployment fields among all the configuration possible. These fields include the [model server](#model-server) and [custom script](#custom-script) (for python models).
A simplified creation form will appear, including the most common deployment fields from all available configurations. These fields include the [model server](#model-server) and [custom script](#custom-script) (for python models).

<p align="center">
<figure>
Expand Down Expand Up @@ -250,7 +250,7 @@ Resources include the number of replicas for the deployment as well as the resou

## API protocol

Hopsworks supports both REST and gRPC as the API protocols to send inference requests to model deployments. To learn about REST and gRPC API protocols for model deployments, see the [API Protocol Guide](api-protocol.md).
Hopsworks supports both REST and gRPC as the API protocols to send inference requests to model deployments. In general, you use gRPC when you need lower latency inference requests. To learn more about the REST and gRPC API protocols for model deployments, see the [API Protocol Guide](api-protocol.md).

## Conclusion

Expand Down
4 changes: 2 additions & 2 deletions docs/user_guides/mlops/serving/resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ If you have at least one model already trained and saved in the Model Registry,
</figure>
</p>

Once in the deployments page, click on `New deployment` if there are not existing deployments or on `Create new deployment` at the top-right corner to open the deployment creation form.
Once in the deployments page, you can create a new deployment by either clicking on `New deployment` (if there are no existing deployments) or on `Create new deployment` it the top-right corner. Both options will open the deployment creation form.

### Step 2: Go to advanced options

A simplified creation form will appear including the most common deployment fields among all the configuration possible. Resource allocation is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.
A simplified creation form will appear including the most common deployment fields from all available configurations. Resource allocation is part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.

<p align="center">
<figure>
Expand Down
4 changes: 2 additions & 2 deletions docs/user_guides/mlops/serving/transformer.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ If you have at least one model already trained and saved in the Model Registry,
</figure>
</p>

Once in the deployments page, click on `New deployment` if there are not existing deployments or on `Create new deployment` at the top-right corner to open the deployment creation form.
Once in the deployments page, you can create a new deployment by either clicking on `New deployment` (if there are no existing deployments) or on `Create new deployment` it the top-right corner. Both options will open the deployment creation form.

### Step 2: Go to advanced options

A simplified creation form will appear including the most common deployment fields among all the configuration possible. Transformers are part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.
A simplified creation form will appear including the most common deployment fields from all available configurations. Transformers are part of the advanced options of a deployment. To navigate to the advanced creation form, click on `Advanced options`.

<p align="center">
<figure>
Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ nav:
- Prediction Services: concepts/mlops/prediction_services.md
- Model Training: concepts/mlops/training.md
- Model Registry: concepts/mlops/registry.md
- Model Serving: concepts/mlops/kserve.md # create, read, version
- Model Serving: concepts/mlops/serving.md
- Vector Database: concepts/mlops/opensearch.md
- BI Tools: concepts/mlops/bi_tools.md
- Development:
Expand Down

0 comments on commit d31dd4d

Please sign in to comment.