Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake agent Doc #5620

Merged
merged 4 commits into from
Aug 2, 2024
Merged

Snowflake agent Doc #5620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
89b4e36
TEST build
Future-Outlier Aug 2, 2024
bb0a935
remove emphasize-lines
Future-Outlier Aug 2, 2024
e792c12
test build
Future-Outlier Aug 2, 2024
dad9784
revert
Future-Outlier Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/deployment/agents/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,12 @@ If you are using a managed deployment of Flyte, you will need to contact your de
- Configuring your Flyte deployment for the BigQuery agent.
* - {ref}`MMCloud Agent <deployment-agent-setup-mmcloud>`
- Configuring your Flyte deployment for the MMCloud agent.
* - {ref}`Sensor Agent <deployment-agent-setup-sensor>`
- Configuring your Flyte deployment for the sensor agent.
* - {ref}`SageMaker Inference <deployment-agent-setup-sagemaker-inference>`
- Deploy models and create, as well as trigger inference endpoints on SageMaker.
* - {ref}`Sensor Agent <deployment-agent-setup-sensor>`
- Configuring your Flyte deployment for the sensor agent.
* - {ref}`Snowflake Agent <deployment-agent-setup-snowflake>`
- Configuring your Flyte deployment for the SnowFlake agent.
* - {ref}`OpenAI Batch <deployment-agent-setup-openai-batch>`
- Submit requests to OpenAI GPT models for asynchronous batch processing.
```
Expand Down
18 changes: 13 additions & 5 deletions docs/deployment/agents/snowflake.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,25 @@
.. _deployment-agent-setup-snowflake:

Snowflake agent
=================
===============

This guide provides an overview of how to set up the Snowflake agent in your Flyte deployment.

1. Set up the key pair authentication in Snowflake. For more details, see the `Snowflake key-pair authentication and key-pair rotation guide <https://docs.snowflake.com/en/user-guide/key-pair-auth>`__.
2. Create a secret with the group "snowflake" and the key "private_key". For more details, see `"Using Secrets in a Task" <https://https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/use_secrets.html>`__.
2. Create a secret with the group "private_key" and the key "snowflake".
This is hardcoded in the flytekit sdk, since we can't know the group and key name in advance.
This is for permission to upload and download data with structured dataset in python task pod.

.. code-block:: bash

kubectl create secret generic snowflake-private-key --namespace=flytesnacks-development --from-file=your_private_key_above
kubectl create secret generic private-key --from-file=snowflake=<YOUR PRIVATE KEY FILE> --namespace=flytesnacks-development

3. Create a secret in the flyteagent's pod, this is for execution snowflake query in the agent pod.

.. code-block:: bash

ENCODED_VALUE=$(cat <YOUR PRIVATE KEY FILE> | base64) && kubectl patch secret flyteagent -n flyte --patch "{\"data\":{\"snowflake_private_key\":\"$ENCODED_VALUE\"}}"


Specify agent configuration
----------------------------
Expand Down Expand Up @@ -73,7 +82,7 @@ Specify agent configuration
supportedTaskTypes:
- snowflake

Ensure that the propeller has the correct service account for BigQuery.
Ensure that the propeller has the correct service account for Snowflake.

Upgrade the Flyte Helm release
------------------------------
Expand All @@ -97,7 +106,6 @@ Upgrade the Flyte Helm release
helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml

Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)

and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).

For Snowflake agent on the Flyte cluster, see `Snowflake agent <https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_agent/index.html>`_.
Loading