New agents guide and deployment config docs updates (#4822)

* small update to start PR Signed-off-by: nikki everett <[email protected]> * add airflow and snowflake agent pages and update existing agents docs Signed-off-by: nikki everett <[email protected]> * small updates Signed-off-by: nikki everett <[email protected]> * [Doc] Airflow agent deployment (#4831) Signed-off-by: Kevin Su <[email protected]> * [Doc] Snowflake agent deployment (#4832) Signed-off-by: Kevin Su <[email protected]> * copyedits Signed-off-by: nikki everett <[email protected]> * move agents guide to flyte Signed-off-by: nikki everett <[email protected]> * rename directory and add guide to toctree Signed-off-by: nikki everett <[email protected]> * stub out async vs sync agents content Signed-off-by: nikki everett <[email protected]> * link to list of agents Signed-off-by: nikki everett <[email protected]> * edits for sync vs async agents Signed-off-by: nikki everett <[email protected]> * add visible TOC table to index page Signed-off-by: nikki everett <[email protected]> * update for async vs sync tasks Signed-off-by: nikki everett <[email protected]> * Update async agent example (#4906) * Add sync agent example Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * formatting and copy updates Signed-off-by: nikki everett <[email protected]> * fix formatting in snowflake doc, small fixes in agents procedural docs Signed-off-by: nikki everett <[email protected]> * fix toctree Signed-off-by: nikki everett <[email protected]> * add link Signed-off-by: nikki everett <[email protected]> * fix links Signed-off-by: nikki everett <[email protected]> * update link Signed-off-by: nikki everett <[email protected]> * import blog content Signed-off-by: nikki everett <[email protected]> * rename agent creation doc, small content updates Signed-off-by: nikki everett <[email protected]> * fix article title Signed-off-by: nikki everett <[email protected]> * fix link Signed-off-by: nikki everett <[email protected]> * undo unnecessary formatting change in unrelated file Signed-off-by: nikki everett <[email protected]> --------- Signed-off-by: nikki everett <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]>
flyteorg · Feb 23, 2024 · 5f92f31 · 5f92f31
1 parent cf992f5
commit 5f92f31
Show file tree

Hide file tree

Showing 12 changed files with 426 additions and 19 deletions.
diff --git a/docs/deployment/agents/airflow.rst b/docs/deployment/agents/airflow.rst
@@ -0,0 +1,97 @@
+.. _deployment-agent-setup-airflow:
+
+Airflow agent
+=================
+
+This guide provides an overview of how to set up the Airflow agent in your Flyte deployment.
+Please note that the you don't need an Airflow cluster to run the Airflow tasks, since Flytekit will
+automatically compile Airflow tasks to Flyte tasks and execute them on the Flyte cluster.
+
+Specify agent configuration
+----------------------------
+
+.. tabs::
+
+    .. group-tab:: Flyte binary
+
+      Edit the relevant YAML file to specify the agent.
+
+      .. code-block:: bash
+
+        kubectl edit configmap flyte-sandbox-config -n flyte
+
+      .. code-block:: yaml
+        :emphasize-lines: 7,11,16
+
+        tasks:
+          task-plugins:
+            enabled-plugins:
+              - container
+              - sidecar
+              - k8s-array
+              - agent-service
+            default-for-task-types:
+              - container: container
+              - container_array: k8s-array
+              - airflow: agent-service
+
+        plugins:
+          agent-service:
+            supportedTaskTypes:
+            - airflow
+
+    .. group-tab:: Flyte core
+
+      Create a file named ``values-override.yaml`` and add the following configuration to it.
+
+      .. code-block:: yaml
+
+        configmap:
+          enabled_plugins:
+            # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
+            tasks:
+              # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
+              task-plugins:
+                # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
+                enabled-plugins:
+                  - container
+                  - sidecar
+                  - k8s-array
+                  - agent-service
+                default-for-task-types:
+                  container: container
+                  sidecar: sidecar
+                  container_array: k8s-array
+                  airflow: agent-service
+            plugins:
+              agent-service:
+                supportedTaskTypes:
+                - airflow
+
+
+Upgrade the Flyte Helm release
+------------------------------
+
+.. tabs::
+
+  .. group-tab:: Flyte binary
+
+    .. code-block:: bash
+
+      helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>
+
+    Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte-backend``),
+    ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``),
+    and ``<YOUR_YAML_FILE>`` with the name of your YAML file.
+
+  .. group-tab:: Flyte core
+
+    .. code-block:: bash
+
+      helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml
+
+    Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)
+
+    and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).
+
+For Airflow agent on the Flyte cluster, see `Airflow agent <https://docs.flyte.org/en/latest/flytesnacks/examples/airflow_agent/index.html>`_.
diff --git a/docs/deployment/agents/bigquery.rst b/docs/deployment/agents/bigquery.rst
@@ -1,6 +1,6 @@
 .. _deployment-agent-setup-bigquery:
 
-Google BigQuery Agent
+Google BigQuery agent
 ======================
 
 This guide provides an overview of setting up BigQuery agent in your Flyte deployment.
@@ -103,4 +103,4 @@ Upgrade the Flyte Helm release
 
     and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).
 
-For BigQuery plugin on the Flyte cluster, please refer to `BigQuery Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_plugin/bigquery.html>`_
+For BigQuery agent on the Flyte cluster, see `BigQuery agent <https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_integration/bigquery_agent.html>`_.
diff --git a/docs/deployment/agents/databricks.rst b/docs/deployment/agents/databricks.rst
@@ -1,6 +1,6 @@
 .. _deployment-agent-setup-databricks:
 
-Databricks Agent
+Databricks agent
 =================
 
 This guide provides an overview of how to set up Databricks agent in your Flyte deployment.
@@ -291,4 +291,4 @@ Wait for the upgrade to complete. You can check the status of the deployment pod
 
   kubectl get pods -n flyte
 
-For databricks plugin on the Flyte cluster, please refer to `Databricks Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_plugin/index.html>`_
+For Databricks agent on the Flyte cluster, see `Databricks agent <https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_integration/databricks_agent.html>`_.
diff --git a/docs/deployment/agents/index.md b/docs/deployment/agents/index.md
@@ -2,31 +2,40 @@
 
 # Agent Setup
 
-.. tags:: Agent, Integration, Data, Advanced
+```{tags} Agent, Integration, Data, Advanced
+```
+
+To set configure your Flyte deployment for agents, see the documentation below.
 
-Discover the process of setting up Agents for Flyte.
+:::{note}
+If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment.
+:::
 
 ```{list-table}
 :header-rows: 0
 :widths: 20 30
 
-* - {ref}`Bigquery Agent <deployment-agent-setup-bigquery>`
-  - Guide to setting up the Bigquery agent.
+* - {ref}`Airflow Agent <deployment-agent-setup-airflow>`
+  - Configuring your Flyte deployment for the Airflow agent
+* - {ref}`Databricks Agent <deployment-agent-setup-databricks>`
+  - Configuring your Flyte deployment for the Databricks agent.
+* - {ref}`Google BigQuery Agent <deployment-agent-setup-bigquery>`
+  - Configuring your Flyte deployment for the BigQuery agent.
 * - {ref}`MMCloud Agent <deployment-agent-setup-mmcloud>`
-  - Guide to setting up the MMCloud agent.
+  - Configuring your Flyte deployment for the MMCloud agent.
 * - {ref}`Sensor Agent <deployment-agent-setup-sensor>`
-  - Guide to setting up the Sensor agent.
-* - {ref}`Databricks Agent <deployment-agent-setup-databricks>`
-  - Guide to setting up the Databricks agent.
+  - Configuring your Flyte deployment for the sensor agent.
 ```
 
 ```{toctree}
 :maxdepth: 1
 :name: Agent setup
 :hidden:
 
+airflow
+databricks
 bigquery
 mmcloud
-databricks
 sensor
+snowflake
 ```
diff --git a/docs/deployment/agents/mmcloud.rst b/docs/deployment/agents/mmcloud.rst
@@ -118,4 +118,4 @@ Wait for the upgrade to complete. You can check the status of the deployment pod
 
   kubectl get pods -n flyte
 
-For MMCloud plugin on the Flyte cluster, please refer to `Memory Machine Cloud Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_plugin/index.html>`_
+For MMCloud agent on the Flyte cluster, see `MMCloud agent <https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_integration/mmcloud_agent.html>`_.
diff --git a/docs/deployment/agents/sensor.rst b/docs/deployment/agents/sensor.rst
@@ -1,13 +1,13 @@
 .. _deployment-agent-setup-sensor:
 
-Sensor Agent
+Sensor agent
 =================
 
-Sensor enables users to continuously check for a file or a condition to be met periodically.
+The `sensor agent <https://docs.flyte.org/en/latest/flytesnacks/examples/sensor/index.html>`_ enables users to continuously check for a file or a condition to be met periodically.
 
 When the condition is met, the sensor will complete.
 
-This guide provides an overview of how to set up Sensor in your Flyte deployment.
+This guide provides an overview of how to set up the sensor agent in your Flyte deployment.
 
 Spin up a cluster
 -----------------
@@ -43,7 +43,7 @@ Spin up a cluster
 Specify agent configuration
 ----------------------------
 
-Enable the Sensor agent by adding the following config to the relevant YAML file(s):
+Enable the sensor agent by adding the following config to the relevant YAML file(s):
 
 .. tabs::
 
@@ -77,7 +77,7 @@ Enable the Sensor agent by adding the following config to the relevant YAML file
 
     .. group-tab:: Flyte core
 
-      Create a file named ``values-override.yaml`` and add the following configuration to it.
+      Create a file named ``values-override.yaml`` and add the following configuration to it:
 
       .. code-block:: yaml
 

diff --git a/docs/deployment/agents/snowflake.rst b/docs/deployment/agents/snowflake.rst
@@ -0,0 +1,103 @@
+.. _deployment-agent-setup-snowflake:
+
+Snowflake agent
+=================
+
+This guide provides an overview of how to set up the Snowflake agent in your Flyte deployment.
+
+1. Set up the key pair authentication in Snowflake. For more details, see the `Snowflake key-pair authentication and key-pair rotation guide <https://docs.snowflake.com/en/user-guide/key-pair-auth>`__.
+2. Create a secret with the group "snowflake" and the key "private_key". For more details, see `"Using Secrets in a Task" <https://https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/use_secrets.html>`__.
+
+.. code-block:: bash
+
+   kubectl create secret generic snowflake-private-key --namespace=flytesnacks-development --from-file=your_private_key_above
+
+Specify agent configuration
+----------------------------
+
+.. tabs::
+
+    .. group-tab:: Flyte binary
+
+      Edit the relevant YAML file to specify the agent.
+
+      .. code-block:: bash
+
+        kubectl edit configmap flyte-sandbox-config -n flyte
+
+      .. code-block:: yaml
+        :emphasize-lines: 7,11,16
+
+        tasks:
+          task-plugins:
+            enabled-plugins:
+              - container
+              - sidecar
+              - k8s-array
+              - agent-service
+            default-for-task-types:
+              - container: container
+              - container_array: k8s-array
+              - snowflake: agent-service
+
+        plugins:
+          agent-service:
+            supportedTaskTypes:
+            - snowflake
+
+    .. group-tab:: Flyte core
+
+      Create a file named ``values-override.yaml`` and add the following configuration to it.
+
+      .. code-block:: yaml
+
+        configmap:
+          enabled_plugins:
+            # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
+            tasks:
+              # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
+              task-plugins:
+                # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
+                enabled-plugins:
+                  - container
+                  - sidecar
+                  - k8s-array
+                  - agent-service
+                default-for-task-types:
+                  container: container
+                  sidecar: sidecar
+                  container_array: k8s-array
+                  snowflake: agent-service
+            plugins:
+              agent-service:
+                supportedTaskTypes:
+                - snowflake
+
+Ensure that the propeller has the correct service account for BigQuery.
+
+Upgrade the Flyte Helm release
+------------------------------
+
+.. tabs::
+
+  .. group-tab:: Flyte binary
+
+    .. code-block:: bash
+
+      helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>
+
+    Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte-backend``),
+    ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``),
+    and ``<YOUR_YAML_FILE>`` with the name of your YAML file.
+
+  .. group-tab:: Flyte core
+
+    .. code-block:: bash
+
+      helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml
+
+    Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)
+
+    and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).
+
+For Snowflake agent on the Flyte cluster, see `Snowflake agent <https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_integration/snowflake_agent.html>`_.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -118,4 +118,4 @@ Wait for the upgrade to complete. You can check the status of the deployment pod

		kubectl get pods -n flyte

		For MMCloud plugin on the Flyte cluster, please refer to `Memory Machine Cloud Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_plugin/index.html>`_
		For MMCloud agent on the Flyte cluster, see `MMCloud agent <https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_integration/mmcloud_agent.html>`_.