Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates for agents integrations docs #1498

Merged
merged 32 commits into from
Feb 23, 2024
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
896a8bc
split bq docs into agents and plugins docs
Feb 14, 2024
acec91b
update docs for plugins with a corresponding agent
Feb 15, 2024
be6806e
update top-level toctree, redirects, and integrations docs landing pa…
Feb 15, 2024
865942c
more updates for bq integration
Feb 15, 2024
37302ed
more updates for databricks integration
Feb 15, 2024
daecbbe
copyedits for databricks plugin page
Feb 15, 2024
9cfd826
lots of cleanup
Feb 15, 2024
1b6a08c
add airflow agent to integrations
Feb 16, 2024
b180562
update integrations landing page
Feb 16, 2024
d62fdb9
more updates for bq and mmcloud agents
Feb 16, 2024
fe1d0b7
formatting fixes
Feb 16, 2024
aa047a9
update toctrees and airflow agents example
Feb 16, 2024
0717ed4
formatting
Feb 16, 2024
2a7c3ee
updates for mmcloud agent
Feb 16, 2024
b2a4c85
lots of cleanup
Feb 18, 2024
8d93290
lots of cleanup
Feb 19, 2024
8907c6c
update note text
Feb 19, 2024
d06b762
add note
Feb 19, 2024
099cdef
add anchor
Feb 19, 2024
9846ac2
merge master
Feb 19, 2024
b26aefa
update links
Feb 20, 2024
b920bd1
add README
Feb 20, 2024
a2d424c
add link to agents guide
Feb 20, 2024
51ca64a
add general info on local testing
Feb 20, 2024
65706e2
update sandbox ephemeral storage to fix failing test
Feb 21, 2024
667d9d5
update correct storage param
Feb 21, 2024
6d3e475
small edit
Feb 22, 2024
bdc9d7e
try bumping flytekit version
Feb 22, 2024
8b1b9e2
try removing all but flytekit version requirement
Feb 22, 2024
cb388a9
remove flytekit version
Feb 22, 2024
8db0bae
reset mnist classifier imagespec and requirements so others can debug
Feb 22, 2024
35a8676
typo
Feb 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -231,8 +231,8 @@
"auto_examples/basics/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/basics/index.html",
"auto_examples/basics/task.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/basics/task.html",
"auto_examples/basics/shell_task.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/basics/shell_task.html",
"auto_examples/bigquery_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_plugin/index.html",
"auto_examples/bigquery_plugin/bigquery.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_plugin/bigquery.html",
"auto_examples/bigquery_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_integration/index.html",
"auto_examples/bigquery_plugin/bigquery.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_integration/bigquery.html",
"auto_examples/blast/blastx_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/blast/blastx_example.html",
"auto_examples/blast/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/blast/index.html",
"auto_examples/customizing_dependencies/image_spec.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/customizing_dependencies/image_spec.html",
Expand All @@ -248,8 +248,8 @@
"auto_examples/data_types_and_io/dataclass.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/dataclass.html",
"auto_examples/data_types_and_io/folder.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/folder.html",
"auto_examples/data_types_and_io/pickle_type.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/pickle_type.html",
"auto_examples/databricks_plugin/databricks_job.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_plugin/databricks_job.html",
"auto_examples/databricks_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_plugin/index.html",
"auto_examples/databricks_plugin/databricks_job.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_integration/databricks_job.html",
"auto_examples/databricks_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_integration/index.html",
"auto_examples/dbt_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/dbt_plugin/index.html",
"auto_examples/dbt_plugin/dbt_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/dbt_plugin/dbt_example.html",
"auto_examples/development_lifecycle/inspecting_executions.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/inspecting_executions.html",
Expand Down Expand Up @@ -317,8 +317,8 @@
"auto_examples/kftensorflow_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/kftensorflow_plugin/index.html",
"auto_examples/mlflow_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mlflow_plugin/index.html",
"auto_examples/mlflow_plugin/mlflow_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mlflow_plugin/mlflow_example.html",
"auto_examples/mmcloud_plugin/example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_plugin/example.html",
"auto_examples/mmcloud_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_plugin/index.html",
"auto_examples/mmcloud_plugin/example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_integration/example.html",
"auto_examples/mmcloud_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_integration/index.html",
"auto_examples/mnist_classifier/pytorch_single_node_and_gpu.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/pytorch_single_node_and_gpu.html",
"auto_examples/mnist_classifier/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/index.html",
"auto_examples/mnist_classifier/pytorch_single_node_multi_gpu.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/pytorch_single_node_multi_gpu.html",
Expand Down Expand Up @@ -358,8 +358,8 @@
"auto_examples/sagemaker_training_plugin/sagemaker_custom_training.htmlhttps://docs.flyte.org/en/latest/": "flytesnacks/examples/sagemaker_training_plugin/sagemaker_custom_training.html",
"auto_examples/sensor/file_sensor_example.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sensor/file_sensor_example.html",
"auto_examples/sensor/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sensor/index.html",
"auto_examples/snowflake_plugin/snowflake.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_plugin/snowflake.html",
"auto_examples/snowflake_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_plugin/index.html",
"auto_examples/snowflake_plugin/snowflake.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_integration/snowflake.html",
"auto_examples/snowflake_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_integration/index.html",
"auto_examples/sql_plugin/sql_alchemy.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sql_plugin/sql_alchemy.html",
"auto_examples/sql_plugin/index.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sql_plugin/index.html",
"auto_examples/sql_plugin/sqlite3_integration.html": "https://docs.flyte.org/en/latest/flytesnacks/examples/sql_plugin/sqlite3_integration.html",
Expand Down
9 changes: 5 additions & 4 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,14 +98,15 @@ flyte_lab
:hidden:

Integrations <integrations>
auto_examples/airflow_agent/index
auto_examples/airflow_plugin/index
auto_examples/athena_plugin/index
auto_examples/aws_batch_plugin/index
auto_examples/sagemaker_pytorch_plugin/index
auto_examples/sagemaker_training_plugin/index
auto_examples/bigquery_plugin/index
auto_examples/bigquery_integration/index
auto_examples/k8s_dask_plugin/index
auto_examples/databricks_plugin/index
auto_examples/databricks_integration/index
auto_examples/dbt_plugin/index
auto_examples/dolt_plugin/index
auto_examples/duckdb_plugin/index
Expand All @@ -114,7 +115,7 @@ auto_examples/greatexpectations_plugin/index
auto_examples/hive_plugin/index
auto_examples/k8s_pod_plugin/index
auto_examples/mlflow_plugin/index
auto_examples/mmcloud_plugin/index
auto_examples/mmcloud_integration/index
auto_examples/modin_plugin/index
auto_examples/kfmpi_plugin/index
auto_examples/onnx_plugin/index
Expand All @@ -123,7 +124,7 @@ auto_examples/pandera_plugin/index
auto_examples/kfpytorch_plugin/index
auto_examples/ray_plugin/index
auto_examples/sensor/index
auto_examples/snowflake_plugin/index
auto_examples/snowflake_integration/index
auto_examples/k8s_spark_plugin/index
auto_examples/sql_plugin/index
auto_examples/kftensorflow_plugin/index
Expand Down
46 changes: 34 additions & 12 deletions docs/integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,35 @@ orchestrated by Flyte itself, within its provisioned Kubernetes clusters.
- Run Ray jobs on a K8s Cluster.
```

(flyte_agents)=

## Flyte agents

[Flyte agents](https://docs.flyte.org/en/latest/flyte_agents/index.html) are long-running, stateless services that receive execution requests via gRPC and initiate jobs with appropriate external or internal services. Each agent service is a Kubernetes deployment that receives gRPC requests from FlytePropeller when users trigger a particular type of task. (For example, the BigQuery agent handles BigQuery tasks.) The agent service then initiates a job with the appropriate service. If you don't see the agent you need below, see "[Developing agents](https://docs.flyte.org/en/latest/flyte_agents/developing_agents.html)" to learn how to develop a new agent.

```{list-table}
:header-rows: 0
:widths: 20 30

* - {ref}`Airflow agent <airflow_agent>`
- Run Airflow jobs in your workflows with the Airflow agent.
* - {ref}`BigQuery agent <bigquery_agent>`
- Run BigQuery jobs in your workflows with the BigQuery agent.
* - {ref}`Databricks <databricks_agent>`
- Run Databricks jobs in your workflows with the Databricks agent.
* - {ref}`Memory Machine Cloud <mmcloud_agent>`
- Execute tasks using the MemVerge Memory Machine Cloud agent.
* - {doc}`Sensor <auto_examples/sensor/index>`
- Run sensor jobs in your workflows with the sensor agent.
* - {ref}`Snowflake <snowflake_agent>`
- Run Snowflake jobs in your workflows with the Snowflake agent.
```

(external_service_backend_plugins)=

## External Service Backend Plugins

As the term suggests, external service backend plugins relies on external services like
As the term suggests, external service backend plugins rely on external services like
[AWS Sagemaker](https://aws.amazon.com/sagemaker),
[Hive](https://docs.qubole.com/en/latest/user-guide/engines/hive/index.html) or
[Snowflake](https://www.snowflake.com/) for handling the workload defined in
Expand All @@ -94,27 +118,25 @@ the Flyte task that use the respective plugin.
:header-rows: 0
:widths: 20 30

* - {doc}`AWS Sagemaker: Model Training <auto_examples/sagemaker_training_plugin/index>`
* - {doc}`AWS Sagemaker: Model Training plugin <auto_examples/sagemaker_training_plugin/index>`
- Train models with built-in or define your own custom algorithms.
* - {doc}`AWS Sagemaker: Pytorch Training <auto_examples/sagemaker_pytorch_plugin/index>`
* - {doc}`AWS Sagemaker: Pytorch Training plugin <auto_examples/sagemaker_pytorch_plugin/index>`
- Train Pytorch models using Sagemaker, with support for distributed training.
* - {doc}`AWS Athena <auto_examples/athena_plugin/index>`
* - {doc}`AWS Athena plugin <auto_examples/athena_plugin/index>`
- Execute queries using AWS Athena
* - {doc}`AWS Batch <auto_examples/aws_batch_plugin/index>`
* - {doc}`AWS Batch plugin <auto_examples/aws_batch_plugin/index>`
- Running tasks and workflows on AWS batch service
* - {doc}`Flyte Interactive <auto_examples/flyteinteractive_plugin/index>`
- Execute tasks using Flyte Interactive to debug.
* - {doc}`Hive <auto_examples/hive_plugin/index>`
* - {doc}`Hive plugin <auto_examples/hive_plugin/index>`
- Run Hive jobs in your workflows.
* - {doc}`MMCloud <auto_examples/mmcloud_plugin/index>`
* - {ref}`MMCloud plugin <mmcloud_plugin>`
- Execute tasks using MemVerge Memory Machine Cloud
* - {doc}`Sensor <auto_examples/sensor/index>`
- Run Sensor jobs in your workflows.
* - {doc}`Snowflake <auto_examples/snowflake_plugin/index>`
* - {ref}`Snowflake <snowflake_plugin>`
- Run Snowflake jobs in your workflows.
* - {doc}`Databricks <auto_examples/databricks_plugin/index>`
* - {ref}`Databricks <databricks_plugin>`
- Run Databricks jobs in your workflows.
* - {doc}`BigQuery <auto_examples/bigquery_plugin/index>`
* - {ref}`BigQuery <bigquery_plugin>`
- Run BigQuery jobs in your workflows.
```

Expand Down
48 changes: 48 additions & 0 deletions examples/airflow_agent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
(airflow_agent)=

# Airflow agent

```{note}
The Airflow agent does not support all [Airflow operators](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html). We have tested many, but if you run into issues, please [file a bug report](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&projects=&template=bug_report.yaml&title=%5BBUG%5D+).
```

## Installation

To install the plugin, run the following command:

`pip install flytekitplugins-airflow`

## Example usage

```{note}

You don't need an Airflow cluster to run Airflow tasks, since Flytekit will
automatically compile Airflow tasks to Flyte tasks and execute them on the Flyte cluster.

```

For a usage example, see the {doc}`Airflow agent example <airflow_agent_example>` page.

## Local testing

To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_locally.html)".

```{note}

In some cases, you will need to store credentials in your local environment when testing locally.

```

## Flyte deployment configuration

```{note}
If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment.
```

To enable the Airflow agent in your Flyte deployment, see the {ref}`Airflow agent deployment guide<deployment-agent-setup-airflow>`.

```{toctree}
:maxdepth: -1
:hidden:
airflow_agent_example
```
23 changes: 23 additions & 0 deletions examples/airflow_agent/airflow_agent/airflow_agent_example.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# %% [markdown]
# (airflow_agent_example)=
# # Airflow agent example
#
# %%

from airflow.sensors.filesystem import FileSensor
from flytekit import task, workflow


@task()
def t1():
print("flyte")


@workflow
def wf():
sensor = FileSensor(task_id="id", filepath="/tmp/1234")
sensor >> t1()


if __name__ == "__main__":
wf()
2 changes: 2 additions & 0 deletions examples/airflow_agent/airflow_agent/requirements.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
apache-airflow-providers-apache-beam[google]
apache-airflow[google]
Loading
Loading