-
Notifications
You must be signed in to change notification settings - Fork 121
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Updates for agents integrations docs (#1498)
* split bq docs into agents and plugins docs Signed-off-by: nikki everett <[email protected]> * update docs for plugins with a corresponding agent Signed-off-by: nikki everett <[email protected]> * update top-level toctree, redirects, and integrations docs landing page for new agents Signed-off-by: nikki everett <[email protected]> * more updates for bq integration Signed-off-by: nikki everett <[email protected]> * more updates for databricks integration Signed-off-by: nikki everett <[email protected]> * copyedits for databricks plugin page Signed-off-by: nikki everett <[email protected]> * lots of cleanup Signed-off-by: nikki everett <[email protected]> * add airflow agent to integrations Signed-off-by: nikki everett <[email protected]> * update integrations landing page Signed-off-by: nikki everett <[email protected]> * more updates for bq and mmcloud agents Signed-off-by: nikki everett <[email protected]> * formatting fixes Signed-off-by: nikki everett <[email protected]> * update toctrees and airflow agents example Signed-off-by: nikki everett <[email protected]> * formatting Signed-off-by: nikki everett <[email protected]> * updates for mmcloud agent Signed-off-by: nikki everett <[email protected]> * lots of cleanup Signed-off-by: nikki everett <[email protected]> * lots of cleanup Signed-off-by: nikki everett <[email protected]> * update note text Signed-off-by: nikki everett <[email protected]> * add note Signed-off-by: nikki everett <[email protected]> * add anchor Signed-off-by: nikki everett <[email protected]> * update links Signed-off-by: nikki everett <[email protected]> * add README Signed-off-by: nikki everett <[email protected]> * add link to agents guide Signed-off-by: nikki everett <[email protected]> * add general info on local testing Signed-off-by: nikki everett <[email protected]> * update sandbox ephemeral storage to fix failing test Signed-off-by: nikki everett <[email protected]> * update correct storage param Signed-off-by: nikki everett <[email protected]> * small edit Signed-off-by: nikki everett <[email protected]> * try bumping flytekit version Signed-off-by: nikki everett <[email protected]> * try removing all but flytekit version requirement Signed-off-by: nikki everett <[email protected]> * remove flytekit version Signed-off-by: nikki everett <[email protected]> * reset mnist classifier imagespec and requirements so others can debug Signed-off-by: nikki everett <[email protected]> * typo Signed-off-by: nikki everett <[email protected]> --------- Signed-off-by: nikki everett <[email protected]>
- Loading branch information
Showing
43 changed files
with
1,457 additions
and
136 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| (airflow_agent)= | ||
|
|
||
| # Airflow agent | ||
|
|
||
| ```{note} | ||
| The Airflow agent does not support all [Airflow operators](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html). We have tested many, but if you run into issues, please [file a bug report](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&projects=&template=bug_report.yaml&title=%5BBUG%5D+). | ||
| ``` | ||
|
|
||
| ## Installation | ||
|
|
||
| To install the plugin, run the following command: | ||
|
|
||
| `pip install flytekitplugins-airflow` | ||
|
|
||
| ## Example usage | ||
|
|
||
| ```{note} | ||
| You don't need an Airflow cluster to run Airflow tasks, since Flytekit will | ||
| automatically compile Airflow tasks to Flyte tasks and execute them on the Flyte cluster. | ||
| ``` | ||
|
|
||
| For a usage example, see the {doc}`Airflow agent example <airflow_agent_example>` page. | ||
|
|
||
| ## Local testing | ||
|
|
||
| To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. For more information, see "[Testing agents locally](https://docs.flyte.org/en/latest/flyte_agents/testing_agents_locally.html)". | ||
|
|
||
| ```{note} | ||
| In some cases, you will need to store credentials in your local environment when testing locally. | ||
| ``` | ||
|
|
||
| ## Flyte deployment configuration | ||
|
|
||
| ```{note} | ||
| If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. | ||
| ``` | ||
|
|
||
| To enable the Airflow agent in your Flyte deployment, see the {ref}`Airflow agent deployment guide<deployment-agent-setup-airflow>`. | ||
|
|
||
| ```{toctree} | ||
| :maxdepth: -1 | ||
| :hidden: | ||
| airflow_agent_example | ||
| ``` |
File renamed without changes.
23 changes: 23 additions & 0 deletions
23
examples/airflow_agent/airflow_agent/airflow_agent_example.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # %% [markdown] | ||
| # (airflow_agent_example)= | ||
| # # Airflow agent example | ||
| # | ||
| # %% | ||
|
|
||
| from airflow.sensors.filesystem import FileSensor | ||
| from flytekit import task, workflow | ||
|
|
||
|
|
||
| @task() | ||
| def t1(): | ||
| print("flyte") | ||
|
|
||
|
|
||
| @workflow | ||
| def wf(): | ||
| sensor = FileSensor(task_id="id", filepath="/tmp/1234") | ||
| sensor >> t1() | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| wf() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| apache-airflow-providers-apache-beam[google] | ||
| apache-airflow[google] |
Oops, something went wrong.