Skip to content

Commit

Permalink
update words
Browse files Browse the repository at this point in the history
Signed-off-by: Yee Hing Tong <[email protected]>
  • Loading branch information
wild-endeavor committed Dec 10, 2024
1 parent 48000bf commit 75a9243
Showing 1 changed file with 22 additions and 47 deletions.
69 changes: 22 additions & 47 deletions docs/user_guide/advanced_composition/eager_workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,60 +210,38 @@ developing your workflows and tasks.

(eager_workflows_remote)=

### Remote Flyte cluster execution
### Setting up remote Flyte cluster access

Under the hood, `@eager` workflows use the {py:class}`~flytekit.remote.remote.FlyteRemote`
object to kick off task, static workflow, and eager workflow executions.

In order to actually execute them on a Flyte cluster, you'll need to configure
eager workflows with a `FlyteRemote` object and secrets configuration that
allows you to authenticate into the cluster via a client secret key:

```{code-block} python
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config
@eager(
remote=FlyteRemote(
config=Config.auto(config_file="config.yaml"),
default_project="flytesnacks",
default_domain="development",
),
client_secret_group="<my_client_secret_group>",
client_secret_key="<my_client_secret_key>",
)
async def eager_workflow_remote(x: int) -> int:
...
object to kick off task, static workflow, and eager workflow executions. In order to create a `FlyteRemote` instance, `Config.auto()` is run and the resulting config object is passed into `FlyteRemote`.

This means that you just need to ensure eager workflow pods are run with the environment variables that including any mounted secrets for a client secret. For instance, the following three are sufficient in the most basic setup.

```{code-block} bash
FLYTE_PLATFORM_URL
FLYTE_CREDENTIALS_CLIENT_ID
FLYTE_CREDENTIALS_CLIENT_SECRET
```

Where `config.yaml` contains a
[flytectl](https://docs.flyte.org/projects/flytectl/en/latest/#configuration)-compatible
config file and `my_client_secret_group` and `my_client_secret_key` are the
{ref}`secret group and key <secrets>` that you've configured for your Flyte
cluster to authenticate via a client key.
See the relevant authentication docs for creating client credentials if using Flyte's internal authorization server.

### Sandbox Flyte cluster execution

When using a sandbox cluster started with `flytectl demo start`, however, the
`client_secret_group` and `client_secret_key` are not required, since the
default sandbox configuration does not require key-based authentication.

```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py
:caption: advanced_composition/eager_workflows.py
:lines: 130-145
```
When using a sandbox cluster started with `flytectl demo start` no authentication is needed and eager workflows should
just work out of the box.

```{important}
When executing eager workflows on a remote Flyte cluster, it will execute the
latest version of tasks, static workflows, and eager workflows that are on
the `default_project` and `default_domain` as specified in the `FlyteRemote`
object. This means that you need to pre-register all Flyte entities that are
invoked inside of the eager workflow.
same version of tasks, static workflows, and eager workflows that are on
the `project` and `domain` as the eager task itself. If an entity is not found, FlyteRemote will attempt to register
it. Please be aware of this and potential naming errors due to difference in folder paths when running in the container.
Future work may be done to allow execution in another project/domain, but reference entities should always be
correctly reflected and invoked.
```

### Registering and running

Assuming that your `flytekit` code is configured correctly, you will need to
Assuming that your `flytekit` code is configured correctly, you should
register all of the task and subworkflows that are used with your eager
workflow with `pyflyte register`:

Expand Down Expand Up @@ -294,7 +272,7 @@ invoked inside of it.

## Eager workflows on Flyte console

Since eager workflows are an experimental feature, there is currently no
Since eager workflows are still in beta, there is currently no
first-class representation of them on Flyte Console, the UI for Flyte.
When you register an eager workflow, you'll be able to see it in the task view:

Expand Down Expand Up @@ -325,23 +303,20 @@ eager workflow:

## Limitations

As this feature is still experimental, there are a few limitations that you
As this feature is still in beta, there are a few limitations that you
need to keep in mind:

- You cannot invoke {ref}`dynamic workflows <dynamic_workflow>`,
{ref}`map tasks <map_task>`, or {ref}`launch plans <launch_plan>` inside an
eager workflow.
- [Context managers](https://docs.python.org/3/library/contextlib.html) will
only work on locally executed functions within the eager workflow, i.e. using a
context manager to modify the behavior of a task or subworkflow will not work
because they are executed on a completely different pod.
- All exceptions raised by Flyte tasks or workflows will be caught and raised
as an {py:class}`~flytekit.experimental.EagerException` at runtime.
as an {py:class}`~flytekit.exceptions.eager.EagerException` at runtime.
- All task/subworkflow outputs are materialized as Python values, which includes
offloaded types like `FlyteFile`, `FlyteDirectory`, `StructuredDataset`, and
`pandas.DataFrame` will be fully downloaded into the pod running the eager workflow.
This prevents you from incrementally downloading or streaming very large datasets
in eager workflows.
in eager workflows. (Please reach out to the team if you are interested in improving this.)
- Flyte entities that are invoked inside of an eager workflow must be registered
under the same project and domain as the eager workflow itself. The eager
workflow will execute the latest version of these entities.
Expand Down

0 comments on commit 75a9243

Please sign in to comment.