Skip to content

Commit

Permalink
[Docs] add streaming support example for file and directory
Browse files Browse the repository at this point in the history
Signed-off-by: DenChenn <[email protected]>
  • Loading branch information
DenChenn committed Oct 21, 2024
1 parent 38af4ec commit 4ba7fdc
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 0 deletions.
17 changes: 17 additions & 0 deletions docs/user_guide/data_types_and_io/flytedirectory.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,4 +86,21 @@ You can run the workflow locally as follows:
:lines: 94-114
```


## Streaming support

Flyte `1.5` introduced support for streaming `FlyteDirectory` types via the `fsspec` library.
The `FlyteDirectory` streaming feature enables efficient streaming and handling of entire directories, simplifying operations involving multiple files.

:::{note}
This feature is marked as experimental. We'd love feedback on the API!
:::

Here is a simple example, you can accept a `FlyteDirectory` as an input, walk through it and copy the files to another `FlyteDirectory` one by one.

```{rli} https://raw.githubusercontent.com/DenChenn/flytesnacks/8dd7bf9708ff56d1fbee37f31763cba3277c102b/examples/data_types_and_io/data_types_and_io/file_streaming.py
:caption: data_types_and_io/file_streaming.py
:lines: 23-33
```{note}

[flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/master/examples/data_types_and_io/
16 changes: 16 additions & 0 deletions docs/user_guide/data_types_and_io/flytefile.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,4 +90,20 @@ You can enable type validation if you have the [python-magic](https://pypi.org/p
Currently, type validation is only supported on the `Mac OS` and `Linux` platforms.
:::

## Streaming support

Flyte `1.5` introduced support for streaming `FlyteFile` types via the `fsspec` library.
This integration enables efficient, on-demand access to remote files, eliminating the need for fully downloading them to local storage.

:::{note}
This feature is marked as experimental. We'd love feedback on the API!
:::

Here is a simple example of removing some columns from a CSV file and writing the result to a new file:

```{rli} https://raw.githubusercontent.com/DenChenn/flytesnacks/8dd7bf9708ff56d1fbee37f31763cba3277c102b/examples/data_types_and_io/data_types_and_io/file_streaming.py
:caption: data_types_and_io/file_streaming.py
:lines: 8-20
```{note}

[flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/master/examples/data_types_and_io/

0 comments on commit 4ba7fdc

Please sign in to comment.