diff --git a/docs/user_guide/data_types_and_io/flytedirectory.md b/docs/user_guide/data_types_and_io/flytedirectory.md index 121a7d9b67..4ad2316ded 100644 --- a/docs/user_guide/data_types_and_io/flytedirectory.md +++ b/docs/user_guide/data_types_and_io/flytedirectory.md @@ -86,4 +86,21 @@ You can run the workflow locally as follows: :lines: 94-114 ``` + +## Streaming support + +Flyte `1.5` introduced support for streaming `FlyteDirectory` types via the `fsspec` library. +The `FlyteDirectory` streaming feature enables efficient streaming and handling of entire directories, simplifying operations involving multiple files. + +:::{note} +This feature is marked as experimental. We'd love feedback on the API! +::: + +Here is a simple example, you can accept a `FlyteDirectory` as an input, walk through it and copy the files to another `FlyteDirectory` one by one. + +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/ddce0448141ea6d2cb148df52bf408874adb15ad/examples/data_types_and_io/data_types_and_io/file_streaming.py +:caption: data_types_and_io/file_streaming.py +:lines: 23-33 +``` + [flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/master/examples/data_types_and_io/ diff --git a/docs/user_guide/data_types_and_io/flytefile.md b/docs/user_guide/data_types_and_io/flytefile.md index e9c02e2132..44378a315a 100644 --- a/docs/user_guide/data_types_and_io/flytefile.md +++ b/docs/user_guide/data_types_and_io/flytefile.md @@ -90,4 +90,20 @@ You can enable type validation if you have the [python-magic](https://pypi.org/p Currently, type validation is only supported on the `Mac OS` and `Linux` platforms. ::: +## Streaming support + +Flyte `1.5` introduced support for streaming `FlyteFile` types via the `fsspec` library. +This integration enables efficient, on-demand access to remote files, eliminating the need for fully downloading them to local storage. + +:::{note} +This feature is marked as experimental. We'd love feedback on the API! +::: + +Here is a simple example of removing some columns from a CSV file and writing the result to a new file: + +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/ddce0448141ea6d2cb148df52bf408874adb15ad/examples/data_types_and_io/data_types_and_io/file_streaming.py +:caption: data_types_and_io/file_streaming.py +:lines: 8-20 +``` + [flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/master/examples/data_types_and_io/