From aaabdf0f45a78de60573d58e3fe1973a0773a69b Mon Sep 17 00:00:00 2001 From: Yee Hing Tong Date: Thu, 5 Sep 2024 21:53:19 -0700 Subject: [PATCH 1/3] note on imagespec/fast relationship Signed-off-by: Yee Hing Tong --- docs/flyte_fundamentals/registering_workflows.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/flyte_fundamentals/registering_workflows.md b/docs/flyte_fundamentals/registering_workflows.md index 1c53a78020..321593db25 100644 --- a/docs/flyte_fundamentals/registering_workflows.md +++ b/docs/flyte_fundamentals/registering_workflows.md @@ -72,7 +72,7 @@ run it with the supplied arguments. As you can see from the expected output, you can visit the link to the Flyte console to see the progress of your running execution. -You may also run `run --remote --copy-all`, which is very similar to the above command. As the name suggests, this will copy the source tree rooted at the top-level `__init__.py` file. With this strategy, any modules discoverable on the `PYTHONPATH` will be importable. +You may also run `run --remote --copy all`, which is very similar to the above command. As the name suggests, this will copy the source tree rooted at the top-level `__init__.py` file. With this strategy, any modules discoverable on the `PYTHONPATH` will be importable. ```{note} `pyflyte run` supports Flyte workflows that import any other user-defined modules that @@ -260,6 +260,14 @@ metadata/configuration, it's more secure if they're private. Learn more about how to pull private image in the {ref}`User Guide `. ``` +##### Relationship between ImageSpec and Fast Registration +The `ImageSpec` construct available in flytekit also has a mechanism to copy files into the image being built. By default, the way it works is +* if fast register is used, then it's assumed that you don't also want to copy source files into built image. +* if fast register is not used (which is the default for `pyflyte package`, or if `pyflyte register --copy none` is specified), then it's assumed that + you do want source files copied. + +In any case, if your `ImageSpec` constructor specifies a `source_root` and the `copy` argument set to something other than `CopyFileDetection.NO_COPY`, then files will be copied regardless of fast registration status. + #### Package your project with `pyflyte package` You can package your project with the `pyflyte package` command like so: From a4a2c38c5fe1581fea187c2f8a8046c0f6c0d7e5 Mon Sep 17 00:00:00 2001 From: Yee Hing Tong Date: Thu, 5 Sep 2024 23:22:57 -0700 Subject: [PATCH 2/3] update for image spec Signed-off-by: Yee Hing Tong --- docs/user_guide/customizing_dependencies/imagespec.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/user_guide/customizing_dependencies/imagespec.md b/docs/user_guide/customizing_dependencies/imagespec.md index ccdd52fe28..bf5d6fc32e 100644 --- a/docs/user_guide/customizing_dependencies/imagespec.md +++ b/docs/user_guide/customizing_dependencies/imagespec.md @@ -103,3 +103,9 @@ or image = ImageSpec(registry="ghcr.io/flyteorg", packages=["pandas"]).force_push() ``` [flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/master/examples/customizing_dependencies/ + +## Getting source files into ImageSpec +Typically, getting source code files into a task's image at run time on a live Flyte backend is done through the fast registration mechanism. +If the `source_root` and `copy` fields to an `ImageSpec` are left blank, then whether or not your source files are copied into the built `ImageSpec` image is basically the inverse of whether fast register is used. Please see [registering workflows](https://docs.flyte.org/en/latest/flyte_fundamentals/registering_workflows.html#containerizing-your-project) for the full explanation. + +Keep in mind that because files are sometimes copied into the built image, the tag that is published for an ImageSpec will change based on whether fast register is enabled, and the contents of any files copied. From e5fbee4942fadb116f5d71b352b62625d57be5a7 Mon Sep 17 00:00:00 2001 From: Yee Hing Tong Date: Sun, 8 Sep 2024 15:56:03 -0700 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Nikki Everett Signed-off-by: Yee Hing Tong --- docs/flyte_fundamentals/registering_workflows.md | 11 +++++------ docs/user_guide/customizing_dependencies/imagespec.md | 6 ++++-- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/flyte_fundamentals/registering_workflows.md b/docs/flyte_fundamentals/registering_workflows.md index 321593db25..8a5ec83c14 100644 --- a/docs/flyte_fundamentals/registering_workflows.md +++ b/docs/flyte_fundamentals/registering_workflows.md @@ -260,13 +260,12 @@ metadata/configuration, it's more secure if they're private. Learn more about how to pull private image in the {ref}`User Guide `. ``` -##### Relationship between ImageSpec and Fast Registration -The `ImageSpec` construct available in flytekit also has a mechanism to copy files into the image being built. By default, the way it works is -* if fast register is used, then it's assumed that you don't also want to copy source files into built image. -* if fast register is not used (which is the default for `pyflyte package`, or if `pyflyte register --copy none` is specified), then it's assumed that - you do want source files copied. +##### Relationship between ImageSpec and fast registration +The `ImageSpec` construct available in flytekit also has a mechanism to copy files into the image being built. Its behavior depends on the type of registration used: +* If fast register is used, then it's assumed that you don't also want to copy source files into the built image. +* If fast register is not used (which is the default for `pyflyte package`, or if `pyflyte register --copy none` is specified), then it's assumed that you do want source files copied into the built image. -In any case, if your `ImageSpec` constructor specifies a `source_root` and the `copy` argument set to something other than `CopyFileDetection.NO_COPY`, then files will be copied regardless of fast registration status. +If your `ImageSpec` constructor specifies a `source_root` and the `copy` argument is set to something other than `CopyFileDetection.NO_COPY`, then files will be copied regardless of fast registration status. #### Package your project with `pyflyte package` diff --git a/docs/user_guide/customizing_dependencies/imagespec.md b/docs/user_guide/customizing_dependencies/imagespec.md index bf5d6fc32e..5128e9601c 100644 --- a/docs/user_guide/customizing_dependencies/imagespec.md +++ b/docs/user_guide/customizing_dependencies/imagespec.md @@ -106,6 +106,8 @@ image = ImageSpec(registry="ghcr.io/flyteorg", packages=["pandas"]).force_push() ## Getting source files into ImageSpec Typically, getting source code files into a task's image at run time on a live Flyte backend is done through the fast registration mechanism. -If the `source_root` and `copy` fields to an `ImageSpec` are left blank, then whether or not your source files are copied into the built `ImageSpec` image is basically the inverse of whether fast register is used. Please see [registering workflows](https://docs.flyte.org/en/latest/flyte_fundamentals/registering_workflows.html#containerizing-your-project) for the full explanation. -Keep in mind that because files are sometimes copied into the built image, the tag that is published for an ImageSpec will change based on whether fast register is enabled, and the contents of any files copied. +However, if your `ImageSpec` constructor specifies a `source_root` and the `copy` argument is set to something other than `CopyFileDetection.NO_COPY`, then files will be copied regardless of fast registration status. +If the `source_root` and `copy` fields to an `ImageSpec` are left blank, then whether or not your source files are copied into the built `ImageSpec` image depends on whether or not you use fast registration. Please see [registering workflows](https://docs.flyte.org/en/latest/flyte_fundamentals/registering_workflows.html#containerizing-your-project) for the full explanation. + +Since files are sometimes copied into the built image, the tag that is published for an ImageSpec will change based on whether fast register is enabled, and the contents of any files copied.