Merge remote-tracking branch 'origin/master' into add-k8s-env-froms

CHANGELOG/CHANGELOG-v0.17.0.md

@@ -10,7 +10,7 @@
 1. Great Expectations Integration ([docs](https://docs.flyte.org/en/latest/flytesnacks/examples/greatexpectations_plugin/index.html)).
 1. Access to durable blob stores (AWS/GCS/etc) are now pluggable.
 1. Local task execution has been updated to also trigger the type engine.
-1. Tasks that have `cache=True` should now be cached when running locally as well ([docs](https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/task_cache.html#how-does-local-caching-work)).
+1. Tasks that have `cache=True` should now be cached when running locally as well ([docs](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/caching.html#how-does-local-caching-work)).
 
 Please see the [flytekit release](https://github.com/flyteorg/flytekit/releases/tag/v0.22.0) for the full list and more details.
 

CHANGELOG/CHANGELOG-v0.5.0.md

@@ -6,7 +6,7 @@
 - Enable CI system to run on forks.
 
 ## Core Platform
-- [Single Task Execution](https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/remote_task.html) to enable registering and launching tasks outside the scope of a workflow to enable faster iteration and a more intuitive development workflow.
+- [Single Task Execution](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/running_tasks.html) to enable registering and launching tasks outside the scope of a workflow to enable faster iteration and a more intuitive development workflow.
 - [Run to completion](https://docs.flyte.org/en/latest/protos/docs/core/core.html#ref-flyteidl-core-workflowmetadata-onfailurepolicy) to enable workflows to continue executing even if one or more branches fail.
 - Fixed retries for dynamically yielded nodes.
 - PreAlpha Support for Raw container with FlyteCoPilot. (docs coming soon). [Sample Notebooks](https://github.com/lyft/flytekit/blob/master/sample-notebooks/raw-container-shell.ipynb). This makes it possible to run workflows with arbitrary containers

CHANGELOG/CHANGELOG-v1.1.0.md

@@ -4,7 +4,7 @@
 ### User Improvements
 Support for [Optional types](https://github.com/flyteorg/flyte/issues/2426). With the inclusion of Union types in flytekit, we can now support optional types.
 
-[Flyte Deck](https://github.com/flyteorg/flyte/issues/2175) is now available. Please take a look at the [documentation](https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/decks.html) and also the [OSS presentation](https://www.youtube.com/watch?v=KqyBYIaAZ7c) that was done a few weeks back.
+[Flyte Deck](https://github.com/flyteorg/flyte/issues/2175) is now available. Please take a look at the [documentation](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/decks.html) and also the [OSS presentation](https://www.youtube.com/watch?v=KqyBYIaAZ7c) that was done a few weeks back.
 
 
 ### Backend Improvements

CHANGELOG/CHANGELOG-v1.10.0.md

@@ -8,7 +8,7 @@ Programmatically consuming inputs and outputs using flyteremote became a lot eas
 
 ![Usage snippet](./images/v1.10.0-flyteconsole-programmatic-access.png)
 
-You'll now be able to use offloaded types in [eager workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/eager_workflows.html).
+You'll now be able to use offloaded types in [eager workflows](https://docs.flyte.org/en/latest/user_guide/advanced_composition/eager_workflows.html).
 
 More ergonomic improvements to [pyflyte](https://docs.flyte.org/en/latest/api/flytekit/pyflyte.html), including the inclusion of a progress bar, the ability to activate launchplans, and the ability to interact with gate nodes in local executions.
 

CHANGELOG/CHANGELOG-v1.11.0-b0.md

@@ -0,0 +1,3 @@
+# Flyte v1.11.0-b0
+
+Beta release to test new idl

CHANGELOG/CHANGELOG-v1.2.0.md

@@ -18,7 +18,7 @@
 - dbt plugin (https://github.com/flyteorg/flyte/issues/2202)
 - cache overriding behavior is now open to all types (https://github.com/flyteorg/flyte/issues/2912)
 - Bug: Fallback to pickling in the case of unknown types used Unions (https://github.com/flyteorg/flyte/issues/2823)
-- [pyflyte run](https://docs.flyte.org/en/latest/api/flytekit/design/clis.html#pyflyte-run) now supports [imperative workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/basics/imperative_workflow.html)
+- [pyflyte run](https://docs.flyte.org/en/latest/api/flytekit/design/clis.html#pyflyte-run) now supports [imperative workflows](https://docs.flyte.org/en/latest/user_guide/basics/imperative_workflows.html)
 - Newlines are now stripped from client secrets (https://github.com/flyteorg/flytekit/pull/1163)
 - Ensure repeatability in the generation of cache keys in the case of dictionaries (https://github.com/flyteorg/flytekit/pull/1126)
 - Support for multiple images in the yaml config file (https://github.com/flyteorg/flytekit/pull/1106)

CHANGELOG/CHANGELOG-v1.5.0.md

@@ -63,7 +63,7 @@ def wf(a: int) -> str:
 
 Notice how calls to `t1_fixed_b` do not need to specify the `b` parameter.
 
-This also works for [Map Tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html) in a limited capacity. For example:
+This also works for [Map Tasks](https://docs.flyte.org/en/latest/user_guide/advanced_composition/map_tasks.html) in a limited capacity. For example:
 
 ```
 from flytekit import task, workflow, partial, map_task
@@ -107,5 +107,5 @@ Map tasks do not support partial tasks with lists as inputs.
 
 ## Flyteconsole
 
-Multiple bug fixes around [waiting for external inputs](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/waiting_for_external_inputs.html#waiting-for-external-inputs).
+Multiple bug fixes around [waiting for external inputs](https://docs.flyte.org/en/latest/user_guide/advanced_composition/waiting_for_external_inputs.html).
 Better support for dataclasses in the launch form.

CHANGELOG/CHANGELOG-v1.9.0.md

@@ -1,11 +1,11 @@
 # Flyte v1.9.0 Release
 
-In this release we're announcing two experimental features, namely (1) ArrayNode map tasks, and (2) Execution Tags. 
+In this release we're announcing two experimental features, namely (1) ArrayNode map tasks, and (2) Execution Tags.
 
 
 ### ArrayNode map tasks
 
-ArrayNodes are described more fully in [RFC 3346](https://github.com/flyteorg/flyte/blob/master/rfc/system/3346-array-node.md), but the summary is that ArrayNode map tasks are a drop-in replacement for [regular map tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html), the only difference being the submodule used to import the `map_task` function. 
+ArrayNodes are described more fully in [RFC 3346](https://github.com/flyteorg/flyte/blob/master/rfc/system/3346-array-node.md), but the summary is that ArrayNode map tasks are a drop-in replacement for [regular map tasks](https://docs.flyte.org/en/latest/user-guide/advanced_composition/map_tasks.html), the only difference being the submodule used to import the `map_task` function.
 More explicitly, let's say you have this code:
 
 ```python
@@ -15,7 +15,7 @@ from flytekit import map_task, task, workflow
 @task
 def t(a: int) -> int:
     ...
-    
+
 @workflow
 def wf(xs: List[int]) -> List[int]:
     return map_task(t)(a=xs)
@@ -31,7 +31,7 @@ from flytekit.experimental import map_task
 @task
 def t(a: int) -> int:
     ...
-    
+
 @workflow
 def wf(xs: List[int]) -> List[int]:
     return map_task(t)(a=xs)
@@ -119,7 +119,7 @@ As mentioned before, this feature is shipped in an experimental capacity, the id
 * chore: remove release git step by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/811
 * fix: union value handling in launch form by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/812
 
-## New Contributors 
+## New Contributors
 * @Nan2018 made their first contribution in https://github.com/flyteorg/flytekit/pull/1751
 * @oliverhu made their first contribution in https://github.com/flyteorg/flytekit/pull/1727
 * @DavidMertz made their first contribution in https://github.com/flyteorg/flytekit/pull/1761

Makefile

@@ -37,6 +37,7 @@ kustomize:
 .PHONY: helm
 helm: ## Generate K8s Manifest from Helm Charts.
 	bash script/generate_helm.sh
+	make -C docker/sandbox-bundled manifests
 
 .PHONY: release_automation
 release_automation:

README.md

@@ -7,7 +7,7 @@
 </h1>
 
 <p align="center">
-  :building_construction: :rocket: :chart_with_upwards_trend: 
+  :building_construction: :rocket: :chart_with_upwards_trend:
 </p>
 
 
@@ -24,7 +24,7 @@
     <img src="https://img.shields.io/badge/openssf%20best%20practices-passing-green?style=for-the-badge" alt="OpenSSF Best Practices label" /></a>
   <a href="https://artifacthub.io/packages/search?repo=flyte">
     <img src="https://img.shields.io/endpoint?style=for-the-badge&url=https://artifacthub.io/badge/repository/flyte" alt="Flyte Helm Chart label" /></a>
-  <a href="https://twitter.com/flyteorg"> 
+  <a href="https://twitter.com/flyteorg">
     <img src="https://img.shields.io/badge/X-000000.svg?style=for-the-badge&logo=X&logoColor=white" height=30px/></a>
   <a href="https://slack.flyte.org" alt="Twitter, formerly X logo label">
     <img src="https://img.shields.io/badge/Slack-Chat-pink?style=for-the-badge&logo=slack" alt="Flyte Slack label" /></a>
@@ -36,7 +36,7 @@ Flyte is an open-source orchestrator that facilitates building production-grade
   Build
 </h2>
 <p>
-Write code in Python or any other language and leverage a robust type engine. 
+Write code in Python or any other language and leverage a robust type engine.
 </p>
 
 <img alt="Getting started with Flyte" src="https://raw.githubusercontent.com/flyteorg/static-resources/main/common/flytereadmebuildv2.gif" style="width: 60%; height: auto;" />
@@ -48,7 +48,7 @@ Write code in Python or any other language and leverage a robust type engine.
 Either locally or on a remote cluster, execute your models with ease.
 </p>
 <img alt="Getting started with Flyte" src="https://raw.githubusercontent.com/flyteorg/static-resources/main/common/flytereadme-deploy.gif" style="width: 60%; height: auto;" />
- 
+
 
 <h3 align="center">
   <a href="#quick-start">Get Started</a>
@@ -107,24 +107,24 @@ Go to the [Deployment guide](https://docs.flyte.org/en/latest/deployment/deploym
 🌐 **Any language**: Write code in any language using raw containers, or choose [Python](https://github.com/flyteorg/flytekit), [Java](https://github.com/flyteorg/flytekit-java), [Scala](https://github.com/flyteorg/flytekit-java) or [JavaScript](https://github.com/NotMatthewGriffin/pterodactyl) SDKs to develop your Flyte workflows. <br />
 🔒 **Immutability**: Immutable executions help ensure reproducibility by preventing any changes to the state of an execution. <br />
 🧬 **Data lineage**: Track the movement and transformation of data throughout the lifecycle of your data and ML workflows. <br />
-📊 **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html). <br />
+📊 **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/en/latest/user_guide/advanced_composition/map_tasks.html). <br />
 🌎 **Multi-tenancy**: Multiple users can share the same platform while maintaining their own distinct data and configurations. <br />
-🌟 **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/dynamic_workflow.html) that can change and evolve as needed, making it easier to respond to changing requirements. <br />
-⏯️ [Wait](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution. <br />
-🌳 **Branching**: [Selectively execute branches](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/conditional.html) of your workflow based on static or dynamic data produced by other tasks or input data. <br />
+🌟 **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/en/latest/user_guide/advanced_composition/dynamic_workflows.html) that can change and evolve as needed, making it easier to respond to changing requirements. <br />
+⏯️ [Wait](https://docs.flyte.org/en/latest/user_guide/advanced_composition/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution. <br />
+🌳 **Branching**: [Selectively execute branches](https://docs.flyte.org/en/latest/user_guide/advanced_composition/conditionals.html) of your workflow based on static or dynamic data produced by other tasks or input data. <br />
 📈 **Data visualization**: Visualize data, monitor models and view training history through plots. <br />
-📂 **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/file.html#file) and [directories](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/folder.html) between local and cloud storage. <br />
-🗃️ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/structured_dataset.html). <br />
+📂 **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/flytefile.html) and [directories](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/flytedirectory.html) between local and cloud storage. <br />
+🗃️ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/structureddataset.html). <br />
 🛡️ **Recover from failures**: Recover only the failed tasks. <br />
 🔁 **Rerun a single task**: Rerun workflows at the most granular level without modifying the previous state of a data/ML workflow. <br />
 🔍 **Cache outputs**: Cache task outputs by passing `cache=True` to the task decorator. <br />
-🚩 **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/checkpoint.html) within a task execution. <br />
+🚩 **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/en/latest/user_guide/advanced_composition/intratask_checkpoints.html) within a task execution. <br />
 ⏰ **Timeout**: Define a timeout period, after which the task is marked as failure. <br />
 🏭 **Dev to prod**: As simple as changing your [domain](https://docs.flyte.org/en/latest/concepts/domains.html) from development or staging to production. <br />
 💸 **Spot or preemptible instances**: Schedule your workflows on spot instances by setting `interruptible` to `True` in the task decorator. <br />
 ☁️ **Cloud-native deployment**: Deploy Flyte on AWS, GCP, Azure and other cloud services. <br />
-📅 **Scheduling**: [Schedule](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/lp_schedules.html) your data and ML workflows to run at a specific time. <br />
-📢 **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/lp_notifications.html) through Slack, PagerDuty or email. <br />
+📅 **Scheduling**: [Schedule](https://docs.flyte.org/en/latest/user_guide/productionizing/schedules.html) your data and ML workflows to run at a specific time. <br />
+📢 **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/en/latest/user_guide/productionizing/notifications.html) through Slack, PagerDuty or email. <br />
 ⌛️ **Timeline view**: Evaluate the duration of each of your Flyte tasks and identify potential bottlenecks. <br />
 💨 **GPU acceleration**: Enable and control your tasks’ GPU demands by requesting resources in the task decorator. <br />
 🐳 **Dependency isolation via containers**: Maintain separate sets of dependencies for your tasks so no dependency conflicts arise. <br />

charts/flyte-binary/README.md

@@ -42,7 +42,7 @@ Chart for basic single Flyte executable deployment
 | configuration.auth.oidc.clientId | string | `""` |  |
 | configuration.auth.oidc.clientSecret | string | `""` |  |
 | configuration.co-pilot.image.repository | string | `"cr.flyte.org/flyteorg/flytecopilot"` |  |
-| configuration.co-pilot.image.tag | string | `"v1.10.7"` |  |
+| configuration.co-pilot.image.tag | string | `"v1.11.0-b0"` |  |
 | configuration.database.dbname | string | `"flyte"` |  |
 | configuration.database.host | string | `"127.0.0.1"` |  |
 | configuration.database.options | string | `"sslmode=disable"` |  |

charts/flyte-binary/values.yaml

@@ -159,7 +159,7 @@ configuration:
       # repository CoPilot sidecar image repository
       repository: cr.flyte.org/flyteorg/flytecopilot # FLYTECOPILOT_IMAGE
       # tag CoPilot sidecar image tag
-      tag: v1.10.7 # FLYTECOPILOT_TAG
+      tag: v1.11.0-b0 # FLYTECOPILOT_TAG
   # agentService Flyte Agent configuration
   agentService:
     defaultAgent: