diff --git a/CHANGELOG/CHANGELOG-v0.17.0.md b/CHANGELOG/CHANGELOG-v0.17.0.md index 2304834992..68c83e8479 100644 --- a/CHANGELOG/CHANGELOG-v0.17.0.md +++ b/CHANGELOG/CHANGELOG-v0.17.0.md @@ -10,7 +10,7 @@ 1. Great Expectations Integration ([docs](https://docs.flyte.org/en/latest/flytesnacks/examples/greatexpectations_plugin/index.html)). 1. Access to durable blob stores (AWS/GCS/etc) are now pluggable. 1. Local task execution has been updated to also trigger the type engine. -1. Tasks that have `cache=True` should now be cached when running locally as well ([docs](https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/task_cache.html#how-does-local-caching-work)). +1. Tasks that have `cache=True` should now be cached when running locally as well ([docs](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/caching.html#how-does-local-caching-work)). Please see the [flytekit release](https://github.com/flyteorg/flytekit/releases/tag/v0.22.0) for the full list and more details. diff --git a/CHANGELOG/CHANGELOG-v0.5.0.md b/CHANGELOG/CHANGELOG-v0.5.0.md index 20382f7050..87a4831f7f 100644 --- a/CHANGELOG/CHANGELOG-v0.5.0.md +++ b/CHANGELOG/CHANGELOG-v0.5.0.md @@ -6,7 +6,7 @@ - Enable CI system to run on forks. ## Core Platform -- [Single Task Execution](https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/remote_task.html) to enable registering and launching tasks outside the scope of a workflow to enable faster iteration and a more intuitive development workflow. +- [Single Task Execution](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/running_tasks.html) to enable registering and launching tasks outside the scope of a workflow to enable faster iteration and a more intuitive development workflow. - [Run to completion](https://docs.flyte.org/en/latest/protos/docs/core/core.html#ref-flyteidl-core-workflowmetadata-onfailurepolicy) to enable workflows to continue executing even if one or more branches fail. - Fixed retries for dynamically yielded nodes. - PreAlpha Support for Raw container with FlyteCoPilot. (docs coming soon). [Sample Notebooks](https://github.com/lyft/flytekit/blob/master/sample-notebooks/raw-container-shell.ipynb). This makes it possible to run workflows with arbitrary containers diff --git a/CHANGELOG/CHANGELOG-v1.1.0.md b/CHANGELOG/CHANGELOG-v1.1.0.md index ebcee3739a..9236270965 100644 --- a/CHANGELOG/CHANGELOG-v1.1.0.md +++ b/CHANGELOG/CHANGELOG-v1.1.0.md @@ -4,7 +4,7 @@ ### User Improvements Support for [Optional types](https://github.com/flyteorg/flyte/issues/2426). With the inclusion of Union types in flytekit, we can now support optional types. -[Flyte Deck](https://github.com/flyteorg/flyte/issues/2175) is now available. Please take a look at the [documentation](https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/decks.html) and also the [OSS presentation](https://www.youtube.com/watch?v=KqyBYIaAZ7c) that was done a few weeks back. +[Flyte Deck](https://github.com/flyteorg/flyte/issues/2175) is now available. Please take a look at the [documentation](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/decks.html) and also the [OSS presentation](https://www.youtube.com/watch?v=KqyBYIaAZ7c) that was done a few weeks back. ### Backend Improvements diff --git a/CHANGELOG/CHANGELOG-v1.10.0.md b/CHANGELOG/CHANGELOG-v1.10.0.md index 48d298ccf7..7791a6fd20 100644 --- a/CHANGELOG/CHANGELOG-v1.10.0.md +++ b/CHANGELOG/CHANGELOG-v1.10.0.md @@ -8,7 +8,7 @@ Programmatically consuming inputs and outputs using flyteremote became a lot eas ![Usage snippet](./images/v1.10.0-flyteconsole-programmatic-access.png) -You'll now be able to use offloaded types in [eager workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/eager_workflows.html). +You'll now be able to use offloaded types in [eager workflows](https://docs.flyte.org/en/latest/user_guide/advanced_composition/eager_workflows.html). More ergonomic improvements to [pyflyte](https://docs.flyte.org/en/latest/api/flytekit/pyflyte.html), including the inclusion of a progress bar, the ability to activate launchplans, and the ability to interact with gate nodes in local executions. diff --git a/CHANGELOG/CHANGELOG-v1.11.0-b0.md b/CHANGELOG/CHANGELOG-v1.11.0-b0.md new file mode 100644 index 0000000000..4d5e5ccb14 --- /dev/null +++ b/CHANGELOG/CHANGELOG-v1.11.0-b0.md @@ -0,0 +1,3 @@ +# Flyte v1.11.0-b0 + +Beta release to test new idl \ No newline at end of file diff --git a/CHANGELOG/CHANGELOG-v1.2.0.md b/CHANGELOG/CHANGELOG-v1.2.0.md index 00a3d8c735..d83bfa4f28 100644 --- a/CHANGELOG/CHANGELOG-v1.2.0.md +++ b/CHANGELOG/CHANGELOG-v1.2.0.md @@ -18,7 +18,7 @@ - dbt plugin (https://github.com/flyteorg/flyte/issues/2202) - cache overriding behavior is now open to all types (https://github.com/flyteorg/flyte/issues/2912) - Bug: Fallback to pickling in the case of unknown types used Unions (https://github.com/flyteorg/flyte/issues/2823) -- [pyflyte run](https://docs.flyte.org/en/latest/api/flytekit/design/clis.html#pyflyte-run) now supports [imperative workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/basics/imperative_workflow.html) +- [pyflyte run](https://docs.flyte.org/en/latest/api/flytekit/design/clis.html#pyflyte-run) now supports [imperative workflows](https://docs.flyte.org/en/latest/user_guide/basics/imperative_workflows.html) - Newlines are now stripped from client secrets (https://github.com/flyteorg/flytekit/pull/1163) - Ensure repeatability in the generation of cache keys in the case of dictionaries (https://github.com/flyteorg/flytekit/pull/1126) - Support for multiple images in the yaml config file (https://github.com/flyteorg/flytekit/pull/1106) diff --git a/CHANGELOG/CHANGELOG-v1.5.0.md b/CHANGELOG/CHANGELOG-v1.5.0.md index a711e38835..1cd809c867 100644 --- a/CHANGELOG/CHANGELOG-v1.5.0.md +++ b/CHANGELOG/CHANGELOG-v1.5.0.md @@ -63,7 +63,7 @@ def wf(a: int) -> str: Notice how calls to `t1_fixed_b` do not need to specify the `b` parameter. -This also works for [Map Tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html) in a limited capacity. For example: +This also works for [Map Tasks](https://docs.flyte.org/en/latest/user_guide/advanced_composition/map_tasks.html) in a limited capacity. For example: ``` from flytekit import task, workflow, partial, map_task @@ -107,5 +107,5 @@ Map tasks do not support partial tasks with lists as inputs. ## Flyteconsole -Multiple bug fixes around [waiting for external inputs](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/waiting_for_external_inputs.html#waiting-for-external-inputs). +Multiple bug fixes around [waiting for external inputs](https://docs.flyte.org/en/latest/user_guide/advanced_composition/waiting_for_external_inputs.html). Better support for dataclasses in the launch form. diff --git a/CHANGELOG/CHANGELOG-v1.9.0.md b/CHANGELOG/CHANGELOG-v1.9.0.md index 90371e5c11..dd7a8f93a3 100644 --- a/CHANGELOG/CHANGELOG-v1.9.0.md +++ b/CHANGELOG/CHANGELOG-v1.9.0.md @@ -1,11 +1,11 @@ # Flyte v1.9.0 Release -In this release we're announcing two experimental features, namely (1) ArrayNode map tasks, and (2) Execution Tags. +In this release we're announcing two experimental features, namely (1) ArrayNode map tasks, and (2) Execution Tags. ### ArrayNode map tasks -ArrayNodes are described more fully in [RFC 3346](https://github.com/flyteorg/flyte/blob/master/rfc/system/3346-array-node.md), but the summary is that ArrayNode map tasks are a drop-in replacement for [regular map tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html), the only difference being the submodule used to import the `map_task` function. +ArrayNodes are described more fully in [RFC 3346](https://github.com/flyteorg/flyte/blob/master/rfc/system/3346-array-node.md), but the summary is that ArrayNode map tasks are a drop-in replacement for [regular map tasks](https://docs.flyte.org/en/latest/user-guide/advanced_composition/map_tasks.html), the only difference being the submodule used to import the `map_task` function. More explicitly, let's say you have this code: ```python @@ -15,7 +15,7 @@ from flytekit import map_task, task, workflow @task def t(a: int) -> int: ... - + @workflow def wf(xs: List[int]) -> List[int]: return map_task(t)(a=xs) @@ -31,7 +31,7 @@ from flytekit.experimental import map_task @task def t(a: int) -> int: ... - + @workflow def wf(xs: List[int]) -> List[int]: return map_task(t)(a=xs) @@ -119,7 +119,7 @@ As mentioned before, this feature is shipped in an experimental capacity, the id * chore: remove release git step by @FrankFlitton in https://github.com/flyteorg/flyteconsole/pull/811 * fix: union value handling in launch form by @ursucarina in https://github.com/flyteorg/flyteconsole/pull/812 -## New Contributors +## New Contributors * @Nan2018 made their first contribution in https://github.com/flyteorg/flytekit/pull/1751 * @oliverhu made their first contribution in https://github.com/flyteorg/flytekit/pull/1727 * @DavidMertz made their first contribution in https://github.com/flyteorg/flytekit/pull/1761 diff --git a/Makefile b/Makefile index 7bcc6e8cf8..64af820787 100644 --- a/Makefile +++ b/Makefile @@ -37,6 +37,7 @@ kustomize: .PHONY: helm helm: ## Generate K8s Manifest from Helm Charts. bash script/generate_helm.sh + make -C docker/sandbox-bundled manifests .PHONY: release_automation release_automation: diff --git a/README.md b/README.md index 43c9e72a82..6049f262ef 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@

- :building_construction: :rocket: :chart_with_upwards_trend: + :building_construction: :rocket: :chart_with_upwards_trend:

@@ -24,7 +24,7 @@ OpenSSF Best Practices label Flyte Helm Chart label - + Flyte Slack label @@ -36,7 +36,7 @@ Flyte is an open-source orchestrator that facilitates building production-grade Build

-Write code in Python or any other language and leverage a robust type engine. +Write code in Python or any other language and leverage a robust type engine.

Getting started with Flyte @@ -48,7 +48,7 @@ Write code in Python or any other language and leverage a robust type engine. Either locally or on a remote cluster, execute your models with ease.

Getting started with Flyte - +

Get Started @@ -107,24 +107,24 @@ Go to the [Deployment guide](https://docs.flyte.org/en/latest/deployment/deploym 🌐 **Any language**: Write code in any language using raw containers, or choose [Python](https://github.com/flyteorg/flytekit), [Java](https://github.com/flyteorg/flytekit-java), [Scala](https://github.com/flyteorg/flytekit-java) or [JavaScript](https://github.com/NotMatthewGriffin/pterodactyl) SDKs to develop your Flyte workflows.
🔒 **Immutability**: Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.
🧬 **Data lineage**: Track the movement and transformation of data throughout the lifecycle of your data and ML workflows.
-📊 **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html).
+📊 **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/en/latest/user_guide/advanced_composition/map_tasks.html).
🌎 **Multi-tenancy**: Multiple users can share the same platform while maintaining their own distinct data and configurations.
-🌟 **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/dynamic_workflow.html) that can change and evolve as needed, making it easier to respond to changing requirements.
-⏯️ [Wait](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution.
-🌳 **Branching**: [Selectively execute branches](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/conditional.html) of your workflow based on static or dynamic data produced by other tasks or input data.
+🌟 **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/en/latest/user_guide/advanced_composition/dynamic_workflows.html) that can change and evolve as needed, making it easier to respond to changing requirements.
+⏯️ [Wait](https://docs.flyte.org/en/latest/user_guide/advanced_composition/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution.
+🌳 **Branching**: [Selectively execute branches](https://docs.flyte.org/en/latest/user_guide/advanced_composition/conditionals.html) of your workflow based on static or dynamic data produced by other tasks or input data.
📈 **Data visualization**: Visualize data, monitor models and view training history through plots.
-📂 **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/file.html#file) and [directories](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/folder.html) between local and cloud storage.
-🗃️ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/structured_dataset.html).
+📂 **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/flytefile.html) and [directories](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/flytedirectory.html) between local and cloud storage.
+🗃️ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/structureddataset.html).
🛡️ **Recover from failures**: Recover only the failed tasks.
🔁 **Rerun a single task**: Rerun workflows at the most granular level without modifying the previous state of a data/ML workflow.
🔍 **Cache outputs**: Cache task outputs by passing `cache=True` to the task decorator.
-🚩 **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/checkpoint.html) within a task execution.
+🚩 **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/en/latest/user_guide/advanced_composition/intratask_checkpoints.html) within a task execution.
⏰ **Timeout**: Define a timeout period, after which the task is marked as failure.
🏭 **Dev to prod**: As simple as changing your [domain](https://docs.flyte.org/en/latest/concepts/domains.html) from development or staging to production.
💸 **Spot or preemptible instances**: Schedule your workflows on spot instances by setting `interruptible` to `True` in the task decorator.
☁️ **Cloud-native deployment**: Deploy Flyte on AWS, GCP, Azure and other cloud services.
-📅 **Scheduling**: [Schedule](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/lp_schedules.html) your data and ML workflows to run at a specific time.
-📢 **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/lp_notifications.html) through Slack, PagerDuty or email.
+📅 **Scheduling**: [Schedule](https://docs.flyte.org/en/latest/user_guide/productionizing/schedules.html) your data and ML workflows to run at a specific time.
+📢 **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/en/latest/user_guide/productionizing/notifications.html) through Slack, PagerDuty or email.
⌛️ **Timeline view**: Evaluate the duration of each of your Flyte tasks and identify potential bottlenecks.
💨 **GPU acceleration**: Enable and control your tasks’ GPU demands by requesting resources in the task decorator.
🐳 **Dependency isolation via containers**: Maintain separate sets of dependencies for your tasks so no dependency conflicts arise.
diff --git a/charts/flyte-binary/README.md b/charts/flyte-binary/README.md index 9d1c3ddb54..99aa1c40b1 100644 --- a/charts/flyte-binary/README.md +++ b/charts/flyte-binary/README.md @@ -42,7 +42,7 @@ Chart for basic single Flyte executable deployment | configuration.auth.oidc.clientId | string | `""` | | | configuration.auth.oidc.clientSecret | string | `""` | | | configuration.co-pilot.image.repository | string | `"cr.flyte.org/flyteorg/flytecopilot"` | | -| configuration.co-pilot.image.tag | string | `"v1.10.7"` | | +| configuration.co-pilot.image.tag | string | `"v1.11.0-b0"` | | | configuration.database.dbname | string | `"flyte"` | | | configuration.database.host | string | `"127.0.0.1"` | | | configuration.database.options | string | `"sslmode=disable"` | | diff --git a/charts/flyte-binary/values.yaml b/charts/flyte-binary/values.yaml index 3b95aed614..0da15a1855 100644 --- a/charts/flyte-binary/values.yaml +++ b/charts/flyte-binary/values.yaml @@ -159,7 +159,7 @@ configuration: # repository CoPilot sidecar image repository repository: cr.flyte.org/flyteorg/flytecopilot # FLYTECOPILOT_IMAGE # tag CoPilot sidecar image tag - tag: v1.10.7 # FLYTECOPILOT_TAG + tag: v1.11.0-b0 # FLYTECOPILOT_TAG # agentService Flyte Agent configuration agentService: defaultAgent: diff --git a/charts/flyte-core/README.md b/charts/flyte-core/README.md index e287e67633..73b5fceae5 100644 --- a/charts/flyte-core/README.md +++ b/charts/flyte-core/README.md @@ -94,8 +94,8 @@ helm install gateway bitnami/contour -n flyte | configmap.clusters.clusterConfigs | list | `[]` | | | configmap.clusters.labelClusterMap | object | `{}` | | | configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI | -| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.7","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | -| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.7","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | +| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | +| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | configmap.core | object | `{"manager":{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"},"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration | | configmap.core.manager | object | `{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config). | | configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). | @@ -115,8 +115,8 @@ helm install gateway bitnami/contour -n flyte | configmap.schedulerConfig.scheduler.profilerPort | int | `10254` | | | configmap.task_logs | object | `{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":false}}}` | Section that configures how the Task logs are displayed on the UI. This has to be changed based on your actual logging provider. Refer to [structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/logs#LogConfig) to understand how to configure various logging engines | | configmap.task_logs.plugins.logs.cloudwatch-enabled | bool | `false` | One option is to enable cloudwatch logging for EKS, update the region and log group accordingly | -| configmap.task_resource_defaults | object | `{"task_resources":{"defaults":{"cpu":"100m","ephemeralStorage":"500Mi","memory":"500Mi"},"limits":{"cpu":2,"ephemeralStorage":"20Mi","gpu":1,"memory":"1Gi"}}}` | Task default resources configuration Refer to the full [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#TaskResourceConfiguration). | -| configmap.task_resource_defaults.task_resources | object | `{"defaults":{"cpu":"100m","ephemeralStorage":"500Mi","memory":"500Mi"},"limits":{"cpu":2,"ephemeralStorage":"20Mi","gpu":1,"memory":"1Gi"}}` | Task default resources parameters | +| configmap.task_resource_defaults | object | `{"task_resources":{"defaults":{"cpu":"100m","memory":"500Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi"}}}` | Task default resources configuration Refer to the full [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#TaskResourceConfiguration). | +| configmap.task_resource_defaults.task_resources | object | `{"defaults":{"cpu":"100m","memory":"500Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi"}}` | Task default resources parameters | | daskoperator | object | `{"enabled":false}` | Optional: Dask Plugin using the Dask Operator | | daskoperator.enabled | bool | `false` | - enable or disable the dask operator deployment installation | | databricks | object | `{"enabled":false,"plugin_config":{"plugins":{"databricks":{"databricksInstance":"dbc-a53b7a3c-614c","entrypointFile":"dbfs:///FileStore/tables/entrypoint.py"}}}}` | Optional: Databricks Plugin allows us to run the spark job on the Databricks platform. | @@ -129,7 +129,7 @@ helm install gateway bitnami/contour -n flyte | datacatalog.extraArgs | object | `{}` | Appends extra command line arguments to the main command | | datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment | -| datacatalog.image.tag | string | `"v1.10.7"` | Docker image tag | +| datacatalog.image.tag | string | `"v1.11.0-b0"` | Docker image tag | | datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | | datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | | datacatalog.podEnv | object | `{}` | Additional Datacatalog container environment variables | @@ -164,7 +164,7 @@ helm install gateway bitnami/contour -n flyte | flyteadmin.extraArgs | object | `{}` | Appends extra command line arguments to the serve command | | flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | | | flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment | -| flyteadmin.image.tag | string | `"v1.10.7"` | | +| flyteadmin.image.tag | string | `"v1.11.0-b0"` | | | flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create | | flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | | flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | @@ -194,13 +194,14 @@ helm install gateway bitnami/contour -n flyte | flyteagent.enabled | bool | `false` | | | flyteagent.plugin_config.plugins.agentService.defaultAgent.endpoint | string | `"dns:///flyteagent.flyte.svc.cluster.local:8000"` | | | flyteagent.plugin_config.plugins.agentService.defaultAgent.insecure | bool | `true` | | +| flyteagent.podLabels | object | `{}` | Labels for flyteagent pods | | flyteconsole.affinity | object | `{}` | affinity for Flyteconsole deployment | | flyteconsole.enabled | bool | `true` | | | flyteconsole.ga.enabled | bool | `false` | | | flyteconsole.ga.tracking_id | string | `"G-0QW4DJWJ20"` | | | flyteconsole.image.pullPolicy | string | `"IfNotPresent"` | | | flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment | -| flyteconsole.image.tag | string | `"v1.10.2"` | | +| flyteconsole.image.tag | string | `"v1.10.3"` | | | flyteconsole.imagePullSecrets | list | `[]` | ImagePullSecrets to assign to the Flyteconsole deployment | | flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment | | flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods | @@ -224,7 +225,7 @@ helm install gateway bitnami/contour -n flyte | flytepropeller.extraArgs | object | `{}` | Appends extra command line arguments to the main command | | flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | | | flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment | -| flytepropeller.image.tag | string | `"v1.10.7"` | | +| flytepropeller.image.tag | string | `"v1.11.0-b0"` | | | flytepropeller.manager | bool | `false` | | | flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment | | flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods | @@ -254,7 +255,7 @@ helm install gateway bitnami/contour -n flyte | flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment | -| flytescheduler.image.tag | string | `"v1.10.7"` | Docker image tag | +| flytescheduler.image.tag | string | `"v1.11.0-b0"` | Docker image tag | | flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment | | flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods | | flytescheduler.podEnv | object | `{}` | Additional Flytescheduler container environment variables | diff --git a/charts/flyte-core/templates/clusterresourcesync/deployment.yaml b/charts/flyte-core/templates/clusterresourcesync/deployment.yaml index 7fb93c9b92..19c0b9c48a 100644 --- a/charts/flyte-core/templates/clusterresourcesync/deployment.yaml +++ b/charts/flyte-core/templates/clusterresourcesync/deployment.yaml @@ -38,9 +38,11 @@ spec: {{- if not .Values.cluster_resource_manager.config.cluster_resources.standaloneDeployment }} {{- include "databaseSecret.volumeMount" . | nindent 10 }} {{- else }} + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth mountPath: /etc/secrets/ {{- end }} + {{- end }} - mountPath: /etc/flyte/clusterresource/templates name: resource-templates - mountPath: /etc/flyte/config @@ -66,10 +68,12 @@ spec: secretName: cluster-credentials {{- end }} {{- if .Values.cluster_resource_manager.config.cluster_resources.standaloneDeployment }} + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth secret: secretName: flyte-secret-auth {{- end }} + {{- end }} {{- with .Values.cluster_resource_manager.nodeSelector }} nodeSelector: {{ tpl (toYaml .) $ | nindent 8 }} {{- end }} diff --git a/charts/flyte-core/templates/flytescheduler/deployment.yaml b/charts/flyte-core/templates/flytescheduler/deployment.yaml index 8e6cd2a4ea..aa22a13e09 100755 --- a/charts/flyte-core/templates/flytescheduler/deployment.yaml +++ b/charts/flyte-core/templates/flytescheduler/deployment.yaml @@ -76,8 +76,10 @@ spec: volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 8 }} - mountPath: /etc/flyte/config name: config-volume + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth mountPath: /etc/secrets/ + {{- end }} {{- with .Values.flytescheduler.additionalVolumeMounts -}} {{ tpl (toYaml .) $ | nindent 8 }} {{- end }} @@ -91,9 +93,11 @@ spec: - configMap: name: flyte-scheduler-config name: config-volume + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth secret: secretName: flyte-secret-auth + {{- end }} {{- with .Values.flytescheduler.additionalVolumes -}} {{ tpl (toYaml .) $ | nindent 6 }} {{- end }} diff --git a/charts/flyte-core/templates/propeller/deployment.yaml b/charts/flyte-core/templates/propeller/deployment.yaml index d24101582b..5fd09e5d5d 100644 --- a/charts/flyte-core/templates/propeller/deployment.yaml +++ b/charts/flyte-core/templates/propeller/deployment.yaml @@ -82,8 +82,10 @@ spec: volumeMounts: - name: config-volume mountPath: /etc/flyte/config + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth mountPath: /etc/secrets/ + {{- end }} {{- with .Values.flytepropeller.additionalVolumeMounts -}} {{ tpl (toYaml .) $ | nindent 8 }} {{- end }} @@ -98,9 +100,11 @@ spec: - configMap: name: flyte-propeller-config name: config-volume + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth secret: secretName: flyte-secret-auth + {{- end }} {{- with .Values.flytepropeller.additionalVolumes -}} {{ tpl (toYaml .) $ | nindent 6 }} {{- end }} diff --git a/charts/flyte-core/templates/propeller/manager.yaml b/charts/flyte-core/templates/propeller/manager.yaml index 875d05dab4..21eb894ba8 100644 --- a/charts/flyte-core/templates/propeller/manager.yaml +++ b/charts/flyte-core/templates/propeller/manager.yaml @@ -43,8 +43,10 @@ template: volumeMounts: - name: config-volume mountPath: /etc/flyte/config + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth mountPath: /etc/secrets/ + {{- end }} {{- if .Values.flytepropeller.terminationMessagePolicy }} terminationMessagePolicy: "{{ .Values.flytepropeller.terminationMessagePolicy }}" {{- end }} @@ -53,9 +55,11 @@ template: - configMap: name: flyte-propeller-config name: config-volume + {{- if .Values.secrets.adminOauthClientCredentials.enabled }} - name: auth secret: secretName: flyte-secret-auth + {{- end }} {{- with .Values.flytepropeller.nodeSelector }} nodeSelector: {{ tpl (toYaml .) $ | nindent 6 }} {{- end }} diff --git a/charts/flyte-core/values.yaml b/charts/flyte-core/values.yaml index 4f6d9d12bc..c104af3e75 100755 --- a/charts/flyte-core/values.yaml +++ b/charts/flyte-core/values.yaml @@ -16,7 +16,7 @@ flyteadmin: image: # -- Docker image for Flyteadmin deployment repository: cr.flyte.org/flyteorg/flyteadmin # FLYTEADMIN_IMAGE - tag: v1.10.7 # FLYTEADMIN_TAG + tag: v1.11.0-b0 # FLYTEADMIN_TAG pullPolicy: IfNotPresent # -- Additional flyteadmin container environment variables # @@ -142,7 +142,7 @@ flytescheduler: # -- Docker image for Flytescheduler deployment repository: cr.flyte.org/flyteorg/flytescheduler # FLYTESCHEDULER_IMAGE # -- Docker image tag - tag: v1.10.7 # FLYTESCHEDULER_TAG + tag: v1.11.0-b0 # FLYTESCHEDULER_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytescheduler deployment @@ -208,7 +208,7 @@ datacatalog: # -- Docker image for Datacatalog deployment repository: cr.flyte.org/flyteorg/datacatalog # DATACATALOG_IMAGE # -- Docker image tag - tag: v1.10.7 # DATACATALOG_TAG + tag: v1.11.0-b0 # DATACATALOG_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Datacatalog deployment @@ -279,6 +279,8 @@ flyteagent: defaultAgent: endpoint: "dns:///flyteagent.flyte.svc.cluster.local:8000" insecure: true + # -- Labels for flyteagent pods + podLabels: {} # # FLYTEPROPELLER SETTINGS @@ -294,7 +296,7 @@ flytepropeller: image: # -- Docker image for Flytepropeller deployment repository: cr.flyte.org/flyteorg/flytepropeller # FLYTEPROPELLER_IMAGE - tag: v1.10.7 # FLYTEPROPELLER_TAG + tag: v1.11.0-b0 # FLYTEPROPELLER_TAG pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytepropeller deployment resources: @@ -377,7 +379,7 @@ flyteconsole: image: # -- Docker image for Flyteconsole deployment repository: cr.flyte.org/flyteorg/flyteconsole # FLYTECONSOLE_IMAGE - tag: v1.10.2 # FLYTECONSOLE_TAG + tag: v1.10.3 # FLYTECONSOLE_TAG pullPolicy: IfNotPresent # -- Default resources requests and limits for Flyteconsole deployment resources: @@ -692,11 +694,9 @@ configmap: defaults: cpu: 100m memory: 500Mi - ephemeralStorage: 500Mi limits: cpu: 2 memory: 1Gi - ephemeralStorage: 20Mi gpu: 1 # -- Admin Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow/launchplan#AdminConfig) @@ -725,7 +725,7 @@ configmap: # -- Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) co-pilot: name: flyte-copilot- - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 # FLYTECOPILOT_IMAGE + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 # FLYTECOPILOT_IMAGE start-timeout: 30s # -- Core propeller configuration diff --git a/charts/flyte/README.md b/charts/flyte/README.md index c0622f1bfc..7b568cb433 100644 --- a/charts/flyte/README.md +++ b/charts/flyte/README.md @@ -71,7 +71,7 @@ helm upgrade -f values-sandbox.yaml flyte . | contour.tolerations | list | `[]` | tolerations for Contour deployment | | daskoperator | object | `{"enabled":false}` | Optional: Dask Plugin using the Dask Operator | | daskoperator.enabled | bool | `false` | - enable or disable the dask operator deployment installation | -| flyte | object | `{"cluster_resource_manager":{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"service_account_name":"flyteadmin","templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]},"common":{"databaseSecret":{"name":"","secretManifest":{}},"flyteNamespaceTemplate":{"enabled":false},"ingress":{"albSSLRedirect":false,"annotations":{"nginx.ingress.kubernetes.io/app-root":"/console"},"enabled":true,"host":"","separateGrpcIngress":false,"separateGrpcIngressAnnotations":{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC"},"tls":{"enabled":false},"webpackHMR":true}},"configmap":{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpcPort":8089,"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.7","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"bigquery_query_job_task":"agent-service","container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-env-vars-from-configmaps":[],"default-env-vars-from-secrets":[],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}},"datacatalog":{"affinity":{},"configPath":"/etc/datacatalog/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/datacatalog","tag":"v1.10.7"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"db":{"admin":{"database":{"dbname":"flyteadmin","host":"postgres","port":5432,"username":"postgres"}},"datacatalog":{"database":{"dbname":"datacatalog","host":"postgres","port":5432,"username":"postgres"}}},"deployRedoc":true,"flyteadmin":{"additionalVolumeMounts":[],"additionalVolumes":[],"affinity":{},"configPath":"/etc/flyte/config/*.yaml","env":[],"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteadmin","tag":"v1.10.7"},"initialProjects":["flytesnacks","flytetester","flyteexamples"],"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"loadBalancerSourceRanges":[],"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flyteconsole":{"affinity":{},"ga":{"enabled":true,"tracking_id":"G-0QW4DJWJ20"},"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteconsole","tag":"v1.10.2"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","memory":"275Mi"},"requests":{"cpu":"10m","memory":"250Mi"}},"service":{"annotations":{},"type":"ClusterIP"},"tolerations":[]},"flytepropeller":{"affinity":{},"cacheSizeMbs":0,"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytepropeller","tag":"v1.10.7"},"manager":false,"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flytescheduler":{"affinity":{},"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytescheduler","tag":"v1.10.7"},"nodeSelector":{},"podAnnotations":{},"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"storage":{"bucketName":"my-s3-bucket","custom":{},"gcs":null,"s3":{"region":"us-east-1"},"type":"sandbox"},"webhook":{"enabled":true,"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]}},"workflow_notifications":{"config":{},"enabled":false},"workflow_scheduler":{"enabled":true,"type":"native"}}` | ------------------------------------------------------------------- Core System settings This section consists of Core components of Flyte and their deployment settings. This includes FlyteAdmin service, Datacatalog, FlytePropeller and Flyteconsole | +| flyte | object | `{"cluster_resource_manager":{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"service_account_name":"flyteadmin","templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]},"common":{"databaseSecret":{"name":"","secretManifest":{}},"flyteNamespaceTemplate":{"enabled":false},"ingress":{"albSSLRedirect":false,"annotations":{"nginx.ingress.kubernetes.io/app-root":"/console"},"enabled":true,"host":"","separateGrpcIngress":false,"separateGrpcIngressAnnotations":{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC"},"tls":{"enabled":false},"webpackHMR":true}},"configmap":{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpcPort":8089,"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"bigquery_query_job_task":"agent-service","container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}},"datacatalog":{"affinity":{},"configPath":"/etc/datacatalog/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/datacatalog","tag":"v1.11.0-b0"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"db":{"admin":{"database":{"dbname":"flyteadmin","host":"postgres","port":5432,"username":"postgres"}},"datacatalog":{"database":{"dbname":"datacatalog","host":"postgres","port":5432,"username":"postgres"}}},"deployRedoc":true,"flyteadmin":{"additionalVolumeMounts":[],"additionalVolumes":[],"affinity":{},"configPath":"/etc/flyte/config/*.yaml","env":[],"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteadmin","tag":"v1.11.0-b0"},"initialProjects":["flytesnacks","flytetester","flyteexamples"],"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"loadBalancerSourceRanges":[],"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flyteconsole":{"affinity":{},"ga":{"enabled":true,"tracking_id":"G-0QW4DJWJ20"},"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteconsole","tag":"v1.10.3"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","memory":"275Mi"},"requests":{"cpu":"10m","memory":"250Mi"}},"service":{"annotations":{},"type":"ClusterIP"},"tolerations":[]},"flytepropeller":{"affinity":{},"cacheSizeMbs":0,"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytepropeller","tag":"v1.11.0-b0"},"manager":false,"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flytescheduler":{"affinity":{},"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytescheduler","tag":"v1.11.0-b0"},"nodeSelector":{},"podAnnotations":{},"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"storage":{"bucketName":"my-s3-bucket","custom":{},"gcs":null,"s3":{"region":"us-east-1"},"type":"sandbox"},"webhook":{"enabled":true,"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]}},"workflow_notifications":{"config":{},"enabled":false},"workflow_scheduler":{"enabled":true,"type":"native"}}` | ------------------------------------------------------------------- Core System settings This section consists of Core components of Flyte and their deployment settings. This includes FlyteAdmin service, Datacatalog, FlytePropeller and Flyteconsole | | flyte.cluster_resource_manager | object | `{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"service_account_name":"flyteadmin","templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]}` | Configuration for the Cluster resource manager component. This is an optional component, that enables automatic cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain | | flyte.cluster_resource_manager.config.cluster_resources | object | `{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}` | ClusterResource parameters Refer to the [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#ClusterResourceConfig) to customize. | | flyte.cluster_resource_manager.config.cluster_resources.standaloneDeployment | bool | `false` | Starts the cluster resource manager in standalone mode with requisite auth credentials to call flyteadmin service endpoints | @@ -91,15 +91,15 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.common.ingress.separateGrpcIngressAnnotations | object | `{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC"}` | - Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled. | | flyte.common.ingress.tls | object | `{"enabled":false}` | - TLS Settings | | flyte.common.ingress.webpackHMR | bool | `true` | - Enable or disable HMR route to flyteconsole. This is useful only for frontend development. | -| flyte.configmap | object | `{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpcPort":8089,"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.7","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"bigquery_query_job_task":"agent-service","container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-env-vars-from-configmaps":[],"default-env-vars-from-secrets":[],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}}` | ----------------------------------------------------------------- CONFIGMAPS SETTINGS | +| flyte.configmap | object | `{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpcPort":8089,"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"bigquery_query_job_task":"agent-service","container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}}` | ----------------------------------------------------------------- CONFIGMAPS SETTINGS | | flyte.configmap.adminServer | object | `{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpcPort":8089,"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}}` | FlyteAdmin server configuration | | flyte.configmap.adminServer.auth | object | `{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}}` | Authentication configuration | | flyte.configmap.adminServer.server.security.secure | bool | `false` | Controls whether to serve requests over SSL/TLS. | | flyte.configmap.adminServer.server.security.useAuth | bool | `false` | Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. | | flyte.configmap.catalog | object | `{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}}` | Catalog Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/catalog#Config) Additional advanced Catalog configuration [here](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/catalog#Config) | | flyte.configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI | -| flyte.configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.7","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | -| flyte.configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.7","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | +| flyte.configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | +| flyte.configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | flyte.configmap.core | object | `{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration | | flyte.configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). | | flyte.configmap.datacatalogServer | object | `{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}}` | Datacatalog server config | @@ -120,7 +120,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.datacatalog.configPath | string | `"/etc/datacatalog/config/*.yaml"` | Default regex string for searching configuration files | | flyte.datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment | -| flyte.datacatalog.image.tag | string | `"v1.10.7"` | Docker image tag | +| flyte.datacatalog.image.tag | string | `"v1.11.0-b0"` | Docker image tag | | flyte.datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | | flyte.datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | | flyte.datacatalog.replicaCount | int | `1` | Replicas count for Datacatalog deployment | @@ -136,7 +136,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.flyteadmin.env | list | `[]` | Additional flyteadmin container environment variables e.g. SendGrid's API key - name: SENDGRID_API_KEY value: "" e.g. secret environment variable (you can combine it with .additionalVolumes): - name: SENDGRID_API_KEY valueFrom: secretKeyRef: name: sendgrid-secret key: api_key | | flyte.flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment | -| flyte.flyteadmin.image.tag | string | `"v1.10.7"` | Docker image tag | +| flyte.flyteadmin.image.tag | string | `"v1.11.0-b0"` | Docker image tag | | flyte.flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create | | flyte.flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | | flyte.flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | @@ -151,7 +151,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.flyteconsole.affinity | object | `{}` | affinity for Flyteconsole deployment | | flyte.flyteconsole.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment | -| flyte.flyteconsole.image.tag | string | `"v1.10.2"` | Docker image tag | +| flyte.flyteconsole.image.tag | string | `"v1.10.3"` | Docker image tag | | flyte.flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment | | flyte.flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods | | flyte.flyteconsole.replicaCount | int | `1` | Replicas count for Flyteconsole deployment | @@ -162,7 +162,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.flytepropeller.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | flyte.flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment | -| flyte.flytepropeller.image.tag | string | `"v1.10.7"` | Docker image tag | +| flyte.flytepropeller.image.tag | string | `"v1.11.0-b0"` | Docker image tag | | flyte.flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment | | flyte.flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods | | flyte.flytepropeller.replicaCount | int | `1` | Replicas count for Flytepropeller deployment | @@ -176,7 +176,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | flyte.flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment | -| flyte.flytescheduler.image.tag | string | `"v1.10.7"` | Docker image tag | +| flyte.flytescheduler.image.tag | string | `"v1.11.0-b0"` | Docker image tag | | flyte.flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment | | flyte.flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods | | flyte.flytescheduler.resources | object | `{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Flytescheduler deployment | diff --git a/charts/flyte/values.yaml b/charts/flyte/values.yaml index 5685c94a8a..469833b8c1 100755 --- a/charts/flyte/values.yaml +++ b/charts/flyte/values.yaml @@ -16,7 +16,7 @@ flyte: # -- Docker image for Flyteadmin deployment repository: cr.flyte.org/flyteorg/flyteadmin # FLYTEADMIN_IMAGE # -- Docker image tag - tag: v1.10.7 # FLYTEADMIN_TAG + tag: v1.11.0-b0 # FLYTEADMIN_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Additional flyteadmin container environment variables @@ -84,7 +84,7 @@ flyte: # -- Docker image for Flytescheduler deployment repository: cr.flyte.org/flyteorg/flytescheduler # FLYTESCHEDULER_IMAGE # -- Docker image tag - tag: v1.10.7 # FLYTESCHEDULER_TAG + tag: v1.11.0-b0 # FLYTESCHEDULER_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytescheduler deployment @@ -129,7 +129,7 @@ flyte: # -- Docker image for Datacatalog deployment repository: cr.flyte.org/flyteorg/datacatalog # DATACATALOG_IMAGE # -- Docker image tag - tag: v1.10.7 # DATACATALOG_TAG + tag: v1.11.0-b0 # DATACATALOG_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Datacatalog deployment @@ -178,7 +178,7 @@ flyte: # -- Docker image for Flytepropeller deployment repository: cr.flyte.org/flyteorg/flytepropeller # FLYTEPROPELLER_IMAGE # -- Docker image tag - tag: v1.10.7 # FLYTEPROPELLER_TAG + tag: v1.11.0-b0 # FLYTEPROPELLER_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytepropeller deployment @@ -223,7 +223,7 @@ flyte: # -- Docker image for Flyteconsole deployment repository: cr.flyte.org/flyteorg/flyteconsole # FLYTECONSOLE_IMAGE # -- Docker image tag - tag: v1.10.2 # FLYTECONSOLE_TAG + tag: v1.10.3 # FLYTECONSOLE_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Flyteconsole deployment @@ -471,7 +471,7 @@ flyte: # -- Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) co-pilot: name: flyte-copilot- - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 # FLYTECOPILOT_IMAGE + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 # FLYTECOPILOT_IMAGE start-timeout: 30s # -- Core propeller configuration diff --git a/charts/flyteagent/README.md b/charts/flyteagent/README.md index cd482d1a20..f889c095bd 100644 --- a/charts/flyteagent/README.md +++ b/charts/flyteagent/README.md @@ -20,7 +20,7 @@ A Helm chart for Flyte agent | fullnameOverride | string | `""` | | | image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | image.repository | string | `"ghcr.io/flyteorg/flyteagent"` | Docker image for flyteagent deployment | -| image.tag | string | `"1.10.3"` | Docker image tag | +| image.tag | string | `"1.10.7"` | Docker image tag | | nameOverride | string | `""` | | | nodeSelector | object | `{}` | nodeSelector for flyteagent deployment | | podAnnotations | object | `{}` | Annotations for flyteagent pods | diff --git a/charts/flyteagent/values.yaml b/charts/flyteagent/values.yaml index 508caf1984..aee84dc2b2 100755 --- a/charts/flyteagent/values.yaml +++ b/charts/flyteagent/values.yaml @@ -23,7 +23,7 @@ image: # -- Docker image for flyteagent deployment repository: ghcr.io/flyteorg/flyteagent # -- Docker image tag - tag: 1.10.3 # FLYTEAGENT_TAG + tag: 1.10.7 # FLYTEAGENT_TAG # -- Docker image pull policy pullPolicy: IfNotPresent ports: diff --git a/cmd/single/console_dist.go b/cmd/single/console_dist.go index ce6c32ce0d..f7ea8b674c 100644 --- a/cmd/single/console_dist.go +++ b/cmd/single/console_dist.go @@ -20,6 +20,7 @@ var consoleHandlers = map[string]handlerFunc{ consoleHandler.ServeHTTP(writer, request) }, consoleRoot + "/": func(writer http.ResponseWriter, request *http.Request) { + writer.Header().Set("Cache-Control", "max-age=604800") // 7 days consoleHandler.ServeHTTP(writer, request) }, } diff --git a/deployment/agent/flyte_agent_helm_generated.yaml b/deployment/agent/flyte_agent_helm_generated.yaml index de54cfda22..46762b4cff 100644 --- a/deployment/agent/flyte_agent_helm_generated.yaml +++ b/deployment/agent/flyte_agent_helm_generated.yaml @@ -78,7 +78,7 @@ spec: - pyflyte - serve - agent - image: "ghcr.io/flyteorg/flyteagent:1.10.3" + image: "ghcr.io/flyteorg/flyteagent:1.10.7" imagePullPolicy: "IfNotPresent" name: flyteagent volumeMounts: diff --git a/deployment/eks/flyte_aws_scheduler_helm_generated.yaml b/deployment/eks/flyte_aws_scheduler_helm_generated.yaml index 18c60208b8..c1865eeb87 100644 --- a/deployment/eks/flyte_aws_scheduler_helm_generated.yaml +++ b/deployment/eks/flyte_aws_scheduler_helm_generated.yaml @@ -192,12 +192,10 @@ data: task_resources: defaults: cpu: 1000m - ephemeralStorage: 500Mi memory: 1000Mi storage: 1000Mi limits: cpu: 2 - ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -431,7 +429,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -849,7 +847,7 @@ spec: template: metadata: annotations: - configChecksum: "85f2694a4138443026b87878dbbc5f1e9f52aa54eb87ef4c64117d1d91e1a7f" + configChecksum: "2b5c85969f2bd85bb51a084f9fd72c20c3aca94be99e53cb4c4e9f78e77ebc5" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -870,7 +868,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -891,7 +889,7 @@ spec: - flytesnacks - flytetester - flyteexamples - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -909,7 +907,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -926,7 +924,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -953,7 +951,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -1058,7 +1056,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -1114,7 +1112,7 @@ spec: seLinuxOptions: type: spc_t containers: - - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.2" + - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.3" imagePullPolicy: "IfNotPresent" name: flyteconsole envFrom: @@ -1188,7 +1186,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -1206,7 +1204,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1269,7 +1267,7 @@ spec: template: metadata: annotations: - configChecksum: "305d6f36301e10e952435f597dbe6700381a43c496a3be2cca60c175439fc9a" + configChecksum: "30e5fce341e4344cb6253ef4321f37c1e0895b9b55a927f94dfbc303d65c15b" labels: app.kubernetes.io/name: flytepropeller app.kubernetes.io/instance: flyte @@ -1295,7 +1293,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -1349,9 +1347,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.10.7 + app.kubernetes.io/version: v1.11.0-b0 annotations: - configChecksum: "305d6f36301e10e952435f597dbe6700381a43c496a3be2cca60c175439fc9a" + configChecksum: "30e5fce341e4344cb6253ef4321f37c1e0895b9b55a927f94dfbc303d65c15b" spec: securityContext: fsGroup: 65534 @@ -1363,7 +1361,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -1390,7 +1388,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/eks/flyte_generated.yaml b/deployment/eks/flyte_generated.yaml index d12576c3af..b4b8f63584 100644 --- a/deployment/eks/flyte_generated.yaml +++ b/deployment/eks/flyte_generated.yaml @@ -8640,7 +8640,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: cr.flyte.org/flyteorg/datacatalog:v1.10.7 + image: cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0 imagePullPolicy: IfNotPresent name: datacatalog ports: @@ -8663,7 +8663,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: cr.flyte.org/flyteorg/datacatalog:v1.10.7 + image: cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0 imagePullPolicy: IfNotPresent name: run-migrations volumeMounts: @@ -8724,7 +8724,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: cr.flyte.org/flyteorg/flytepropeller:v1.10.7 + image: cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0 imagePullPolicy: IfNotPresent name: webhook volumeMounts: @@ -8751,7 +8751,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: cr.flyte.org/flyteorg/flytepropeller:v1.10.7 + image: cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0 imagePullPolicy: IfNotPresent name: generate-secrets volumeMounts: @@ -8799,7 +8799,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: cr.flyte.org/flyteorg/flyteadmin:v1.10.7 + image: cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0 imagePullPolicy: IfNotPresent name: flyteadmin ports: @@ -8846,7 +8846,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: cr.flyte.org/flyteorg/flyteadmin:v1.10.7 + image: cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0 imagePullPolicy: IfNotPresent name: run-migrations volumeMounts: @@ -8863,7 +8863,7 @@ spec: - flytesnacks - flytetester - flyteexamples - image: cr.flyte.org/flyteorg/flyteadmin:v1.10.7 + image: cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0 imagePullPolicy: IfNotPresent name: seed-projects volumeMounts: @@ -8877,7 +8877,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: cr.flyte.org/flyteorg/flyteadmin:v1.10.7 + image: cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0 imagePullPolicy: IfNotPresent name: sync-cluster-resources volumeMounts: @@ -8897,7 +8897,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: cr.flyte.org/flyteorg/flyteadmin:v1.10.7 + image: cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0 imagePullPolicy: IfNotPresent name: generate-secrets volumeMounts: @@ -8951,7 +8951,7 @@ spec: - envFrom: - configMapRef: name: flyte-console-config - image: cr.flyte.org/flyteorg/flyteconsole:v1.10.2 + image: cr.flyte.org/flyteorg/flyteconsole:v1.10.3 name: flyteconsole ports: - containerPort: 8080 @@ -9002,7 +9002,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: cr.flyte.org/flyteorg/flytepropeller:v1.10.7 + image: cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0 imagePullPolicy: IfNotPresent name: flytepropeller ports: @@ -9270,7 +9270,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: cr.flyte.org/flyteorg/flyteadmin:v1.10.7 + image: cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0 imagePullPolicy: IfNotPresent name: sync-cluster-resources volumeMounts: diff --git a/deployment/eks/flyte_helm_controlplane_generated.yaml b/deployment/eks/flyte_helm_controlplane_generated.yaml index 520e7f2a03..1ae984cf69 100644 --- a/deployment/eks/flyte_helm_controlplane_generated.yaml +++ b/deployment/eks/flyte_helm_controlplane_generated.yaml @@ -173,12 +173,10 @@ data: task_resources: defaults: cpu: 1000m - ephemeralStorage: 500Mi memory: 1000Mi storage: 1000Mi limits: cpu: 2 - ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -555,7 +553,7 @@ spec: template: metadata: annotations: - configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" + configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -576,7 +574,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -597,7 +595,7 @@ spec: - flytesnacks - flytetester - flyteexamples - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -615,7 +613,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -632,7 +630,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -659,7 +657,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -764,7 +762,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -820,7 +818,7 @@ spec: seLinuxOptions: type: spc_t containers: - - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.2" + - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.3" imagePullPolicy: "IfNotPresent" name: flyteconsole envFrom: @@ -894,7 +892,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -912,7 +910,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -975,7 +973,7 @@ spec: template: metadata: annotations: - configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" + configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte @@ -995,7 +993,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1015,7 +1013,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: diff --git a/deployment/eks/flyte_helm_dataplane_generated.yaml b/deployment/eks/flyte_helm_dataplane_generated.yaml index b6dd553ba4..510ef5c3c8 100644 --- a/deployment/eks/flyte_helm_dataplane_generated.yaml +++ b/deployment/eks/flyte_helm_dataplane_generated.yaml @@ -94,7 +94,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -427,7 +427,7 @@ spec: template: metadata: annotations: - configChecksum: "305d6f36301e10e952435f597dbe6700381a43c496a3be2cca60c175439fc9a" + configChecksum: "30e5fce341e4344cb6253ef4321f37c1e0895b9b55a927f94dfbc303d65c15b" labels: app.kubernetes.io/name: flytepropeller app.kubernetes.io/instance: flyte @@ -453,7 +453,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -507,9 +507,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.10.7 + app.kubernetes.io/version: v1.11.0-b0 annotations: - configChecksum: "305d6f36301e10e952435f597dbe6700381a43c496a3be2cca60c175439fc9a" + configChecksum: "30e5fce341e4344cb6253ef4321f37c1e0895b9b55a927f94dfbc303d65c15b" spec: securityContext: fsGroup: 65534 @@ -521,7 +521,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -548,7 +548,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/eks/flyte_helm_generated.yaml b/deployment/eks/flyte_helm_generated.yaml index ab009db4cb..4cd67923ba 100644 --- a/deployment/eks/flyte_helm_generated.yaml +++ b/deployment/eks/flyte_helm_generated.yaml @@ -204,12 +204,10 @@ data: task_resources: defaults: cpu: 1000m - ephemeralStorage: 500Mi memory: 1000Mi storage: 1000Mi limits: cpu: 2 - ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -462,7 +460,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -880,7 +878,7 @@ spec: template: metadata: annotations: - configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" + configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -901,7 +899,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -922,7 +920,7 @@ spec: - flytesnacks - flytetester - flyteexamples - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -940,7 +938,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -957,7 +955,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -984,7 +982,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -1089,7 +1087,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -1145,7 +1143,7 @@ spec: seLinuxOptions: type: spc_t containers: - - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.2" + - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.3" imagePullPolicy: "IfNotPresent" name: flyteconsole envFrom: @@ -1219,7 +1217,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -1237,7 +1235,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1300,7 +1298,7 @@ spec: template: metadata: annotations: - configChecksum: "61fa8a4eebe7e96a3e25b0b2c4baaf7d6af84924167f57e569632fdd282b442" + configChecksum: "053b20ebc40227f6ed8ddc61f5997ee7997c604158f773779f20ec61af11a2f" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte @@ -1320,7 +1318,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1340,7 +1338,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: @@ -1399,7 +1397,7 @@ spec: template: metadata: annotations: - configChecksum: "305d6f36301e10e952435f597dbe6700381a43c496a3be2cca60c175439fc9a" + configChecksum: "30e5fce341e4344cb6253ef4321f37c1e0895b9b55a927f94dfbc303d65c15b" labels: app.kubernetes.io/name: flytepropeller app.kubernetes.io/instance: flyte @@ -1425,7 +1423,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -1479,9 +1477,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.10.7 + app.kubernetes.io/version: v1.11.0-b0 annotations: - configChecksum: "305d6f36301e10e952435f597dbe6700381a43c496a3be2cca60c175439fc9a" + configChecksum: "30e5fce341e4344cb6253ef4321f37c1e0895b9b55a927f94dfbc303d65c15b" spec: securityContext: fsGroup: 65534 @@ -1493,7 +1491,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -1520,7 +1518,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/gcp/flyte_helm_controlplane_generated.yaml b/deployment/gcp/flyte_helm_controlplane_generated.yaml index 3ede3cd5be..0f1ebf1381 100644 --- a/deployment/gcp/flyte_helm_controlplane_generated.yaml +++ b/deployment/gcp/flyte_helm_controlplane_generated.yaml @@ -178,12 +178,10 @@ data: task_resources: defaults: cpu: 500m - ephemeralStorage: 500Mi memory: 500Mi storage: 500Mi limits: cpu: 2 - ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -570,7 +568,7 @@ spec: template: metadata: annotations: - configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" + configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -591,7 +589,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -612,7 +610,7 @@ spec: - flytesnacks - flytetester - flyteexamples - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -630,7 +628,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -647,7 +645,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -674,7 +672,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -779,7 +777,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -835,7 +833,7 @@ spec: seLinuxOptions: type: spc_t containers: - - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.2" + - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.3" imagePullPolicy: "IfNotPresent" name: flyteconsole envFrom: @@ -909,7 +907,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -927,7 +925,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -990,7 +988,7 @@ spec: template: metadata: annotations: - configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" + configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte @@ -1010,7 +1008,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1030,7 +1028,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: diff --git a/deployment/gcp/flyte_helm_dataplane_generated.yaml b/deployment/gcp/flyte_helm_dataplane_generated.yaml index 4ba186eb48..59a0fca4f6 100644 --- a/deployment/gcp/flyte_helm_dataplane_generated.yaml +++ b/deployment/gcp/flyte_helm_dataplane_generated.yaml @@ -94,7 +94,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -435,7 +435,7 @@ spec: template: metadata: annotations: - configChecksum: "74e9568d4bf785885a1103d7c39c1b2ede648fc59f8f714c28ba6578e5d5ca1" + configChecksum: "bfe89fce66aa8eee9543c676ab07345b9c05c4ec7859daefd51da6bf414f0f4" labels: app.kubernetes.io/name: flytepropeller app.kubernetes.io/instance: flyte @@ -460,7 +460,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -514,9 +514,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.10.7 + app.kubernetes.io/version: v1.11.0-b0 annotations: - configChecksum: "74e9568d4bf785885a1103d7c39c1b2ede648fc59f8f714c28ba6578e5d5ca1" + configChecksum: "bfe89fce66aa8eee9543c676ab07345b9c05c4ec7859daefd51da6bf414f0f4" spec: securityContext: fsGroup: 65534 @@ -528,7 +528,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -555,7 +555,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/gcp/flyte_helm_generated.yaml b/deployment/gcp/flyte_helm_generated.yaml index a3ead16e95..f220536479 100644 --- a/deployment/gcp/flyte_helm_generated.yaml +++ b/deployment/gcp/flyte_helm_generated.yaml @@ -209,12 +209,10 @@ data: task_resources: defaults: cpu: 500m - ephemeralStorage: 500Mi memory: 500Mi storage: 500Mi limits: cpu: 2 - ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 2000Mi @@ -475,7 +473,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -903,7 +901,7 @@ spec: template: metadata: annotations: - configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" + configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -924,7 +922,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -945,7 +943,7 @@ spec: - flytesnacks - flytetester - flyteexamples - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -963,7 +961,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -980,7 +978,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -1007,7 +1005,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -1112,7 +1110,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -1168,7 +1166,7 @@ spec: seLinuxOptions: type: spc_t containers: - - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.2" + - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.3" imagePullPolicy: "IfNotPresent" name: flyteconsole envFrom: @@ -1242,7 +1240,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -1260,7 +1258,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1323,7 +1321,7 @@ spec: template: metadata: annotations: - configChecksum: "c7d43aa7ff4bf67124616d00a83d3c45926ea5ca36bdebdfac1cbcd0e465270" + configChecksum: "2e169a911a8234dd42d06ca0887279093f4ed36033d0543749ce126b26b50f3" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte @@ -1343,7 +1341,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1363,7 +1361,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: @@ -1422,7 +1420,7 @@ spec: template: metadata: annotations: - configChecksum: "74e9568d4bf785885a1103d7c39c1b2ede648fc59f8f714c28ba6578e5d5ca1" + configChecksum: "bfe89fce66aa8eee9543c676ab07345b9c05c4ec7859daefd51da6bf414f0f4" labels: app.kubernetes.io/name: flytepropeller app.kubernetes.io/instance: flyte @@ -1447,7 +1445,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -1501,9 +1499,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.10.7 + app.kubernetes.io/version: v1.11.0-b0 annotations: - configChecksum: "74e9568d4bf785885a1103d7c39c1b2ede648fc59f8f714c28ba6578e5d5ca1" + configChecksum: "bfe89fce66aa8eee9543c676ab07345b9c05c4ec7859daefd51da6bf414f0f4" spec: securityContext: fsGroup: 65534 @@ -1515,7 +1513,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -1542,7 +1540,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml b/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml index a8c637e0a3..2d93910a5c 100644 --- a/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml +++ b/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml @@ -116,7 +116,7 @@ data: stackdriver-enabled: false k8s: co-pilot: - image: "cr.flyte.org/flyteorg/flytecopilot:v1.10.7" + image: "cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0" k8s-array: logs: config: @@ -358,7 +358,7 @@ spec: app.kubernetes.io/instance: flyte app.kubernetes.io/component: flyte-binary annotations: - checksum/configuration: da323d1ce8e93e67668afc8b940ef2ee926464950f41ef618ed65b7ca1c42ada + checksum/configuration: 882c31ec18bdac7aa4f1a9057f9e549b1307b60b5d76839dfb6bc526958bee57 checksum/configuration-secret: d5d93f4e67780b21593dc3799f0f6682aab0765e708e4020939975d14d44f929 checksum/cluster-resource-templates: 7dfa59f3d447e9c099b8f8ffad3af466fecbc9cf9f8c97295d9634254a55d4ae spec: diff --git a/deployment/sandbox/flyte_helm_generated.yaml b/deployment/sandbox/flyte_helm_generated.yaml index 84cb265052..a4d9ece75f 100644 --- a/deployment/sandbox/flyte_helm_generated.yaml +++ b/deployment/sandbox/flyte_helm_generated.yaml @@ -334,12 +334,10 @@ data: task_resources: defaults: cpu: 100m - ephemeralStorage: 500Mi memory: 200Mi storage: 5Mi limits: cpu: 2 - ephemeralStorage: 20Mi gpu: 1 memory: 1Gi storage: 20Mi @@ -587,7 +585,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.10.7 + image: cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -6688,7 +6686,7 @@ spec: template: metadata: annotations: - configChecksum: "82d6ffa2a2dd83eb11c491a95af43fdede659d6b5b400b6edcd88291a28c4f4" + configChecksum: "45f0232531c0d1494809cf83387a95b2fc802019ea095de7a24ccd4f8de86ec" labels: app.kubernetes.io/name: flyteadmin app.kubernetes.io/instance: flyte @@ -6709,7 +6707,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -6729,7 +6727,7 @@ spec: - flytesnacks - flytetester - flyteexamples - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -6746,7 +6744,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -6762,7 +6760,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -6789,7 +6787,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -6884,7 +6882,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.10.7" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -6937,7 +6935,7 @@ spec: seLinuxOptions: type: spc_t containers: - - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.2" + - image: "cr.flyte.org/flyteorg/flyteconsole:v1.10.3" imagePullPolicy: "IfNotPresent" name: flyteconsole envFrom: @@ -7009,7 +7007,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -7026,7 +7024,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.10.7" + image: "cr.flyte.org/flyteorg/datacatalog:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -7079,7 +7077,7 @@ spec: template: metadata: annotations: - configChecksum: "82d6ffa2a2dd83eb11c491a95af43fdede659d6b5b400b6edcd88291a28c4f4" + configChecksum: "45f0232531c0d1494809cf83387a95b2fc802019ea095de7a24ccd4f8de86ec" labels: app.kubernetes.io/name: flytescheduler app.kubernetes.io/instance: flyte @@ -7099,7 +7097,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -7118,7 +7116,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.10.7" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: @@ -7174,7 +7172,11 @@ spec: template: metadata: annotations: +<<<<<<< HEAD configChecksum: "8aecf4367155815507c20281571bb08e78ea9ed12376fff6f7b9ff2f8f669d9" +======= + configChecksum: "8d992b3c2174350d363ddbf3b1ac0d7f8017a546ec794a9551a4f2b1f4e6ea7" +>>>>>>> origin/master labels: app.kubernetes.io/name: flytepropeller app.kubernetes.io/instance: flyte @@ -7199,7 +7201,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -7246,9 +7248,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.10.7 + app.kubernetes.io/version: v1.11.0-b0 annotations: - configChecksum: "8aecf4367155815507c20281571bb08e78ea9ed12376fff6f7b9ff2f8f669d9" + configChecksum: "8d992b3c2174350d363ddbf3b1ac0d7f8017a546ec794a9551a4f2b1f4e6ea7" spec: securityContext: fsGroup: 65534 @@ -7260,7 +7262,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -7287,7 +7289,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.10.7" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.11.0-b0" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/docker/sandbox-bundled/Makefile b/docker/sandbox-bundled/Makefile index 0b4eac7e0a..d2ec89de8b 100644 --- a/docker/sandbox-bundled/Makefile +++ b/docker/sandbox-bundled/Makefile @@ -15,12 +15,15 @@ flyte: FLYTECONSOLE_VERSION := latest flyte: $(foreach arch,amd64 arm64,$(call FLYTE_BINARY_BUILD,$(arch))) -.PHONY: manifests -manifests: - mkdir -p manifests +.PHONY: dep_update +dep_update: helm dependency update ../../charts/flyteagent helm dependency update ../../charts/flyte-binary helm dependency update ../../charts/flyte-sandbox + +.PHONY: manifests +manifests: dep_update + mkdir -p manifests kustomize build \ --enable-helm \ --load-restrictor=LoadRestrictionsNone \ @@ -35,7 +38,7 @@ manifests: kustomize/complete-agent > manifests/complete-agent.yaml .PHONY: build -build: flyte manifests +build: flyte dep_update manifests [ -n "$(shell docker buildx ls | awk '/^flyte-sandbox / {print $$1}')" ] || \ docker buildx create --name flyte-sandbox \ --driver docker-container --driver-opt image=moby/buildkit:master \ diff --git a/docker/sandbox-bundled/manifests/complete-agent.yaml b/docker/sandbox-bundled/manifests/complete-agent.yaml index 85eb73622d..6de7c86be9 100644 --- a/docker/sandbox-bundled/manifests/complete-agent.yaml +++ b/docker/sandbox-bundled/manifests/complete-agent.yaml @@ -468,7 +468,7 @@ data: stackdriver-enabled: false k8s: co-pilot: - image: "cr.flyte.org/flyteorg/flytecopilot:v1.10.7" + image: "cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0" k8s-array: logs: config: @@ -816,7 +816,7 @@ type: Opaque --- apiVersion: v1 data: - haSharedSecret: ZzlBSjNLWDhDcTdqZ05xUg== + haSharedSecret: UDI2NklEa2dSNUhNeTFteA== proxyPassword: "" proxyUsername: "" kind: Secret @@ -1246,7 +1246,7 @@ spec: metadata: annotations: checksum/cluster-resource-templates: 6fd9b172465e3089fcc59f738b92b8dc4d8939360c19de8ee65f68b0e7422035 - checksum/configuration: 475406181c84abf6c22db03375314bebedd360d52cc923e32579238d93075b2b + checksum/configuration: 9ab632fe7ac69bcf63d6965a44986a05e23798beda4a3175d1601e61057a9832 checksum/configuration-secret: 09216ffaa3d29e14f88b1f30af580d02a2a5e014de4d750b7f275cc07ed4e914 labels: app.kubernetes.io/component: flyte-binary @@ -1412,7 +1412,7 @@ spec: metadata: annotations: checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81 - checksum/secret: 883bf21ceceed4d8d6b24949d400a6df8eb33b71e5056782a702fcf3baaa7f01 + checksum/secret: d57403ae8ea0fce27bceda25f6af446fe51652e99e95a07fddae387006ee29f1 labels: app: docker-registry release: flyte-sandbox @@ -1755,7 +1755,7 @@ spec: value: minio - name: FLYTE_AWS_SECRET_ACCESS_KEY value: miniostorage - image: ghcr.io/flyteorg/flyteagent:1.10.3 + image: ghcr.io/flyteorg/flyteagent:1.10.7 imagePullPolicy: IfNotPresent name: flyteagent ports: diff --git a/docker/sandbox-bundled/manifests/complete.yaml b/docker/sandbox-bundled/manifests/complete.yaml index 8bd0ca2b00..b56e367ac4 100644 --- a/docker/sandbox-bundled/manifests/complete.yaml +++ b/docker/sandbox-bundled/manifests/complete.yaml @@ -457,7 +457,7 @@ data: stackdriver-enabled: false k8s: co-pilot: - image: "cr.flyte.org/flyteorg/flytecopilot:v1.10.7" + image: "cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0" k8s-array: logs: config: @@ -796,7 +796,7 @@ type: Opaque --- apiVersion: v1 data: - haSharedSecret: aGtXbUVsYnhhcVRRS0RwRA== + haSharedSecret: T21pWVJOUEdxMXBTSVE1RQ== proxyPassword: "" proxyUsername: "" kind: Secret @@ -1194,7 +1194,7 @@ spec: metadata: annotations: checksum/cluster-resource-templates: 6fd9b172465e3089fcc59f738b92b8dc4d8939360c19de8ee65f68b0e7422035 - checksum/configuration: ebc0c801b378ad16b6df2e54a8796fb57e71130935130b9f8e3201faf2fd09e2 + checksum/configuration: 11cd65708fd872839c6e561e84c30e045567486f06757f4549c69cc22aea5697 checksum/configuration-secret: 09216ffaa3d29e14f88b1f30af580d02a2a5e014de4d750b7f275cc07ed4e914 labels: app.kubernetes.io/component: flyte-binary @@ -1360,7 +1360,7 @@ spec: metadata: annotations: checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81 - checksum/secret: 9f699df433a7f3227784261437025f01a0ddb97d1514041ab1d3a93533b70135 + checksum/secret: b0e1d465fbab24856443e463cb7846c898d03f1e00ac443b08e5474d28418ba3 labels: app: docker-registry release: flyte-sandbox diff --git a/docker/sandbox-bundled/manifests/dev.yaml b/docker/sandbox-bundled/manifests/dev.yaml index a5fe4c4109..2a8383a1dd 100644 --- a/docker/sandbox-bundled/manifests/dev.yaml +++ b/docker/sandbox-bundled/manifests/dev.yaml @@ -499,7 +499,7 @@ metadata: --- apiVersion: v1 data: - haSharedSecret: R2NGVWU3dmpId2prNHFlbw== + haSharedSecret: bGRYdlJtdmZ5Qm14ZEJnNg== proxyPassword: "" proxyUsername: "" kind: Secret @@ -934,7 +934,7 @@ spec: metadata: annotations: checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81 - checksum/secret: ddccf9a515ebaf4fcc214a064ef0223cca9d7c0b063247810d7f1e5c5ef51311 + checksum/secret: b3f9230da427e818d5a63cbbf15159f2b165c98e6f56e269983c0a8fff6b6099 labels: app: docker-registry release: flyte-sandbox diff --git a/docs/community/contribute.rst b/docs/community/contribute.rst index 12cbf38b01..e866be5a2c 100644 --- a/docs/community/contribute.rst +++ b/docs/community/contribute.rst @@ -282,7 +282,7 @@ The resulting ``html`` files will be in ``docs/_build/html``. You can view them * - **Purpose**: Examples, Tips, and Tricks to use Flytekit SDKs * - **Language**: Python (In the future, Java examples will be added) * - **Guidelines**: Refer to the `Flytesnacks Contribution Guide `__ - + ``flytectl`` ************ @@ -291,7 +291,7 @@ The resulting ``html`` files will be in ``docs/_build/html``. You can view them * - `Repo `__ * - **Purpose**: A standalone Flyte CLI * - **Language**: Go - * - **Guidelines**: Refer to the `FlyteCTL Contribution Guide `__ + * - **Guidelines**: Refer to the `FlyteCTL Contribution Guide `__ 🔮 Development Environment Setup Guide @@ -677,7 +677,7 @@ You can access it via http://localhost:30080/console. Core Flyte components, such as admin, propeller, and datacatalog, as well as user runtime containers rely on an object store (in this case, minio) to hold files. -During development, you might need to examine files such as `input.pb/output.pb `__, or `deck.html `__ stored in minio. +During development, you might need to examine files such as `input.pb/output.pb `__, or `deck.html `__ stored in minio. Access the minio console at: http://localhost:30080/minio/login. The default credentials are: diff --git a/docs/community/troubleshoot.rst b/docs/community/troubleshoot.rst index b4f6c271d4..1228b5f5a0 100644 --- a/docs/community/troubleshoot.rst +++ b/docs/community/troubleshoot.rst @@ -133,3 +133,21 @@ Example output: $ kubectl annotate serviceaccount -n eks.amazonaws.com/role-arn=arn:aws:iam::xxxx:role/ - Refer to this community-maintained `guides `_ for further information about Flyte deployment on EKS + +``FlyteScopedUserException: 'JavaPackage' object is not callable`` when running a Spark task +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Please add ``spark`` to the list of `enabled-plugins` in the config yaml file. For example, + +.. code-block:: yaml + + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - K8S-ARRAY + - spark + default-for-task-types: + - container: container + - container_array: K8S-ARRAY diff --git a/docs/concepts/admin.rst b/docs/concepts/admin.rst index 4e6ee67a8e..f33ef51364 100644 --- a/docs/concepts/admin.rst +++ b/docs/concepts/admin.rst @@ -237,44 +237,6 @@ Permitted project operations include: - Register - List -.. _divedeep-admin-matchable-resources: - -Matchable resources -+++++++++++++++++++ - -A thorough background on :ref:`matchable resources ` explains -their purpose and application logic. As a summary, these are used to override system level defaults for Kubernetes cluster -resource management, default execution values, and more across different levels of specificity. - -These entities consist of: - -- ProjectDomainAttributes -- WorkflowAttributes - -``ProjectDomainAttributes`` configure customizable overrides at the project and domain level, and ``WorkflowAttributes`` configure customizable overrides at the project, domain and workflow level. - -Permitted attribute operations include: - -- Update (implicitly creates if there is no existing override) -- Get -- Delete - - -Defaults --------- - -Task resource defaults -++++++++++++++++++++++ - -User-facing documentation on configuring task resource requests and limits can be found in :std:ref:`cookbook:customizing task resources`. - -As a system administrator you may want to define default task resource requests and limits across your Flyte deployment. -This can be done through the flyteadmin config. - -**Default** values get injected as the task requests and limits when a task definition omits a specific resource. -**Limit** values are only used as validation. Neither a task request nor limit can exceed the limit for a resource type. - - Using the Admin Service ----------------------- diff --git a/docs/concepts/tasks.rst b/docs/concepts/tasks.rst index f3ae87709e..94807d3632 100644 --- a/docs/concepts/tasks.rst +++ b/docs/concepts/tasks.rst @@ -30,7 +30,7 @@ When deciding if a unit of execution constitutes a Flyte task, consider these qu - Is there a well-defined graceful/successful exit criteria for the task? A task is expected to exit after completion of input processing. - Is it repeatable? Under certain circumstances, a task might be retried, rerun, etc. with the same inputs. It is expected - to produce the same output every single time. For example, avoid using random number generators with current clock as seed. Use a system-provided clock as the seed instead. + to produce the same output every single time. For example, avoid using random number generators with current clock as seed. Use a system-provided clock as the seed instead. - Is it a pure function, i.e., does it have side effects that are unknown to the system (calls a web-service)? It is recommended to avoid side-effects in tasks. When side-effects are evident, ensure that the operations are idempotent. Dynamic Tasks @@ -38,7 +38,7 @@ Dynamic Tasks "Dynamic tasks" is a misnomer. Flyte is one-of-a-kind workflow engine that ships with the concept of truly `Dynamic Workflows `__! -Users can generate workflows in reaction to user inputs or computed values at runtime. +Users can generate workflows in reaction to user inputs or computed values at runtime. These executions are evaluated to generate a static graph before execution. Extending Task @@ -47,9 +47,9 @@ Extending Task Plugins ^^^^^^^ -Flyte exposes an extensible model to express tasks in an execution-independent language. -It contains first-class task plugins (for example: `Papermill `__, -`Great Expectations `__, and :ref:`more `.) +Flyte exposes an extensible model to express tasks in an execution-independent language. +It contains first-class task plugins (for example: `Papermill `__, +`Great Expectations `__, and :ref:`more `.) that execute the Flyte tasks. Almost any action can be implemented and introduced into Flyte as a "Plugin", which includes: @@ -58,7 +58,7 @@ Almost any action can be implemented and introduced into Flyte as a "Plugin", wh - Tasks that call web services. Flyte ships with certain defaults, for example, running a simple Python function does not need any hosted service. Flyte knows how to -execute these kinds of tasks on Kubernetes. It turns out these are the vast majority of tasks in machine learning, and Flyte is adept at +execute these kinds of tasks on Kubernetes. It turns out these are the vast majority of tasks in machine learning, and Flyte is adept at handling an enormous scale on Kubernetes. This is achieved by implementing a unique scheduler on Kubernetes. Types @@ -74,14 +74,14 @@ Inherent Features Fault tolerance ^^^^^^^^^^^^^^^ -In any distributed system, failure is inevitable. Allowing users to design a fault-tolerant system (e.g. workflow) is an inherent goal of Flyte. +In any distributed system, failure is inevitable. Allowing users to design a fault-tolerant system (e.g. workflow) is an inherent goal of Flyte. At a high level, tasks offer two parameters to achieve fault tolerance: **Retries** - -Tasks can define a retry strategy to let the system know how to handle failures (For example: retry 3 times on any kind of error). -There are two kinds of retries: +Tasks can define a retry strategy to let the system know how to handle failures (For example: retry 3 times on any kind of error). + +There are two kinds of retries: 1. System retry: It is a system-defined, recoverable failure that is used when system failures occur. The number of retries is validated against the number of system retries. @@ -91,7 +91,7 @@ System retry can be of two types: - **Downstream System Retry**: When a downstream system (or service) fails, or remote service is not contactable, the failure is retried against the number of retries set `here `__. This performs end-to-end system retry against the node whenever the task fails with a system error. This is useful when the downstream service throws a 500 error, abrupt network failure, etc. -- **Transient Failure Retry**: This retry mechanism offers resiliency against transient failures, which are opaque to the user. It is tracked across the entire duration of execution. It helps Flyte entities and the additional services connected to Flyte like S3, to continue operating despite a system failure. Indeed, all transient failures are handled gracefully by Flyte! Moreover, in case of a transient failure retry, Flyte does not necessarily retry the entire task. “Retrying an entire task” means that the entire pod associated with the Flyte task would be rerun with a clean slate; instead, it just retries the atomic operation. For example, Flyte tries to persist the state until it can, exhausts the max retries, and backs off. +- **Transient Failure Retry**: This retry mechanism offers resiliency against transient failures, which are opaque to the user. It is tracked across the entire duration of execution. It helps Flyte entities and the additional services connected to Flyte like S3, to continue operating despite a system failure. Indeed, all transient failures are handled gracefully by Flyte! Moreover, in case of a transient failure retry, Flyte does not necessarily retry the entire task. “Retrying an entire task” means that the entire pod associated with the Flyte task would be rerun with a clean slate; instead, it just retries the atomic operation. For example, Flyte tries to persist the state until it can, exhausts the max retries, and backs off. To set a transient failure retry: @@ -102,17 +102,17 @@ System retry can be of two types: 2. User retry: If a task fails to execute, it is retried for a specific number of times, and this number is set by the user in `TaskMetadata `__. The number of retries must be less than or equal to 10. .. note:: - + Recoverable vs. Non-Recoverable failures: Recoverable failures will be retried and counted against the task's retry count. Non-recoverable failures will just fail, i.e., the task isn’t retried irrespective of user/system retry configurations. All user exceptions are considered non-recoverable unless the exception is a subclass of FlyteRecoverableException. .. note:: - `RFC 3902 `_ implements an alternative, simplified retry behaviour with which both system and user retries are counted towards a single retry budget defined in the task decorator (thus, without a second retry budget defined in the platform configuration). The last retries are always performed on non-spot instances to guarantee completion. To activate this behaviour, set ``configmap.core.propeller.node-config.ignore-retry-cause`` to ``true`` in the helm values. + `RFC 3902 `_ implements an alternative, simplified retry behavior with which both system and user retries are counted towards a single retry budget defined in the task decorator (thus, without a second retry budget defined in the platform configuration). The last retries are always performed on non-spot instances to guarantee completion. To activate this behaviour, set ``configmap.core.propeller.node-config.ignore-retry-cause`` to ``true`` in the helm values. **Timeouts** - + To ensure that the system is always making progress, tasks must be guaranteed to end gracefully/successfully. The system defines a default timeout period for the tasks. It is possible for task authors to define a timeout period, after which the task is marked as ``failure``. Note that a timed-out task will be retried if it has a retry strategy defined. The timeout can be handled in the `TaskMetadata `__. @@ -120,4 +120,4 @@ Caching/Memoization ^^^^^^^^^^^^^^^^^^^ Flyte supports memoization of task outputs to ensure that identical invocations of a task are not executed repeatedly, thereby saving compute resources and execution time. For example, if you wish to run the same piece of code multiple times, you can reuse the output instead of re-computing it. -For more information on memoization, refer to the :std:doc:`Caching Example `. +For more information on memoization, refer to the :std:doc:`/user_guide/development_lifecycle/caching`. diff --git a/docs/conf.py b/docs/conf.py index d9e38e5806..b4666d94a4 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -35,7 +35,7 @@ # The short X.Y version version = "" # The full version, including alpha/beta/rc tags -release = "1.10.7" +release = "1.11.0-b0" # -- General configuration --------------------------------------------------- @@ -313,6 +313,7 @@ # These patterns are used to replace values in source files that are imported # from other repos. REPLACE_PATTERNS = { + r"": r"", r"": r"", INTERSPHINX_REFS_PATTERN: INTERSPHINX_REFS_REPLACE, @@ -328,17 +329,18 @@ PROTO_REF_PATTERN: PROTO_REF_REPLACE, r"/protos/docs/service/index": r"/protos/docs/service/service", r"": r"", - r"": r"" } +# r"": r"", + import_projects_config = { "clone_dir": "_projects", "flytekit_api_dir": "_src/flytekit/", "source_regex_mapping": REPLACE_PATTERNS, "list_table_toc": [ - "flytesnacks/userguide", - "flytesnacks/tutorials", - "flytesnacks/integrations", + "flytesnacks/tutorials", + "flytesnacks/integrations", + "flytesnacks/deprecated_integrations" ], "dev_build": bool(int(os.environ.get("MONODOCS_DEV_BUILD", 1))), } @@ -369,6 +371,25 @@ "flytesnacks/_build", "flytesnacks/_tags", "flytesnacks/getting_started", + "flytesnacks/userguide.md", + "flytesnacks/environment_setup.md", + "flytesnacks/index.md", + "examples/advanced_composition", + "examples/basics", + "examples/customizing_dependencies", + "examples/data_types_and_io", + "examples/development_lifecycle", + "examples/extending", + "examples/productionizing", + "examples/testing", + "flytesnacks/examples/advanced_composition", + "flytesnacks/examples/basics", + "flytesnacks/examples/customizing_dependencies", + "flytesnacks/examples/data_types_and_io", + "flytesnacks/examples/development_lifecycle", + "flytesnacks/examples/extending", + "flytesnacks/examples/productionizing", + "flytesnacks/examples/testing", ] ], "local": flytesnacks_local_path is not None, diff --git a/docs/core_use_cases/analytics.md b/docs/core_use_cases/analytics.md index 58b6ab770c..886b75618d 100644 --- a/docs/core_use_cases/analytics.md +++ b/docs/core_use_cases/analytics.md @@ -173,7 +173,7 @@ and [DBT](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt) integrations. If you need to connect to a database, Flyte provides first-party -support for {ref}`AWS Athena `, {ref}`Google Bigquery `, -{ref}`Snowflake `, {ref}`SQLAlchemy `, and +support for {ref}`AWS Athena `, {ref}`Google Bigquery `, +{ref}`Snowflake `, {ref}`SQLAlchemy `, and {ref}`SQLite3 `. ``` diff --git a/docs/core_use_cases/data_engineering.md b/docs/core_use_cases/data_engineering.md index 25eb802fc2..9cbfca430c 100644 --- a/docs/core_use_cases/data_engineering.md +++ b/docs/core_use_cases/data_engineering.md @@ -170,6 +170,6 @@ and [DBT](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt) integrations. For database connectors, Flyte provides first-party support for {ref}`AWS Athena `, -{ref}`Google Bigquery `, {ref}`Snowflake `, +{ref}`Google BigQuery `, {ref}`Snowflake `, {ref}`SQLAlchemy `, and {ref}`SQLite3 `. ``` diff --git a/docs/deployment/agents/airflow.rst b/docs/deployment/agents/airflow.rst new file mode 100644 index 0000000000..ad6a6dab36 --- /dev/null +++ b/docs/deployment/agents/airflow.rst @@ -0,0 +1,97 @@ +.. _deployment-agent-setup-airflow: + +Airflow agent +================= + +This guide provides an overview of how to set up the Airflow agent in your Flyte deployment. +Please note that the you don't need an Airflow cluster to run the Airflow tasks, since Flytekit will +automatically compile Airflow tasks to Flyte tasks and execute them on the Flyte cluster. + +Specify agent configuration +---------------------------- + +.. tabs:: + + .. group-tab:: Flyte binary + + Edit the relevant YAML file to specify the agent. + + .. code-block:: bash + + kubectl edit configmap flyte-sandbox-config -n flyte + + .. code-block:: yaml + :emphasize-lines: 7,11,16 + + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + - container: container + - container_array: k8s-array + - airflow: agent-service + + plugins: + agent-service: + supportedTaskTypes: + - airflow + + .. group-tab:: Flyte core + + Create a file named ``values-override.yaml`` and add the following configuration to it. + + .. code-block:: yaml + + configmap: + enabled_plugins: + # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) + tasks: + # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) + task-plugins: + # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + container: container + sidecar: sidecar + container_array: k8s-array + airflow: agent-service + plugins: + agent-service: + supportedTaskTypes: + - airflow + + +Upgrade the Flyte Helm release +------------------------------ + +.. tabs:: + + .. group-tab:: Flyte binary + + .. code-block:: bash + + helm upgrade flyteorg/flyte-binary -n --values + + Replace ```` with the name of your release (e.g., ``flyte-backend``), + ```` with the name of your namespace (e.g., ``flyte``), + and ```` with the name of your YAML file. + + .. group-tab:: Flyte core + + .. code-block:: bash + + helm upgrade flyte/flyte-core -n --values values-override.yaml + + Replace ```` with the name of your release (e.g., ``flyte``) + + and ```` with the name of your namespace (e.g., ``flyte``). + +For Airflow agent on the Flyte cluster, see `Airflow agent `_. diff --git a/docs/deployment/agents/bigquery.rst b/docs/deployment/agents/bigquery.rst index 9835c3d47a..d706ac7c37 100644 --- a/docs/deployment/agents/bigquery.rst +++ b/docs/deployment/agents/bigquery.rst @@ -1,6 +1,6 @@ .. _deployment-agent-setup-bigquery: -Google BigQuery Agent +Google BigQuery agent ====================== This guide provides an overview of setting up BigQuery agent in your Flyte deployment. @@ -103,4 +103,4 @@ Upgrade the Flyte Helm release and ```` with the name of your namespace (e.g., ``flyte``). -For BigQuery plugin on the Flyte cluster, please refer to `BigQuery Plugin Example `_ +For BigQuery agent on the Flyte cluster, see `BigQuery agent `_. diff --git a/docs/deployment/agents/databricks.rst b/docs/deployment/agents/databricks.rst index 00a5e97a47..3dbf7731c5 100644 --- a/docs/deployment/agents/databricks.rst +++ b/docs/deployment/agents/databricks.rst @@ -1,6 +1,6 @@ .. _deployment-agent-setup-databricks: -Databricks Agent +Databricks agent ================= This guide provides an overview of how to set up Databricks agent in your Flyte deployment. @@ -291,4 +291,4 @@ Wait for the upgrade to complete. You can check the status of the deployment pod kubectl get pods -n flyte -For databricks plugin on the Flyte cluster, please refer to `Databricks Plugin Example `_ +For Databricks agent on the Flyte cluster, see `Databricks agent `_. diff --git a/docs/deployment/agents/index.md b/docs/deployment/agents/index.md index e27644570a..0e114c8d06 100644 --- a/docs/deployment/agents/index.md +++ b/docs/deployment/agents/index.md @@ -2,22 +2,29 @@ # Agent Setup -.. tags:: Agent, Integration, Data, Advanced +```{tags} Agent, Integration, Data, Advanced +``` + +To set configure your Flyte deployment for agents, see the documentation below. -Discover the process of setting up Agents for Flyte. +:::{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment. +::: ```{list-table} :header-rows: 0 :widths: 20 30 -* - {ref}`Bigquery Agent ` - - Guide to setting up the Bigquery agent. +* - {ref}`Airflow Agent ` + - Configuring your Flyte deployment for the Airflow agent +* - {ref}`Databricks Agent ` + - Configuring your Flyte deployment for the Databricks agent. +* - {ref}`Google BigQuery Agent ` + - Configuring your Flyte deployment for the BigQuery agent. * - {ref}`MMCloud Agent ` - - Guide to setting up the MMCloud agent. + - Configuring your Flyte deployment for the MMCloud agent. * - {ref}`Sensor Agent ` - - Guide to setting up the Sensor agent. -* - {ref}`Databricks Agent ` - - Guide to setting up the Databricks agent. + - Configuring your Flyte deployment for the sensor agent. ``` ```{toctree} @@ -25,8 +32,10 @@ Discover the process of setting up Agents for Flyte. :name: Agent setup :hidden: +airflow +databricks bigquery mmcloud -databricks sensor +snowflake ``` diff --git a/docs/deployment/agents/mmcloud.rst b/docs/deployment/agents/mmcloud.rst index 217beab8ed..ac08f4fcdf 100644 --- a/docs/deployment/agents/mmcloud.rst +++ b/docs/deployment/agents/mmcloud.rst @@ -118,4 +118,4 @@ Wait for the upgrade to complete. You can check the status of the deployment pod kubectl get pods -n flyte -For MMCloud plugin on the Flyte cluster, please refer to `Memory Machine Cloud Plugin Example `_ +For MMCloud agent on the Flyte cluster, see `MMCloud agent `_. diff --git a/docs/deployment/agents/sensor.rst b/docs/deployment/agents/sensor.rst index ecb45e426f..958e5d896a 100644 --- a/docs/deployment/agents/sensor.rst +++ b/docs/deployment/agents/sensor.rst @@ -1,13 +1,13 @@ .. _deployment-agent-setup-sensor: -Sensor Agent +Sensor agent ================= -Sensor enables users to continuously check for a file or a condition to be met periodically. +The `sensor agent `_ enables users to continuously check for a file or a condition to be met periodically. When the condition is met, the sensor will complete. -This guide provides an overview of how to set up Sensor in your Flyte deployment. +This guide provides an overview of how to set up the sensor agent in your Flyte deployment. Spin up a cluster ----------------- @@ -43,7 +43,7 @@ Spin up a cluster Specify agent configuration ---------------------------- -Enable the Sensor agent by adding the following config to the relevant YAML file(s): +Enable the sensor agent by adding the following config to the relevant YAML file(s): .. tabs:: @@ -77,7 +77,7 @@ Enable the Sensor agent by adding the following config to the relevant YAML file .. group-tab:: Flyte core - Create a file named ``values-override.yaml`` and add the following configuration to it. + Create a file named ``values-override.yaml`` and add the following configuration to it: .. code-block:: yaml diff --git a/docs/deployment/agents/snowflake.rst b/docs/deployment/agents/snowflake.rst new file mode 100644 index 0000000000..f4d82c0eb2 --- /dev/null +++ b/docs/deployment/agents/snowflake.rst @@ -0,0 +1,103 @@ +.. _deployment-agent-setup-snowflake: + +Snowflake agent +================= + +This guide provides an overview of how to set up the Snowflake agent in your Flyte deployment. + +1. Set up the key pair authentication in Snowflake. For more details, see the `Snowflake key-pair authentication and key-pair rotation guide `__. +2. Create a secret with the group "snowflake" and the key "private_key". For more details, see `"Using Secrets in a Task" `__. + +.. code-block:: bash + + kubectl create secret generic snowflake-private-key --namespace=flytesnacks-development --from-file=your_private_key_above + +Specify agent configuration +---------------------------- + +.. tabs:: + + .. group-tab:: Flyte binary + + Edit the relevant YAML file to specify the agent. + + .. code-block:: bash + + kubectl edit configmap flyte-sandbox-config -n flyte + + .. code-block:: yaml + :emphasize-lines: 7,11,16 + + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + - container: container + - container_array: k8s-array + - snowflake: agent-service + + plugins: + agent-service: + supportedTaskTypes: + - snowflake + + .. group-tab:: Flyte core + + Create a file named ``values-override.yaml`` and add the following configuration to it. + + .. code-block:: yaml + + configmap: + enabled_plugins: + # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) + tasks: + # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) + task-plugins: + # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + container: container + sidecar: sidecar + container_array: k8s-array + snowflake: agent-service + plugins: + agent-service: + supportedTaskTypes: + - snowflake + +Ensure that the propeller has the correct service account for BigQuery. + +Upgrade the Flyte Helm release +------------------------------ + +.. tabs:: + + .. group-tab:: Flyte binary + + .. code-block:: bash + + helm upgrade flyteorg/flyte-binary -n --values + + Replace ```` with the name of your release (e.g., ``flyte-backend``), + ```` with the name of your namespace (e.g., ``flyte``), + and ```` with the name of your YAML file. + + .. group-tab:: Flyte core + + .. code-block:: bash + + helm upgrade flyte/flyte-core -n --values values-override.yaml + + Replace ```` with the name of your release (e.g., ``flyte``) + + and ```` with the name of your namespace (e.g., ``flyte``). + +For Snowflake agent on the Flyte cluster, see `Snowflake agent `_. diff --git a/docs/deployment/configuration/customizable_resources.rst b/docs/deployment/configuration/customizable_resources.rst index 5e41863a7a..6fb1318ac6 100644 --- a/docs/deployment/configuration/customizable_resources.rst +++ b/docs/deployment/configuration/customizable_resources.rst @@ -1,12 +1,292 @@ .. _deployment-configuration-customizable-resources: -################################# -Adding New Customizable Resources -################################# +################################################################# +Customizing project, domain, and workflow resources with flytectl +################################################################# + +For critical projects and workflows, you can use the :ref:`flytectl update ` command to configure +settings for task, cluster, and workflow execution resources, set matching executions to execute on specific clusters, set execution queue attributes, and :ref:`more ` +that differ from the default values set for your global Flyte installation. These customizable settings are created, updated, and deleted via the API and stored in the FlyteAdmin database. + +In code, these settings are sometimes called `matchable attributes` or `matchable resources`, because we use a hierarchy for matching the customizations to applicable Flyte inventory and executions. + +******************************* +Configuring existing resources +******************************* + + +About the resource hierarchy +============================ + +Many platform specifications set in the FlyteAdmin config are applied to every project and domain. Although these values are customizable as part of your helm installation, they are still applied to every user project and domain combination. + +You can choose to customize these settings along increasing levels of specificity with Flyte: + +- Domain +- Project and Domain +- Project, Domain, and Workflow name +- Project, Domain, Workflow name and LaunchPlan name + +See :ref:`control-plane` to understand projects and domains. +The following section will show you how to configure the settings along +these dimensions. + +Task resources +============== + +As a system administrator you may want to define default task resource requests and limits across your Flyte deployment. This can be set globally in the FlyteAdmin `config `__ +in `task_resource_defaults`. + +**Default** values get injected as the task requests and limits when a task definition omits a specific :py:class:`resource `. + +**Limit** values are only used as validation. Neither a task request nor limit can exceed the limit for a resource type. + +Configuring task resources +-------------------------- +Available resources for configuration include: + +- CPU +- GPU +- Memory +- `Ephemeral Storage `__ + +In the absence of a customization, the global +`default values `__ +in `task_resource_defaults` are used. + +The customized values from the database are assigned at execution, rather than registration time. + +Customizing task resource configuration +--------------------------------------- + +To customize resources for project-domain attributes using `flytectl`, define a ``tra.yaml`` file with your customizations: + +.. code-block:: yaml + + project: flyteexamples + domain: development + defaults: + cpu: "1" + memory: 150Mi + limits: + cpu: "2" + memory: 450Mi + +Update the task resource attributes for a project-domain combination: + +.. prompt:: bash $ + + flytectl update task-resource-attribute --attrFile tra.yaml + +.. note:: + + Refer to the :ref:`docs ` to + learn more about the command and its supported flag(s). + +To fetch and verify the individual project-domain attributes: + +.. prompt:: bash $ + + flytectl get task-resource-attribute -p flyteexamples -d development + +.. note:: + + Refer to the :ref:`docs ` to learn + more about the command and its supported flag(s). + +You can view all custom task-resource-attributes by visiting +``protocol://`` and substitute +the protocol and host appropriately. + +Cluster resources +================= + +Cluster resources are how you configure Kubernetes namespace attributes that are applied at execution time. This includes per-namespace resource quota, patching the default service account with a bounded IAM role, or attaching `imagePullSecrets` to the default service account for accessing a private container registry + + +Configuring cluster resources +----------------------------- +The format of all these parameters are free-form key-value pairs used for populating the Kubernetes object templates consumed by the cluster resource controller. The cluster resource controller ensures these fully rendered object templates are applied as Kubernetes resources for each execution namespace. + +The keys represent templatized variables in the +`cluster resource template `__ +and the values are what you want to see filled in. + +In the absence of custom customized values, your Flyte installation will use ``templateData`` from the +`FlyteAdmin config `__ +as the per-domain defaults. Flyte specifies these defaults by domain and applies them to every +project-domain namespace combination. + + +Customizing cluster resource configuration +------------------------------------------ +.. note:: + The cluster resource template values can be specified on domain, and project-and-domain. + Since Flyte execution namespaces are never on a per-workflow or a launch plan basis, specifying a workflow or launch plan level customization is non-actionable. + This is a departure from the usual hierarchy for customizable resources. + + +Define an attributes file, ``cra.yaml``: + +.. code-block:: yaml + + domain: development + project: flyteexamples + attributes: + projectQuotaCpu: "1000" + projectQuotaMemory: 5Ti + +To ensure that the customizations reflect in the Kubernetes namespace +``flyteexamples-development`` (that is, the namespace has a resource quota of +1000 CPU cores and 5TB of memory) when the admin fills in cluster resource +templates: + +.. prompt:: bash $ + + flytectl update cluster-resource-attribute --attrFile cra.yaml + +.. note:: + + Refer to the :ref:`docs ` + to learn more about the command and its supported flag(s). + +To fetch and verify the individual project-domain attributes: + +.. prompt:: bash $ + + flytectl get cluster-resource-attribute -p flyteexamples -d development + +.. note:: + + Refer to the :ref:`docs ` to + learn more about the command and its supported flag(s). + +Flyte uses these updated values to fill the template fields for the +``flyteexamples-development`` namespace. + +For other namespaces, the +`platform defaults `__ +apply. + +.. note:: + The template values, for example, ``projectQuotaCpu`` or ``projectQuotaMemory`` are free-form strings. + Ensure that they match the template placeholders in your `template file `__ + for your changes to take effect and custom values to be substituted. + +You can view all custom cluster-resource-attributes by visiting ``protocol://`` +and substitute the protocol and host appropriately. + + +Workflow execution configuration +================================ + + +Although many execution-time parameters can be overridden at execution time itself, it is helpful to set defaults on a per-project or per-workflow basis. This config includes +- `annotations and labels `__ +etc. in the `Workflow execution config `__. +- `max_parallelism`: Limits maximum number of nodes that can be evaluated for an individual workflow in parallel +- `security context `__: configures the pod identity and auth credentials for task pods at execution time +- `raw_output_data_config`: where offloaded user data is stored +- `interruptible`: whether to use [spot instances](https://docs.flyte.org/en/user_guide/productionizing/spot_instances.html) +- `overwrite_cache`: Allows for all cached values of a workflow and its tasks to be overwritten for a single execution. +- `envs`: Custom environment variables to apply for task pods brought up during execution + +Customizing workflow execution configuration +-------------------------------------------- + +These can be defined at two levels of project-domain or project-domain-workflow: + +.. prompt:: bash $ + + flytectl update workflow-execution-config + +.. note:: + + Refer to the :ref:`docs ` + to learn more about the command and its supported flag(s). + +Execution cluster label +======================= + +This matchable attributes allows forcing a matching execution to consistently execute on a specific Kubernetes cluster for multi-cluster Flyte deployment set-up. In lieu of an explicit customization, cluster assignment is random. + +For setting up a multi-cluster environment, follow :ref:`the guide ` + + +Customizing execution cluster label configuration +------------------------------------------------- + +Define an attributes file in `ec.yaml`: + +.. code-block:: yaml + + value: mycluster + domain: development + project: flyteexamples + +Ensure that admin places executions in the flyteexamples project and development domain onto ``mycluster``: + +.. prompt:: bash $ + + flytectl update execution-cluster-label --attrFile ec.yaml + +.. note:: + + Refer to the :ref:`docs ` + to learn more about the command and its supported flag(s). + +To fetch and verify the individual project-domain attributes: + +.. prompt:: bash $ + + flytectl get execution-cluster-label -p flyteexamples -d development + +.. note:: + + Refer to the :ref:`docs ` to + learn more about the command and its supported flag(s). + +You can view all custom execution cluster attributes by visiting +``protocol://`` and substitute +the protocol and host appropriately. + +.. _deployment-customizable-resources-execution-queues: + +Execution queues +================ + +Execution queues are defined in +`FlyteAdmin config `__. +These are used for execution placement for constructs like AWS Batch. + +The **attributes** associated with an execution queue must match the **tags** +for workflow executions. The tags associated with configurable resources are +stored in the admin database. + +Customizing execution queue configuration +----------------------------------------- + +.. prompt:: bash $ + + flytectl update execution-queue-attribute + +.. note:: + + Refer to the :ref:`docs ` + to learn more about the command and its supported flag(s). + +You can view existing attributes for which tags can be assigned by visiting +``protocol:///api/v1/matchable_attributes?resource_type=2`` and substitute +the protocol and host appropriately. + + +********************************* +Adding new customizable resources +********************************* .. tags:: Infrastructure, Advanced -As a quick refresher, custom resources allow you to manage configurations for specific combinations of user projects, domains and workflows that override default values. +As a quick refresher, custom resources allow you to manage configurations for specific combinations of user projects, domains and workflows that customize default values. Examples of such resources include execution clusters, task resource defaults, and :std:ref:`more `. .. note:: @@ -16,8 +296,9 @@ In a :ref:`multi-cluster setup `, an example Here's how you could go about building a customizable priority designation. + Example -------- +======= Let's say you want to inject a default priority annotation for your workflows. Perhaps you start off with a model where everything has a default priority but soon you realize it makes sense that workflows in your production domain should take higher priority than those in your development domain. @@ -27,7 +308,7 @@ Now, one of your user teams requires critical workflows to have a higher priorit Here's how you could do that. Flyte IDL -^^^^^^^^^ +--------- Introduce a new :std:ref:`matchable resource ` that includes a unique enum value and proto message definition. @@ -56,7 +337,7 @@ See the changes in this `file `__ your new matchable priority resource and use it while creating executions or in relevant use cases. @@ -84,119 +365,8 @@ For example: Flytekit -^^^^^^^^ +-------- For convenience, add a FlyteCTL wrapper to update the new attributes. Refer to `this PR `__ for the entire set of changes required. That's it! You now have a new matchable attribute to configure as the needs of your users evolve. - -Flyte ResourceManager ---------------------- - -**Flyte ResourceManager** is a configurable component that allows plugins to manage resource allocations independently. It helps track resource utilization of tasks that run on Flyte. The default deployments are configured as ``noop``, which indicates that the ResourceManager provided by Flyte is disabled and plugins rely on each independent platform to manage resource utilization. In situations like the K8s plugin, where the platform has a robust mechanism to manage resource scheduling, this may work well. However, in a scenario like a simple web API plugin, the rate at which Flyte sends requests may overwhelm a service and benefit from additional resource management. - -The below attribute is configurable within FlytePropeller, which can be disabled with: - -.. code-block:: yaml - - resourcemanager: - type: noop - -The ResourceManager provides a task-type-specific pooling system for Flyte tasks. Optionally, plugin writers can request resource allocation in their tasks. - -A plugin defines a collection of resource pools using its configuration. Flyte uses tokens as a placeholder to represent a unit of resource. - -How does a Flyte plugin request for resources? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The Flyte plugin registers the resource and the desired quota of every resource with the **ResourceRegistrar** when setting up FlytePropeller. When a plugin is invoked, FlytePropeller provides a proxy for the plugin. This proxy facilitates the plugin's view of the resource pool by controlling operations to allocate and deallocate resources. - -.. dropdown:: :fa:`info-circle` Enabling Redis instance - :title: text-muted - :animate: fade-in-slide-down - - The ResourceManager can use a Redis instance as an external store to track and manage resource pool allocation. By default, it is disabled, and can be enabled with: - - .. code-block:: yaml - - resourcemanager: - type: redis - resourceMaxQuota: 100 - redis: - hostPaths: - - foo - hostKey: bar - maxRetries: 0 - -Once the setup is complete, FlytePropeller builds a ResourceManager based on the previously requested resource registration. Based on the plugin implementation's logic, resources are allocated and deallocated. - -During runtime, the ResourceManager: - -#. Allocates tokens to the plugin. -#. Releases tokens once the task is completed. - -How are resources allocated? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When a Flyte task execution needs to send a request to an external service, the plugin claims a unit of the corresponding resource. This is done using a **ResourceName**, which is a unique token and a fully qualified resource request (which is typically an integer). The execution generates this unique token and registers this token with the ResourceManager by calling the ResourceManager’s **"AllocateResource function"**. If the resource pool has sufficient capacity to fulfil your request, then the resources requested are allocated, and the plugin proceeds further. - -When the status is **"AllocationGranted"**, the execution moves forward and sends out the request for those resources. - -The granted token is recorded in a token pool which corresponds to the resource that is managed by the ResourceManager. - -How are resources deallocated? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -When the request is completed, the plugin asks the ResourceManager to release the token by calling the ReleaseResource() function present in the ResourceManager. Upon calling the function, the token is eliminated from the token pool. -In this manner, Flyte plugins intelligently throttle resource usage during parallel execution of nodes. - -Example -^^^^^^^^ -Let's take an example to understand resource allocation and deallocation when a plugin requests resources. - -Flyte has a built-in `Qubole `__ plugin. This plugin allows Flyte tasks to send Hive commands to Qubole. In the plugin, a single Qubole cluster is considered a resource, and sending a single Hive command to a Qubole cluster consumes a token of the corresponding resource. -The resource is allocated when the status is **“AllocationGranted”**. Qubole plugin calls: - -.. code-block:: go - - status, err := AllocateResource(ctx, , , ) - -Wherein the placeholders are occupied by: - -.. code-block:: go - - status, err := AllocateResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0", ResourceConstraintsSpec{}) - -The resource is deallocated when the Hive command completes its execution and the corresponding token is released. The plugin calls: - -.. code-block:: go - - status, err := AllocateResource(ctx, , , ) - -Wherein the placeholders are occupied by: - -.. code-block:: go - - err := ReleaseResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0") - -Below is an example interface that shows allocation and deallocation of resources. - -.. code-block:: go - - type ResourceManager interface { - GetID() string - // During execution, the plugin calls AllocateResource() to register a token in the token pool associated with a resource - // If it is granted an allocation, the token is recorded in the token pool until the same plugin releases it. - // When calling AllocateResource, the plugin has to specify a ResourceConstraintsSpec that contains resource capping constraints at different project and namespace levels. - // The ResourceConstraint pointers in ResourceConstraintsSpec can be set to nil to not have a constraint at that level - AllocateResource(ctx context.Context, namespace ResourceNamespace, allocationToken string, constraintsSpec ResourceConstraintsSpec) (AllocationStatus, error) - // During execution, after an outstanding request is completed, the plugin uses ReleaseResource() to release the allocation of the token from the token pool. This way, it redeems the quota taken by the token - ReleaseResource(ctx context.Context, namespace ResourceNamespace, allocationToken string) error - } - -How can you force ResourceManager to force runtime quota allocation constraints? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Runtime quota allocation constraints can be achieved using ResourceConstraintsSpec. It is a contact that a plugin can specify at different project and namespace levels. - -Let's take an example to understand it. - -You can set ResourceConstraintsSpec to ``nil`` objects, which means there would be no allocation constraints at the respective project and namespace level. When ResourceConstraintsSpec specifies ``nil`` ProjectScopeResourceConstraint, and a non-nil NamespaceScopeResourceConstraint, it suggests no constraints specified at any project or namespace level. diff --git a/docs/deployment/configuration/general.rst b/docs/deployment/configuration/general.rst index 828ff3871b..b5278365d2 100644 --- a/docs/deployment/configuration/general.rst +++ b/docs/deployment/configuration/general.rst @@ -1,319 +1,31 @@ .. _deployment-configuration-general: -################################# -Configuring Custom K8s Resources -################################# - -*************************** -Configurable Resource Types -*************************** - -Many platform specifications such as task resource defaults, project namespace Kubernetes quota, and more can be -assigned using default values or custom overrides. Defaults are specified in the FlyteAdmin config and -overrides for specific projects are registered with the FlyteAdmin service. - -You can customize these settings along increasing levels of specificity with Flyte: - -- Domain -- Project and Domain -- Project, Domain, and Workflow name -- Project, Domain, Workflow name and LaunchPlan name - -See :ref:`control-plane` to understand projects and domains. -The following section will show you how to configure the settings along -these dimensions. - -Task Resources -============== - -Configuring task :py:class:`resources ` includes -setting default values for unspecified task requests and limits. Task resources -also include limits which specify the maximum value that a task request or a limit can have. - -- CPU -- GPU -- Memory -- Storage -- `Ephemeral Storage `__ - -In the absence of an override, the global -`default values `__ -in `task_resource_defaults` are used. - -The override values from the database are assigned at execution, rather than registration time. - -To customize resources for project-domain attributes, define a ``tra.yaml`` file with your overrides: - -.. code-block:: yaml - - project: flyteexamples - domain: development - defaults: - cpu: "1" - memory: 150Mi - limits: - cpu: "2" - memory: 450Mi - -Update the task resource attributes for a project-domain combination: - -.. prompt:: bash $ - - flytectl update task-resource-attribute --attrFile tra.yaml - -.. note:: - - Refer to the :ref:`docs ` to - learn more about the command and its supported flag(s). - -To fetch and verify the individual project-domain attributes: - -.. prompt:: bash $ - - flytectl get task-resource-attribute -p flyteexamples -d development - -.. note:: - - Refer to the :ref:`docs ` to learn - more about the command and its supported flag(s). - -You can view all custom task-resource-attributes by visiting -``protocol://`` and substitute -the protocol and host appropriately. - -Cluster Resources -================= -These are free-form key-value pairs used when filling the templates that the -admin feeds into the cluster manager, which is the process that syncs Kubernetes -resources. - -The keys represent templatized variables in the -`cluster resource template `__ -and the values are what you want to see filled in. - -In the absence of custom override values, you can use ``templateData`` from the -`FlyteAdmin config `__ -as a default. Flyte specifies these defaults by domain and applies them to every -project-domain namespace combination. - -.. note:: - The settings above can be specified on domain, and project-and-domain. - Since Flyte hasn't tied the notion of a workflow or a launch plan to any Kubernetes construct, specifying a workflow or launch plan name doesn't make sense. - This is a departure from the usual hierarchy for customizable resources. - -Define an attributes file, ``cra.yaml``: - -.. code-block:: yaml - - domain: development - project: flyteexamples - attributes: - projectQuotaCpu: "1000" - projectQuotaMemory: 5Ti - -To ensure that the overrides reflect in the Kubernetes namespace -``flyteexamples-development`` (that is, the namespace has a resource quota of -1000 CPU cores and 5TB of memory) when the admin fills in cluster resource -templates: - -.. prompt:: bash $ - - flytectl update cluster-resource-attribute --attrFile cra.yaml - -.. note:: - - Refer to the :ref:`docs ` - to learn more about the command and its supported flag(s). - -To fetch and verify the individual project-domain attributes: - -.. prompt:: bash $ - - flytectl get cluster-resource-attribute -p flyteexamples -d development - -.. note:: - - Refer to the :ref:`docs ` to - learn more about the command and its supported flag(s). - -Flyte uses these updated values to fill the template fields for the -``flyteexamples-development`` namespace. - -For other namespaces, the -`platform defaults `__ -apply. - -.. note:: - The template values, for example, ``projectQuotaCpu`` or ``projectQuotaMemory`` are free-form strings. - Ensure that they match the template placeholders in your `template file `__ - for your changes to take effect and custom values to be substituted. - -You can view all custom cluster-resource-attributes by visiting ``protocol://`` -and substitute the protocol and host appropriately. - -Execution Cluster Label -======================= -This allows forcing a matching execution to consistently execute on a specific -Kubernetes cluster for multi-cluster Flyte deployment set-up. - -Define an attributes file in `ec.yaml`: - -.. code-block:: yaml - - value: mycluster - domain: development - project: flyteexamples - -Ensure that admin places executions in the flyteexamples project and development domain onto ``mycluster``: - -.. prompt:: bash $ - - flytectl update execution-cluster-label --attrFile ec.yaml - -.. note:: - - Refer to the :ref:`docs ` - to learn more about the command and its supported flag(s). - -To fetch and verify the individual project-domain attributes: - -.. prompt:: bash $ - - flytectl get execution-cluster-label -p flyteexamples -d development - -.. note:: - - Refer to the :ref:`docs ` to - learn more about the command and its supported flag(s). - -You can view all custom execution cluster attributes by visiting -``protocol://`` and substitute -the protocol and host appropriately. - -Execution Queues -================ -Execution queues are defined in -`flyteadmin config `__. -These are used for execution placement for constructs like AWS Batch. - -The **attributes** associated with an execution queue must match the **tags** -for workflow executions. The tags associated with configurable resources are -stored in the admin database. - -.. prompt:: bash $ - - flytectl update execution-queue-attribute - -.. note:: - - Refer to the :ref:`docs ` - to learn more about the command and its supported flag(s). - -You can view existing attributes for which tags can be assigned by visiting -``protocol:///api/v1/matchable_attributes?resource_type=2`` and substitute -the protocol and host appropriately. - -Workflow Execution Config -========================= - -This helps with overriding the config used for workflows execution which includes -`security context `__, `annotations or labels `__ -etc. in the `Workflow execution config `__. -These can be defined at two levels of project-domain or project-domain-workflow: - -.. prompt:: bash $ - - flytectl update workflow-execution-config - -.. note:: - - Refer to the :ref:`docs ` - to learn more about the command and its supported flag(s). - -Configuring Service Roles -========================= -You can configure service roles along 3 levels: - -#. Project + domain defaults (every execution launched in this project/domain uses this service account) - -#. Launch plan default (every invocation of this launch plan uses this service account) - -#. Execution time override (overrides at invocation for a specific execution only) - -********* -Hierarchy -********* - -Increasing specificity defines how matchable resource attributes get applied. -The available configurations, in order of decreasing specificity are: - -#. Domain, Project, Workflow name, and LaunchPlan - -#. Domain, Project, and Workflow name - -#. Domain and Project - -#. Domain - -Default values for all and per-domain attributes may be specified in the -FlyteAdmin config as documented in the :std:ref:`deployment-configuration-customizable-resources`. - -Example -======= -If the database includes the following: - -+------------+--------------+----------+-------------+-----------+ -| Domain | Project | Workflow | Launch Plan | Tags | -+============+==============+==========+=============+===========+ -| production | widgetmodels | | | critical | -+------------+--------------+----------+-------------+-----------+ -| production | widgetmodels | Demand | | supply | -+------------+--------------+----------+-------------+-----------+ - -- Any inbound ``CreateExecution`` requests with **[Domain: Production, Project: widgetmodels, Workflow: Demand]** for any launch plan will have a tag value of "supply". -- Any inbound ``CreateExecution`` requests with **[Domain: Production, Project: widgetmodels]** for any workflow other than ``Demand`` and any launch plan will have a tag value "critical". -- All other inbound CreateExecution requests will use the default values specified in the FlyteAdmin config (if any). - - -Configuring K8s Pod -=================== - -There are two approaches to applying the K8s Pod configuration. The **recommended** -method is to use Flyte's Compile-time and Runtime PodTemplate schemes. You can do this by creating -K8s PodTemplate resource/s that serves as the base configuration for all the -task Pods that Flyte initializes. This solution ensures completeness regarding -support configuration options and maintainability as new features are added to K8s. - -The legacy technique is to set configuration options in Flyte's K8s plugin configuration. - -.. note :: - - These two approaches can be used simultaneously, where the K8s plugin configuration will override the default PodTemplate values. - -.. _using-k8s-podtemplates: - -******************************* -Using K8s PodTemplates -******************************* +########################################### +Configuring task pods with K8s PodTemplates +########################################### `PodTemplate `__ is a K8s native resource used to define a K8s Pod. It contains all the fields in the PodSpec, in addition to ObjectMeta to control resource-specific metadata -such as Labels or Annotations. They are commonly applied in Deployments, +such as Labels or Annotations. PodTemplates are commonly applied in Deployments, ReplicaSets, etc to define the managed Pod configuration of the resources. -Within Flyte, you can leverage this resource to configure Pods created as part -of Flyte's task execution. It ensures complete control over Pod configuration, +Within Flyte, you can use PodTemplates to configure Pods created as part +of Flyte's task execution. This ensures complete control over Pod configuration, supporting all options available through the resource and ensuring maintainability in future versions. -Starting with the Flyte 1.4 release, we now have 2 ways of defining `PodTemplate `__: +Starting with the Flyte 1.4 release, there are two ways of defining `PodTemplate `__: 1. Compile-time PodTemplate defined at the task level 2. Runtime PodTemplates +.. note :: + + The legacy technique is to set configuration options in Flyte's K8s plugin configuration. These two approaches can be used simultaneously, where the K8s plugin configuration will override the default PodTemplate values. +************************* Compile-time PodTemplates -========================= +************************* We can define a compile-time pod template, as part of the definition of a `Task `__, for example: @@ -356,8 +68,9 @@ the name of the primary container, labels, and annotations. The term compile-time here refers to the fact that the pod template definition is part of the `TaskSpec `__. +******************** Runtime PodTemplates -==================== +******************** Runtime PodTemplates, as the name suggests, are applied during runtime, as part of building the resultant Pod. In terms of how they are applied, you have two choices: (1) you either elect one specific PodTemplate to be considered as default, or (2) you @@ -367,7 +80,7 @@ PodTemplate name will be used. Set the ``default-pod-template-name`` in FlytePropeller --------------------------------------------------------- +======================================================= This `option `__ initializes a K8s informer internally to track system PodTemplate updates @@ -390,7 +103,7 @@ An example configuration is: default-pod-template-name: Create a PodTemplate resource ------------------------------- +============================= Flyte recognizes PodTemplate definitions with the ``default-pod-template-name`` at two granularities. @@ -425,7 +138,7 @@ set to anything. However, we recommend using a real image, for example ``docker.io/rwgrim/docker-noop``. Using ``pod_template_name`` in a Task --------------------------------------- +===================================== It's also possible to use PodTemplate in tasks by specifying ``pod_template_name`` in the task definition. For example: diff --git a/docs/deployment/configuration/generated/flytepropeller_config.rst b/docs/deployment/configuration/generated/flytepropeller_config.rst index 6ddf08273c..db4f9a543e 100644 --- a/docs/deployment/configuration/generated/flytepropeller_config.rst +++ b/docs/deployment/configuration/generated/flytepropeller_config.rst @@ -1195,6 +1195,7 @@ ray (`ray.Config`_) enabled: false endpoint: "" name: "" + serviceAccount: default serviceType: NodePort shutdownAfterJobFinishes: true ttlSecondsAfterFinished: 3600 @@ -1342,7 +1343,7 @@ resourceConstraints (`core.ResourceConstraintsSpec`_) Value: 100 -defaultAgent (`agent.Agent`_) +defaultAgent (`agent.Deployment`_) """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" The default agent. @@ -1358,7 +1359,7 @@ The default agent. timeouts: null -agents (map[string]*agent.Agent) +agents (map[string]*agent.Deployment) """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" The agents. @@ -1391,7 +1392,7 @@ supportedTaskTypes ([]string) - task_type_2 -agent.Agent +agent.Deployment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ endpoint (string) @@ -3567,6 +3568,18 @@ Version of the Ray CRD to use when creating RayClusters or RayJobs. v1alpha1 +serviceAccount (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The k8s service account to run as + +**Default Value**: + +.. code-block:: yaml + + default + + ray.DefaultConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/deployment/configuration/index.md b/docs/deployment/configuration/index.md index 9eec838596..b5758679d7 100644 --- a/docs/deployment/configuration/index.md +++ b/docs/deployment/configuration/index.md @@ -33,10 +33,10 @@ you'll need to add the configuration settings under the `inline` section in the - Migration guide to move to Admin's own authorization server. * - {ref}`Understanding Authentication ` - Migration guide to move to Admin's own authorization server. -* - {ref}`Configuring Custom K8s Resources ` +* - {ref}`Configuring task pods with K8s PodTemplates ` - Use Flyte's cluster-resource-controller to control specific Kubernetes resources and administer project/domain-specific CPU/GPU/memory resource quotas. -* - {ref}`Adding New Customizable Resources ` - - Create new default configurations or overriding certain values for specific combinations of user projects, domains and workflows through Flyte APIs. +* - {ref}`Customizing project, domain, and workflow resources with flytectl ` + - Use the Flyte APIs to create new default configurations to override certain values for specific combinations of user projects, domains and workflows. * - {ref}`Notifications ` - Guide to setting up and configuring notifications. * - {ref}`External Events ` @@ -47,6 +47,8 @@ you'll need to add the configuration settings under the `inline` section in the - Improve the performance of the core Flyte engine. * - {ref}`Platform Events ` - Configure Flyte to to send events to external pub/sub systems. +* - {ref}`Resource Manager ` + - Manage external resource pooling ``` ```{toctree} @@ -63,4 +65,5 @@ monitoring notifications performance cloud_event +resource_manager ``` diff --git a/docs/deployment/configuration/notifications.rst b/docs/deployment/configuration/notifications.rst index 386e19a406..2e4a77ac53 100644 --- a/docs/deployment/configuration/notifications.rst +++ b/docs/deployment/configuration/notifications.rst @@ -39,7 +39,7 @@ For example ) -See detailed usage examples in the :std:doc:`User Guide ` +See detailed usage examples in the :std:doc:`/user_guide/productionizing/notifications` Notifications can be combined with schedules to automatically alert you when a scheduled job succeeds or fails. diff --git a/docs/deployment/configuration/resource_manager.rst b/docs/deployment/configuration/resource_manager.rst new file mode 100644 index 0000000000..3bb3d079d4 --- /dev/null +++ b/docs/deployment/configuration/resource_manager.rst @@ -0,0 +1,109 @@ +.. _deployment-configuration-resource-manager: + +##################### +Flyte ResourceManager +##################### + +**Flyte ResourceManager** is a configurable component that helps track resource utilization of tasks that run on Flyte and allows plugins to manage resource allocations independently. Default deployments are configured with the ResourceManager disabled, which means plugins rely on each independent platform to manage resource utilization. See below for the default ResourceManager configuration: + +.. code-block:: yaml + + resourcemanager: + type: noop + +When using a plugin that connects to a platform with a robust resource scheduling mechanism, like the K8s plugin, we recommend leaving the default ``flyteresourcemanager`` configuration in place. However, with web API plugins (for example), the rate at which Flyte sends requests may overwhelm a service, and we recommend changing the ``resourcemanager`` configuration. + +The ResourceManager provides a task-type-specific pooling system for Flyte tasks. Optionally, plugin writers can request resource allocation in their tasks. + +A plugin defines a collection of resource pools using its configuration. Flyte uses tokens as a placeholder to represent a unit of resource. + +How Flyte plugins request resources +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Flyte plugins register the desired resource and resource quota with the **ResourceRegistrar** when setting up FlytePropeller. When a plugin is invoked, FlytePropeller provides a proxy for the plugin. This proxy facilitates the plugin's view of the resource pool by controlling operations to allocate and deallocate resources. + +Once the setup is complete, FlytePropeller builds a ResourceManager based on the previously requested resource registration. Based on the plugin implementation's logic, resources are allocated and deallocated. + +During runtime, the ResourceManager: + +#. Allocates tokens to the plugin. +#. Releases tokens once the task is completed. + +In this manner, Flyte plugins intelligently throttle resource usage during parallel execution of nodes. + +.. note :: + + The ResourceManager can use a Redis instance as an external store to track and manage resource pool allocation. By default, it is disabled, and can be enabled with: + + .. code-block:: yaml + + resourcemanager: + type: redis + resourceMaxQuota: 100 + redis: + hostPaths: + - foo + hostKey: bar + maxRetries: 0 + +Plugin resource allocation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a Flyte task execution needs to send a request to an external service, the plugin claims a unit of the corresponding resource using a **ResourceName**, which is a unique token and a fully qualified resource request (typically an integer). The task execution generates this unique token and registers the token with the ResourceManager by calling the ResourceManager’s ``AllocateResource`` function. If the resource pool has sufficient capacity to fulfill the request, then the requested resources are allocated, and the plugin proceeds further. + +When the status changes to **"AllocationGranted"**, the execution sends out the request for those resources. + +The granted token is recorded in a token pool which corresponds to the resource that is managed by the ResourceManager. + +Plugin resource deallocation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +When the request is completed, the plugin asks the ResourceManager to release the token by calling the ResourceManager's ``ReleaseResource()`` function, which eliminates the token from the token pool. + +Example +^^^^^^^^ + +Flyte has a built-in `Qubole `__ plugin which allows Flyte tasks to send Hive commands to Qubole. In the plugin, a single Qubole cluster is considered a resource, and sending a single Hive command to a Qubole cluster consumes a token of the corresponding resource. +The resource is allocated when the status is **“AllocationGranted”**. The Qubole plugin calls: + +.. code-block:: go + + status, err := AllocateResource(ctx, , , ) + +In our example scenario, the placeholder values are replaced with the following: + +.. code-block:: go + + status, err := AllocateResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0", ResourceConstraintsSpec{}) + +The resource is deallocated when the Hive command completes its execution and the corresponding token is released. The plugin calls: + +.. code-block:: go + + status, err := AllocateResource(ctx, , , ) + +In our example scenario, the placeholder values are replaced with the following: + +.. code-block:: go + + err := ReleaseResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0") + +See below for an example interface that shows allocation and deallocation of resources: + +.. code-block:: go + + type ResourceManager interface { + GetID() string + // During execution, the plugin calls AllocateResource() to register a token in the token pool associated with a resource + // If it is granted an allocation, the token is recorded in the token pool until the same plugin releases it. + // When calling AllocateResource, the plugin has to specify a ResourceConstraintsSpec that contains resource capping constraints at different project and namespace levels. + // The ResourceConstraint pointers in ResourceConstraintsSpec can be set to nil to not have a constraint at that level + AllocateResource(ctx context.Context, namespace ResourceNamespace, allocationToken string, constraintsSpec ResourceConstraintsSpec) (AllocationStatus, error) + // During execution, after an outstanding request is completed, the plugin uses ReleaseResource() to release the allocation of the token from the token pool. This way, it redeems the quota taken by the token + ReleaseResource(ctx context.Context, namespace ResourceNamespace, allocationToken string) error + } + +Configuring ResourceManager to force runtime quota allocation constraints +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Runtime quota allocation constraints can be achieved using ResourceConstraintsSpec. It is a contact that a plugin can specify at different project and namespace levels. + +For example, you can set ResourceConstraintsSpec to ``nil`` objects, which means there would be no allocation constraints at the respective project and namespace level. When ResourceConstraintsSpec specifies ``nil`` ProjectScopeResourceConstraint, and a non-nil NamespaceScopeResourceConstraint, it suggests no constraints specified at any project or namespace level. diff --git a/docs/deployment/plugins/aws/batch.rst b/docs/deployment/plugins/aws/batch.rst index 15bfe5939d..a3cad36d0e 100644 --- a/docs/deployment/plugins/aws/batch.rst +++ b/docs/deployment/plugins/aws/batch.rst @@ -8,7 +8,7 @@ and single tasks running on AWS Batch. .. note:: - For single [non-map] task use, please take note of + For single [non-map] task use, please take note of the additional code when updating the flytepropeller config. AWS Batch simplifies the process for developers, scientists and engineers to run @@ -21,7 +21,7 @@ optimizing AWS Batch job queues for load distribution and priority coordination. Set up AWS Batch ---------------- -Follow the guide `Running batch jobs +Follow the guide `Running batch jobs at scale for less `__. By the end of this step, your AWS Account should have a configured compute environment @@ -30,7 +30,7 @@ and one or more AWS Batch Job Queues. Modify users' AWS IAM role trust policy document ------------------------------------------------ -Follow the guide `AWS Batch Execution +Follow the guide `AWS Batch Execution IAM role `__. When running workflows in Flyte, users can specify a Kubernetes service account and/or an IAM Role to run as. @@ -40,11 +40,11 @@ to allow elastic container service (ECS) to assume the role. Modify system's AWS IAM role policies ------------------------------------- -Follow the guide `Granting a user permissions to pass a +Follow the guide `Granting a user permissions to pass a role to an AWS service `__. The best practice for granting permissions to Flyte components is by utilizing OIDC, -as described in the +as described in the `OIDC documentation `__. This approach entails assigning an IAM Role to each service account being used. To proceed, identify the IAM Role associated with the flytepropeller's Kubernetes service account, @@ -113,7 +113,7 @@ with distinct attributes and matching logic based on project/domain/workflowName - default These settings can also be dynamically altered through ``flytectl`` (or FlyteAdmin API). -Learn about the :ref:`core concept here `. +Learn about the :ref:`core concept here `. For guidance on how to dynamically update these configurations, refer to the :ref:`Flytectl docs `. Update FlytePropeller's configuration @@ -145,10 +145,10 @@ These configurations reside within FlytePropeller's configMap. Modify the config .. note:: - To register the `map task - `__ on Flyte, + To register the `map task + `__ on Flyte, use the command ``pyflyte register ``. - Launch the execution through the FlyteConsole by selecting the appropriate ``IAM Role`` and entering the full + Launch the execution through the FlyteConsole by selecting the appropriate ``IAM Role`` and entering the full ``AWS Arn`` of an IAM Role configured according to the above guide. Once the task starts executing, you'll find a link for the AWS Array Job in the log links section of the Flyte Console. diff --git a/docs/deployment/plugins/k8s/index.rst b/docs/deployment/plugins/k8s/index.rst index 2199f099e8..7e1c103eec 100644 --- a/docs/deployment/plugins/k8s/index.rst +++ b/docs/deployment/plugins/k8s/index.rst @@ -440,14 +440,14 @@ Install the Kubernetes operator 3. Use a Flyte pod template with ``template.spec.schedulerName: scheduler-plugins-scheduler`` to use the new gang scheduler for your tasks. - See the :ref:`using-k8s-podtemplates` section for more information on pod templates in Flyte. + See :ref:`deployment-configuration-general` for more information on pod templates in Flyte. You can set the scheduler name in the pod template passed to the ``@task`` decorator. However, to prevent the two different schedulers from competing for resources, it is recommended to set the scheduler name in the pod template in the ``flyte`` namespace which is applied to all tasks. Non distributed training tasks can be scheduled by the gang scheduler as well. - For more information on pod templates in Flyte, refer to the :ref:`using-k8s-podtemplates` section. + For more information on pod templates in Flyte, see :ref:`deployment-configuration-general`. You can set the scheduler name in the pod template passed to the ``@task`` decorator. However, to avoid resource competition between the two different schedulers, it is recommended to set the scheduler name in the pod template in the ``flyte`` namespace, diff --git a/docs/flyte_agents/developing_agents.md b/docs/flyte_agents/developing_agents.md new file mode 100644 index 0000000000..ba114be6c7 --- /dev/null +++ b/docs/flyte_agents/developing_agents.md @@ -0,0 +1,100 @@ +--- +jupytext: + formats: md:myst + text_representation: + extension: .md + format_name: myst +--- + +(developing_agents)= +# Developing agents + +The Flyte agent framework enables rapid agent development, since agents are decoupled from the core FlytePropeller engine. Rather than building a complete gRPC service from scratch, you can implement an agent as a Python class, easing development. Agents can be tested independently and deployed privately, making maintenance easier and giving you more flexibility and control over development. + +If you need to create a new type of task, we recommend creating a new agent to run it rather than running the task in a pod. After testing the new agent, you can update your FlytePropeller configMap to specify the type of task that the agent should run. + +```{note} + +We strongly encourage you to contribute your agent to the Flyte community. To do so, follow the steps in "[Contributing to Flyte](https://docs.flyte.org/en/latest/community/contribute.html)", and reach out to us on [Slack](https://docs.flyte.org/en/latest/community/contribute.html#) if you have any questions. + +``` + +There are two types of agents: **async** and **sync**. +* **Async agents** enable long-running jobs that execute on an external platform over time. They communicate with external services that have asynchronous APIs that support `create`, `get`, and `delete` operations. The vast majority of agents are async agents. +* **Sync agents** enable request/response services that return immediate outputs (e.g. calling an internal API to fetch data or communicating with the OpenAI API). + +```{note} + +While agents can be written in any programming language, we currently only support Python agents. We may support other languages in the future. + +``` + +## Async agent interface specification + +To create a new async agent, extend the `AsyncAgentBase` and implement `create`, `get`, and `delete` methods. These methods must be idempotent. + +- `create`: This method is used to initiate a new job. Users have the flexibility to use gRPC, REST, or an SDK to create a job. +- `get`: This method retrieves the job resource (jobID or output literal) associated with the task, such as a BigQuery job ID or Databricks task ID. +- `delete`: Invoking this method will send a request to delete the corresponding job. + +```python +from flytekit.extend.backend.base_agent import AsyncAgentBase, AgentRegistry, Resource +from flytekit import StructuredDataset +from dataclasses import dataclass + +@dataclass +class BigQueryMetadata(ResourceMeta): + """ + This is the metadata for the job. For example, the id of the job. + """ + job_id: str + +class BigQueryAgent(AsyncAgentBase): + def __init__(self): + super().__init__(task_type_name="bigquery", metadata_type=BigQueryMetadata) + + def create( + self, + task_template: TaskTemplate, + inputs: typing.Optional[LiteralMap] = None, + **kwargs, + ) -> BigQueryMetadata: + # Submit the job to BigQuery here. + return BigQueryMetadata(job_id=job_id, outputs={"o0": StructuredDataset(uri=result_table_uri))} + + def get(self, resource_meta: BigQueryMetadata, **kwargs) -> Resource: + # Get the job status from BigQuery. + return Resource(phase=res.phase) + + def delete(self, resource_meta: BigQueryMetadata, **kwargs): + # Delete the job from BigQuery. + ... + +# To register the custom agent +AgentRegistry.register(BigQueryAgent()) +``` + +For an example implementation, see the [BigQuery agent](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-bigquery/flytekitplugins/bigquery/agent.py#L43). + +## Sync agent interface specification + +To create a new sync agent, extend the `SyncAgentBase` class and implement a `do` method. This method must be idempotent. + +- `do`: This method is used to execute the synchronous task, and the worker in Flyte will be blocked until the method returns. + +```python +from flytekit.extend.backend.base_agent import SyncAgentBase, AgentRegistry, Resource + +class OpenAIAgent(SyncAgentBase): + def __init__(self): + super().__init__(task_type_name="openai") + + def do(self, task_template: TaskTemplate, inputs: Optional[LiteralMap], **kwargs) -> Resource: + # Convert the literal map to python value. + ctx = FlyteContextManager.current_context() + python_inputs = TypeEngine.literal_map_to_kwargs(ctx, inputs, literal_types=task_template.interface.inputs) + # Call the OpenAI API here. + return Resource(phase=phaseTaskExecution.SUCCEEDED, outputs={"o0": "Hello world!"}) + +AgentRegistry.register(OpenAIAgent()) +``` diff --git a/docs/flyte_agents/enabling_agents_in_your_flyte_deployment.md b/docs/flyte_agents/enabling_agents_in_your_flyte_deployment.md new file mode 100644 index 0000000000..f50b740a21 --- /dev/null +++ b/docs/flyte_agents/enabling_agents_in_your_flyte_deployment.md @@ -0,0 +1,16 @@ +--- +jupytext: + formats: md:myst + text_representation: + extension: .md + format_name: myst +--- + +(enabling_agents_in_your_flyte_deploymen)= +# Enabling agents in your Flyte deployment + +After you have finished {ref}`testing an agent locally `, you can enable the agent in your Flyte deployment to use it in production. To enable a particular agent in your Flyte deployment, see the [Agent setup guide](https://docs.flyte.org/en/latest/deployment/agents/index.html) for the agent. + +:::{note} +If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to enable agents in your deployment. +::: diff --git a/docs/flyte_agents/index.md b/docs/flyte_agents/index.md new file mode 100644 index 0000000000..293f661be9 --- /dev/null +++ b/docs/flyte_agents/index.md @@ -0,0 +1,49 @@ +--- +# override the toc-determined page navigation order +prev-page: getting_started/extending_flyte +prev-page-title: Extending Flyte +--- + +(flyte_agents_guide)= +# Flyte agents + +Flyte agents are long-running, stateless services that receive execution requests via gRPC and initiate jobs with appropriate external or internal services. They enable two key workflows: asynchronously launching jobs on hosted platforms (e.g. Databricks or Snowflake) and calling external synchronous services, such as access control, data retrieval, and model inferencing. + +Each agent service is a Kubernetes deployment that receives gRPC requests from FlytePropeller when users trigger a particular type of task (for example, the BigQuery agent handles BigQuery tasks). The agent service then initiates a job with the appropriate service. Since agents can be spawned in process, they allow for running all services locally as long as the connection secrets are available. Moreover, agents use a protobuf interface, thus can be implemented in any language, enabling flexibility, reuse of existing libraries, and simpler testing. + +You can create different agent services that host different agents, e.g., a production and a development agent service: + +:::{figure} https://i.ibb.co/vXhBDjP/Screen-Shot-2023-05-29-at-2-54-14-PM.png +:alt: Agent Service +:class: with-shadow +::: + +(using_agents_in_tasks)= +## Using agents in tasks + +If you need to connect to an external service in your workflow, we recommend using the corresponding agent rather than a web API plugin. Agents are designed to be scalable and can handle large workloads efficiently, and decrease load on FlytePropeller, since they run outside of it. You can also test agents locally without having to change the Flyte backend configuration, streamlining development. + +For a list of agents you can use in your tasks and example usage for each, see the [Integrations](https://docs.flyte.org/en/latest/flytesnacks/integrations.html#agents) documentation. + +## Table of contents + +```{list-table} +:header-rows: 0 +:widths: 20 30 + +* - {doc}`Developing agents ` + - If the agent you need doesn't exist, follow these steps to create it. +* - {doc}`Testing agents locally ` + - Whether using an existing agent or developing a new one, you can test the agent locally without needing to configure your Flyte deployment. +* - {doc}`Enabling agents in your Flyte deployment ` + - Once you have tested an agent locally and want to use it in production, you must configure your Flyte deployment for the agent. +``` + +```{toctree} +:maxdepth: -1 +:hidden: + +developing_agents +testing_agents_locally +enabling_agents_in_your_flyte_deployment +``` diff --git a/docs/flyte_agents/testing_agents_locally.md b/docs/flyte_agents/testing_agents_locally.md new file mode 100644 index 0000000000..2d7b98ba3e --- /dev/null +++ b/docs/flyte_agents/testing_agents_locally.md @@ -0,0 +1,79 @@ +--- +jupytext: + formats: md:myst + text_representation: + extension: .md + format_name: myst +--- + +(testing_agents_locally)= +# Testing agents locally + +You can test agents locally without running the backend server, making agent development easier. + +To test an agent locally, create a class for the agent task that inherits from [AsyncAgentExecutorMixin](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py#L155). This mixin can handle both asynchronous tasks and synchronous tasks and allows flytekit to mimic FlytePropeller's behavior in calling the agent. + +## BigQuery example + +To test the BigQuery agent, copy the following code to a file called `bigquery_task.py`, modifying as needed. + +```{note} + +In some cases, you will need to store credentials in your local environment when testing locally. +For example, you need to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable when running BigQuery tasks to test the BigQuery agent. + +``` + +Add `AsyncAgentExecutorMixin` to this class to tell flytekit to use the agent to run the task. +```python +class BigQueryTask(AsyncAgentExecutorMixin, SQLTask[BigQueryConfig]): + def __init__(self, name: str, **kwargs): + ... +``` + +Flytekit will automatically use the agent to run the task in the local execution. +```python +bigquery_doge_coin = BigQueryTask( + name=f"bigquery.doge_coin", + inputs=kwtypes(version=int), + query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;", + output_structured_dataset_type=StructuredDataset, + task_config=BigQueryConfig(ProjectID="flyte-test-340607") +) +``` + +You can run the above example task locally and test the agent with the following command: + +```bash +pyflyte run bigquery_task.py bigquery_doge_coin --version 10 +``` + +## Databricks example +To test the Databricks agent, copy the following code to a file called `databricks_task.py`, modifying as needed. + +```python +@task(task_config=Databricks(...)) +def hello_spark(partitions: int) -> float: + print("Starting Spark with Partitions: {}".format(partitions)) + + n = 100000 * partitions + sess = flytekit.current_context().spark_session + count = ( + sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add) + ) + pi_val = 4.0 * count / n + print("Pi val is :{}".format(pi_val)) + return pi_val +``` + +To execute the Spark task on the agent, you must configure the `raw-output-data-prefix` with a remote path. +This configuration ensures that flytekit transfers the input data to the blob storage and allows the Spark job running on Databricks to access the input data directly from the designated bucket. + +```{note} +The Spark task will run locally if the `raw-output-data-prefix` is not set. +``` + +```bash +pyflyte run --raw-output-data-prefix s3://my-s3-bucket/databricks databricks_task.py hello_spark +``` + diff --git a/docs/flyte_fundamentals/optimizing_tasks.md b/docs/flyte_fundamentals/optimizing_tasks.md index 508767d05f..00c27c881f 100644 --- a/docs/flyte_fundamentals/optimizing_tasks.md +++ b/docs/flyte_fundamentals/optimizing_tasks.md @@ -243,7 +243,7 @@ When this task is executed on a Flyte cluster, it automatically provisions all o the resources that you need. In this case, that need is distributed training, but Flyte also provides integrations for {ref}`Spark `, {ref}`Ray `, {ref}`MPI `, {ref}`Sagemaker `, -{ref}`Snowflake `, and more. +{ref}`Snowflake `, and more. Even though Flyte itself is a powerful compute engine and orchestrator for data engineering, machine learning, and analytics, perhaps you have existing diff --git a/docs/index.md b/docs/index.md index 3a8d38e6ba..4720be51f7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -79,7 +79,7 @@ contribute its architecture and design. You can also access the * - {doc}`🔤 Introduction to Flyte ` - Get your first workflow running, learn about the Flyte development lifecycle and core use cases. -* - {doc}`📖 User Guide ` +* - {doc}`📖 User Guide ` - A comprehensive view of Flyte's functionality for data and ML practitioners. * - {doc}`📚 Tutorials ` - End-to-end examples of Flyte for data/feature engineering, machine learning, @@ -138,6 +138,7 @@ Introduction Quickstart guide Getting started with workflow development Flyte fundamentals +Flyte agents Core use cases ``` @@ -147,9 +148,10 @@ Core use cases :name: examples-guides :hidden: -User Guide +User Guide Tutorials Integrations +Deprecated integrations ``` ```{toctree} diff --git a/docs/user_guide/advanced_composition/chaining_flyte_entities.md b/docs/user_guide/advanced_composition/chaining_flyte_entities.md new file mode 100644 index 0000000000..f51b45a2d0 --- /dev/null +++ b/docs/user_guide/advanced_composition/chaining_flyte_entities.md @@ -0,0 +1,112 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(chain_flyte_entities)= + +# Chaining Flyte entities + +```{eval-rst} +.. tags:: Basic +``` + +Flytekit offers a mechanism for chaining Flyte entities using the `>>` operator. +This is particularly valuable when chaining tasks and subworkflows without the need for data flow between the entities. + +## Tasks + +Let's establish a sequence where `t1()` occurs after `t0()`, and `t2()` follows `t1()`. + +```{code-cell} +from flytekit import task, workflow + + +@task +def t2(): + print("Running t2") + return + + +@task +def t1(): + print("Running t1") + return + + +@task +def t0(): + print("Running t0") + return + + +@workflow +def chain_tasks_wf(): + t2_promise = t2() + t1_promise = t1() + t0_promise = t0() + + t0_promise >> t1_promise + t1_promise >> t2_promise +``` + ++++ {"lines_to_next_cell": 0} + +(chain_subworkflow)= +## Subworkflows + +Just like tasks, you can chain {ref}`subworkflows `. + +```{code-cell} +:lines_to_next_cell: 2 + +@workflow +def sub_workflow_1(): + t1() + + +@workflow +def sub_workflow_0(): + t0() + + +@workflow +def chain_workflows_wf(): + sub_wf1 = sub_workflow_1() + sub_wf0 = sub_workflow_0() + + sub_wf0 >> sub_wf1 +``` + +To run the provided workflows on the Flyte cluster, use the following commands: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/chain_entities.py \ + chain_tasks_wf +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/chain_entities.py \ + chain_workflows_wf +``` + +:::{note} +Chaining tasks and subworkflows is not supported in local environments. +Follow the progress of this issue [here](https://github.com/flyteorg/flyte/issues/4080). +::: diff --git a/docs/user_guide/advanced_composition/conditionals.md b/docs/user_guide/advanced_composition/conditionals.md new file mode 100644 index 0000000000..88c447a05c --- /dev/null +++ b/docs/user_guide/advanced_composition/conditionals.md @@ -0,0 +1,323 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(conditional)= + +# Conditionals + +```{eval-rst} +.. tags:: Intermediate +``` + +Flytekit elevates conditions to a first-class construct named `conditional`, providing a powerful mechanism for selectively +executing branches in a workflow. Conditions leverage static or dynamic data generated by tasks or +received as workflow inputs. While conditions are highly performant in their evaluation, +it's important to note that they are restricted to specific binary and logical operators +and are applicable only to primitive values. + +To begin, import the necessary libraries. + +```{code-cell} +import random + +from flytekit import conditional, task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +## Simple branch + +In this example, we introduce two tasks, `calculate_circle_circumference` and +`calculate_circle_area`. The workflow dynamically chooses between these tasks based on whether the input +falls within the fraction range (0-1) or not. + +```{code-cell} +@task +def calculate_circle_circumference(radius: float) -> float: + return 2 * 3.14 * radius # Task to calculate the circumference of a circle + + +@task +def calculate_circle_area(radius: float) -> float: + return 3.14 * radius * radius # Task to calculate the area of a circle + + +@workflow +def shape_properties(radius: float) -> float: + return ( + conditional("shape_properties") + .if_((radius >= 0.1) & (radius < 1.0)) + .then(calculate_circle_circumference(radius=radius)) + .else_() + .then(calculate_circle_area(radius=radius)) + ) + + +if __name__ == "__main__": + radius_small = 0.5 + print(f"Circumference of circle (radius={radius_small}): {shape_properties(radius=radius_small)}") + + radius_large = 3.0 + print(f"Area of circle (radius={radius_large}): {shape_properties(radius=radius_large)}") +``` + ++++ {"lines_to_next_cell": 0} + +## Multiple branches + +We establish an `if` condition with multiple branches, which will result in a failure if none of the conditions is met. +It's important to note that any `conditional` statement in Flyte is expected to be complete, +meaning that all possible branches must be accounted for. + +```{code-cell} +@workflow +def shape_properties_with_multiple_branches(radius: float) -> float: + return ( + conditional("shape_properties_with_multiple_branches") + .if_((radius >= 0.1) & (radius < 1.0)) + .then(calculate_circle_circumference(radius=radius)) + .elif_((radius >= 1.0) & (radius <= 10.0)) + .then(calculate_circle_area(radius=radius)) + .else_() + .fail("The input must be within the range of 0 to 10.") + ) +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +Take note of the usage of bitwise operators (`&`). Due to Python's PEP-335, +the logical `and`, `or` and `not` operators cannot be overloaded. +Flytekit employs bitwise `&` and `|` as equivalents for logical `and` and `or` operators, +a convention also observed in other libraries. +::: + +## Consuming the output of a conditional +Here, we write a task that consumes the output returned by a `conditional`. + +```{code-cell} +@workflow +def shape_properties_accept_conditional_output(radius: float) -> float: + result = ( + conditional("shape_properties_accept_conditional_output") + .if_((radius >= 0.1) & (radius < 1.0)) + .then(calculate_circle_circumference(radius=radius)) + .elif_((radius >= 1.0) & (radius <= 10.0)) + .then(calculate_circle_area(radius=radius)) + .else_() + .fail("The input must exist between 0 and 10.") + ) + return calculate_circle_area(radius=result) + + +if __name__ == "__main__": + print(f"Circumference of circle x Area of circle (radius={radius_small}): {shape_properties(radius=5.0)}") +``` + ++++ {"lines_to_next_cell": 0} + +## Using the output of a previous task in a conditional + +You can check if a boolean returned from the previous task is `True`, +but unary operations are not supported directly. Instead, use the `is_true`, +`is_false` and `is_none` methods on the result. + +```{code-cell} +@task +def coin_toss(seed: int) -> bool: + """ + Mimic a condition to verify the successful execution of an operation + """ + r = random.Random(seed) + if r.random() < 0.5: + return True + return False + + +@task +def failed() -> int: + """ + Mimic a task that handles failure + """ + return -1 + + +@task +def success() -> int: + """ + Mimic a task that handles success + """ + return 0 + + +@workflow +def boolean_wf(seed: int = 5) -> int: + result = coin_toss(seed=seed) + return conditional("coin_toss").if_(result.is_true()).then(success()).else_().then(failed()) +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +*How do output values acquire these methods?* In a workflow, direct access to outputs is not permitted. +Inputs and outputs are automatically encapsulated in a special object known as {py:class}`flytekit.extend.Promise`. +::: + +## Using boolean workflow inputs in a conditional +You can directly pass a boolean to a workflow. + +```{code-cell} +@workflow +def boolean_input_wf(boolean_input: bool) -> int: + return conditional("boolean_input_conditional").if_(boolean_input.is_true()).then(success()).else_().then(failed()) +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +Observe that the passed boolean possesses a method called `is_true`. +This boolean resides within the workflow context and is encapsulated in a specialized Flytekit object. +This special object enables it to exhibit additional behavior. +::: + +You can run the workflows locally as follows: + +```{code-cell} +if __name__ == "__main__": + print("Running boolean_wf a few times...") + for index in range(0, 5): + print(f"The output generated by boolean_wf = {boolean_wf(seed=index)}") + print( + f"Boolean input: {True if index < 2 else False}; workflow output: {boolean_input_wf(boolean_input=True if index < 2 else False)}" + ) +``` + ++++ {"lines_to_next_cell": 0} + +## Nested conditionals + +You can nest conditional sections arbitrarily inside other conditional sections. +However, these nested sections can only be in the `then` part of a `conditional` block. + +```{code-cell} +@workflow +def nested_conditions(radius: float) -> float: + return ( + conditional("nested_conditions") + .if_((radius >= 0.1) & (radius < 1.0)) + .then( + conditional("inner_nested_conditions") + .if_(radius < 0.5) + .then(calculate_circle_circumference(radius=radius)) + .elif_((radius >= 0.5) & (radius < 0.9)) + .then(calculate_circle_area(radius=radius)) + .else_() + .fail("0.9 is an outlier.") + ) + .elif_((radius >= 1.0) & (radius <= 10.0)) + .then(calculate_circle_area(radius=radius)) + .else_() + .fail("The input must be within the range of 0 to 10.") + ) + + +if __name__ == "__main__": + print(f"nested_conditions(0.4): {nested_conditions(radius=0.4)}") +``` + ++++ {"lines_to_next_cell": 0} + +## Using the output of a task in a conditional + +Let's write a fun workflow that triggers the `calculate_circle_circumference` task in the event of a "heads" outcome, +and alternatively, runs the `calculate_circle_area` task in the event of a "tail" outcome. + +```{code-cell} +@workflow +def consume_task_output(radius: float, seed: int = 5) -> float: + is_heads = coin_toss(seed=seed) + return ( + conditional("double_or_square") + .if_(is_heads.is_true()) + .then(calculate_circle_circumference(radius=radius)) + .else_() + .then(calculate_circle_area(radius=radius)) + ) +``` + ++++ {"lines_to_next_cell": 0} + +You can run the workflow locally as follows: + +```{code-cell} +if __name__ == "__main__": + default_seed_output = consume_task_output(radius=0.4) + print( + f"Executing consume_task_output(0.4) with default seed=5. Expected output: calculate_circle_circumference => {default_seed_output}" + ) + + custom_seed_output = consume_task_output(radius=0.4, seed=7) + print(f"Executing consume_task_output(0.4, seed=7). Expected output: calculate_circle_area => {custom_seed_output}") +``` + +## Run the example on the Flyte cluster + +To run the provided workflows on the Flyte cluster, use the following commands: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + shape_properties --radius 3.0 +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + shape_properties_with_multiple_branches --radius 11.0 +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + shape_properties_accept_conditional_output --radius 0.5 +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + boolean_wf +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + boolean_input_wf --boolean_input +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + nested_conditions --radius 0.7 +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/conditional.py \ + consume_task_output --radius 0.4 --seed 7 +``` diff --git a/docs/user_guide/advanced_composition/decorating_tasks.md b/docs/user_guide/advanced_composition/decorating_tasks.md new file mode 100644 index 0000000000..50135ee8ab --- /dev/null +++ b/docs/user_guide/advanced_composition/decorating_tasks.md @@ -0,0 +1,152 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(decorating_tasks)= + +# Decorating tasks + +```{eval-rst} +.. tags:: Intermediate +``` + +You can easily change how tasks behave by using decorators to wrap your task functions. + +In order to make sure that your decorated function contains all the type annotation and docstring +information that Flyte needs, you will need to use the built-in {py:func}`~functools.wraps` decorator. + +To begin, import the required dependencies. + +```{code-cell} +import logging +from functools import partial, wraps + +from flytekit import task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +Create a logger to monitor the execution's progress. + +```{code-cell} +logger = logging.getLogger(__file__) +``` + ++++ {"lines_to_next_cell": 0} + +## Using a single decorator + +We define a decorator that logs the input and output details for a decorated task. + +```{code-cell} +def log_io(fn): + @wraps(fn) + def wrapper(*args, **kwargs): + logger.info(f"task {fn.__name__} called with args: {args}, kwargs: {kwargs}") + out = fn(*args, **kwargs) + logger.info(f"task {fn.__name__} output: {out}") + return out + + return wrapper +``` + ++++ {"lines_to_next_cell": 0} + +We create a task named `t1` that is decorated with `log_io`. + +:::{note} +The order of invoking the decorators is important. `@task` should always be the outer-most decorator. +::: + +```{code-cell} +@task +@log_io +def t1(x: int) -> int: + return x + 1 +``` + ++++ {"lines_to_next_cell": 0} + +(stacking_decorators)= + +## Stacking multiple decorators + +You can also stack multiple decorators on top of each other as long as `@task` is the outer-most decorator. + +We define a decorator that verifies if the output from the decorated function is a positive number before it's returned. +If this assumption is violated, it raises a `ValueError` exception. + +```{code-cell} +def validate_output(fn=None, *, floor=0): + @wraps(fn) + def wrapper(*args, **kwargs): + out = fn(*args, **kwargs) + if out <= floor: + raise ValueError(f"output of task {fn.__name__} must be a positive number, found {out}") + return out + + if fn is None: + return partial(validate_output, floor=floor) + + return wrapper +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +The output of the `validate_output` task uses {py:func}`~functools.partial` to implement parameterized decorators. +::: + +We define a function that uses both the logging and validator decorators. + +```{code-cell} +@task +@log_io +@validate_output(floor=10) +def t2(x: int) -> int: + return x + 10 +``` + ++++ {"lines_to_next_cell": 0} + +Finally, we compose a workflow that calls `t1` and `t2`. + +```{code-cell} +@workflow +def decorating_task_wf(x: int) -> int: + return t2(x=t1(x=x)) + + +if __name__ == "__main__": + print(f"Running decorating_task_wf(x=10) {decorating_task_wf(x=10)}") +``` + +## Run the example on the Flyte cluster + +To run the provided workflow on the Flyte cluster, use the following command: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_tasks.py \ + decorating_task_wf --x 10 +``` + +In this example, you learned how to modify the behavior of tasks via function decorators using the built-in +{py:func}`~functools.wraps` decorator pattern. To learn more about how to extend Flyte at a deeper level, for +example creating custom types, custom tasks or backend plugins, +see {ref}`Extending Flyte `. diff --git a/docs/user_guide/advanced_composition/decorating_workflows.md b/docs/user_guide/advanced_composition/decorating_workflows.md new file mode 100644 index 0000000000..3a369cc433 --- /dev/null +++ b/docs/user_guide/advanced_composition/decorating_workflows.md @@ -0,0 +1,180 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(decorating_workflows)= + +# Decorating workflows + +```{eval-rst} +.. tags:: Intermediate +``` + +The behavior of workflows can be modified in a light-weight fashion by using the built-in {py:func}`~functools.wraps` +decorator pattern, similar to using decorators to +{ref}`customize task behavior `. However, unlike in the case of +tasks, we need to do a little extra work to make sure that the DAG underlying the workflow executes tasks in the +correct order. + +## Setup-teardown pattern + +The main use case of decorating `@workflow`-decorated functions is to establish a setup-teardown pattern to execute task +before and after your main workflow logic. This is useful when integrating with other external services +like [wandb](https://wandb.ai/site) or [clearml](https://clear.ml/), which enable you to track metrics of model +training runs. + +To begin, import the necessary libraries. + +```{code-cell} +from functools import partial, wraps +from unittest.mock import MagicMock + +import flytekit +from flytekit import FlyteContextManager, task, workflow +from flytekit.core.node_creation import create_node +``` + ++++ {"lines_to_next_cell": 0} + +Let's define the tasks we need for setup and teardown. In this example, we use the +{py:class}`unittest.mock.MagicMock` class to create a fake external service that we want to initialize at the +beginning of our workflow and finish at the end. + +```{code-cell} +external_service = MagicMock() + + +@task +def setup(): + print("initializing external service") + external_service.initialize(id=flytekit.current_context().execution_id) + + +@task +def teardown(): + print("finish external service") + external_service.complete(id=flytekit.current_context().execution_id) +``` + ++++ {"lines_to_next_cell": 0} + +As you can see, you can even use Flytekit's current context to access the `execution_id` of the current workflow +if you need to link Flyte with the external service so that you reference the same unique identifier in both the +external service and Flyte. + +## Workflow decorator + +We create a decorator that we want to use to wrap our workflow function. + +```{code-cell} +def setup_teardown(fn=None, *, before, after): + @wraps(fn) + def wrapper(*args, **kwargs): + # get the current flyte context to obtain access to the compilation state of the workflow DAG. + ctx = FlyteContextManager.current_context() + + # defines before node + before_node = create_node(before) + # ctx.compilation_state.nodes == [before_node] + + # under the hood, flytekit compiler defines and threads + # together nodes within the `my_workflow` function body + outputs = fn(*args, **kwargs) + # ctx.compilation_state.nodes == [before_node, *nodes_created_by_fn] + + # defines the after node + after_node = create_node(after) + # ctx.compilation_state.nodes == [before_node, *nodes_created_by_fn, after_node] + + # compile the workflow correctly by making sure `before_node` + # runs before the first workflow node and `after_node` + # runs after the last workflow node. + if ctx.compilation_state is not None: + # ctx.compilation_state.nodes is a list of nodes defined in the + # order of execution above + workflow_node0 = ctx.compilation_state.nodes[1] + workflow_node1 = ctx.compilation_state.nodes[-2] + before_node >> workflow_node0 + workflow_node1 >> after_node + return outputs + + if fn is None: + return partial(setup_teardown, before=before, after=after) + + return wrapper +``` + ++++ {"lines_to_next_cell": 0} + +There are a few key pieces to note in the `setup_teardown` decorator above: + +1. It takes a `before` and `after` argument, both of which need to be `@task`-decorated functions. These + tasks will run before and after the main workflow function body. +2. The [create_node](https://github.com/flyteorg/flytekit/blob/9e156bb0cf3d1441c7d1727729e8f9b4bbc3f168/flytekit/core/node_creation.py#L18) function + to create nodes associated with the `before` and `after` tasks. +3. When `fn` is called, under the hood Flytekit creates all the nodes associated with the workflow function body +4. The code within the `if ctx.compilation_state is not None:` conditional is executed at compile time, which + is where we extract the first and last nodes associated with the workflow function body at index `1` and `-2`. +5. The `>>` right shift operator ensures that `before_node` executes before the + first node and `after_node` executes after the last node of the main workflow function body. + +## Defining the DAG + +We define two tasks that will constitute the workflow. + +```{code-cell} +@task +def t1(x: float) -> float: + return x - 1 + + +@task +def t2(x: float) -> float: + return x**2 +``` + ++++ {"lines_to_next_cell": 0} + +And then create our decorated workflow: + +```{code-cell} +:lines_to_next_cell: 2 + +@workflow +@setup_teardown(before=setup, after=teardown) +def decorating_workflow(x: float) -> float: + return t2(x=t1(x=x)) + + +if __name__ == "__main__": + print(decorating_workflow(x=10.0)) +``` + +## Run the example on the Flyte cluster + +To run the provided workflow on the Flyte cluster, use the following command: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/decorating_workflows.py \ + decorating_workflow --x 10.0 +``` + +To define workflows imperatively, refer to {ref}`this example `, +and to learn more about how to extend Flyte at a deeper level, for example creating custom types, custom tasks or +backend plugins, see {ref}`Extending Flyte `. diff --git a/docs/user_guide/advanced_composition/dynamic_workflows.md b/docs/user_guide/advanced_composition/dynamic_workflows.md new file mode 100644 index 0000000000..99bc88a372 --- /dev/null +++ b/docs/user_guide/advanced_composition/dynamic_workflows.md @@ -0,0 +1,292 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(dynamic_workflow)= + +# Dynamic workflows + +```{eval-rst} +.. tags:: Intermediate +``` + +A workflow whose directed acyclic graph (DAG) is computed at run-time is a {py:func}`~flytekit.dynamic` workflow. +The tasks in a dynamic workflow are executed at runtime using dynamic inputs. +This type of workflow shares similarities with the {py:func}`~flytekit.workflow`, as it employs a Python-esque DSL +to declare dependencies between the tasks or define new workflows. A key distinction lies in the dynamic workflow being assessed at runtime. +This means that the inputs are initially materialized and forwarded to dynamic workflow, resembling the behavior of a task. +However, the return value from a dynamic workflow is a {py:class}`~flytekit.extend.Promise` object, +which can be materialized by the subsequent tasks. + +Think of a dynamic workflow as a combination of a task and a workflow. +It is used to dynamically decide the parameters of a workflow at runtime. +It is both compiled and executed at run-time. You can define a dynamic workflow using the `@dynamic` decorator. + +Within the `@dynamic` context, each invocation of a {py:func}`~flytekit.task` or a derivative of +{py:class}`~flytekit.core.base_task.Task` class leads to deferred evaluation using a promise, +rather than the immediate materialization of the actual value. While nesting other `@dynamic` and +`@workflow` constructs within this task is possible, direct interaction with the outputs of a task/workflow is limited, +as they are lazily evaluated. If interaction with the outputs is desired, it is recommended to separate the +logic in a dynamic workflow and create a new task to read and resolve the outputs. + +Dynamic workflows become essential when you require: + +- Modifying the logic of the code at runtime +- Changing or deciding on feature extraction parameters on-the-go +- Building AutoML pipelines +- Tuning hyperparameters during execution + +This example utilizes dynamic workflow to count the common characters between any two strings. + +To begin, we import the required libraries. + +```{code-cell} +from flytekit import dynamic, task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +We define a task that returns the index of a character, where A-Z/a-z is equivalent to 0-25. + +```{code-cell} +@task +def return_index(character: str) -> int: + if character.islower(): + return ord(character) - ord("a") + else: + return ord(character) - ord("A") +``` + ++++ {"lines_to_next_cell": 0} + +We also create a task that prepares a list of 26 characters by populating the frequency of each character. + +```{code-cell} +@task +def update_list(freq_list: list[int], list_index: int) -> list[int]: + freq_list[list_index] += 1 + return freq_list +``` + ++++ {"lines_to_next_cell": 0} + +We define a task to calculate the number of common characters between the two strings. + +```{code-cell} +@task +def derive_count(freq1: list[int], freq2: list[int]) -> int: + count = 0 + for i in range(26): + count += min(freq1[i], freq2[i]) + return count +``` + ++++ {"lines_to_next_cell": 0} + +We define a dynamic workflow to accomplish the following: + +1. Initialize an empty 26-character list to be passed to the `update_list` task +2. Iterate through each character of the first string (`s1`) and populate the frequency list +3. Iterate through each character of the second string (`s2`) and populate the frequency list +4. Determine the number of common characters by comparing the two frequency lists + +The looping process is contingent on the number of characters in both strings, which is unknown until runtime. + +```{code-cell} +@dynamic +def count_characters(s1: str, s2: str) -> int: + # s1 and s2 should be accessible + + # Initialize empty lists with 26 slots each, corresponding to every alphabet (lower and upper case) + freq1 = [0] * 26 + freq2 = [0] * 26 + + # Loop through characters in s1 + for i in range(len(s1)): + # Calculate the index for the current character in the alphabet + index = return_index(character=s1[i]) + # Update the frequency list for s1 + freq1 = update_list(freq_list=freq1, list_index=index) + # index and freq1 are not accessible as they are promises + + # looping through the string s2 + for i in range(len(s2)): + # Calculate the index for the current character in the alphabet + index = return_index(character=s2[i]) + # Update the frequency list for s2 + freq2 = update_list(freq_list=freq2, list_index=index) + # index and freq2 are not accessible as they are promises + + # Count the common characters between s1 and s2 + return derive_count(freq1=freq1, freq2=freq2) +``` + ++++ {"lines_to_next_cell": 0} + +A dynamic workflow is modeled as a task in the backend, +but the body of the function is executed to produce a workflow at run-time. +In both dynamic and static workflows, the output of tasks are promise objects. + +Propeller executes the dynamic task within its Kubernetes pod, resulting in a compiled DAG, which is then accessible in the console. +It utilizes the information acquired during the dynamic task's execution to schedule and execute each node within the dynamic task. +Visualization of the dynamic workflow's graph in the UI becomes available only after the dynamic task has completed its execution. + +When a dynamic task is executed, it generates the entire workflow as its output, termed the *futures file*. +This nomenclature reflects the anticipation that the workflow is yet to be executed, and all subsequent outputs are considered futures. + +:::{note} +Local execution works when a `@dynamic` decorator is used because Flytekit treats it as a task that runs with native Python inputs. +::: + +Define a workflow that triggers the dynamic workflow. + +```{code-cell} +@workflow +def dynamic_wf(s1: str, s2: str) -> int: + return count_characters(s1=s1, s2=s2) +``` + ++++ {"lines_to_next_cell": 0} + +You can run the workflow locally as follows: + +```{code-cell} +:lines_to_next_cell: 2 + +if __name__ == "__main__": + print(dynamic_wf(s1="Pear", s2="Earth")) +``` + ++++ {"lines_to_next_cell": 0} + +## Why use dynamic workflows? + +### Flexibility + +Dynamic workflows streamline the process of building pipelines, offering the flexibility to design workflows +according to the unique requirements of your project. This level of adaptability is not achievable with static workflows. + +### Lower pressure on etcd + +The workflow Custom Resource Definition (CRD) and the states associated with static workflows are stored in etcd, +the Kubernetes database. This database maintains Flyte workflow CRDs as key-value pairs, tracking the status of each node's execution. + +However, there is a limitation with etcd — a hard limit on data size, encompassing the workflow and node status sizes. +Consequently, it's crucial to ensure that static workflows don't excessively consume memory. + +In contrast, dynamic workflows offload the workflow specification (including node/task definitions and connections) to the blobstore. +Still, the statuses of nodes are stored in the workflow CRD within etcd. + +Dynamic workflows help alleviate some of the pressure on etcd storage space, providing a solution to mitigate storage constraints. + +## Dynamic workflows vs. map tasks + +Dynamic tasks come with overhead for large fan-out tasks as they store metadata for the entire workflow. +In contrast, {ref}`map tasks ` prove efficient for such extensive fan-out scenarios since they refrain from storing metadata, +resulting in less noticeable overhead. + +(advanced_merge_sort)= +## Merge sort + +Merge sort is a perfect example to showcase how to seamlessly achieve recursion using dynamic workflows. +Flyte imposes limitations on the depth of recursion to prevent misuse and potential impacts on the overall stability of the system. + +```{code-cell} +:lines_to_next_cell: 2 + +from typing import Tuple + +from flytekit import conditional, dynamic, task, workflow + + +@task +def split(numbers: list[int]) -> Tuple[list[int], list[int], int, int]: + return ( + numbers[0 : int(len(numbers) / 2)], + numbers[int(len(numbers) / 2) :], + int(len(numbers) / 2), + int(len(numbers)) - int(len(numbers) / 2), + ) + + +@task +def merge(sorted_list1: list[int], sorted_list2: list[int]) -> list[int]: + result = [] + while len(sorted_list1) > 0 and len(sorted_list2) > 0: + # Compare the current element of the first array with the current element of the second array. + # If the element in the first array is smaller, append it to the result and increment the first array index. + # Otherwise, do the same with the second array. + if sorted_list1[0] < sorted_list2[0]: + result.append(sorted_list1.pop(0)) + else: + result.append(sorted_list2.pop(0)) + + # Extend the result with the remaining elements from both arrays + result.extend(sorted_list1) + result.extend(sorted_list2) + + return result + + +@task +def sort_locally(numbers: list[int]) -> list[int]: + return sorted(numbers) + + +@dynamic +def merge_sort_remotely(numbers: list[int], run_local_at_count: int) -> list[int]: + split1, split2, new_count1, new_count2 = split(numbers=numbers) + sorted1 = merge_sort(numbers=split1, numbers_count=new_count1, run_local_at_count=run_local_at_count) + sorted2 = merge_sort(numbers=split2, numbers_count=new_count2, run_local_at_count=run_local_at_count) + return merge(sorted_list1=sorted1, sorted_list2=sorted2) + + +@workflow +def merge_sort(numbers: list[int], numbers_count: int, run_local_at_count: int = 5) -> list[int]: + return ( + conditional("terminal_case") + .if_(numbers_count <= run_local_at_count) + .then(sort_locally(numbers=numbers)) + .else_() + .then(merge_sort_remotely(numbers=numbers, run_local_at_count=run_local_at_count)) + ) +``` + +By simply adding the `@dynamic` annotation, the `merge_sort_remotely` function transforms into a plan of execution, +generating a Flyte workflow with four distinct nodes. These nodes run remotely on potentially different hosts, +with Flyte ensuring proper data reference passing and maintaining execution order with maximum possible parallelism. + +`@dynamic` is essential in this context because the number of times `merge_sort` needs to be triggered is unknown at compile time. +The dynamic workflow calls a static workflow, which subsequently calls the dynamic workflow again, +creating a recursive and flexible execution structure. + +## Run the example on the Flyte cluster + +To run the provided workflows on the Flyte cluster, you can use the following commands: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py \ + dynamic_wf --s1 "Pear" --s2 "Earth" +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/dynamic_workflow.py \ + merge_sort --numbers '[1813, 3105, 3260, 2634, 383, 7037, 3291, 2403, 315, 7164]' --numbers_count 10 +``` diff --git a/docs/user_guide/advanced_composition/eager_workflows.md b/docs/user_guide/advanced_composition/eager_workflows.md new file mode 100644 index 0000000000..c2cc1dc542 --- /dev/null +++ b/docs/user_guide/advanced_composition/eager_workflows.md @@ -0,0 +1,495 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(eager_workflows)= + +# Eager workflows + +```{eval-rst} +.. tags:: Intermediate +``` + +```{important} +This feature is experimental and the API is subject to breaking changes. +If you encounter any issues please consider submitting a +[bug report](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&projects=&template=bug_report.yaml&title=%5BBUG%5D+). +``` + +So far, the two types of workflows you've seen are static workflows, which +are defined with `@workflow`-decorated functions or imperative `Workflow` class, +and dynamic workflows, which are defined with the `@dynamic` decorator. + +{ref}`Static workflows ` are created at compile time when you call `pyflyte run`, +`pyflyte register`, or `pyflyte serialize`. This means that the workflow is static +and cannot change its shape at any point: all of the variables defined as an input +to the workflow or as an output of a task or subworkflow are promises. +{ref}`Dynamic workflows `, on the other hand, are compiled +at runtime so that they can materialize the inputs of the workflow as Python values +and use them to determine the shape of the execution graph. + +In this guide you'll learn how to use eager workflows, which allow you to +create extremely flexible workflows that give you run-time access to +intermediary task/subworkflow outputs. + +## Why eager workflows? + +Both static and dynamic workflows have a key limitation: while they provide +compile-time and run-time type safety, respectively, they both suffer from +inflexibility in expressing asynchronous execution graphs that many Python +programmers may be accustomed to by using, for example, the +[asyncio](https://docs.python.org/3/library/asyncio.html) library. + +Unlike static and dynamic workflows, eager workflows allow you to use all of +the python constructs that you're familiar with via the `asyncio` API. To +understand what this looks like, let's define a very basic eager workflow +using the `@eager` decorator. + +```{code-cell} +:lines_to_next_cell: 2 + +from flytekit import task, workflow +from flytekit.experimental import eager + + +@task +def add_one(x: int) -> int: + return x + 1 + + +@task +def double(x: int) -> int: + return x * 2 + + +@eager +async def simple_eager_workflow(x: int) -> int: + out = await add_one(x=x) + if out < 0: + return -1 + return await double(x=out) +``` + ++++ {"lines_to_next_cell": 2} + +As we can see in the code above, we're defining an `async` function called +`simple_eager_workflow` that takes an integer as input and returns an integer. +By decorating this function with `@eager`, we now have the ability to invoke +tasks, static subworkflows, and even other eager subworkflows in an _eager_ +fashion such that we can materialize their outputs and use them inside the +parent eager workflow itself. + +In the `simple_eager_workflow` function, we can see that we're `await`ing +the output of the `add_one` task and assigning it to the `out` variable. If +`out` is a negative integer, the workflow will return `-1`. Otherwise, it +will double the output of `add_one` and return it. + +Unlike in static and dynamic workflows, this variable is actually +the Python integer that is the result of `x + 1` and not a promise. + +## How it works + +When you decorate a function with `@eager`, any function invoked within it +that's decorated with `@task`, `@workflow`, or `@eager` becomes +an [awaitable](https://docs.python.org/3/library/asyncio-task.html#awaitables) +object within the lifetime of the parent eager workflow execution. Note that +this happens automatically and you don't need to use the `async` keyword when +defining a task or workflow that you want to invoke within an eager workflow. + +```{important} +With eager workflows, you basically have access to the Python `asyncio` +interface to define extremely flexible execution graphs! The trade-off is that +you lose the compile-time type safety that you get with regular static workflows +and to a lesser extent, dynamic workflows. + +We're leveraging Python's native `async` capabilities in order to: + +1. Materialize the output of flyte tasks and subworkflows so you can operate + on them without spinning up another pod and also determine the shape of the + workflow graph in an extremely flexible manner. +2. Provide an alternative way of achieving concurrency in Flyte. Flyte has + concurrency built into it, so all tasks/subworkflows will execute concurrently + assuming that they don't have any dependencies on each other. However, eager + workflows provide a python-native way of doing this, with the main downside + being that you lose the benefits of statically compiled workflows such as + compile-time analysis and first-class data lineage tracking. +``` + +Similar to {ref}`dynamic workflows `, eager workflows are +actually tasks. The main difference is that, while dynamic workflows compile +a static workflow at runtime using materialized inputs, eager workflows do +not compile any workflow at all. Instead, they use the {py:class}`~flytekit.remote.remote.FlyteRemote` +object together with Python's `asyncio` API to kick off tasks and subworkflow +executions eagerly whenever you `await` on a coroutine. This means that eager +workflows can materialize an output of a task or subworkflow and use it as a +Python object in the underlying runtime environment. We'll see how to configure +`@eager` functions to run on a remote Flyte cluster +{ref}`later in this guide `. + +## What can you do with eager workflows? + +In this section we'll cover a few of the use cases that you can accomplish +with eager workflows, some of which you can't accomplish with static or dynamic +workflows. + +### Operating on task and subworkflow outputs + +One of the biggest benefits of eager workflows is that you can now materialize +task and subworkflow outputs as Python values and do operations on them just +like you would in any other Python function. Let's look at another example: + +```{code-cell} +@eager +async def another_eager_workflow(x: int) -> int: + out = await add_one(x=x) + + # out is a Python integer + out = out - 1 + + return await double(x=out) +``` + ++++ {"lines_to_next_cell": 0} + +Since out is an actual Python integer and not a promise, we can do operations +on it at runtime, inside the eager workflow function body. This is not possible +with static or dynamic workflows. + +### Pythonic conditionals + +As you saw in the `simple_eager_workflow` workflow above, you can use regular +Python conditionals in your eager workflows. Let's look at a more complicated +example: + +```{code-cell} +:lines_to_next_cell: 2 + +@task +def gt_100(x: int) -> bool: + return x > 100 + + +@eager +async def eager_workflow_with_conditionals(x: int) -> int: + out = await add_one(x=x) + + if out < 0: + return -1 + elif await gt_100(x=out): + return 100 + else: + out = await double(x=out) + + assert out >= -1 + return out +``` + +In the above example, we're using the eager workflow's Python runtime +to check if `out` is negative, but we're also using the `gt_100` task in the +`elif` statement, which will be executed in a separate Flyte task. + +### Loops + +You can also gather the outputs of multiple tasks or subworkflows into a list: + +```{code-cell} +import asyncio + + +@eager +async def eager_workflow_with_for_loop(x: int) -> int: + outputs = [] + + for i in range(x): + outputs.append(add_one(x=i)) + + outputs = await asyncio.gather(*outputs) + return await double(x=sum(outputs)) +``` + ++++ {"lines_to_next_cell": 0} + +### Static subworkflows + +You can also invoke static workflows from within an eager workflow: + +```{code-cell} +:lines_to_next_cell: 2 + +@workflow +def subworkflow(x: int) -> int: + out = add_one(x=x) + return double(x=out) + + +@eager +async def eager_workflow_with_static_subworkflow(x: int) -> int: + out = await subworkflow(x=x) + assert out == (x + 1) * 2 + return out +``` + ++++ {"lines_to_next_cell": 0} + +### Eager subworkflows + +You can have nest eager subworkflows inside a parent eager workflow: + +```{code-cell} +:lines_to_next_cell: 2 + +@eager +async def eager_subworkflow(x: int) -> int: + return await add_one(x=x) + + +@eager +async def nested_eager_workflow(x: int) -> int: + out = await eager_subworkflow(x=x) + return await double(x=out) +``` + ++++ {"lines_to_next_cell": 0} + +### Catching exceptions + +You can also catch exceptions in eager workflows through `EagerException`: + +```{code-cell} +:lines_to_next_cell: 2 + +from flytekit.experimental import EagerException + + +@task +def raises_exc(x: int) -> int: + if x <= 0: + raise TypeError + return x + + +@eager +async def eager_workflow_with_exception(x: int) -> int: + try: + return await raises_exc(x=x) + except EagerException: + return -1 +``` + +Even though the `raises_exc` exception task raises a `TypeError`, the +`eager_workflow_with_exception` runtime will raise an `EagerException` and +you'll need to specify `EagerException` as the exception type in your `try... except` +block. + +```{note} +This is a current limitation in the `@eager` workflow implementation. +```` + +## Executing eager workflows + +As with most Flyte constructs, you can execute eager workflows both locally +and remotely. + +### Local execution + +You can execute eager workflows locally by simply calling them like a regular +`async` function: + +```{code-cell} +if __name__ == "__main__": + result = asyncio.run(simple_eager_workflow(x=5)) + print(f"Result: {result}") # "Result: 12" +``` + +This just uses the `asyncio.run` function to execute the eager workflow just +like any other Python async code. This is useful for local debugging as you're +developing your workflows and tasks. + +(eager_workflows_remote)= + +### Remote Flyte cluster execution + +Under the hood, `@eager` workflows use the {py:class}`~flytekit.remote.remote.FlyteRemote` +object to kick off task, static workflow, and eager workflow executions. + +In order to actually execute them on a Flyte cluster, you'll need to configure +eager workflows with a `FlyteRemote` object and secrets configuration that +allows you to authenticate into the cluster via a client secret key. + +```{code-block} python +from flytekit.remote import FlyteRemote +from flytekit.configuration import Config + +@eager( + remote=FlyteRemote( + config=Config.auto(config_file="config.yaml"), + default_project="flytesnacks", + default_domain="development", + ), + client_secret_group="", + client_secret_key="", +) +async def eager_workflow_remote(x: int) -> int: + ... +``` + ++++ + +Where `config.yaml` contains a +[flytectl](https://docs.flyte.org/projects/flytectl/en/latest/#configuration)-compatible +config file and `my_client_secret_group` and `my_client_secret_key` are the +{ref}`secret group and key ` that you've configured for your Flyte +cluster to authenticate via a client key. + ++++ + +### Sandbox Flyte cluster execution + +When using a sandbox cluster started with `flytectl demo start`, however, the +`client_secret_group` and `client_secret_key` are not required, since the +default sandbox configuration does not require key-based authentication. + +```{code-cell} +:lines_to_next_cell: 2 + +from flytekit.configuration import Config +from flytekit.remote import FlyteRemote + + +@eager( + remote=FlyteRemote( + config=Config.for_sandbox(), + default_project="flytesnacks", + default_domain="development", + ) +) +async def eager_workflow_sandbox(x: int) -> int: + out = await add_one(x=x) + if out < 0: + return -1 + return await double(x=out) +``` + +```{important} +When executing eager workflows on a remote Flyte cluster, it will execute the +latest version of tasks, static workflows, and eager workflows that are on +the `default_project` and `default_domain` as specified in the `FlyteRemote` +object. This means that you need to pre-register all Flyte entities that are +invoked inside of the eager workflow. +``` + +### Registering and running + +Assuming that your `flytekit` code is configured correctly, you will need to +register all of the task and subworkflows that are used with your eager +workflow with `pyflyte register`: + +```{prompt} bash +pyflyte --config register \ + --project \ + --domain \ + --image \ + path/to/eager_workflows.py +``` + +And then run it with `pyflyte run`: + +```{prompt} bash +pyflyte --config run \ + --project \ + --domain \ + --image \ + path/to/eager_workflows.py simple_eager_workflow --x 10 +``` + +```{note} +You need to register the tasks/workflows associated with your eager workflow +because eager workflows are actually flyte tasks under the hood, which means +that `pyflyte run` has no way of knowing what tasks and subworkflows are +invoked inside of it. +``` + +## Eager workflows on Flyte console + +Since eager workflows are an experimental feature, there is currently no +first-class representation of them on Flyte Console, the UI for Flyte. +When you register an eager workflow, you'll be able to see it in the task view: + +:::{figure} https://github.com/flyteorg/static-resources/blob/main/flytesnacks/user_guide/flyte_eager_workflow_ui_view.png?raw=true +:alt: Eager Workflow UI View +:class: with-shadow +::: + +When you execute an eager workflow, the tasks and subworkflows invoked within +it **won't show up** on the node, graph, or timeline view. As mentioned above, +this is because eager workflows are actually Flyte tasks under the hood and +Flyte has no way of knowing the shape of the execution graph before actually +executing them. + +:::{figure} https://github.com/flyteorg/static-resources/blob/main/flytesnacks/user_guide/flyte_eager_workflow_execution.png?raw=true +:alt: Eager Workflow Execution +:class: with-shadow +::: + +However, at the end of execution, you'll be able to use {ref}`Flyte Decks ` +to see a list of all the tasks and subworkflows that were executed within the +eager workflow: + +:::{figure} https://github.com/flyteorg/static-resources/blob/main/flytesnacks/user_guide/flyte_eager_workflow_deck.png?raw=true +:alt: Eager Workflow Deck +:class: with-shadow +::: + +## Limitations + +As this feature is still experimental, there are a few limitations that you +need to keep in mind: + +- You cannot invoke {ref}`dynamic workflows `, + {ref}`map tasks `, or {ref}`launch plans ` inside an + eager workflow. +- [Context managers](https://docs.python.org/3/library/contextlib.html) will + only work on locally executed functions within the eager workflow, i.e. using a + context manager to modify the behavior of a task or subworkflow will not work + because they are executed on a completely different pod. +- All exceptions raised by Flyte tasks or workflows will be caught and raised + as an {py:class}`~flytekit.experimental.EagerException` at runtime. +- All task/subworkflow outputs are materialized as Python values, which includes + offloaded types like `FlyteFile`, `FlyteDirectory`, `StructuredDataset`, and + `pandas.DataFrame` will be fully downloaded into the pod running the eager workflow. + This prevents you from incrementally downloading or streaming very large datasets + in eager workflows. +- Flyte entities that are invoked inside of an eager workflow must be registered + under the same project and domain as the eager workflow itself. The eager + workflow will execute the latest version of these entities. +- Flyte console currently does not have a first-class way of viewing eager + workflows, but it can be accessed via the task list view and the execution + graph is viewable via Flyte Decks. + +## Summary of workflows + +Eager workflows are a powerful new construct that trades-off compile-time type +safety for flexibility in the shape of the execution graph. The table below +will help you to reason about the different workflow constructs in Flyte in terms +of promises and materialized values: + +| Construct | Description | Flyte Promises | Pro | Con | +|--------|--------|--------|----|----| +| `@workflow` | Compiled at compile-time | All inputs and intermediary outputs are promises | Type errors caught at compile-time | Constrained by Flyte DSL | +| `@dynamic` | Compiled at run-time | Inputs are materialized, but outputs of all Flyte entities are Promises | More flexible than `@workflow`, e.g. can do Python operations on inputs | Can't use a lot of Python constructs (e.g. try/except) | +| `@eager` | Never compiled | Everything is materialized! | Can effectively use all Python constructs via `asyncio` syntax | No compile-time benefits, this is the wild west 🏜 | diff --git a/docs/user_guide/advanced_composition/index.md b/docs/user_guide/advanced_composition/index.md new file mode 100644 index 0000000000..26eb8df33c --- /dev/null +++ b/docs/user_guide/advanced_composition/index.md @@ -0,0 +1,24 @@ +(advanced_composition)= + +# Advanced composition + +This section of the user guide introduces the advanced features of the Flytekit Python SDK. +These examples cover more complex aspects of Flyte, including conditions, subworkflows, +dynamic workflows, map tasks, gate nodes and more. + +```{toctree} +:maxdepth: -1 +:name: advanced_composition_toc +:hidden: + +conditionals +chaining_flyte_entities +subworkflows +dynamic_workflows +map_tasks +eager_workflows +decorating_tasks +decorating_workflows +intratask_checkpoints +waiting_for_external_inputs +``` diff --git a/docs/user_guide/advanced_composition/intratask_checkpoints.md b/docs/user_guide/advanced_composition/intratask_checkpoints.md new file mode 100644 index 0000000000..703279abcb --- /dev/null +++ b/docs/user_guide/advanced_composition/intratask_checkpoints.md @@ -0,0 +1,137 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +# Intratask checkpoints + +```{eval-rst} +.. tags:: MachineLearning, Intermediate +``` + +A checkpoint in Flyte serves to recover a task from a previous failure by preserving the task's state before the failure +and resuming from the latest recorded state. + +## Why intratask checkpoints? + +The inherent design of Flyte, being a workflow engine, allows users to break down operations, programs or ideas +into smaller tasks within workflows. In the event of a task failure, the workflow doesn't need to rerun the +previously completed tasks. Instead, it can retry the specific task that encountered an issue. +Once the problematic task succeeds, it won't be rerun. Consequently, the natural boundaries between tasks act as implicit checkpoints. + +However, there are scenarios where breaking a task into smaller tasks is either challenging or undesirable due to the associated overhead. +This is especially true when running a substantial computation in a tight loop. +In such cases, users may consider splitting each loop iteration into individual tasks using dynamic workflows. +Yet, the overhead of spawning new tasks, recording intermediate results, and reconstructing the state can incur additional expenses. + +### Use case: Model training + +An exemplary scenario illustrating the utility of intra-task checkpointing is during model training. +In situations where executing multiple epochs or iterations with the same dataset might be time-consuming, +setting task boundaries can incur a high bootstrap time and be costly. + +Flyte addresses this challenge by providing a mechanism to checkpoint progress within a task execution, +saving it as a file or set of files. In the event of a failure, the checkpoint file can be re-read to +resume most of the state without rerunning the entire task. +This feature opens up possibilities to leverage alternate, more cost-effective compute systems, +such as [AWS spot instances](https://aws.amazon.com/ec2/spot/), +[GCP pre-emptible instances](https://cloud.google.com/compute/docs/instances/preemptible) and others. + +These instances offer great performance at significantly lower price points compared to their on-demand or reserved counterparts. +This becomes feasible when tasks are constructed in a fault-tolerant manner. +For tasks running within a short duration, e.g., less than 10 minutes, the likelihood of failure is negligible, +and task-boundary-based recovery provides substantial fault tolerance for successful completion. + +However, as the task execution time increases, the cost of re-running it also increases, +reducing the chances of successful completion. This is precisely where Flyte's intra-task checkpointing proves to be highly beneficial. + +Here's an example illustrating how to develop tasks that leverage intra-task checkpointing. +It's important to note that Flyte currently offers the low-level API for checkpointing. +Future integrations aim to incorporate higher-level checkpointing APIs from popular training frameworks +like Keras, PyTorch, Scikit-learn, and big-data frameworks such as Spark and Flink, enhancing their fault-tolerance capabilities. + +To begin, import the necessary libraries and set the number of task retries to `3`. + +```{code-cell} +from flytekit import current_context, task, workflow +from flytekit.exceptions.user import FlyteRecoverableException + +RETRIES = 3 +``` + ++++ {"lines_to_next_cell": 0} + +We define a task to iterate precisely `n_iterations`, checkpoint its state, and recover from simulated failures. + +```{code-cell} +@task(retries=RETRIES) +def use_checkpoint(n_iterations: int) -> int: + cp = current_context().checkpoint + prev = cp.read() + + start = 0 + if prev: + start = int(prev.decode()) + + # Create a failure interval to simulate failures across 'n' iterations and then succeed after configured retries + failure_interval = n_iterations // RETRIES + index = 0 + for index in range(start, n_iterations): + # Simulate a deterministic failure for demonstration. Showcasing how it eventually completes within the given retries + if index > start and index % failure_interval == 0: + raise FlyteRecoverableException(f"Failed at iteration {index}, failure_interval {failure_interval}.") + # Save progress state. It is also entirely possible to save state every few intervals + cp.write(f"{index + 1}".encode()) + return index +``` + ++++ {"lines_to_next_cell": 0} + +The checkpoint system offers additional APIs, documented in the code accessible at +[checkpointer code](https://github.com/flyteorg/flytekit/blob/master/flytekit/core/checkpointer.py). + +Create a workflow that invokes the task. +The task will automatically undergo retries in the event of a {ref}`FlyteRecoverableException `. + +```{code-cell} +@workflow +def checkpointing_example(n_iterations: int) -> int: + return use_checkpoint(n_iterations=n_iterations) +``` + ++++ {"lines_to_next_cell": 0} + +The local checkpoint is not utilized here because retries are not supported. + +```{code-cell} +if __name__ == "__main__": + try: + checkpointing_example(n_iterations=10) + except RuntimeError as e: # noqa : F841 + # Since no retries are performed, an exception is expected when run locally + pass +``` + +## Run the example on the Flyte cluster + +To run the provided workflow on the Flyte cluster, use the following command: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/checkpoint.py \ + checkpointing_example --n_iterations 10 +``` diff --git a/docs/user_guide/advanced_composition/map_tasks.md b/docs/user_guide/advanced_composition/map_tasks.md new file mode 100644 index 0000000000..6449b6d124 --- /dev/null +++ b/docs/user_guide/advanced_composition/map_tasks.md @@ -0,0 +1,278 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(map_task)= + +# Map tasks + +```{eval-rst} +.. tags:: Intermediate +``` + +Using a map task in Flyte allows for the execution of a pod task or a regular task across a series of inputs within a single workflow node. +This capability eliminates the need to create individual nodes for each instance, leading to substantial performance improvements. + +Map tasks find utility in diverse scenarios, such as: + +1. Executing the same code logic on multiple inputs +2. Concurrent processing of multiple data batches +3. Hyperparameter optimization + +The following examples demonstrate how to use map tasks with both single and multiple inputs. + +To begin, import the required libraries. + +```{code-cell} +from flytekit import map_task, task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +Here's a simple workflow that uses {py:func}`map_task `. + +```{code-cell} +threshold = 11 + + +@task +def detect_anomalies(data_point: int) -> bool: + return data_point > threshold + + +@workflow +def map_workflow(data: list[int] = [10, 12, 11, 10, 13, 12, 100, 11, 12, 10]) -> list[bool]: + # Use the map task to apply the anomaly detection function to each data point + return map_task(detect_anomalies)(data_point=data) + + +if __name__ == "__main__": + print(f"Anomalies Detected: {map_workflow()}") +``` + ++++ {"lines_to_next_cell": 0} + +To customize resource allocations, such as memory usage for individual map tasks, +you can leverage `with_overrides`. Here's an example using the `detect_anomalies` map task within a workflow: + +```python +from flytekit import Resources + + +@workflow +def map_workflow_with_resource_overrides(data: list[int] = [10, 12, 11, 10, 13, 12, 100, 11, 12, 10]) -> list[bool]: + return map_task(detect_anomalies)(data_point=data).with_overrides(requests=Resources(mem="2Gi")) +``` + +You can use {py:class}`~flytekit.TaskMetadata` to set attributes such as `cache`, `cache_version`, `interruptible`, `retries` and `timeout`. +```python +from flytekit import TaskMetadata + + +@workflow +def map_workflow_with_metadata(data: list[int] = [10, 12, 11, 10, 13, 12, 100, 11, 12, 10]) -> list[bool]: + return map_task(detect_anomalies, metadata=TaskMetadata(cache=True, cache_version="0.1", retries=1))( + data_point=data + ) +``` + +You can also configure `concurrency` and `min_success_ratio` for a map task: +- `concurrency` limits the number of mapped tasks that can run in parallel to the specified batch size. +If the input size exceeds the concurrency value, multiple batches will run serially until all inputs are processed. +If left unspecified, it implies unbounded concurrency. +- `min_success_ratio` determines the minimum fraction of total jobs that must complete successfully before terminating +the map task and marking it as successful. + +```python +@workflow +def map_workflow_with_additional_params(data: list[int] = [10, 12, 11, 10, 13, 12, 100, 11, 12, 10]) -> list[bool]: + return map_task(detect_anomalies, concurrency=1, min_success_ratio=0.75)(data_point=data) +``` + +A map task internally uses a compression algorithm (bitsets) to handle every Flyte workflow node’s metadata, +which would have otherwise been in the order of 100s of bytes. + +When defining a map task, avoid calling other tasks in it. Flyte +can't accurately register tasks that call other tasks. While Flyte +will correctly execute a task that calls other tasks, it will not be +able to give full performance advantages. This is +especially true for map tasks. + +In this example, the map task `suboptimal_mappable_task` would not +give you the best performance. + +```{code-cell} +@task +def upperhalf(a: int) -> int: + return a / 2 + 1 + + +@task +def suboptimal_mappable_task(a: int) -> str: + inc = upperhalf(a=a) + stringified = str(inc) + return stringified +``` + ++++ {"lines_to_next_cell": 0} + +By default, the map task utilizes the Kubernetes array plugin for execution. +However, map tasks can also be run on alternate execution backends. +For example, you can configure the map task to run on +[AWS Batch](https://docs.flyte.org/en/latest/deployment/plugin_setup/aws/batch.html#deployment-plugin-setup-aws-array), +a provisioned service that offers scalability for handling large-scale tasks. + +## Map a task with multiple inputs + +You might need to map a task with multiple inputs. + +For instance, consider a task that requires three inputs. + +```{code-cell} +@task +def multi_input_task(quantity: int, price: float, shipping: float) -> float: + return quantity * price * shipping +``` + ++++ {"lines_to_next_cell": 0} + +You may want to map this task with only the ``quantity`` input, while keeping the other inputs unchanged. +Since a map task accepts only one input, you can achieve this by partially binding values to the map task. +This can be done using the {py:func}`functools.partial` function. + +```{code-cell} +import functools + + +@workflow +def multiple_inputs_map_workflow(list_q: list[int] = [1, 2, 3, 4, 5], p: float = 6.0, s: float = 7.0) -> list[float]: + partial_task = functools.partial(multi_input_task, price=p, shipping=s) + return map_task(partial_task)(quantity=list_q) +``` + ++++ {"lines_to_next_cell": 0} + +Another possibility is to bind the outputs of a task to partials. + +```{code-cell} +@task +def get_price() -> float: + return 7.0 + + +@workflow +def map_workflow_partial_with_task_output(list_q: list[int] = [1, 2, 3, 4, 5], s: float = 6.0) -> list[float]: + p = get_price() + partial_task = functools.partial(multi_input_task, price=p, shipping=s) + return map_task(partial_task)(quantity=list_q) +``` + ++++ {"lines_to_next_cell": 0} + +You can also provide multiple lists as input to a ``map_task``. + +```{code-cell} +:lines_to_next_cell: 2 + +@workflow +def map_workflow_with_lists( + list_q: list[int] = [1, 2, 3, 4, 5], list_p: list[float] = [6.0, 9.0, 8.7, 6.5, 1.2], s: float = 6.0 +) -> list[float]: + partial_task = functools.partial(multi_input_task, shipping=s) + return map_task(partial_task)(quantity=list_q, price=list_p) +``` + +```{note} +It is important to note that you cannot provide a list as an input to a partial task. +``` + +## Run the example on the Flyte cluster + +To run the provided workflows on the Flyte cluster, use the following commands: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + map_workflow +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + map_workflow_with_additional_params +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + multiple_inputs_map_workflow +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + map_workflow_partial_with_task_output +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/map_task.py \ + map_workflow_with_lists +``` + +## ArrayNode + +:::{important} +This feature is experimental and the API is subject to breaking changes. +If you encounter any issues please consider submitting a +[bug report](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&projects=&template=bug_report.yaml&title=%5BBUG%5D+). +::: + +ArrayNode map tasks serve as a seamless substitution for regular map tasks, differing solely in the submodule +utilized to import the `map_task` function. Specifically, you will need to import `map_task` from the experimental module as illustrated below: + +```python +from flytekit import task, workflow +from flytekit.experimental import map_task + +@task +def t(a: int) -> int: + ... + +@workflow +def array_node_wf(xs: list[int]) -> list[int]: + return map_task(t)(a=xs) +``` + +Flyte introduces map task to enable parallelization of homogeneous operations, +offering efficient evaluation and a user-friendly API. Because it’s implemented as a backend plugin, +its evaluation is independent of core Flyte logic, which generates subtask executions that lack full Flyte functionality. +ArrayNode tackles this issue by offering robust support for subtask executions. +It also extends mapping capabilities across all plugins and Flyte node types. +This enhancement will be a part of our move from the experimental phase to general availability. + +In contrast to map tasks, an ArrayNode provides the following enhancements: + +- **Wider mapping support**. ArrayNode extends mapping capabilities beyond Kubernetes tasks, encompassing tasks such as Python tasks, container tasks and pod tasks. +- **Cache management**. It supports both cache serialization and cache overwriting for subtask executions. +- **Intra-task checkpointing**. ArrayNode enables intra-task checkpointing, contributing to improved execution reliability. +- **Workflow recovery**. Subtasks remain recoverable during the workflow recovery process. (This is a work in progress.) +- **Subtask failure handling**. The mechanism handles subtask failures effectively, ensuring that running subtasks are appropriately aborted. +- **Multiple input values**. Subtasks can be defined with multiple input values, enhancing their versatility. + +We expect the performance of ArrayNode map tasks to compare closely to standard map tasks. diff --git a/docs/user_guide/advanced_composition/subworkflows.md b/docs/user_guide/advanced_composition/subworkflows.md new file mode 100644 index 0000000000..59826aa491 --- /dev/null +++ b/docs/user_guide/advanced_composition/subworkflows.md @@ -0,0 +1,182 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(subworkflow)= + +# Subworkflows + +```{eval-rst} +.. tags:: Intermediate +``` + +Subworkflows share similarities with {ref}`launch plans `, as both enable users to initiate one workflow from within another. +The distinction lies in the analogy: think of launch plans as "pass by pointer" and subworkflows as "pass by value." + +## When to use subworkflows? + +Subworkflows offer an elegant solution for managing parallelism between a workflow and its launched sub-flows, +as they execute within the same context as the parent workflow. +Consequently, all nodes of a subworkflow adhere to the overall constraints imposed by the parent workflow. + +Consider this scenario: when workflow `A` is integrated as a subworkflow of workflow `B`, +running workflow `B` results in the entire graph of workflow `A` being duplicated into workflow `B` at the point of invocation. + +Here's an example illustrating the calculation of slope, intercept and the corresponding y-value. + +```{code-cell} +from flytekit import task, workflow + + +@task +def slope(x: list[int], y: list[int]) -> float: + sum_xy = sum([x[i] * y[i] for i in range(len(x))]) + sum_x_squared = sum([x[i] ** 2 for i in range(len(x))]) + n = len(x) + return (n * sum_xy - sum(x) * sum(y)) / (n * sum_x_squared - sum(x) ** 2) + + +@task +def intercept(x: list[int], y: list[int], slope: float) -> float: + mean_x = sum(x) / len(x) + mean_y = sum(y) / len(y) + intercept = mean_y - slope * mean_x + return intercept + + +@workflow +def slope_intercept_wf(x: list[int], y: list[int]) -> (float, float): + slope_value = slope(x=x, y=y) + intercept_value = intercept(x=x, y=y, slope=slope_value) + return (slope_value, intercept_value) + + +@task +def regression_line(val: int, slope_value: float, intercept_value: float) -> float: + return (slope_value * val) + intercept_value # y = mx + c + + +@workflow +def regression_line_wf(val: int = 5, x: list[int] = [-3, 0, 3], y: list[int] = [7, 4, -2]) -> float: + slope_value, intercept_value = slope_intercept_wf(x=x, y=y) + return regression_line(val=val, slope_value=slope_value, intercept_value=intercept_value) +``` + ++++ {"lines_to_next_cell": 0} + +The `slope_intercept_wf` computes the slope and intercept of the regression line. +Subsequently, the `regression_line_wf` triggers `slope_intercept_wf` and then computes the y-value. + +To execute the workflow locally, use the following: + +```{code-cell} +if __name__ == "__main__": + print(f"Executing regression_line_wf(): {regression_line_wf()}") +``` + ++++ {"lines_to_next_cell": 0} + +It's possible to nest a workflow that contains a subworkflow within another workflow. +Workflows can be easily constructed from other workflows, even if they function as standalone entities. +Each workflow in this module has the capability to exist and run independently. + +```{code-cell} +@workflow +def nested_regression_line_wf() -> float: + return regression_line_wf() +``` + ++++ {"lines_to_next_cell": 0} + +You can run the nested workflow locally as well. + +```{code-cell} +if __name__ == "__main__": + print(f"Running nested_regression_line_wf(): {nested_regression_line_wf()}") +``` + ++++ {"lines_to_next_cell": 0} + +## External workflow + +When launch plans are employed within a workflow to initiate the execution of a pre-defined workflow, +a new external execution is triggered. This results in a distinct execution ID and can be identified +as a separate entity. + +These external invocations of a workflow, initiated using launch plans from a parent workflow, +are termed as external workflows. They may have separate parallelism constraints since the context is not shared. + +:::{tip} +If your deployment uses {ref}`multiple Kubernetes clusters `, +external workflows may offer a way to distribute the workload of a workflow across multiple clusters. +::: + +Here's an example that illustrates the concept of external workflows: + +```{code-cell} + +from flytekit import LaunchPlan + +launch_plan = LaunchPlan.get_or_create( + regression_line_wf, "regression_line_workflow", default_inputs={"val": 7, "x": [-3, 0, 3], "y": [7, 4, -2]} +) + + +@workflow +def nested_regression_line_lp() -> float: + # Trigger launch plan from within a workflow + return launch_plan() +``` + ++++ {"lines_to_next_cell": 0} + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_external_workflow_execution.png +:alt: External workflow execution +:class: with-shadow +::: + +In the console screenshot above, note that the launch plan execution ID differs from that of the workflow. + +You can run a workflow containing an external workflow locally as follows: + +```{code-cell} +if __name__ == "__main__": + print(f"Running nested_regression_line_lp(): {nested_regression_line_lp}") +``` + +## Run the example on a Flyte cluster + +To run the provided workflows on a Flyte cluster, use the following commands: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py \ + regression_line_wf +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py \ + nested_regression_line_wf +``` + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/advanced_composition/advanced_composition/subworkflow.py \ + nested_regression_line_lp +``` diff --git a/docs/user_guide/advanced_composition/waiting_for_external_inputs.md b/docs/user_guide/advanced_composition/waiting_for_external_inputs.md new file mode 100644 index 0000000000..d694b62443 --- /dev/null +++ b/docs/user_guide/advanced_composition/waiting_for_external_inputs.md @@ -0,0 +1,314 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +# Waiting for external inputs + +*New in Flyte 1.3.0* + +There are use cases where you may want a workflow execution to pause, only to continue +when some time has passed or when it receives some inputs that are external to +the workflow execution inputs. You can think of these as execution-time inputs, +since they need to be supplied to the workflow after it's launched. Examples of +this use case would be: + +1. **Model Deployment**: A hyperparameter-tuning workflow that + trains `n` models, where a human needs to inspect a report before approving + the model for downstream deployment to some serving layer. +2. **Data Labeling**: A workflow that iterates through an image dataset, + presenting individual images to a human annotator for them to label. +3. **Active Learning**: An [active learning]() + workflow that trains a model, shows examples for a human annotator to label + based which examples it's least/most certain about or would provide the most + information to the model. + +These use cases can be achieved in Flyte with the {func}`~flytekit.sleep`, +{func}`~flytekit.wait_for_input`, and {func}`~flytekit.approve` workflow nodes. +Although all of the examples above are human-in-the-loop processes, these +constructs allow you to pass inputs into a workflow from some arbitrary external +process (👩 human or 🤖 machine) in order to continue. + +:::{important} +These functions can only be used inside {func}`@workflow `-decorated +functions, {func}`@dynamic `-decorated functions, or +{ref}`imperative workflows `. +::: + +## Pause executions with the `sleep` node + +The simplest case is when you want your workflow to {py:func}`~flytekit.sleep` +for some specified amount of time before continuing. + +Though this type of node may not be used often in a production setting, +you might want to use it, for example, if you want to simulate a delay in +your workflow to mock out the behavior of some long-running computation. + +```{code-cell} +from datetime import timedelta + +from flytekit import sleep, task, workflow + + +@task +def long_running_computation(num: int) -> int: + """A mock task pretending to be a long-running computation.""" + return num + + +@workflow +def sleep_wf(num: int) -> int: + """Simulate a "long-running" computation with sleep.""" + + # increase the sleep duration to actually make it long-running + sleeping = sleep(timedelta(seconds=10)) + result = long_running_computation(num=num) + sleeping >> result + return result +``` + ++++ {"lines_to_next_cell": 0} + +As you can see above, we define a simple `add_one` task and a `sleep_wf` +workflow. We first create a `sleeping` and `result` node, then +order the dependencies with the `>>` operator such that the workflow sleeps +for 10 seconds before kicking off the `result` computation. Finally, we +return the `result`. + +:::{note} +You can learn more about the `>>` chaining operator +{ref}`here `. +::: + +Now that you have a general sense of how this works, let's move onto the +{func}`~flytekit.wait_for_input` workflow node. + +## Supply external inputs with `wait_for_input` + +With the {py:func}`~flytekit.wait_for_input` node, you can pause a +workflow execution that requires some external input signal. For example, +suppose that you have a workflow that publishes an automated analytics report, +but before publishing it you want to give it a custom title. You can achieve +this by defining a `wait_for_input` node that takes a `str` input and +finalizes the report: + +```{code-cell} +import typing + +from flytekit import wait_for_input + + +@task +def create_report(data: typing.List[float]) -> dict: # o0 + """A toy report task.""" + return { + "mean": sum(data) / len(data), + "length": len(data), + "max": max(data), + "min": min(data), + } + + +@task +def finalize_report(report: dict, title: str) -> dict: + return {"title": title, **report} + + +@workflow +def reporting_wf(data: typing.List[float]) -> dict: + report = create_report(data=data) + title_input = wait_for_input("title", timeout=timedelta(hours=1), expected_type=str) + return finalize_report(report=report, title=title_input) +``` + +Let's breakdown what's happening in the code above: + +- In `reporting_wf` we first create the raw `report` +- Then, we define a `title` node that will wait for a string to be provided + through the Flyte API, which can be done through the Flyte UI or through + `FlyteRemote` (more on that later). This node will time out after 1 hour. +- Finally, we pass the `title_input` promise into `finalize_report`, which + attaches the custom title to the report. + +:::{note} +The `create_report` task is just toy example. In a realistic example, this +report might be an html file or set of visualizations. This can be rendered +in the Flyte UI with {ref}`Flyte Decks `. +::: + +As mentioned in the beginning of this page, this construct can be used for +selecting the best-performing model in cases where there isn't a clear single +metric to determine the best model, or if you're doing data labeling using +a Flyte workflow. + +## Continue executions with `approve` + +Finally, the {py:func}`~flytekit.approve` workflow node allows you to wait on +an explicit approval signal before continuing execution. Going back to our +report-publishing use case, suppose that we want to block the publishing of +a report for some reason (e.g. if they don't appear to be valid): + +```{code-cell} +from flytekit import approve + + +@workflow +def reporting_with_approval_wf(data: typing.List[float]) -> dict: + report = create_report(data=data) + title_input = wait_for_input("title", timeout=timedelta(hours=1), expected_type=str) + final_report = finalize_report(report=report, title=title_input) + + # approve the final report, where the output of approve is the final_report + # dictionary. + return approve(final_report, "approve-final-report", timeout=timedelta(hours=2)) +``` + ++++ {"lines_to_next_cell": 0} + +The `approve` node will pass the `final_report` promise through as the +output of the workflow, provided that the `approve-final-report` gets an +approval input via the Flyte UI or Flyte API. + +You can also use the output of the `approve` function as a promise, feeding +it to a subsequent task. Let's create a version of our report-publishing +workflow where the approval happens after `create_report`: + +```{code-cell} +@workflow +def approval_as_promise_wf(data: typing.List[float]) -> dict: + report = create_report(data=data) + title_input = wait_for_input("title", timeout=timedelta(hours=1), expected_type=str) + + # wait for report to run so that the user can view it before adding a custom + # title to the report + report >> title_input + + final_report = finalize_report( + report=approve(report, "raw-report-approval", timeout=timedelta(hours=2)), + title=title_input, + ) + return final_report +``` + ++++ {"lines_to_next_cell": 0} + +## Working with conditionals + +The node constructs by themselves are useful, but they become even more +useful when we combine them with other Flyte constructs, like {ref}`conditionals `. + +To illustrate this, let's extend the report-publishing use case so that we +produce an "invalid report" output in case we don't approve the final report: + +```{code-cell} +:lines_to_next_cell: 2 + +from flytekit import conditional + + +@task +def invalid_report() -> dict: + return {"invalid_report": True} + + +@workflow +def conditional_wf(data: typing.List[float]) -> dict: + report = create_report(data=data) + title_input = wait_for_input("title-input", timeout=timedelta(hours=1), expected_type=str) + + # Define a "review-passes" wait_for_input node so that a human can review + # the report before finalizing it. + review_passed = wait_for_input("review-passes", timeout=timedelta(hours=2), expected_type=bool) + report >> review_passed + + # This conditional returns the finalized report if the review passes, + # otherwise it returns an invalid report output. + return ( + conditional("final-report-condition") + .if_(review_passed.is_true()) + .then(finalize_report(report=report, title=title_input)) + .else_() + .then(invalid_report()) + ) +``` + +On top of the `approved` node, which we use in the `conditional` to +determine which branch to execute, we also define a `disapprove_reason` +gate node, which will be used as an input to the `invalid_report` task. + +## Sending inputs to `wait_for_input` and `approve` nodes + +Assuming that you've registered the above workflows on a Flyte cluster that's +been started with {ref}`flytectl demo start `, +there are two ways of using `wait_for_input` and `approve` nodes: + +### Using the Flyte UI + +If you launch the `reporting_wf` workflow on the Flyte UI, you'll see a +**Graph** view of the workflow execution like this: + +```{image} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/wait_for_input_graph.png +:alt: reporting workflow wait for input graph +``` + +Clicking on the {fa}`play-circle,style=far` icon of the `title` task node or the +**Resume** button on the sidebar will create a modal form that you can use to +provide the custom title input. + +```{image} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/wait_for_input_form.png +:alt: reporting workflow wait for input form +``` + +### Using `FlyteRemote` + +For many cases it's enough to use Flyte UI to provide inputs/approvals on +gate nodes. However, if you want to pass inputs to `wait_for_input` and +`approve` nodes programmatically, you can use the +{py:meth}`FlyteRemote.set_signal ` +method. Using the `gate_node_with_conditional_wf` workflow, the example +below allows you to set values for `title-input` and `review-passes` nodes. + +```python +import typing +from flytekit.remote.remote import FlyteRemote +from flytekit.configuration import Config + +remote = FlyteRemote( + Config.for_sandbox(), + default_project="flytesnacks", + default_domain="development", +) + +# First kick off the wotrkflow +flyte_workflow = remote.fetch_workflow( + name="core.control_flow.waiting_for_external_inputs.conditional_wf" +) + +# Execute the workflow +execution = remote.execute(flyte_workflow, inputs={"data": [1.0, 2.0, 3.0, 4.0, 5.0]}) + +# Get a list of signals available for the execution +signals = remote.list_signals(execution.id.name) + +# Set a signal value for the "title" node. Make sure that the "title-input" +# node is in the `signals` list above +remote.set_signal("title-input", execution.id.name, "my report") + +# Set signal value for the "review-passes" node. Make sure that the "review-passes" +# node is in the `signals` list above +remote.set_signal("review-passes", execution.id.name, True) +``` diff --git a/docs/user_guide/basics/documenting_workflows.md b/docs/user_guide/basics/documenting_workflows.md new file mode 100644 index 0000000000..d6a561c532 --- /dev/null +++ b/docs/user_guide/basics/documenting_workflows.md @@ -0,0 +1,157 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +# Documenting workflows + +```{eval-rst} +.. tags:: Basic +``` + +Well-documented code significantly improves code readability. +Flyte enables the use of docstrings to document your code. +Docstrings are stored in [FlyteAdmin](https://docs.flyte.org/en/latest/concepts/admin.html) +and displayed on the UI. + +To begin, import the relevant libraries. + +```{code-cell} +from typing import Tuple + +from flytekit import workflow +``` + ++++ {"lines_to_next_cell": 0} + +We import the `slope` and `intercept` tasks from the `workflow.py` file. + +```{code-cell} +from .workflow import intercept, slope +``` + ++++ {"lines_to_next_cell": 0} + +## Sphinx-style docstring + +An example to demonstrate Sphinx-style docstring. + +The initial section of the docstring provides a concise overview of the workflow. +The subsequent section provides a comprehensive explanation. +The last part of the docstring outlines the parameters and return type. + +```{code-cell} +@workflow +def sphinx_docstring_wf(x: list[int] = [-3, 0, 3], y: list[int] = [7, 4, -2]) -> Tuple[float, float]: + """ + Slope and intercept of a regression line + + This workflow accepts a list of coefficient pairs for a regression line. + It calculates both the slope and intercept of the regression line. + + :param x: List of x-coefficients + :param y: List of y-coefficients + :return: Slope and intercept values + """ + slope_value = slope(x=x, y=y) + intercept_value = intercept(x=x, y=y, slope=slope_value) + return slope_value, intercept_value +``` + ++++ {"lines_to_next_cell": 0} + +## NumPy-style docstring + +An example to demonstrate NumPy-style docstring. + +The first part of the docstring provides a concise overview of the workflow. +The next section offers a comprehensive description. +The third section of the docstring details all parameters along with their respective data types. +The final section of the docstring explains the return type and its associated data type. + +```{code-cell} +@workflow +def numpy_docstring_wf(x: list[int] = [-3, 0, 3], y: list[int] = [7, 4, -2]) -> Tuple[float, float]: + """ + Slope and intercept of a regression line + + This workflow accepts a list of coefficient pairs for a regression line. + It calculates both the slope and intercept of the regression line. + + Parameters + ---------- + x : list[int] + List of x-coefficients + y : list[int] + List of y-coefficients + + Returns + ------- + out : Tuple[float, float] + Slope and intercept values + """ + slope_value = slope(x=x, y=y) + intercept_value = intercept(x=x, y=y, slope=slope_value) + return slope_value, intercept_value +``` + ++++ {"lines_to_next_cell": 0} + +## Google-style docstring + +An example to demonstrate Google-style docstring. + +The initial section of the docstring offers a succinct one-liner summary of the workflow. +The subsequent section of the docstring provides an extensive explanation. +The third segment of the docstring outlines the parameters and return type, +including their respective data types. + +```{code-cell} +:lines_to_next_cell: 2 + +@workflow +def google_docstring_wf(x: list[int] = [-3, 0, 3], y: list[int] = [7, 4, -2]) -> Tuple[float, float]: + """ + Slope and intercept of a regression line + + This workflow accepts a list of coefficient pairs for a regression line. + It calculates both the slope and intercept of the regression line. + + Args: + x (list[int]): List of x-coefficients + y (list[int]): List of y-coefficients + + Returns: + Tuple[float, float]: Slope and intercept values + """ + slope_value = slope(x=x, y=y) + intercept_value = intercept(x=x, y=y, slope=slope_value) + return slope_value, intercept_value +``` + +Here are two screenshots showcasing how the description appears on the UI: +1. On the workflow page, you'll find the short description: +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/document_wf_short.png +:alt: Short description +:class: with-shadow +::: + +2. If you click into the workflow, you'll see the long description in the basic information section: +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/document_wf_long.png +:alt: Long description +:class: with-shadow +::: diff --git a/docs/user_guide/basics/hello_world.md b/docs/user_guide/basics/hello_world.md new file mode 100644 index 0000000000..45e5e89c4d --- /dev/null +++ b/docs/user_guide/basics/hello_world.md @@ -0,0 +1,75 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + + +# Hello, World! + +```{eval-rst} +.. tags:: Basic +``` + +Let's write a Flyte {py:func}`~flytekit.workflow` that invokes a +{py:func}`~flytekit.task` to generate the output "Hello, World!". + +Flyte tasks are the core building blocks of larger, more complex workflows. +Workflows compose multiple tasks – or other workflows – +into meaningful steps of computation to produce some useful set of outputs or outcomes. + +To begin, import `task` and `workflow` from the `flytekit` library. + +```{code-cell} +from flytekit import task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +Define a task that produces the string "Hello, World!". +Simply using the `@task` decorator to annotate the Python function. + +```{code-cell} +@task +def say_hello() -> str: + return "Hello, World!" +``` + ++++ {"lines_to_next_cell": 0} + +You can handle the output of a task in the same way you would with a regular Python function. +Store the output in a variable and use it as a return value for a Flyte workflow. + +```{code-cell} +@workflow +def hello_world_wf() -> str: + res = say_hello() + return res +``` + ++++ {"lines_to_next_cell": 0} + +Run the workflow by simply calling it like a Python function. + +```{code-cell} +:lines_to_next_cell: 2 + +if __name__ == "__main__": + print(f"Running hello_world_wf() {hello_world_wf()}") +``` + +Next, let's delve into the specifics of {ref}`tasks `, +{ref}`workflows ` and {ref}`launch plans `. diff --git a/docs/user_guide/basics/imperative_workflows.md b/docs/user_guide/basics/imperative_workflows.md new file mode 100644 index 0000000000..b5da5b6336 --- /dev/null +++ b/docs/user_guide/basics/imperative_workflows.md @@ -0,0 +1,119 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(imperative_workflow)= + +# Imperative workflows + +```{eval-rst} +.. tags:: Basic +``` + +Workflows are commonly created by applying the `@workflow` decorator to Python functions. +During compilation, this involves processing the function's body and utilizing subsequent calls to +underlying tasks to establish and record the workflow structure. This approach is known as declarative +and is suitable when manually drafting the workflow. + +However, in cases where workflows are constructed programmatically, an imperative style is more appropriate. +For instance, if tasks have been defined already, their sequence and dependencies might have been specified +in textual form (perhaps during a transition from a legacy system). +In such scenarios, you want to orchestrate these tasks. +This is where Flyte's imperative workflows come into play, allowing you to programmatically construct workflows. + +To begin, import the necessary dependencies. + +```{code-cell} +from flytekit import Workflow +``` + ++++ {"lines_to_next_cell": 0} + +We import the `slope` and `intercept` tasks from the `workflow.py` file. + +```{code-cell} +from .workflow import intercept, slope +``` + ++++ {"lines_to_next_cell": 0} + +Create an imperative workflow. + +```{code-cell} +imperative_wf = Workflow(name="imperative_workflow") +``` + ++++ {"lines_to_next_cell": 0} + +Add the workflow inputs to the imperative workflow. + +```{code-cell} +imperative_wf.add_workflow_input("x", list[int]) +imperative_wf.add_workflow_input("y", list[int]) +``` + ++++ {"lines_to_next_cell": 0} + +::: {note} +If you want to assign default values to the workflow inputs, +you can create a {ref}`launch plan `. +::: + +Add the tasks that need to be triggered from within the workflow. + +```{code-cell} +node_t1 = imperative_wf.add_entity(slope, x=imperative_wf.inputs["x"], y=imperative_wf.inputs["y"]) +node_t2 = imperative_wf.add_entity( + intercept, x=imperative_wf.inputs["x"], y=imperative_wf.inputs["y"], slope=node_t1.outputs["o0"] +) +``` + ++++ {"lines_to_next_cell": 0} + +Lastly, add the workflow output. + +```{code-cell} +imperative_wf.add_workflow_output("wf_output", node_t2.outputs["o0"]) +``` + ++++ {"lines_to_next_cell": 0} + +You can execute the workflow locally as follows: + +```{code-cell} +if __name__ == "__main__": + print(f"Running imperative_wf() {imperative_wf(x=[-3, 0, 3], y=[7, 4, -2])}") +``` + +:::{note} +You also have the option to provide a list of inputs and +retrieve a list of outputs from the workflow. + +```python +wf_input_y = imperative_wf.add_workflow_input("y", list[str]) +node_t3 = wf.add_entity(some_task, a=[wf.inputs["x"], wf_input_y]) +``` + +```python +wf.add_workflow_output( + "list_of_outputs", + [node_t1.outputs["o0"], node_t2.outputs["o0"]], + python_type=list[str], +) +``` +::: diff --git a/docs/user_guide/basics/index.md b/docs/user_guide/basics/index.md new file mode 100644 index 0000000000..bc97b74cc9 --- /dev/null +++ b/docs/user_guide/basics/index.md @@ -0,0 +1,25 @@ +# Basics + +This section introduces you to the basic building blocks of Flyte +using `flytekit`. `flytekit` is a Python SDK for developing Flyte workflows and +tasks, and can be used generally, whenever stateful computation is desirable. +`flytekit` workflows and tasks are completely runnable locally, unless they need +some advanced backend functionality like starting a distributed Spark cluster. + +Here, you will learn how to write Flyte tasks, assemble them into workflows, +run bash scripts, and document workflows. + +```{toctree} +:maxdepth: -1 +:name: basics_toc +:hidden: + +hello_world +tasks +workflows +launch_plans +imperative_workflows +documenting_workflows +shell_tasks +named_outputs +``` diff --git a/docs/user_guide/basics/launch_plans.md b/docs/user_guide/basics/launch_plans.md new file mode 100644 index 0000000000..01eb9d1051 --- /dev/null +++ b/docs/user_guide/basics/launch_plans.md @@ -0,0 +1,116 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(launch_plan)= + +# Launch plans + +```{eval-rst} +.. tags:: Basic +``` + +Launch plans link a partial or complete list of inputs required to initiate a workflow, +accompanied by optional run-time overrides like notifications, schedules and more. +They serve various purposes: + +- Schedule the same workflow multiple times, with optional predefined inputs. +- Run a specific workflow but with altered notifications. +- Share a workflow with predefined inputs, allowing another user to initiate an execution. +- Share a workflow with the option for the other user to override certain inputs. +- Share a workflow, ensuring specific inputs remain unchanged. + +Launch plans are the only means for invoking workflow executions. +When a workflow is serialized and registered, a _default launch plan_ is generated. +This default launch plan can bind default workflow inputs and runtime options defined +in the project's flytekit configuration (such as user role). + +To begin, import the necessary libraries. + +```{code-cell} +from flytekit import LaunchPlan, current_context +``` + ++++ {"lines_to_next_cell": 0} + +We import the workflow from the `workflow.py` file for which we're going to create a launch plan. + +```{code-cell} +from .workflow import simple_wf +``` + ++++ {"lines_to_next_cell": 0} + +Create a default launch plan with no inputs during serialization. + +```{code-cell} +default_lp = LaunchPlan.get_default_launch_plan(current_context(), simple_wf) +``` + ++++ {"lines_to_next_cell": 0} + +You can run the launch plan locally as follows: + +```{code-cell} +default_lp(x=[-3, 0, 3], y=[7, 4, -2]) +``` + ++++ {"lines_to_next_cell": 0} + +Create a launch plan and specify the default inputs. + +```{code-cell} +simple_wf_lp = LaunchPlan.create( + name="simple_wf_lp", workflow=simple_wf, default_inputs={"x": [-3, 0, 3], "y": [7, 4, -2]} +) +``` + ++++ {"lines_to_next_cell": 0} + +You can trigger the launch plan locally as follows: + +```{code-cell} +simple_wf_lp() +``` + ++++ {"lines_to_next_cell": 0} + +You can override the defaults as follows: + +```{code-cell} +simple_wf_lp(x=[3, 5, 3], y=[-3, 2, -2]) +``` + ++++ {"lines_to_next_cell": 0} + +It's possible to lock launch plan inputs, preventing them from being overridden during execution. + +```{code-cell} +simple_wf_lp_fixed_inputs = LaunchPlan.get_or_create( + name="fixed_inputs", workflow=simple_wf, fixed_inputs={"x": [-3, 0, 3]} +) +``` + +Attempting to modify the inputs will result in an error being raised by Flyte. + +:::{note} +You can employ default and fixed inputs in conjunction in a launch plan. +::: + +Launch plans can also be used to run workflows on a specific cadence. +For more information, refer to the {ref}`scheduling_launch_plan` documentation. diff --git a/docs/user_guide/basics/named_outputs.md b/docs/user_guide/basics/named_outputs.md new file mode 100644 index 0000000000..a609cd50a9 --- /dev/null +++ b/docs/user_guide/basics/named_outputs.md @@ -0,0 +1,116 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(named_outputs)= + +# Named outputs + +```{eval-rst} +.. tags:: Basic +``` + +By default, Flyte employs a standardized convention to assign names to the outputs of tasks or workflows. +Each output is sequentially labeled as `o1`, `o2`, `o3`, ... `on`, where `o` serves as the standard prefix, +and `1`, `2`, ... `n` indicates the positional index within the returned values. + +However, Flyte allows the customization of output names for tasks or workflows. +This customization becomes beneficial when you're returning multiple outputs +and you wish to assign a distinct name to each of them. + +The following example illustrates the process of assigning names to outputs for both a task and a workflow. + +To begin, import the required dependencies. + +```{code-cell} +from typing import NamedTuple + +from flytekit import task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +Define a `NamedTuple` and assign it as an output to a task. + +```{code-cell} +slope_value = NamedTuple("slope_value", [("slope", float)]) + + +@task +def slope(x: list[int], y: list[int]) -> slope_value: + sum_xy = sum([x[i] * y[i] for i in range(len(x))]) + sum_x_squared = sum([x[i] ** 2 for i in range(len(x))]) + n = len(x) + return (n * sum_xy - sum(x) * sum(y)) / (n * sum_x_squared - sum(x) ** 2) +``` + ++++ {"lines_to_next_cell": 0} + +Likewise, assign a `NamedTuple` to the output of `intercept` task. + +```{code-cell} +intercept_value = NamedTuple("intercept_value", [("intercept", float)]) + + +@task +def intercept(x: list[int], y: list[int], slope: float) -> intercept_value: + mean_x = sum(x) / len(x) + mean_y = sum(y) / len(y) + intercept = mean_y - slope * mean_x + return intercept +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +While it's possible to create `NamedTuple`s directly within the code, +it's often better to declare them explicitly. This helps prevent potential linting errors in tools like mypy. + +``` +def slope() -> NamedTuple("slope_value", slope=float): + pass +``` +::: + +You can easily unpack the `NamedTuple` outputs directly within a workflow. +Additionally, you can also have the workflow return a `NamedTuple` as an output. + +:::{note} +Remember that we are extracting individual task execution outputs by dereferencing them. +This is necessary because `NamedTuple`s function as tuples and require this dereferencing. +::: + +```{code-cell} +slope_and_intercept_values = NamedTuple("slope_and_intercept_values", [("slope", float), ("intercept", float)]) + + +@workflow +def simple_wf_with_named_outputs(x: list[int] = [-3, 0, 3], y: list[int] = [7, 4, -2]) -> slope_and_intercept_values: + slope_value = slope(x=x, y=y) + intercept_value = intercept(x=x, y=y, slope=slope_value.slope) + return slope_and_intercept_values(slope=slope_value.slope, intercept=intercept_value.intercept) +``` + ++++ {"lines_to_next_cell": 0} + +You can run the workflow locally as follows: + +```{code-cell} +if __name__ == "__main__": + print(f"Running simple_wf_with_named_outputs() {simple_wf_with_named_outputs()}") +``` diff --git a/docs/user_guide/basics/shell_tasks.md b/docs/user_guide/basics/shell_tasks.md new file mode 100644 index 0000000000..73cc5ab6b8 --- /dev/null +++ b/docs/user_guide/basics/shell_tasks.md @@ -0,0 +1,145 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(shell_task)= + +# Shell tasks + +```{eval-rst} +.. tags:: Basic +``` + +To execute bash scripts within Flyte, you can utilize the {py:class}`~flytekit.extras.tasks.shell.ShellTask` class. +This example includes three shell tasks to execute bash commands. + +First, import the necessary libraries. + +```{code-cell} +from pathlib import Path +from typing import Tuple + +import flytekit +from flytekit import kwtypes, task, workflow +from flytekit.extras.tasks.shell import OutputLocation, ShellTask +from flytekit.types.directory import FlyteDirectory +from flytekit.types.file import FlyteFile +``` + ++++ {"lines_to_next_cell": 0} + +With the required imports in place, you can proceed to define a shell task. +To create a shell task, provide a name for it, specify the bash script to be executed, +and define inputs and outputs if needed. + +```{code-cell} +t1 = ShellTask( + name="task_1", + debug=True, + script=""" + set -ex + echo "Hey there! Let's run some bash scripts using Flyte's ShellTask." + echo "Showcasing Flyte's Shell Task." >> {inputs.x} + if grep "Flyte" {inputs.x} + then + echo "Found it!" >> {inputs.x} + else + echo "Not found!" + fi + """, + inputs=kwtypes(x=FlyteFile), + output_locs=[OutputLocation(var="i", var_type=FlyteFile, location="{inputs.x}")], +) + + +t2 = ShellTask( + name="task_2", + debug=True, + script=""" + set -ex + cp {inputs.x} {inputs.y} + tar -zcvf {outputs.j} {inputs.y} + """, + inputs=kwtypes(x=FlyteFile, y=FlyteDirectory), + output_locs=[OutputLocation(var="j", var_type=FlyteFile, location="{inputs.y}.tar.gz")], +) + + +t3 = ShellTask( + name="task_3", + debug=True, + script=""" + set -ex + tar -zxvf {inputs.z} + cat {inputs.y}/$(basename {inputs.x}) | wc -m > {outputs.k} + """, + inputs=kwtypes(x=FlyteFile, y=FlyteDirectory, z=FlyteFile), + output_locs=[OutputLocation(var="k", var_type=FlyteFile, location="output.txt")], +) +``` + ++++ {"lines_to_next_cell": 0} + +Here's a breakdown of the parameters of the `ShellTask`: + +- The `inputs` parameter allows you to specify the types of inputs that the task will accept +- The `output_locs` parameter is used to define the output locations, which can be `FlyteFile` or `FlyteDirectory` +- The `script` parameter contains the actual bash script that will be executed + (`{inputs.x}`, `{outputs.j}`, etc. will be replaced with the actual input and output values). +- The `debug` parameter is helpful for debugging purposes + +We define a task to instantiate `FlyteFile` and `FlyteDirectory`. +A `.gitkeep` file is created in the FlyteDirectory as a placeholder to ensure the directory exists. + +```{code-cell} +@task +def create_entities() -> Tuple[FlyteFile, FlyteDirectory]: + working_dir = Path(flytekit.current_context().working_directory) + flytefile = working_dir / "test.txt" + flytefile.touch() + + flytedir = working_dir / "testdata" + flytedir.mkdir(exist_ok=True) + + flytedir_file = flytedir / ".gitkeep" + flytedir_file.touch() + return flytefile, flytedir +``` + ++++ {"lines_to_next_cell": 0} + +We create a workflow to define the dependencies between the tasks. + +```{code-cell} +@workflow +def shell_task_wf() -> FlyteFile: + x, y = create_entities() + t1_out = t1(x=x) + t2_out = t2(x=t1_out, y=y) + t3_out = t3(x=x, y=y, z=t2_out) + return t3_out +``` + ++++ {"lines_to_next_cell": 0} + +You can run the workflow locally. + +```{code-cell} +if __name__ == "__main__": + print(f"Running shell_task_wf() {shell_task_wf()}") +``` diff --git a/docs/user_guide/basics/tasks.md b/docs/user_guide/basics/tasks.md new file mode 100644 index 0000000000..3f9fcb493d --- /dev/null +++ b/docs/user_guide/basics/tasks.md @@ -0,0 +1,108 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(task)= + +# Tasks + +```{eval-rst} +.. tags:: Basic +``` + +A task serves as the fundamental building block and an extension point within Flyte. +It exhibits the following characteristics: + +1. Versioned (typically aligned with the git sha) +2. Strong interfaces (annotated inputs and outputs) +3. Declarative +4. Independently executable +5. Suitable for unit testing + +A Flyte task operates within its own container and runs on a [Kubernetes pod](https://kubernetes.io/docs/concepts/workloads/pods/). +It can be classified into two types: + +1. A task associated with a Python function. Executing the task is the same as executing the function. +2. A task without a Python function, such as a SQL query or a portable task like prebuilt + algorithms in SageMaker, or a service calling an API. + +Flyte offers numerous plugins for tasks, including backend plugins like +[Athena](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-aws-athena/flytekitplugins/athena/task.py). + +This example demonstrates how to write and execute a +[Python function task](https://github.com/flyteorg/flytekit/blob/master/flytekit/core/python_function_task.py#L75). + +To begin, import `task` from the `flytekit` library. + +```{code-cell} +from flytekit import task +``` + ++++ {"lines_to_next_cell": 0} + +The use of the {py:func}`~flytekit.task` decorator is mandatory for a ``PythonFunctionTask``. +A task is essentially a regular Python function, with the exception that all inputs and outputs must be clearly annotated with their types. +Learn more about the supported types in the {ref}`type-system section `. + +We create a task that computes the slope of a regression line. + +```{code-cell} +@task +def slope(x: list[int], y: list[int]) -> float: + sum_xy = sum([x[i] * y[i] for i in range(len(x))]) + sum_x_squared = sum([x[i] ** 2 for i in range(len(x))]) + n = len(x) + return (n * sum_xy - sum(x) * sum(y)) / (n * sum_x_squared - sum(x) ** 2) +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +Flytekit will assign a default name to the output variable like `out0`. +In case of multiple outputs, each output will be numbered in the order +starting with 0, e.g., -> `out0, out1, out2, ...`. +::: + +You can execute a Flyte task just like any regular Python function. + +```{code-cell} +if __name__ == "__main__": + print(slope(x=[-3, 0, 3], y=[7, 4, -2])) +``` + +:::{note} +When invoking a Flyte task, you need to use keyword arguments to specify +the values for the corresponding parameters. +::: + +(single_task_execution)= + +To run it locally, you can use the following `pyflyte run` command: +``` +pyflyte run \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/task.py \ + slope --x '[-3,0,3]' --y '[7,4,-2]' +``` + +If you want to run it remotely on the Flyte cluster, +simply add the `--remote flag` to the `pyflyte run` command: +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/task.py \ + slope --x '[-3,0,3]' --y '[7,4,-2]' +``` diff --git a/docs/user_guide/basics/workflows.md b/docs/user_guide/basics/workflows.md new file mode 100644 index 0000000000..1f750c9da8 --- /dev/null +++ b/docs/user_guide/basics/workflows.md @@ -0,0 +1,151 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(workflow)= + +# Workflows + +```{eval-rst} +.. tags:: Basic +``` + +Workflows link multiple tasks together. They can be written as Python functions, +but it's important to distinguish tasks and workflows. + +A task's body executes at run-time on a Kubernetes cluster, in a Query Engine like BigQuery, +or on hosted services like AWS Batch or Sagemaker. + +In contrast, a workflow's body doesn't perform computations; it's used to structure tasks. +A workflow's body executes at registration time, during the workflow's registration process. +Registration involves uploading the packaged (serialized) code to the Flyte backend, +enabling the workflow to be triggered. + +For more information, see the {std:ref}`registration documentation `. + +To begin, import {py:func}`~flytekit.task` and {py:func}`~flytekit.workflow` from the flytekit library. + +```{code-cell} +from flytekit import task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +We define `slope` and `intercept` tasks to compute the slope and +intercept of the regression line, respectively. + +```{code-cell} +@task +def slope(x: list[int], y: list[int]) -> float: + sum_xy = sum([x[i] * y[i] for i in range(len(x))]) + sum_x_squared = sum([x[i] ** 2 for i in range(len(x))]) + n = len(x) + return (n * sum_xy - sum(x) * sum(y)) / (n * sum_x_squared - sum(x) ** 2) + + +@task +def intercept(x: list[int], y: list[int], slope: float) -> float: + mean_x = sum(x) / len(x) + mean_y = sum(y) / len(y) + intercept = mean_y - slope * mean_x + return intercept +``` + ++++ {"lines_to_next_cell": 0} + +Define a workflow to establish the task dependencies. +Just like a task, a workflow is also strongly typed. + +```{code-cell} +@workflow +def simple_wf(x: list[int], y: list[int]) -> float: + slope_value = slope(x=x, y=y) + intercept_value = intercept(x=x, y=y, slope=slope_value) + return intercept_value +``` + ++++ {"lines_to_next_cell": 0} + +The {py:func}`~flytekit.workflow` decorator encapsulates Flyte tasks, +essentially representing lazily evaluated promises. +During parsing, function calls are deferred until execution time. +These function calls generate {py:class}`~flytekit.extend.Promise`s that can be propagated to downstream functions, +yet remain inaccessible within the workflow itself. +The actual evaluation occurs when the workflow is executed. + +Workflows can be executed locally, resulting in immediate evaluation, or through tools like +[`pyflyte`](https://docs.flyte.org/projects/flytekit/en/latest/pyflyte.html), +[`flytectl`](https://docs.flyte.org/projects/flytectl/en/latest/index.html) or UI, triggering evaluation. +While workflows decorated with `@workflow` resemble Python functions, +they function as python-esque Domain Specific Language (DSL). +When encountering a @task-decorated Python function, a promise object is created. +This promise doesn't store the task's actual output. Its fulfillment only occurs during execution. +Additionally, the inputs to a workflow are also promises, you can only pass promises into +tasks, workflows and other Flyte constructs. + +:::{note} +You can learn more about creating dynamic Flyte workflows by referring +to {ref}`dynamic workflows `. +In a dynamic workflow, unlike a simple workflow, the inputs are pre-materialized. +However, each task invocation within the dynamic workflow still generates a promise that is evaluated lazily. +Bear in mind that a workflow can have tasks, other workflows and dynamic workflows. +::: + +You can run a workflow by calling it as you would with a Python function and providing the necessary inputs. + +```{code-cell} +if __name__ == "__main__": + print(f"Running simple_wf() {simple_wf(x=[-3, 0, 3], y=[7, 4, -2])}") +``` + ++++ {"lines_to_next_cell": 0} + +To run the workflow locally, you can use the following `pyflyte run` command: +``` +pyflyte run \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py \ + simple_wf --x '[-3,0,3]' --y '[7,4,-2]' +``` + +If you want to run it remotely on the Flyte cluster, +simply add the `--remote flag` to the `pyflyte run` command: +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py \ + simple_wf --x '[-3,0,3]' --y '[7,4,-2]' +``` + +While workflows are usually constructed from multiple tasks with dependencies established through +shared inputs and outputs, there are scenarios where isolating the execution of a single task +proves advantageous during the development and iteration of its logic. +Crafting a new workflow definition each time for this purpose can be cumbersome. +However, {ref}`executing an individual task ` independently, +without the confines of a workflow, offers a convenient approach for iterating on task logic effortlessly. + +## Use `partial` to provide default arguments to tasks +You can use the {py:func}`functools.partial` function to assign default or constant values to the parameters of your tasks. + +```{code-cell} +import functools + + +@workflow +def simple_wf_with_partial(x: list[int], y: list[int]) -> float: + partial_task = functools.partial(slope, x=x) + return partial_task(y=y) +``` diff --git a/docs/user_guide/customizing_dependencies/imagespec.md b/docs/user_guide/customizing_dependencies/imagespec.md new file mode 100644 index 0000000000..5a9ef93736 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/imagespec.md @@ -0,0 +1,162 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(image_spec_example)= + +# ImageSpec + +```{eval-rst} +.. tags:: Containerization, Intermediate +``` + +:::{note} +This is an experimental feature, which is subject to change the API in the future. +::: + +`ImageSpec` is a way to specify how to build a container image without a Dockerfile. The `ImageSpec` by default will be +converted to an [Envd](https://envd.tensorchord.ai/) config, and the [Envd builder](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-envd/flytekitplugins/envd/image_builder.py#L12-L34) will build the image for you. However, you can also register your own builder to build +the image using other tools. + +For every {py:class}`flytekit.PythonFunctionTask` task or a task decorated with the `@task` decorator, +you can specify rules for binding container images. By default, flytekit binds a single container image, i.e., +the [default Docker image](https://ghcr.io/flyteorg/flytekit), to all tasks. To modify this behavior, +use the `container_image` parameter available in the {py:func}`flytekit.task` decorator, and pass an +`ImageSpec`. + +Before building the image, Flytekit checks the container registry first to see if the image already exists. By doing +so, it avoids having to rebuild the image over and over again. If the image does not exist, flytekit will build the +image before registering the workflow, and replace the image name in the task template with the newly built image name. + +```{code-cell} +import typing + +import pandas as pd +from flytekit import ImageSpec, Resources, task, workflow +``` + +:::{admonition} Prerequisites +:class: important + +- Install [flytekitplugins-envd](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-envd) to build the `ImageSpec`. +- To build the image on remote machine, check this [doc](https://envd.tensorchord.ai/teams/context.html#start-remote-buildkitd-on-builder-machine). +- When using a registry in ImageSpec, `docker login` is required to push the image +::: + ++++ {"lines_to_next_cell": 0} + +You can specify python packages, apt packages, and environment variables in the `ImageSpec`. +These specified packages will be added on top of the [default image](https://github.com/flyteorg/flytekit/blob/master/Dockerfile), which can be found in the Flytekit Dockerfile. +More specifically, flytekit invokes [DefaultImages.default_image()](https://github.com/flyteorg/flytekit/blob/f2cfef0ec098d4ae8f042ab915b0b30d524092c6/flytekit/configuration/default_images.py#L26-L27) function. +This function determines and returns the default image based on the Python version and flytekit version. For example, if you are using python 3.8 and flytekit 0.16.0, the default image assigned will be `ghcr.io/flyteorg/flytekit:py3.8-1.6.0`. +If desired, you can also override the default image by providing a custom `base_image` parameter when using the `ImageSpec`. + +```{code-cell} +pandas_image_spec = ImageSpec( + base_image="ghcr.io/flyteorg/flytekit:py3.8-1.6.2", + packages=["pandas", "numpy"], + python_version="3.9", + apt_packages=["git"], + env={"Debug": "True"}, + registry="ghcr.io/flyteorg", +) + +sklearn_image_spec = ImageSpec( + base_image="ghcr.io/flyteorg/flytekit:py3.8-1.6.2", + packages=["scikit-learn"], + registry="ghcr.io/flyteorg", +) +``` + ++++ {"lines_to_next_cell": 0} + +:::{important} +Replace `ghcr.io/flyteorg` with a container registry you've access to publish to. +To upload the image to the local registry in the demo cluster, indicate the registry as `localhost:30000`. +::: + +`is_container` is used to determine whether the task is utilizing the image constructed from the `ImageSpec`. +If the task is indeed using the image built from the `ImageSpec`, it will then import Tensorflow. +This approach helps minimize module loading time and prevents unnecessary dependency installation within a single image. + +```{code-cell} +if sklearn_image_spec.is_container(): + from sklearn.linear_model import LogisticRegression +``` + ++++ {"lines_to_next_cell": 0} + +To enable tasks to utilize the images built with `ImageSpec`, you can specify the `container_image` parameter for those tasks. + +```{code-cell} +@task(container_image=pandas_image_spec) +def get_pandas_dataframe() -> typing.Tuple[pd.DataFrame, pd.Series]: + df = pd.read_csv("https://storage.googleapis.com/download.tensorflow.org/data/heart.csv") + print(df.head()) + return df[["age", "thalach", "trestbps", "chol", "oldpeak"]], df.pop("target") + + +@task(container_image=sklearn_image_spec, requests=Resources(cpu="1", mem="1Gi")) +def get_model(max_iter: int, multi_class: str) -> typing.Any: + return LogisticRegression(max_iter=max_iter, multi_class=multi_class) + + +# Get a basic model to train. +@task(container_image=sklearn_image_spec, requests=Resources(cpu="1", mem="1Gi")) +def train_model(model: typing.Any, feature: pd.DataFrame, target: pd.Series) -> typing.Any: + model.fit(feature, target) + return model + + +# Lastly, let's define a workflow to capture the dependencies between the tasks. +@workflow() +def wf(): + feature, target = get_pandas_dataframe() + model = get_model(max_iter=3000, multi_class="auto") + train_model(model=model, feature=feature, target=target) + + +if __name__ == "__main__": + wf() +``` + +There exists an option to override the container image by providing an Image Spec YAML file to the `pyflyte run` or `pyflyte register` command. +This allows for greater flexibility in specifying a custom container image. For example: + +```yaml +# imageSpec.yaml +python_version: 3.11 +registry: pingsutw +packages: + - sklearn +env: + Debug: "True" +``` + +``` +# Use pyflyte to register the workflow +pyflyte run --remote --image image.yaml image_spec.py wf +``` + ++++ + +If you only want to build the image without registering the workflow, you can use the `pyflyte build` command. + +``` +pyflyte build --remote image_spec.py wf +``` diff --git a/docs/user_guide/customizing_dependencies/index.md b/docs/user_guide/customizing_dependencies/index.md new file mode 100644 index 0000000000..0c5262dd67 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/index.md @@ -0,0 +1,17 @@ +# Customizing dependencies + +In this section, you will uncover how Flyte utilizes Docker images to construct containers under the hood, +and you'll learn how to craft your own images to encompass all the necessary dependencies for your tasks or workflows. +You will explore how to execute a raw container with custom commands, +indicate multiple container images within a single workflow, +and get familiar with the ins and outs of `ImageSpec`! + +```{toctree} +:maxdepth: -1 +:name: customizing_dependencies_toc +:hidden: + +imagespec +raw_containers +multiple_images_in_a_workflow +``` diff --git a/docs/user_guide/customizing_dependencies/multiple_images_in_a_workflow.md b/docs/user_guide/customizing_dependencies/multiple_images_in_a_workflow.md new file mode 100644 index 0000000000..0c323cada9 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/multiple_images_in_a_workflow.md @@ -0,0 +1,110 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(multi_images)= + +# Multiple images in a workflow + +```{eval-rst} +.. tags:: Containerization, Intermediate +``` + +For every {py:class}`flytekit.PythonFunctionTask` task or a task decorated with the `@task` decorator, you can specify rules for binding container images. +By default, flytekit binds a single container image, i.e., the [default Docker image](https://ghcr.io/flyteorg/flytekit), to all tasks. +To modify this behavior, use the `container_image` parameter available in the {py:func}`flytekit.task` decorator. + +:::{note} +If the Docker image is not available publicly, refer to {ref}`Pulling Private Images`. +::: + +```{code-cell} +:lines_to_next_cell: 2 + +import numpy as np +from flytekit import task, workflow + + +@task(container_image="{{.image.mindmeld.fqn}}:{{.image.mindmeld.version}}") +def get_data() -> np.ndarray: + # here we're importing scikit learn within the Flyte task + from sklearn import datasets + + iris = datasets.load_iris() + X = iris.data[:, :2] + return X + + +@task(container_image="{{.image.borebuster.fqn}}:{{.image.borebuster.version}}") +def normalize(X: np.ndarray) -> np.ndarray: + return (X - X.mean(axis=0)) / X.std(axis=0) + + +@workflow +def multi_images_wf() -> np.ndarray: + X = get_data() + X = normalize(X=X) + return X +``` + +Observe how the `sklearn` library is imported in the context of a Flyte task. +This approach is beneficial when creating tasks in a single module, where some tasks have dependencies that others do not require. + +## Configuring image parameters + +The following parameters can be used to configure images in the `@task` decorator: + +1. `image` refers to the name of the image in the image configuration. The name `default` is a reserved keyword and will automatically apply to the default image name for this repository. +2. `fqn` refers to the fully qualified name of the image. For example, it includes the repository and domain URL of the image. Example: docker.io/my_repo/xyz. +3. `version` refers to the tag of the image. For example: latest, or python-3.9 etc. If `container_image` is not specified, then the default configured image for the project is used. + +## Sending images to `pyflyte` command + +You can pass Docker images to the `pyflyte run` or `pyflyte register` command. +For instance: + +``` +pyflyte run --remote --image mindmeld="ghcr.io/flyteorg/flytecookbook:core-latest" --image borebuster="ghcr.io/flyteorg/flytekit:py3.9-latest" multi_images.py multi_images_wf +``` + +## Configuring images in `$HOME/.flyte/config.yaml` + +To specify images in your `$HOME/.flyte/config.yaml` file (or whichever configuration file you are using), include an "images" section in the configuration. +For example: + +```{code-block} yaml +:emphasize-lines: 6-8 + +admin: + # For GRPC endpoints you might want to use dns:///flyte.myexample.com + endpoint: localhost:30080 + authType: Pkce + insecure: true +images: + mindmeld: ghcr.io/flyteorg/flytecookbook:core-latest + borebuster: ghcr.io/flyteorg/flytekit:py3.9-latest +console: + endpoint: http://localhost:30080 +logger: + show-source: true + level: 0 +``` + +Send the name of the configuration file to your `pyflyte run` command as follows: + +``` +pyflyte --config $HOME/.flyte/config.yaml run --remote multi_images.py multi_images_wf +``` diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/haskell/Dockerfile b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/haskell/Dockerfile new file mode 100644 index 0000000000..8d9679088d --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/haskell/Dockerfile @@ -0,0 +1,7 @@ +FROM haskell:9 + +WORKDIR /root + +COPY calculate-ellipse-area.hs /root + +RUN ghc calculate-ellipse-area.hs diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/haskell/calculate-ellipse-area.hs b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/haskell/calculate-ellipse-area.hs new file mode 100644 index 0000000000..c31d4a10ad --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/haskell/calculate-ellipse-area.hs @@ -0,0 +1,19 @@ +import System.IO +import System.Environment +import Text.Read +import Text.Printf + +calculateEllipseArea :: Float -> Float -> Float +calculateEllipseArea a b = pi * a * b + +main = do + args <- getArgs + let a = args!!0 + b = args!!1 + + let area = calculateEllipseArea (read a::Float) (read b::Float) + + let output_area = args!!2 ++ "/area" + output_metadata = args!!2 ++ "/metadata" + writeFile output_area (show area) + writeFile output_metadata "[from haskell rawcontainer]" diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/julia/Dockerfile b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/julia/Dockerfile new file mode 100644 index 0000000000..caaf85a2ab --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/julia/Dockerfile @@ -0,0 +1,5 @@ +FROM julia:1.6.4-buster + +WORKDIR /root + +COPY calculate-ellipse-area.jl /root diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/julia/calculate-ellipse-area.jl b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/julia/calculate-ellipse-area.jl new file mode 100644 index 0000000000..c26ffdfea7 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/julia/calculate-ellipse-area.jl @@ -0,0 +1,31 @@ + +using Printf + +function calculate_area(a, b) + π * a * b +end + +function write_output(output_dir, output_file, v) + output_path = @sprintf "%s/%s" output_dir output_file + open(output_path, "w") do file + write(file, string(v)) + end +end + +function main(a, b, output_dir) + a = parse.(Float64, a) + b = parse.(Float64, b) + + area = calculate_area(a, b) + + write_output(output_dir, "area", area) + write_output(output_dir, "metadata", "[from julia rawcontainer]") +end + +# the keyword ARGS is a special value that contains the command-line arguments +# julia arrays are 1-indexed +a = ARGS[1] +b = ARGS[2] +output_dir = ARGS[3] + +main(a, b, output_dir) diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/python/Dockerfile b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/python/Dockerfile new file mode 100644 index 0000000000..2174b2d997 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/python/Dockerfile @@ -0,0 +1,5 @@ +FROM python:3.10-slim-buster + +WORKDIR /root + +COPY *.py /root/ diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/python/calculate-ellipse-area.py b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/python/calculate-ellipse-area.py new file mode 100644 index 0000000000..7c589da7c8 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/python/calculate-ellipse-area.py @@ -0,0 +1,29 @@ +import math +import sys + + +def write_output(output_dir, output_file, v): + with open(f"{output_dir}/{output_file}", "w") as f: + f.write(str(v)) + + +def calculate_area(a, b): + return math.pi * a * b + + +def main(a, b, output_dir): + a = float(a) + b = float(b) + + area = calculate_area(a, b) + + write_output(output_dir, "area", area) + write_output(output_dir, "metadata", "[from python rawcontainer]") + + +if __name__ == "__main__": + a = sys.argv[1] + b = sys.argv[2] + output_dir = sys.argv[3] + + main(a, b, output_dir) diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/Dockerfile b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/Dockerfile new file mode 100644 index 0000000000..b1dad09c08 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/Dockerfile @@ -0,0 +1,5 @@ +FROM r-base + +WORKDIR /root + +COPY *.R /root/ diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/calculate-ellipse-area.R b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/calculate-ellipse-area.R new file mode 100644 index 0000000000..d2650d826b --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/calculate-ellipse-area.R @@ -0,0 +1,13 @@ +#!/usr/bin/env Rscript + +args = commandArgs(trailingOnly=TRUE) + +a = args[1] +b = args[2] +output_dir = args[3] + +area <- pi * as.double(a) * as.double(b) +print(area) + +writeLines(as.character(area), sprintf("%s/%s", output_dir, 'area')) +writeLines("[from R rawcontainer]", sprintf("%s/%s", output_dir, 'metadata')) diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/install-readr.R b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/install-readr.R new file mode 100644 index 0000000000..3308314e4e --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/r/install-readr.R @@ -0,0 +1 @@ +install.packages("readr") diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/shell/Dockerfile b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/shell/Dockerfile new file mode 100644 index 0000000000..856160ba11 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/shell/Dockerfile @@ -0,0 +1,6 @@ +FROM alpine + +WORKDIR /root + +COPY calculate-ellipse-area.sh /root +RUN chmod +x /root/calculate-ellipse-area.sh diff --git a/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/shell/calculate-ellipse-area.sh b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/shell/calculate-ellipse-area.sh new file mode 100755 index 0000000000..5096e14035 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw-containers-supporting-files/per-language/shell/calculate-ellipse-area.sh @@ -0,0 +1,5 @@ +#! /usr/bin/env sh + +echo "4*a(1) * $1 * $2" | bc -l | tee "$3/area" + +echo "[from shell rawcontainer]" | tee "$3/metadata" diff --git a/docs/user_guide/customizing_dependencies/raw_containers.md b/docs/user_guide/customizing_dependencies/raw_containers.md new file mode 100644 index 0000000000..2ba6cfec55 --- /dev/null +++ b/docs/user_guide/customizing_dependencies/raw_containers.md @@ -0,0 +1,227 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(raw_container)= + +# Raw containers + +```{eval-rst} +.. tags:: Containerization, Advanced +``` + +This example demonstrates how to use arbitrary containers in 5 different languages, all orchestrated in flytekit seamlessly. +Flyte mounts an input data volume where all the data needed by the container is available, and an output data volume +for the container to write all the data which will be stored away. + +The data is written as separate files, one per input variable. The format of the file is serialized strings. +Refer to the raw protocol to understand how to leverage this. + +```{code-cell} +import logging + +from flytekit import ContainerTask, kwtypes, task, workflow + +logger = logging.getLogger(__file__) +``` + ++++ {"lines_to_next_cell": 0} + +## Container tasks + +A {py:class}`flytekit.ContainerTask` denotes an arbitrary container. In the following example, the name of the task +is `calculate_ellipse_area_shell`. This name has to be unique in the entire project. Users can specify: + +- `input_data_dir` -> where inputs will be written to. +- `output_data_dir` -> where Flyte will expect the outputs to exist. + +`inputs` and `outputs` specify the interface for the task; thus it should be an ordered dictionary of typed input and +output variables. + +```{code-cell} +calculate_ellipse_area_shell = ContainerTask( + name="ellipse-area-metadata-shell", + input_data_dir="/var/inputs", + output_data_dir="/var/outputs", + inputs=kwtypes(a=float, b=float), + outputs=kwtypes(area=float, metadata=str), + image="ghcr.io/flyteorg/rawcontainers-shell:v2", + command=[ + "./calculate-ellipse-area.sh", + "{{.inputs.a}}", + "{{.inputs.b}}", + "/var/outputs", + ], +) + +calculate_ellipse_area_python = ContainerTask( + name="ellipse-area-metadata-python", + input_data_dir="/var/inputs", + output_data_dir="/var/outputs", + inputs=kwtypes(a=float, b=float), + outputs=kwtypes(area=float, metadata=str), + image="ghcr.io/flyteorg/rawcontainers-python:v2", + command=[ + "python", + "calculate-ellipse-area.py", + "{{.inputs.a}}", + "{{.inputs.b}}", + "/var/outputs", + ], +) + +calculate_ellipse_area_r = ContainerTask( + name="ellipse-area-metadata-r", + input_data_dir="/var/inputs", + output_data_dir="/var/outputs", + inputs=kwtypes(a=float, b=float), + outputs=kwtypes(area=float, metadata=str), + image="ghcr.io/flyteorg/rawcontainers-r:v2", + command=[ + "Rscript", + "--vanilla", + "calculate-ellipse-area.R", + "{{.inputs.a}}", + "{{.inputs.b}}", + "/var/outputs", + ], +) + +calculate_ellipse_area_haskell = ContainerTask( + name="ellipse-area-metadata-haskell", + input_data_dir="/var/inputs", + output_data_dir="/var/outputs", + inputs=kwtypes(a=float, b=float), + outputs=kwtypes(area=float, metadata=str), + image="ghcr.io/flyteorg/rawcontainers-haskell:v2", + command=[ + "./calculate-ellipse-area", + "{{.inputs.a}}", + "{{.inputs.b}}", + "/var/outputs", + ], +) + +calculate_ellipse_area_julia = ContainerTask( + name="ellipse-area-metadata-julia", + input_data_dir="/var/inputs", + output_data_dir="/var/outputs", + inputs=kwtypes(a=float, b=float), + outputs=kwtypes(area=float, metadata=str), + image="ghcr.io/flyteorg/rawcontainers-julia:v2", + command=[ + "julia", + "calculate-ellipse-area.jl", + "{{.inputs.a}}", + "{{.inputs.b}}", + "/var/outputs", + ], +) + + +@task +def report_all_calculated_areas( + area_shell: float, + metadata_shell: str, + area_python: float, + metadata_python: str, + area_r: float, + metadata_r: str, + area_haskell: float, + metadata_haskell: str, + area_julia: float, + metadata_julia: str, +): + logger.info(f"shell: area={area_shell}, metadata={metadata_shell}") + logger.info(f"python: area={area_python}, metadata={metadata_python}") + logger.info(f"r: area={area_r}, metadata={metadata_r}") + logger.info(f"haskell: area={area_haskell}, metadata={metadata_haskell}") + logger.info(f"julia: area={area_julia}, metadata={metadata_julia}") +``` + ++++ {"lines_to_next_cell": 0} + +As can be seen in this example, `ContainerTask`s can be interacted with like normal Python functions, whose inputs +correspond to the declared input variables. All data returned by the tasks are consumed and logged by a Flyte task. + +```{code-cell} +:lines_to_next_cell: 2 + +@workflow +def wf(a: float, b: float): + # Calculate area in all languages + area_shell, metadata_shell = calculate_ellipse_area_shell(a=a, b=b) + area_python, metadata_python = calculate_ellipse_area_python(a=a, b=b) + area_r, metadata_r = calculate_ellipse_area_r(a=a, b=b) + area_haskell, metadata_haskell = calculate_ellipse_area_haskell(a=a, b=b) + area_julia, metadata_julia = calculate_ellipse_area_julia(a=a, b=b) + + # Report on all results in a single task to simplify comparison + report_all_calculated_areas( + area_shell=area_shell, + metadata_shell=metadata_shell, + area_python=area_python, + metadata_python=metadata_python, + area_r=area_r, + metadata_r=metadata_r, + area_haskell=area_haskell, + metadata_haskell=metadata_haskell, + area_julia=area_julia, + metadata_julia=metadata_julia, + ) +``` + +One of the benefits of raw container tasks is that Flytekit does not need to be installed in the target container. + +:::{note} +Raw containers cannot be run locally at the moment. +::: + +## Scripts + +The contents of each script specified in the `ContainerTask` is as follows: + +### calculate-ellipse-area.sh + +```{literalinclude} raw-containers-supporting-files/per-language/shell/calculate-ellipse-area.sh +:language: shell +``` + +### calculate-ellipse-area.py + +```{literalinclude} raw-containers-supporting-files/per-language/python/calculate-ellipse-area.py +:language: python +``` + +### calculate-ellipse-area.R + +```{literalinclude} raw-containers-supporting-files/per-language/r/calculate-ellipse-area.R +:language: r +``` + +### calculate-ellipse-area.hs + +```{literalinclude} raw-containers-supporting-files/per-language/haskell/calculate-ellipse-area.hs +:language: haskell +``` + +### calculate-ellipse-area.jl + +```{literalinclude} raw-containers-supporting-files/per-language/julia/calculate-ellipse-area.jl +:language: julia +``` diff --git a/docs/user_guide/data_types_and_io/accessing_attributes.md b/docs/user_guide/data_types_and_io/accessing_attributes.md new file mode 100644 index 0000000000..42706a3d1d --- /dev/null +++ b/docs/user_guide/data_types_and_io/accessing_attributes.md @@ -0,0 +1,176 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(attribute_access)= + +# Accessing attributes + +```{eval-rst} +.. tags:: Basic +``` + +You can directly access attributes on output promises for lists, dicts, dataclasses and combinations of these types in Flyte. +This functionality facilitates the direct passing of output attributes within workflows, +enhancing the convenience of working with complex data structures. + +To begin, import the required dependencies and define a common task for subsequent use. + +```{code-cell} +from dataclasses import dataclass + +from dataclasses_json import dataclass_json +from flytekit import task, workflow + + +@task +def print_message(message: str): + print(message) + return +``` + ++++ {"lines_to_next_cell": 0} + +## List +You can access an output list using index notation. + +:::{important} +Flyte currently does not support output promise access through list slicing. +::: + +```{code-cell} +@task +def list_task() -> list[str]: + return ["apple", "banana"] + + +@workflow +def list_wf(): + items = list_task() + first_item = items[0] + print_message(message=first_item) +``` + ++++ {"lines_to_next_cell": 0} + +## Dictionary +Access the output dictionary by specifying the key. + +```{code-cell} +@task +def dict_task() -> dict[str, str]: + return {"fruit": "banana"} + + +@workflow +def dict_wf(): + fruit_dict = dict_task() + print_message(message=fruit_dict["fruit"]) +``` + ++++ {"lines_to_next_cell": 0} + +## Data class +Directly access an attribute of a dataclass. + +```{code-cell} +@dataclass_json +@dataclass +class Fruit: + name: str + + +@task +def dataclass_task() -> Fruit: + return Fruit(name="banana") + + +@workflow +def dataclass_wf(): + fruit_instance = dataclass_task() + print_message(message=fruit_instance.name) +``` + ++++ {"lines_to_next_cell": 0} + +## Complex type +Combinations of list, dict and dataclass also work effectively. + +```{code-cell} +@task +def advance_task() -> (dict[str, list[str]], list[dict[str, str]], dict[str, Fruit]): + return {"fruits": ["banana"]}, [{"fruit": "banana"}], {"fruit": Fruit(name="banana")} + + +@task +def print_list(fruits: list[str]): + print(fruits) + + +@task +def print_dict(fruit_dict: dict[str, str]): + print(fruit_dict) + + +@workflow +def advanced_workflow(): + dictionary_list, list_dict, dict_dataclass = advance_task() + print_message(message=dictionary_list["fruits"][0]) + print_message(message=list_dict[0]["fruit"]) + print_message(message=dict_dataclass["fruit"].name) + + print_list(fruits=dictionary_list["fruits"]) + print_dict(fruit_dict=list_dict[0]) +``` + ++++ {"lines_to_next_cell": 0} + +You can run all the workflows locally as follows: + +```{code-cell} +:lines_to_next_cell: 2 + +if __name__ == "__main__": + list_wf() + dict_wf() + dataclass_wf() + advanced_workflow() +``` + +## Failure scenario +The following workflow fails because it attempts to access indices and keys that are out of range: + +```python +from flytekit import WorkflowFailurePolicy + + +@task +def failed_task() -> (list[str], dict[str, str], Fruit): + return ["apple", "banana"], {"fruit": "banana"}, Fruit(name="banana") + + +@workflow( + # The workflow remains unaffected if one of the nodes encounters an error, as long as other executable nodes are still available + failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE +) +def failed_workflow(): + fruits_list, fruit_dict, fruit_instance = failed_task() + print_message(message=fruits_list[100]) # Accessing an index that doesn't exist + print_message(message=fruit_dict["fruits"]) # Accessing a non-existent key + print_message(message=fruit_instance.fruit) # Accessing a non-existent param +``` diff --git a/docs/user_guide/data_types_and_io/dataclass.md b/docs/user_guide/data_types_and_io/dataclass.md new file mode 100644 index 0000000000..fdb9f1d992 --- /dev/null +++ b/docs/user_guide/data_types_and_io/dataclass.md @@ -0,0 +1,172 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(dataclass)= + +# Dataclass + +```{eval-rst} +.. tags:: Basic +``` + +When you've multiple values that you want to send across Flyte entities, you can use a `dataclass`. + +Flytekit uses the [Mashumaro library](https://github.com/Fatal1ty/mashumaro) +to serialize and deserialize dataclasses. + +:::{important} +If you're using Flytekit version below v1.10, you'll need to decorate with `@dataclass_json` using +`from dataclass_json import dataclass_json` instead of inheriting from Mashumaro's `DataClassJSONMixin`. +::: + +To begin, import the necessary dependencies. + +```{code-cell} +import os +import tempfile +from dataclasses import dataclass + +import pandas as pd +from flytekit import task, workflow +from flytekit.types.directory import FlyteDirectory +from flytekit.types.file import FlyteFile +from flytekit.types.structured import StructuredDataset +from mashumaro.mixins.json import DataClassJSONMixin +``` + ++++ {"lines_to_next_cell": 0} + +## Python types +We define a `dataclass` with `int`, `str` and `dict` as the data types. + +```{code-cell} +@dataclass +class Datum(DataClassJSONMixin): + x: int + y: str + z: dict[int, str] +``` + ++++ {"lines_to_next_cell": 0} + +You can send a `dataclass` between different tasks written in various languages, and input it through the Flyte console as raw JSON. + +:::{note} +All variables in a data class should be **annotated with their type**. Failure to do should will result in an error. +::: + +Once declared, a dataclass can be returned as an output or accepted as an input. + +```{code-cell} +@task +def stringify(s: int) -> Datum: + """ + A dataclass return will be treated as a single complex JSON return. + """ + return Datum(x=s, y=str(s), z={s: str(s)}) + + +@task +def add(x: Datum, y: Datum) -> Datum: + """ + Flytekit automatically converts the provided JSON into a data class. + If the structures don't match, it triggers a runtime failure. + """ + x.z.update(y.z) + return Datum(x=x.x + y.x, y=x.y + y.y, z=x.z) +``` + ++++ {"lines_to_next_cell": 0} + +## Flyte types +We also define a data class that accepts {std:ref}`StructuredDataset `, +{std:ref}`FlyteFile ` and {std:ref}`FlyteDirectory `. + +```{code-cell} +@dataclass +class FlyteTypes(DataClassJSONMixin): + dataframe: StructuredDataset + file: FlyteFile + directory: FlyteDirectory + + +@task +def upload_data() -> FlyteTypes: + """ + Flytekit will upload FlyteFile, FlyteDirectory and StructuredDataset to the blob store, + such as GCP or S3. + """ + # 1. StructuredDataset + df = pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [20, 22]}) + + # 2. FlyteDirectory + temp_dir = tempfile.mkdtemp(prefix="flyte-") + df.to_parquet(temp_dir + "/df.parquet") + + # 3. FlyteFile + file_path = tempfile.NamedTemporaryFile(delete=False) + file_path.write(b"Hello, World!") + + fs = FlyteTypes( + dataframe=StructuredDataset(dataframe=df), + file=FlyteFile(file_path.name), + directory=FlyteDirectory(temp_dir), + ) + return fs + + +@task +def download_data(res: FlyteTypes): + assert pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [20, 22]}).equals(res.dataframe.open(pd.DataFrame).all()) + f = open(res.file, "r") + assert f.read() == "Hello, World!" + assert os.listdir(res.directory) == ["df.parquet"] +``` + ++++ {"lines_to_next_cell": 0} + +A data class supports the usage of data associated with Python types, data classes, +flyte file, flyte directory and structured dataset. + +We define a workflow that calls the tasks created above. + +```{code-cell} +@workflow +def dataclass_wf(x: int, y: int) -> (Datum, FlyteTypes): + o1 = add(x=stringify(s=x), y=stringify(s=y)) + o2 = upload_data() + download_data(res=o2) + return o1, o2 +``` + ++++ {"lines_to_next_cell": 0} + +You can run the workflow locally as follows: + +```{code-cell} +if __name__ == "__main__": + dataclass_wf(x=10, y=20) +``` + +To trigger a task that accepts a dataclass as an input with `pyflyte run`, you can provide a JSON file as an input: +``` +pyflyte run \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/dataclass.py \ + add --x dataclass_input.json --y dataclass_input.json +``` diff --git a/docs/user_guide/data_types_and_io/enum_type.md b/docs/user_guide/data_types_and_io/enum_type.md new file mode 100644 index 0000000000..b4727c508f --- /dev/null +++ b/docs/user_guide/data_types_and_io/enum_type.md @@ -0,0 +1,100 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +# Enum type + +```{eval-rst} +.. tags:: Basic +``` + +At times, you might need to limit the acceptable values for inputs or outputs to a predefined set. +This common requirement is usually met by using Enum types in programming languages. + +You can create a Python Enum type and utilize it as an input or output for a task. +Flytekit will automatically convert it and constrain the inputs and outputs to the predefined set of values. + +:::{important} +Currently, only string values are supported as valid enum values. +Flyte assumes the first value in the list as the default, and Enum types cannot be optional. +Therefore, when defining enums, it's important to design them with the first value as a valid default. +::: + +To begin, import the dependencies. + +```{code-cell} +from enum import Enum + +from flytekit import task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +We define an enum and a simple coffee maker workflow that accepts an order and brews coffee ☕️ accordingly. +The assumption is that the coffee maker only understands enum inputs. + +```{code-cell} +class Coffee(Enum): + ESPRESSO = "espresso" + AMERICANO = "americano" + LATTE = "latte" + CAPPUCCINO = "cappucccino" + + +@task +def take_order(coffee: str) -> Coffee: + return Coffee(coffee) + + +@task +def prep_order(coffee_enum: Coffee) -> str: + return f"Preparing {coffee_enum.value} ..." + + +@workflow +def coffee_maker(coffee: str) -> str: + coffee_enum = take_order(coffee=coffee) + return prep_order(coffee_enum=coffee_enum) +``` + ++++ {"lines_to_next_cell": 0} + +The workflow can also accept an enum value. + +```{code-cell} +@workflow +def coffee_maker_enum(coffee_enum: Coffee) -> str: + return prep_order(coffee_enum=coffee_enum) +``` + ++++ {"lines_to_next_cell": 0} + +You can send a string to the `coffee_maker_enum` workflow during its execution, like this: +``` +pyflyte run \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/data_types_and_io/data_types_and_io/enum_type.py \ + coffee_maker_enum --coffee_enum="latte" +``` + +You can run the workflows locally. + +```{code-cell} +if __name__ == "__main__": + print(coffee_maker(coffee="latte")) + print(coffee_maker_enum(coffee_enum=Coffee.LATTE)) +``` diff --git a/docs/user_guide/data_types_and_io/flytedirectory.md b/docs/user_guide/data_types_and_io/flytedirectory.md new file mode 100644 index 0000000000..6dd75ed159 --- /dev/null +++ b/docs/user_guide/data_types_and_io/flytedirectory.md @@ -0,0 +1,199 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(folder)= + +# FlyteDirectory + +```{eval-rst} +.. tags:: Data, Basic +``` + +In addition to files, folders are another fundamental operating system primitive. +Flyte supports folders in the form of +[multi-part blobs](https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/core/types.proto#L73). + +To begin, import the libraries. + +```{code-cell} +import csv +import os +import urllib.request +from collections import defaultdict +from pathlib import Path +from typing import List + +import flytekit +from flytekit import task, workflow +from flytekit.types.directory import FlyteDirectory +``` + ++++ {"lines_to_next_cell": 0} + +Building upon the previous example demonstrated in the {std:ref}`file ` section, +let's continue by considering the normalization of columns in a CSV file. + +The following task downloads a list of URLs pointing to CSV files +and returns the folder path in a `FlyteDirectory` object. + +```{code-cell} +@task +def download_files(csv_urls: List[str]) -> FlyteDirectory: + working_dir = flytekit.current_context().working_directory + local_dir = Path(os.path.join(working_dir, "csv_files")) + local_dir.mkdir(exist_ok=True) + + # get the number of digits needed to preserve the order of files in the local directory + zfill_len = len(str(len(csv_urls))) + for idx, remote_location in enumerate(csv_urls): + local_image = os.path.join( + # prefix the file name with the index location of the file in the original csv_urls list + local_dir, + f"{str(idx).zfill(zfill_len)}_{os.path.basename(remote_location)}", + ) + urllib.request.urlretrieve(remote_location, local_image) + return FlyteDirectory(path=str(local_dir)) +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +You can annotate a `FlyteDirectory` when you want to download or upload the contents of the directory in batches. +For example, + +```{code-block} +@task +def t1(directory: Annotated[FlyteDirectory, BatchSize(10)]) -> Annotated[FlyteDirectory, BatchSize(100)]: + ... + return FlyteDirectory(...) +``` + +Flytekit efficiently downloads files from the specified input directory in 10-file chunks. +It then loads these chunks into memory before writing them to the local disk. +The process repeats for subsequent sets of 10 files. +Similarly, for outputs, Flytekit uploads the resulting directory in chunks of 100. +::: + +We define a helper function to normalize the columns in-place. + +:::{note} +This is a plain Python function that will be called in a subsequent Flyte task. This example +demonstrates how Flyte tasks are simply entrypoints of execution, which can themselves call +other functions and routines that are written in pure Python. +::: + +```{code-cell} +def normalize_columns( + local_csv_file: str, + column_names: List[str], + columns_to_normalize: List[str], +): + # read the data from the raw csv file + parsed_data = defaultdict(list) + with open(local_csv_file, newline="\n") as input_file: + reader = csv.DictReader(input_file, fieldnames=column_names) + for row in (x for i, x in enumerate(reader) if i > 0): + for column in columns_to_normalize: + parsed_data[column].append(float(row[column].strip())) + + # normalize the data + normalized_data = defaultdict(list) + for colname, values in parsed_data.items(): + mean = sum(values) / len(values) + std = (sum([(x - mean) ** 2 for x in values]) / len(values)) ** 0.5 + normalized_data[colname] = [(x - mean) / std for x in values] + + # overwrite the csv file with the normalized columns + with open(local_csv_file, mode="w") as output_file: + writer = csv.DictWriter(output_file, fieldnames=columns_to_normalize) + writer.writeheader() + for row in zip(*normalized_data.values()): + writer.writerow({k: row[i] for i, k in enumerate(columns_to_normalize)}) +``` + ++++ {"lines_to_next_cell": 0} + +We then define a task that accepts the previously downloaded folder, along with some metadata about the +column names of each file in the directory and the column names that we want to normalize. + +```{code-cell} +@task +def normalize_all_files( + csv_files_dir: FlyteDirectory, + columns_metadata: List[List[str]], + columns_to_normalize_metadata: List[List[str]], +) -> FlyteDirectory: + for local_csv_file, column_names, columns_to_normalize in zip( + # make sure we sort the files in the directory to preserve the original order of the csv urls + [os.path.join(csv_files_dir, x) for x in sorted(os.listdir(csv_files_dir))], + columns_metadata, + columns_to_normalize_metadata, + ): + normalize_columns(local_csv_file, column_names, columns_to_normalize) + return FlyteDirectory(path=csv_files_dir.path) +``` + ++++ {"lines_to_next_cell": 0} + +Compose all of the above tasks into a workflow. This workflow accepts a list +of URL strings pointing to a remote location containing a CSV file, a list of column names +associated with each CSV file, and a list of columns that we want to normalize. + +```{code-cell} +@workflow +def download_and_normalize_csv_files( + csv_urls: List[str], + columns_metadata: List[List[str]], + columns_to_normalize_metadata: List[List[str]], +) -> FlyteDirectory: + directory = download_files(csv_urls=csv_urls) + return normalize_all_files( + csv_files_dir=directory, + columns_metadata=columns_metadata, + columns_to_normalize_metadata=columns_to_normalize_metadata, + ) +``` + ++++ {"lines_to_next_cell": 0} + +You can run the workflow locally as follows: + +```{code-cell} +if __name__ == "__main__": + csv_urls = [ + "https://people.sc.fsu.edu/~jburkardt/data/csv/biostats.csv", + "https://people.sc.fsu.edu/~jburkardt/data/csv/faithful.csv", + ] + columns_metadata = [ + ["Name", "Sex", "Age", "Heights (in)", "Weight (lbs)"], + ["Index", "Eruption length (mins)", "Eruption wait (mins)"], + ] + columns_to_normalize_metadata = [ + ["Age"], + ["Eruption length (mins)"], + ] + + print(f"Running {__file__} main...") + directory = download_and_normalize_csv_files( + csv_urls=csv_urls, + columns_metadata=columns_metadata, + columns_to_normalize_metadata=columns_to_normalize_metadata, + ) + print(f"Running download_and_normalize_csv_files on {csv_urls}: " f"{directory}") +``` diff --git a/docs/user_guide/data_types_and_io/flytefile.md b/docs/user_guide/data_types_and_io/flytefile.md new file mode 100644 index 0000000000..474cad4041 --- /dev/null +++ b/docs/user_guide/data_types_and_io/flytefile.md @@ -0,0 +1,168 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(file)= + +# FlyteFile + +```{eval-rst} +.. tags:: Data, Basic +``` + +Files are one of the most fundamental entities that users of Python work with, +and they are fully supported by Flyte. In the IDL, they are known as +[Blob](https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/core/literals.proto#L33) +literals which are backed by the +[blob type](https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/core/types.proto#L47). + +Let's assume our mission here is pretty simple. We download a few CSV file +links, read them with the python built-in {py:class}`csv.DictReader` function, +normalize some pre-specified columns, and output the normalized columns to +another csv file. + +First, import the libraries. + +```{code-cell} +import csv +import os +from collections import defaultdict +from typing import List + +import flytekit +from flytekit import task, workflow +from flytekit.types.file import FlyteFile +``` + ++++ {"lines_to_next_cell": 0} + +Define a task that accepts {py:class}`~flytekit.types.file.FlyteFile` as an input. +The following is a task that accepts a `FlyteFile`, a list of column names, +and a list of column names to normalize. The task then outputs a CSV file +containing only the normalized columns. For this example, we use z-score normalization, +which involves mean-centering and standard-deviation-scaling. + +:::{note} +The `FlyteFile` literal can be scoped with a string, which gets inserted +into the format of the Blob type ("jpeg" is the string in +`FlyteFile[typing.TypeVar("jpeg")]`). The format is entirely optional, +and if not specified, defaults to `""`. +Predefined aliases for commonly used flyte file formats are also available. +You can find them [here](https://github.com/flyteorg/flytekit/blob/master/flytekit/types/file/__init__.py). +::: + +```{code-cell} +@task +def normalize_columns( + csv_url: FlyteFile, + column_names: List[str], + columns_to_normalize: List[str], + output_location: str, +) -> FlyteFile: + # read the data from the raw csv file + parsed_data = defaultdict(list) + with open(csv_url, newline="\n") as input_file: + reader = csv.DictReader(input_file, fieldnames=column_names) + next(reader) # Skip header + for row in reader: + for column in columns_to_normalize: + parsed_data[column].append(float(row[column].strip())) + + # normalize the data + normalized_data = defaultdict(list) + for colname, values in parsed_data.items(): + mean = sum(values) / len(values) + std = (sum([(x - mean) ** 2 for x in values]) / len(values)) ** 0.5 + normalized_data[colname] = [(x - mean) / std for x in values] + + # write to local path + out_path = os.path.join( + flytekit.current_context().working_directory, + f"normalized-{os.path.basename(csv_url.path).rsplit('.')[0]}.csv", + ) + with open(out_path, mode="w") as output_file: + writer = csv.DictWriter(output_file, fieldnames=columns_to_normalize) + writer.writeheader() + for row in zip(*normalized_data.values()): + writer.writerow({k: row[i] for i, k in enumerate(columns_to_normalize)}) + + if output_location: + return FlyteFile(path=out_path, remote_path=output_location) + else: + return FlyteFile(path=out_path) +``` + ++++ {"lines_to_next_cell": 0} + +When the image URL is sent to the task, the Flytekit engine translates it into a `FlyteFile` object on the local +drive (but doesn't download it). The act of calling the `download()` method should trigger the download, and the `path` +attribute enables to `open` the file. + +If the `output_location` argument is specified, it will be passed to the `remote_path` argument of `FlyteFile`, +which will use that path as the storage location instead of a random location (Flyte's object store). + +When this task finishes, the Flytekit engine returns the `FlyteFile` instance, uploads the file to the location, and +creates a blob literal pointing to it. + +Lastly, define a workflow. The `normalize_csv_files` workflow has an `output_location` argument which is passed +to the `location` input of the task. If it's not an empty string, the task attempts to +upload its file to that location. + +```{code-cell} +@workflow +def normalize_csv_file( + csv_url: FlyteFile, + column_names: List[str], + columns_to_normalize: List[str], + output_location: str = "", +) -> FlyteFile: + return normalize_columns( + csv_url=csv_url, + column_names=column_names, + columns_to_normalize=columns_to_normalize, + output_location=output_location, + ) +``` + ++++ {"lines_to_next_cell": 0} + +You can run the workflow locally as follows: + +```{code-cell} +if __name__ == "__main__": + default_files = [ + ( + "https://people.sc.fsu.edu/~jburkardt/data/csv/biostats.csv", + ["Name", "Sex", "Age", "Heights (in)", "Weight (lbs)"], + ["Age"], + ), + ( + "https://people.sc.fsu.edu/~jburkardt/data/csv/faithful.csv", + ["Index", "Eruption length (mins)", "Eruption wait (mins)"], + ["Eruption length (mins)"], + ), + ] + print(f"Running {__file__} main...") + for index, (csv_url, column_names, columns_to_normalize) in enumerate(default_files): + normalized_columns = normalize_csv_file( + csv_url=csv_url, + column_names=column_names, + columns_to_normalize=columns_to_normalize, + ) + print(f"Running normalize_csv_file workflow on {csv_url}: " f"{normalized_columns}") +``` diff --git a/docs/user_guide/data_types_and_io/index.md b/docs/user_guide/data_types_and_io/index.md new file mode 100644 index 0000000000..f419c99bd3 --- /dev/null +++ b/docs/user_guide/data_types_and_io/index.md @@ -0,0 +1,152 @@ +(data_types_and_io)= + +# Data Types and IO + +Flyte being a data-aware orchestration platform, types play a vital role within it. +This section provides an introduction to the wide range of data types that Flyte supports. +These types serve a dual purpose by not only validating the data but also enabling seamless +transfer of data between local and cloud storage. +They enable: + +- Data lineage +- Memoization +- Auto parallelization +- Simplifying access to data +- Auto generated CLI and launch UI + +For a more comprehensive understanding of how Flyte manages data, refer to the +{std:ref}`Understand How Flyte Handles Data ` guide. + +(python_to_flyte_type_mapping)= + +## Mapping Python to Flyte types + +Flytekit automatically translates most Python types into Flyte types. +Here's a breakdown of these mappings: + +```{eval-rst} +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Python Type + - Flyte Type + - Conversion + - Comment + * - ``int`` + - ``Integer`` + - Automatic + - Use Python 3 type hints. + * - ``float`` + - ``Float`` + - Automatic + - Use Python 3 type hints. + * - ``str`` + - ``String`` + - Automatic + - Use Python 3 type hints. + * - ``bool`` + - ``Boolean`` + - Automatic + - Use Python 3 type hints. + * - ``bytes``/``bytearray`` + - ``Binary`` + - Not Supported + - You have the option to employ your own custom type transformer. + * - ``complex`` + - NA + - Not Supported + - You have the option to employ your own custom type transformer. + * - ``datetime.timedelta`` + - ``Duration`` + - Automatic + - Use Python 3 type hints. + * - ``datetime.datetime`` + - ``Datetime`` + - Automatic + - Use Python 3 type hints. + * - ``datetime.date`` + - ``Datetime`` + - Automatic + - Use Python 3 type hints. + * - ``typing.List[T]`` / ``list[T]`` + - ``Collection [T]`` + - Automatic + - Use ``typing.List[T]`` or ``list[T]``, where ``T`` can represent one of the other supported types listed in the table. + * - ``typing.Iterator[T]`` + - ``Collection [T]`` + - Automatic + - Use ``typing.Iterator[T]``, where ``T`` can represent one of the other supported types listed in the table. + * - File / file-like / ``os.PathLike`` + - ``FlyteFile`` + - Automatic + - If you're using ``file`` or ``os.PathLike`` objects, Flyte will default to the binary protocol for the file. + When using ``FlyteFile["protocol"]``, it is assumed that the file is in the specified protocol, such as 'jpg', 'png', 'hdf5', etc. + * - Directory + - ``FlyteDirectory`` + - Automatic + - When using ``FlyteDirectory["protocol"]``, it is assumed that all the files belong to the specified protocol. + * - ``typing.Dict[str, V]`` / ``dict[str, V]`` + - ``Map[str, V]`` + - Automatic + - Use ``typing.Dict[str, V]`` or ``dict[str, V]``, where ``V`` can be one of the other supported types in the table, + including a nested dictionary. + * - ``dict`` + - JSON (``struct.pb``) + - Automatic + - Use ``dict``. It's assumed that the untyped dictionary can be converted to JSON. + However, this may not always be possible and could result in a ``RuntimeError``. + * - ``@dataclass`` + - ``Struct`` + - Automatic + - The class should be a pure value class that inherits from Mashumaro's DataClassJSONMixin, + and be annotated with the ``@dataclass`` decorator. + * - ``np.ndarray`` + - File + - Automatic + - Use ``np.ndarray`` as a type hint. + * - ``pandas.DataFrame`` + - Structured Dataset + - Automatic + - Use ``pandas.DataFrame`` as a type hint. Pandas column types aren't preserved. + * - ``pyspark.DataFrame`` + - Structured Dataset + - To utilize the type, install the ``flytekitplugins-spark`` plugin. + - Use ``pyspark.DataFrame`` as a type hint. + * - ``pydantic.BaseModel`` + - ``Map`` + - To utilize the type, install the ``flytekitplugins-pydantic`` plugin. + - Use ``pydantic.BaseModel`` as a type hint. + * - ``torch.Tensor`` / ``torch.nn.Module`` + - File + - To utilize the type, install the ``torch`` library. + - Use ``torch.Tensor`` or ``torch.nn.Module`` as a type hint, and you can use their derived types. + * - ``tf.keras.Model`` + - File + - To utilize the type, install the ``tensorflow`` library. + - Use ``tf.keras.Model`` and its derived types. + * - ``sklearn.base.BaseEstimator`` + - File + - To utilize the type, install the ``scikit-learn`` library. + - Use ``sklearn.base.BaseEstimator`` and its derived types. + * - User defined types + - Any + - Custom transformers + - The ``FlytePickle`` transformer is the default option, but you can also define custom transformers. + **For instructions on building custom type transformers, please refer to :ref:`this section `**. +``` + +```{toctree} +:maxdepth: -1 +:name: data_types_and_io_toc +:hidden: + +flytefile +flytedirectory +structureddataset +dataclass +accessing_attributes +pytorch_type +enum_type +pickle_type +``` diff --git a/docs/user_guide/data_types_and_io/pickle_type.md b/docs/user_guide/data_types_and_io/pickle_type.md new file mode 100644 index 0000000000..b5cbb89f5a --- /dev/null +++ b/docs/user_guide/data_types_and_io/pickle_type.md @@ -0,0 +1,131 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(pickle_type)= + +# Pickle type + +```{eval-rst} +.. tags:: Basic +``` + +Flyte enforces type safety by utilizing type information for compiling tasks and workflows, +enabling various features such as static analysis and conditional branching. + +However, we also strive to offer flexibility to end-users so they don't have to invest heavily +in understanding their data structures upfront before experiencing the value Flyte has to offer. + +Flyte supports the `FlytePickle` transformer, which converts any unrecognized type hint into `FlytePickle`, +enabling the serialization/deserialization of Python values to/from a pickle file. + +:::{important} +Pickle can only be used to send objects between the exact same Python version. +For optimal performance, it's advisable to either employ Python types that are supported by Flyte +or register a custom transformer, as using pickle types can result in lower performance. +::: + +This example demonstrates how you can utilize custom objects without registering a transformer. + +```{code-cell} +from flytekit import task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +`Superhero` represents a user-defined complex type that can be serialized to a pickle file by Flytekit +and transferred between tasks as both input and output data. + +:::{note} +Alternatively, you can {ref}`turn this object into a dataclass ` for improved performance. +We have used a simple object here for demonstration purposes. +::: + +```{code-cell} +class Superhero: + def __init__(self, name, power): + self.name = name + self.power = power + + +@task +def welcome_superhero(name: str, power: str) -> Superhero: + return Superhero(name, power) + + +@task +def greet_superhero(superhero: Superhero) -> str: + return f"👋 Hello {superhero.name}! Your superpower is {superhero.power}." + + +@workflow +def superhero_wf(name: str = "Thor", power: str = "Flight") -> str: + superhero = welcome_superhero(name=name, power=power) + return greet_superhero(superhero=superhero) +``` + ++++ {"lines_to_next_cell": 0} + +## Batch size + +By default, if the list subtype is unrecognized, a single pickle file is generated. +To optimize serialization and deserialization performance for scenarios involving a large number of items +or significant list elements, you can specify a batch size. +This feature allows for the processing of each batch as a separate pickle file. +The following example demonstrates how to set the batch size. + +```{code-cell} +from typing import Iterator + +from flytekit.types.pickle.pickle import BatchSize +from typing_extensions import Annotated + + +@task +def welcome_superheroes(names: list[str], powers: list[str]) -> Annotated[list[Superhero], BatchSize(3)]: + return [Superhero(name, power) for name, power in zip(names, powers)] + + +@task +def greet_superheroes(superheroes: list[Superhero]) -> Iterator[str]: + for superhero in superheroes: + yield f"👋 Hello {superhero.name}! Your superpower is {superhero.power}." + + +@workflow +def superheroes_wf( + names: list[str] = ["Thor", "Spiderman", "Hulk"], + powers: list[str] = ["Flight", "Surface clinger", "Shapeshifting"], +) -> Iterator[str]: + superheroes = welcome_superheroes(names=names, powers=powers) + return greet_superheroes(superheroes=superheroes) +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +The `welcome_superheroes` task will generate two pickle files: one containing two superheroes and the other containing one superhero. +::: + +You can run the workflows locally as follows: + +```{code-cell} +if __name__ == "__main__": + print(f"Superhero wf: {superhero_wf()}") + print(f"Superhero(es) wf: {superheroes_wf()}") +``` diff --git a/docs/user_guide/data_types_and_io/pytorch_type.md b/docs/user_guide/data_types_and_io/pytorch_type.md new file mode 100644 index 0000000000..4e5715d128 --- /dev/null +++ b/docs/user_guide/data_types_and_io/pytorch_type.md @@ -0,0 +1,219 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(pytorch_type)= + +# PyTorch type + +```{eval-rst} +.. tags:: MachineLearning, Basic +``` + +Flyte advocates for the use of strongly-typed data to simplify the development of robust and testable pipelines. +In addition to its application in data engineering, Flyte is primarily used for machine learning. +To streamline the communication between Flyte tasks, particularly when dealing with tensors and models, +we have introduced support for PyTorch types. + +## Tensors and modules + +At times, you may find the need to pass tensors and modules (models) within your workflow. +Without native support for PyTorch tensors and modules, Flytekit relies on {std:ref}`pickle ` for serializing +and deserializing these entities, as well as any unknown types. +However, this approach isn't the most efficient. As a result, we've integrated PyTorch's +serialization and deserialization support into the Flyte type system. + +```{code-cell} +import torch +from flytekit import task, workflow + + +@task +def generate_tensor_2d() -> torch.Tensor: + return torch.tensor([[1.0, -1.0, 2], [1.0, -1.0, 9], [0, 7.0, 3]]) + + +@task +def reshape_tensor(tensor: torch.Tensor) -> torch.Tensor: + # convert 2D to 3D + tensor.unsqueeze_(-1) + return tensor.expand(3, 3, 2) + + +@task +def generate_module() -> torch.nn.Module: + bn = torch.nn.BatchNorm1d(3, track_running_stats=True) + return bn + + +@task +def get_model_weight(model: torch.nn.Module) -> torch.Tensor: + return model.weight + + +class MyModel(torch.nn.Module): + def __init__(self): + super(MyModel, self).__init__() + self.l0 = torch.nn.Linear(4, 2) + self.l1 = torch.nn.Linear(2, 1) + + def forward(self, input): + out0 = self.l0(input) + out0_relu = torch.nn.functional.relu(out0) + return self.l1(out0_relu) + + +@task +def get_l1() -> torch.nn.Module: + model = MyModel() + return model.l1 + + +@workflow +def pytorch_native_wf(): + reshape_tensor(tensor=generate_tensor_2d()) + get_model_weight(model=generate_module()) + get_l1() +``` + ++++ {"lines_to_next_cell": 0} + +Passing around tensors and modules is no more a hassle! + +## Checkpoint + +`PyTorchCheckpoint` is a specialized checkpoint used for serializing and deserializing PyTorch models. +It checkpoints `torch.nn.Module`'s state, hyperparameters and optimizer state. + +This module checkpoint differs from the standard checkpoint as it specifically captures the module's `state_dict`. +Therefore, when restoring the module, the module's `state_dict` must be used in conjunction with the actual module. +According to the PyTorch [docs](https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model), +it's recommended to store the module's `state_dict` rather than the module itself, +although the serialization should work in either case. + +```{code-cell} +:lines_to_next_cell: 2 + +from dataclasses import dataclass + +import torch.nn as nn +import torch.nn.functional as F +import torch.optim as optim +from dataclasses_json import dataclass_json +from flytekit.extras.pytorch import PyTorchCheckpoint + + +@dataclass_json +@dataclass +class Hyperparameters: + epochs: int + loss: float + + +class Net(nn.Module): + def __init__(self): + super(Net, self).__init__() + self.conv1 = nn.Conv2d(3, 6, 5) + self.pool = nn.MaxPool2d(2, 2) + self.conv2 = nn.Conv2d(6, 16, 5) + self.fc1 = nn.Linear(16 * 5 * 5, 120) + self.fc2 = nn.Linear(120, 84) + self.fc3 = nn.Linear(84, 10) + + def forward(self, x): + x = self.pool(F.relu(self.conv1(x))) + x = self.pool(F.relu(self.conv2(x))) + x = x.view(-1, 16 * 5 * 5) + x = F.relu(self.fc1(x)) + x = F.relu(self.fc2(x)) + x = self.fc3(x) + return x + + +@task +def generate_model(hyperparameters: Hyperparameters) -> PyTorchCheckpoint: + bn = Net() + optimizer = optim.SGD(bn.parameters(), lr=0.001, momentum=0.9) + return PyTorchCheckpoint(module=bn, hyperparameters=hyperparameters, optimizer=optimizer) + + +@task +def load(checkpoint: PyTorchCheckpoint): + new_bn = Net() + new_bn.load_state_dict(checkpoint["module_state_dict"]) + optimizer = optim.SGD(new_bn.parameters(), lr=0.001, momentum=0.9) + optimizer.load_state_dict(checkpoint["optimizer_state_dict"]) + + +@workflow +def pytorch_checkpoint_wf(): + checkpoint = generate_model(hyperparameters=Hyperparameters(epochs=10, loss=0.1)) + load(checkpoint=checkpoint) +``` + +:::{note} +`PyTorchCheckpoint` supports serializing hyperparameters of types `dict`, `NamedTuple` and `dataclass`. +::: + +## Auto GPU to CPU and CPU to GPU conversion + +Not all PyTorch computations require a GPU. In some cases, it can be advantageous to transfer the +computation to a CPU, especially after training the model on a GPU. +To utilize the power of a GPU, the typical construct to use is: `to(torch.device("cuda"))`. + +When working with GPU variables on a CPU, variables need to be transferred to the CPU using the `to(torch.device("cpu"))` construct. +However, this manual conversion recommended by PyTorch may not be very user-friendly. +To address this, we added support for automatic GPU to CPU conversion (and vice versa) for PyTorch types. + +```python +from flytekit import Resources +from typing import Tuple + + +@task(requests=Resources(gpu="1")) +def train() -> Tuple[PyTorchCheckpoint, torch.Tensor, torch.Tensor, torch.Tensor]: + ... + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + model = Model(X_train.shape[1]) + model.to(device) + ... + X_train, X_test = X_train.to(device), X_test.to(device) + y_train, y_test = y_train.to(device), y_test.to(device) + ... + return PyTorchCheckpoint(module=model), X_train, X_test, y_test + +@task +def predict( + checkpoint: PyTorchCheckpoint, + X_train: torch.Tensor, + X_test: torch.Tensor, + y_test: torch.Tensor, +): + new_bn = Model(X_train.shape[1]) + new_bn.load_state_dict(checkpoint["module_state_dict"]) + + accuracy_list = np.zeros((5,)) + + with torch.no_grad(): + y_pred = new_bn(X_test) + correct = (torch.argmax(y_pred, dim=1) == y_test).type(torch.FloatTensor) + accuracy_list = correct.mean() +``` + +The `predict` task will run on a CPU, and +the device conversion from GPU to CPU will be automatically handled by Flytekit. diff --git a/docs/user_guide/data_types_and_io/structureddataset.md b/docs/user_guide/data_types_and_io/structureddataset.md new file mode 100644 index 0000000000..6639d1b2a7 --- /dev/null +++ b/docs/user_guide/data_types_and_io/structureddataset.md @@ -0,0 +1,376 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(structured_dataset)= + +# StructuredDataset + +```{eval-rst} +.. tags:: Basic, DataFrame +``` +```{currentmodule} flytekit.types.structured +``` + +As with most type systems, Python has primitives, container types like maps and tuples, and support for user-defined structures. +However, while there’s a rich variety of dataframe classes (Pandas, Spark, Pandera, etc.), there’s no native Python type that +represents a dataframe in the abstract. This is the gap that the {py:class}`StructuredDataset` type is meant to fill. +It offers the following benefits: + +- Eliminate boilerplate code you would otherwise need to write to serialize/deserialize from file objects into dataframe instances, +- Eliminate additional inputs/outputs that convey metadata around the format of the tabular data held in those files, +- Add flexibility around how dataframe files are loaded, +- Offer a range of dataframe specific functionality - enforce compatibility of different schemas + (not only at compile time, but also runtime since type information is carried along in the literal), + store third-party schema definitions, and potentially in the future, render sample data, provide summary statistics, etc. + +## Usage + +To use the `StructuredDataset` type, import `pandas` and define a task that returns a Pandas Dataframe. +Flytekit will detect the Pandas DataFrame return signature and convert the interface for the task to +the {py:class}`StructuredDataset` type. + +## Example + +This example demonstrates how to work with a structured dataset using Flyte entities. + +```{note} +To use the `StructuredDataset` type, you only need to import `pandas`. +The other imports specified below are only necessary for this specific example. +``` + +To begin, import the dependencies for the example: + +```{code-cell} +import os +import typing + +import numpy as np +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq +from flytekit import FlyteContext, StructuredDatasetType, kwtypes, task, workflow +from flytekit.models import literals +from flytekit.models.literals import StructuredDatasetMetadata +from flytekit.types.structured.structured_dataset import ( + PARQUET, + StructuredDataset, + StructuredDatasetDecoder, + StructuredDatasetEncoder, + StructuredDatasetTransformerEngine, +) +from typing_extensions import Annotated +``` + ++++ {"lines_to_next_cell": 0} + +Define a task that returns a Pandas DataFrame. + +```{code-cell} +@task +def generate_pandas_df(a: int) -> pd.DataFrame: + return pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [a, 22], "Height": [160, 178]}) +``` + ++++ {"lines_to_next_cell": 0} + +Using this simplest form, however, the user is not able to set the additional dataframe information alluded to above, + +- Column type information +- Serialized byte format +- Storage driver and location +- Additional third party schema information + +This is by design as we wanted the default case to suffice for the majority of use-cases, and to require +as few changes to existing code as possible. Specifying these is simple, however, and relies on Python variable annotations, +which is designed explicitly to supplement types with arbitrary metadata. + +## Column type information +If you want to extract a subset of actual columns of the dataframe and specify their types for type validation, +you can just specify the column names and their types in the structured dataset type annotation. + +First, initialize column types you want to extract from the `StructuredDataset`. + +```{code-cell} +all_cols = kwtypes(Name=str, Age=int, Height=int) +col = kwtypes(Age=int) +``` + ++++ {"lines_to_next_cell": 0} + +Define a task that opens a structured dataset by calling `all()`. +When you invoke `all()` with ``pandas.DataFrame``, the Flyte engine downloads the parquet file on S3, and deserializes it to `pandas.DataFrame`. +Keep in mind that you can invoke ``open()`` with any dataframe type that's supported or added to structured dataset. +For instance, you can use ``pa.Table`` to convert the Pandas DataFrame to a PyArrow table. + +```{code-cell} +@task +def get_subset_pandas_df(df: Annotated[StructuredDataset, all_cols]) -> Annotated[StructuredDataset, col]: + df = df.open(pd.DataFrame).all() + df = pd.concat([df, pd.DataFrame([[30]], columns=["Age"])]) + return StructuredDataset(dataframe=df) + + +@workflow +def simple_sd_wf(a: int = 19) -> Annotated[StructuredDataset, col]: + pandas_df = generate_pandas_df(a=a) + return get_subset_pandas_df(df=pandas_df) +``` + ++++ {"lines_to_next_cell": 0} + +The code may result in runtime failures if the columns do not match. +The input ``df`` has ``Name``, ``Age`` and ``Height`` columns, whereas the output structured dataset will only have the ``Age`` column. + +## Serialized byte format +You can use a custom serialization format to serialize your dataframes. +Here's how you can register the Pandas to CSV handler, which is already available, +and enable the CSV serialization by annotating the structured dataset with the CSV format: + +```{code-cell} +from flytekit.types.structured import register_csv_handlers +from flytekit.types.structured.structured_dataset import CSV + +register_csv_handlers() + + +@task +def pandas_to_csv(df: pd.DataFrame) -> Annotated[StructuredDataset, CSV]: + return StructuredDataset(dataframe=df) + + +@workflow +def pandas_to_csv_wf() -> Annotated[StructuredDataset, CSV]: + pandas_df = generate_pandas_df(a=19) + return pandas_to_csv(df=pandas_df) +``` + ++++ {"lines_to_next_cell": 0} + +## Storage driver and location +By default, the data will be written to the same place that all other pointer-types (FlyteFile, FlyteDirectory, etc.) are written to. +This is controlled by the output data prefix option in Flyte which is configurable on multiple levels. + +That is to say, in the simple default case, Flytekit will, + +- Look up the default format for say, Pandas dataframes, +- Look up the default storage location based on the raw output prefix setting, +- Use these two settings to select an encoder and invoke it. + +So what's an encoder? To understand that, let's look into how the structured dataset plugin works. + +## Inner workings of a structured dataset plugin + +Two things need to happen with any dataframe instance when interacting with Flyte: + +- Serialization/deserialization from/to the Python instance to bytes (in the format specified above). +- Transmission/retrieval of those bits to/from somewhere. + +Each structured dataset plugin (called encoder or decoder) needs to perform both of these steps. +Flytekit decides which of the loaded plugins to invoke based on three attributes: + +- The byte format +- The storage location +- The Python type in the task or workflow signature. + +These three keys uniquely identify which encoder (used when converting a dataframe in Python memory to a Flyte value, +e.g. when a task finishes and returns a dataframe) or decoder (used when hydrating a dataframe in memory from a Flyte value, +e.g. when a task starts and has a dataframe input) to invoke. + +However, it is awkward to require users to use `typing.Annotated` on every signature. +Therefore, Flytekit has a default byte-format for every Python dataframe type registered with flytekit. + +## The `uri` argument + +BigQuery `uri` allows you to load and retrieve data from cloud using the `uri` argument. +The `uri` comprises of the bucket name and the filename prefixed with `gs://`. +If you specify BigQuery `uri` for structured dataset, BigQuery creates a table in the location specified by the `uri`. +The `uri` in structured dataset reads from or writes to S3, GCP, BigQuery or any storage. + +Before writing DataFrame to a BigQuery table, + +1. Create a [GCP account](https://cloud.google.com/docs/authentication/getting-started) and create a service account. +2. Create a project and add the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to your `.bashrc` file. +3. Create a dataset in your project. + +Here's how you can define a task that converts a pandas DataFrame to a BigQuery table: + +```python +@task +def pandas_to_bq() -> StructuredDataset: + df = pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [20, 22]}) + return StructuredDataset(dataframe=df, uri="gs:///") +``` + +Replace `BUCKET_NAME` with the name of your GCS bucket and `FILE_NAME` with the name of the file the dataframe should be copied to. + +### Note that no format was specified in the structured dataset constructor, or in the signature. So how did the BigQuery encoder get invoked? +This is because the stock BigQuery encoder is loaded into Flytekit with an empty format. +The Flytekit `StructuredDatasetTransformerEngine` interprets that to mean that it is a generic encoder +(or decoder) and can work across formats, if a more specific format is not found. + +And here's how you can define a task that converts the BigQuery table to a pandas DataFrame: + +```python +@task +def bq_to_pandas(sd: StructuredDataset) -> pd.DataFrame: + return sd.open(pd.DataFrame).all() +``` + +:::{note} +Flyte creates a table inside the dataset in the project upon BigQuery query execution. +::: + +## How to return multiple dataframes from a task? +For instance, how would a task return say two dataframes: +- The first dataframe be written to BigQuery and serialized by one of their libraries, +- The second needs to be serialized to CSV and written at a specific location in GCS different from the generic pointer-data bucket + +If you want the default behavior (which is itself configurable based on which plugins are loaded), +you can work just with your current raw dataframe classes. + +```python +@task +def t1() -> typing.Tuple[StructuredDataset, StructuredDataset]: + ... + return StructuredDataset(df1, uri="bq://project:flyte.table"), \ + StructuredDataset(df2, uri="gs://auxiliary-bucket/data") +``` + +If you want to customize the Flyte interaction behavior, you'll need to wrap your dataframe in a `StructuredDataset` wrapper object. + +## How to define a custom structured dataset plugin? + +`StructuredDataset` ships with an encoder and a decoder that handles the conversion of a +Python value to a Flyte literal and vice-versa, respectively. +Here is a quick demo showcasing how one might build a NumPy encoder and decoder, +enabling the use of a 2D NumPy array as a valid type within structured datasets. + +### NumPy encoder + +Extend `StructuredDatasetEncoder` and implement the `encode` function. +The `encode` function converts NumPy array to an intermediate format (parquet file format in this case). + +```{code-cell} +class NumpyEncodingHandler(StructuredDatasetEncoder): + def encode( + self, + ctx: FlyteContext, + structured_dataset: StructuredDataset, + structured_dataset_type: StructuredDatasetType, + ) -> literals.StructuredDataset: + df = typing.cast(np.ndarray, structured_dataset.dataframe) + name = ["col" + str(i) for i in range(len(df))] + table = pa.Table.from_arrays(df, name) + path = ctx.file_access.get_random_remote_directory() + local_dir = ctx.file_access.get_random_local_directory() + local_path = os.path.join(local_dir, f"{0:05}") + pq.write_table(table, local_path) + ctx.file_access.upload_directory(local_dir, path) + return literals.StructuredDataset( + uri=path, + metadata=StructuredDatasetMetadata(structured_dataset_type=StructuredDatasetType(format=PARQUET)), + ) +``` + ++++ {"lines_to_next_cell": 0} + +### NumPy decoder + +Extend {py:class}`StructuredDatasetDecoder` and implement the {py:meth}`~StructuredDatasetDecoder.decode` function. +The {py:meth}`~StructuredDatasetDecoder.decode` function converts the parquet file to a `numpy.ndarray`. + +```{code-cell} +class NumpyDecodingHandler(StructuredDatasetDecoder): + def decode( + self, + ctx: FlyteContext, + flyte_value: literals.StructuredDataset, + current_task_metadata: StructuredDatasetMetadata, + ) -> np.ndarray: + local_dir = ctx.file_access.get_random_local_directory() + ctx.file_access.get_data(flyte_value.uri, local_dir, is_multipart=True) + table = pq.read_table(local_dir) + return table.to_pandas().to_numpy() +``` + ++++ {"lines_to_next_cell": 0} + +### NumPy renderer + +Create a default renderer for numpy array, then Flytekit will use this renderer to +display schema of NumPy array on the Flyte deck. + +```{code-cell} +class NumpyRenderer: + def to_html(self, df: np.ndarray) -> str: + assert isinstance(df, np.ndarray) + name = ["col" + str(i) for i in range(len(df))] + table = pa.Table.from_arrays(df, name) + return pd.DataFrame(table.schema).to_html(index=False) +``` + ++++ {"lines_to_next_cell": 0} + +In the end, register the encoder, decoder and renderer with the `StructuredDatasetTransformerEngine`. +Specify the Python type you want to register this encoder with (`np.ndarray`), +the storage engine to register this against (if not specified, it is assumed to work for all the storage backends), +and the byte format, which in this case is `PARQUET`. + +```{code-cell} +StructuredDatasetTransformerEngine.register(NumpyEncodingHandler(np.ndarray, None, PARQUET)) +StructuredDatasetTransformerEngine.register(NumpyDecodingHandler(np.ndarray, None, PARQUET)) +StructuredDatasetTransformerEngine.register_renderer(np.ndarray, NumpyRenderer()) +``` + ++++ {"lines_to_next_cell": 0} + +You can now use `numpy.ndarray` to deserialize the parquet file to NumPy and serialize a task's output (NumPy array) to a parquet file. + +```{code-cell} +@task +def generate_pd_df_with_str() -> pd.DataFrame: + return pd.DataFrame({"Name": ["Tom", "Joseph"]}) + + +@task +def to_numpy(sd: StructuredDataset) -> Annotated[StructuredDataset, None, PARQUET]: + numpy_array = sd.open(np.ndarray).all() + return StructuredDataset(dataframe=numpy_array) + + +@workflow +def numpy_wf() -> Annotated[StructuredDataset, None, PARQUET]: + return to_numpy(sd=generate_pd_df_with_str()) +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +`pyarrow` raises an `Expected bytes, got a 'int' object` error when the dataframe contains integers. +::: + +You can run the code locally as follows: + +```{code-cell} +if __name__ == "__main__": + sd = simple_sd_wf() + print(f"A simple Pandas dataframe workflow: {sd.open(pd.DataFrame).all()}") + print(f"Using CSV as the serializer: {pandas_to_csv_wf().open(pd.DataFrame).all()}") + print(f"NumPy encoder and decoder: {numpy_wf().open(np.ndarray).all()}") +``` diff --git a/docs/user_guide/development_lifecycle/agents.md b/docs/user_guide/development_lifecycle/agents.md new file mode 100644 index 0000000000..2ec1309272 --- /dev/null +++ b/docs/user_guide/development_lifecycle/agents.md @@ -0,0 +1,234 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(extend-agent-service)= + +# Agents + +```{eval-rst} +.. tags:: Extensibility, Contribute, Intermediate +``` + +:::{note} +This is an experimental feature, which is subject to change the API in the future. +::: + +## What is an agent? + +In Flyte, an agent is a long-running stateless service that can be used to execute tasks. It reduces the overhead of creating a pod for each task. +In addition, it's easy to scale up and down the agent service based on the workload. Agent services are designed to be language-agnostic. +For now, we only support Python agent, but we may support other languages in the future. + +Agent is designed to run a specific type of task. For example, you can create a BigQuery agent to run BigQuery task. Therefore, if you create a new type of task, you can +either run the task in the pod, or you can create a new agent to run it. You can determine how the task will be executed in the FlytePropeller configMap. + +Key goals of the agent service include: + +- Support for communication with external services: The focus is on enabling agents that seamlessly interact with external services. +- Independent testing and private deployment: Agents can be tested independently and deployed privately, providing flexibility and control over development. +- Flyte Agent usage in local development: Users, especially in `flytekit` and `unionml`, can leverage backend agents for local development, streamlining the development process. +- Language-agnostic: Agents can be authored in any programming language, allowing users to work with their preferred language and tools. +- Scalability: Agents are designed to be scalable, ensuring they can handle large-scale workloads effectively. +- Simple API: Agents offer a straightforward API, making integration and usage straightforward for developers. + +## Why do we need an agent service? + +Without agents, people need to implement a backend plugin in the propeller. The backend plugin is responsible for +creating a CRD and submitting a http request to the external service. However, it increases the complexity of flytepropeller, and +it's hard to maintain the backend plugin. For example, if we want to add a new plugin, we need to update and compile +flytepropeller, and it's also hard to test. In addition, the backend plugin is running in flytepropeller itself, so it +increases the load of the flytepropeller engine. + +Furthermore, implementing backend plugins can be challenging, particularly for data scientists and ML engineers who may lack proficiency in +Golang. Additionally, managing performance requirements, maintenance, and development can be burdensome. +To address these issues, we introduced the "Agent Service" in Flyte. This system enables rapid plugin +development while decoupling them from the core flytepropeller engine. + +## Overview + +The Flyte agent service is a Python-based agent registry powered by a gRPC server. It allows users and flytepropeller +to send gRPC requests to the registry for executing jobs such as BigQuery and Databricks. Each Agent service is a Kubernetes +deployment. You can create two different Agent services hosting different Agents. For example, you can create one production +agent service and one development agent service. + +:::{figure} https://i.ibb.co/vXhBDjP/Screen-Shot-2023-05-29-at-2-54-14-PM.png +:alt: Agent Service +:class: with-shadow +::: + +## How to register a new agent + +### Flytekit interface specification + +To register a new agent, you can extend the `AgentBase` class in the flytekit backend module. Implementing the following three methods is necessary, and it's important to ensure that all calls are idempotent: + +- `create`: This method is used to initiate a new task. Users have the flexibility to use gRPC, REST, or an SDK to create a task. +- `get`: This method allows retrieving the job Resource (jobID or output literal) associated with the task, such as a BigQuery Job ID or Databricks task ID. +- `delete`: Invoking this method will send a request to delete the corresponding job. + +```python +from flytekit.extend.backend.base_agent import AgentBase, AgentRegistry +from dataclasses import dataclass +import requests + +@dataclass +class Metadata: + # you can add any metadata you want, propeller will pass the metadata to the agent to get the job status. + # For example, you can add the job_id to the metadata, and the agent will use the job_id to get the job status. + # You could also add the s3 file path, and the agent can check if the file exists. + job_id: str + +class CustomAgent(AgentBase): + def __init__(self, task_type: str): + # Each agent should have a unique task type. Agent service will use the task type to find the corresponding agent. + self._task_type = task_type + + def create( + self, + context: grpc.ServicerContext, + output_prefix: str, + task_template: TaskTemplate, + inputs: typing.Optional[LiteralMap] = None, + ) -> TaskCreateResponse: + # 1. Submit the task to the external service (BigQuery, DataBricks, etc.) + # 2. Create a task metadata such as jobID. + # 3. Return the task metadata, and keep in mind that the metadata should be serialized to bytes. + res = requests.post(url, json=data) + return CreateTaskResponse(resource_meta=json.dumps(asdict(Metadata(job_id=str(res.job_id)))).encode("utf-8")) + + def get(self, context: grpc.ServicerContext, resource_meta: bytes) -> TaskGetResponse: + # 1. Deserialize the metadata. + # 2. Use the metadata to get the job status. + # 3. Return the job status. + metadata = Metadata(**json.loads(resource_meta.decode("utf-8"))) + res = requests.get(url, json={"job_id": metadata.job_id}) + return GetTaskResponse(resource=Resource(state=res.state) + + def delete(self, context: grpc.ServicerContext, resource_meta: bytes) -> TaskDeleteResponse: + # 1. Deserialize the metadata. + # 2. Use the metadata to delete the job. + # 3. If failed to delete the job, add the error message to the grpc context. + # context.set_code(grpc.StatusCode.INTERNAL) + # context.set_details(f"failed to create task with error {e}") + try: + metadata = Metadata(**json.loads(resource_meta.decode("utf-8"))) + requests.delete(url, json={"job_id": metadata.job_id}) + except Exception as e: + logger.error(f"failed to delete task with error {e}") + context.set_code(grpc.StatusCode.INTERNAL) + context.set_details(f"failed to delete task with error {e}") + return DeleteTaskResponse() + +# To register the custom agent +AgentRegistry.register(CustomAgent()) +``` + +Here is an example of [BigQuery Agent](https://github.com/flyteorg/flytekit/blob/9977aac26242ebbede8e00d476c2fbc59ac5487a/plugins/flytekit-bigquery/flytekitplugins/bigquery/agent.py#L35) implementation. + +### How to test the agent + +Agent can be tested locally without running backend server. It makes the development of the agent easier. + +The task inherited from AsyncAgentExecutorMixin can be executed locally, allowing flytekit to mimic the propeller's behavior to call the agent. +In some cases, you should store credentials in your local environment when testing locally. +For example, you need to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable when testing the BigQuery task. +After setting up the CREDENTIALS, you can run the task locally. Flytekit will automatically call the agent to create, get, or delete the task. + +```python +bigquery_doge_coin = BigQueryTask( + name=f"bigquery.doge_coin", + inputs=kwtypes(version=int), + query_template="SELECT * FROM `bigquery-public-data.crypto_dogecoin.transactions` WHERE version = @version LIMIT 10;", + output_structured_dataset_type=StructuredDataset, + task_config=BigQueryConfig(ProjectID="flyte-test-340607") +) +``` + +Task above task as an example, you can run the task locally and test agent with the following command: + +```bash +pyflyte run wf.py bigquery_doge_coin --version 10 +``` + +### Build a new image + +The following is a sample Dockerfile for building an image for a flyte agent. + +```Dockerfile +FROM python:3.9-slim-buster + +MAINTAINER Flyte Team +LABEL org.opencontainers.image.source=https://github.com/flyteorg/flytekit + +WORKDIR /root +ENV PYTHONPATH /root + +# flytekit will autoload the agent if package is installed. +RUN pip install flytekitplugins-bigquery +CMD pyflyte serve agent --port 8000 +``` + +:::{note} +For flytekit versions `<=v1.10.2`, use `pyflyte serve`. +For flytekit versions `>v1.10.2`, use `pyflyte serve agent`. +::: + +### Update FlyteAgent + +1. Update the FlyteAgent deployment's [image](https://github.com/flyteorg/flyte/blob/c049865cba017ad826405c7145cd3eccbc553232/charts/flyteagent/templates/agent/deployment.yaml#L26) +2. Update the FlytePropeller configmap. + +```YAML +tasks: + task-plugins: + enabled-plugins: + - agent-service + default-for-task-types: + - bigquery_query_job_task: agent-service + - custom_task: agent-service + +plugins: + agent-service: + supportedTaskTypes: + - bigquery_query_job_task + - default_task + - custom_task + # By default, all the request will be sent to the default agent. + defaultAgent: + endpoint: "dns:///flyteagent.flyte.svc.cluster.local:8000" + insecure: true + timeouts: + GetTask: 200ms + defaultTimeout: 50ms + agents: + custom_agent: + endpoint: "dns:///custom-flyteagent.flyte.svc.cluster.local:8000" + insecure: false + defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}' + timeouts: + GetTask: 100ms + defaultTimeout: 20ms + agentForTaskTypes: + # It will override the default agent for custom_task, which means propeller will send the request to this agent. + - custom_task: custom_agent +``` + +3. Restart the FlytePropeller + +``` +kubectl rollout restart deployment flytepropeller -n flyte +``` diff --git a/docs/user_guide/development_lifecycle/cache_serializing.md b/docs/user_guide/development_lifecycle/cache_serializing.md new file mode 100644 index 0000000000..f570b1c351 --- /dev/null +++ b/docs/user_guide/development_lifecycle/cache_serializing.md @@ -0,0 +1,74 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Cache serializing + +```{eval-rst} +.. tags:: Intermediate +``` + +Serializing means only executing a single instance of a unique cacheable task (determined by the cache_version parameter and task signature) at a time. Using this mechanism, Flyte ensures that during multiple concurrent executions of a task only a single instance is evaluated and all others wait until completion and reuse the resulting cached outputs. + +Ensuring serialized evaluation requires a small degree of overhead to coordinate executions using a lightweight artifact reservation system. Therefore, this should be viewed as an extension to rather than a replacement for non-serialized cacheable tasks. It is particularly well fit for long running or otherwise computationally expensive tasks executed in scenarios similar to the following examples: + +- Periodically scheduled workflow where a single task evaluation duration may span multiple scheduled executions. +- Running a commonly shared task within different workflows (which receive the same inputs). + ++++ {"lines_to_next_cell": 0} + +For any {py:func}`flytekit.task` in Flyte, there is always one required import, which is: + +```{code-cell} +from flytekit import task +``` + ++++ {"lines_to_next_cell": 0} + +Task cache serializing is disabled by default to avoid unexpected behavior for task executions. To enable use the `cache_serialize` parameter. +`cache_serialize` is a switch to enable or disable serialization of the task +This operation is only useful for cacheable tasks, where one may reuse output from a previous execution. Flyte requires implicitly enabling the `cache` parameter on all cache serializable tasks. +Cache key definitions follow the same rules as non-serialized cache tasks. It is important to understand the implications of the task signature and `cache_version` parameter in defining cached results. + +```{code-cell} +:lines_to_next_cell: 2 + +@task(cache=True, cache_serialize=True, cache_version="1.0") +def square(n: int) -> int: + """ + Parameters: + n (int): name of the parameter for the task will be derived from the name of the input variable. + The type will be automatically deduced to Types.Integer + + Return: + int: The label for the output will be automatically assigned, and the type will be deduced from the annotation + + """ + return n * n +``` + +In the above example calling `square(n=2)` multiple times concurrently (even in different executions or workflows) will only execute the multiplication operation once. +Concurrently evaluated tasks will wait for completion of the first instance before reusing the cached results and subsequent evaluations will instantly reuse existing cache results. + ++++ + +## How does serializing caches work? + +The cache serialize paradigm introduces a new artifact reservation system. Tasks may use this reservation system to acquire an artifact reservation, indicating that they are actively evaluating the task, and release the reservation, once the execution is completed. Flyte uses a clock-skew algorithm to define reservation timeouts. Therefore, tasks are required to periodically extend the reservation during execution. + +The first execution of a serializable cached task will successfully acquire the artifact reservation. Execution will be performed as usual and upon completion, the results are written to the cache and reservation is released. Concurrently executed task instances (i.e. in parallel with the initial execution) will observe an active reservation, in which case the execution will wait until the next reevaluation and perform another check. Once the initial execution completes it will reuse the cached results. Subsequently executed task instances (i.e. after an execution has already completed successfully) will immediately reuse the existing cached results. + +Flyte handles task execution failures using a timeout on the reservation. If the task currently holding the reservation fails to extend it before it times out, another task may acquire the reservation and begin executing the task. diff --git a/docs/user_guide/development_lifecycle/caching.md b/docs/user_guide/development_lifecycle/caching.md new file mode 100644 index 0000000000..b0cc202cd9 --- /dev/null +++ b/docs/user_guide/development_lifecycle/caching.md @@ -0,0 +1,240 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Caching + +```{eval-rst} +.. tags:: Basic +``` + +Flyte provides the ability to cache the output of task executions to make the subsequent executions faster. A well-behaved Flyte task should generate deterministic output given the same inputs and task functionality. + +Task caching is useful when a user knows that many executions with the same inputs may occur. For example, consider the following scenarios: + +- Running a task periodically on a schedule +- Running the code multiple times when debugging workflows +- Running the commonly shared tasks amongst different workflows, which receive the same inputs + +Let's watch a brief explanation of caching and a demo in this video, followed by how task caching can be enabled . + +```{eval-rst} +.. youtube:: WNkThCp-gqo + +``` + ++++ {"lines_to_next_cell": 0} + +Import the necessary libraries. + +```{code-cell} +import time + +import pandas +``` + ++++ {"lines_to_next_cell": 0} + +For any {py:func}`flytekit.task` in Flyte, there is always one required import, which is: + +```{code-cell} +:lines_to_next_cell: 1 + +from flytekit import HashMethod, task, workflow +from flytekit.core.node_creation import create_node +from typing_extensions import Annotated +``` + +Task caching is disabled by default to avoid unintended consequences of caching tasks with side effects. To enable caching and control its behavior, use the `cache` and `cache_version` parameters when constructing a task. +`cache` is a switch to enable or disable the cache, and `cache_version` pertains to the version of the cache. +`cache_version` field indicates that the task functionality has changed. +Bumping the `cache_version` is akin to invalidating the cache. +You can manually update this version and Flyte caches the next execution instead of relying on the old cache. + +```{code-cell} +@task(cache=True, cache_version="1.0") # noqa: F841 +def square(n: int) -> int: + """ + Parameters: + n (int): name of the parameter for the task will be derived from the name of the input variable. + The type will be automatically deduced to ``Types.Integer``. + + Return: + int: The label for the output will be automatically assigned, and the type will be deduced from the annotation. + + """ + return n * n +``` + +In the above example, calling `square(n=2)` twice (even if it's across different executions or different workflows) will only execute the multiplication operation once. +The next time, the output will be made available immediately since it is captured from the previous execution with the same inputs. + ++++ + +If in a subsequent code update, you update the signature of the task to return the original number along with the result, it'll automatically invalidate the cache (even though the cache version remains the same). + +```python +@task(cache=True, cache_version="1.0") +def square(n: int) -> Tuple[int, int]: + ... +``` + ++++ + +:::{note} +If the user changes the task interface in any way (such as adding, removing, or editing inputs/outputs), Flyte treats that as a task functionality change. In the subsequent execution, Flyte runs the task and stores the outputs as newly cached values. +::: + +## How does caching work? + +Caching is implemented differently depending on the mode the user is running, i.e. whether they are running locally or using remote Flyte. + +### How does remote caching work? + +The cache keys for remote task execution are composed of **Project**, **Domain**, **Cache Version**, **Task Signature**, and **Inputs** associated with the execution of the task, as per the following definitions: + +- **Project:** A task run under one project cannot use the cached task execution from another project which would cause inadvertent results between project teams that could result in data corruption. +- **Domain:** To separate test, staging, and production data, task executions are not shared across these environments. +- **Cache Version:** When task functionality changes, you can change the `cache_version` of the task. Flyte will know not to use older cached task executions and create a new cache entry on the subsequent execution. +- **Task Signature:** The cache is specific to the task signature associated with the execution. The signature constitutes the task name, input parameter names/types, and the output parameter name/type. +- **Task Input Values:** A well-formed Flyte task always produces deterministic outputs. This means, given a set of input values, every execution should have identical outputs. When task execution is cached, the input values are part of the cache key. + +The remote cache for a particular task is invalidated in two ways: + +1. Modifying the `cache_version`; +2. Updating the task signature. + +:::{note} +Task executions can be cached across different versions of the task because a change in SHA does not necessarily mean that it correlates to a change in the task functionality. +::: + +### How does local caching work? + +The flytekit package uses the [diskcache](https://github.com/grantjenks/python-diskcache) package, specifically [diskcache.Cache](http://www.grantjenks.com/docs/diskcache/tutorial.html#cache), to aid in the memoization of task executions. The results of local task executions are stored under `~/.flyte/local-cache/` and cache keys are composed of **Cache Version**, **Task Signature**, and **Task Input Values**. + +Similar to the remote case, a local cache entry for a task will be invalidated if either the `cache_version` or the task signature is modified. In addition, the local cache can also be emptied by running the following command: `pyflyte local-cache clear`, which essentially obliterates the contents of the `~/.flyte/local-cache/` directory. + +:::{note} +The format used by the store is opaque and not meant to be inspectable. +::: + +(cache-offloaded-objects)= + +## Caching of non-Flyte offloaded objects + +The default behavior displayed by Flyte's memoization feature might not match the user intuition. For example, this code makes use of pandas dataframes: + +```{code-cell} +@task +def foo(a: int, b: str) -> pandas.DataFrame: + df = pandas.DataFrame(...) + ... + return df + + +@task(cache=True, cache_version="1.0") +def bar(df: pandas.DataFrame) -> int: + ... + + +@workflow +def wf(a: int, b: str): + df = foo(a=a, b=b) + v = bar(df=df) # noqa: F841 +``` + +If run twice with the same inputs, one would expect that `bar` would trigger a cache hit, but it turns out that's not the case because of how dataframes are represented in Flyte. +However, with release 1.2.0, Flyte provides a new way to control memoization behavior of literals. This is done via a `typing.Annotated` call on the task signature. +For example, in order to cache the result of calls to `bar`, you can rewrite the code above like this: + +```{code-cell} +def hash_pandas_dataframe(df: pandas.DataFrame) -> str: + return str(pandas.util.hash_pandas_object(df)) + + +@task +def foo_1( # noqa: F811 + a: int, b: str # noqa: F821 +) -> Annotated[pandas.DataFrame, HashMethod(hash_pandas_dataframe)]: # noqa: F821 # noqa: F821 + df = pandas.DataFrame(...) # noqa: F821 + ... + return df + + +@task(cache=True, cache_version="1.0") # noqa: F811 +def bar_1(df: pandas.DataFrame) -> int: # noqa: F811 + ... # noqa: F811 + + +@workflow +def wf_1(a: int, b: str): # noqa: F811 + df = foo(a=a, b=b) # noqa: F811 + v = bar(df=df) # noqa: F841 +``` + +Note how the output of task `foo` is annotated with an object of type `HashMethod`. Essentially, it represents a function that produces a hash that is used as part of the cache key calculation in calling the task `bar`. + +### How does caching of offloaded objects work? + +Recall how task input values are taken into account to derive a cache key. +This is done by turning the literal representation into a string and using that string as part of the cache key. In the case of dataframes annotated with `HashMethod` we use the hash as the representation of the Literal. In other words, the literal hash is used in the cache key. + +This feature also works in local execution. + ++++ + +Here's a complete example of the feature: + +```{code-cell} +def hash_pandas_dataframe(df: pandas.DataFrame) -> str: + return str(pandas.util.hash_pandas_object(df)) + + +@task +def uncached_data_reading_task() -> Annotated[pandas.DataFrame, HashMethod(hash_pandas_dataframe)]: + return pandas.DataFrame({"column_1": [1, 2, 3]}) + + +@task(cache=True, cache_version="1.0") +def cached_data_processing_task(df: pandas.DataFrame) -> pandas.DataFrame: + time.sleep(1) + return df * 2 + + +@task +def compare_dataframes(df1: pandas.DataFrame, df2: pandas.DataFrame): + assert df1.equals(df2) + + +@workflow +def cached_dataframe_wf(): + raw_data = uncached_data_reading_task() + + # Execute `cached_data_processing_task` twice, but force those + # two executions to happen serially to demonstrate how the second run + # hits the cache. + t1_node = create_node(cached_data_processing_task, df=raw_data) + t2_node = create_node(cached_data_processing_task, df=raw_data) + t1_node >> t2_node + + # Confirm that the dataframes actually match + compare_dataframes(df1=t1_node.o0, df2=t2_node.o0) + + +if __name__ == "__main__": + df1 = cached_dataframe_wf() + print(f"Running cached_dataframe_wf once : {df1}") +``` diff --git a/docs/user_guide/development_lifecycle/creating_a_new_project.md b/docs/user_guide/development_lifecycle/creating_a_new_project.md new file mode 100644 index 0000000000..7741b810d2 --- /dev/null +++ b/docs/user_guide/development_lifecycle/creating_a_new_project.md @@ -0,0 +1,28 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Creating a new project + +Creates project to be used as a home for the flyte resources of tasks and workflows. +Refer to the [flytectl API reference](https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_project.html) +for more details. + +```{eval-rst} +.. prompt:: bash + + flytectl create project --id "my-flyte-project-name" --labels "my-label=my-project-label" --description "my-flyte-project-name" --name "my-flyte-project-name" +``` diff --git a/docs/user_guide/development_lifecycle/debugging_executions.md b/docs/user_guide/development_lifecycle/debugging_executions.md new file mode 100644 index 0000000000..7c8f9562d3 --- /dev/null +++ b/docs/user_guide/development_lifecycle/debugging_executions.md @@ -0,0 +1,46 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Debugging executions + +The inspection of task and workflow execution would provide you log links to debug things further + +Using `--details` flag would shows you node executions view with log links. + +``` +└── n1 - FAILED - 2021-06-30 08:51:07.3111846 +0000 UTC - 2021-06-30 08:51:17.192852 +0000 UTC +│ ├── Attempt :0 +│ └── Task - FAILED - 2021-06-30 08:51:07.3111846 +0000 UTC - 2021-06-30 08:51:17.192852 +0000 UTC +│ └── Logs : +│ └── Name :Kubernetes Logs (User) +│ └── URI :http://localhost:30082/#/log/flytectldemo-development/f3a5a4034960f4aa1a09-n1-0/pod?namespace=flytectldemo-development +``` + +Additionally you can check the pods launched by flyte in \-\ namespace + +``` +kubectl get pods -n - +``` + +The launched pods will have a prefix of execution name along with suffix of nodeId + +``` +NAME READY STATUS RESTARTS AGE +f65009af77f284e50959-n0-0 0/1 ErrImagePull 0 18h +``` + +So here the investigation can move ahead by describing the pod and checking the issue with Image pull. diff --git a/docs/user_guide/development_lifecycle/decks.md b/docs/user_guide/development_lifecycle/decks.md new file mode 100644 index 0000000000..5aae4955b1 --- /dev/null +++ b/docs/user_guide/development_lifecycle/decks.md @@ -0,0 +1,316 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(decks)= + +# Decks + +```{eval-rst} +.. tags:: UI, Intermediate +``` + +The Decks feature enables you to obtain customizable and default visibility into your tasks. +Think of it as a visualization tool that you can utilize within your Flyte tasks. + +Decks are equipped with a variety of {ref}`renderers `, +such as FrameRenderer and MarkdownRenderer. These renderers produce HTML files. +As an example, FrameRenderer transforms a DataFrame into an HTML table, and MarkdownRenderer converts Markdown text into HTML. + +Each task has a minimum of three decks: input, output and default. +The input/output decks are used to render the input/output data of tasks, +while the default deck can be used to render line plots, scatter plots or Markdown text. +Additionally, you can create new decks to render your data using custom renderers. + +:::{note} +Flyte Decks is an opt-in feature; to enable it, set `enable_deck` to `True` in the task parameters. +::: + +To begin, import the dependencies. + +```{code-cell} +import flytekit +from flytekit import ImageSpec, task +from flytekitplugins.deck.renderer import MarkdownRenderer +from sklearn.decomposition import PCA +``` + ++++ {"lines_to_next_cell": 0} + +We create a new deck named `pca` and render Markdown content along with a +[PCA](https://en.wikipedia.org/wiki/Principal_component_analysis) plot. + +You can begin by initializing an {ref}`ImageSpec ` object to encompass all the necessary dependencies. +This approach automatically triggers a Docker build, alleviating the need for you to manually create a Docker image. + +```{code-cell} +custom_image = ImageSpec(name="flyte-decks-example", packages=["plotly"], registry="ghcr.io/flyteorg") + +if custom_image.is_container(): + import plotly + import plotly.express as px +``` + ++++ {"lines_to_next_cell": 0} + +:::{important} +Replace `ghcr.io/flyteorg` with a container registry you've access to publish to. +To upload the image to the local registry in the demo cluster, indicate the registry as `localhost:30000`. +::: + +Note the usage of `append` to append the Plotly deck to the Markdown deck. + +```{code-cell} +@task(enable_deck=True, container_image=custom_image) +def pca_plot(): + iris_df = px.data.iris() + X = iris_df[["sepal_length", "sepal_width", "petal_length", "petal_width"]] + pca = PCA(n_components=3) + components = pca.fit_transform(X) + total_var = pca.explained_variance_ratio_.sum() * 100 + fig = px.scatter_3d( + components, + x=0, + y=1, + z=2, + color=iris_df["species"], + title=f"Total Explained Variance: {total_var:.2f}%", + labels={"0": "PC 1", "1": "PC 2", "2": "PC 3"}, + ) + main_deck = flytekit.Deck("pca", MarkdownRenderer().to_html("### Principal Component Analysis")) + main_deck.append(plotly.io.to_html(fig)) +``` + ++++ {"lines_to_next_cell": 0} + +:::{Important} +To view the log output locally, the `FLYTE_SDK_LOGGING_LEVEL` environment variable should be set to 20. +::: + +The following is the expected output containing the path to the `deck.html` file: + +``` +{"asctime": "2023-07-11 13:16:04,558", "name": "flytekit", "levelname": "INFO", "message": "pca_plot task creates flyte deck html to file:///var/folders/6f/xcgm46ds59j7g__gfxmkgdf80000gn/T/flyte-0_8qfjdd/sandbox/local_flytekit/c085853af5a175edb17b11cd338cbd61/deck.html"} +``` + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_deck_plot_local.webp +:alt: Flyte deck plot +:class: with-shadow +::: + +Once you execute this task on the Flyte cluster, you can access the deck by clicking the _Flyte Deck_ button: + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_deck_button.png +:alt: Flyte deck button +:class: with-shadow +::: + +(deck_renderer)= + +## Deck renderer + +The Deck renderer is an integral component of the Deck plugin, which offers both default and personalized task visibility. +Within the Deck, an array of renderers is present, responsible for generating HTML files. + +These renderers showcase HTML in the user interface, facilitating the visualization and documentation of task-associated data. + +In the Flyte context, a collection of deck objects is stored. +When the task connected with a deck object is executed, these objects employ renderers to transform data into HTML files. + +### Available renderers + +#### Frame renderer + +Creates a profile report from a Pandas DataFrame. + +```{code-cell} +import pandas as pd +from flytekitplugins.deck.renderer import FrameProfilingRenderer + + +@task(enable_deck=True) +def frame_renderer() -> None: + df = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]}) + flytekit.Deck("Frame Renderer", FrameProfilingRenderer().to_html(df=df)) +``` + ++++ {"lines_to_next_cell": 0} + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_frame_renderer.png +:alt: Frame renderer +:class: with-shadow +::: + ++++ {"lines_to_next_cell": 0} + +#### Top-frame renderer + +Renders DataFrame as an HTML table. +This renderer doesn't necessitate plugin installation since it's accessible within the flytekit library. + +```{code-cell} +from typing import Annotated + +from flytekit.deck import TopFrameRenderer + + +@task(enable_deck=True) +def top_frame_renderer() -> Annotated[pd.DataFrame, TopFrameRenderer(1)]: + return pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]}) +``` + ++++ {"lines_to_next_cell": 0} + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_top_frame_renderer.png +:alt: Top frame renderer +:class: with-shadow +::: + +#### Markdown renderer + +Converts a Markdown string into HTML, producing HTML as a Unicode string. + +```{code-cell} +@task(enable_deck=True) +def markdown_renderer() -> None: + flytekit.current_context().default_deck.append( + MarkdownRenderer().to_html("You can install flytekit using this command: ```import flytekit```") + ) +``` + ++++ {"lines_to_next_cell": 0} + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_markdown_renderer.png +:alt: Markdown renderer +:class: with-shadow +::: + +#### Box renderer + +Groups rows of DataFrame together into a +box-and-whisker mark to visualize their distribution. + +Each box extends from the first quartile (Q1) to the third quartile (Q3). +The median (Q2) is indicated by a line within the box. +Typically, the whiskers extend to the edges of the box, +plus or minus 1.5 times the interquartile range (IQR: Q3-Q1). + +```{code-cell} +from flytekitplugins.deck.renderer import BoxRenderer + + +@task(enable_deck=True) +def box_renderer() -> None: + iris_df = px.data.iris() + flytekit.Deck("Box Plot", BoxRenderer("sepal_length").to_html(iris_df)) +``` + ++++ {"lines_to_next_cell": 0} + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_box_renderer.png +:alt: Box renderer +:class: with-shadow +::: + +#### Image renderer + +Converts a {ref}`FlyteFile ` or `PIL.Image.Image` object into an HTML string, +where the image data is encoded as a base64 string. + +```{code-cell} +from flytekit import workflow +from flytekit.types.file import FlyteFile +from flytekitplugins.deck.renderer import ImageRenderer + + +@task(enable_deck=True) +def image_renderer(image: FlyteFile) -> None: + flytekit.Deck("Image Renderer", ImageRenderer().to_html(image_src=image)) + + +@workflow +def image_renderer_wf( + image: FlyteFile = "https://bit.ly/3KZ95q4", +) -> None: + image_renderer(image=image) +``` + ++++ {"lines_to_next_cell": 0} + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_image_renderer.png +:alt: Image renderer +:class: with-shadow +::: + +#### Table renderer + +Converts a Pandas dataframe into an HTML table. + +```{code-cell} +from flytekitplugins.deck.renderer import TableRenderer + + +@task(enable_deck=True) +def table_renderer() -> None: + flytekit.Deck( + "Table Renderer", + TableRenderer().to_html(df=pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]}), table_width=50), + ) +``` + ++++ {"lines_to_next_cell": 0} + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_table_renderer.png +:alt: Table renderer +:class: with-shadow +::: + +#### Source code renderer + +Converts source code to HTML and renders it as a Unicode string on the deck. + +```{code-cell} +:lines_to_next_cell: 2 + +import inspect + +from flytekitplugins.deck.renderer import SourceCodeRenderer + + +@task(enable_deck=True) +def source_code_renderer() -> None: + file_path = inspect.getsourcefile(frame_renderer.__wrapped__) + with open(file_path, "r") as f: + source_code = f.read() + flytekit.Deck( + "Source Code Renderer", + SourceCodeRenderer().to_html(source_code), + ) +``` + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_source_code_renderer.png +:alt: Source code renderer +:class: with-shadow +::: + +### Contribute to renderers + +Don't hesitate to integrate a new renderer into +[renderer.py](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-deck-standard/flytekitplugins/deck/renderer.py) +if your deck renderers can enhance data visibility. +Feel encouraged to open a pull request and play a part in enhancing the Flyte deck renderer ecosystem! diff --git a/docs/user_guide/development_lifecycle/failure_node.md b/docs/user_guide/development_lifecycle/failure_node.md new file mode 100644 index 0000000000..61756e4a8a --- /dev/null +++ b/docs/user_guide/development_lifecycle/failure_node.md @@ -0,0 +1,89 @@ +(failure_node)= +# Failure node + +```{eval-rst} + .. tags:: FailureNode, Intermediate +``` + +The failure node feature enables you to designate a specific node to execute in the event of a failure within your workflow. + +For example, a workflow involves creating a cluster at the beginning, followed by the execution of tasks, and concludes with the deletion of the cluster once all tasks are completed. However, if any task within the workflow encounters an error, flyte will abort the entire workflow and won’t delete the cluster. This poses a challenge if you still need to clean up the cluster even in a task failure. + +To address this issue, you can add a failure node into your workflow. This ensures that critical actions, such as deleting the cluster, are executed even in the event of failures occurring throughout the workflow execution: + +```python +from flytekit import WorkflowFailurePolicy, task, workflow + + +@task +def create_cluster(name: str): + print(f"Creating cluster: {name}") + +``` + +Create a task that will fail during execution: + +```python +@task +def t1(a: int, b: str): + print(f"{a} {b}") + raise ValueError("Fail!") + + +@task +def delete_cluster(name: str): + print(f"Deleting cluster {name}") +``` + +Create a task that will be executed if any of the tasks in the workflow fail: + +```python +@task +def clean_up(name: str): + print(f"Cleaning up cluster {name}") + +``` + +Specify the `on_failure` to a cleanup task. This task will be executed if any of the tasks in the workflow fail: + + +:::{note} +The input of `clean_up` should be the exact same as the input of the workflow. +::: + +```python +@workflow(on_failure=clean_up) +def subwf(name: str): + c = create_cluster(name=name) + t = t1(a=1, b="2") + d = delete_cluster(name=name) + c >> t >> d +``` + +By setting the failure policy to `FAIL_AFTER_EXECUTABLE_NODES_COMPLETE` to ensure that the `wf1` is executed even if the subworkflow fails. In this case, both parent and child workflows will fail, resulting in the `clean_up` task being executed twice: + +```python +@workflow(on_failure=clean_up, failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE) +def wf1(name: str = "my_cluster"): + c = create_cluster(name=name) + subwf(name="another_cluster") + t = t1(a=1, b="2") + d = delete_cluster(name=name) + c >> t >> d + + +@workflow +def clean_up_wf(name: str): + return clean_up(name=name) +``` + +You can also set the `on_failure` to a workflow. This workflow will be executed if any of the tasks in the workflow fail: + +```python +@workflow(on_failure=clean_up_wf) +def wf2(name: str = "my_cluster"): + c = create_cluster(name=name) + t = t1(a=1, b="2") + d = delete_cluster(name=name) + c >> t >> d +``` diff --git a/docs/user_guide/development_lifecycle/index.md b/docs/user_guide/development_lifecycle/index.md new file mode 100644 index 0000000000..8c21abc291 --- /dev/null +++ b/docs/user_guide/development_lifecycle/index.md @@ -0,0 +1,23 @@ +# Development lifecycle + +In this section, you will discover Flyte's features that aid in local workflow development. +You will gain an understanding of concepts like caching, the Flyte remote API, Agents, Decks and more. + +```{toctree} +:maxdepth: -1 +:name: development_lifecycle_toc +:hidden: + +agents +private_images +caching +cache_serializing +decks +failure_node +creating_a_new_project +running_tasks +running_workflows +running_launch_plans +inspecting_executions +debugging_executions +``` diff --git a/docs/user_guide/development_lifecycle/inspecting_executions.md b/docs/user_guide/development_lifecycle/inspecting_executions.md new file mode 100644 index 0000000000..1ce09ae155 --- /dev/null +++ b/docs/user_guide/development_lifecycle/inspecting_executions.md @@ -0,0 +1,83 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Inspecting executions + +## Flytectl + +Flytectl supports inspecting execution by retrieving its details. For a deeper dive, refer to the +[API reference](https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_get_execution.html) guide. + +Monitor the execution by providing the execution id from create command which can be task or workflow execution. + +``` +flytectl get execution -p flytesnacks -d development +``` + +For more details use `--details` flag which shows node executions along with task executions on them. + +``` +flytectl get execution -p flytesnacks -d development --details +``` + +If you prefer to see yaml/json view for the details then change the output format using the -o flag. + +``` +flytectl get execution -p flytesnacks -d development --details -o yaml +``` + +To see the results of the execution you can inspect the node closure outputUri in detailed yaml output. + +``` +"outputUri": "s3://my-s3-bucket/metadata/propeller/flytesnacks-development-/n0/data/0/outputs.pb" +``` + +## FlyteRemote + +With FlyteRemote, you can fetch the inputs and outputs of executions and inspect them. + +```python +from flytekit.remote import FlyteRemote + +# FlyteRemote object is the main entrypoint to API +remote = FlyteRemote( + config=Config.for_endpoint(endpoint="flyte.example.net"), + default_project="flytesnacks", + default_domain="development", +) + +execution = remote.fetch_execution( + name="fb22e306a0d91e1c6000", project="flytesnacks", domain="development" +) + +input_keys = execution.inputs.keys() +output_keys = execution.outputs.keys() + +# The inputs and outputs correspond to the top-level execution or the workflow itself. +# To fetch a specific output, say, a model file: +model_file = execution.outputs["model_file"] +with open(model_file) as f: + ... + +# You can use FlyteRemote.sync() to sync the entity object's state with the remote state during the execution run. +synced_execution = remote.sync(execution, sync_nodes=True) +node_keys = synced_execution.node_executions.keys() + +# node_executions will fetch all the underlying node executions recursively. +# To fetch output of a specific node execution: +node_execution_output = synced_execution.node_executions["n1"].outputs["model_file"] +``` diff --git a/docs/user_guide/development_lifecycle/private_images.md b/docs/user_guide/development_lifecycle/private_images.md new file mode 100644 index 0000000000..5ebd41ea73 --- /dev/null +++ b/docs/user_guide/development_lifecycle/private_images.md @@ -0,0 +1,72 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(private_images)= + +# Private images + +As we learned in the {ref}`Flyte Fundamentals ` guide, +Flyte uses OCI-compatible containers to package up your code and third-party +dependencies. For production use-cases your images may require proprietary code +and configuration that you want to keep private. + +You can use different private container registries to host your images, such as +[AWS ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html), +[Docker Hub](https://docs.docker.com/docker-hub/repos/#private-repositories), +[GitLab Container Registry](https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#access-an-image-from-a-private-container-registry), +and [GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry). + +To pull private images, ensure that you have the command line tools and login +information associated with the registry. + +## Create a secret + +First [create a secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) +that contains all the credentials needed to log into the registry. + +## Configure `imagePullSecrets` + +Then, you'll need to specify a `imagePullSecrets` configuration to pull a +private image using one of two methods below. + +```{eval-rst} +.. tabs:: + + .. tab:: Service Account + + You can use the default or new service account for this option: + + 1. Add your ``imagePullSecrets`` configuration to the + `service account `__. + 2. Use this service account to log into the private registry and pull the image. + 3. When you create a task/workflow execution this service account should + be specified to access the private image. + + .. tab:: Custom Pod Template + + This option uses a `custom pod template `__ + to create a pod. This template is automatically added to every ``pod`` that + Flyte creates. + + 1. Add your ``imagePullSecrets`` configuration to this custom pod template. + 2. Update `FlytePropeller `__ about the pod created in the previous step. + 3. FlytePropeller adds ``imagePullSecrets``, along with other customization for the pod, + to the PodSpec, which should look similar to this + `manifest `__. + 4. The pods with their keys can log in and access the images in the private registry. + Once you set up the token to authenticate with the private registry, you can pull images from them. +``` diff --git a/docs/user_guide/development_lifecycle/running_launch_plans.md b/docs/user_guide/development_lifecycle/running_launch_plans.md new file mode 100644 index 0000000000..1fb8bb4c2c --- /dev/null +++ b/docs/user_guide/development_lifecycle/running_launch_plans.md @@ -0,0 +1,84 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(remote_launchplan)= + +# Running launch plans + +## Flytectl + +This is multi-steps process where we create an execution spec file, update the spec file and then create the execution. +More details can be found [here](https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_execution.html). + +**Generate an execution spec file** + +``` +flytectl get launchplan -p flytesnacks -d development myapp.workflows.example.my_wf --execFile exec_spec.yaml +``` + +**Update the input spec file for arguments to the workflow** + +``` +.... +inputs: + name: "adam" +.... +``` + +**Create execution using the exec spec file** + +``` +flytectl create execution -p flytesnacks -d development --execFile exec_spec.yaml +``` + +**Monitor the execution by providing the execution id from create command** + +``` +flytectl get execution -p flytesnacks -d development +``` + +## FlyteRemote + +A launch plan can be launched via FlyteRemote programmatically. + +```python +from flytekit.remote import FlyteRemote +from flytekit.configuration import Config +from flytekit import LaunchPlan + +# FlyteRemote object is the main entrypoint to API +remote = FlyteRemote( + config=Config.for_endpoint(endpoint="flyte.example.net"), + default_project="flytesnacks", + default_domain="development", +) + +# Fetch launch plan +flyte_lp = remote.fetch_launch_plan( + name="workflows.example.wf", version="v1", project="flytesnacks", domain="development" +) + +# Execute +execution = remote.execute( + flyte_lp, inputs={"mean": 1}, execution_name="lp-execution", wait=True +) + +# Or use execution_name_prefix to avoid repeated execution names +execution = remote.execute( + flyte_lp, inputs={"mean": 1}, execution_name_prefix="flyte", wait=True +) +``` diff --git a/docs/user_guide/development_lifecycle/running_tasks.md b/docs/user_guide/development_lifecycle/running_tasks.md new file mode 100644 index 0000000000..882380109d --- /dev/null +++ b/docs/user_guide/development_lifecycle/running_tasks.md @@ -0,0 +1,98 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(remote_task)= + +# Running tasks + +## Flytectl + +This is a multi-step process where we create an execution spec file, update the spec file, and then create the execution. +More details can be found in the [Flytectl API reference](https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_execution.html). + +**Generate execution spec file** + +``` +flytectl get tasks -d development -p flytesnacks workflows.example.generate_normal_df --latest --execFile exec_spec.yaml +``` + +**Update the input spec file for arguments to the workflow** + +``` +iamRoleARN: 'arn:aws:iam::12345678:role/defaultrole' +inputs: + n: 200 + mean: 0.0 + sigma: 1.0 +kubeServiceAcct: "" +targetDomain: "" +targetProject: "" +task: workflows.example.generate_normal_df +version: "v1" +``` + +**Create execution using the exec spec file** + +``` +flytectl create execution -p flytesnacks -d development --execFile exec_spec.yaml +``` + +**Monitor the execution by providing the execution id from create command** + +``` +flytectl get execution -p flytesnacks -d development +``` + +## FlyteRemote + +A task can be launched via FlyteRemote programmatically. + +```python +from flytekit.remote import FlyteRemote +from flytekit.configuration import Config, SerializationSettings + +# FlyteRemote object is the main entrypoint to API +remote = FlyteRemote( + config=Config.for_endpoint(endpoint="flyte.example.net"), + default_project="flytesnacks", + default_domain="development", +) + +# Get Task +flyte_task = remote.fetch_task(name="workflows.example.generate_normal_df", version="v1") + +flyte_task = remote.register_task( + entity=flyte_task, + serialization_settings=SerializationSettings(image_config=None), + version="v2", +) + +# Run Task +execution = remote.execute( + flyte_task, inputs={"n": 200, "mean": 0.0, "sigma": 1.0}, execution_name="task-execution", wait=True +) + +# Or use execution_name_prefix to avoid repeated execution names +execution = remote.execute( + flyte_task, inputs={"n": 200, "mean": 0.0, "sigma": 1.0}, execution_name_prefix="flyte", wait=True +) + +# Inspecting execution +# The 'inputs' and 'outputs' correspond to the task execution. +input_keys = execution.inputs.keys() +output_keys = execution.outputs.keys() +``` diff --git a/docs/user_guide/development_lifecycle/running_workflows.md b/docs/user_guide/development_lifecycle/running_workflows.md new file mode 100644 index 0000000000..2e04714adc --- /dev/null +++ b/docs/user_guide/development_lifecycle/running_workflows.md @@ -0,0 +1,59 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Running workflows + +Workflows on their own are not runnable directly. However, a launchplan is always bound to a workflow and you can use +launchplans to **launch** a workflow. For cases in which you want the launchplan to have the same arguments as a workflow, +if you are using one of the SDK's to author your workflows - like flytekit, flytekit-java etc, then they should +automatically create a `default launchplan` for the workflow. + +A `default launchplan` has the same name as the workflow and all argument defaults are similar. See +{ref}`Launch Plans` to run a workflow via the default launchplan. + +{ref}`Tasks also can be executed ` using the launch command. +One difference between running a task and a workflow via launchplans is that launchplans cannot be associated with a +task. This is to avoid triggers and scheduling. + +## FlyteRemote + +Workflows can be executed with FlyteRemote because under the hood it fetches and triggers a default launch plan. + +```python +from flytekit.remote import FlyteRemote +from flytekit.configuration import Config + +# FlyteRemote object is the main entrypoint to API +remote = FlyteRemote( + config=Config.for_endpoint(endpoint="flyte.example.net"), + default_project="flytesnacks", + default_domain="development", +) + +# Fetch workflow +flyte_workflow = remote.fetch_workflow(name="workflows.example.wf", version="v1") + +# Execute +execution = remote.execute( + flyte_workflow, inputs={"mean": 1}, execution_name="workflow-execution", wait=True +) + +# Or use execution_name_prefix to avoid repeated execution names +execution = remote.execute( + flyte_workflow, inputs={"mean": 1}, execution_name_prefix="flyte", wait=True +) +``` diff --git a/docs/user_guide/environment_setup.md b/docs/user_guide/environment_setup.md new file mode 100644 index 0000000000..1fc41e749b --- /dev/null +++ b/docs/user_guide/environment_setup.md @@ -0,0 +1,245 @@ +(env_setup)= + +# Environment setup + +## Prerequisites + +- Make sure you have [docker](https://docs.docker.com/get-docker/) and [git](https://git-scm.com/) installed. +- Install {doc}`flytectl `, the commandline interface for Flyte. + +## Repo setup + +As we intend to execute the code locally, duplicate this code block into `hello_world.py`. + +```python +from flytekit import task, workflow + +@task +def say_hello() -> str: + return "Hello, World!" + +@workflow +def hello_world_wf() -> str: + res = say_hello() + return res + +if __name__ == "__main__": + print(f"Running hello_world_wf() {hello_world_wf()}") +``` + +To install `flytekit`, run the following command: + +``` +pip install flytekit +``` + +:::{tip} +**Recommended**: Create a new python virtual environment to make sure it doesn't interfere with your +development environment. You can do this by running the following commands in your terminal: + +```{prompt} bash +python -m venv ~/venvs/flyte-examples +source ~/venvs/flyte-examples/bin/activate +``` + +::: + +To make sure everything is working in your virtual environment, run `hello_world.py` locally: + +```{prompt} bash +python hello_world.py +``` + +Expected output: + +```{prompt} +Running hello_world_wf() Hello, World! +``` + +## Create a local demo Flyte cluster + +```{important} +Make sure the Docker daemon is running before starting the demo cluster. +``` + +Use `flytectl` to start a demo Flyte cluster: + +```{prompt} bash +flytectl demo start +``` + +After this completes, be sure to export the Flyte config as it will be essential later. Run the command in the output that looks like this: + +```{prompt} bash +export FLYTECTL_CONFIG= ~//.flyte/config-sandbox.yaml +``` + +## Running workflows + +Now you can run the example workflow locally using the default Docker image bundled with `flytekit`: + +```{prompt} bash +pyflyte run hello_world.py hello_world_wf +``` + +:::{note} +The initial arguments of `pyflyte run` take the form of +`path/to/script.py `, where `` +refers to the function decorated with `@task` or `@workflow` that you wish to run. +::: + +To run the workflow on the demo Flyte cluster, all you need to do is supply the `--remote` flag: + +``` +pyflyte run --remote hello_world.py hello_world_wf +``` + +You can also run the code directly from a remote source: + +``` +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py \ + hello_world_wf +``` + +You should see an output that looks like: + +```{prompt} +Go to https:///console/projects/flytesnacks/domains/development/executions/ to see execution in the console. +``` + +You can visit this URL to inspect the execution as it runs: + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/common/first_run_console.gif +:alt: A quick visual tour for launching your first Workflow. +::: + +Finally, run a workflow that takes some inputs, for example the `workflow.py` example: + +```{prompt} bash +pyflyte run --remote \ + https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/workflow.py \ + simple_wf --x '[-3,0,3]' --y '[7,4,-2]' +``` + +:::{note} +We're passing in the workflow inputs as additional options to `pyflyte run`. In the above example, the +inputs are `--x '[-3,0,3]'` and `--y '[7,4,-2]'`. For snake-case argument names like `arg_name`, you can provide the +option as `--arg-name`. +::: + +## Visualizing workflows + +Workflows can be visualized as DAGs on the UI. +However, you can visualize workflows on the browser and in the terminal by _just_ using your terminal. + +To view workflow on the browser: + +```{prompt} bash $ +flytectl get workflows \ + --project flytesnacks \ + --domain development \ + --version \ + -o doturl \ + basics.workflow.simple_wf +``` + +To view workflow as a `strict digraph` on the command line: + +```{prompt} bash $ +flytectl get workflows \ + --project flytesnacks \ + --domain development \ + --version \ + -o dot \ + basics.workflow.simple_wf +``` + +Replace `` with the version obtained from the console UI, +which might resemble something like `BLrGKJaYsW2ME1PaoirK1g==`. + +:::{tip} +Running most of the examples in the **User guide** only requires the default Docker image that ships with Flyte. +Many examples in the {ref}`tutorials` and {ref}`integrations` section depend on additional libraries such as +`sklearn`, `pytorch` or `tensorflow`, which will not work with the default docker image used by `pyflyte run`. + +These examples will explicitly show you which images to use for running these examples by passing in the Docker +image you want to use with the `--image` option in `pyflyte run`. +::: + +🎉 Congrats! Now you can run all the examples in the {ref}`userguide` 🎉! + +## Configuring the demo cluster to use additional resources + +Depending on how resource intensive your workflows are, you may encounter errors such as +OOM (Out of Memory) errors or find pods with the status OOMKilled. +It is crucial to understand that the demo cluster is not set up to immediately accommodate +all workflow requirements, and some resource requests may be ignored based on the cluster's limits. + +:::{tip} +Keep in mind that, for production deployments, you should give careful consideration to +these configurations rather than simply setting large numbers. +::: + +Here's how you can go about modifying the configurations: + +1. Add cluster resource attributes to `cra.yaml`: + +``` +attributes: + projectQuotaCpu: "1000" + projectQuotaMemory: 5Ti +project: flytesnacks +domain: development +``` + +2. Add task resource attributes to `tra.yaml`: + +``` +defaults: + cpu: "2" + memory: 1Gi +limits: + cpu: "1000" + memory: 5Ti +project: flytesnacks +domain: development +``` + +3. Apply the two configuration files using the following commands: + +``` +$ flytectl update task-resource-attribute --attrFile tra.yaml +$ flytectl update cluster-resource-attribute --attrFile cra.yaml +``` + +4. Confirm that the configuration is applied using the following commands: + +``` +$ flytectl get task-resource-attribute -p flytesnacks -d development +{"project":"flytesnacks","domain":"development","defaults":{"cpu":"2","memory":"1Gi"},"limits":{"cpu":"1000","memory":"5Ti"}} + +$ flytectl get cluster-resource-attribute -p flytesnacks -d development +{"project":"flytesnacks","domain":"development","attributes":{"projectQuotaCpu":"1000","projectQuotaMemory":"5Ti"}} +``` + +And that's it! You have successfully modified your Flyte demo cluster to accommodate resource intensive workloads. + +For more information, refer to the +[Configuring Custom K8s Resources](https://docs.flyte.org/en/latest/deployment/configuration/general.html) guide. + +## Local registry + +If you find yourself using tasks dependent on `ImageSpec` containers built with `envd` on the demo cluster, +before you submit your workflow, you will need to inform `envd` how to push the images it builds to the cluster. +This can be done via: + +``` +envd context create --name flyte-sandbox --builder tcp --builder-address localhost:30003 --use +``` + +You will also need to update your `ImageSpec` instances to set `registry="localhost:30000"`. + +## What's next? + +Try out the examples in the {doc}`Basics ` section. diff --git a/docs/user_guide/extending/backend_plugins.md b/docs/user_guide/extending/backend_plugins.md new file mode 100644 index 0000000000..876ef8019e --- /dev/null +++ b/docs/user_guide/extending/backend_plugins.md @@ -0,0 +1,91 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +%% [markdown] +(extend-plugin-flyte-backend)= + +# Backend plugins + +```{eval-rst} +.. tags:: Extensibility, Contribute, Intermediate +``` + +This guide will take you through the why and how of writing a backend plugin for +Flyte. + +To recap, here are a few examples of why you would want to implement a backend plugin: + +1. We want to add a new capability to the Flyte Platform, for example we might want to: + - Talk to a new service like AWS Sagemaker, Snowflake, Redshift, Athena, BigQuery, etc. + - Orchestrate a set of containers in a new way like Spark, Flink, Distributed + training on Kubernetes (usually using a Kubernetes operator). + - Use a new container orchestration engine like AWS Batch/ECS, Hashicorp' Nomad + - Use a completely new runtime like AWS Lambda, KNative, etc. +3. You want to retain the capability to update the plugin implementation and roll + out new changes and fixes without affecting the users code or requiring them to update + versions of their plugins. +4. You want the same plugin to be accessible across multiple language SDK's. + +```{note} +Talking to a new service can be done using flytekit extensions and usually is the better way to get started. But, once matured, most of these extensions are better to be migrated to the backend. For the rest of the cases, it is possible to extend flytekit to achieve these scenarios, but this is less desirable, because of the associated overhead of first launching a container that launches these jobs downstream. +``` + +## Basics + +In this section we'll go through the components of a backend plugin using the {ref}`K8s Spark plugin` as a reference. A Flyte backend extension consists of 3 parts: interface +specification, `flytekit` plugin implementation, and `flytepropeller` plugin implementation. + +### Interface specification + +Usually Flyte extensions need information that is not covered by a {std:ref}`Flyte TaskTemplate `. The TaskTemplate consists of a +the interface, task_type identifier, some metadata and other fields. + +```{note} +An important field to note here is {std:ref}`custom `. The custom field is essentially an unstructured JSON. This makes it possible to extend a task-template beyond the default supported targets {std:ref}`container `. + +The motivation of the `custom`` field is to marshal a JSON structure that specifies information beyond what a regular TaskTemplate can capture. The actual structure of the JSON is known only to the implemented backend-plugin and the SDK components. The core Flyte platform, does not understand of look into the specifics of this structure. +``` + +It is highly recommended to use an interface definition language like Protobuf, OpenAPISpec etc to declare specify the structure of the JSON. From here, on we refer to this as the **Plugin Specification**. + +```{note} +For Spark we decided to use Protobuf to specify the plugin as can be seen [here](https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/plugins/spark.proto). Note it isn't necessary to have the Plugin structure specified in `flyteidl`, but we do it for simplicity, ease of maintenance alongside the core platform, and convenience leveraging existing tooling to generate code for protobuf. +``` + +### Flytekit plugin implementation + +Now that you have a specification, we have to implement a method to generate this new TaskTemplate, with the special custom field. Also, this is where the UX design comes into play. You want to write the best possible interface in the SDK that users are delighted to use. The end goal is to create the TaskTemplate with the Custom field populated with the actual JSON structure. + +We will currently refer to the Python `flytekit` SDK as an example for extending and +implementing the SDK. + +The SDK task should be implemented as an extension of {py:class}`flytekit.core.base_task.PythonTask`, or more commonly {py:class}`flytekit.PythonFunctionTask`. +In the case of Spark, we extend the {py:class}`flytekit.PythonFunctionTask`, as shown [here](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-spark/flytekitplugins/spark/task.py#L77-L123). + +The `SparkTask` is implemented as a regular flytekit plugin, with one exception: the `custom` field is now actually the `SparkJob` protocol buffer. When serializing a task, `flytekit` base classes will automatically invoke the [`get_custom` method](https://github.com/flyteorg/flytekit/blob/c02075d472b5587d199630bcfc7f9937673c6a0e/flytekit/core/base_task.py#L255). + +### FlytePropeller backend plugin + +The backend plugin is where the actual logic of the execution is implemented. The backend plugin uses the **Flyte PluginMachinery** interface to implement a plugin which can be one of the following supported types: + +1. [Kubernetes operator Plugin](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.26/go/tasks/pluginmachinery/k8s#Plugin): The demo in the video below shows two examples of K8s backend plugins: flytekit `Athena` & `Spark`, and Flyte K8s `Pod` & `Spark`. + + ```{youtube} oK2RGQuP94k + ``` + +2. **A Web API plugin:** [Async](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.26/go/tasks/pluginmachinery/webapi#AsyncPlugin) or [Sync](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.26/go/tasks/pluginmachinery/webapi#SyncPlugin). +3. [Core Plugin](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/core#Plugin): if none of the above fits diff --git a/docs/user_guide/extending/container_interface.md b/docs/user_guide/extending/container_interface.md new file mode 100644 index 0000000000..c0be559d76 --- /dev/null +++ b/docs/user_guide/extending/container_interface.md @@ -0,0 +1,81 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(core-extend-flyte-container-interface)= + +# Container interface + +```{eval-rst} +.. tags:: Extensibility, Contribute, Intermediate +``` + +Flyte typically interacts with containers in the course of its task execution (since most tasks are container +tasks). This is what that process looks like: + +1. At compilation time for a container task, the arguments to that container (and the container image itself) are set. + + 1. This is done by flytekit for instance for your run of the mill `@task`. This step is **crucial** - the task needs to specify an image available in the registry configured in the flyte installation. + +2. At runtime, Flyte will execute your task via a plugin. The default container plugin will do the following: + + 1. Set a series of environment variables. + 2. Before running the container, search/replace values in the container arguments. The command templating section below details how this happens. + + :::{note} + This templating process *should* be done by **all** plugins, even plugins that don't run a container but need + some information from the execution side. For example, a query task that submits a query to an engine that + writes the output to the raw output location. Or a query that uses the unique retry key as a temp table name, etc. + ::: + +## Command templating + +The templating of container arguments at run-time is one of the more advanced constructs of Flyte, but one that +authors of new task types should be aware of. For example, when looking at the hello world task in the UI, +if you click the Task tab, you'd see JSON that contains something like the following: + +```json +"container": { + "command": [], + "args": [ + "pyflyte-execute", + "--inputs", + "{{.input}}", + "--output-prefix", + "{{.outputPrefix}}", + "--raw-output-data-prefix", + "{{.rawOutputDataPrefix}}", + "--resolver", + "flytekit.core.python_auto_container.default_task_resolver", + "--", + "task-module", + "core.basic.hello_world", + "task-name", + "say_hello" + ], +``` + +The following table explains what each of the `{{}}` items mean, along with some others. + +| Template | Example | Description | +| ------------------------ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| {{.Input}} | `s3://my-bucket/inputs.pb` | Pb file containing a LiteralMap containing the inputs | +| {{.InputPrefix}} | `s3://my-bucket` | Just the bucket where the inputs.pb file can be found | +| {{.Inputs.\}} | `"hello world"` | For primitive inputs, the task can request that Flyte unpack the actual literal value, saving the task from having to download the file. Note that for Blob, Schema and StructuredDataset types, the uri where the data is stored will be filled in as the value. | +| {{.OutputPrefix}} | `s3://my-bucket/abc/data` | Location where the task should write a LiteralMap of output values in a file called `outputs.pb` | +| {{.RawOutputDataPrefix}} | `s3://your-data/` | Bucket where off-loaded data types (schemas, files, structureddatasets, etc.) are written. | +| {{.PerRetryUniqueKey}} | (random characters) | This is a random string that allows the task to differentiate between different executions of a task. Values will be unique per retry as well. | +| {{.TaskTemplatePath}} | `s3://my-bucket/task.pb` | For tasks that need the full task definition, use this template to access the full TaskTemplate IDL message. To ensure performance, propeller will not upload this file if this template was not requested by the task. | diff --git a/docs/user_guide/extending/custom_types.md b/docs/user_guide/extending/custom_types.md new file mode 100644 index 0000000000..af82d1a6ec --- /dev/null +++ b/docs/user_guide/extending/custom_types.md @@ -0,0 +1,195 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(advanced_custom_types)= + +# Custom types + +```{eval-rst} +.. tags:: Extensibility, Contribute, Intermediate +``` + +Flyte is a strongly-typed framework for authoring tasks and workflows. But there are situations when the existing +types do not directly work. This is true with any programming language! + +Similar to a programming language enabling higher-level concepts to describe user-specific objects such as classes in Python/Java/C++, struct in C/Golang, etc., +Flytekit allows modeling user classes. The idea is to make an interface that is more productive for the +use case, while writing a transformer that converts the user-defined type into one of the generic constructs in Flyte's type system. + +This example will try to model an example user-defined dataset and show how it can be seamlessly integrated with Flytekit's type engine. + +The example is demonstrated in the video below: + +```{eval-rst} +.. youtube:: 1xExpRzz8Tw + + +``` + ++++ {"lines_to_next_cell": 0} + +First, we import the dependencies. + +```{code-cell} +import os +import tempfile +import typing +from typing import Type + +from flytekit import Blob, BlobMetadata, BlobType, FlyteContext, Literal, LiteralType, Scalar, task, workflow +from flytekit.extend import TypeEngine, TypeTransformer +``` + ++++ {"lines_to_next_cell": 0} + +:::{note} +`FlyteContext` is used to access a random local directory. +::: + +Defined type here represents a list of files on the disk. We will refer to it as `MyDataset`. + +```{code-cell} +class MyDataset(object): + """ + ``MyDataset`` is a collection of files. In Flyte, this maps to a multi-part blob or directory. + """ + + def __init__(self, base_dir: str = None): + if base_dir is None: + self._tmp_dir = tempfile.TemporaryDirectory() + self._base_dir = self._tmp_dir.name + self._files = [] + else: + self._base_dir = base_dir + files = os.listdir(base_dir) + self._files = [os.path.join(base_dir, f) for f in files] + + @property + def base_dir(self) -> str: + return self._base_dir + + @property + def files(self) -> typing.List[str]: + return self._files + + def new_file(self, name: str) -> str: + new_file = os.path.join(self._base_dir, name) + self._files.append(new_file) + return new_file +``` + ++++ {"lines_to_next_cell": 0} + +`MyDataset` represents a set of files locally. However, when a workflow consists of multiple steps, we want the data to +flow between different steps. To achieve this, it is necessary to explain how the data will be transformed to +Flyte's remote references. To do this, we create a new instance of +{py:class}`~flytekit:flytekit.extend.TypeTransformer`, for the type `MyDataset` as follows: + +:::{note} +The `TypeTransformer` is a Generic abstract base class. The `Generic` type argument refers to the actual object +that we want to work with. In this case, it is the `MyDataset` object. +::: + +```{code-cell} +class MyDatasetTransformer(TypeTransformer[MyDataset]): + _TYPE_INFO = BlobType(format="binary", dimensionality=BlobType.BlobDimensionality.MULTIPART) + + def __init__(self): + super(MyDatasetTransformer, self).__init__(name="mydataset-transform", t=MyDataset) + + def get_literal_type(self, t: Type[MyDataset]) -> LiteralType: + """ + This is useful to tell the Flytekit type system that ``MyDataset`` actually refers to what corresponding type. + In this example, we say its of format binary (do not try to introspect) and there is more than one file in it. + """ + return LiteralType(blob=self._TYPE_INFO) + + def to_literal( + self, + ctx: FlyteContext, + python_val: MyDataset, + python_type: Type[MyDataset], + expected: LiteralType, + ) -> Literal: + """ + This method is used to convert from the given python type object ``MyDataset`` to the Literal representation. + """ + # Step 1: Upload all the data into a remote place recommended by Flyte + remote_dir = ctx.file_access.get_random_remote_directory() + ctx.file_access.upload_directory(python_val.base_dir, remote_dir) + # Step 2: Return a pointer to this remote_dir in the form of a Literal + return Literal(scalar=Scalar(blob=Blob(uri=remote_dir, metadata=BlobMetadata(type=self._TYPE_INFO)))) + + def to_python_value(self, ctx: FlyteContext, lv: Literal, expected_python_type: Type[MyDataset]) -> MyDataset: + """ + In this method, we want to be able to re-hydrate the custom object from Flyte Literal value. + """ + # Step 1: Download remote data locally + local_dir = ctx.file_access.get_random_local_directory() + ctx.file_access.download_directory(lv.scalar.blob.uri, local_dir) + # Step 2: Create the ``MyDataset`` object + return MyDataset(base_dir=local_dir) +``` + ++++ {"lines_to_next_cell": 0} + +Before we can use MyDataset in our tasks, we need to let Flytekit know that `MyDataset` should be considered as a valid type. +This is done using {py:class}`~flytekit:flytekit.extend.TypeEngine`'s `register` method. + +```{code-cell} +TypeEngine.register(MyDatasetTransformer()) +``` + ++++ {"lines_to_next_cell": 0} + +The new type should be ready to use! Let us write an example generator and consumer for this new datatype. + +```{code-cell} +@task +def generate() -> MyDataset: + d = MyDataset() + for i in range(3): + fp = d.new_file(f"x{i}") + with open(fp, "w") as f: + f.write(f"Contents of file{i}") + + return d + + +@task +def consume(d: MyDataset) -> str: + s = "" + for f in d.files: + with open(f) as fp: + s += fp.read() + s += "\n" + return s + + +@workflow +def wf() -> str: + return consume(d=generate()) +``` + ++++ {"lines_to_next_cell": 0} + +This workflow can be executed and tested locally. Flytekit will exercise the entire path even if you run it locally. + +```{code-cell} +if __name__ == "__main__": + print(wf()) +``` diff --git a/docs/user_guide/extending/index.md b/docs/user_guide/extending/index.md new file mode 100644 index 0000000000..19a553ddfc --- /dev/null +++ b/docs/user_guide/extending/index.md @@ -0,0 +1,218 @@ +(plugins_extend)= + +# Extending Flyte + +The core of Flyte is a container execution engine, where you can write one or more tasks and compose them together to +form a data dependency DAG, called a `workflow`. If your work involves writing simple Python or Java tasks that can +either perform operations on their own or call out to {ref}`Supported external services `, +then there's _no need to extend Flyte_. + +## Define a Custom Type + +Flyte, just like a programming language, has a core type-system, which can be extended by adding user-defined data types. +For example, Flyte supports adding support for a dataframe type from a new library, a custom user data structure, or a +grouping of images in a specific encoding. + +Flytekit natively supports structured data like {py:func}`~dataclasses.dataclass` using JSON as the +representation format (see {ref}`Using Custom Python Objects `). + +Flytekit allows users to extend Flyte's type system and implement types in Python that are not representable as JSON documents. The user has to implement a {py:class}`~flytekit.extend.TypeTransformer` +class to enable the translation of type from user type to Flyte-understood type. + +As an example, instead of using {py:class}`pandas.DataFrame` directly, you may want to use +[Pandera](https://pandera.readthedocs.io/en/stable/) to perform validation of an input or output dataframe +(see {ref}`Basic Schema Example `). + +To extend the type system, refer to {ref}`advanced_custom_types`. + +## Add a New Task Plugin + +Often you want to interact with services like: + +- Databases (e.g., Postgres, MySQL, etc.) +- DataWarehouses (e.g., Snowflake, BigQuery, Redshift etc.) +- Computation (e.g., AWS EMR, Databricks etc.) + +You might want this interaction to be available as a template for the open-source community or in your organization. This +can be done by creating a task plugin, which makes it possible to reuse the task's underlying functionality within Flyte +workflows. + +If you want users to write code simply using the {py:func}`~flytekit.task` decorator, but want to provide the +capability of running the function as a spark job or a sagemaker training job, then you can extend Flyte's task system. + +```{code-block} python +@task(task_config=MyContainerExecutionTask( + plugin_specific_config_a=..., + plugin_specific_config_b=..., + ... +)) +def foo(...) -> ...: + ... +``` + +Alternatively, you can provide an interface like this: + +```{code-block} python +query_task = SnowflakeTask( + query="Select * from x where x.time < {{.inputs.time}}", + inputs=kwtypes(time=datetime), + output_schema_type=pandas.DataFrame, +) + +@workflow +def my_wf(t: datetime) -> ...: + df = query_task(time=t) + return process(df=df) +``` + +There are two options when writing a new task plugin: you can write a task plugin as an extension in Flytekit or you can go deeper and write a plugin in the Flyte backend. + +## Flytekit-Only Task Plugin + +Flytekit is designed to be extremely extensible. You can add new task-types that are useful only for your use-case. +Flyte does come with the capability of extending the backend, but that is only required if you want the capability to be +extended to all users of Flyte, or there is a cost/visibility benefit of doing so. + +Writing your own Flytekit plugin is simple and is typically where you want to start when enabling custom task functionality. + +```{list-table} +:widths: 50 50 +:header-rows: 1 + +* - Pros + - Cons +* - Simple to write — implement in Python. Flyte will treat it like a container execution and blindly pass + control to the plugin. + - Limited ways of providing additional visibility in progress, external links, etc. +* - Simple to publish: `flytekitplugins` can be published as independent libraries and they follow a simple API. + - Has to be implemented in every language as these are SDK-side plugins only. +* - Simple to perform testing: test locally in flytekit. + - In case of side-effects, it could lead to resource leaks. For example, if the plugin runs a BigQuery job, + it is possible that the plugin may crash after running the job and Flyte cannot guarantee that the BigQuery job + will be successfully terminated. +* - + - Potentially expensive: in cases where the plugin runs a remote job, running a new pod for every task execution + causes severe strain on Kubernetes and the task itself uses almost no CPUs. Also because of its stateful nature, + using spot-instances is not trivial. +* - + - A bug fix to the runtime needs a new library version of the plugin. +* - + - Not trivial to implement resource controls, like throttling, resource pooling, etc. +``` + +### User Container vs. Pre-built Container Task Plugin + +A Flytekit-only task plugin can be a {ref}`user container ` or {ref}`pre-built container ` task plugin. + +```{list-table} +:widths: 10 50 50 +:header-rows: 1 + +* - + - User Container + - Pre-built Container +* - Serialization + - At serialization time, a Docker container image is required. The assumption is that this Docker image has the task code. + - The Docker container image is hardcoded at serialization time into the task definition by the author of that task plugin. +* - Serialization + - The serialized task contains instructions to the container on how to reconstitute the task. + - Serialized task should contain all the information needed to run that task instance (but not necessarily to reconstitute it). +* - Run-time + - When Flyte runs the task, the container is launched, and the user-given instructions recreate a Python object representing the task. + - When Flyte runs the task, the container is launched. The container should have an executor built into it that knows how to execute the task. +* - Run-time + - The task object that gets serialized at compile-time is recreated using the user's code at run time. + - The task object that gets serialized at compile-time does not exist at run time. +* - Run-time + - At platform-run-time, the user-decorated function is executed. + - At platform-run-time, there is no user function, and the executor is responsible for producing outputs, given the inputs to the task. +``` + +### Backend Plugin + +{ref}`Writing a Backend plugin ` makes it possible for users to write extensions for FlytePropeller - Flyte's scheduling engine. This enables complete control of the visualization and availability +of the plugin. + +```{list-table} +:widths: 50 50 +:header-rows: 1 + +* - Pros + - Cons +* - Service oriented way of deploying new plugins - strong contracts. Maintainers can deploy new versions of the backend plugin, fix bugs, without needing the users to upgrade libraries, etc. + - Need to be implemented in Golang. +* - Drastically cheaper and more efficient to execute. FlytePropeller is written in Golang and uses an event loop model. Each process of FlytePropeller can execute thousands of tasks concurrently. + - Needs a FlytePropeller build (*currently*). +* - Flyte guarantees resource cleanup. + - Need to implement contract in a spec language like protobuf, OpenAPI, etc. +* - Flyteconsole plugins (capability coming soon!) can be added to customize visualization and progress tracking of the execution. + - Development cycle can be much slower than flytekit-only plugins. +* - Resource controls and backpressure management is available. + - +* - Implement once, use in any SDK or language! + - +``` + +#### Flyte Agent Service + +_New in Flyte 1.7.0_ + +{ref}`Flyte Agent Service ` allows you to write backend +plugins in Python. + +### Summary + +```{mermaid} + +flowchart LR + U{Use Case} + F([Python Flytekit Plugin]) + B([Golang
Backend Plugin]) + + subgraph WFTP[Writing Flytekit Task Plugins] + UCP([User Container Plugin]) + PCP([Pre-built Container Plugin]) + end + + subgraph WBE[Writing Backend Extensions] + K8S([K8s Plugin]) + WP([WebAPI Plugin]) + CP([Complex Plugin]) + end + + subgraph WCFT[Writing Custom Flyte Types] + T([Flytekit
Type Transformer]) + end + + U -- Light-weight
Extensions --> F + U -- Performant
Multi-language
Extensions --> B + U -- Specialized
Domain-specific Types --> T + F -- Require
user-defined
container --> UCP + F -- Provide
prebuilt
container --> PCP + B --> K8S + B --> WP + B --> CP + + style WCFT fill:#eee,stroke:#aaa + style WFTP fill:#eee,stroke:#aaa + style WBE fill:#eee,stroke:#aaa + style U fill:#fff2b2,stroke:#333 + style B fill:#EAD1DC,stroke:#333 + style K8S fill:#EAD1DC,stroke:#333 + style WP fill:#EAD1DC,stroke:#333 + style CP fill:#EAD1DC,stroke:#333 +``` + +Use the flow-chart above to point you to one of these examples: + +```{toctree} +:maxdepth: -1 +:name: extending_toc +:hidden: + +custom_types +prebuilt_container_task_plugins +user_container_task_plugins +backend_plugins +container_interface +``` diff --git a/docs/user_guide/extending/prebuilt_container_task_plugins.md b/docs/user_guide/extending/prebuilt_container_task_plugins.md new file mode 100644 index 0000000000..ed51f1b7a6 --- /dev/null +++ b/docs/user_guide/extending/prebuilt_container_task_plugins.md @@ -0,0 +1,105 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(prebuilt_container)= + +# Prebuilt container task plugins + +```{eval-rst} +.. tags:: Extensibility, Contribute, Intermediate +``` + +A prebuilt container task plugin runs a prebuilt container. The following are the advantages of using a prebuilt container in comparison to a user-defined container: + +- Shifts the burden of writing Dockerfile from the user who uses the task in workflows to the author of the task type. +- Allows the author to optimize the image that the task runs on. +- Makes it possible to (largely) extend the Flyte task execution behavior without using the backend Golang plugin. + The caveat is that these tasks can't access the K8s cluster, so you'll still need a backend plugin if you want a custom task type that generates CRD. + +## Usage + +Take a look at the [example PR](https://github.com/flyteorg/flytekit/pull/470), where we switched the built-in SQLite3 task from the old (user-container) to the new style of writing tasks. + +There aren't many changes from the user's standpoint: +\- Install whichever Python library has the task type definition (in the case of SQLite3, it's bundled in Flytekit, but this isn't always the case (for example, [SQLAlchemy](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-sqlalchemy))). +\- Import and instantiate the task as you would for any other type of non-function-based task. + +## How to write a task + +Writing a pre-built container task consists of three steps: + +1. Defining a Task class +2. Defining an Executor class +3. Creating a Dockerfile that is executed when any user runs your task. It'll most likely include Flytekit, Python, and your task extension code. + +To follow along, use the [PR (mentioned above)](https://github.com/flyteorg/flytekit/pull/470) where we migrate the SQLite3 task. + +## Python library + +### Defining a task + +New tasks of this type must be created as a subclass of the `PythonCustomizedContainerTask` class. + +Specifically, you need to customize the following three arguments which would be sent to the parent class constructor: + +- `container_image`: This is the container image that will run on a Flyte platform when the user invokes the job. +- `executor_type`: This should be the Python class that inherits the `ShimTaskExecutor`. +- `task_type`: All types have a task type. Flyte engine uses this string to identify which plugin to use when running a task. + +The container plugin will be used for everything that doesn't have an explicit match (which is correct in this case). +So you may call it whatever you want, just not something that's already been claimed (like "spark"). + +Referring to the SQLite3 example, + +``` +container_image="ghcr.io/flyteorg/flytekit:py38-v0.19.0b7", +executor_type=SQLite3TaskExecutor, +task_type="sqlite", +``` + +Note that the container is special in this case since we utilize the Flytekit image. + +Furthermore, you need to override the `get_custom` function to include all the information the executor will need to run. + +Keep in mind that the task's execution behavior is entirely defined by the task's serialized form (that is, the serialized `TaskTemplate`). +This function stores and inserts the data into the template's [custom field](https://github.com/flyteorg/flyteidl/blob/7302971c064b6061a148f2bee79f673bc8cf30ee/protos/flyteidl/core/tasks.proto#L114). +However, keep the task template's overall size to a minimum. + +### Executor + +You must subclass and override the `execute_from_model` function for the `ShimTaskExecutor` abstract class. +This function will be invoked in both local workflow execution and platform-run-time execution, and will include all of the business logic of your task. + +The signature of this execute function differs from the `execute` functions of most other tasks since the `TaskTemplate` determines all the business logic, including how the task is run. + +### Image + +This is the custom image that you specified in the subclass `PythonCustomizedContainerTask`. Out of the box, when Flyte runs the container, these tasks will run a command that looks like this + +``` +pyflyte-execute --inputs s3://inputs.pb --output-prefix s3://outputs --raw-output-data-prefix s3://user-data --resolver flytekit.core.python_customized_container_task.default_task_template_resolver -- {{.taskTemplatePath}} path.to.your.executor.subclass +``` + +This means that your [Docker image](https://github.com/flyteorg/flytekit/blob/master/Dockerfile) will need Python and Flytekit installed. +The container's Python interpreter should be able to find your custom executor class at the import path `path.to.your.executor.subclass`. + +______________________________________________________________________ + +The key takeaways of a pre-built container task plugin are: + +- The task object serialized at compile time does not exist at run time. +- There is no user function at platform run time, and the executor is responsible for producing outputs based on the task's inputs. diff --git a/docs/user_guide/extending/user_container_task_plugins.md b/docs/user_guide/extending/user_container_task_plugins.md new file mode 100644 index 0000000000..68cc1a859d --- /dev/null +++ b/docs/user_guide/extending/user_container_task_plugins.md @@ -0,0 +1,164 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +(user_container)= + +# User container task plugins + +```{eval-rst} +.. tags:: Extensibility, Contribute, Intermediate +``` + +A user container task plugin runs a user-defined container that has the user code. + +This tutorial will walk you through writing your own sensor-style plugin that allows users to wait for a file to land +in the object store. Remember that if you follow the flyte/flytekit constructs, you will automatically make your plugin portable +across all cloud platforms that Flyte supports. + +## Sensor plugin + +A sensor plugin waits for some event to happen before marking the task as success. You need not worry about the +timeout as that will be handled by the flyte engine itself when running in production. + +### Plugin API + +```python +sensor = WaitForObjectStoreFile(metadata=metadata(timeout="1H", retries=10)) + +@workflow +def wait_and_run(path: str) -> int: + # To demonstrate how to create outputs, we will also + # return the output from the sensor. The output will be the + # same as the path + path = sensor(path=path) + return do_next(path=path) +``` + +```{code-cell} +import typing +from datetime import timedelta +from time import sleep + +from flytekit import TaskMetadata, task, workflow +from flytekit.extend import Interface, PythonTask, context_manager +``` + ++++ {"lines_to_next_cell": 0} + +### Plugin structure + +As illustrated above, to achieve this structure we need to create a class named `WaitForObjectStoreFile`, which +derives from {py:class}`flytekit.PythonFunctionTask` as follows. + +```{code-cell} +class WaitForObjectStoreFile(PythonTask): + """ + Add documentation here for your plugin. + This plugin creates an object store file sensor that waits and exits only when the file exists. + """ + + _VAR_NAME: str = "path" + + def __init__( + self, + name: str, + poll_interval: timedelta = timedelta(seconds=10), + **kwargs, + ): + super(WaitForObjectStoreFile, self).__init__( + task_type="object-store-sensor", + name=name, + task_config=None, + interface=Interface(inputs={self._VAR_NAME: str}, outputs={self._VAR_NAME: str}), + **kwargs, + ) + self._poll_interval = poll_interval + + def execute(self, **kwargs) -> typing.Any: + # No need to check for existence, as that is guaranteed. + path = kwargs[self._VAR_NAME] + ctx = context_manager.FlyteContext.current_context() + user_context = ctx.user_space_params + while True: + user_context.logging.info(f"Sensing file in path {path}...") + if ctx.file_access.exists(path): + user_context.logging.info(f"file in path {path} exists!") + return path + user_context.logging.warning(f"file in path {path} does not exists!") + sleep(self._poll_interval.seconds) +``` + +#### Config objects + +Flytekit routes to the right plugin based on the type of `task_config` class if using the `@task` decorator. +Config is very useful for cases when you want to customize the behavior of the plugin or pass the config information +to the backend plugin; however, in this case there's no real configuration. The config object can be any class that your +plugin understands. + +:::{note} +Observe that the base class is Generic; it is parameterized with the desired config class. +::: + +:::{note} +To create a task decorator-based plugin, `task_config` is required. +In this example, we are creating a named class plugin, and hence, this construct does not need a plugin. +::: + +Refer to the [spark plugin](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-spark) for an example of a config object. + ++++ + +### Actual usage + +```{code-cell} +sensor = WaitForObjectStoreFile( + name="my-objectstore-sensor", + metadata=TaskMetadata(retries=10, timeout=timedelta(minutes=20)), + poll_interval=timedelta(seconds=1), +) + + +@task +def print_file(path: str) -> str: + print(path) + return path + + +@workflow +def my_workflow(path: str) -> str: + return print_file(path=sensor(path=path)) +``` + ++++ {"lines_to_next_cell": 0} + +And of course, you can run the workflow locally using your own new shiny plugin! + +```{code-cell} +if __name__ == "__main__": + f = "/tmp/some-file" + with open(f, "w") as w: + w.write("Hello World!") + + print(my_workflow(path=f)) +``` + +The key takeaways of a user container task plugin are: + +- The task object that gets serialized at compile-time is recreated using the user's code at run time. +- At platform-run-time, the user-decorated function is executed. diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md new file mode 100644 index 0000000000..9c4009c9fe --- /dev/null +++ b/docs/user_guide/index.md @@ -0,0 +1,72 @@ +--- +:next-page: environment_setup +:next-page-title: Environment Setup +:prev-page: getting_started/analytics +:prev-page-title: Analytics +--- + +(userguide)= + +# User guide + +If this is your first time using Flyte, check out the {doc}`Getting Started ` guide. + +This _User guide_, the {ref}`Tutorials ` and the {ref}`Integrations ` examples cover all of +the key features of Flyte for data analytics, data science and machine learning practitioners, organized by topic. Each +section below introduces a core feature of Flyte and how you can use it to address specific use cases. Code for all +of the examples can be found in the [flytesnacks repo](https://github.com/flyteorg/flytesnacks). + +It comes with a specific environment to make running, documenting +and contributing samples easy. If this is your first time running these examples, follow the +{doc}`environment setup guide ` to get started. + +```{tip} +To learn about how to spin up and manage a Flyte cluster in the cloud, see the +{doc}`Deployment Guides `. +``` + +```{note} +Want to contribute or update an example? Check out the {doc}`Contribution Guide <../flytesnacks/contribute>`. +``` + +## Table of contents + +```{list-table} +:header-rows: 0 +:widths: 20 30 + +* - {doc}`🌳 Environment Setup ` + - Set up a development environment to run the examples in the user guide. +* - {doc}`🔤 Basics ` + - Learn about tasks, workflows, launch plans, caching and managing files and directories. +* - {doc}`⌨️ Data Types and IO ` + - Improve pipeline robustness with Flyte's portable and extensible type system. +* - {doc}`🔮 Advanced Composition ` + - Implement conditionals, nested and dynamic workflows, map tasks and even recursion! +* - {doc}`🧩 Customizing Dependencies ` + - Provide custom dependencies to run your Flyte entities. +* - {doc}`🏡 Development Lifecycle ` + - Develop and test locally on the demo cluster. +* - {doc}`⚗️ Testing ` + - Test tasks and workflows with Flyte's testing utilities. +* - {doc}`🚢 Productionizing ` + - Ship and configure your machine learning pipelines on a production Flyte installation. +* - {doc}`🏗 Extending ` + - Define custom plugins that aren't currently supported in the Flyte ecosystem. +``` + +```{toctree} +:maxdepth: -1 +:name: user_guide_toc +:hidden: + +environment_setup +basics/index +data_types_and_io/index +advanced_composition/index +customizing_dependencies/index +development_lifecycle/index +testing/index +productionizing/index +extending/index +``` \ No newline at end of file diff --git a/docs/user_guide/productionizing/configuring_access_to_gpus.md b/docs/user_guide/productionizing/configuring_access_to_gpus.md new file mode 100644 index 0000000000..f2575b5adb --- /dev/null +++ b/docs/user_guide/productionizing/configuring_access_to_gpus.md @@ -0,0 +1,53 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(configure-gpus)= + +# Configuring access to GPUs + +```{eval-rst} +.. tags:: Deployment, Infrastructure, GPU, Intermediate +``` + +Along with the simpler resources like CPU/Memory, you may want to configure and access GPU resources. Flyte +allows you to configure the GPU access poilcy for your cluster. GPUs are expensive and it would not be ideal to +treat machines with GPUs and machines with CPUs equally. You may want to reserve machines with GPUs for tasks +that explicitly request GPUs. To achieve this, Flyte uses the Kubernetes concept of [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). + +Kubernetes can automatically apply tolerations for extended resources like GPUs using the [ExtendedResourceToleration plugin](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#extendedresourcetoleration), enabled by default in some cloud environments. Make sure the GPU nodes are tainted with a key matching the resource name, i.e., `key: nvidia.com/gpu`. + +You can also configure Flyte backend to apply specific tolerations. This configuration is controlled under generic k8s plugin configuration as can be found [here](https://github.com/flyteorg/flyteplugins/blob/5a00b19d88b93f9636410a41f81a73356a711482/go/tasks/pluginmachinery/flytek8s/config/config.go#L120). + +The idea of this configuration is that whenever a task that can execute on Kubernetes requests for GPUs, it automatically +adds the matching toleration for that resource (in this case, `gpu`) to the generated PodSpec. +As it follows here, you can configure it to access specific resources using the tolerations for all resources supported by +Kubernetes. + +Here's an example configuration: + +```yaml +plugins: + k8s: + resource-tolerations: + - nvidia.com/gpu: + - key: "key1" + operator: "Equal" + value: "value1" + effect: "NoSchedule" +``` + +Getting this configuration into your deployment will depend on how Flyte is deployed on your cluster. If you use the default Opta/Helm route, you'll need to amend your Helm chart values ([example](https://github.com/flyteorg/flyte/blob/cc127265aec490ad9537d29bd7baff828043c6f5/charts/flyte-core/values.yaml#L629)) so that they end up [here](https://github.com/flyteorg/flyte/blob/3d265f166fcdd8e20b07ff82b494c0a7f6b7b108/deployment/eks/flyte_helm_generated.yaml#L521). diff --git a/docs/user_guide/productionizing/configuring_logging_links_in_the_ui.md b/docs/user_guide/productionizing/configuring_logging_links_in_the_ui.md new file mode 100644 index 0000000000..57502a8c77 --- /dev/null +++ b/docs/user_guide/productionizing/configuring_logging_links_in_the_ui.md @@ -0,0 +1,138 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(configure-logging)= + +# Configuring logging links in the UI + +```{eval-rst} +.. tags:: Deployment, Intermediate, UI +``` + +To debug your workflows in production, you want to access logs from your tasks as they run. +These logs are different from the core Flyte platform logs, are specific to execution, and may vary from plugin to plugin; +for example, Spark may have driver and executor logs. + +Every organization potentially uses different log aggregators, making it hard to create a one-size-fits-all solution. +Some examples of the log aggregators include cloud-hosted solutions like AWS CloudWatch, GCP Stackdriver, Splunk, Datadog, etc. + +Flyte provides a simplified interface to configure your log provider. Flyte-sandbox +ships with the Kubernetes dashboard to visualize the logs. This may not be safe for production, hence we recommend users +explore other log aggregators. + +## How to configure? + +To configure your log provider, the provider needs to support `URL` links that are shareable and can be templatized. +The templating engine has access to [these](https://github.com/flyteorg/flyteplugins/blob/b0684d97a1cf240f1a44f310f4a79cc21844caa9/go/tasks/pluginmachinery/tasklog/plugin.go#L7-L16) parameters. + +The parameters can be used to generate a unique URL to the logs using a templated URI that pertain to a specific task. The templated URI has access to the following parameters: + +```{eval-rst} +.. list-table:: Parameters to generate a templated URI + :widths: 25 50 + :header-rows: 1 + + * - Parameter + - Description + * - ``{{ .podName }}`` + - Gets the pod name as it shows in k8s dashboard + * - ``{{ .podUID }}`` + - The pod UID generated by the k8s at runtime + * - ``{{ .namespace }}`` + - K8s namespace where the pod runs + * - ``{{ .containerName }}`` + - The container name that generated the log + * - ``{{ .containerId }}`` + - The container id docker/crio generated at run time + * - ``{{ .logName }}`` + - A deployment specific name where to expect the logs to be + * - ``{{ .hostname }}`` + - The hostname where the pod is running and logs reside + * - ``{{ .podRFC3339StartTime }}`` + - The pod creation time (in RFC3339 format, e.g. "2021-01-01T02:07:14Z", also conforming to ISO 8601) + * - ``{{ .podRFC3339FinishTime }}`` + - Don't have a good mechanism for this yet, but approximating with ``time.Now`` for now + * - ``{{ .podUnixStartTime }}`` + - The pod creation time (in unix seconds, not millis) + * - ``{{ .podUnixFinishTime }}`` + - Don't have a good mechanism for this yet, but approximating with ``time.Now`` for now +``` + +The parameterization engine uses Golangs native templating format and hence uses `{{ }}`. An example configuration can be seen as follows: + +```yaml +task_logs: + plugins: + logs: + templates: + - displayName: + templateUris: + - "https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/flyte-production/kubernetes;stream=var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log" + - "https://some-other-source/home?region=us-east-1#logEventViewer:group=/flyte-production/kubernetes;stream=var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log" + messageFormat: 0 # this parameter is optional, but use 0 for "unknown", 1 for "csv", or 2 for "json" +``` + +:::{tip} +Since helm chart uses the same templating syntax for args (like `{{ }}`), compiling the chart results in helm replacing Flyte log link templates as well. To avoid this, you can use escaped templating for Flyte logs in the helm chart. +This ensures that Flyte log link templates remain in place during helm chart compilation. +For example: + +If your configuration looks like this: + +`https://someexample.com/app/podName={{ "{{" }} .podName {{ "}}" }}&containerName={{ .containerName }}` + +Helm chart will generate: + +`https://someexample.com/app/podName={{.podName}}&containerName={{.containerName}}` + +Flytepropeller pod would be created as: + +`https://someexample.com/app/podName=pname&containerName=cname` +::: + +This code snippet will output two logs per task that use the log plugin. +However, not all task types use the log plugin; for example, the SageMaker plugin uses the log output provided by Sagemaker, and the Snowflake plugin will use a link to the snowflake console. + +## Datadog integration + +To send your Flyte workflow logs to Datadog, you can follow these steps: + +1. Enable collection of logs from containers and collection of logs using files. The precise configuration steps will vary depending on your specific setup. + +For instance, if you're using Helm, use the following config: + +```yaml +logs: + enabled: true + containerCollectAll: true + containerCollectUsingFiles: true +``` + +If you're using environment variables, use the following config: + +```yaml +DD_LOGS_ENABLED: "false" +DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL: "true" +DD_LOGS_CONFIG_K8S_CONTAINER_USE_FILE: "true" +DD_CONTAINER_EXCLUDE_LOGS: "name:datadog-agent" # This is to avoid tracking logs produced by the datadog agent itself +``` + +:::{warning} +The boolean values have to be represented as strings. +::: + +2. The Datadog [guide](https://docs.datadoghq.com/containers/kubernetes/log/?tab=daemonset) includes a section on mounting volumes. It is essential (and a prerequisite for proper functioning) to map the volumes "logpodpath" and "logcontainerpath" as illustrated in the linked example. While the "pointerdir" volume is optional, it is recommended that you map it to prevent the loss of container logs during restarts or network issues (as stated in the guide). diff --git a/docs/user_guide/productionizing/customizing_task_resources.md b/docs/user_guide/productionizing/customizing_task_resources.md new file mode 100644 index 0000000000..39fee64d55 --- /dev/null +++ b/docs/user_guide/productionizing/customizing_task_resources.md @@ -0,0 +1,181 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Customizing task resources + +```{eval-rst} +.. tags:: Deployment, Infrastructure, Basic +``` + +One of the reasons to use a hosted Flyte environment is the potential of leveraging CPU, memory and storage resources, far greater than what's available locally. +Flytekit makes it possible to specify these requirements declaratively and close to where the task itself is declared. + ++++ + +In this example, the memory required by the function increases as the dataset size increases. +Large datasets may not be able to run locally, so we would want to provide hints to the Flyte backend to request for more memory. +This is done by decorating the task with the hints as shown in the following code sample. + +Tasks can have `requests` and `limits` which mirror the native [equivalents in Kubernetes](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits). +A task can possibly be allocated more resources than it requests, but never more than its limit. +Requests are treated as hints to schedule tasks on nodes with available resources, whereas limits +are hard constraints. + +For either a request or limit, refer to the {py:class}`flytekit:flytekit.Resources` documentation. + +The following attributes can be specified for a `Resource`. + +1. `cpu` +2. `mem` +3. `gpu` + +To ensure that regular tasks that don't require GPUs are not scheduled on GPU nodes, a separate node group for GPU nodes can be configured with [taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). + +To ensure that tasks that require GPUs get the needed tolerations on their pods, set up FlytePropeller using the following [configuration](https://github.com/flyteorg/flytepropeller/blob/v0.10.5/config.yaml#L51,L56). Ensure that this toleration config matches the taint config you have configured to protect your GPU providing nodes from dealing with regular non-GPU workloads (pods). + +The actual values follow the [Kubernetes convention](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes). +Let's look at an example to understand how to customize resources. + ++++ {"lines_to_next_cell": 0} + +Import the dependencies. + +```{code-cell} +import typing + +from flytekit import Resources, task, workflow +``` + ++++ {"lines_to_next_cell": 0} + +Define a task and configure the resources to be allocated to it. + +```{code-cell} +@task(requests=Resources(cpu="1", mem="100Mi"), limits=Resources(cpu="2", mem="150Mi")) +def count_unique_numbers(x: typing.List[int]) -> int: + s = set() + for i in x: + s.add(i) + return len(s) +``` + ++++ {"lines_to_next_cell": 0} + +Define a task that computes the square of a number. + +```{code-cell} +@task +def square(x: int) -> int: + return x * x +``` + ++++ {"lines_to_next_cell": 0} + +You can use the tasks decorated with memory and storage hints like regular tasks in a workflow. + +```{code-cell} +@workflow +def my_workflow(x: typing.List[int]) -> int: + return square(x=count_unique_numbers(x=x)) +``` + ++++ {"lines_to_next_cell": 0} + +You can execute the workflow locally. + +```{code-cell} +if __name__ == "__main__": + print(count_unique_numbers(x=[1, 1, 2])) + print(my_workflow(x=[1, 1, 2])) +``` + +:::{note} +To alter the limits of the default platform configuration, change the [admin config](https://github.com/flyteorg/flyte/blob/b16ffd76934d690068db1265ac9907a278fba2ee/deployment/eks/flyte_helm_generated.yaml#L203-L213) and [namespace level quota](https://github.com/flyteorg/flyte/blob/b16ffd76934d690068db1265ac9907a278fba2ee/deployment/eks/flyte_helm_generated.yaml#L214-L240) on the cluster. +::: + ++++ + +(resource_with_overrides)= + +## Using `with_overrides` + +You can use the `with_overrides` method to override the resources allocated to the tasks dynamically. +Let's understand how the resources can be initialized with an example. + ++++ {"lines_to_next_cell": 0} + +Import the dependencies. + +```{code-cell} +import typing # noqa: E402 + +from flytekit import Resources, task, workflow # noqa: E402 +``` + ++++ {"lines_to_next_cell": 0} + +Define a task and configure the resources to be allocated to it. +You can use tasks decorated with memory and storage hints like regular tasks in a workflow. + +```{code-cell} +@task(requests=Resources(cpu="1", mem="200Mi"), limits=Resources(cpu="2", mem="350Mi")) +def count_unique_numbers_1(x: typing.List[int]) -> int: + s = set() + for i in x: + s.add(i) + return len(s) +``` + ++++ {"lines_to_next_cell": 0} + +Define a task that computes the square of a number. + +```{code-cell} +@task +def square_1(x: int) -> int: + return x * x +``` + ++++ {"lines_to_next_cell": 0} + +The `with_overrides` method overrides the old resource allocations. + +```{code-cell} +@workflow +def my_pipeline(x: typing.List[int]) -> int: + return square_1(x=count_unique_numbers_1(x=x)).with_overrides(limits=Resources(cpu="6", mem="500Mi")) +``` + ++++ {"lines_to_next_cell": 0} + +You can execute the workflow locally. + +```{code-cell} +if __name__ == "__main__": + print(count_unique_numbers_1(x=[1, 1, 2])) + print(my_pipeline(x=[1, 1, 2])) +``` + +You can see the memory allocation below. The memory limit is `500Mi` rather than `350Mi`, and the +CPU limit is 4, whereas it should have been 6 as specified using `with_overrides`. +This is because the default platform CPU quota for every pod is 4. + +:::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/core/resource_allocation.png +:alt: Resource allocated using "with_overrides" method + +Resource allocated using "with_overrides" method +::: diff --git a/docs/user_guide/productionizing/index.md b/docs/user_guide/productionizing/index.md new file mode 100644 index 0000000000..a7748b1256 --- /dev/null +++ b/docs/user_guide/productionizing/index.md @@ -0,0 +1,24 @@ +(deployment_workflow)= + +# Productionize + +In this section, you will learn how to take Flyte pipelines into production. +You will explore concepts such as customizing resources, notifications, scheduling, +GPU configuration, secrets, spot instances and more. + +```{toctree} +:maxdepth: -1 +:name: productionizing_toc +:hidden: + +customizing_task_resources +reference_tasks +reference_launch_plans +notifications +schedules +configuring_logging_links_in_the_ui +configuring_access_to_gpus +spot_instances +secrets +workflow_labels_and_annotations +``` diff --git a/docs/user_guide/productionizing/notifications.md b/docs/user_guide/productionizing/notifications.md new file mode 100644 index 0000000000..133a402c43 --- /dev/null +++ b/docs/user_guide/productionizing/notifications.md @@ -0,0 +1,225 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Notifications + +```{eval-rst} +.. tags:: Intermediate + +``` + ++++ + +When a workflow is completed, users can be notified by: + +- Email +- [Pagerduty](https://support.pagerduty.com/docs/email-integration-guide#integrating-with-a-pagerduty-service) +- [Slack](https://slack.com/help/articles/206819278-Send-emails-to-Slack) + +The content of these notifications is configurable at the platform level. + +## Code example + +When a workflow reaches a specified [terminal workflow execution phase](https://github.com/flyteorg/flytekit/blob/v0.16.0b7/flytekit/core/notification.py#L10,L15), +the {py:class}`flytekit:flytekit.Email`, {py:class}`flytekit:flytekit.PagerDuty`, or {py:class}`flytekit:flytekit.Slack` +objects can be used in the construction of a {py:class}`flytekit:flytekit.LaunchPlan`. + +```{code-cell} +from datetime import timedelta +``` + ++++ {"lines_to_next_cell": 0} + +Consider the following example workflow: + +```{code-cell} +from flytekit import Email, FixedRate, LaunchPlan, PagerDuty, Slack, WorkflowExecutionPhase, task, workflow + + +@task +def double_int_and_print(a: int) -> str: + return str(a * 2) + + +@workflow +def int_doubler_wf(a: int) -> str: + doubled = double_int_and_print(a=a) + return doubled +``` + ++++ {"lines_to_next_cell": 0} + +Here are three scenarios that can help deepen your understanding of how notifications work: + +1. Launch Plan triggers email notifications when the workflow execution reaches the `SUCCEEDED` phase. + +```{code-cell} +int_doubler_wf_lp = LaunchPlan.get_or_create( + name="email_notifications_lp", + workflow=int_doubler_wf, + default_inputs={"a": 4}, + notifications=[ + Email( + phases=[WorkflowExecutionPhase.SUCCEEDED], + recipients_email=["admin@example.com"], + ) + ], +) +``` + ++++ {"lines_to_next_cell": 0} + +2. Notifications shine when used for scheduled workflows to alert for failures. + +```{code-cell} +:lines_to_next_cell: 2 + +int_doubler_wf_scheduled_lp = LaunchPlan.get_or_create( + name="int_doubler_wf_scheduled", + workflow=int_doubler_wf, + default_inputs={"a": 4}, + notifications=[ + PagerDuty( + phases=[WorkflowExecutionPhase.FAILED, WorkflowExecutionPhase.TIMED_OUT], + recipients_email=["abc@pagerduty.com"], + ) + ], + schedule=FixedRate(duration=timedelta(days=1)), +) +``` + +3. Notifications can be combined with different permutations of terminal phases and recipient targets. + +```{code-cell} +wacky_int_doubler_lp = LaunchPlan.get_or_create( + name="wacky_int_doubler", + workflow=int_doubler_wf, + default_inputs={"a": 4}, + notifications=[ + Email( + phases=[WorkflowExecutionPhase.FAILED], + recipients_email=["me@example.com", "you@example.com"], + ), + Email( + phases=[WorkflowExecutionPhase.SUCCEEDED], + recipients_email=["myboss@example.com"], + ), + Slack( + phases=[ + WorkflowExecutionPhase.SUCCEEDED, + WorkflowExecutionPhase.ABORTED, + WorkflowExecutionPhase.TIMED_OUT, + ], + recipients_email=["myteam@slack.com"], + ), + ], +) +``` + +4. You can use pyflyte register to register the launch plan and launch it in the web console to get the notifications. + +``` +pyflyte register lp_notifications.py +``` + +Choose the launch plan with notifications config +:::{figure} https://i.ibb.co/cLT5tRX/lp.png +:alt: Notifications Launch Plan +:class: with-shadow +::: + ++++ + +### Future work + +Work is ongoing to support a generic event egress system that can be used to publish events for tasks, workflows, and +workflow nodes. When this is complete, generic event subscribers can asynchronously process these events for a rich +and fully customizable experience. + +## Platform configuration changes + +The `notifications` top-level portion of the Flyteadmin config specifies how to handle notifications. + +As in schedules, the handling of notifications is composed of two parts— one part handles enqueuing notifications asynchronously. The other part handles processing pending notifications and sends out emails and alerts. + +This is only supported for Flyte instances running on AWS. + +### Config +#### For Sandbox +To publish notifications, you'll need to register a sendgrid api key from [sendgrid](https://sendgrid.com/), it's free for 100 emails per day. +You have to add notifications config in your sandbox config file. + +```yaml +# config-sandbox.yaml +notifications: + type: sandbox # noqa: F821 + emailer: + emailServerConfig: + serviceName: sendgrid + apiKeyEnvVar: SENDGRID_API_KEY + subject: "Notice: Execution \"{{ workflow.name }}\" has {{ phase }} in \"{{ domain }}\"." + sender: "flyte-notifications@company.com" + body: > + Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at + + http://flyte.company.com/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}. {{ error }} +``` + +Note that you should set and export the `SENDGRID_API_KEY` environment variable in your shell. + +#### For AWS +To publish notifications, you'll need to set up an [SNS topic](https://aws.amazon.com/sns/?whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc). + +To process notifications, you'll need to set up an [AWS SQS](https://aws.amazon.com/sqs/) queue to consume notification events. This queue must be configured as a subscription to your SNS topic you created above. + +To publish notifications, you'll need a [verified SES email address](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-addresses-and-domains.html) which will be used to send notification emails and alerts using email APIs. + +The role you use to run Flyteadmin must have permissions to read and write to your SNS topic and SQS queue. + +Let's look into the following config section and explain what each value represents: + +```bash +notifications: + type: "aws" # noqa: F821 + region: "us-east-1" + publisher: + topicName: "arn:aws:sns:us-east-1:{{ YOUR ACCOUNT ID }}:{{ YOUR TOPIC }}" + processor: + queueName: "{{ YOUR QUEUE NAME }}" + accountId: "{{ YOUR ACCOUNT ID }}" + emailer: + subject: "Notice: Execution \"{{ workflow.name }}\" has {{ phase }} in \"{{ domain }}\"." + sender: "flyte-notifications@company.com" + body: > + Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at + + http://flyte.company.com/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}. {{ error }} +``` + +- **type**: AWS is the only cloud back-end supported for executing scheduled workflows; hence `"aws"` is the only valid value. By default, the no-op executor is used. +- **region**: Specifies the region AWS clients should use when creating SNS and SQS clients. +- **publisher**: Handles pushing notification events to your SNS topic. + : - **topicName**: This is the arn of your SNS topic. +- **processor**: Handles recording notification events and enqueueing them to be processed asynchronously. + : - **queueName**: Name of the SQS queue which will capture pending notification events. + - **accountId**: AWS [account id](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#FindingYourAWSId). +- **emailer**: Encloses config details for sending and formatting emails used as notifications. + : - **subject**: Configurable subject line used in notification emails. + - **sender**: Your verified SES email sender. + - **body**: Configurable email body used in notifications. + +The complete set of parameters that can be used for email templating are checked in [here](https://github.com/flyteorg/flyteadmin/blob/a84223dab00dfa52d8ba1ed2d057e77b6c6ab6a7/pkg/async/notifications/email.go#L18,L30). diff --git a/docs/user_guide/productionizing/reference_launch_plans.md b/docs/user_guide/productionizing/reference_launch_plans.md new file mode 100644 index 0000000000..8ea476ef3e --- /dev/null +++ b/docs/user_guide/productionizing/reference_launch_plans.md @@ -0,0 +1,88 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Reference launch plans + +```{eval-rst} +.. tags:: Intermediate +``` + +A {py:func}`flytekit.reference_launch_plan` references previously defined, serialized, and registered Flyte launch plans. +You can reference launch plans from other projects and create workflows that use launch plans declared by others. + +The following example illustrates how to use reference launch plans. + +:::{note} +Reference launch plans cannot be run locally. You must mock them out. +::: + +```{code-cell} +:lines_to_next_cell: 2 + +from typing import List + +from flytekit import reference_launch_plan, workflow +from flytekit.types.file import FlyteFile + + +@reference_launch_plan( + project="flytesnacks", + domain="development", + name="data_types_and_io.file.normalize_csv_file", + version="{{ registration.version }}", +) +def normalize_csv_file( + csv_url: FlyteFile, + column_names: List[str], + columns_to_normalize: List[str], + output_location: str, +) -> FlyteFile: + ... + + +@workflow +def reference_lp_wf() -> FlyteFile: + return normalize_csv_file( + csv_url="https://people.sc.fsu.edu/~jburkardt/data/csv/biostats.csv", + column_names=["Name", "Sex", "Age", "Heights (in)", "Weight (lbs)"], + columns_to_normalize=["Age"], + output_location="", + ) +``` + +It's important to verify that the workflow interface corresponds to that of the referenced workflow. + +:::{note} +The macro `{{ registration.version }}` is populated by `flytectl register` during registration. +Generally, it is unnecessary for reference launch plans, as it is preferable to bind to a specific version of the task or launch plan. +However, in this example, we are registering both the launch plan `core.flyte_basics.files.normalize_csv_file` and the workflow that references it. +Therefore, we need the macro to be updated to the version of a specific Flytesnacks release. +This is why `{{ registration.version }}` is used. + +A typical reference launch plan would resemble the following: + +```python +@reference_launch_plan( + project="flytesnacks", + domain="development", + name="core.flyte_basics.files.normalize_csv_file", + version="d06cebcfbeabc02b545eefa13a01c6ca992940c8", # If using GIT for versioning OR 0.16.0, if semver +) +def normalize_csv_file(...): + ... +``` +::: diff --git a/docs/user_guide/productionizing/reference_tasks.md b/docs/user_guide/productionizing/reference_tasks.md new file mode 100644 index 0000000000..057986d74a --- /dev/null +++ b/docs/user_guide/productionizing/reference_tasks.md @@ -0,0 +1,89 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +# Reference tasks + +```{eval-rst} +.. tags:: Intermediate +``` + +A {py:func}`flytekit.reference_task` references the Flyte tasks that have already been defined, serialized, and registered. +You can reference tasks from other projects and create workflows that use tasks declared by others. +These tasks can be in their own containers, python runtimes, flytekit versions, and even different languages. + +The following example illustrates how to use reference tasks. + +:::{note} +Reference tasks cannot be run locally. You must mock them out. +::: + +```{code-cell} +:lines_to_next_cell: 2 + +from typing import List + +from flytekit import reference_task, workflow +from flytekit.types.file import FlyteFile + + +@reference_task( + project="flytesnacks", + domain="development", + name="data_types_and_io.file.normalize_columns", + version="{{ registration.version }}", +) +def normalize_columns( + csv_url: FlyteFile, + column_names: List[str], + columns_to_normalize: List[str], + output_location: str, +) -> FlyteFile: + ... + + +@workflow +def wf() -> FlyteFile: + return normalize_columns( + csv_url="https://people.sc.fsu.edu/~jburkardt/data/csv/biostats.csv", + column_names=["Name", "Sex", "Age", "Heights (in)", "Weight (lbs)"], + columns_to_normalize=["Age"], + output_location="", + ) +``` + +:::{note} +The macro `{{ registration.version }}` is populated by `flytectl register` during registration. +Generally, it is unnecessary for reference tasks, as it is preferable to bind to a specific version of the task or launch plan. +However, in this example, we are registering both the task `core.flyte_basics.files.normalize_columns` and the workflow that references it. +Therefore, we need the macro to be updated to the version of a specific Flytesnacks release. +This is why `{{ registration.version }}` is used. + +A typical reference task would resemble the following: + +```python +@reference_task( + project="flytesnacks", + domain="development", + name="core.flyte_basics.files.normalize_columns", + version="d06cebcfbeabc02b545eefa13a01c6ca992940c8", # If using GIT for versioning OR 0.16.0, if semver + ) + def normalize_columns(...): + ... +``` +::: diff --git a/docs/user_guide/productionizing/schedules.md b/docs/user_guide/productionizing/schedules.md new file mode 100644 index 0000000000..0deda30d04 --- /dev/null +++ b/docs/user_guide/productionizing/schedules.md @@ -0,0 +1,227 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(scheduling_launch_plan)= + +# Schedules + +```{eval-rst} +.. tags:: Basic +``` + +{ref}`flyte:divedeep-launchplans` can be set to run automatically on a schedule using the Flyte Native Scheduler. +For workflows that depend on knowing the kick-off time, Flyte supports passing in the scheduled time (not the actual time, which may be a few seconds off) as an argument to the workflow. + +Check out a demo of how the Native Scheduler works: + +```{eval-rst} +.. youtube:: sQoCp2qSQK4 +``` + +:::{note} +Native scheduler doesn't support [AWS syntax](http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html#CronExpressions). +::: + ++++ {"lines_to_next_cell": 0} + +Consider the following example workflow: + +```{code-cell} +from datetime import datetime + +from flytekit import task, workflow + + +@task +def format_date(run_date: datetime) -> str: + return run_date.strftime("%Y-%m-%d %H:%M") + + +@workflow +def date_formatter_wf(kickoff_time: datetime): + formatted_kickoff_time = format_date(run_date=kickoff_time) + print(formatted_kickoff_time) +``` + ++++ {"lines_to_next_cell": 0} + +The `date_formatter_wf` workflow can be scheduled using either the `CronSchedule` or the `FixedRate` object. + +(cron-schedules)= + +## Cron schedules + +[Cron](https://en.wikipedia.org/wiki/Cron) expression strings use this {ref}`syntax `. +An incorrect cron schedule expression would lead to failure in triggering the schedule. + +```{code-cell} +from flytekit import CronSchedule, LaunchPlan # noqa: E402 + +# creates a launch plan that runs every minute. +cron_lp = LaunchPlan.get_or_create( + name="my_cron_scheduled_lp", + workflow=date_formatter_wf, + schedule=CronSchedule( + # Note that the ``kickoff_time_input_arg`` matches the workflow input we defined above: kickoff_time + # But in case you are using the AWS scheme of schedules and not using the native scheduler then switch over the schedule parameter with cron_expression + schedule="*/1 * * * *", # Following schedule runs every min + kickoff_time_input_arg="kickoff_time", + ), +) +``` + +The `kickoff_time_input_arg` corresponds to the workflow input `kickoff_time`. +Specifying this argument means that Flyte will pass in the kick-off time of the +cron schedule into the `kickoff_time` argument of the `date_formatter_wf` workflow. + ++++ + +## Fixed rate intervals + +If you prefer to use an interval rather than a cron scheduler to schedule your workflows, you can use the fixed-rate scheduler. +A fixed-rate scheduler runs at the specified interval. + +Here's an example: + +```{code-cell} +from datetime import timedelta # noqa: E402 + +from flytekit import FixedRate, LaunchPlan # noqa: E402 + + +@task +def be_positive(name: str) -> str: + return f"You're awesome, {name}" + + +@workflow +def positive_wf(name: str): + reminder = be_positive(name=name) + print(f"{reminder}") + + +fixed_rate_lp = LaunchPlan.get_or_create( + name="my_fixed_rate_lp", + workflow=positive_wf, + # Note that the workflow above doesn't accept any kickoff time arguments. + # We just omit the ``kickoff_time_input_arg`` from the FixedRate schedule invocation + schedule=FixedRate(duration=timedelta(minutes=10)), + fixed_inputs={"name": "you"}, +) +``` + +This fixed-rate scheduler runs every ten minutes. Similar to a cron scheduler, a fixed-rate scheduler also accepts `kickoff_time_input_arg` (which is omitted in this example). + +(activating-schedules)= + +## Activating a schedule + +After initializing your launch plan, [activate the specific version of the launch plan](https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_launchplan.html) so that the schedule runs. + +```bash +flytectl update launchplan -p flyteexamples -d development {{ name_of_lp }} --version --activate +``` + ++++ + +Verify if your launch plan was activated: + +```bash +flytectl get launchplan -p flytesnacks -d development +``` + ++++ + +## Deactivating a schedule + +You can [archive/deactivate the launch plan](https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_launchplan.html) to deschedule any scheduled job associated with it. + +```bash +flytectl update launchplan -p flyteexamples -d development {{ name_of_lp }} --version --archive +``` + ++++ + +## Platform configuration changes for AWS scheduler + +The Scheduling feature can be run using the Flyte native scheduler which comes with Flyte. If you intend to use the AWS scheduler then it requires additional infrastructure to run, so these will have to be created and configured. The following sections are only required if you use the AWS scheme for the scheduler. You can still run the Flyte native scheduler on AWS. + +### Setting up scheduled workflows + +To run workflow executions based on user-specified schedules, you'll need to fill out the top-level `scheduler` portion of the flyteadmin application configuration. + +In particular, you'll need to configure the two components responsible for scheduling workflows and processing schedule event triggers. + +:::{note} +This functionality is currently only supported for AWS installs. +::: + +#### Event scheduler + +To schedule workflow executions, you'll need to set up an [AWS SQS](https://aws.amazon.com/sqs/) queue. A standard-type queue should suffice. The flyteadmin event scheduler creates [AWS CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html) event rules that invoke your SQS queue as a target. + +With that in mind, let's take a look at an example `eventScheduler` config section and dive into what each value represents: + +```bash +scheduler: + eventScheduler: + scheme: "aws" + region: "us-east-1" + scheduleRole: "arn:aws:iam::{{ YOUR ACCOUNT ID }}:role/{{ ROLE }}" + targetName: "arn:aws:sqs:us-east-1:{{ YOUR ACCOUNT ID }}:{{ YOUR QUEUE NAME }}" + scheduleNamePrefix: "flyte" +``` + ++++ + +- **scheme**: in this case because AWS is the only cloud back-end supported for scheduling workflows, only `"aws"` is a valid value. By default, the no-op scheduler is used. +- **region**: this specifies which region initialized AWS clients should use when creating CloudWatch rules. +- **scheduleRole** This is the IAM role ARN with permissions set to `Allow` + : - `events:PutRule` + - `events:PutTargets` + - `events:DeleteRule` + - `events:RemoveTargets` +- **targetName** this is the ARN for the SQS Queue you've allocated to scheduling workflows. +- **scheduleNamePrefix** this is an entirely optional prefix used when creating schedule rules. Because of AWS naming length restrictions, scheduled rules are a random hash and having a shared prefix makes these names more readable and indicates who generated the rules. + +#### Workflow executor + +Scheduled events which trigger need to be handled by the workflow executor, which subscribes to triggered events from the SQS queue configured above. + +:::{NOTE} +Failure to configure a workflow executor will result in all your scheduled events piling up silently without ever kicking off workflow executions. +::: + +Again, let's break down a sample config: + +```bash +scheduler: + eventScheduler: + ... + workflowExecutor: + scheme: "aws" + region: "us-east-1" + scheduleQueueName: "{{ YOUR QUEUE NAME }}" + accountId: "{{ YOUR ACCOUNT ID }}" +``` + ++++ + +- **scheme**: in this case because AWS is the only cloud back-end supported for executing scheduled workflows, only `"aws"` is a valid value. By default, the no-op executor is used and in case of sandbox we use `"local"` scheme which uses the Flyte native scheduler. +- **region**: this specifies which region AWS clients should use when creating an SQS subscriber client. +- **scheduleQueueName**: this is the name of the SQS Queue you've allocated to scheduling workflows. +- **accountId**: Your AWS [account id](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#FindingYourAWSId). diff --git a/docs/user_guide/productionizing/secrets.md b/docs/user_guide/productionizing/secrets.md new file mode 100644 index 0000000000..dba145c483 --- /dev/null +++ b/docs/user_guide/productionizing/secrets.md @@ -0,0 +1,447 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +(secrets)= + +# Secrets + +```{eval-rst} +.. tags:: Kubernetes, Intermediate +``` + +Flyte supports running a variety of tasks, from containers to SQL queries and +service calls, and it provides a native Secret construct to request and access +secrets. + +This example explains how you can access secrets in a Flyte Task. Flyte provides +different types of secrets, but for users writing Python tasks, you can only access +secure secrets either as environment variables or as a file injected into the +running container. + ++++ + +## Creating secrets with a secrets manager + +:::{admonition} Prerequisites +:class: important + +- Install [kubectl](https://kubernetes.io/docs/tasks/tools/). +- Have access to a Flyte cluster, for e.g. with `flytectl demo start` as + described {ref}`here `. +::: + +The first step to using secrets in Flyte is to create one on the backend. +By default, Flyte uses the K8s-native secrets manager, which we'll use in this +example, but you can also {ref}`configure different secret managers `. + +First, we use `kubectl` to create a secret called `user-info` with a +`user_secret` key: + +```{eval-rst} +.. prompt:: bash $ + + kubectl create secret -n - generic user-info --from-literal=user_secret=mysecret +``` + +:::{note} +Be sure to specify the correct Kubernetes namespace when creating a secret. If you plan on accessing +the secret in the `flytesnacks` project under the `development` domain, replace `-` +with `flytesnacks-development`. This is because secrets need to be in the same namespace as the +workflow execution. +::: + +:::{important} +The imperative command above is useful for creating secrets in an ad hoc manner, +but it may not be the most secure or sustainable way to do so. You can, however, +define secrets using a [configuration file](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-config-file/) +or tools like [Kustomize](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kustomize/). +::: + ++++ + +## Using secrets in tasks + +Once you've defined a secret on the Flyte backend, `flytekit` exposes a class +called {py:class}`~flytekit.Secret`s, which allows you to request a secret +from the configured secret manager. + +```{code-cell} +import os +from typing import Tuple + +import flytekit +from flytekit import Secret, task, workflow +from flytekit.testing import SecretsManager + +secret = Secret( + group="", + key="", + mount_requirement=Secret.MountType.ENV_VAR, +) +``` + +Secrets consists of `group`, `key`, and `mounting_requirement` arguments, +where a secret group can have multiple secrets associated with it. +If the `mounting_requirement` argument is not specified, the secret will +be injected as an environment variable by default. + +In the code below we specify two variables, `SECRET_GROUP` and +`SECRET_NAME`, which maps onto the `user-info` secret that we created +with `kubectl` above, with a key called `user_secret`. + +```{code-cell} +SECRET_GROUP = "user-info" +SECRET_NAME = "user_secret" +``` + +Now we declare the secret in the `secret_requests` argument of the +{py:func}`@task ` decorator. The request tells Flyte to make +the secret available to the task. + +The secret can then be accessed inside the task using the +{py:class}`~flytekit.ExecutionParameters` object, which is returned by +invoking the {py:func}`flytekit.current_context` function, as shown below. + +At runtime, flytekit looks inside the task pod for an environment variable or +a mounted file with a predefined name/path and loads the value. + +```{code-cell} +@task(secret_requests=[Secret(group=SECRET_GROUP, key=SECRET_NAME)]) +def secret_task() -> str: + context = flytekit.current_context() + secret_val = context.secrets.get(SECRET_GROUP, SECRET_NAME) + print(secret_val) + return secret_val +``` + +:::{warning} +Never print secret values! The example above is just for demonstration purposes. +::: + +:::{note} +- In case Flyte fails to access the secret, an error is raised. +- The `Secret` group and key are required parameters during declaration + and usage. Failure to specify will cause a {py:class}`ValueError`. +::: + +### Multiple keys grouped into one secret + +In some cases you may have multiple secrets and sometimes, they maybe grouped +as one secret in the SecretStore. + +For example, In Kubernetes secrets, it is possible to nest multiple keys under +the same secret: + +```{eval-rst} +.. prompt:: bash $ + + kubectl create secret generic user-info \ + --from-literal=user_secret=mysecret \ + --from-literal=username=my_username \ + --from-literal=password=my_password +``` + +In this case, the secret group will be `user-info`, with three available +secret keys: `user_secret`, `username`, and `password`. + +```{code-cell} +USERNAME_SECRET = "username" +PASSWORD_SECRET = "password" +``` + ++++ {"lines_to_next_cell": 0} + +The Secret structure allows passing two fields, matching the key and the group, as previously described: + +```{code-cell} +@task( + secret_requests=[ + Secret(key=USERNAME_SECRET, group=SECRET_GROUP), + Secret(key=PASSWORD_SECRET, group=SECRET_GROUP), + ] +) +def user_info_task() -> Tuple[str, str]: + context = flytekit.current_context() + secret_username = context.secrets.get(SECRET_GROUP, USERNAME_SECRET) + secret_pwd = context.secrets.get(SECRET_GROUP, PASSWORD_SECRET) + print(f"{secret_username}={secret_pwd}") + return secret_username, secret_pwd +``` + +:::{warning} +Never print secret values! The example above is just for demonstration purposes. +::: + +### Mounting secrets as files or environment variables + +It is also possible to make Flyte mount the secret as a file or an environment +variable. + +The file type is useful for large secrets that do not fit in environment variables, +which are typically asymmetric keys (like certs, etc). Another reason may be that a +dependent library requires the secret to be available as a file. +In these scenarios you can specify the `mount_requirement=Secret.MountType.FILE`. + +In the following example we force the mounting to be an environment variable: + +```{code-cell} +@task( + secret_requests=[ + Secret( + group=SECRET_GROUP, + key=SECRET_NAME, + mount_requirement=Secret.MountType.ENV_VAR, + ) + ] +) +def secret_file_task() -> Tuple[str, str]: + secret_manager = flytekit.current_context().secrets + + # get the secrets filename + f = secret_manager.get_secrets_file(SECRET_GROUP, SECRET_NAME) + + # get secret value from an environment variable + secret_val = secret_manager.get(SECRET_GROUP, SECRET_NAME) + + # returning the filename and the secret_val + return f, secret_val +``` + ++++ {"lines_to_next_cell": 0} + +These tasks can be used in your workflow as usual + +```{code-cell} +@workflow +def my_secret_workflow() -> Tuple[str, str, str, str, str]: + x = secret_task() + y, z = user_info_task() + f, s = secret_file_task() + return x, y, z, f, s +``` + +### Testing with mock secrets + +The simplest way to test secret accessibility is to export the secret as an +environment variable. There are some helper methods available to do so: + +```{code-cell} +if __name__ == "__main__": + sec = SecretsManager() + os.environ[sec.get_secrets_env_var(SECRET_GROUP, SECRET_NAME)] = "value" + os.environ[sec.get_secrets_env_var(SECRET_GROUP, USERNAME_SECRET)] = "username_value" + os.environ[sec.get_secrets_env_var(SECRET_GROUP, PASSWORD_SECRET)] = "password_value" + x, y, z, f, s = my_secret_workflow() + assert x == "value" + assert y == "username_value" + assert z == "password_value" + assert f == sec.get_secrets_file(SECRET_GROUP, SECRET_NAME) + assert s == "value" +``` + +## Using secrets in task templates + +For task types that connect to a remote database, you'll need to specify +secret request as well. For example, for the {py:class}`~flytekitplugins.sqlalchemy.SQLAlchemyTask` +you need to: + +1. Specify the `secret_requests` argument. +2. Configure the {py:class}`~flytekitplugins.sqlalchemy.SQLAlchemyConfig` to + declare which secret maps onto which connection argument. + +```python +from flytekit import kwtypes +from flytekitplugins.sqlalchemy import SQLAlchemyTask, SQLAlchemyConfig + + +# define the secrets +secrets = { + "username": Secret(group="", key=""), + "password": Secret(group="", key=""), +} + + +sql_query = SQLAlchemyTask( + name="sql_query", + query_template="""SELECT * FROM my_table LIMIT {{ .inputs.limit }}""", + inputs=kwtypes(limit=int), + + # request secrets + secret_requests=[*secrets.values()], + + # specify username and password credentials in the configuration + task_config=SQLAlchemyConfig( + uri="", + secret_connect_args=secrets, + ), +) +``` + ++++ + +:::{note} +Here the `secret_connect_args` map to the +[SQLAlchemy engine configuration](https://docs.sqlalchemy.org/en/20/core/engines.html) +argument names for the username and password. +::: + +You can then use the `sql_query` task inside a workflow to grab data and +perform downstream transformations on it. + ++++ + +## How secrets injection works + +The rest of this page describes how secrets injection works under the hood. +For a simple task that launches a Pod, the flow would look something like this: + +```{image} https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gICAgUHJvcGVsbGVyLT4-K1BsdWdpbnM6IENyZWF0ZSBLOHMgUmVzb3VyY2VcbiAgICBQbHVnaW5zLT4-LVByb3BlbGxlcjogUmVzb3VyY2UgT2JqZWN0XG4gICAgUHJvcGVsbGVyLT4-K1Byb3BlbGxlcjogU2V0IExhYmVscyAmIEFubm90YXRpb25zXG4gICAgUHJvcGVsbGVyLT4-K0FwaVNlcnZlcjogQ3JlYXRlIE9iamVjdCAoZS5nLiBQb2QpXG4gICAgQXBpU2VydmVyLT4-K1BvZCBXZWJob29rOiAvbXV0YXRlXG4gICAgUG9kIFdlYmhvb2stPj4rUG9kIFdlYmhvb2s6IExvb2t1cCBnbG9iYWxzXG4gICAgUG9kIFdlYmhvb2stPj4rUG9kIFdlYmhvb2s6IEluamVjdCBTZWNyZXQgQW5ub3RhdGlvbnMgKGUuZy4gSzhzLCBWYXVsdC4uLiBldGMuKVxuICAgIFBvZCBXZWJob29rLT4-LUFwaVNlcnZlcjogTXV0YXRlZCBQb2RcbiAgICBcbiAgICAgICAgICAgICIsIm1lcm1haWQiOnt9LCJ1cGRhdGVFZGl0b3IiOmZhbHNlfQ +:target: https://mermaid.ink/img/eyJjb2RlIjoic2VxdWVuY2VEaWFncmFtXG4gICAgUHJvcGVsbGVyLT4-K1BsdWdpbnM6IENyZWF0ZSBLOHMgUmVzb3VyY2VcbiAgICBQbHVnaW5zLT4-LVByb3BlbGxlcjogUmVzb3VyY2UgT2JqZWN0XG4gICAgUHJvcGVsbGVyLT4-K1Byb3BlbGxlcjogU2V0IExhYmVscyAmIEFubm90YXRpb25zXG4gICAgUHJvcGVsbGVyLT4-K0FwaVNlcnZlcjogQ3JlYXRlIE9iamVjdCAoZS5nLiBQb2QpXG4gICAgQXBpU2VydmVyLT4-K1BvZCBXZWJob29rOiAvbXV0YXRlXG4gICAgUG9kIFdlYmhvb2stPj4rUG9kIFdlYmhvb2s6IExvb2t1cCBnbG9iYWxzXG4gICAgUG9kIFdlYmhvb2stPj4rUG9kIFdlYmhvb2s6IEluamVjdCBTZWNyZXQgQW5ub3RhdGlvbnMgKGUuZy4gSzhzLCBWYXVsdC4uLiBldGMuKVxuICAgIFBvZCBXZWJob29rLT4-LUFwaVNlcnZlcjogTXV0YXRlZCBQb2RcbiAgICBcbiAgICAgICAgICAgICIsIm1lcm1haWQiOnt9LCJ1cGRhdGVFZGl0b3IiOmZhbHNlfQ +``` + +Breaking down this sequence diagram: + +1. Flyte invokes a plugin to create the K8s object. This can be a Pod or a more complex CRD (e.g. Spark, PyTorch, etc.) + + :::{note} + The plugin ensures that the labels and annotations are passed to any Pod that is spawned due to the creation of the CRD. + ::: + +2. Flyte applies labels and annotations that are referenced to all secrets the task is requesting access to. Note that secrets are not case sensitive. + +3. Flyte sends a `POST` request to `ApiServer` to create the object. + +4. Before persisting the Pod, `ApiServer` invokes all the registered Pod Webhooks and Flyte's Pod Webhook is called. + +5. Using the labels and annotiations attached in **step 2**, Flyte Pod Webhook looks up globally mounted secrets for each of the requested secrets. + +6. If found, the Pod Webhook mounts them directly in the Pod. If not found, the Pod Webhook injects the appropriate annotations to load the secrets for K8s (or Vault or Confidant or any secret management system plugin configured) into the task pod. + +Once the secret is injected into the task pod, Flytekit can read it using the secret manager. + +The webhook is included in all overlays in the Flytekit repo. The deployment file creates two things; a **Job** and a **Deployment**. + +1. `flyte-pod-webhook-secrets` **Job**: This job runs `flytepropeller webhook init-certs` command that issues self-signed CA Certificate as well as a derived TLS certificate and its private key. Ensure that the private key is in lower case, that is, `my_token` in contrast to `MY_TOKEN`. It stores them into a new secret `flyte-pod-webhook-secret`. +2. `flyte-pod-webhook` **Deployment**: This deployment creates the Webhook pod which creates a MutatingWebhookConfiguration on startup. This serves as the registration contract with the ApiServer to know about the Webhook before it starts serving traffic. + +## Secret discovery + +Flyte identifies secrets using a secret group and a secret key, which can +be accessed by {py:func}`flytekit.current_context` in the task function +body, as shown in the code examples above. + +Flytekit relies on the following environment variables to load secrets (defined [here](https://github.com/flyteorg/flytekit/blob/9d313429c577a919ec0ad4cd397a5db356a1df0d/flytekit/configuration/internal.py#L141-L159)). When running tasks and workflows locally you should make sure to store your secrets accordingly or to modify these: + +- `FLYTE_SECRETS_DEFAULT_DIR`: The directory Flytekit searches for secret files. **Default:** `"/etc/secrets"` +- `FLYTE_SECRETS_FILE_PREFIX`: a common file prefix for Flyte secrets. **Default:** `""` +- `FLYTE_SECRETS_ENV_PREFIX`: a common env var prefix for Flyte secrets. **Default:** `"_FSEC_"` + +When running a workflow on a Flyte cluster, the configured secret manager will use the secret Group and Key to try and retrieve a secret. +If successful, it will make the secret available as either file or environment variable and will if necessary modify the above variables automatically so that the task can load and use the secrets. + +(configure_secret_management)= + +## Configuring a secret management system plugin + +When a task requests a secret, Flytepropeller will try to retrieve secrets in the following order: + +1. Checking for global secrets, i.e. secrets mounted as files or environment variables on the `flyte-pod-webhook` pod +2. Checking with an additional configurable secret manager. + +:::{important} +The global secrets take precedence over any secret discoverable by the secret manager plugins. +::: + +The following secret managers are available at the time of writing: + +- [K8s secrets](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret) (**default**): `flyte-pod-webhook` will try to look for a K8s secret named after the secret Group and retrieve the value for the secret Key. +- [AWS Secret Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html): `flyte-pod-webhook` will add the AWS Secret Manager sidecar container to a task Pod which will mount the secret. +- [Vault Agent Injector](https://developer.hashicorp.com/vault/tutorials/getting-started/getting-started-first-secret#write-a-secret) : `flyte-pod-webhook` will annotate the task Pod with the respective Vault annotations that trigger an existing Vault Agent Injector to retrieve the specified secret Key from a vault path defined as secret Group. + +You can configure the additional secret manager by defining `secretManagerType` to be either 'K8s', 'AWS' or 'Vault' in +the [core config](https://github.com/flyteorg/flyte/blob/master/kustomize/base/single_cluster/headless/config/propeller/core.yaml#L34) of the Flytepropeller. + +When using the K8s secret manager plugin, which is enabled by default, the secrets need to be available in the same namespace as the task execution +(for example `flytesnacks-development`). K8s secrets can be mounted as either files or injected as environment variables into the task pod, +so if you need to make larger files available to the task, then this might be the better option. + +Furthermore, this method also allows you to have separate credentials for different domains but still using the same name for the secret. + +### AWS secrets manager + +When using the AWS secret management plugin, secrets need to be specified by naming them in the format +`:`, where the secret string is a plain-text value, **not** key/value json. + +### Vault secrets manager + +When using the Vault secret manager, make sure you have Vault Agent deployed on your cluster as described in this +[step-by-step tutorial](https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar). +Vault secrets can only be mounted as files and will become available under `"/etc/flyte/secrets/SECRET_GROUP/SECRET_NAME"`. + +Vault comes with various secrets engines. Currently Flyte supports working with both version 1 and 2 of the `Key Vault engine ` as well as the `databases secrets engine `. +You can use use the `group_version` parameter to specify which secret backend engine to use. Available choices are: "kv1", "kv2", "db": + ++++ {"lines_to_next_cell": 0} + +How to request secrets with the Vault secret manager + +```{code-cell} +secret = Secret( + group="", + key="", + group_version="", +) +``` + +The group parameter is used to specify the path to the secret in the Vault backend. For example, if you have a secret stored in Vault at `"secret/data/flyte/secret"` then the group parameter should be `"secret/data/flyte"`. +When using either of the Key Vault engine versions, the secret key is the name of a specific secret entry to be retrieved from the group path. +When using the database secrets engine, the secret key itself is arbitrary but is required by Flyte to name and identify the secret file. It is arbitrary because the database secrets engine returns always two keys, `username` and `password` and we need to retrieve a matching pair in one request. + +**Configuration** + +You can configure the Vault role under which Flyte will try to read the secret by setting webhook.vaultSecretManager.role (default: `"flyte"`). +There is also a deprecated `webhook.vaultSecretManager.kvVersion` setting in the configmap that can be used to specify the version but only for the Key Vault backend engine. +Available choices are: "1", "2". Note that the version number needs to be an explicit string (e.g. `"1"`). + +**Annotations** + +By default, `flyte-pod-webhook` injects following annotations to task pod: + +1. `vault.hashicorp.com/agent-inject` to configure whether injection is explicitly enabled or disabled for a pod. +2. `vault.hashicorp.com/secret-volume-path` to configure where on the filesystem a secret will be rendered. +3. `vault.hashicorp.com/role` to configure the Vault role used by the Vault Agent auto-auth method. +4. `vault.hashicorp.com/agent-pre-populate-only` to configure whether an init container is the only injected container. +5. `vault.hashicorp.com/agent-inject-secret` to configure Vault Agent to retrieve the secrets from Vault required by the container. +6. `vault.hashicorp.com/agent-inject-file` to configure the filename and path in the secrets volume where a Vault secret will be written. +7. `vault.hashicorp.com/agent-inject-template` to configure the template Vault Agent should use for rendering a secret. + +It is possible to add extra annotations or override the existing ones in Flyte either at the task level using pod annotations or at the installation level. +If Flyte administrator wants to set up annotations for the entire system, they can utilize `webhook.vaultSecretManager.annotations` to accomplish this. + +## Scaling the webhook + +### Vertical scaling + +To scale the Webhook to be able to process the number/rate of pods you need, you may need to configure a vertical [pod +autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler). + +### Horizontal scaling + +The Webhook does not make any external API Requests in response to Pod mutation requests. It should be able to handle traffic +quickly. For horizontal scaling, adding additional replicas for the Pod in the +deployment should be sufficient. A single `MutatingWebhookConfiguration` object will be used, the same TLS certificate +will be shared across the pods and the Service created will automatically load balance traffic across the available pods. diff --git a/docs/user_guide/productionizing/spot_instances.md b/docs/user_guide/productionizing/spot_instances.md new file mode 100644 index 0000000000..85024f6996 --- /dev/null +++ b/docs/user_guide/productionizing/spot_instances.md @@ -0,0 +1,92 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + ++++ {"lines_to_next_cell": 0} + +# Spot instances + +```{eval-rst} +.. tags:: AWS, GCP, Intermediate + +``` + ++++ + +## What are spot instances? + +Spot instances are unused EC2 capacity in AWS. [Spot instances](https://aws.amazon.com/ec2/spot/?cards.sort-by=item.additionalFields.startDateTime&cards.sort-order=asc) can result in up to 90% savings on on-demand prices. The caveat is that these instances can be preempted at any point and no longer be available for use. This can happen due to: + +- Price – The spot price is greater than your maximum price. +- Capacity – If there are not enough unused EC2 instances to meet the demand for spot instances, Amazon EC2 interrupts spot instances. Amazon EC2 determines the order in which the instances are interrupted. +- Constraints – If your request includes a constraint such as a launch group or an Availability Zone group, these spot instances are terminated as a group when the constraint can no longer be met. + +Generally, most spot instances are obtained for around 2 hours (median), with the floor being about 20 minutes and the ceiling of unbounded duration. + +:::{note} +Spot Instances are called `Preemptible Instances` in the GCP terminology. +::: + +### Setting up spot instances + +- AWS: +- GCP: + +If an auto-scaling group (ASG) is set up, you may want to isolate the tasks you want to trigger on spot/preemptible instances from the regular workloads. +This can be done by setting taints and tolerations using the [config](https://github.com/flyteorg/flyteplugins/blob/60b94c688ef2b98aa53a9224b529ac672af04540/go/tasks/pluginmachinery/flytek8s/config/config.go#L84-L92) available at `flyteorg/flyteplugins` repo. + +:::{admonition} What's an ASG for a spot/preemptible instance? +When your spot/preemptible instance is terminated, ASG attempts to launch a replacement instance to maintain the desired capacity for the group. +::: + ++++ + +## What are interruptible tasks? + +If specified, the `interruptible flag` is added to the task definition and signals to the Flyte engine that it may be scheduled on machines that may be preempted, such as AWS spot instances. This is low-hanging fruit for any cost-savings initiative. + +### Setting interruptible tasks + +To run your workload on a spot/preemptible instance, you can set interruptible to `True`. In case you would like to automatically retry in case the node gets preemted, please also make sure to set at least one retry. For example: + +```python +@task(cache_version='1', interruptible=True, retries=1) +def add_one_and_print(value_to_print: int) -> int: + return value_to_print + 1 +``` + ++++ + +By setting this value, Flyte will schedule your task on an auto-scaling group (ASG) with only spot instances. + +:::{note} +If you set `retries=n`, for instance, and the task gets preempted repeatedly, Flyte will retry on a preemptible/spot instance `n-1` times and for the last attempt will retry your task on a non-spot (regular) instance. Please note that tasks will only be retried if at least one retry is allowed using the `retries` parameter in the `task` decorator. +::: + +### Which tasks should be set to interruptible? + +Most Flyte workloads should be good candidates for spot instances. +If your task does NOT exhibit the following properties, you can set `interruptible` to true. + +- Time-sensitive: It needs to run now and can not have any unexpected delays. +- Side Effects: The task is not idempotent, and retrying will cause issues. +- Long-Running Tasks: The task takes > 2 hours. Having an interruption during this time frame could potentially waste a lot of computation. + +In a nutshell, you should use spot/preemptible instances when you want to reduce the total cost of running jobs at the expense of potential delays in execution due to restarts. + ++++ + +% TODO: Write "How to Recover From Interruptions?" section diff --git a/docs/user_guide/productionizing/workflow_labels_and_annotations.md b/docs/user_guide/productionizing/workflow_labels_and_annotations.md new file mode 100644 index 0000000000..5b9fe6d4c6 --- /dev/null +++ b/docs/user_guide/productionizing/workflow_labels_and_annotations.md @@ -0,0 +1,75 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Workflow labels and annotations + +```{eval-rst} +.. tags:: Kubernetes, Intermediate +``` + +In Flyte, workflow executions are created as Kubernetes resources. These can be extended with +[labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) and +[annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). + +**Labels** and **annotations** are key value pairs which can be used to identify workflows for your own uses. + +:::{Warning} +Note that adding labels and annotations to your K8s resources may have side-effects depending on webhook behavior on your execution clusters. +::: + +Labels are meant to be used as identifying attributes, whereas annotations are arbitrary, *non-identifying* metadata. + +Using labels and annotations is entirely optional. They can be used to categorize and identify workflow executions. + +Labels and annotations are optional parameters to launch plan and execution invocations. When an execution +defines labels and/or annotations *and* the launch plan does as well, the execution spec values will be preferred. + +## Launch plan usage example + +```python +from flytekit import Labels, Annotations + +@workflow +class MyWorkflow(object): + ... + +my_launch_plan = MyWorkflow.create_launch_plan( + labels=Labels({"myexecutionlabel": "bar", ...}), + annotations=Annotations({"region": "SEA", ...}), + ... +) + +my_launch_plan.execute(...) +``` + +## Execution example + +```python +from flytekit import Labels, Annotations + +@workflow +class MyWorkflow(object): + ... + +my_launch_plan = MyWorkflow.create_launch_plan(...) + +my_launch_plan.execute( + labels=Labels({"myexecutionlabel": "bar", ...}), + annotations=Annotations({"region": "SEA", ...}), + ... +) +``` diff --git a/docs/user_guide/testing/index.md b/docs/user_guide/testing/index.md new file mode 100644 index 0000000000..80d3f6ef4e --- /dev/null +++ b/docs/user_guide/testing/index.md @@ -0,0 +1,16 @@ +(testing)= + +# Testing + +The `flytekit` python SDK provides a few utilities for making it easier to test +your tasks and workflows in your test suite. For more details, you can also refer +to the {py:mod}`~flytekit.testing` module in the API reference. + + +```{toctree} +:maxdepth: -1 +:name: testing_toc +:hidden: + +mocking_tasks +``` diff --git a/docs/user_guide/testing/mocking_tasks.md b/docs/user_guide/testing/mocking_tasks.md new file mode 100644 index 0000000000..07b4f0239e --- /dev/null +++ b/docs/user_guide/testing/mocking_tasks.md @@ -0,0 +1,101 @@ +--- +jupytext: + cell_metadata_filter: all + formats: md:myst + main_language: python + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 + language: python + name: python3 +--- + +# Mocking tasks + +A lot of the tasks that you write you can run locally, but some of them you will not be able to, usually because they +are tasks that depend on a third-party only available on the backend. Hive tasks are a common example, as most users +will not have access to the service that executes Hive queries from their development environment. However, it's still +useful to be able to locally run a workflow that calls such a task. In these instances, flytekit provides a couple +of utilities to help navigate this. + +```{code-cell} +import datetime + +import pandas +from flytekit import SQLTask, TaskMetadata, kwtypes, task, workflow +from flytekit.testing import patch, task_mock +from flytekit.types.schema import FlyteSchema +``` + ++++ {"lines_to_next_cell": 0} + +This is a generic SQL task (and is by default not hooked up to any datastore nor handled by any plugin), and must +be mocked. + +```{code-cell} +sql = SQLTask( + "my-query", + query_template="SELECT * FROM hive.city.fact_airport_sessions WHERE ds = '{{ .Inputs.ds }}' LIMIT 10", + inputs=kwtypes(ds=datetime.datetime), + outputs=kwtypes(results=FlyteSchema), + metadata=TaskMetadata(retries=2), +) +``` + ++++ {"lines_to_next_cell": 0} + +This is a task that can run locally + +```{code-cell} +@task +def t1() -> datetime.datetime: + return datetime.datetime.now() +``` + ++++ {"lines_to_next_cell": 0} + +Declare a workflow that chains these two tasks together. + +```{code-cell} +@workflow +def my_wf() -> FlyteSchema: + dt = t1() + return sql(ds=dt) +``` + ++++ {"lines_to_next_cell": 0} + +Without a mock, calling the workflow would typically raise an exception, but with the `task_mock` construct, which +returns a `MagicMock` object, we can override the return value. + +```{code-cell} +def main_1(): + with task_mock(sql) as mock: + mock.return_value = pandas.DataFrame(data={"x": [1, 2], "y": ["3", "4"]}) + assert (my_wf().open().all() == pandas.DataFrame(data={"x": [1, 2], "y": ["3", "4"]})).all().all() +``` + ++++ {"lines_to_next_cell": 0} + +There is another utility as well called `patch` which offers the same functionality, but in the traditional Python +patching style, where the first argument is the `MagicMock` object. + +```{code-cell} +def main_2(): + @patch(sql) + def test_user_demo_test(mock_sql): + mock_sql.return_value = pandas.DataFrame(data={"x": [1, 2], "y": ["3", "4"]}) + assert (my_wf().open().all() == pandas.DataFrame(data={"x": [1, 2], "y": ["3", "4"]})).all().all() + + test_user_demo_test() + + +if __name__ == "__main__": + main_1() + main_2() +``` diff --git a/flyte-single-binary-local.yaml b/flyte-single-binary-local.yaml index c369424795..bd8964da35 100644 --- a/flyte-single-binary-local.yaml +++ b/flyte-single-binary-local.yaml @@ -85,3 +85,11 @@ storage: access_key_id: minio secret_key: miniostorage container: my-s3-bucket + +task_resources: + defaults: + cpu: 500m + memory: 500Mi + limits: + cpu: 800m + memory: 800Mi diff --git a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go index 732c00666a..6a4a3b137f 100644 --- a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go +++ b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go @@ -54,12 +54,18 @@ func ValidateLaunchPlan(ctx context.Context, if err := validateSchedule(request, expectedInputs); err != nil { return err } + // Augment default inputs with the unbound workflow inputs. request.Spec.DefaultInputs = expectedInputs if request.Spec.EntityMetadata != nil { if err := validateNotifications(request.Spec.EntityMetadata.Notifications); err != nil { return err } + if request.GetSpec().GetEntityMetadata().GetLaunchConditions() != nil { + return errors.NewFlyteAdminErrorf( + codes.InvalidArgument, + "Launch condition must be empty, found %v", request.GetSpec().GetEntityMetadata().GetLaunchConditions()) + } } return nil } diff --git a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go index 8c2c768d42..86bfc5c6b7 100644 --- a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go +++ b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go @@ -317,7 +317,7 @@ func TestValidateSchedule_ArgNotFixed(t *testing.T) { }) t.Run("with rate", func(t *testing.T) { request := testutils.GetLaunchPlanRequestWithFixedRateSchedule(2, admin.FixedRateUnit_HOUR) - + err := validateSchedule(request, inputMap) assert.NotNil(t, err) }) diff --git a/flyteadmin/pkg/repositories/config/migrations.go b/flyteadmin/pkg/repositories/config/migrations.go index 891a80db1d..a3e5484a7d 100644 --- a/flyteadmin/pkg/repositories/config/migrations.go +++ b/flyteadmin/pkg/repositories/config/migrations.go @@ -1182,7 +1182,48 @@ var NoopMigrations = []*gormigrate.Migration{ }, } -var Migrations = append(LegacyMigrations, NoopMigrations...) +// ContinuedMigrations - Above are a series of migrations labeled as no-op migrations. These are migrations that we +// wrote to bring the then-existing migrations up to the Gorm standard, which is to write from scratch, each struct +// that we want auto-migrated, inside each function. Previously we had not been doing that. The idea is that we will +// one day delete the migrations prior to the no-op series. New migrations should continue in this array here, again +// using the proper Gorm methodology of including the struct definitions inside each migration function. +var ContinuedMigrations = []*gormigrate.Migration{ + { + ID: "pg-continue-2024-02-launchplan", + Migrate: func(tx *gorm.DB) error { + type LaunchPlanScheduleType string + type LaunchConditionType string + + type LaunchPlan struct { + ID uint `gorm:"index;autoIncrement;not null"` + CreatedAt time.Time `gorm:"type:time"` + UpdatedAt time.Time `gorm:"type:time"` + DeletedAt *time.Time `gorm:"index"` + Project string `gorm:"primary_key;index:lp_project_domain_name_idx,lp_project_domain_idx" valid:"length(0|255)"` + Domain string `gorm:"primary_key;index:lp_project_domain_name_idx,lp_project_domain_idx" valid:"length(0|255)"` + Name string `gorm:"primary_key;index:lp_project_domain_name_idx" valid:"length(0|255)"` + Version string `gorm:"primary_key" valid:"length(0|255)"` + Spec []byte `gorm:"not null"` + WorkflowID uint `gorm:"index"` + Closure []byte `gorm:"not null"` + // GORM doesn't save the zero value for ints, so we use a pointer for the State field + State *int32 `gorm:"default:0"` + // Hash of the launch plan + Digest []byte + ScheduleType LaunchPlanScheduleType + // store the type of event that this launch plan is triggered by, can be empty, or SCHED + LaunchConditionType *LaunchConditionType `gorm:"type:varchar(32);index:idx_launch_plans_launch_conditions,where:launch_condition_type is not null"` + } + return tx.AutoMigrate(&LaunchPlan{}) + }, + Rollback: func(tx *gorm.DB) error { + return tx.Table("launch_plans").Migrator().DropColumn(&models.LaunchPlan{}, "launch_condition_type") + }, + }, +} + +var m = append(LegacyMigrations, NoopMigrations...) +var Migrations = append(m, ContinuedMigrations...) func alterTableColumnType(db *sql.DB, columnName, columnType string) error { var err error diff --git a/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo_test.go b/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo_test.go index b43b9d75ff..05487fe4c9 100644 --- a/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo_test.go +++ b/flyteadmin/pkg/repositories/gormimpl/launch_plan_repo_test.go @@ -264,7 +264,7 @@ func TestListLaunchPlans_Pagination(t *testing.T) { GlobalMock := mocket.Catcher.Reset() GlobalMock.NewMock().WithQuery( - `SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 LIMIT 2 OFFSET 1`).WithReply(launchPlans) + `SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type","launch_plans"."launch_condition_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 LIMIT 2 OFFSET 1`).WithReply(launchPlans) collection, err := launchPlanRepo.List(context.Background(), interfaces.ListResourceInput{ InlineFilters: []common.InlineFilter{ @@ -311,7 +311,7 @@ func TestListLaunchPlans_Filters(t *testing.T) { GlobalMock := mocket.Catcher.Reset() GlobalMock.Logging = true // Only match on queries that append the name filter - GlobalMock.NewMock().WithQuery(`SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 AND launch_plans.version = $4 LIMIT 20`).WithReply(launchPlans[0:1]) + GlobalMock.NewMock().WithQuery(`SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type","launch_plans"."launch_condition_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 AND launch_plans.version = $4 LIMIT 20`).WithReply(launchPlans[0:1]) collection, err := launchPlanRepo.List(context.Background(), interfaces.ListResourceInput{ InlineFilters: []common.InlineFilter{ @@ -403,8 +403,8 @@ func TestListLaunchPlansForWorkflow(t *testing.T) { // HACK: gorm orders the filters on join clauses non-deterministically. Ordering of filters doesn't affect // correctness, but because the mocket library only pattern matches on substrings, both variations of the (valid) // SQL that gorm produces are checked below. - query := `SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 AND workflows.deleted_at = $4 LIMIT 20` - alternateQuery := `SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 AND workflows.deleted_at = $4 LIMIT 20` + query := `SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type","launch_plans"."launch_condition_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 AND workflows.deleted_at = $4 LIMIT 20` + alternateQuery := `SELECT "launch_plans"."id","launch_plans"."created_at","launch_plans"."updated_at","launch_plans"."deleted_at","launch_plans"."project","launch_plans"."domain","launch_plans"."name","launch_plans"."version","launch_plans"."spec","launch_plans"."workflow_id","launch_plans"."closure","launch_plans"."state","launch_plans"."digest","launch_plans"."schedule_type","launch_plans"."launch_condition_type" FROM "launch_plans" inner join workflows on launch_plans.workflow_id = workflows.id WHERE launch_plans.project = $1 AND launch_plans.domain = $2 AND launch_plans.name = $3 AND workflows.deleted_at = $4 LIMIT 20` GlobalMock.NewMock().WithQuery(query).WithReply(launchPlans) GlobalMock.NewMock().WithQuery(alternateQuery).WithReply(launchPlans) diff --git a/flyteadmin/pkg/repositories/models/launch_plan.go b/flyteadmin/pkg/repositories/models/launch_plan.go index 1a12dd259a..75d4c63669 100644 --- a/flyteadmin/pkg/repositories/models/launch_plan.go +++ b/flyteadmin/pkg/repositories/models/launch_plan.go @@ -19,7 +19,14 @@ const ( LaunchPlanScheduleTypeRATE LaunchPlanScheduleType = "RATE" ) -// Database model to encapsulate a launch plan. +type LaunchConditionType string + +const ( + // LaunchConditionTypeSCHED is the const representing the launch plan has a trigger type of schedule + LaunchConditionTypeSCHED LaunchConditionType = "SCHED" +) + +// LaunchPlan Database model to encapsulate a launch plan. type LaunchPlan struct { BaseModel LaunchPlanKey @@ -29,8 +36,9 @@ type LaunchPlan struct { // GORM doesn't save the zero value for ints, so we use a pointer for the State field State *int32 `gorm:"default:0"` // Hash of the launch plan - Digest []byte - ScheduleType LaunchPlanScheduleType + Digest []byte + ScheduleType LaunchPlanScheduleType + LaunchConditionType *LaunchConditionType `gorm:"type:varchar(32);index:idx_launch_plans_launch_conditions,where:launch_condition_type is not null"` } var LaunchPlanColumns = modelColumns(LaunchPlan{}) diff --git a/flyteadmin/pkg/repositories/transformers/launch_plan.go b/flyteadmin/pkg/repositories/transformers/launch_plan.go index f11cdacb5f..f716f3cc4e 100644 --- a/flyteadmin/pkg/repositories/transformers/launch_plan.go +++ b/flyteadmin/pkg/repositories/transformers/launch_plan.go @@ -40,18 +40,21 @@ func CreateLaunchPlanModel( return models.LaunchPlan{}, errors.NewFlyteAdminError(codes.Internal, "Failed to serialize launch plan closure") } + var launchConditionType models.LaunchConditionType scheduleType := models.LaunchPlanScheduleTypeNONE if launchPlan.Spec.EntityMetadata != nil && launchPlan.Spec.EntityMetadata.Schedule != nil { if launchPlan.Spec.EntityMetadata.Schedule.GetCronExpression() != "" || launchPlan.Spec.EntityMetadata.Schedule.GetCronSchedule() != nil { scheduleType = models.LaunchPlanScheduleTypeCRON + launchConditionType = models.LaunchConditionTypeSCHED } else if launchPlan.Spec.EntityMetadata.Schedule.GetRate() != nil { scheduleType = models.LaunchPlanScheduleTypeRATE + launchConditionType = models.LaunchConditionTypeSCHED } } state := int32(initState) - return models.LaunchPlan{ + lpModel := models.LaunchPlan{ LaunchPlanKey: models.LaunchPlanKey{ Project: launchPlan.Id.Project, Domain: launchPlan.Id.Domain, @@ -64,7 +67,11 @@ func CreateLaunchPlanModel( WorkflowID: workflowRepoID, Digest: digest, ScheduleType: scheduleType, - }, nil + } + if launchConditionType != "" { + lpModel.LaunchConditionType = &launchConditionType + } + return lpModel, nil } // Transforms a LaunchPlanModel to a LaunchPlan diff --git a/flyteadmin/pkg/repositories/transformers/node_execution.go b/flyteadmin/pkg/repositories/transformers/node_execution.go index 59ea230f7e..817f53290a 100644 --- a/flyteadmin/pkg/repositories/transformers/node_execution.go +++ b/flyteadmin/pkg/repositories/transformers/node_execution.go @@ -284,7 +284,7 @@ func UpdateNodeExecutionModel( // In the case of dynamic nodes reporting DYNAMIC_RUNNING, the IsParent and IsDynamic bits will be set for this event. // Update the node execution metadata accordingly. - if request.Event.IsParent || request.Event.IsDynamic { + if request.Event.IsParent || request.Event.IsDynamic || request.Event.IsArray { var nodeExecutionMetadata admin.NodeExecutionMetaData if len(nodeExecutionModel.NodeExecutionMetadata) > 0 { if err := proto.Unmarshal(nodeExecutionModel.NodeExecutionMetadata, &nodeExecutionMetadata); err != nil { diff --git a/flyteadmin/pkg/server/service.go b/flyteadmin/pkg/server/service.go index 07eb193883..d51f1c4f31 100644 --- a/flyteadmin/pkg/server/service.go +++ b/flyteadmin/pkg/server/service.go @@ -4,6 +4,7 @@ import ( "context" "crypto/tls" "fmt" + "google.golang.org/protobuf/encoding/protojson" "net" "net/http" "strings" @@ -201,6 +202,19 @@ func newHTTPServer(ctx context.Context, pluginRegistry *plugins.Registry, cfg *c var gwmuxOptions = make([]runtime.ServeMuxOption, 0) // This option means that http requests are served with protobufs, instead of json. We always want this. gwmuxOptions = append(gwmuxOptions, runtime.WithMarshalerOption("application/octet-stream", &runtime.ProtoMarshaller{})) + // grpc-gateway v2 switched the marshaller used to encode JSON messages in 2.5.0. This changed the + // default encoding from snake_case (the v1 behavior) to lowerCamelCase, which is the case recommended + // by protobuf. However the protobuf docs do mention that JSON printers may provide a way to use + // the proto names as field names instead. This option in grpc-gateway v2 does just that, + // by setting a custom marshaler. We are enabling this narrowly however, by applying it only for + // the application/json content type. + gwmuxOptions = append(gwmuxOptions, runtime.WithMarshalerOption("application/json", &runtime.JSONPb{ + MarshalOptions: protojson.MarshalOptions{ + UseProtoNames: true, + EmitUnpopulated: true, + EmitDefaultValues: true, + }, + })) // This option sets subject in the user info response gwmuxOptions = append(gwmuxOptions, runtime.WithForwardResponseOption(auth.GetUserInfoForwardResponseHandler())) diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentServiceClient.go b/flyteidl/clients/go/admin/mocks/AsyncAgentServiceClient.go index f11ef1adfe..0103e3f293 100644 --- a/flyteidl/clients/go/admin/mocks/AsyncAgentServiceClient.go +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentServiceClient.go @@ -10,6 +10,8 @@ import ( grpc "google.golang.org/grpc" mock "github.com/stretchr/testify/mock" + + service "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service" ) // AsyncAgentServiceClient is an autogenerated mock type for the AsyncAgentServiceClient type @@ -165,7 +167,7 @@ type AsyncAgentServiceClient_GetTaskLogs struct { *mock.Call } -func (_m AsyncAgentServiceClient_GetTaskLogs) Return(_a0 *admin.GetTaskLogsResponse, _a1 error) *AsyncAgentServiceClient_GetTaskLogs { +func (_m AsyncAgentServiceClient_GetTaskLogs) Return(_a0 service.AsyncAgentService_GetTaskLogsClient, _a1 error) *AsyncAgentServiceClient_GetTaskLogs { return &AsyncAgentServiceClient_GetTaskLogs{Call: _m.Call.Return(_a0, _a1)} } @@ -180,7 +182,7 @@ func (_m *AsyncAgentServiceClient) OnGetTaskLogsMatch(matchers ...interface{}) * } // GetTaskLogs provides a mock function with given fields: ctx, in, opts -func (_m *AsyncAgentServiceClient) GetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) (*admin.GetTaskLogsResponse, error) { +func (_m *AsyncAgentServiceClient) GetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) (service.AsyncAgentService_GetTaskLogsClient, error) { _va := make([]interface{}, len(opts)) for _i := range opts { _va[_i] = opts[_i] @@ -190,12 +192,12 @@ func (_m *AsyncAgentServiceClient) GetTaskLogs(ctx context.Context, in *admin.Ge _ca = append(_ca, _va...) ret := _m.Called(_ca...) - var r0 *admin.GetTaskLogsResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.GetTaskLogsRequest, ...grpc.CallOption) *admin.GetTaskLogsResponse); ok { + var r0 service.AsyncAgentService_GetTaskLogsClient + if rf, ok := ret.Get(0).(func(context.Context, *admin.GetTaskLogsRequest, ...grpc.CallOption) service.AsyncAgentService_GetTaskLogsClient); ok { r0 = rf(ctx, in, opts...) } else { if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.GetTaskLogsResponse) + r0 = ret.Get(0).(service.AsyncAgentService_GetTaskLogsClient) } } diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentServiceServer.go b/flyteidl/clients/go/admin/mocks/AsyncAgentServiceServer.go index 1803e286eb..76b618f791 100644 --- a/flyteidl/clients/go/admin/mocks/AsyncAgentServiceServer.go +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentServiceServer.go @@ -8,6 +8,8 @@ import ( admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" mock "github.com/stretchr/testify/mock" + + service "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service" ) // AsyncAgentServiceServer is an autogenerated mock type for the AsyncAgentServiceServer type @@ -142,11 +144,11 @@ type AsyncAgentServiceServer_GetTaskLogs struct { *mock.Call } -func (_m AsyncAgentServiceServer_GetTaskLogs) Return(_a0 *admin.GetTaskLogsResponse, _a1 error) *AsyncAgentServiceServer_GetTaskLogs { - return &AsyncAgentServiceServer_GetTaskLogs{Call: _m.Call.Return(_a0, _a1)} +func (_m AsyncAgentServiceServer_GetTaskLogs) Return(_a0 error) *AsyncAgentServiceServer_GetTaskLogs { + return &AsyncAgentServiceServer_GetTaskLogs{Call: _m.Call.Return(_a0)} } -func (_m *AsyncAgentServiceServer) OnGetTaskLogs(_a0 context.Context, _a1 *admin.GetTaskLogsRequest) *AsyncAgentServiceServer_GetTaskLogs { +func (_m *AsyncAgentServiceServer) OnGetTaskLogs(_a0 *admin.GetTaskLogsRequest, _a1 service.AsyncAgentService_GetTaskLogsServer) *AsyncAgentServiceServer_GetTaskLogs { c_call := _m.On("GetTaskLogs", _a0, _a1) return &AsyncAgentServiceServer_GetTaskLogs{Call: c_call} } @@ -157,26 +159,17 @@ func (_m *AsyncAgentServiceServer) OnGetTaskLogsMatch(matchers ...interface{}) * } // GetTaskLogs provides a mock function with given fields: _a0, _a1 -func (_m *AsyncAgentServiceServer) GetTaskLogs(_a0 context.Context, _a1 *admin.GetTaskLogsRequest) (*admin.GetTaskLogsResponse, error) { +func (_m *AsyncAgentServiceServer) GetTaskLogs(_a0 *admin.GetTaskLogsRequest, _a1 service.AsyncAgentService_GetTaskLogsServer) error { ret := _m.Called(_a0, _a1) - var r0 *admin.GetTaskLogsResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.GetTaskLogsRequest) *admin.GetTaskLogsResponse); ok { + var r0 error + if rf, ok := ret.Get(0).(func(*admin.GetTaskLogsRequest, service.AsyncAgentService_GetTaskLogsServer) error); ok { r0 = rf(_a0, _a1) } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.GetTaskLogsResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.GetTaskLogsRequest) error); ok { - r1 = rf(_a0, _a1) - } else { - r1 = ret.Error(1) + r0 = ret.Error(0) } - return r0, r1 + return r0 } type AsyncAgentServiceServer_GetTaskMetrics struct { diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_CreateTaskClient.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_CreateTaskClient.go new file mode 100644 index 0000000000..52c063d197 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_CreateTaskClient.go @@ -0,0 +1,296 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_CreateTaskClient is an autogenerated mock type for the AsyncAgentService_CreateTaskClient type +type AsyncAgentService_CreateTaskClient struct { + mock.Mock +} + +type AsyncAgentService_CreateTaskClient_CloseAndRecv struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_CloseAndRecv) Return(_a0 *admin.CreateTaskResponse, _a1 error) *AsyncAgentService_CreateTaskClient_CloseAndRecv { + return &AsyncAgentService_CreateTaskClient_CloseAndRecv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnCloseAndRecv() *AsyncAgentService_CreateTaskClient_CloseAndRecv { + c_call := _m.On("CloseAndRecv") + return &AsyncAgentService_CreateTaskClient_CloseAndRecv{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnCloseAndRecvMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_CloseAndRecv { + c_call := _m.On("CloseAndRecv", matchers...) + return &AsyncAgentService_CreateTaskClient_CloseAndRecv{Call: c_call} +} + +// CloseAndRecv provides a mock function with given fields: +func (_m *AsyncAgentService_CreateTaskClient) CloseAndRecv() (*admin.CreateTaskResponse, error) { + ret := _m.Called() + + var r0 *admin.CreateTaskResponse + if rf, ok := ret.Get(0).(func() *admin.CreateTaskResponse); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.CreateTaskResponse) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_CreateTaskClient_CloseSend struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_CloseSend) Return(_a0 error) *AsyncAgentService_CreateTaskClient_CloseSend { + return &AsyncAgentService_CreateTaskClient_CloseSend{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnCloseSend() *AsyncAgentService_CreateTaskClient_CloseSend { + c_call := _m.On("CloseSend") + return &AsyncAgentService_CreateTaskClient_CloseSend{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnCloseSendMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_CloseSend { + c_call := _m.On("CloseSend", matchers...) + return &AsyncAgentService_CreateTaskClient_CloseSend{Call: c_call} +} + +// CloseSend provides a mock function with given fields: +func (_m *AsyncAgentService_CreateTaskClient) CloseSend() error { + ret := _m.Called() + + var r0 error + if rf, ok := ret.Get(0).(func() error); ok { + r0 = rf() + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskClient_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_Context) Return(_a0 context.Context) *AsyncAgentService_CreateTaskClient_Context { + return &AsyncAgentService_CreateTaskClient_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnContext() *AsyncAgentService_CreateTaskClient_Context { + c_call := _m.On("Context") + return &AsyncAgentService_CreateTaskClient_Context{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnContextMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_CreateTaskClient_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_CreateTaskClient) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_CreateTaskClient_Header struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_Header) Return(_a0 metadata.MD, _a1 error) *AsyncAgentService_CreateTaskClient_Header { + return &AsyncAgentService_CreateTaskClient_Header{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnHeader() *AsyncAgentService_CreateTaskClient_Header { + c_call := _m.On("Header") + return &AsyncAgentService_CreateTaskClient_Header{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnHeaderMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_Header { + c_call := _m.On("Header", matchers...) + return &AsyncAgentService_CreateTaskClient_Header{Call: c_call} +} + +// Header provides a mock function with given fields: +func (_m *AsyncAgentService_CreateTaskClient) Header() (metadata.MD, error) { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_CreateTaskClient_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_RecvMsg) Return(_a0 error) *AsyncAgentService_CreateTaskClient_RecvMsg { + return &AsyncAgentService_CreateTaskClient_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnRecvMsg(m interface{}) *AsyncAgentService_CreateTaskClient_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_CreateTaskClient_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_CreateTaskClient_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_CreateTaskClient) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskClient_Send struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_Send) Return(_a0 error) *AsyncAgentService_CreateTaskClient_Send { + return &AsyncAgentService_CreateTaskClient_Send{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnSend(_a0 *admin.CreateTaskRequest) *AsyncAgentService_CreateTaskClient_Send { + c_call := _m.On("Send", _a0) + return &AsyncAgentService_CreateTaskClient_Send{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnSendMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_Send { + c_call := _m.On("Send", matchers...) + return &AsyncAgentService_CreateTaskClient_Send{Call: c_call} +} + +// Send provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_CreateTaskClient) Send(_a0 *admin.CreateTaskRequest) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.CreateTaskRequest) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskClient_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_SendMsg) Return(_a0 error) *AsyncAgentService_CreateTaskClient_SendMsg { + return &AsyncAgentService_CreateTaskClient_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnSendMsg(m interface{}) *AsyncAgentService_CreateTaskClient_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_CreateTaskClient_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_CreateTaskClient_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_CreateTaskClient) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskClient_Trailer struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskClient_Trailer) Return(_a0 metadata.MD) *AsyncAgentService_CreateTaskClient_Trailer { + return &AsyncAgentService_CreateTaskClient_Trailer{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnTrailer() *AsyncAgentService_CreateTaskClient_Trailer { + c_call := _m.On("Trailer") + return &AsyncAgentService_CreateTaskClient_Trailer{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskClient) OnTrailerMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskClient_Trailer { + c_call := _m.On("Trailer", matchers...) + return &AsyncAgentService_CreateTaskClient_Trailer{Call: c_call} +} + +// Trailer provides a mock function with given fields: +func (_m *AsyncAgentService_CreateTaskClient) Trailer() metadata.MD { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + return r0 +} diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_CreateTaskServer.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_CreateTaskServer.go new file mode 100644 index 0000000000..68268ce939 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_CreateTaskServer.go @@ -0,0 +1,258 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_CreateTaskServer is an autogenerated mock type for the AsyncAgentService_CreateTaskServer type +type AsyncAgentService_CreateTaskServer struct { + mock.Mock +} + +type AsyncAgentService_CreateTaskServer_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskServer_Context) Return(_a0 context.Context) *AsyncAgentService_CreateTaskServer_Context { + return &AsyncAgentService_CreateTaskServer_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnContext() *AsyncAgentService_CreateTaskServer_Context { + c_call := _m.On("Context") + return &AsyncAgentService_CreateTaskServer_Context{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnContextMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskServer_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_CreateTaskServer_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_CreateTaskServer) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_CreateTaskServer_Recv struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskServer_Recv) Return(_a0 *admin.CreateTaskRequest, _a1 error) *AsyncAgentService_CreateTaskServer_Recv { + return &AsyncAgentService_CreateTaskServer_Recv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnRecv() *AsyncAgentService_CreateTaskServer_Recv { + c_call := _m.On("Recv") + return &AsyncAgentService_CreateTaskServer_Recv{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnRecvMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskServer_Recv { + c_call := _m.On("Recv", matchers...) + return &AsyncAgentService_CreateTaskServer_Recv{Call: c_call} +} + +// Recv provides a mock function with given fields: +func (_m *AsyncAgentService_CreateTaskServer) Recv() (*admin.CreateTaskRequest, error) { + ret := _m.Called() + + var r0 *admin.CreateTaskRequest + if rf, ok := ret.Get(0).(func() *admin.CreateTaskRequest); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.CreateTaskRequest) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_CreateTaskServer_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskServer_RecvMsg) Return(_a0 error) *AsyncAgentService_CreateTaskServer_RecvMsg { + return &AsyncAgentService_CreateTaskServer_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnRecvMsg(m interface{}) *AsyncAgentService_CreateTaskServer_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_CreateTaskServer_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskServer_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_CreateTaskServer_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_CreateTaskServer) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskServer_SendAndClose struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskServer_SendAndClose) Return(_a0 error) *AsyncAgentService_CreateTaskServer_SendAndClose { + return &AsyncAgentService_CreateTaskServer_SendAndClose{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSendAndClose(_a0 *admin.CreateTaskResponse) *AsyncAgentService_CreateTaskServer_SendAndClose { + c_call := _m.On("SendAndClose", _a0) + return &AsyncAgentService_CreateTaskServer_SendAndClose{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSendAndCloseMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskServer_SendAndClose { + c_call := _m.On("SendAndClose", matchers...) + return &AsyncAgentService_CreateTaskServer_SendAndClose{Call: c_call} +} + +// SendAndClose provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_CreateTaskServer) SendAndClose(_a0 *admin.CreateTaskResponse) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.CreateTaskResponse) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskServer_SendHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskServer_SendHeader) Return(_a0 error) *AsyncAgentService_CreateTaskServer_SendHeader { + return &AsyncAgentService_CreateTaskServer_SendHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSendHeader(_a0 metadata.MD) *AsyncAgentService_CreateTaskServer_SendHeader { + c_call := _m.On("SendHeader", _a0) + return &AsyncAgentService_CreateTaskServer_SendHeader{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSendHeaderMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskServer_SendHeader { + c_call := _m.On("SendHeader", matchers...) + return &AsyncAgentService_CreateTaskServer_SendHeader{Call: c_call} +} + +// SendHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_CreateTaskServer) SendHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskServer_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskServer_SendMsg) Return(_a0 error) *AsyncAgentService_CreateTaskServer_SendMsg { + return &AsyncAgentService_CreateTaskServer_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSendMsg(m interface{}) *AsyncAgentService_CreateTaskServer_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_CreateTaskServer_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskServer_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_CreateTaskServer_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_CreateTaskServer) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_CreateTaskServer_SetHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_CreateTaskServer_SetHeader) Return(_a0 error) *AsyncAgentService_CreateTaskServer_SetHeader { + return &AsyncAgentService_CreateTaskServer_SetHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSetHeader(_a0 metadata.MD) *AsyncAgentService_CreateTaskServer_SetHeader { + c_call := _m.On("SetHeader", _a0) + return &AsyncAgentService_CreateTaskServer_SetHeader{Call: c_call} +} + +func (_m *AsyncAgentService_CreateTaskServer) OnSetHeaderMatch(matchers ...interface{}) *AsyncAgentService_CreateTaskServer_SetHeader { + c_call := _m.On("SetHeader", matchers...) + return &AsyncAgentService_CreateTaskServer_SetHeader{Call: c_call} +} + +// SetHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_CreateTaskServer) SetHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// SetTrailer provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_CreateTaskServer) SetTrailer(_a0 metadata.MD) { + _m.Called(_a0) +} diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_ExecuteTaskSyncClient.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_ExecuteTaskSyncClient.go new file mode 100644 index 0000000000..d75c24464e --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_ExecuteTaskSyncClient.go @@ -0,0 +1,296 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_ExecuteTaskSyncClient is an autogenerated mock type for the AsyncAgentService_ExecuteTaskSyncClient type +type AsyncAgentService_ExecuteTaskSyncClient struct { + mock.Mock +} + +type AsyncAgentService_ExecuteTaskSyncClient_CloseSend struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_CloseSend) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncClient_CloseSend { + return &AsyncAgentService_ExecuteTaskSyncClient_CloseSend{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnCloseSend() *AsyncAgentService_ExecuteTaskSyncClient_CloseSend { + c_call := _m.On("CloseSend") + return &AsyncAgentService_ExecuteTaskSyncClient_CloseSend{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnCloseSendMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_CloseSend { + c_call := _m.On("CloseSend", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_CloseSend{Call: c_call} +} + +// CloseSend provides a mock function with given fields: +func (_m *AsyncAgentService_ExecuteTaskSyncClient) CloseSend() error { + ret := _m.Called() + + var r0 error + if rf, ok := ret.Get(0).(func() error); ok { + r0 = rf() + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncClient_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_Context) Return(_a0 context.Context) *AsyncAgentService_ExecuteTaskSyncClient_Context { + return &AsyncAgentService_ExecuteTaskSyncClient_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnContext() *AsyncAgentService_ExecuteTaskSyncClient_Context { + c_call := _m.On("Context") + return &AsyncAgentService_ExecuteTaskSyncClient_Context{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnContextMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_ExecuteTaskSyncClient) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncClient_Header struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_Header) Return(_a0 metadata.MD, _a1 error) *AsyncAgentService_ExecuteTaskSyncClient_Header { + return &AsyncAgentService_ExecuteTaskSyncClient_Header{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnHeader() *AsyncAgentService_ExecuteTaskSyncClient_Header { + c_call := _m.On("Header") + return &AsyncAgentService_ExecuteTaskSyncClient_Header{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnHeaderMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_Header { + c_call := _m.On("Header", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_Header{Call: c_call} +} + +// Header provides a mock function with given fields: +func (_m *AsyncAgentService_ExecuteTaskSyncClient) Header() (metadata.MD, error) { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_ExecuteTaskSyncClient_Recv struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_Recv) Return(_a0 *admin.ExecuteTaskSyncResponse, _a1 error) *AsyncAgentService_ExecuteTaskSyncClient_Recv { + return &AsyncAgentService_ExecuteTaskSyncClient_Recv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnRecv() *AsyncAgentService_ExecuteTaskSyncClient_Recv { + c_call := _m.On("Recv") + return &AsyncAgentService_ExecuteTaskSyncClient_Recv{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnRecvMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_Recv { + c_call := _m.On("Recv", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_Recv{Call: c_call} +} + +// Recv provides a mock function with given fields: +func (_m *AsyncAgentService_ExecuteTaskSyncClient) Recv() (*admin.ExecuteTaskSyncResponse, error) { + ret := _m.Called() + + var r0 *admin.ExecuteTaskSyncResponse + if rf, ok := ret.Get(0).(func() *admin.ExecuteTaskSyncResponse); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.ExecuteTaskSyncResponse) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_ExecuteTaskSyncClient_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_RecvMsg) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncClient_RecvMsg { + return &AsyncAgentService_ExecuteTaskSyncClient_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnRecvMsg(m interface{}) *AsyncAgentService_ExecuteTaskSyncClient_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_ExecuteTaskSyncClient_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_ExecuteTaskSyncClient) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncClient_Send struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_Send) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncClient_Send { + return &AsyncAgentService_ExecuteTaskSyncClient_Send{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnSend(_a0 *admin.ExecuteTaskSyncRequest) *AsyncAgentService_ExecuteTaskSyncClient_Send { + c_call := _m.On("Send", _a0) + return &AsyncAgentService_ExecuteTaskSyncClient_Send{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnSendMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_Send { + c_call := _m.On("Send", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_Send{Call: c_call} +} + +// Send provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_ExecuteTaskSyncClient) Send(_a0 *admin.ExecuteTaskSyncRequest) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.ExecuteTaskSyncRequest) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncClient_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_SendMsg) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncClient_SendMsg { + return &AsyncAgentService_ExecuteTaskSyncClient_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnSendMsg(m interface{}) *AsyncAgentService_ExecuteTaskSyncClient_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_ExecuteTaskSyncClient_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_ExecuteTaskSyncClient) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncClient_Trailer struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncClient_Trailer) Return(_a0 metadata.MD) *AsyncAgentService_ExecuteTaskSyncClient_Trailer { + return &AsyncAgentService_ExecuteTaskSyncClient_Trailer{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnTrailer() *AsyncAgentService_ExecuteTaskSyncClient_Trailer { + c_call := _m.On("Trailer") + return &AsyncAgentService_ExecuteTaskSyncClient_Trailer{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncClient) OnTrailerMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncClient_Trailer { + c_call := _m.On("Trailer", matchers...) + return &AsyncAgentService_ExecuteTaskSyncClient_Trailer{Call: c_call} +} + +// Trailer provides a mock function with given fields: +func (_m *AsyncAgentService_ExecuteTaskSyncClient) Trailer() metadata.MD { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + return r0 +} diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_ExecuteTaskSyncServer.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_ExecuteTaskSyncServer.go new file mode 100644 index 0000000000..a42eb507a3 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_ExecuteTaskSyncServer.go @@ -0,0 +1,258 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_ExecuteTaskSyncServer is an autogenerated mock type for the AsyncAgentService_ExecuteTaskSyncServer type +type AsyncAgentService_ExecuteTaskSyncServer struct { + mock.Mock +} + +type AsyncAgentService_ExecuteTaskSyncServer_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncServer_Context) Return(_a0 context.Context) *AsyncAgentService_ExecuteTaskSyncServer_Context { + return &AsyncAgentService_ExecuteTaskSyncServer_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnContext() *AsyncAgentService_ExecuteTaskSyncServer_Context { + c_call := _m.On("Context") + return &AsyncAgentService_ExecuteTaskSyncServer_Context{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnContextMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncServer_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_ExecuteTaskSyncServer_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_ExecuteTaskSyncServer) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncServer_Recv struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncServer_Recv) Return(_a0 *admin.ExecuteTaskSyncRequest, _a1 error) *AsyncAgentService_ExecuteTaskSyncServer_Recv { + return &AsyncAgentService_ExecuteTaskSyncServer_Recv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnRecv() *AsyncAgentService_ExecuteTaskSyncServer_Recv { + c_call := _m.On("Recv") + return &AsyncAgentService_ExecuteTaskSyncServer_Recv{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnRecvMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncServer_Recv { + c_call := _m.On("Recv", matchers...) + return &AsyncAgentService_ExecuteTaskSyncServer_Recv{Call: c_call} +} + +// Recv provides a mock function with given fields: +func (_m *AsyncAgentService_ExecuteTaskSyncServer) Recv() (*admin.ExecuteTaskSyncRequest, error) { + ret := _m.Called() + + var r0 *admin.ExecuteTaskSyncRequest + if rf, ok := ret.Get(0).(func() *admin.ExecuteTaskSyncRequest); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.ExecuteTaskSyncRequest) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_ExecuteTaskSyncServer_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncServer_RecvMsg) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncServer_RecvMsg { + return &AsyncAgentService_ExecuteTaskSyncServer_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnRecvMsg(m interface{}) *AsyncAgentService_ExecuteTaskSyncServer_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_ExecuteTaskSyncServer_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncServer_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_ExecuteTaskSyncServer_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_ExecuteTaskSyncServer) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncServer_Send struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncServer_Send) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncServer_Send { + return &AsyncAgentService_ExecuteTaskSyncServer_Send{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSend(_a0 *admin.ExecuteTaskSyncResponse) *AsyncAgentService_ExecuteTaskSyncServer_Send { + c_call := _m.On("Send", _a0) + return &AsyncAgentService_ExecuteTaskSyncServer_Send{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSendMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncServer_Send { + c_call := _m.On("Send", matchers...) + return &AsyncAgentService_ExecuteTaskSyncServer_Send{Call: c_call} +} + +// Send provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_ExecuteTaskSyncServer) Send(_a0 *admin.ExecuteTaskSyncResponse) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.ExecuteTaskSyncResponse) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncServer_SendHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncServer_SendHeader) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncServer_SendHeader { + return &AsyncAgentService_ExecuteTaskSyncServer_SendHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSendHeader(_a0 metadata.MD) *AsyncAgentService_ExecuteTaskSyncServer_SendHeader { + c_call := _m.On("SendHeader", _a0) + return &AsyncAgentService_ExecuteTaskSyncServer_SendHeader{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSendHeaderMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncServer_SendHeader { + c_call := _m.On("SendHeader", matchers...) + return &AsyncAgentService_ExecuteTaskSyncServer_SendHeader{Call: c_call} +} + +// SendHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_ExecuteTaskSyncServer) SendHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncServer_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncServer_SendMsg) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncServer_SendMsg { + return &AsyncAgentService_ExecuteTaskSyncServer_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSendMsg(m interface{}) *AsyncAgentService_ExecuteTaskSyncServer_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_ExecuteTaskSyncServer_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncServer_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_ExecuteTaskSyncServer_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_ExecuteTaskSyncServer) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_ExecuteTaskSyncServer_SetHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_ExecuteTaskSyncServer_SetHeader) Return(_a0 error) *AsyncAgentService_ExecuteTaskSyncServer_SetHeader { + return &AsyncAgentService_ExecuteTaskSyncServer_SetHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSetHeader(_a0 metadata.MD) *AsyncAgentService_ExecuteTaskSyncServer_SetHeader { + c_call := _m.On("SetHeader", _a0) + return &AsyncAgentService_ExecuteTaskSyncServer_SetHeader{Call: c_call} +} + +func (_m *AsyncAgentService_ExecuteTaskSyncServer) OnSetHeaderMatch(matchers ...interface{}) *AsyncAgentService_ExecuteTaskSyncServer_SetHeader { + c_call := _m.On("SetHeader", matchers...) + return &AsyncAgentService_ExecuteTaskSyncServer_SetHeader{Call: c_call} +} + +// SetHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_ExecuteTaskSyncServer) SetHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// SetTrailer provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_ExecuteTaskSyncServer) SetTrailer(_a0 metadata.MD) { + _m.Called(_a0) +} diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskClient.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskClient.go new file mode 100644 index 0000000000..d163efb098 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskClient.go @@ -0,0 +1,264 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_GetTaskClient is an autogenerated mock type for the AsyncAgentService_GetTaskClient type +type AsyncAgentService_GetTaskClient struct { + mock.Mock +} + +type AsyncAgentService_GetTaskClient_CloseSend struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskClient_CloseSend) Return(_a0 error) *AsyncAgentService_GetTaskClient_CloseSend { + return &AsyncAgentService_GetTaskClient_CloseSend{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskClient) OnCloseSend() *AsyncAgentService_GetTaskClient_CloseSend { + c_call := _m.On("CloseSend") + return &AsyncAgentService_GetTaskClient_CloseSend{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskClient) OnCloseSendMatch(matchers ...interface{}) *AsyncAgentService_GetTaskClient_CloseSend { + c_call := _m.On("CloseSend", matchers...) + return &AsyncAgentService_GetTaskClient_CloseSend{Call: c_call} +} + +// CloseSend provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskClient) CloseSend() error { + ret := _m.Called() + + var r0 error + if rf, ok := ret.Get(0).(func() error); ok { + r0 = rf() + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskClient_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskClient_Context) Return(_a0 context.Context) *AsyncAgentService_GetTaskClient_Context { + return &AsyncAgentService_GetTaskClient_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskClient) OnContext() *AsyncAgentService_GetTaskClient_Context { + c_call := _m.On("Context") + return &AsyncAgentService_GetTaskClient_Context{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskClient) OnContextMatch(matchers ...interface{}) *AsyncAgentService_GetTaskClient_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_GetTaskClient_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskClient) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_GetTaskClient_Header struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskClient_Header) Return(_a0 metadata.MD, _a1 error) *AsyncAgentService_GetTaskClient_Header { + return &AsyncAgentService_GetTaskClient_Header{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_GetTaskClient) OnHeader() *AsyncAgentService_GetTaskClient_Header { + c_call := _m.On("Header") + return &AsyncAgentService_GetTaskClient_Header{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskClient) OnHeaderMatch(matchers ...interface{}) *AsyncAgentService_GetTaskClient_Header { + c_call := _m.On("Header", matchers...) + return &AsyncAgentService_GetTaskClient_Header{Call: c_call} +} + +// Header provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskClient) Header() (metadata.MD, error) { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_GetTaskClient_Recv struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskClient_Recv) Return(_a0 *admin.GetTaskResponse, _a1 error) *AsyncAgentService_GetTaskClient_Recv { + return &AsyncAgentService_GetTaskClient_Recv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_GetTaskClient) OnRecv() *AsyncAgentService_GetTaskClient_Recv { + c_call := _m.On("Recv") + return &AsyncAgentService_GetTaskClient_Recv{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskClient) OnRecvMatch(matchers ...interface{}) *AsyncAgentService_GetTaskClient_Recv { + c_call := _m.On("Recv", matchers...) + return &AsyncAgentService_GetTaskClient_Recv{Call: c_call} +} + +// Recv provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskClient) Recv() (*admin.GetTaskResponse, error) { + ret := _m.Called() + + var r0 *admin.GetTaskResponse + if rf, ok := ret.Get(0).(func() *admin.GetTaskResponse); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.GetTaskResponse) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_GetTaskClient_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskClient_RecvMsg) Return(_a0 error) *AsyncAgentService_GetTaskClient_RecvMsg { + return &AsyncAgentService_GetTaskClient_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskClient) OnRecvMsg(m interface{}) *AsyncAgentService_GetTaskClient_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_GetTaskClient_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskClient) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskClient_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_GetTaskClient_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskClient) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskClient_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskClient_SendMsg) Return(_a0 error) *AsyncAgentService_GetTaskClient_SendMsg { + return &AsyncAgentService_GetTaskClient_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskClient) OnSendMsg(m interface{}) *AsyncAgentService_GetTaskClient_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_GetTaskClient_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskClient) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskClient_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_GetTaskClient_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskClient) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskClient_Trailer struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskClient_Trailer) Return(_a0 metadata.MD) *AsyncAgentService_GetTaskClient_Trailer { + return &AsyncAgentService_GetTaskClient_Trailer{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskClient) OnTrailer() *AsyncAgentService_GetTaskClient_Trailer { + c_call := _m.On("Trailer") + return &AsyncAgentService_GetTaskClient_Trailer{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskClient) OnTrailerMatch(matchers ...interface{}) *AsyncAgentService_GetTaskClient_Trailer { + c_call := _m.On("Trailer", matchers...) + return &AsyncAgentService_GetTaskClient_Trailer{Call: c_call} +} + +// Trailer provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskClient) Trailer() metadata.MD { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + return r0 +} diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskLogsClient.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskLogsClient.go new file mode 100644 index 0000000000..a28d4ac497 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskLogsClient.go @@ -0,0 +1,264 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_GetTaskLogsClient is an autogenerated mock type for the AsyncAgentService_GetTaskLogsClient type +type AsyncAgentService_GetTaskLogsClient struct { + mock.Mock +} + +type AsyncAgentService_GetTaskLogsClient_CloseSend struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsClient_CloseSend) Return(_a0 error) *AsyncAgentService_GetTaskLogsClient_CloseSend { + return &AsyncAgentService_GetTaskLogsClient_CloseSend{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnCloseSend() *AsyncAgentService_GetTaskLogsClient_CloseSend { + c_call := _m.On("CloseSend") + return &AsyncAgentService_GetTaskLogsClient_CloseSend{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnCloseSendMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsClient_CloseSend { + c_call := _m.On("CloseSend", matchers...) + return &AsyncAgentService_GetTaskLogsClient_CloseSend{Call: c_call} +} + +// CloseSend provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskLogsClient) CloseSend() error { + ret := _m.Called() + + var r0 error + if rf, ok := ret.Get(0).(func() error); ok { + r0 = rf() + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsClient_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsClient_Context) Return(_a0 context.Context) *AsyncAgentService_GetTaskLogsClient_Context { + return &AsyncAgentService_GetTaskLogsClient_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnContext() *AsyncAgentService_GetTaskLogsClient_Context { + c_call := _m.On("Context") + return &AsyncAgentService_GetTaskLogsClient_Context{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnContextMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsClient_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_GetTaskLogsClient_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskLogsClient) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsClient_Header struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsClient_Header) Return(_a0 metadata.MD, _a1 error) *AsyncAgentService_GetTaskLogsClient_Header { + return &AsyncAgentService_GetTaskLogsClient_Header{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnHeader() *AsyncAgentService_GetTaskLogsClient_Header { + c_call := _m.On("Header") + return &AsyncAgentService_GetTaskLogsClient_Header{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnHeaderMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsClient_Header { + c_call := _m.On("Header", matchers...) + return &AsyncAgentService_GetTaskLogsClient_Header{Call: c_call} +} + +// Header provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskLogsClient) Header() (metadata.MD, error) { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_GetTaskLogsClient_Recv struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsClient_Recv) Return(_a0 *admin.GetTaskLogsResponse, _a1 error) *AsyncAgentService_GetTaskLogsClient_Recv { + return &AsyncAgentService_GetTaskLogsClient_Recv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnRecv() *AsyncAgentService_GetTaskLogsClient_Recv { + c_call := _m.On("Recv") + return &AsyncAgentService_GetTaskLogsClient_Recv{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnRecvMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsClient_Recv { + c_call := _m.On("Recv", matchers...) + return &AsyncAgentService_GetTaskLogsClient_Recv{Call: c_call} +} + +// Recv provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskLogsClient) Recv() (*admin.GetTaskLogsResponse, error) { + ret := _m.Called() + + var r0 *admin.GetTaskLogsResponse + if rf, ok := ret.Get(0).(func() *admin.GetTaskLogsResponse); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.GetTaskLogsResponse) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type AsyncAgentService_GetTaskLogsClient_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsClient_RecvMsg) Return(_a0 error) *AsyncAgentService_GetTaskLogsClient_RecvMsg { + return &AsyncAgentService_GetTaskLogsClient_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnRecvMsg(m interface{}) *AsyncAgentService_GetTaskLogsClient_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_GetTaskLogsClient_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsClient_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_GetTaskLogsClient_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskLogsClient) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsClient_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsClient_SendMsg) Return(_a0 error) *AsyncAgentService_GetTaskLogsClient_SendMsg { + return &AsyncAgentService_GetTaskLogsClient_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnSendMsg(m interface{}) *AsyncAgentService_GetTaskLogsClient_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_GetTaskLogsClient_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsClient_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_GetTaskLogsClient_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskLogsClient) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsClient_Trailer struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsClient_Trailer) Return(_a0 metadata.MD) *AsyncAgentService_GetTaskLogsClient_Trailer { + return &AsyncAgentService_GetTaskLogsClient_Trailer{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnTrailer() *AsyncAgentService_GetTaskLogsClient_Trailer { + c_call := _m.On("Trailer") + return &AsyncAgentService_GetTaskLogsClient_Trailer{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsClient) OnTrailerMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsClient_Trailer { + c_call := _m.On("Trailer", matchers...) + return &AsyncAgentService_GetTaskLogsClient_Trailer{Call: c_call} +} + +// Trailer provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskLogsClient) Trailer() metadata.MD { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + return r0 +} diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskLogsServer.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskLogsServer.go new file mode 100644 index 0000000000..00d5ad6b48 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskLogsServer.go @@ -0,0 +1,217 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_GetTaskLogsServer is an autogenerated mock type for the AsyncAgentService_GetTaskLogsServer type +type AsyncAgentService_GetTaskLogsServer struct { + mock.Mock +} + +type AsyncAgentService_GetTaskLogsServer_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsServer_Context) Return(_a0 context.Context) *AsyncAgentService_GetTaskLogsServer_Context { + return &AsyncAgentService_GetTaskLogsServer_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnContext() *AsyncAgentService_GetTaskLogsServer_Context { + c_call := _m.On("Context") + return &AsyncAgentService_GetTaskLogsServer_Context{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnContextMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsServer_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_GetTaskLogsServer_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskLogsServer) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsServer_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsServer_RecvMsg) Return(_a0 error) *AsyncAgentService_GetTaskLogsServer_RecvMsg { + return &AsyncAgentService_GetTaskLogsServer_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnRecvMsg(m interface{}) *AsyncAgentService_GetTaskLogsServer_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_GetTaskLogsServer_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsServer_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_GetTaskLogsServer_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskLogsServer) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsServer_Send struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsServer_Send) Return(_a0 error) *AsyncAgentService_GetTaskLogsServer_Send { + return &AsyncAgentService_GetTaskLogsServer_Send{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSend(_a0 *admin.GetTaskLogsResponse) *AsyncAgentService_GetTaskLogsServer_Send { + c_call := _m.On("Send", _a0) + return &AsyncAgentService_GetTaskLogsServer_Send{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSendMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsServer_Send { + c_call := _m.On("Send", matchers...) + return &AsyncAgentService_GetTaskLogsServer_Send{Call: c_call} +} + +// Send provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskLogsServer) Send(_a0 *admin.GetTaskLogsResponse) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.GetTaskLogsResponse) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsServer_SendHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsServer_SendHeader) Return(_a0 error) *AsyncAgentService_GetTaskLogsServer_SendHeader { + return &AsyncAgentService_GetTaskLogsServer_SendHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSendHeader(_a0 metadata.MD) *AsyncAgentService_GetTaskLogsServer_SendHeader { + c_call := _m.On("SendHeader", _a0) + return &AsyncAgentService_GetTaskLogsServer_SendHeader{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSendHeaderMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsServer_SendHeader { + c_call := _m.On("SendHeader", matchers...) + return &AsyncAgentService_GetTaskLogsServer_SendHeader{Call: c_call} +} + +// SendHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskLogsServer) SendHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsServer_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsServer_SendMsg) Return(_a0 error) *AsyncAgentService_GetTaskLogsServer_SendMsg { + return &AsyncAgentService_GetTaskLogsServer_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSendMsg(m interface{}) *AsyncAgentService_GetTaskLogsServer_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_GetTaskLogsServer_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsServer_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_GetTaskLogsServer_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskLogsServer) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskLogsServer_SetHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskLogsServer_SetHeader) Return(_a0 error) *AsyncAgentService_GetTaskLogsServer_SetHeader { + return &AsyncAgentService_GetTaskLogsServer_SetHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSetHeader(_a0 metadata.MD) *AsyncAgentService_GetTaskLogsServer_SetHeader { + c_call := _m.On("SetHeader", _a0) + return &AsyncAgentService_GetTaskLogsServer_SetHeader{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskLogsServer) OnSetHeaderMatch(matchers ...interface{}) *AsyncAgentService_GetTaskLogsServer_SetHeader { + c_call := _m.On("SetHeader", matchers...) + return &AsyncAgentService_GetTaskLogsServer_SetHeader{Call: c_call} +} + +// SetHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskLogsServer) SetHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// SetTrailer provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskLogsServer) SetTrailer(_a0 metadata.MD) { + _m.Called(_a0) +} diff --git a/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskServer.go b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskServer.go new file mode 100644 index 0000000000..e16fd3591a --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/AsyncAgentService_GetTaskServer.go @@ -0,0 +1,217 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// AsyncAgentService_GetTaskServer is an autogenerated mock type for the AsyncAgentService_GetTaskServer type +type AsyncAgentService_GetTaskServer struct { + mock.Mock +} + +type AsyncAgentService_GetTaskServer_Context struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskServer_Context) Return(_a0 context.Context) *AsyncAgentService_GetTaskServer_Context { + return &AsyncAgentService_GetTaskServer_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskServer) OnContext() *AsyncAgentService_GetTaskServer_Context { + c_call := _m.On("Context") + return &AsyncAgentService_GetTaskServer_Context{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskServer) OnContextMatch(matchers ...interface{}) *AsyncAgentService_GetTaskServer_Context { + c_call := _m.On("Context", matchers...) + return &AsyncAgentService_GetTaskServer_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *AsyncAgentService_GetTaskServer) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type AsyncAgentService_GetTaskServer_RecvMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskServer_RecvMsg) Return(_a0 error) *AsyncAgentService_GetTaskServer_RecvMsg { + return &AsyncAgentService_GetTaskServer_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskServer) OnRecvMsg(m interface{}) *AsyncAgentService_GetTaskServer_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &AsyncAgentService_GetTaskServer_RecvMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskServer) OnRecvMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskServer_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &AsyncAgentService_GetTaskServer_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskServer) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskServer_Send struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskServer_Send) Return(_a0 error) *AsyncAgentService_GetTaskServer_Send { + return &AsyncAgentService_GetTaskServer_Send{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSend(_a0 *admin.GetTaskResponse) *AsyncAgentService_GetTaskServer_Send { + c_call := _m.On("Send", _a0) + return &AsyncAgentService_GetTaskServer_Send{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSendMatch(matchers ...interface{}) *AsyncAgentService_GetTaskServer_Send { + c_call := _m.On("Send", matchers...) + return &AsyncAgentService_GetTaskServer_Send{Call: c_call} +} + +// Send provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskServer) Send(_a0 *admin.GetTaskResponse) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.GetTaskResponse) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskServer_SendHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskServer_SendHeader) Return(_a0 error) *AsyncAgentService_GetTaskServer_SendHeader { + return &AsyncAgentService_GetTaskServer_SendHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSendHeader(_a0 metadata.MD) *AsyncAgentService_GetTaskServer_SendHeader { + c_call := _m.On("SendHeader", _a0) + return &AsyncAgentService_GetTaskServer_SendHeader{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSendHeaderMatch(matchers ...interface{}) *AsyncAgentService_GetTaskServer_SendHeader { + c_call := _m.On("SendHeader", matchers...) + return &AsyncAgentService_GetTaskServer_SendHeader{Call: c_call} +} + +// SendHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskServer) SendHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskServer_SendMsg struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskServer_SendMsg) Return(_a0 error) *AsyncAgentService_GetTaskServer_SendMsg { + return &AsyncAgentService_GetTaskServer_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSendMsg(m interface{}) *AsyncAgentService_GetTaskServer_SendMsg { + c_call := _m.On("SendMsg", m) + return &AsyncAgentService_GetTaskServer_SendMsg{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSendMsgMatch(matchers ...interface{}) *AsyncAgentService_GetTaskServer_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &AsyncAgentService_GetTaskServer_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *AsyncAgentService_GetTaskServer) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type AsyncAgentService_GetTaskServer_SetHeader struct { + *mock.Call +} + +func (_m AsyncAgentService_GetTaskServer_SetHeader) Return(_a0 error) *AsyncAgentService_GetTaskServer_SetHeader { + return &AsyncAgentService_GetTaskServer_SetHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSetHeader(_a0 metadata.MD) *AsyncAgentService_GetTaskServer_SetHeader { + c_call := _m.On("SetHeader", _a0) + return &AsyncAgentService_GetTaskServer_SetHeader{Call: c_call} +} + +func (_m *AsyncAgentService_GetTaskServer) OnSetHeaderMatch(matchers ...interface{}) *AsyncAgentService_GetTaskServer_SetHeader { + c_call := _m.On("SetHeader", matchers...) + return &AsyncAgentService_GetTaskServer_SetHeader{Call: c_call} +} + +// SetHeader provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskServer) SetHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// SetTrailer provides a mock function with given fields: _a0 +func (_m *AsyncAgentService_GetTaskServer) SetTrailer(_a0 metadata.MD) { + _m.Called(_a0) +} diff --git a/flyteidl/clients/go/admin/mocks/SyncAgentServiceClient.go b/flyteidl/clients/go/admin/mocks/SyncAgentServiceClient.go new file mode 100644 index 0000000000..6faf4c2ad9 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/SyncAgentServiceClient.go @@ -0,0 +1,66 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + grpc "google.golang.org/grpc" + + mock "github.com/stretchr/testify/mock" + + service "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service" +) + +// SyncAgentServiceClient is an autogenerated mock type for the SyncAgentServiceClient type +type SyncAgentServiceClient struct { + mock.Mock +} + +type SyncAgentServiceClient_ExecuteTaskSync struct { + *mock.Call +} + +func (_m SyncAgentServiceClient_ExecuteTaskSync) Return(_a0 service.SyncAgentService_ExecuteTaskSyncClient, _a1 error) *SyncAgentServiceClient_ExecuteTaskSync { + return &SyncAgentServiceClient_ExecuteTaskSync{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *SyncAgentServiceClient) OnExecuteTaskSync(ctx context.Context, opts ...grpc.CallOption) *SyncAgentServiceClient_ExecuteTaskSync { + c_call := _m.On("ExecuteTaskSync", ctx, opts) + return &SyncAgentServiceClient_ExecuteTaskSync{Call: c_call} +} + +func (_m *SyncAgentServiceClient) OnExecuteTaskSyncMatch(matchers ...interface{}) *SyncAgentServiceClient_ExecuteTaskSync { + c_call := _m.On("ExecuteTaskSync", matchers...) + return &SyncAgentServiceClient_ExecuteTaskSync{Call: c_call} +} + +// ExecuteTaskSync provides a mock function with given fields: ctx, opts +func (_m *SyncAgentServiceClient) ExecuteTaskSync(ctx context.Context, opts ...grpc.CallOption) (service.SyncAgentService_ExecuteTaskSyncClient, error) { + _va := make([]interface{}, len(opts)) + for _i := range opts { + _va[_i] = opts[_i] + } + var _ca []interface{} + _ca = append(_ca, ctx) + _ca = append(_ca, _va...) + ret := _m.Called(_ca...) + + var r0 service.SyncAgentService_ExecuteTaskSyncClient + if rf, ok := ret.Get(0).(func(context.Context, ...grpc.CallOption) service.SyncAgentService_ExecuteTaskSyncClient); ok { + r0 = rf(ctx, opts...) + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(service.SyncAgentService_ExecuteTaskSyncClient) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func(context.Context, ...grpc.CallOption) error); ok { + r1 = rf(ctx, opts...) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} diff --git a/flyteidl/clients/go/admin/mocks/SyncAgentServiceServer.go b/flyteidl/clients/go/admin/mocks/SyncAgentServiceServer.go new file mode 100644 index 0000000000..ee7b4a78e6 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/SyncAgentServiceServer.go @@ -0,0 +1,45 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + service "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service" + mock "github.com/stretchr/testify/mock" +) + +// SyncAgentServiceServer is an autogenerated mock type for the SyncAgentServiceServer type +type SyncAgentServiceServer struct { + mock.Mock +} + +type SyncAgentServiceServer_ExecuteTaskSync struct { + *mock.Call +} + +func (_m SyncAgentServiceServer_ExecuteTaskSync) Return(_a0 error) *SyncAgentServiceServer_ExecuteTaskSync { + return &SyncAgentServiceServer_ExecuteTaskSync{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentServiceServer) OnExecuteTaskSync(_a0 service.SyncAgentService_ExecuteTaskSyncServer) *SyncAgentServiceServer_ExecuteTaskSync { + c_call := _m.On("ExecuteTaskSync", _a0) + return &SyncAgentServiceServer_ExecuteTaskSync{Call: c_call} +} + +func (_m *SyncAgentServiceServer) OnExecuteTaskSyncMatch(matchers ...interface{}) *SyncAgentServiceServer_ExecuteTaskSync { + c_call := _m.On("ExecuteTaskSync", matchers...) + return &SyncAgentServiceServer_ExecuteTaskSync{Call: c_call} +} + +// ExecuteTaskSync provides a mock function with given fields: _a0 +func (_m *SyncAgentServiceServer) ExecuteTaskSync(_a0 service.SyncAgentService_ExecuteTaskSyncServer) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(service.SyncAgentService_ExecuteTaskSyncServer) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} diff --git a/flyteidl/clients/go/admin/mocks/SyncAgentService_ExecuteTaskSyncClient.go b/flyteidl/clients/go/admin/mocks/SyncAgentService_ExecuteTaskSyncClient.go new file mode 100644 index 0000000000..c88068293f --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/SyncAgentService_ExecuteTaskSyncClient.go @@ -0,0 +1,296 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// SyncAgentService_ExecuteTaskSyncClient is an autogenerated mock type for the SyncAgentService_ExecuteTaskSyncClient type +type SyncAgentService_ExecuteTaskSyncClient struct { + mock.Mock +} + +type SyncAgentService_ExecuteTaskSyncClient_CloseSend struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_CloseSend) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncClient_CloseSend { + return &SyncAgentService_ExecuteTaskSyncClient_CloseSend{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnCloseSend() *SyncAgentService_ExecuteTaskSyncClient_CloseSend { + c_call := _m.On("CloseSend") + return &SyncAgentService_ExecuteTaskSyncClient_CloseSend{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnCloseSendMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_CloseSend { + c_call := _m.On("CloseSend", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_CloseSend{Call: c_call} +} + +// CloseSend provides a mock function with given fields: +func (_m *SyncAgentService_ExecuteTaskSyncClient) CloseSend() error { + ret := _m.Called() + + var r0 error + if rf, ok := ret.Get(0).(func() error); ok { + r0 = rf() + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncClient_Context struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_Context) Return(_a0 context.Context) *SyncAgentService_ExecuteTaskSyncClient_Context { + return &SyncAgentService_ExecuteTaskSyncClient_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnContext() *SyncAgentService_ExecuteTaskSyncClient_Context { + c_call := _m.On("Context") + return &SyncAgentService_ExecuteTaskSyncClient_Context{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnContextMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_Context { + c_call := _m.On("Context", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *SyncAgentService_ExecuteTaskSyncClient) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncClient_Header struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_Header) Return(_a0 metadata.MD, _a1 error) *SyncAgentService_ExecuteTaskSyncClient_Header { + return &SyncAgentService_ExecuteTaskSyncClient_Header{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnHeader() *SyncAgentService_ExecuteTaskSyncClient_Header { + c_call := _m.On("Header") + return &SyncAgentService_ExecuteTaskSyncClient_Header{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnHeaderMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_Header { + c_call := _m.On("Header", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_Header{Call: c_call} +} + +// Header provides a mock function with given fields: +func (_m *SyncAgentService_ExecuteTaskSyncClient) Header() (metadata.MD, error) { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type SyncAgentService_ExecuteTaskSyncClient_Recv struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_Recv) Return(_a0 *admin.ExecuteTaskSyncResponse, _a1 error) *SyncAgentService_ExecuteTaskSyncClient_Recv { + return &SyncAgentService_ExecuteTaskSyncClient_Recv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnRecv() *SyncAgentService_ExecuteTaskSyncClient_Recv { + c_call := _m.On("Recv") + return &SyncAgentService_ExecuteTaskSyncClient_Recv{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnRecvMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_Recv { + c_call := _m.On("Recv", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_Recv{Call: c_call} +} + +// Recv provides a mock function with given fields: +func (_m *SyncAgentService_ExecuteTaskSyncClient) Recv() (*admin.ExecuteTaskSyncResponse, error) { + ret := _m.Called() + + var r0 *admin.ExecuteTaskSyncResponse + if rf, ok := ret.Get(0).(func() *admin.ExecuteTaskSyncResponse); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.ExecuteTaskSyncResponse) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type SyncAgentService_ExecuteTaskSyncClient_RecvMsg struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_RecvMsg) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncClient_RecvMsg { + return &SyncAgentService_ExecuteTaskSyncClient_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnRecvMsg(m interface{}) *SyncAgentService_ExecuteTaskSyncClient_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &SyncAgentService_ExecuteTaskSyncClient_RecvMsg{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnRecvMsgMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *SyncAgentService_ExecuteTaskSyncClient) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncClient_Send struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_Send) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncClient_Send { + return &SyncAgentService_ExecuteTaskSyncClient_Send{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnSend(_a0 *admin.ExecuteTaskSyncRequest) *SyncAgentService_ExecuteTaskSyncClient_Send { + c_call := _m.On("Send", _a0) + return &SyncAgentService_ExecuteTaskSyncClient_Send{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnSendMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_Send { + c_call := _m.On("Send", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_Send{Call: c_call} +} + +// Send provides a mock function with given fields: _a0 +func (_m *SyncAgentService_ExecuteTaskSyncClient) Send(_a0 *admin.ExecuteTaskSyncRequest) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.ExecuteTaskSyncRequest) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncClient_SendMsg struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_SendMsg) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncClient_SendMsg { + return &SyncAgentService_ExecuteTaskSyncClient_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnSendMsg(m interface{}) *SyncAgentService_ExecuteTaskSyncClient_SendMsg { + c_call := _m.On("SendMsg", m) + return &SyncAgentService_ExecuteTaskSyncClient_SendMsg{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnSendMsgMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *SyncAgentService_ExecuteTaskSyncClient) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncClient_Trailer struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncClient_Trailer) Return(_a0 metadata.MD) *SyncAgentService_ExecuteTaskSyncClient_Trailer { + return &SyncAgentService_ExecuteTaskSyncClient_Trailer{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnTrailer() *SyncAgentService_ExecuteTaskSyncClient_Trailer { + c_call := _m.On("Trailer") + return &SyncAgentService_ExecuteTaskSyncClient_Trailer{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncClient) OnTrailerMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncClient_Trailer { + c_call := _m.On("Trailer", matchers...) + return &SyncAgentService_ExecuteTaskSyncClient_Trailer{Call: c_call} +} + +// Trailer provides a mock function with given fields: +func (_m *SyncAgentService_ExecuteTaskSyncClient) Trailer() metadata.MD { + ret := _m.Called() + + var r0 metadata.MD + if rf, ok := ret.Get(0).(func() metadata.MD); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(metadata.MD) + } + } + + return r0 +} diff --git a/flyteidl/clients/go/admin/mocks/SyncAgentService_ExecuteTaskSyncServer.go b/flyteidl/clients/go/admin/mocks/SyncAgentService_ExecuteTaskSyncServer.go new file mode 100644 index 0000000000..de1579d7a8 --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/SyncAgentService_ExecuteTaskSyncServer.go @@ -0,0 +1,258 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" + + metadata "google.golang.org/grpc/metadata" + + mock "github.com/stretchr/testify/mock" +) + +// SyncAgentService_ExecuteTaskSyncServer is an autogenerated mock type for the SyncAgentService_ExecuteTaskSyncServer type +type SyncAgentService_ExecuteTaskSyncServer struct { + mock.Mock +} + +type SyncAgentService_ExecuteTaskSyncServer_Context struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncServer_Context) Return(_a0 context.Context) *SyncAgentService_ExecuteTaskSyncServer_Context { + return &SyncAgentService_ExecuteTaskSyncServer_Context{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnContext() *SyncAgentService_ExecuteTaskSyncServer_Context { + c_call := _m.On("Context") + return &SyncAgentService_ExecuteTaskSyncServer_Context{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnContextMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncServer_Context { + c_call := _m.On("Context", matchers...) + return &SyncAgentService_ExecuteTaskSyncServer_Context{Call: c_call} +} + +// Context provides a mock function with given fields: +func (_m *SyncAgentService_ExecuteTaskSyncServer) Context() context.Context { + ret := _m.Called() + + var r0 context.Context + if rf, ok := ret.Get(0).(func() context.Context); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(context.Context) + } + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncServer_Recv struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncServer_Recv) Return(_a0 *admin.ExecuteTaskSyncRequest, _a1 error) *SyncAgentService_ExecuteTaskSyncServer_Recv { + return &SyncAgentService_ExecuteTaskSyncServer_Recv{Call: _m.Call.Return(_a0, _a1)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnRecv() *SyncAgentService_ExecuteTaskSyncServer_Recv { + c_call := _m.On("Recv") + return &SyncAgentService_ExecuteTaskSyncServer_Recv{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnRecvMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncServer_Recv { + c_call := _m.On("Recv", matchers...) + return &SyncAgentService_ExecuteTaskSyncServer_Recv{Call: c_call} +} + +// Recv provides a mock function with given fields: +func (_m *SyncAgentService_ExecuteTaskSyncServer) Recv() (*admin.ExecuteTaskSyncRequest, error) { + ret := _m.Called() + + var r0 *admin.ExecuteTaskSyncRequest + if rf, ok := ret.Get(0).(func() *admin.ExecuteTaskSyncRequest); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*admin.ExecuteTaskSyncRequest) + } + } + + var r1 error + if rf, ok := ret.Get(1).(func() error); ok { + r1 = rf() + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +type SyncAgentService_ExecuteTaskSyncServer_RecvMsg struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncServer_RecvMsg) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncServer_RecvMsg { + return &SyncAgentService_ExecuteTaskSyncServer_RecvMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnRecvMsg(m interface{}) *SyncAgentService_ExecuteTaskSyncServer_RecvMsg { + c_call := _m.On("RecvMsg", m) + return &SyncAgentService_ExecuteTaskSyncServer_RecvMsg{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnRecvMsgMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncServer_RecvMsg { + c_call := _m.On("RecvMsg", matchers...) + return &SyncAgentService_ExecuteTaskSyncServer_RecvMsg{Call: c_call} +} + +// RecvMsg provides a mock function with given fields: m +func (_m *SyncAgentService_ExecuteTaskSyncServer) RecvMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncServer_Send struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncServer_Send) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncServer_Send { + return &SyncAgentService_ExecuteTaskSyncServer_Send{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSend(_a0 *admin.ExecuteTaskSyncResponse) *SyncAgentService_ExecuteTaskSyncServer_Send { + c_call := _m.On("Send", _a0) + return &SyncAgentService_ExecuteTaskSyncServer_Send{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSendMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncServer_Send { + c_call := _m.On("Send", matchers...) + return &SyncAgentService_ExecuteTaskSyncServer_Send{Call: c_call} +} + +// Send provides a mock function with given fields: _a0 +func (_m *SyncAgentService_ExecuteTaskSyncServer) Send(_a0 *admin.ExecuteTaskSyncResponse) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(*admin.ExecuteTaskSyncResponse) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncServer_SendHeader struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncServer_SendHeader) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncServer_SendHeader { + return &SyncAgentService_ExecuteTaskSyncServer_SendHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSendHeader(_a0 metadata.MD) *SyncAgentService_ExecuteTaskSyncServer_SendHeader { + c_call := _m.On("SendHeader", _a0) + return &SyncAgentService_ExecuteTaskSyncServer_SendHeader{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSendHeaderMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncServer_SendHeader { + c_call := _m.On("SendHeader", matchers...) + return &SyncAgentService_ExecuteTaskSyncServer_SendHeader{Call: c_call} +} + +// SendHeader provides a mock function with given fields: _a0 +func (_m *SyncAgentService_ExecuteTaskSyncServer) SendHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncServer_SendMsg struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncServer_SendMsg) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncServer_SendMsg { + return &SyncAgentService_ExecuteTaskSyncServer_SendMsg{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSendMsg(m interface{}) *SyncAgentService_ExecuteTaskSyncServer_SendMsg { + c_call := _m.On("SendMsg", m) + return &SyncAgentService_ExecuteTaskSyncServer_SendMsg{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSendMsgMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncServer_SendMsg { + c_call := _m.On("SendMsg", matchers...) + return &SyncAgentService_ExecuteTaskSyncServer_SendMsg{Call: c_call} +} + +// SendMsg provides a mock function with given fields: m +func (_m *SyncAgentService_ExecuteTaskSyncServer) SendMsg(m interface{}) error { + ret := _m.Called(m) + + var r0 error + if rf, ok := ret.Get(0).(func(interface{}) error); ok { + r0 = rf(m) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +type SyncAgentService_ExecuteTaskSyncServer_SetHeader struct { + *mock.Call +} + +func (_m SyncAgentService_ExecuteTaskSyncServer_SetHeader) Return(_a0 error) *SyncAgentService_ExecuteTaskSyncServer_SetHeader { + return &SyncAgentService_ExecuteTaskSyncServer_SetHeader{Call: _m.Call.Return(_a0)} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSetHeader(_a0 metadata.MD) *SyncAgentService_ExecuteTaskSyncServer_SetHeader { + c_call := _m.On("SetHeader", _a0) + return &SyncAgentService_ExecuteTaskSyncServer_SetHeader{Call: c_call} +} + +func (_m *SyncAgentService_ExecuteTaskSyncServer) OnSetHeaderMatch(matchers ...interface{}) *SyncAgentService_ExecuteTaskSyncServer_SetHeader { + c_call := _m.On("SetHeader", matchers...) + return &SyncAgentService_ExecuteTaskSyncServer_SetHeader{Call: c_call} +} + +// SetHeader provides a mock function with given fields: _a0 +func (_m *SyncAgentService_ExecuteTaskSyncServer) SetHeader(_a0 metadata.MD) error { + ret := _m.Called(_a0) + + var r0 error + if rf, ok := ret.Get(0).(func(metadata.MD) error); ok { + r0 = rf(_a0) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// SetTrailer provides a mock function with given fields: _a0 +func (_m *SyncAgentService_ExecuteTaskSyncServer) SetTrailer(_a0 metadata.MD) { + _m.Called(_a0) +} diff --git a/flyteidl/clients/go/admin/mocks/UnsafeSyncAgentServiceServer.go b/flyteidl/clients/go/admin/mocks/UnsafeSyncAgentServiceServer.go new file mode 100644 index 0000000000..c4f4b4099c --- /dev/null +++ b/flyteidl/clients/go/admin/mocks/UnsafeSyncAgentServiceServer.go @@ -0,0 +1,15 @@ +// Code generated by mockery v1.0.1. DO NOT EDIT. + +package mocks + +import mock "github.com/stretchr/testify/mock" + +// UnsafeSyncAgentServiceServer is an autogenerated mock type for the UnsafeSyncAgentServiceServer type +type UnsafeSyncAgentServiceServer struct { + mock.Mock +} + +// mustEmbedUnimplementedSyncAgentServiceServer provides a mock function with given fields: +func (_m *UnsafeSyncAgentServiceServer) mustEmbedUnimplementedSyncAgentServiceServer() { + _m.Called() +} diff --git a/flyteidl/clients/go/assets/admin.swagger.json b/flyteidl/clients/go/assets/admin.swagger.json index 73dbd3b8dc..191143dd89 100644 --- a/flyteidl/clients/go/assets/admin.swagger.json +++ b/flyteidl/clients/go/assets/admin.swagger.json @@ -4273,10 +4273,11 @@ "SYSTEM", "RELAUNCH", "CHILD_WORKFLOW", - "RECOVERED" + "RECOVERED", + "TRIGGER" ], "default": "MANUAL", - "description": "The method by which this execution was launched.\n\n - MANUAL: The default execution mode, MANUAL implies that an execution was launched by an individual.\n - SCHEDULED: A schedule triggered this execution launch.\n - SYSTEM: A system process was responsible for launching this execution rather an individual.\n - RELAUNCH: This execution was launched with identical inputs as a previous execution.\n - CHILD_WORKFLOW: This execution was triggered by another execution.\n - RECOVERED: This execution was recovered from another execution." + "description": "The method by which this execution was launched.\n\n - MANUAL: The default execution mode, MANUAL implies that an execution was launched by an individual.\n - SCHEDULED: A schedule triggered this execution launch.\n - SYSTEM: A system process was responsible for launching this execution rather an individual.\n - RELAUNCH: This execution was launched with identical inputs as a previous execution.\n - CHILD_WORKFLOW: This execution was triggered by another execution.\n - RECOVERED: This execution was recovered from another execution.\n - TRIGGER: Execution was kicked off by the artifact trigger system" }, "IOStrategyDownloadMode": { "type": "string", @@ -6474,10 +6475,6 @@ "coreArtifactBindingData": { "type": "object", "properties": { - "index": { - "type": "integer", - "format": "int64" - }, "partition_key": { "type": "string" }, @@ -8080,6 +8077,10 @@ "extended_resources": { "$ref": "#/definitions/coreExtendedResources", "description": "Overrides for all non-standard resources, not captured by\nv1.ResourceRequirements, to allocate to a task." + }, + "container_image": { + "type": "string", + "description": "Override for the image used by task pods." } }, "description": "Optional task node overrides that will be applied at task execution time." diff --git a/flyteidl/gen/pb-es/flyteidl/admin/agent_pb.ts b/flyteidl/gen/pb-es/flyteidl/admin/agent_pb.ts index 834197a299..deb3601202 100644 --- a/flyteidl/gen/pb-es/flyteidl/admin/agent_pb.ts +++ b/flyteidl/gen/pb-es/flyteidl/admin/agent_pb.ts @@ -4,8 +4,9 @@ // @ts-nocheck import type { BinaryReadOptions, FieldList, JsonReadOptions, JsonValue, PartialMessage, PlainMessage } from "@bufbuild/protobuf"; -import { Duration, Message, proto3, protoInt64, Timestamp } from "@bufbuild/protobuf"; +import { Duration, Message, proto3, protoInt64, Struct, Timestamp } from "@bufbuild/protobuf"; import { TaskExecutionIdentifier } from "../core/identifier_pb.js"; +import { TaskNodeOverrides } from "../core/workflow_pb.js"; import { LiteralMap } from "../core/literals_pb.js"; import { TaskTemplate } from "../core/tasks_pb.js"; import { TaskExecution_Phase, TaskLog } from "../core/execution_pb.js"; @@ -15,6 +16,7 @@ import { ExecutionMetricResult } from "../core/metrics_pb.js"; * The state of the execution is used to control its visibility in the UI/CLI. * * @generated from enum flyteidl.admin.State + * @deprecated */ export enum State { /** @@ -99,6 +101,39 @@ export class TaskExecutionMetadata extends Message { */ environmentVariables: { [key: string]: string } = {}; + /** + * Represents the maximum number of attempts allowed for a task. + * If a task fails, it can be retried up to this maximum number of attempts. + * + * @generated from field: int32 max_attempts = 7; + */ + maxAttempts = 0; + + /** + * Indicates whether the task execution can be interrupted. + * If set to true, the task can be stopped before completion. + * + * @generated from field: bool interruptible = 8; + */ + interruptible = false; + + /** + * Specifies the threshold for failure count at which the interruptible property + * will take effect. If the number of consecutive task failures exceeds this threshold, + * interruptible behavior will be activated. + * + * @generated from field: int32 interruptible_failure_threshold = 9; + */ + interruptibleFailureThreshold = 0; + + /** + * Overrides for specific properties of the task node. + * These overrides can be used to customize the behavior of the task node. + * + * @generated from field: flyteidl.core.TaskNodeOverrides overrides = 10; + */ + overrides?: TaskNodeOverrides; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -113,6 +148,10 @@ export class TaskExecutionMetadata extends Message { { no: 4, name: "annotations", kind: "map", K: 9 /* ScalarType.STRING */, V: {kind: "scalar", T: 9 /* ScalarType.STRING */} }, { no: 5, name: "k8s_service_account", kind: "scalar", T: 9 /* ScalarType.STRING */ }, { no: 6, name: "environment_variables", kind: "map", K: 9 /* ScalarType.STRING */, V: {kind: "scalar", T: 9 /* ScalarType.STRING */} }, + { no: 7, name: "max_attempts", kind: "scalar", T: 5 /* ScalarType.INT32 */ }, + { no: 8, name: "interruptible", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, + { no: 9, name: "interruptible_failure_threshold", kind: "scalar", T: 5 /* ScalarType.INT32 */ }, + { no: 10, name: "overrides", kind: "message", T: TaskNodeOverrides }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): TaskExecutionMetadata { @@ -206,24 +245,11 @@ export class CreateTaskRequest extends Message { */ export class CreateTaskResponse extends Message { /** - * Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). - * Resource is for synchronous task execution. + * ResourceMeta is created by the agent. It could be a string (jobId) or a dict (more complex metadata). * - * @generated from oneof flyteidl.admin.CreateTaskResponse.res + * @generated from field: bytes resource_meta = 1; */ - res: { - /** - * @generated from field: bytes resource_meta = 1; - */ - value: Uint8Array; - case: "resourceMeta"; - } | { - /** - * @generated from field: flyteidl.admin.Resource resource = 2; - */ - value: Resource; - case: "resource"; - } | { case: undefined; value?: undefined } = { case: undefined }; + resourceMeta = new Uint8Array(0); constructor(data?: PartialMessage) { super(); @@ -233,8 +259,7 @@ export class CreateTaskResponse extends Message { static readonly runtime: typeof proto3 = proto3; static readonly typeName = "flyteidl.admin.CreateTaskResponse"; static readonly fields: FieldList = proto3.util.newFieldList(() => [ - { no: 1, name: "resource_meta", kind: "scalar", T: 12 /* ScalarType.BYTES */, oneof: "res" }, - { no: 2, name: "resource", kind: "message", T: Resource, oneof: "res" }, + { no: 1, name: "resource_meta", kind: "scalar", T: 12 /* ScalarType.BYTES */ }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): CreateTaskResponse { @@ -254,6 +279,209 @@ export class CreateTaskResponse extends Message { } } +/** + * @generated from message flyteidl.admin.CreateRequestHeader + */ +export class CreateRequestHeader extends Message { + /** + * Template of the task that encapsulates all the metadata of the task. + * + * @generated from field: flyteidl.core.TaskTemplate template = 1; + */ + template?: TaskTemplate; + + /** + * Prefix for where task output data will be written. (e.g. s3://my-bucket/randomstring) + * + * @generated from field: string output_prefix = 2; + */ + outputPrefix = ""; + + /** + * subset of runtime task execution metadata. + * + * @generated from field: flyteidl.admin.TaskExecutionMetadata task_execution_metadata = 3; + */ + taskExecutionMetadata?: TaskExecutionMetadata; + + /** + * MaxDatasetSizeBytes is the maximum size of the dataset that can be generated by the task. + * + * @generated from field: int64 max_dataset_size_bytes = 4; + */ + maxDatasetSizeBytes = protoInt64.zero; + + constructor(data?: PartialMessage) { + super(); + proto3.util.initPartial(data, this); + } + + static readonly runtime: typeof proto3 = proto3; + static readonly typeName = "flyteidl.admin.CreateRequestHeader"; + static readonly fields: FieldList = proto3.util.newFieldList(() => [ + { no: 1, name: "template", kind: "message", T: TaskTemplate }, + { no: 2, name: "output_prefix", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 3, name: "task_execution_metadata", kind: "message", T: TaskExecutionMetadata }, + { no: 4, name: "max_dataset_size_bytes", kind: "scalar", T: 3 /* ScalarType.INT64 */ }, + ]); + + static fromBinary(bytes: Uint8Array, options?: Partial): CreateRequestHeader { + return new CreateRequestHeader().fromBinary(bytes, options); + } + + static fromJson(jsonValue: JsonValue, options?: Partial): CreateRequestHeader { + return new CreateRequestHeader().fromJson(jsonValue, options); + } + + static fromJsonString(jsonString: string, options?: Partial): CreateRequestHeader { + return new CreateRequestHeader().fromJsonString(jsonString, options); + } + + static equals(a: CreateRequestHeader | PlainMessage | undefined, b: CreateRequestHeader | PlainMessage | undefined): boolean { + return proto3.util.equals(CreateRequestHeader, a, b); + } +} + +/** + * @generated from message flyteidl.admin.ExecuteTaskSyncRequest + */ +export class ExecuteTaskSyncRequest extends Message { + /** + * @generated from oneof flyteidl.admin.ExecuteTaskSyncRequest.part + */ + part: { + /** + * @generated from field: flyteidl.admin.CreateRequestHeader header = 1; + */ + value: CreateRequestHeader; + case: "header"; + } | { + /** + * @generated from field: flyteidl.core.LiteralMap inputs = 2; + */ + value: LiteralMap; + case: "inputs"; + } | { case: undefined; value?: undefined } = { case: undefined }; + + constructor(data?: PartialMessage) { + super(); + proto3.util.initPartial(data, this); + } + + static readonly runtime: typeof proto3 = proto3; + static readonly typeName = "flyteidl.admin.ExecuteTaskSyncRequest"; + static readonly fields: FieldList = proto3.util.newFieldList(() => [ + { no: 1, name: "header", kind: "message", T: CreateRequestHeader, oneof: "part" }, + { no: 2, name: "inputs", kind: "message", T: LiteralMap, oneof: "part" }, + ]); + + static fromBinary(bytes: Uint8Array, options?: Partial): ExecuteTaskSyncRequest { + return new ExecuteTaskSyncRequest().fromBinary(bytes, options); + } + + static fromJson(jsonValue: JsonValue, options?: Partial): ExecuteTaskSyncRequest { + return new ExecuteTaskSyncRequest().fromJson(jsonValue, options); + } + + static fromJsonString(jsonString: string, options?: Partial): ExecuteTaskSyncRequest { + return new ExecuteTaskSyncRequest().fromJsonString(jsonString, options); + } + + static equals(a: ExecuteTaskSyncRequest | PlainMessage | undefined, b: ExecuteTaskSyncRequest | PlainMessage | undefined): boolean { + return proto3.util.equals(ExecuteTaskSyncRequest, a, b); + } +} + +/** + * @generated from message flyteidl.admin.ExecuteTaskSyncResponseHeader + */ +export class ExecuteTaskSyncResponseHeader extends Message { + /** + * @generated from field: flyteidl.admin.Resource resource = 1; + */ + resource?: Resource; + + constructor(data?: PartialMessage) { + super(); + proto3.util.initPartial(data, this); + } + + static readonly runtime: typeof proto3 = proto3; + static readonly typeName = "flyteidl.admin.ExecuteTaskSyncResponseHeader"; + static readonly fields: FieldList = proto3.util.newFieldList(() => [ + { no: 1, name: "resource", kind: "message", T: Resource }, + ]); + + static fromBinary(bytes: Uint8Array, options?: Partial): ExecuteTaskSyncResponseHeader { + return new ExecuteTaskSyncResponseHeader().fromBinary(bytes, options); + } + + static fromJson(jsonValue: JsonValue, options?: Partial): ExecuteTaskSyncResponseHeader { + return new ExecuteTaskSyncResponseHeader().fromJson(jsonValue, options); + } + + static fromJsonString(jsonString: string, options?: Partial): ExecuteTaskSyncResponseHeader { + return new ExecuteTaskSyncResponseHeader().fromJsonString(jsonString, options); + } + + static equals(a: ExecuteTaskSyncResponseHeader | PlainMessage | undefined, b: ExecuteTaskSyncResponseHeader | PlainMessage | undefined): boolean { + return proto3.util.equals(ExecuteTaskSyncResponseHeader, a, b); + } +} + +/** + * @generated from message flyteidl.admin.ExecuteTaskSyncResponse + */ +export class ExecuteTaskSyncResponse extends Message { + /** + * Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). + * Resource is for synchronous task execution. + * + * @generated from oneof flyteidl.admin.ExecuteTaskSyncResponse.res + */ + res: { + /** + * @generated from field: flyteidl.admin.ExecuteTaskSyncResponseHeader header = 1; + */ + value: ExecuteTaskSyncResponseHeader; + case: "header"; + } | { + /** + * @generated from field: flyteidl.core.LiteralMap outputs = 2; + */ + value: LiteralMap; + case: "outputs"; + } | { case: undefined; value?: undefined } = { case: undefined }; + + constructor(data?: PartialMessage) { + super(); + proto3.util.initPartial(data, this); + } + + static readonly runtime: typeof proto3 = proto3; + static readonly typeName = "flyteidl.admin.ExecuteTaskSyncResponse"; + static readonly fields: FieldList = proto3.util.newFieldList(() => [ + { no: 1, name: "header", kind: "message", T: ExecuteTaskSyncResponseHeader, oneof: "res" }, + { no: 2, name: "outputs", kind: "message", T: LiteralMap, oneof: "res" }, + ]); + + static fromBinary(bytes: Uint8Array, options?: Partial): ExecuteTaskSyncResponse { + return new ExecuteTaskSyncResponse().fromBinary(bytes, options); + } + + static fromJson(jsonValue: JsonValue, options?: Partial): ExecuteTaskSyncResponse { + return new ExecuteTaskSyncResponse().fromJson(jsonValue, options); + } + + static fromJsonString(jsonString: string, options?: Partial): ExecuteTaskSyncResponse { + return new ExecuteTaskSyncResponse().fromJsonString(jsonString, options); + } + + static equals(a: ExecuteTaskSyncResponse | PlainMessage | undefined, b: ExecuteTaskSyncResponse | PlainMessage | undefined): boolean { + return proto3.util.equals(ExecuteTaskSyncResponse, a, b); + } +} + /** * A message used to fetch a job resource from flyte agent server. * @@ -263,7 +491,8 @@ export class GetTaskRequest extends Message { /** * A predefined yet extensible Task type identifier. * - * @generated from field: string task_type = 1; + * @generated from field: string task_type = 1 [deprecated = true]; + * @deprecated */ taskType = ""; @@ -274,6 +503,13 @@ export class GetTaskRequest extends Message { */ resourceMeta = new Uint8Array(0); + /** + * A predefined yet extensible Task type identifier. + * + * @generated from field: flyteidl.admin.TaskCategory task_category = 3; + */ + taskCategory?: TaskCategory; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -284,6 +520,7 @@ export class GetTaskRequest extends Message { static readonly fields: FieldList = proto3.util.newFieldList(() => [ { no: 1, name: "task_type", kind: "scalar", T: 9 /* ScalarType.STRING */ }, { no: 2, name: "resource_meta", kind: "scalar", T: 12 /* ScalarType.BYTES */ }, + { no: 3, name: "task_category", kind: "message", T: TaskCategory }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): GetTaskRequest { @@ -314,13 +551,6 @@ export class GetTaskResponse extends Message { */ resource?: Resource; - /** - * log information for the task execution - * - * @generated from field: repeated flyteidl.core.TaskLog log_links = 2; - */ - logLinks: TaskLog[] = []; - constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -330,7 +560,6 @@ export class GetTaskResponse extends Message { static readonly typeName = "flyteidl.admin.GetTaskResponse"; static readonly fields: FieldList = proto3.util.newFieldList(() => [ { no: 1, name: "resource", kind: "message", T: Resource }, - { no: 2, name: "log_links", kind: "message", T: TaskLog, repeated: true }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): GetTaskResponse { @@ -392,6 +621,13 @@ export class Resource extends Message { */ phase = TaskExecution_Phase.UNDEFINED; + /** + * Custom data specific to the agent. + * + * @generated from field: google.protobuf.Struct custom_info = 6; + */ + customInfo?: Struct; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -405,6 +641,7 @@ export class Resource extends Message { { no: 3, name: "message", kind: "scalar", T: 9 /* ScalarType.STRING */ }, { no: 4, name: "log_links", kind: "message", T: TaskLog, repeated: true }, { no: 5, name: "phase", kind: "enum", T: proto3.getEnumType(TaskExecution_Phase) }, + { no: 6, name: "custom_info", kind: "message", T: Struct }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): Resource { @@ -433,7 +670,8 @@ export class DeleteTaskRequest extends Message { /** * A predefined yet extensible Task type identifier. * - * @generated from field: string task_type = 1; + * @generated from field: string task_type = 1 [deprecated = true]; + * @deprecated */ taskType = ""; @@ -444,6 +682,13 @@ export class DeleteTaskRequest extends Message { */ resourceMeta = new Uint8Array(0); + /** + * A predefined yet extensible Task type identifier. + * + * @generated from field: flyteidl.admin.TaskCategory task_category = 3; + */ + taskCategory?: TaskCategory; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -454,6 +699,7 @@ export class DeleteTaskRequest extends Message { static readonly fields: FieldList = proto3.util.newFieldList(() => [ { no: 1, name: "task_type", kind: "scalar", T: 9 /* ScalarType.STRING */ }, { no: 2, name: "resource_meta", kind: "scalar", T: 12 /* ScalarType.BYTES */ }, + { no: 3, name: "task_category", kind: "message", T: TaskCategory }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): DeleteTaskRequest { @@ -522,10 +768,29 @@ export class Agent extends Message { /** * SupportedTaskTypes are the types of the tasks that the agent can handle. * - * @generated from field: repeated string supported_task_types = 2; + * @generated from field: repeated string supported_task_types = 2 [deprecated = true]; + * @deprecated */ supportedTaskTypes: string[] = []; + /** + * IsSync indicates whether this agent is a sync agent. Sync agents are expected to return their + * results synchronously when called by propeller. Given that sync agents can affect the performance + * of the system, it's important to enforce strict timeout policies. + * An Async agent, on the other hand, is required to be able to identify jobs by an + * identifier and query for job statuses as jobs progress. + * + * @generated from field: bool is_sync = 3; + */ + isSync = false; + + /** + * Supported_task_categories are the categories of the tasks that the agent can handle. + * + * @generated from field: repeated flyteidl.admin.TaskCategory supported_task_categories = 4; + */ + supportedTaskCategories: TaskCategory[] = []; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -536,6 +801,8 @@ export class Agent extends Message { static readonly fields: FieldList = proto3.util.newFieldList(() => [ { no: 1, name: "name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, { no: 2, name: "supported_task_types", kind: "scalar", T: 9 /* ScalarType.STRING */, repeated: true }, + { no: 3, name: "is_sync", kind: "scalar", T: 8 /* ScalarType.BOOL */ }, + { no: 4, name: "supported_task_categories", kind: "message", T: TaskCategory, repeated: true }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): Agent { @@ -555,6 +822,53 @@ export class Agent extends Message { } } +/** + * @generated from message flyteidl.admin.TaskCategory + */ +export class TaskCategory extends Message { + /** + * The name of the task type. + * + * @generated from field: string name = 1; + */ + name = ""; + + /** + * The version of the task type. + * + * @generated from field: int32 version = 2; + */ + version = 0; + + constructor(data?: PartialMessage) { + super(); + proto3.util.initPartial(data, this); + } + + static readonly runtime: typeof proto3 = proto3; + static readonly typeName = "flyteidl.admin.TaskCategory"; + static readonly fields: FieldList = proto3.util.newFieldList(() => [ + { no: 1, name: "name", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 2, name: "version", kind: "scalar", T: 5 /* ScalarType.INT32 */ }, + ]); + + static fromBinary(bytes: Uint8Array, options?: Partial): TaskCategory { + return new TaskCategory().fromBinary(bytes, options); + } + + static fromJson(jsonValue: JsonValue, options?: Partial): TaskCategory { + return new TaskCategory().fromJson(jsonValue, options); + } + + static fromJsonString(jsonString: string, options?: Partial): TaskCategory { + return new TaskCategory().fromJsonString(jsonString, options); + } + + static equals(a: TaskCategory | PlainMessage | undefined, b: TaskCategory | PlainMessage | undefined): boolean { + return proto3.util.equals(TaskCategory, a, b); + } +} + /** * A request to get an agent. * @@ -716,7 +1030,8 @@ export class GetTaskMetricsRequest extends Message { /** * A predefined yet extensible Task type identifier. * - * @generated from field: string task_type = 1; + * @generated from field: string task_type = 1 [deprecated = true]; + * @deprecated */ taskType = ""; @@ -756,6 +1071,13 @@ export class GetTaskMetricsRequest extends Message { */ step?: Duration; + /** + * A predefined yet extensible Task type identifier. + * + * @generated from field: flyteidl.admin.TaskCategory task_category = 7; + */ + taskCategory?: TaskCategory; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -770,6 +1092,7 @@ export class GetTaskMetricsRequest extends Message { { no: 4, name: "start_time", kind: "message", T: Timestamp }, { no: 5, name: "end_time", kind: "message", T: Timestamp }, { no: 6, name: "step", kind: "message", T: Duration }, + { no: 7, name: "task_category", kind: "message", T: TaskCategory }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): GetTaskMetricsRequest { @@ -839,7 +1162,8 @@ export class GetTaskLogsRequest extends Message { /** * A predefined yet extensible Task type identifier. * - * @generated from field: string task_type = 1; + * @generated from field: string task_type = 1 [deprecated = true]; + * @deprecated */ taskType = ""; @@ -865,6 +1189,13 @@ export class GetTaskLogsRequest extends Message { */ token = ""; + /** + * A predefined yet extensible Task type identifier. + * + * @generated from field: flyteidl.admin.TaskCategory task_category = 5; + */ + taskCategory?: TaskCategory; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -877,6 +1208,7 @@ export class GetTaskLogsRequest extends Message { { no: 2, name: "resource_meta", kind: "scalar", T: 12 /* ScalarType.BYTES */ }, { no: 3, name: "lines", kind: "scalar", T: 4 /* ScalarType.UINT64 */ }, { no: 4, name: "token", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 5, name: "task_category", kind: "message", T: TaskCategory }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): GetTaskLogsRequest { @@ -897,11 +1229,49 @@ export class GetTaskLogsRequest extends Message { } /** - * A response containing the logs for a task execution. - * - * @generated from message flyteidl.admin.GetTaskLogsResponse + * @generated from message flyteidl.admin.GetTaskLogsResponseHeader */ -export class GetTaskLogsResponse extends Message { +export class GetTaskLogsResponseHeader extends Message { + /** + * In the case of multiple pages of results, the server-provided token can be used to fetch the next page + * in a query. If there are no more results, this value will be empty. + * + * @generated from field: string token = 1; + */ + token = ""; + + constructor(data?: PartialMessage) { + super(); + proto3.util.initPartial(data, this); + } + + static readonly runtime: typeof proto3 = proto3; + static readonly typeName = "flyteidl.admin.GetTaskLogsResponseHeader"; + static readonly fields: FieldList = proto3.util.newFieldList(() => [ + { no: 1, name: "token", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + ]); + + static fromBinary(bytes: Uint8Array, options?: Partial): GetTaskLogsResponseHeader { + return new GetTaskLogsResponseHeader().fromBinary(bytes, options); + } + + static fromJson(jsonValue: JsonValue, options?: Partial): GetTaskLogsResponseHeader { + return new GetTaskLogsResponseHeader().fromJson(jsonValue, options); + } + + static fromJsonString(jsonString: string, options?: Partial): GetTaskLogsResponseHeader { + return new GetTaskLogsResponseHeader().fromJsonString(jsonString, options); + } + + static equals(a: GetTaskLogsResponseHeader | PlainMessage | undefined, b: GetTaskLogsResponseHeader | PlainMessage | undefined): boolean { + return proto3.util.equals(GetTaskLogsResponseHeader, a, b); + } +} + +/** + * @generated from message flyteidl.admin.GetTaskLogsResponseBody + */ +export class GetTaskLogsResponseBody extends Message { /** * The execution log results. * @@ -909,13 +1279,56 @@ export class GetTaskLogsResponse extends Message { */ results: string[] = []; + constructor(data?: PartialMessage) { + super(); + proto3.util.initPartial(data, this); + } + + static readonly runtime: typeof proto3 = proto3; + static readonly typeName = "flyteidl.admin.GetTaskLogsResponseBody"; + static readonly fields: FieldList = proto3.util.newFieldList(() => [ + { no: 1, name: "results", kind: "scalar", T: 9 /* ScalarType.STRING */, repeated: true }, + ]); + + static fromBinary(bytes: Uint8Array, options?: Partial): GetTaskLogsResponseBody { + return new GetTaskLogsResponseBody().fromBinary(bytes, options); + } + + static fromJson(jsonValue: JsonValue, options?: Partial): GetTaskLogsResponseBody { + return new GetTaskLogsResponseBody().fromJson(jsonValue, options); + } + + static fromJsonString(jsonString: string, options?: Partial): GetTaskLogsResponseBody { + return new GetTaskLogsResponseBody().fromJsonString(jsonString, options); + } + + static equals(a: GetTaskLogsResponseBody | PlainMessage | undefined, b: GetTaskLogsResponseBody | PlainMessage | undefined): boolean { + return proto3.util.equals(GetTaskLogsResponseBody, a, b); + } +} + +/** + * A response containing the logs for a task execution. + * + * @generated from message flyteidl.admin.GetTaskLogsResponse + */ +export class GetTaskLogsResponse extends Message { /** - * In the case of multiple pages of results, the server-provided token can be used to fetch the next page - * in a query. If there are no more results, this value will be empty. - * - * @generated from field: string token = 2; + * @generated from oneof flyteidl.admin.GetTaskLogsResponse.part */ - token = ""; + part: { + /** + * @generated from field: flyteidl.admin.GetTaskLogsResponseHeader header = 1; + */ + value: GetTaskLogsResponseHeader; + case: "header"; + } | { + /** + * @generated from field: flyteidl.admin.GetTaskLogsResponseBody body = 2; + */ + value: GetTaskLogsResponseBody; + case: "body"; + } | { case: undefined; value?: undefined } = { case: undefined }; constructor(data?: PartialMessage) { super(); @@ -925,8 +1338,8 @@ export class GetTaskLogsResponse extends Message { static readonly runtime: typeof proto3 = proto3; static readonly typeName = "flyteidl.admin.GetTaskLogsResponse"; static readonly fields: FieldList = proto3.util.newFieldList(() => [ - { no: 1, name: "results", kind: "scalar", T: 9 /* ScalarType.STRING */, repeated: true }, - { no: 2, name: "token", kind: "scalar", T: 9 /* ScalarType.STRING */ }, + { no: 1, name: "header", kind: "message", T: GetTaskLogsResponseHeader, oneof: "part" }, + { no: 2, name: "body", kind: "message", T: GetTaskLogsResponseBody, oneof: "part" }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): GetTaskLogsResponse { diff --git a/flyteidl/gen/pb-es/flyteidl/admin/execution_pb.ts b/flyteidl/gen/pb-es/flyteidl/admin/execution_pb.ts index 409a27d96f..bce98af489 100644 --- a/flyteidl/gen/pb-es/flyteidl/admin/execution_pb.ts +++ b/flyteidl/gen/pb-es/flyteidl/admin/execution_pb.ts @@ -914,6 +914,13 @@ export enum ExecutionMetadata_ExecutionMode { * @generated from enum value: RECOVERED = 5; */ RECOVERED = 5, + + /** + * Execution was kicked off by the artifact trigger system + * + * @generated from enum value: TRIGGER = 6; + */ + TRIGGER = 6, } // Retrieve enum metadata with: proto3.getEnumType(ExecutionMetadata_ExecutionMode) proto3.util.setEnumType(ExecutionMetadata_ExecutionMode, "flyteidl.admin.ExecutionMetadata.ExecutionMode", [ @@ -923,6 +930,7 @@ proto3.util.setEnumType(ExecutionMetadata_ExecutionMode, "flyteidl.admin.Executi { no: 3, name: "RELAUNCH" }, { no: 4, name: "CHILD_WORKFLOW" }, { no: 5, name: "RECOVERED" }, + { no: 6, name: "TRIGGER" }, ]); /** diff --git a/flyteidl/gen/pb-es/flyteidl/core/artifact_id_pb.ts b/flyteidl/gen/pb-es/flyteidl/core/artifact_id_pb.ts index 2e03e3916f..42d063161d 100644 --- a/flyteidl/gen/pb-es/flyteidl/core/artifact_id_pb.ts +++ b/flyteidl/gen/pb-es/flyteidl/core/artifact_id_pb.ts @@ -69,11 +69,6 @@ export class ArtifactKey extends Message { * @generated from message flyteidl.core.ArtifactBindingData */ export class ArtifactBindingData extends Message { - /** - * @generated from field: uint32 index = 1; - */ - index = 0; - /** * These two fields are only relevant in the partition value case * @@ -108,7 +103,6 @@ export class ArtifactBindingData extends Message { static readonly runtime: typeof proto3 = proto3; static readonly typeName = "flyteidl.core.ArtifactBindingData"; static readonly fields: FieldList = proto3.util.newFieldList(() => [ - { no: 1, name: "index", kind: "scalar", T: 13 /* ScalarType.UINT32 */ }, { no: 2, name: "partition_key", kind: "scalar", T: 9 /* ScalarType.STRING */, oneof: "partition_data" }, { no: 3, name: "bind_to_time_partition", kind: "scalar", T: 8 /* ScalarType.BOOL */, oneof: "partition_data" }, { no: 4, name: "transform", kind: "scalar", T: 9 /* ScalarType.STRING */ }, diff --git a/flyteidl/gen/pb-es/flyteidl/core/workflow_pb.ts b/flyteidl/gen/pb-es/flyteidl/core/workflow_pb.ts index 644f792e63..414e6eb319 100644 --- a/flyteidl/gen/pb-es/flyteidl/core/workflow_pb.ts +++ b/flyteidl/gen/pb-es/flyteidl/core/workflow_pb.ts @@ -1121,6 +1121,13 @@ export class TaskNodeOverrides extends Message { */ extendedResources?: ExtendedResources; + /** + * Override for the image used by task pods. + * + * @generated from field: string container_image = 3; + */ + containerImage = ""; + constructor(data?: PartialMessage) { super(); proto3.util.initPartial(data, this); @@ -1131,6 +1138,7 @@ export class TaskNodeOverrides extends Message { static readonly fields: FieldList = proto3.util.newFieldList(() => [ { no: 1, name: "resources", kind: "message", T: Resources }, { no: 2, name: "extended_resources", kind: "message", T: ExtendedResources }, + { no: 3, name: "container_image", kind: "scalar", T: 9 /* ScalarType.STRING */ }, ]); static fromBinary(bytes: Uint8Array, options?: Partial): TaskNodeOverrides { diff --git a/flyteidl/gen/pb-es/flyteidl/service/agent_connect.ts b/flyteidl/gen/pb-es/flyteidl/service/agent_connect.ts index 3068a3181a..cec0c0aabc 100644 --- a/flyteidl/gen/pb-es/flyteidl/service/agent_connect.ts +++ b/flyteidl/gen/pb-es/flyteidl/service/agent_connect.ts @@ -3,11 +3,33 @@ /* eslint-disable */ // @ts-nocheck -import { CreateTaskRequest, CreateTaskResponse, DeleteTaskRequest, DeleteTaskResponse, GetAgentRequest, GetAgentResponse, GetTaskLogsRequest, GetTaskLogsResponse, GetTaskMetricsRequest, GetTaskMetricsResponse, GetTaskRequest, GetTaskResponse, ListAgentsRequest, ListAgentsResponse } from "../admin/agent_pb.js"; +import { CreateTaskRequest, CreateTaskResponse, DeleteTaskRequest, DeleteTaskResponse, ExecuteTaskSyncRequest, ExecuteTaskSyncResponse, GetAgentRequest, GetAgentResponse, GetTaskLogsRequest, GetTaskLogsResponse, GetTaskMetricsRequest, GetTaskMetricsResponse, GetTaskRequest, GetTaskResponse, ListAgentsRequest, ListAgentsResponse } from "../admin/agent_pb.js"; import { MethodKind } from "@bufbuild/protobuf"; /** - * AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server. + * SyncAgentService defines an RPC Service that allows propeller to send the request to the agent server synchronously. + * + * @generated from service flyteidl.service.SyncAgentService + */ +export const SyncAgentService = { + typeName: "flyteidl.service.SyncAgentService", + methods: { + /** + * ExecuteTaskSync streams the create request and inputs to the agent service and streams the outputs back. + * + * @generated from rpc flyteidl.service.SyncAgentService.ExecuteTaskSync + */ + executeTaskSync: { + name: "ExecuteTaskSync", + I: ExecuteTaskSyncRequest, + O: ExecuteTaskSyncResponse, + kind: MethodKind.BiDiStreaming, + }, + } +} as const; + +/** + * AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server asynchronously. * * @generated from service flyteidl.service.AsyncAgentService */ @@ -15,7 +37,7 @@ export const AsyncAgentService = { typeName: "flyteidl.service.AsyncAgentService", methods: { /** - * Send a task create request to the agent server. + * CreateTask sends a task create request to the agent service. * * @generated from rpc flyteidl.service.AsyncAgentService.CreateTask */ @@ -71,7 +93,7 @@ export const AsyncAgentService = { name: "GetTaskLogs", I: GetTaskLogsRequest, O: GetTaskLogsResponse, - kind: MethodKind.Unary, + kind: MethodKind.ServerStreaming, }, } } as const; diff --git a/flyteidl/gen/pb-go/flyteidl/admin/agent.pb.go b/flyteidl/gen/pb-go/flyteidl/admin/agent.pb.go index de0a0f0244..01aee40e95 100644 --- a/flyteidl/gen/pb-go/flyteidl/admin/agent.pb.go +++ b/flyteidl/gen/pb-go/flyteidl/admin/agent.pb.go @@ -11,6 +11,7 @@ import ( protoreflect "google.golang.org/protobuf/reflect/protoreflect" protoimpl "google.golang.org/protobuf/runtime/protoimpl" durationpb "google.golang.org/protobuf/types/known/durationpb" + structpb "google.golang.org/protobuf/types/known/structpb" timestamppb "google.golang.org/protobuf/types/known/timestamppb" reflect "reflect" sync "sync" @@ -24,6 +25,8 @@ const ( ) // The state of the execution is used to control its visibility in the UI/CLI. +// +// Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. type State int32 const ( @@ -97,6 +100,19 @@ type TaskExecutionMetadata struct { K8SServiceAccount string `protobuf:"bytes,5,opt,name=k8s_service_account,json=k8sServiceAccount,proto3" json:"k8s_service_account,omitempty"` // Environment variables attached to the task execution EnvironmentVariables map[string]string `protobuf:"bytes,6,rep,name=environment_variables,json=environmentVariables,proto3" json:"environment_variables,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` + // Represents the maximum number of attempts allowed for a task. + // If a task fails, it can be retried up to this maximum number of attempts. + MaxAttempts int32 `protobuf:"varint,7,opt,name=max_attempts,json=maxAttempts,proto3" json:"max_attempts,omitempty"` + // Indicates whether the task execution can be interrupted. + // If set to true, the task can be stopped before completion. + Interruptible bool `protobuf:"varint,8,opt,name=interruptible,proto3" json:"interruptible,omitempty"` + // Specifies the threshold for failure count at which the interruptible property + // will take effect. If the number of consecutive task failures exceeds this threshold, + // interruptible behavior will be activated. + InterruptibleFailureThreshold int32 `protobuf:"varint,9,opt,name=interruptible_failure_threshold,json=interruptibleFailureThreshold,proto3" json:"interruptible_failure_threshold,omitempty"` + // Overrides for specific properties of the task node. + // These overrides can be used to customize the behavior of the task node. + Overrides *core.TaskNodeOverrides `protobuf:"bytes,10,opt,name=overrides,proto3" json:"overrides,omitempty"` } func (x *TaskExecutionMetadata) Reset() { @@ -173,6 +189,34 @@ func (x *TaskExecutionMetadata) GetEnvironmentVariables() map[string]string { return nil } +func (x *TaskExecutionMetadata) GetMaxAttempts() int32 { + if x != nil { + return x.MaxAttempts + } + return 0 +} + +func (x *TaskExecutionMetadata) GetInterruptible() bool { + if x != nil { + return x.Interruptible + } + return false +} + +func (x *TaskExecutionMetadata) GetInterruptibleFailureThreshold() int32 { + if x != nil { + return x.InterruptibleFailureThreshold + } + return 0 +} + +func (x *TaskExecutionMetadata) GetOverrides() *core.TaskNodeOverrides { + if x != nil { + return x.Overrides + } + return nil +} + // Represents a request structure to create task. type CreateTaskRequest struct { state protoimpl.MessageState @@ -257,14 +301,8 @@ type CreateTaskResponse struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - // Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). - // Resource is for synchronous task execution. - // - // Types that are assignable to Res: - // - // *CreateTaskResponse_ResourceMeta - // *CreateTaskResponse_Resource - Res isCreateTaskResponse_Res `protobuf_oneof:"res"` + // ResourceMeta is created by the agent. It could be a string (jobId) or a dict (more complex metadata). + ResourceMeta []byte `protobuf:"bytes,1,opt,name=resource_meta,json=resourceMeta,proto3" json:"resource_meta,omitempty"` } func (x *CreateTaskResponse) Reset() { @@ -299,42 +337,299 @@ func (*CreateTaskResponse) Descriptor() ([]byte, []int) { return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{2} } -func (m *CreateTaskResponse) GetRes() isCreateTaskResponse_Res { +func (x *CreateTaskResponse) GetResourceMeta() []byte { + if x != nil { + return x.ResourceMeta + } + return nil +} + +type CreateRequestHeader struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Template of the task that encapsulates all the metadata of the task. + Template *core.TaskTemplate `protobuf:"bytes,1,opt,name=template,proto3" json:"template,omitempty"` + // Prefix for where task output data will be written. (e.g. s3://my-bucket/randomstring) + OutputPrefix string `protobuf:"bytes,2,opt,name=output_prefix,json=outputPrefix,proto3" json:"output_prefix,omitempty"` + // subset of runtime task execution metadata. + TaskExecutionMetadata *TaskExecutionMetadata `protobuf:"bytes,3,opt,name=task_execution_metadata,json=taskExecutionMetadata,proto3" json:"task_execution_metadata,omitempty"` + // MaxDatasetSizeBytes is the maximum size of the dataset that can be generated by the task. + MaxDatasetSizeBytes int64 `protobuf:"varint,4,opt,name=max_dataset_size_bytes,json=maxDatasetSizeBytes,proto3" json:"max_dataset_size_bytes,omitempty"` +} + +func (x *CreateRequestHeader) Reset() { + *x = CreateRequestHeader{} + if protoimpl.UnsafeEnabled { + mi := &file_flyteidl_admin_agent_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *CreateRequestHeader) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*CreateRequestHeader) ProtoMessage() {} + +func (x *CreateRequestHeader) ProtoReflect() protoreflect.Message { + mi := &file_flyteidl_admin_agent_proto_msgTypes[3] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use CreateRequestHeader.ProtoReflect.Descriptor instead. +func (*CreateRequestHeader) Descriptor() ([]byte, []int) { + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{3} +} + +func (x *CreateRequestHeader) GetTemplate() *core.TaskTemplate { + if x != nil { + return x.Template + } + return nil +} + +func (x *CreateRequestHeader) GetOutputPrefix() string { + if x != nil { + return x.OutputPrefix + } + return "" +} + +func (x *CreateRequestHeader) GetTaskExecutionMetadata() *TaskExecutionMetadata { + if x != nil { + return x.TaskExecutionMetadata + } + return nil +} + +func (x *CreateRequestHeader) GetMaxDatasetSizeBytes() int64 { + if x != nil { + return x.MaxDatasetSizeBytes + } + return 0 +} + +type ExecuteTaskSyncRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Types that are assignable to Part: + // + // *ExecuteTaskSyncRequest_Header + // *ExecuteTaskSyncRequest_Inputs + Part isExecuteTaskSyncRequest_Part `protobuf_oneof:"part"` +} + +func (x *ExecuteTaskSyncRequest) Reset() { + *x = ExecuteTaskSyncRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_flyteidl_admin_agent_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ExecuteTaskSyncRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ExecuteTaskSyncRequest) ProtoMessage() {} + +func (x *ExecuteTaskSyncRequest) ProtoReflect() protoreflect.Message { + mi := &file_flyteidl_admin_agent_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ExecuteTaskSyncRequest.ProtoReflect.Descriptor instead. +func (*ExecuteTaskSyncRequest) Descriptor() ([]byte, []int) { + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{4} +} + +func (m *ExecuteTaskSyncRequest) GetPart() isExecuteTaskSyncRequest_Part { if m != nil { - return m.Res + return m.Part } return nil } -func (x *CreateTaskResponse) GetResourceMeta() []byte { - if x, ok := x.GetRes().(*CreateTaskResponse_ResourceMeta); ok { - return x.ResourceMeta +func (x *ExecuteTaskSyncRequest) GetHeader() *CreateRequestHeader { + if x, ok := x.GetPart().(*ExecuteTaskSyncRequest_Header); ok { + return x.Header + } + return nil +} + +func (x *ExecuteTaskSyncRequest) GetInputs() *core.LiteralMap { + if x, ok := x.GetPart().(*ExecuteTaskSyncRequest_Inputs); ok { + return x.Inputs } return nil } -func (x *CreateTaskResponse) GetResource() *Resource { - if x, ok := x.GetRes().(*CreateTaskResponse_Resource); ok { +type isExecuteTaskSyncRequest_Part interface { + isExecuteTaskSyncRequest_Part() +} + +type ExecuteTaskSyncRequest_Header struct { + Header *CreateRequestHeader `protobuf:"bytes,1,opt,name=header,proto3,oneof"` +} + +type ExecuteTaskSyncRequest_Inputs struct { + Inputs *core.LiteralMap `protobuf:"bytes,2,opt,name=inputs,proto3,oneof"` +} + +func (*ExecuteTaskSyncRequest_Header) isExecuteTaskSyncRequest_Part() {} + +func (*ExecuteTaskSyncRequest_Inputs) isExecuteTaskSyncRequest_Part() {} + +type ExecuteTaskSyncResponseHeader struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Resource *Resource `protobuf:"bytes,1,opt,name=resource,proto3" json:"resource,omitempty"` +} + +func (x *ExecuteTaskSyncResponseHeader) Reset() { + *x = ExecuteTaskSyncResponseHeader{} + if protoimpl.UnsafeEnabled { + mi := &file_flyteidl_admin_agent_proto_msgTypes[5] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ExecuteTaskSyncResponseHeader) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ExecuteTaskSyncResponseHeader) ProtoMessage() {} + +func (x *ExecuteTaskSyncResponseHeader) ProtoReflect() protoreflect.Message { + mi := &file_flyteidl_admin_agent_proto_msgTypes[5] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ExecuteTaskSyncResponseHeader.ProtoReflect.Descriptor instead. +func (*ExecuteTaskSyncResponseHeader) Descriptor() ([]byte, []int) { + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{5} +} + +func (x *ExecuteTaskSyncResponseHeader) GetResource() *Resource { + if x != nil { return x.Resource } return nil } -type isCreateTaskResponse_Res interface { - isCreateTaskResponse_Res() +type ExecuteTaskSyncResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). + // Resource is for synchronous task execution. + // + // Types that are assignable to Res: + // + // *ExecuteTaskSyncResponse_Header + // *ExecuteTaskSyncResponse_Outputs + Res isExecuteTaskSyncResponse_Res `protobuf_oneof:"res"` +} + +func (x *ExecuteTaskSyncResponse) Reset() { + *x = ExecuteTaskSyncResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_flyteidl_admin_agent_proto_msgTypes[6] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ExecuteTaskSyncResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ExecuteTaskSyncResponse) ProtoMessage() {} + +func (x *ExecuteTaskSyncResponse) ProtoReflect() protoreflect.Message { + mi := &file_flyteidl_admin_agent_proto_msgTypes[6] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ExecuteTaskSyncResponse.ProtoReflect.Descriptor instead. +func (*ExecuteTaskSyncResponse) Descriptor() ([]byte, []int) { + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{6} +} + +func (m *ExecuteTaskSyncResponse) GetRes() isExecuteTaskSyncResponse_Res { + if m != nil { + return m.Res + } + return nil +} + +func (x *ExecuteTaskSyncResponse) GetHeader() *ExecuteTaskSyncResponseHeader { + if x, ok := x.GetRes().(*ExecuteTaskSyncResponse_Header); ok { + return x.Header + } + return nil +} + +func (x *ExecuteTaskSyncResponse) GetOutputs() *core.LiteralMap { + if x, ok := x.GetRes().(*ExecuteTaskSyncResponse_Outputs); ok { + return x.Outputs + } + return nil +} + +type isExecuteTaskSyncResponse_Res interface { + isExecuteTaskSyncResponse_Res() } -type CreateTaskResponse_ResourceMeta struct { - ResourceMeta []byte `protobuf:"bytes,1,opt,name=resource_meta,json=resourceMeta,proto3,oneof"` +type ExecuteTaskSyncResponse_Header struct { + Header *ExecuteTaskSyncResponseHeader `protobuf:"bytes,1,opt,name=header,proto3,oneof"` } -type CreateTaskResponse_Resource struct { - Resource *Resource `protobuf:"bytes,2,opt,name=resource,proto3,oneof"` +type ExecuteTaskSyncResponse_Outputs struct { + Outputs *core.LiteralMap `protobuf:"bytes,2,opt,name=outputs,proto3,oneof"` } -func (*CreateTaskResponse_ResourceMeta) isCreateTaskResponse_Res() {} +func (*ExecuteTaskSyncResponse_Header) isExecuteTaskSyncResponse_Res() {} -func (*CreateTaskResponse_Resource) isCreateTaskResponse_Res() {} +func (*ExecuteTaskSyncResponse_Outputs) isExecuteTaskSyncResponse_Res() {} // A message used to fetch a job resource from flyte agent server. type GetTaskRequest struct { @@ -343,15 +638,19 @@ type GetTaskRequest struct { unknownFields protoimpl.UnknownFields // A predefined yet extensible Task type identifier. + // + // Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. TaskType string `protobuf:"bytes,1,opt,name=task_type,json=taskType,proto3" json:"task_type,omitempty"` // Metadata about the resource to be pass to the agent. ResourceMeta []byte `protobuf:"bytes,2,opt,name=resource_meta,json=resourceMeta,proto3" json:"resource_meta,omitempty"` + // A predefined yet extensible Task type identifier. + TaskCategory *TaskCategory `protobuf:"bytes,3,opt,name=task_category,json=taskCategory,proto3" json:"task_category,omitempty"` } func (x *GetTaskRequest) Reset() { *x = GetTaskRequest{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[3] + mi := &file_flyteidl_admin_agent_proto_msgTypes[7] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -364,7 +663,7 @@ func (x *GetTaskRequest) String() string { func (*GetTaskRequest) ProtoMessage() {} func (x *GetTaskRequest) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[3] + mi := &file_flyteidl_admin_agent_proto_msgTypes[7] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -377,9 +676,10 @@ func (x *GetTaskRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetTaskRequest.ProtoReflect.Descriptor instead. func (*GetTaskRequest) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{3} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{7} } +// Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. func (x *GetTaskRequest) GetTaskType() string { if x != nil { return x.TaskType @@ -394,6 +694,13 @@ func (x *GetTaskRequest) GetResourceMeta() []byte { return nil } +func (x *GetTaskRequest) GetTaskCategory() *TaskCategory { + if x != nil { + return x.TaskCategory + } + return nil +} + // Response to get an individual task resource. type GetTaskResponse struct { state protoimpl.MessageState @@ -401,14 +708,12 @@ type GetTaskResponse struct { unknownFields protoimpl.UnknownFields Resource *Resource `protobuf:"bytes,1,opt,name=resource,proto3" json:"resource,omitempty"` - // log information for the task execution - LogLinks []*core.TaskLog `protobuf:"bytes,2,rep,name=log_links,json=logLinks,proto3" json:"log_links,omitempty"` } func (x *GetTaskResponse) Reset() { *x = GetTaskResponse{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[4] + mi := &file_flyteidl_admin_agent_proto_msgTypes[8] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -421,7 +726,7 @@ func (x *GetTaskResponse) String() string { func (*GetTaskResponse) ProtoMessage() {} func (x *GetTaskResponse) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[4] + mi := &file_flyteidl_admin_agent_proto_msgTypes[8] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -434,7 +739,7 @@ func (x *GetTaskResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetTaskResponse.ProtoReflect.Descriptor instead. func (*GetTaskResponse) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{4} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{8} } func (x *GetTaskResponse) GetResource() *Resource { @@ -444,13 +749,6 @@ func (x *GetTaskResponse) GetResource() *Resource { return nil } -func (x *GetTaskResponse) GetLogLinks() []*core.TaskLog { - if x != nil { - return x.LogLinks - } - return nil -} - type Resource struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -470,12 +768,14 @@ type Resource struct { LogLinks []*core.TaskLog `protobuf:"bytes,4,rep,name=log_links,json=logLinks,proto3" json:"log_links,omitempty"` // The phase of the execution is used to determine the phase of the plugin's execution. Phase core.TaskExecution_Phase `protobuf:"varint,5,opt,name=phase,proto3,enum=flyteidl.core.TaskExecution_Phase" json:"phase,omitempty"` + // Custom data specific to the agent. + CustomInfo *structpb.Struct `protobuf:"bytes,6,opt,name=custom_info,json=customInfo,proto3" json:"custom_info,omitempty"` } func (x *Resource) Reset() { *x = Resource{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[5] + mi := &file_flyteidl_admin_agent_proto_msgTypes[9] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -488,7 +788,7 @@ func (x *Resource) String() string { func (*Resource) ProtoMessage() {} func (x *Resource) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[5] + mi := &file_flyteidl_admin_agent_proto_msgTypes[9] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -501,7 +801,7 @@ func (x *Resource) ProtoReflect() protoreflect.Message { // Deprecated: Use Resource.ProtoReflect.Descriptor instead. func (*Resource) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{5} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{9} } // Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. @@ -540,6 +840,13 @@ func (x *Resource) GetPhase() core.TaskExecution_Phase { return core.TaskExecution_Phase(0) } +func (x *Resource) GetCustomInfo() *structpb.Struct { + if x != nil { + return x.CustomInfo + } + return nil +} + // A message used to delete a task. type DeleteTaskRequest struct { state protoimpl.MessageState @@ -547,15 +854,19 @@ type DeleteTaskRequest struct { unknownFields protoimpl.UnknownFields // A predefined yet extensible Task type identifier. + // + // Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. TaskType string `protobuf:"bytes,1,opt,name=task_type,json=taskType,proto3" json:"task_type,omitempty"` // Metadata about the resource to be pass to the agent. ResourceMeta []byte `protobuf:"bytes,2,opt,name=resource_meta,json=resourceMeta,proto3" json:"resource_meta,omitempty"` + // A predefined yet extensible Task type identifier. + TaskCategory *TaskCategory `protobuf:"bytes,3,opt,name=task_category,json=taskCategory,proto3" json:"task_category,omitempty"` } func (x *DeleteTaskRequest) Reset() { *x = DeleteTaskRequest{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[6] + mi := &file_flyteidl_admin_agent_proto_msgTypes[10] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -568,7 +879,7 @@ func (x *DeleteTaskRequest) String() string { func (*DeleteTaskRequest) ProtoMessage() {} func (x *DeleteTaskRequest) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[6] + mi := &file_flyteidl_admin_agent_proto_msgTypes[10] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -581,9 +892,10 @@ func (x *DeleteTaskRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use DeleteTaskRequest.ProtoReflect.Descriptor instead. func (*DeleteTaskRequest) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{6} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{10} } +// Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. func (x *DeleteTaskRequest) GetTaskType() string { if x != nil { return x.TaskType @@ -598,6 +910,13 @@ func (x *DeleteTaskRequest) GetResourceMeta() []byte { return nil } +func (x *DeleteTaskRequest) GetTaskCategory() *TaskCategory { + if x != nil { + return x.TaskCategory + } + return nil +} + // Response to delete a task. type DeleteTaskResponse struct { state protoimpl.MessageState @@ -608,7 +927,7 @@ type DeleteTaskResponse struct { func (x *DeleteTaskResponse) Reset() { *x = DeleteTaskResponse{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[7] + mi := &file_flyteidl_admin_agent_proto_msgTypes[11] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -621,7 +940,7 @@ func (x *DeleteTaskResponse) String() string { func (*DeleteTaskResponse) ProtoMessage() {} func (x *DeleteTaskResponse) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[7] + mi := &file_flyteidl_admin_agent_proto_msgTypes[11] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -634,7 +953,7 @@ func (x *DeleteTaskResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use DeleteTaskResponse.ProtoReflect.Descriptor instead. func (*DeleteTaskResponse) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{7} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{11} } // A message containing the agent metadata. @@ -646,13 +965,23 @@ type Agent struct { // Name is the developer-assigned name of the agent. Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` // SupportedTaskTypes are the types of the tasks that the agent can handle. + // + // Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. SupportedTaskTypes []string `protobuf:"bytes,2,rep,name=supported_task_types,json=supportedTaskTypes,proto3" json:"supported_task_types,omitempty"` + // IsSync indicates whether this agent is a sync agent. Sync agents are expected to return their + // results synchronously when called by propeller. Given that sync agents can affect the performance + // of the system, it's important to enforce strict timeout policies. + // An Async agent, on the other hand, is required to be able to identify jobs by an + // identifier and query for job statuses as jobs progress. + IsSync bool `protobuf:"varint,3,opt,name=is_sync,json=isSync,proto3" json:"is_sync,omitempty"` + // Supported_task_categories are the categories of the tasks that the agent can handle. + SupportedTaskCategories []*TaskCategory `protobuf:"bytes,4,rep,name=supported_task_categories,json=supportedTaskCategories,proto3" json:"supported_task_categories,omitempty"` } func (x *Agent) Reset() { *x = Agent{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[8] + mi := &file_flyteidl_admin_agent_proto_msgTypes[12] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -665,7 +994,7 @@ func (x *Agent) String() string { func (*Agent) ProtoMessage() {} func (x *Agent) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[8] + mi := &file_flyteidl_admin_agent_proto_msgTypes[12] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -678,21 +1007,93 @@ func (x *Agent) ProtoReflect() protoreflect.Message { // Deprecated: Use Agent.ProtoReflect.Descriptor instead. func (*Agent) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{8} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{12} +} + +func (x *Agent) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +// Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. +func (x *Agent) GetSupportedTaskTypes() []string { + if x != nil { + return x.SupportedTaskTypes + } + return nil +} + +func (x *Agent) GetIsSync() bool { + if x != nil { + return x.IsSync + } + return false +} + +func (x *Agent) GetSupportedTaskCategories() []*TaskCategory { + if x != nil { + return x.SupportedTaskCategories + } + return nil +} + +type TaskCategory struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The name of the task type. + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // The version of the task type. + Version int32 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"` +} + +func (x *TaskCategory) Reset() { + *x = TaskCategory{} + if protoimpl.UnsafeEnabled { + mi := &file_flyteidl_admin_agent_proto_msgTypes[13] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TaskCategory) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TaskCategory) ProtoMessage() {} + +func (x *TaskCategory) ProtoReflect() protoreflect.Message { + mi := &file_flyteidl_admin_agent_proto_msgTypes[13] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TaskCategory.ProtoReflect.Descriptor instead. +func (*TaskCategory) Descriptor() ([]byte, []int) { + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{13} } -func (x *Agent) GetName() string { +func (x *TaskCategory) GetName() string { if x != nil { return x.Name } return "" } -func (x *Agent) GetSupportedTaskTypes() []string { +func (x *TaskCategory) GetVersion() int32 { if x != nil { - return x.SupportedTaskTypes + return x.Version } - return nil + return 0 } // A request to get an agent. @@ -708,7 +1109,7 @@ type GetAgentRequest struct { func (x *GetAgentRequest) Reset() { *x = GetAgentRequest{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[9] + mi := &file_flyteidl_admin_agent_proto_msgTypes[14] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -721,7 +1122,7 @@ func (x *GetAgentRequest) String() string { func (*GetAgentRequest) ProtoMessage() {} func (x *GetAgentRequest) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[9] + mi := &file_flyteidl_admin_agent_proto_msgTypes[14] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -734,7 +1135,7 @@ func (x *GetAgentRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetAgentRequest.ProtoReflect.Descriptor instead. func (*GetAgentRequest) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{9} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{14} } func (x *GetAgentRequest) GetName() string { @@ -756,7 +1157,7 @@ type GetAgentResponse struct { func (x *GetAgentResponse) Reset() { *x = GetAgentResponse{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[10] + mi := &file_flyteidl_admin_agent_proto_msgTypes[15] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -769,7 +1170,7 @@ func (x *GetAgentResponse) String() string { func (*GetAgentResponse) ProtoMessage() {} func (x *GetAgentResponse) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[10] + mi := &file_flyteidl_admin_agent_proto_msgTypes[15] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -782,7 +1183,7 @@ func (x *GetAgentResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetAgentResponse.ProtoReflect.Descriptor instead. func (*GetAgentResponse) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{10} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{15} } func (x *GetAgentResponse) GetAgent() *Agent { @@ -802,7 +1203,7 @@ type ListAgentsRequest struct { func (x *ListAgentsRequest) Reset() { *x = ListAgentsRequest{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[11] + mi := &file_flyteidl_admin_agent_proto_msgTypes[16] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -815,7 +1216,7 @@ func (x *ListAgentsRequest) String() string { func (*ListAgentsRequest) ProtoMessage() {} func (x *ListAgentsRequest) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[11] + mi := &file_flyteidl_admin_agent_proto_msgTypes[16] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -828,7 +1229,7 @@ func (x *ListAgentsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ListAgentsRequest.ProtoReflect.Descriptor instead. func (*ListAgentsRequest) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{11} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{16} } // A response containing a list of agents. @@ -843,7 +1244,7 @@ type ListAgentsResponse struct { func (x *ListAgentsResponse) Reset() { *x = ListAgentsResponse{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[12] + mi := &file_flyteidl_admin_agent_proto_msgTypes[17] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -856,7 +1257,7 @@ func (x *ListAgentsResponse) String() string { func (*ListAgentsResponse) ProtoMessage() {} func (x *ListAgentsResponse) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[12] + mi := &file_flyteidl_admin_agent_proto_msgTypes[17] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -869,7 +1270,7 @@ func (x *ListAgentsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ListAgentsResponse.ProtoReflect.Descriptor instead. func (*ListAgentsResponse) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{12} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{17} } func (x *ListAgentsResponse) GetAgents() []*Agent { @@ -886,6 +1287,8 @@ type GetTaskMetricsRequest struct { unknownFields protoimpl.UnknownFields // A predefined yet extensible Task type identifier. + // + // Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. TaskType string `protobuf:"bytes,1,opt,name=task_type,json=taskType,proto3" json:"task_type,omitempty"` // Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). ResourceMeta []byte `protobuf:"bytes,2,opt,name=resource_meta,json=resourceMeta,proto3" json:"resource_meta,omitempty"` @@ -898,12 +1301,14 @@ type GetTaskMetricsRequest struct { EndTime *timestamppb.Timestamp `protobuf:"bytes,5,opt,name=end_time,json=endTime,proto3" json:"end_time,omitempty"` // Query resolution step width in duration format or float number of seconds. Step *durationpb.Duration `protobuf:"bytes,6,opt,name=step,proto3" json:"step,omitempty"` + // A predefined yet extensible Task type identifier. + TaskCategory *TaskCategory `protobuf:"bytes,7,opt,name=task_category,json=taskCategory,proto3" json:"task_category,omitempty"` } func (x *GetTaskMetricsRequest) Reset() { *x = GetTaskMetricsRequest{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[13] + mi := &file_flyteidl_admin_agent_proto_msgTypes[18] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -916,7 +1321,7 @@ func (x *GetTaskMetricsRequest) String() string { func (*GetTaskMetricsRequest) ProtoMessage() {} func (x *GetTaskMetricsRequest) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[13] + mi := &file_flyteidl_admin_agent_proto_msgTypes[18] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -929,9 +1334,10 @@ func (x *GetTaskMetricsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetTaskMetricsRequest.ProtoReflect.Descriptor instead. func (*GetTaskMetricsRequest) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{13} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{18} } +// Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. func (x *GetTaskMetricsRequest) GetTaskType() string { if x != nil { return x.TaskType @@ -974,6 +1380,13 @@ func (x *GetTaskMetricsRequest) GetStep() *durationpb.Duration { return nil } +func (x *GetTaskMetricsRequest) GetTaskCategory() *TaskCategory { + if x != nil { + return x.TaskCategory + } + return nil +} + // A response containing a list of metrics for a task execution. type GetTaskMetricsResponse struct { state protoimpl.MessageState @@ -987,7 +1400,7 @@ type GetTaskMetricsResponse struct { func (x *GetTaskMetricsResponse) Reset() { *x = GetTaskMetricsResponse{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[14] + mi := &file_flyteidl_admin_agent_proto_msgTypes[19] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1000,7 +1413,7 @@ func (x *GetTaskMetricsResponse) String() string { func (*GetTaskMetricsResponse) ProtoMessage() {} func (x *GetTaskMetricsResponse) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[14] + mi := &file_flyteidl_admin_agent_proto_msgTypes[19] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1013,7 +1426,7 @@ func (x *GetTaskMetricsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetTaskMetricsResponse.ProtoReflect.Descriptor instead. func (*GetTaskMetricsResponse) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{14} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{19} } func (x *GetTaskMetricsResponse) GetResults() []*core.ExecutionMetricResult { @@ -1030,6 +1443,8 @@ type GetTaskLogsRequest struct { unknownFields protoimpl.UnknownFields // A predefined yet extensible Task type identifier. + // + // Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. TaskType string `protobuf:"bytes,1,opt,name=task_type,json=taskType,proto3" json:"task_type,omitempty"` // Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). ResourceMeta []byte `protobuf:"bytes,2,opt,name=resource_meta,json=resourceMeta,proto3" json:"resource_meta,omitempty"` @@ -1038,12 +1453,14 @@ type GetTaskLogsRequest struct { // In the case of multiple pages of results, the server-provided token can be used to fetch the next page // in a query. If there are no more results, this value will be empty. Token string `protobuf:"bytes,4,opt,name=token,proto3" json:"token,omitempty"` + // A predefined yet extensible Task type identifier. + TaskCategory *TaskCategory `protobuf:"bytes,5,opt,name=task_category,json=taskCategory,proto3" json:"task_category,omitempty"` } func (x *GetTaskLogsRequest) Reset() { *x = GetTaskLogsRequest{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[15] + mi := &file_flyteidl_admin_agent_proto_msgTypes[20] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1056,7 +1473,7 @@ func (x *GetTaskLogsRequest) String() string { func (*GetTaskLogsRequest) ProtoMessage() {} func (x *GetTaskLogsRequest) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[15] + mi := &file_flyteidl_admin_agent_proto_msgTypes[20] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1069,9 +1486,10 @@ func (x *GetTaskLogsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetTaskLogsRequest.ProtoReflect.Descriptor instead. func (*GetTaskLogsRequest) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{15} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{20} } +// Deprecated: Marked as deprecated in flyteidl/admin/agent.proto. func (x *GetTaskLogsRequest) GetTaskType() string { if x != nil { return x.TaskType @@ -1100,23 +1518,127 @@ func (x *GetTaskLogsRequest) GetToken() string { return "" } -// A response containing the logs for a task execution. -type GetTaskLogsResponse struct { +func (x *GetTaskLogsRequest) GetTaskCategory() *TaskCategory { + if x != nil { + return x.TaskCategory + } + return nil +} + +type GetTaskLogsResponseHeader struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - // The execution log results. - Results []string `protobuf:"bytes,1,rep,name=results,proto3" json:"results,omitempty"` // In the case of multiple pages of results, the server-provided token can be used to fetch the next page // in a query. If there are no more results, this value will be empty. - Token string `protobuf:"bytes,2,opt,name=token,proto3" json:"token,omitempty"` + Token string `protobuf:"bytes,1,opt,name=token,proto3" json:"token,omitempty"` +} + +func (x *GetTaskLogsResponseHeader) Reset() { + *x = GetTaskLogsResponseHeader{} + if protoimpl.UnsafeEnabled { + mi := &file_flyteidl_admin_agent_proto_msgTypes[21] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetTaskLogsResponseHeader) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetTaskLogsResponseHeader) ProtoMessage() {} + +func (x *GetTaskLogsResponseHeader) ProtoReflect() protoreflect.Message { + mi := &file_flyteidl_admin_agent_proto_msgTypes[21] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetTaskLogsResponseHeader.ProtoReflect.Descriptor instead. +func (*GetTaskLogsResponseHeader) Descriptor() ([]byte, []int) { + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{21} +} + +func (x *GetTaskLogsResponseHeader) GetToken() string { + if x != nil { + return x.Token + } + return "" +} + +type GetTaskLogsResponseBody struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The execution log results. + Results []string `protobuf:"bytes,1,rep,name=results,proto3" json:"results,omitempty"` +} + +func (x *GetTaskLogsResponseBody) Reset() { + *x = GetTaskLogsResponseBody{} + if protoimpl.UnsafeEnabled { + mi := &file_flyteidl_admin_agent_proto_msgTypes[22] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetTaskLogsResponseBody) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetTaskLogsResponseBody) ProtoMessage() {} + +func (x *GetTaskLogsResponseBody) ProtoReflect() protoreflect.Message { + mi := &file_flyteidl_admin_agent_proto_msgTypes[22] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetTaskLogsResponseBody.ProtoReflect.Descriptor instead. +func (*GetTaskLogsResponseBody) Descriptor() ([]byte, []int) { + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{22} +} + +func (x *GetTaskLogsResponseBody) GetResults() []string { + if x != nil { + return x.Results + } + return nil +} + +// A response containing the logs for a task execution. +type GetTaskLogsResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Types that are assignable to Part: + // + // *GetTaskLogsResponse_Header + // *GetTaskLogsResponse_Body + Part isGetTaskLogsResponse_Part `protobuf_oneof:"part"` } func (x *GetTaskLogsResponse) Reset() { *x = GetTaskLogsResponse{} if protoimpl.UnsafeEnabled { - mi := &file_flyteidl_admin_agent_proto_msgTypes[16] + mi := &file_flyteidl_admin_agent_proto_msgTypes[23] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1129,7 +1651,7 @@ func (x *GetTaskLogsResponse) String() string { func (*GetTaskLogsResponse) ProtoMessage() {} func (x *GetTaskLogsResponse) ProtoReflect() protoreflect.Message { - mi := &file_flyteidl_admin_agent_proto_msgTypes[16] + mi := &file_flyteidl_admin_agent_proto_msgTypes[23] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1142,23 +1664,46 @@ func (x *GetTaskLogsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetTaskLogsResponse.ProtoReflect.Descriptor instead. func (*GetTaskLogsResponse) Descriptor() ([]byte, []int) { - return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{16} + return file_flyteidl_admin_agent_proto_rawDescGZIP(), []int{23} } -func (x *GetTaskLogsResponse) GetResults() []string { - if x != nil { - return x.Results +func (m *GetTaskLogsResponse) GetPart() isGetTaskLogsResponse_Part { + if m != nil { + return m.Part } return nil } -func (x *GetTaskLogsResponse) GetToken() string { - if x != nil { - return x.Token +func (x *GetTaskLogsResponse) GetHeader() *GetTaskLogsResponseHeader { + if x, ok := x.GetPart().(*GetTaskLogsResponse_Header); ok { + return x.Header } - return "" + return nil +} + +func (x *GetTaskLogsResponse) GetBody() *GetTaskLogsResponseBody { + if x, ok := x.GetPart().(*GetTaskLogsResponse_Body); ok { + return x.Body + } + return nil } +type isGetTaskLogsResponse_Part interface { + isGetTaskLogsResponse_Part() +} + +type GetTaskLogsResponse_Header struct { + Header *GetTaskLogsResponseHeader `protobuf:"bytes,1,opt,name=header,proto3,oneof"` +} + +type GetTaskLogsResponse_Body struct { + Body *GetTaskLogsResponseBody `protobuf:"bytes,2,opt,name=body,proto3,oneof"` +} + +func (*GetTaskLogsResponse_Header) isGetTaskLogsResponse_Part() {} + +func (*GetTaskLogsResponse_Body) isGetTaskLogsResponse_Part() {} + var File_flyteidl_admin_agent_proto protoreflect.FileDescriptor var file_flyteidl_admin_agent_proto_rawDesc = []byte{ @@ -1168,189 +1713,286 @@ var file_flyteidl_admin_agent_proto_rawDesc = []byte{ 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x6c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x19, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x74, 0x61, 0x73, 0x6b, 0x73, 0x2e, - 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, - 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x69, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x2e, - 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1d, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, - 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1b, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, - 0x6f, 0x72, 0x65, 0x2f, 0x6d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, - 0x6f, 0x1a, 0x1e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, - 0x75, 0x66, 0x2f, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x70, 0x72, 0x6f, 0x74, - 0x6f, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, - 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e, 0x70, 0x72, 0x6f, - 0x74, 0x6f, 0x22, 0x98, 0x05, 0x0a, 0x15, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, - 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x52, 0x0a, 0x11, - 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, - 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x26, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, - 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, - 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, - 0x0f, 0x74, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, - 0x12, 0x1c, 0x0a, 0x09, 0x6e, 0x61, 0x6d, 0x65, 0x73, 0x70, 0x61, 0x63, 0x65, 0x18, 0x02, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x09, 0x6e, 0x61, 0x6d, 0x65, 0x73, 0x70, 0x61, 0x63, 0x65, 0x12, 0x49, - 0x0a, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x31, + 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1c, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, + 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x77, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x1a, 0x1e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, + 0x72, 0x65, 0x2f, 0x69, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x2e, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x1a, 0x1d, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, + 0x72, 0x65, 0x2f, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x1a, 0x1b, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, 0x72, + 0x65, 0x2f, 0x6d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, + 0x1e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, + 0x2f, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, + 0x1f, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, + 0x2f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, + 0x1a, 0x1c, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, + 0x66, 0x2f, 0x73, 0x74, 0x72, 0x75, 0x63, 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0xe9, + 0x06, 0x0a, 0x15, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, + 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x52, 0x0a, 0x11, 0x74, 0x61, 0x73, 0x6b, + 0x5f, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x26, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, + 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, 0x0f, 0x74, 0x61, 0x73, + 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x1c, 0x0a, 0x09, + 0x6e, 0x61, 0x6d, 0x65, 0x73, 0x70, 0x61, 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x09, 0x6e, 0x61, 0x6d, 0x65, 0x73, 0x70, 0x61, 0x63, 0x65, 0x12, 0x49, 0x0a, 0x06, 0x6c, 0x61, + 0x62, 0x65, 0x6c, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x31, 0x2e, 0x66, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, + 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x6c, + 0x61, 0x62, 0x65, 0x6c, 0x73, 0x12, 0x58, 0x0a, 0x0b, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, + 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x36, 0x2e, 0x66, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, + 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2e, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x45, 0x6e, 0x74, + 0x72, 0x79, 0x52, 0x0b, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, + 0x2e, 0x0a, 0x13, 0x6b, 0x38, 0x73, 0x5f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x5f, 0x61, + 0x63, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x11, 0x6b, 0x38, + 0x73, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x41, 0x63, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x12, + 0x74, 0x0a, 0x15, 0x65, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x5f, 0x76, + 0x61, 0x72, 0x69, 0x61, 0x62, 0x6c, 0x65, 0x73, 0x18, 0x06, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, - 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, - 0x79, 0x52, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x12, 0x58, 0x0a, 0x0b, 0x61, 0x6e, 0x6e, - 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x36, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, - 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, - 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, - 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x0b, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, - 0x6f, 0x6e, 0x73, 0x12, 0x2e, 0x0a, 0x13, 0x6b, 0x38, 0x73, 0x5f, 0x73, 0x65, 0x72, 0x76, 0x69, - 0x63, 0x65, 0x5f, 0x61, 0x63, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, - 0x52, 0x11, 0x6b, 0x38, 0x73, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x41, 0x63, 0x63, 0x6f, - 0x75, 0x6e, 0x74, 0x12, 0x74, 0x0a, 0x15, 0x65, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, - 0x6e, 0x74, 0x5f, 0x76, 0x61, 0x72, 0x69, 0x61, 0x62, 0x6c, 0x65, 0x73, 0x18, 0x06, 0x20, 0x03, - 0x28, 0x0b, 0x32, 0x3f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, - 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, - 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x45, 0x6e, 0x76, 0x69, 0x72, 0x6f, - 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x56, 0x61, 0x72, 0x69, 0x61, 0x62, 0x6c, 0x65, 0x73, 0x45, 0x6e, - 0x74, 0x72, 0x79, 0x52, 0x14, 0x65, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, - 0x56, 0x61, 0x72, 0x69, 0x61, 0x62, 0x6c, 0x65, 0x73, 0x1a, 0x39, 0x0a, 0x0b, 0x4c, 0x61, 0x62, - 0x65, 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, - 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, - 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x3e, 0x0a, 0x10, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, - 0x6f, 0x6e, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, - 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, - 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x47, 0x0a, 0x19, 0x45, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, - 0x65, 0x6e, 0x74, 0x56, 0x61, 0x72, 0x69, 0x61, 0x62, 0x6c, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, - 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, - 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, - 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x83, 0x02, - 0x0a, 0x11, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, - 0x65, 0x73, 0x74, 0x12, 0x31, 0x0a, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, - 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x06, - 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x12, 0x37, 0x0a, 0x08, 0x74, 0x65, 0x6d, 0x70, 0x6c, 0x61, - 0x74, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1b, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, - 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x54, 0x65, 0x6d, - 0x70, 0x6c, 0x61, 0x74, 0x65, 0x52, 0x08, 0x74, 0x65, 0x6d, 0x70, 0x6c, 0x61, 0x74, 0x65, 0x12, - 0x23, 0x0a, 0x0d, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, - 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x50, 0x72, - 0x65, 0x66, 0x69, 0x78, 0x12, 0x5d, 0x0a, 0x17, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x65, 0x78, 0x65, - 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, - 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x25, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, - 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, - 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x15, 0x74, 0x61, - 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, - 0x61, 0x74, 0x61, 0x22, 0x7a, 0x0a, 0x12, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, - 0x6b, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x25, 0x0a, 0x0d, 0x72, 0x65, 0x73, - 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, - 0x48, 0x00, 0x52, 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4d, 0x65, 0x74, 0x61, - 0x12, 0x36, 0x0a, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, - 0x6d, 0x69, 0x6e, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x48, 0x00, 0x52, 0x08, - 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x42, 0x05, 0x0a, 0x03, 0x72, 0x65, 0x73, 0x22, - 0x52, 0x0a, 0x0e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x23, - 0x0a, 0x0d, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, - 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4d, - 0x65, 0x74, 0x61, 0x22, 0x7c, 0x0a, 0x0f, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, - 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x34, 0x0a, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, - 0x63, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, - 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, - 0x63, 0x65, 0x52, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x33, 0x0a, 0x09, - 0x6c, 0x6f, 0x67, 0x5f, 0x6c, 0x69, 0x6e, 0x6b, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, - 0x16, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, - 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x52, 0x08, 0x6c, 0x6f, 0x67, 0x4c, 0x69, 0x6e, 0x6b, - 0x73, 0x22, 0xf9, 0x01, 0x0a, 0x08, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x2f, - 0x0a, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x15, 0x2e, - 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x53, - 0x74, 0x61, 0x74, 0x65, 0x42, 0x02, 0x18, 0x01, 0x52, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x12, - 0x33, 0x0a, 0x07, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x45, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, + 0x74, 0x56, 0x61, 0x72, 0x69, 0x61, 0x62, 0x6c, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, + 0x14, 0x65, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x56, 0x61, 0x72, 0x69, + 0x61, 0x62, 0x6c, 0x65, 0x73, 0x12, 0x21, 0x0a, 0x0c, 0x6d, 0x61, 0x78, 0x5f, 0x61, 0x74, 0x74, + 0x65, 0x6d, 0x70, 0x74, 0x73, 0x18, 0x07, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0b, 0x6d, 0x61, 0x78, + 0x41, 0x74, 0x74, 0x65, 0x6d, 0x70, 0x74, 0x73, 0x12, 0x24, 0x0a, 0x0d, 0x69, 0x6e, 0x74, 0x65, + 0x72, 0x72, 0x75, 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, 0x18, 0x08, 0x20, 0x01, 0x28, 0x08, 0x52, + 0x0d, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x72, 0x75, 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, 0x12, 0x46, + 0x0a, 0x1f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x72, 0x75, 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, 0x5f, + 0x66, 0x61, 0x69, 0x6c, 0x75, 0x72, 0x65, 0x5f, 0x74, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, + 0x64, 0x18, 0x09, 0x20, 0x01, 0x28, 0x05, 0x52, 0x1d, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x72, 0x75, + 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, 0x46, 0x61, 0x69, 0x6c, 0x75, 0x72, 0x65, 0x54, 0x68, 0x72, + 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64, 0x12, 0x3e, 0x0a, 0x09, 0x6f, 0x76, 0x65, 0x72, 0x72, 0x69, + 0x64, 0x65, 0x73, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x20, 0x2e, 0x66, 0x6c, 0x79, 0x74, + 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x4e, 0x6f, + 0x64, 0x65, 0x4f, 0x76, 0x65, 0x72, 0x72, 0x69, 0x64, 0x65, 0x73, 0x52, 0x09, 0x6f, 0x76, 0x65, + 0x72, 0x72, 0x69, 0x64, 0x65, 0x73, 0x1a, 0x39, 0x0a, 0x0b, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, + 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, + 0x01, 0x1a, 0x3e, 0x0a, 0x10, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, + 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, + 0x01, 0x1a, 0x47, 0x0a, 0x19, 0x45, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, + 0x56, 0x61, 0x72, 0x69, 0x61, 0x62, 0x6c, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, + 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, + 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x83, 0x02, 0x0a, 0x11, 0x43, + 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, + 0x12, 0x31, 0x0a, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, - 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x07, 0x6f, 0x75, 0x74, - 0x70, 0x75, 0x74, 0x73, 0x12, 0x18, 0x0a, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, - 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x12, 0x33, - 0x0a, 0x09, 0x6c, 0x6f, 0x67, 0x5f, 0x6c, 0x69, 0x6e, 0x6b, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, - 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, - 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x52, 0x08, 0x6c, 0x6f, 0x67, 0x4c, 0x69, - 0x6e, 0x6b, 0x73, 0x12, 0x38, 0x0a, 0x05, 0x70, 0x68, 0x61, 0x73, 0x65, 0x18, 0x05, 0x20, 0x01, - 0x28, 0x0e, 0x32, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, - 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, - 0x2e, 0x50, 0x68, 0x61, 0x73, 0x65, 0x52, 0x05, 0x70, 0x68, 0x61, 0x73, 0x65, 0x22, 0x55, 0x0a, - 0x11, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, - 0x23, 0x0a, 0x0d, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, - 0x4d, 0x65, 0x74, 0x61, 0x22, 0x14, 0x0a, 0x12, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, - 0x73, 0x6b, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x4d, 0x0a, 0x05, 0x41, 0x67, - 0x65, 0x6e, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, - 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x30, 0x0a, 0x14, 0x73, 0x75, 0x70, 0x70, 0x6f, - 0x72, 0x74, 0x65, 0x64, 0x5f, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x18, - 0x02, 0x20, 0x03, 0x28, 0x09, 0x52, 0x12, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x64, - 0x54, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x73, 0x22, 0x25, 0x0a, 0x0f, 0x47, 0x65, 0x74, - 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x12, 0x0a, 0x04, - 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, - 0x22, 0x3f, 0x0a, 0x10, 0x47, 0x65, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x65, 0x73, 0x70, - 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x2b, 0x0a, 0x05, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, - 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x05, 0x61, 0x67, 0x65, 0x6e, - 0x74, 0x22, 0x13, 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x52, - 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x43, 0x0a, 0x12, 0x4c, 0x69, 0x73, 0x74, 0x41, 0x67, - 0x65, 0x6e, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x2d, 0x0a, 0x06, - 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x66, - 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x41, 0x67, - 0x65, 0x6e, 0x74, 0x52, 0x06, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x22, 0x94, 0x02, 0x0a, 0x15, - 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, - 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, - 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, - 0x70, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, - 0x65, 0x74, 0x61, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, - 0x72, 0x63, 0x65, 0x4d, 0x65, 0x74, 0x61, 0x12, 0x18, 0x0a, 0x07, 0x71, 0x75, 0x65, 0x72, 0x69, - 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x09, 0x52, 0x07, 0x71, 0x75, 0x65, 0x72, 0x69, 0x65, - 0x73, 0x12, 0x39, 0x0a, 0x0a, 0x73, 0x74, 0x61, 0x72, 0x74, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, - 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, - 0x70, 0x52, 0x09, 0x73, 0x74, 0x61, 0x72, 0x74, 0x54, 0x69, 0x6d, 0x65, 0x12, 0x35, 0x0a, 0x08, - 0x65, 0x6e, 0x64, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, - 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, - 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x07, 0x65, 0x6e, 0x64, 0x54, - 0x69, 0x6d, 0x65, 0x12, 0x2d, 0x0a, 0x04, 0x73, 0x74, 0x65, 0x70, 0x18, 0x06, 0x20, 0x01, 0x28, - 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, - 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x04, 0x73, 0x74, - 0x65, 0x70, 0x22, 0x58, 0x0a, 0x16, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, - 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x3e, 0x0a, 0x07, - 0x72, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x24, 0x2e, - 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x45, 0x78, - 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x52, 0x65, 0x73, - 0x75, 0x6c, 0x74, 0x52, 0x07, 0x72, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x73, 0x22, 0x82, 0x01, 0x0a, - 0x12, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, 0x65, 0x71, 0x75, - 0x65, 0x73, 0x74, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, - 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, - 0x12, 0x23, 0x0a, 0x0d, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, - 0x61, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, - 0x65, 0x4d, 0x65, 0x74, 0x61, 0x12, 0x14, 0x0a, 0x05, 0x6c, 0x69, 0x6e, 0x65, 0x73, 0x18, 0x03, - 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x6c, 0x69, 0x6e, 0x65, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x74, - 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x74, 0x6f, 0x6b, 0x65, - 0x6e, 0x22, 0x45, 0x0a, 0x13, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, - 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x72, 0x65, 0x73, 0x75, - 0x6c, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x07, 0x72, 0x65, 0x73, 0x75, 0x6c, - 0x74, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, - 0x09, 0x52, 0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x2a, 0x5e, 0x0a, 0x05, 0x53, 0x74, 0x61, 0x74, - 0x65, 0x12, 0x15, 0x0a, 0x11, 0x52, 0x45, 0x54, 0x52, 0x59, 0x41, 0x42, 0x4c, 0x45, 0x5f, 0x46, - 0x41, 0x49, 0x4c, 0x55, 0x52, 0x45, 0x10, 0x00, 0x12, 0x15, 0x0a, 0x11, 0x50, 0x45, 0x52, 0x4d, - 0x41, 0x4e, 0x45, 0x4e, 0x54, 0x5f, 0x46, 0x41, 0x49, 0x4c, 0x55, 0x52, 0x45, 0x10, 0x01, 0x12, - 0x0b, 0x0a, 0x07, 0x50, 0x45, 0x4e, 0x44, 0x49, 0x4e, 0x47, 0x10, 0x02, 0x12, 0x0b, 0x0a, 0x07, - 0x52, 0x55, 0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, 0x03, 0x12, 0x0d, 0x0a, 0x09, 0x53, 0x55, 0x43, - 0x43, 0x45, 0x45, 0x44, 0x45, 0x44, 0x10, 0x04, 0x42, 0xb6, 0x01, 0x0a, 0x12, 0x63, 0x6f, 0x6d, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x42, - 0x0a, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3b, 0x67, - 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x6f, - 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, 0x62, 0x2d, 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, 0x74, - 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0xa2, 0x02, 0x03, 0x46, 0x41, 0x58, - 0xaa, 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x41, 0x64, 0x6d, 0x69, - 0x6e, 0xca, 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, 0x6d, - 0x69, 0x6e, 0xe2, 0x02, 0x1a, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, - 0x6d, 0x69, 0x6e, 0x5c, 0x47, 0x50, 0x42, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, - 0x02, 0x0f, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x41, 0x64, 0x6d, 0x69, - 0x6e, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x06, 0x69, 0x6e, 0x70, + 0x75, 0x74, 0x73, 0x12, 0x37, 0x0a, 0x08, 0x74, 0x65, 0x6d, 0x70, 0x6c, 0x61, 0x74, 0x65, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1b, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x54, 0x65, 0x6d, 0x70, 0x6c, 0x61, + 0x74, 0x65, 0x52, 0x08, 0x74, 0x65, 0x6d, 0x70, 0x6c, 0x61, 0x74, 0x65, 0x12, 0x23, 0x0a, 0x0d, + 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x18, 0x03, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x0c, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x50, 0x72, 0x65, 0x66, 0x69, + 0x78, 0x12, 0x5d, 0x0a, 0x17, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, + 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x04, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x25, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, + 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, + 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x15, 0x74, 0x61, 0x73, 0x6b, 0x45, + 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x22, 0x39, 0x0a, 0x12, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, + 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72, + 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4d, 0x65, 0x74, 0x61, 0x22, 0x87, 0x02, 0x0a, 0x13, + 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x48, 0x65, 0x61, + 0x64, 0x65, 0x72, 0x12, 0x37, 0x0a, 0x08, 0x74, 0x65, 0x6d, 0x70, 0x6c, 0x61, 0x74, 0x65, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1b, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x54, 0x65, 0x6d, 0x70, 0x6c, 0x61, + 0x74, 0x65, 0x52, 0x08, 0x74, 0x65, 0x6d, 0x70, 0x6c, 0x61, 0x74, 0x65, 0x12, 0x23, 0x0a, 0x0d, + 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x0c, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x50, 0x72, 0x65, 0x66, 0x69, + 0x78, 0x12, 0x5d, 0x0a, 0x17, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, + 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x03, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x25, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, + 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, + 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x15, 0x74, 0x61, 0x73, 0x6b, 0x45, + 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x12, 0x33, 0x0a, 0x16, 0x6d, 0x61, 0x78, 0x5f, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, + 0x73, 0x69, 0x7a, 0x65, 0x5f, 0x62, 0x79, 0x74, 0x65, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, + 0x52, 0x13, 0x6d, 0x61, 0x78, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x69, 0x7a, 0x65, + 0x42, 0x79, 0x74, 0x65, 0x73, 0x22, 0x94, 0x01, 0x0a, 0x16, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, + 0x65, 0x54, 0x61, 0x73, 0x6b, 0x53, 0x79, 0x6e, 0x63, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, + 0x12, 0x3d, 0x0a, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x23, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, + 0x6e, 0x2e, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x48, + 0x65, 0x61, 0x64, 0x65, 0x72, 0x48, 0x00, 0x52, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, + 0x33, 0x0a, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x48, 0x00, 0x52, 0x06, 0x69, 0x6e, + 0x70, 0x75, 0x74, 0x73, 0x42, 0x06, 0x0a, 0x04, 0x70, 0x61, 0x72, 0x74, 0x22, 0x55, 0x0a, 0x1d, + 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x53, 0x79, 0x6e, 0x63, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x34, 0x0a, + 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x18, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, + 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x52, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, + 0x72, 0x63, 0x65, 0x22, 0xa0, 0x01, 0x0a, 0x17, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x65, 0x54, + 0x61, 0x73, 0x6b, 0x53, 0x79, 0x6e, 0x63, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, + 0x47, 0x0a, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x2d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, + 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x53, 0x79, 0x6e, 0x63, + 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x48, 0x00, + 0x52, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x35, 0x0a, 0x07, 0x6f, 0x75, 0x74, 0x70, + 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, + 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, + 0x6c, 0x4d, 0x61, 0x70, 0x48, 0x00, 0x52, 0x07, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x42, + 0x05, 0x0a, 0x03, 0x72, 0x65, 0x73, 0x22, 0x99, 0x01, 0x0a, 0x0e, 0x47, 0x65, 0x74, 0x54, 0x61, + 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1f, 0x0a, 0x09, 0x74, 0x61, 0x73, + 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x42, 0x02, 0x18, 0x01, + 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x72, 0x65, + 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x02, 0x20, 0x01, 0x28, + 0x0c, 0x52, 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4d, 0x65, 0x74, 0x61, 0x12, + 0x41, 0x0a, 0x0d, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, + 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, + 0x67, 0x6f, 0x72, 0x79, 0x52, 0x0c, 0x74, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, + 0x72, 0x79, 0x22, 0x47, 0x0a, 0x0f, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x34, 0x0a, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, + 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, + 0x65, 0x52, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x22, 0xb3, 0x02, 0x0a, 0x08, + 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x2f, 0x0a, 0x05, 0x73, 0x74, 0x61, 0x74, + 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x15, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x65, 0x42, 0x02, + 0x18, 0x01, 0x52, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x12, 0x33, 0x0a, 0x07, 0x6f, 0x75, 0x74, + 0x70, 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, + 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x07, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x12, 0x18, + 0x0a, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x12, 0x33, 0x0a, 0x09, 0x6c, 0x6f, 0x67, 0x5f, + 0x6c, 0x69, 0x6e, 0x6b, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, 0x6b, + 0x4c, 0x6f, 0x67, 0x52, 0x08, 0x6c, 0x6f, 0x67, 0x4c, 0x69, 0x6e, 0x6b, 0x73, 0x12, 0x38, 0x0a, + 0x05, 0x70, 0x68, 0x61, 0x73, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x22, 0x2e, 0x66, + 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x61, 0x73, + 0x6b, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x50, 0x68, 0x61, 0x73, 0x65, + 0x52, 0x05, 0x70, 0x68, 0x61, 0x73, 0x65, 0x12, 0x38, 0x0a, 0x0b, 0x63, 0x75, 0x73, 0x74, 0x6f, + 0x6d, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x67, + 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x53, + 0x74, 0x72, 0x75, 0x63, 0x74, 0x52, 0x0a, 0x63, 0x75, 0x73, 0x74, 0x6f, 0x6d, 0x49, 0x6e, 0x66, + 0x6f, 0x22, 0x9c, 0x01, 0x0a, 0x11, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1f, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, + 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x42, 0x02, 0x18, 0x01, 0x52, 0x08, + 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x72, 0x65, 0x73, 0x6f, + 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, + 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4d, 0x65, 0x74, 0x61, 0x12, 0x41, 0x0a, + 0x0d, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x18, 0x03, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, + 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, + 0x72, 0x79, 0x52, 0x0c, 0x74, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, + 0x22, 0x14, 0x0a, 0x12, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0xc4, 0x01, 0x0a, 0x05, 0x41, 0x67, 0x65, 0x6e, 0x74, + 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, + 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x34, 0x0a, 0x14, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x65, + 0x64, 0x5f, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x18, 0x02, 0x20, 0x03, + 0x28, 0x09, 0x42, 0x02, 0x18, 0x01, 0x52, 0x12, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x65, + 0x64, 0x54, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x73, 0x12, 0x17, 0x0a, 0x07, 0x69, 0x73, + 0x5f, 0x73, 0x79, 0x6e, 0x63, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x52, 0x06, 0x69, 0x73, 0x53, + 0x79, 0x6e, 0x63, 0x12, 0x58, 0x0a, 0x19, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x64, + 0x5f, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x69, 0x65, 0x73, + 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, + 0x67, 0x6f, 0x72, 0x79, 0x52, 0x17, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x65, 0x64, 0x54, + 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x69, 0x65, 0x73, 0x22, 0x3c, 0x0a, + 0x0c, 0x54, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x12, 0x12, 0x0a, + 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, + 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x05, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x25, 0x0a, 0x0f, 0x47, + 0x65, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x12, + 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, + 0x6d, 0x65, 0x22, 0x3f, 0x0a, 0x10, 0x47, 0x65, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x65, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x2b, 0x0a, 0x05, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, + 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x05, 0x61, 0x67, + 0x65, 0x6e, 0x74, 0x22, 0x13, 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, + 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x43, 0x0a, 0x12, 0x4c, 0x69, 0x73, 0x74, + 0x41, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x2d, + 0x0a, 0x06, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x15, + 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, + 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x06, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x22, 0xdb, 0x02, + 0x0a, 0x15, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1f, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, + 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x42, 0x02, 0x18, 0x01, 0x52, 0x08, + 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x72, 0x65, 0x73, 0x6f, + 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, + 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4d, 0x65, 0x74, 0x61, 0x12, 0x18, 0x0a, + 0x07, 0x71, 0x75, 0x65, 0x72, 0x69, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x09, 0x52, 0x07, + 0x71, 0x75, 0x65, 0x72, 0x69, 0x65, 0x73, 0x12, 0x39, 0x0a, 0x0a, 0x73, 0x74, 0x61, 0x72, 0x74, + 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, + 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, + 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x09, 0x73, 0x74, 0x61, 0x72, 0x74, 0x54, 0x69, + 0x6d, 0x65, 0x12, 0x35, 0x0a, 0x08, 0x65, 0x6e, 0x64, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x05, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, + 0x52, 0x07, 0x65, 0x6e, 0x64, 0x54, 0x69, 0x6d, 0x65, 0x12, 0x2d, 0x0a, 0x04, 0x73, 0x74, 0x65, + 0x70, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, + 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, + 0x6f, 0x6e, 0x52, 0x04, 0x73, 0x74, 0x65, 0x70, 0x12, 0x41, 0x0a, 0x0d, 0x74, 0x61, 0x73, 0x6b, + 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x18, 0x07, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, + 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x52, 0x0c, 0x74, + 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x22, 0x58, 0x0a, 0x16, 0x47, + 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x3e, 0x0a, 0x07, 0x72, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x73, + 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, + 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x52, 0x07, 0x72, 0x65, + 0x73, 0x75, 0x6c, 0x74, 0x73, 0x22, 0xc9, 0x01, 0x0a, 0x12, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, + 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1f, 0x0a, 0x09, + 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x42, + 0x02, 0x18, 0x01, 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x23, 0x0a, + 0x0d, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x02, + 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0c, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4d, 0x65, + 0x74, 0x61, 0x12, 0x14, 0x0a, 0x05, 0x6c, 0x69, 0x6e, 0x65, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, + 0x04, 0x52, 0x05, 0x6c, 0x69, 0x6e, 0x65, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x74, 0x6f, 0x6b, 0x65, + 0x6e, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x12, 0x41, + 0x0a, 0x0d, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x18, + 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, + 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x54, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, + 0x6f, 0x72, 0x79, 0x52, 0x0c, 0x74, 0x61, 0x73, 0x6b, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, + 0x79, 0x22, 0x31, 0x0a, 0x19, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, + 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x14, + 0x0a, 0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x74, + 0x6f, 0x6b, 0x65, 0x6e, 0x22, 0x33, 0x0a, 0x17, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, + 0x6f, 0x67, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x12, + 0x18, 0x0a, 0x07, 0x72, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, + 0x52, 0x07, 0x72, 0x65, 0x73, 0x75, 0x6c, 0x74, 0x73, 0x22, 0xa1, 0x01, 0x0a, 0x13, 0x47, 0x65, + 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, + 0x65, 0x12, 0x43, 0x0a, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x29, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, + 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, 0x65, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x48, 0x00, 0x52, 0x06, + 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x3d, 0x0a, 0x04, 0x62, 0x6f, 0x64, 0x79, 0x18, 0x02, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, + 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, + 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x48, 0x00, 0x52, + 0x04, 0x62, 0x6f, 0x64, 0x79, 0x42, 0x06, 0x0a, 0x04, 0x70, 0x61, 0x72, 0x74, 0x2a, 0x62, 0x0a, + 0x05, 0x53, 0x74, 0x61, 0x74, 0x65, 0x12, 0x15, 0x0a, 0x11, 0x52, 0x45, 0x54, 0x52, 0x59, 0x41, + 0x42, 0x4c, 0x45, 0x5f, 0x46, 0x41, 0x49, 0x4c, 0x55, 0x52, 0x45, 0x10, 0x00, 0x12, 0x15, 0x0a, + 0x11, 0x50, 0x45, 0x52, 0x4d, 0x41, 0x4e, 0x45, 0x4e, 0x54, 0x5f, 0x46, 0x41, 0x49, 0x4c, 0x55, + 0x52, 0x45, 0x10, 0x01, 0x12, 0x0b, 0x0a, 0x07, 0x50, 0x45, 0x4e, 0x44, 0x49, 0x4e, 0x47, 0x10, + 0x02, 0x12, 0x0b, 0x0a, 0x07, 0x52, 0x55, 0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, 0x03, 0x12, 0x0d, + 0x0a, 0x09, 0x53, 0x55, 0x43, 0x43, 0x45, 0x45, 0x44, 0x45, 0x44, 0x10, 0x04, 0x1a, 0x02, 0x18, + 0x01, 0x42, 0xb6, 0x01, 0x0a, 0x12, 0x63, 0x6f, 0x6d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x42, 0x0a, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x50, + 0x72, 0x6f, 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3b, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, + 0x6f, 0x6d, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, + 0x65, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, + 0x62, 0x2d, 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x61, 0x64, + 0x6d, 0x69, 0x6e, 0xa2, 0x02, 0x03, 0x46, 0x41, 0x58, 0xaa, 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, + 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0xca, 0x02, 0x0e, 0x46, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0xe2, 0x02, 0x1a, 0x46, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0x5c, 0x47, 0x50, 0x42, + 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, 0x02, 0x0f, 0x46, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, + 0x6f, 0x33, } var ( @@ -1366,64 +2008,87 @@ func file_flyteidl_admin_agent_proto_rawDescGZIP() []byte { } var file_flyteidl_admin_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1) -var file_flyteidl_admin_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 20) +var file_flyteidl_admin_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 27) var file_flyteidl_admin_agent_proto_goTypes = []interface{}{ - (State)(0), // 0: flyteidl.admin.State - (*TaskExecutionMetadata)(nil), // 1: flyteidl.admin.TaskExecutionMetadata - (*CreateTaskRequest)(nil), // 2: flyteidl.admin.CreateTaskRequest - (*CreateTaskResponse)(nil), // 3: flyteidl.admin.CreateTaskResponse - (*GetTaskRequest)(nil), // 4: flyteidl.admin.GetTaskRequest - (*GetTaskResponse)(nil), // 5: flyteidl.admin.GetTaskResponse - (*Resource)(nil), // 6: flyteidl.admin.Resource - (*DeleteTaskRequest)(nil), // 7: flyteidl.admin.DeleteTaskRequest - (*DeleteTaskResponse)(nil), // 8: flyteidl.admin.DeleteTaskResponse - (*Agent)(nil), // 9: flyteidl.admin.Agent - (*GetAgentRequest)(nil), // 10: flyteidl.admin.GetAgentRequest - (*GetAgentResponse)(nil), // 11: flyteidl.admin.GetAgentResponse - (*ListAgentsRequest)(nil), // 12: flyteidl.admin.ListAgentsRequest - (*ListAgentsResponse)(nil), // 13: flyteidl.admin.ListAgentsResponse - (*GetTaskMetricsRequest)(nil), // 14: flyteidl.admin.GetTaskMetricsRequest - (*GetTaskMetricsResponse)(nil), // 15: flyteidl.admin.GetTaskMetricsResponse - (*GetTaskLogsRequest)(nil), // 16: flyteidl.admin.GetTaskLogsRequest - (*GetTaskLogsResponse)(nil), // 17: flyteidl.admin.GetTaskLogsResponse - nil, // 18: flyteidl.admin.TaskExecutionMetadata.LabelsEntry - nil, // 19: flyteidl.admin.TaskExecutionMetadata.AnnotationsEntry - nil, // 20: flyteidl.admin.TaskExecutionMetadata.EnvironmentVariablesEntry - (*core.TaskExecutionIdentifier)(nil), // 21: flyteidl.core.TaskExecutionIdentifier - (*core.LiteralMap)(nil), // 22: flyteidl.core.LiteralMap - (*core.TaskTemplate)(nil), // 23: flyteidl.core.TaskTemplate - (*core.TaskLog)(nil), // 24: flyteidl.core.TaskLog - (core.TaskExecution_Phase)(0), // 25: flyteidl.core.TaskExecution.Phase - (*timestamppb.Timestamp)(nil), // 26: google.protobuf.Timestamp - (*durationpb.Duration)(nil), // 27: google.protobuf.Duration - (*core.ExecutionMetricResult)(nil), // 28: flyteidl.core.ExecutionMetricResult + (State)(0), // 0: flyteidl.admin.State + (*TaskExecutionMetadata)(nil), // 1: flyteidl.admin.TaskExecutionMetadata + (*CreateTaskRequest)(nil), // 2: flyteidl.admin.CreateTaskRequest + (*CreateTaskResponse)(nil), // 3: flyteidl.admin.CreateTaskResponse + (*CreateRequestHeader)(nil), // 4: flyteidl.admin.CreateRequestHeader + (*ExecuteTaskSyncRequest)(nil), // 5: flyteidl.admin.ExecuteTaskSyncRequest + (*ExecuteTaskSyncResponseHeader)(nil), // 6: flyteidl.admin.ExecuteTaskSyncResponseHeader + (*ExecuteTaskSyncResponse)(nil), // 7: flyteidl.admin.ExecuteTaskSyncResponse + (*GetTaskRequest)(nil), // 8: flyteidl.admin.GetTaskRequest + (*GetTaskResponse)(nil), // 9: flyteidl.admin.GetTaskResponse + (*Resource)(nil), // 10: flyteidl.admin.Resource + (*DeleteTaskRequest)(nil), // 11: flyteidl.admin.DeleteTaskRequest + (*DeleteTaskResponse)(nil), // 12: flyteidl.admin.DeleteTaskResponse + (*Agent)(nil), // 13: flyteidl.admin.Agent + (*TaskCategory)(nil), // 14: flyteidl.admin.TaskCategory + (*GetAgentRequest)(nil), // 15: flyteidl.admin.GetAgentRequest + (*GetAgentResponse)(nil), // 16: flyteidl.admin.GetAgentResponse + (*ListAgentsRequest)(nil), // 17: flyteidl.admin.ListAgentsRequest + (*ListAgentsResponse)(nil), // 18: flyteidl.admin.ListAgentsResponse + (*GetTaskMetricsRequest)(nil), // 19: flyteidl.admin.GetTaskMetricsRequest + (*GetTaskMetricsResponse)(nil), // 20: flyteidl.admin.GetTaskMetricsResponse + (*GetTaskLogsRequest)(nil), // 21: flyteidl.admin.GetTaskLogsRequest + (*GetTaskLogsResponseHeader)(nil), // 22: flyteidl.admin.GetTaskLogsResponseHeader + (*GetTaskLogsResponseBody)(nil), // 23: flyteidl.admin.GetTaskLogsResponseBody + (*GetTaskLogsResponse)(nil), // 24: flyteidl.admin.GetTaskLogsResponse + nil, // 25: flyteidl.admin.TaskExecutionMetadata.LabelsEntry + nil, // 26: flyteidl.admin.TaskExecutionMetadata.AnnotationsEntry + nil, // 27: flyteidl.admin.TaskExecutionMetadata.EnvironmentVariablesEntry + (*core.TaskExecutionIdentifier)(nil), // 28: flyteidl.core.TaskExecutionIdentifier + (*core.TaskNodeOverrides)(nil), // 29: flyteidl.core.TaskNodeOverrides + (*core.LiteralMap)(nil), // 30: flyteidl.core.LiteralMap + (*core.TaskTemplate)(nil), // 31: flyteidl.core.TaskTemplate + (*core.TaskLog)(nil), // 32: flyteidl.core.TaskLog + (core.TaskExecution_Phase)(0), // 33: flyteidl.core.TaskExecution.Phase + (*structpb.Struct)(nil), // 34: google.protobuf.Struct + (*timestamppb.Timestamp)(nil), // 35: google.protobuf.Timestamp + (*durationpb.Duration)(nil), // 36: google.protobuf.Duration + (*core.ExecutionMetricResult)(nil), // 37: flyteidl.core.ExecutionMetricResult } var file_flyteidl_admin_agent_proto_depIdxs = []int32{ - 21, // 0: flyteidl.admin.TaskExecutionMetadata.task_execution_id:type_name -> flyteidl.core.TaskExecutionIdentifier - 18, // 1: flyteidl.admin.TaskExecutionMetadata.labels:type_name -> flyteidl.admin.TaskExecutionMetadata.LabelsEntry - 19, // 2: flyteidl.admin.TaskExecutionMetadata.annotations:type_name -> flyteidl.admin.TaskExecutionMetadata.AnnotationsEntry - 20, // 3: flyteidl.admin.TaskExecutionMetadata.environment_variables:type_name -> flyteidl.admin.TaskExecutionMetadata.EnvironmentVariablesEntry - 22, // 4: flyteidl.admin.CreateTaskRequest.inputs:type_name -> flyteidl.core.LiteralMap - 23, // 5: flyteidl.admin.CreateTaskRequest.template:type_name -> flyteidl.core.TaskTemplate - 1, // 6: flyteidl.admin.CreateTaskRequest.task_execution_metadata:type_name -> flyteidl.admin.TaskExecutionMetadata - 6, // 7: flyteidl.admin.CreateTaskResponse.resource:type_name -> flyteidl.admin.Resource - 6, // 8: flyteidl.admin.GetTaskResponse.resource:type_name -> flyteidl.admin.Resource - 24, // 9: flyteidl.admin.GetTaskResponse.log_links:type_name -> flyteidl.core.TaskLog - 0, // 10: flyteidl.admin.Resource.state:type_name -> flyteidl.admin.State - 22, // 11: flyteidl.admin.Resource.outputs:type_name -> flyteidl.core.LiteralMap - 24, // 12: flyteidl.admin.Resource.log_links:type_name -> flyteidl.core.TaskLog - 25, // 13: flyteidl.admin.Resource.phase:type_name -> flyteidl.core.TaskExecution.Phase - 9, // 14: flyteidl.admin.GetAgentResponse.agent:type_name -> flyteidl.admin.Agent - 9, // 15: flyteidl.admin.ListAgentsResponse.agents:type_name -> flyteidl.admin.Agent - 26, // 16: flyteidl.admin.GetTaskMetricsRequest.start_time:type_name -> google.protobuf.Timestamp - 26, // 17: flyteidl.admin.GetTaskMetricsRequest.end_time:type_name -> google.protobuf.Timestamp - 27, // 18: flyteidl.admin.GetTaskMetricsRequest.step:type_name -> google.protobuf.Duration - 28, // 19: flyteidl.admin.GetTaskMetricsResponse.results:type_name -> flyteidl.core.ExecutionMetricResult - 20, // [20:20] is the sub-list for method output_type - 20, // [20:20] is the sub-list for method input_type - 20, // [20:20] is the sub-list for extension type_name - 20, // [20:20] is the sub-list for extension extendee - 0, // [0:20] is the sub-list for field type_name + 28, // 0: flyteidl.admin.TaskExecutionMetadata.task_execution_id:type_name -> flyteidl.core.TaskExecutionIdentifier + 25, // 1: flyteidl.admin.TaskExecutionMetadata.labels:type_name -> flyteidl.admin.TaskExecutionMetadata.LabelsEntry + 26, // 2: flyteidl.admin.TaskExecutionMetadata.annotations:type_name -> flyteidl.admin.TaskExecutionMetadata.AnnotationsEntry + 27, // 3: flyteidl.admin.TaskExecutionMetadata.environment_variables:type_name -> flyteidl.admin.TaskExecutionMetadata.EnvironmentVariablesEntry + 29, // 4: flyteidl.admin.TaskExecutionMetadata.overrides:type_name -> flyteidl.core.TaskNodeOverrides + 30, // 5: flyteidl.admin.CreateTaskRequest.inputs:type_name -> flyteidl.core.LiteralMap + 31, // 6: flyteidl.admin.CreateTaskRequest.template:type_name -> flyteidl.core.TaskTemplate + 1, // 7: flyteidl.admin.CreateTaskRequest.task_execution_metadata:type_name -> flyteidl.admin.TaskExecutionMetadata + 31, // 8: flyteidl.admin.CreateRequestHeader.template:type_name -> flyteidl.core.TaskTemplate + 1, // 9: flyteidl.admin.CreateRequestHeader.task_execution_metadata:type_name -> flyteidl.admin.TaskExecutionMetadata + 4, // 10: flyteidl.admin.ExecuteTaskSyncRequest.header:type_name -> flyteidl.admin.CreateRequestHeader + 30, // 11: flyteidl.admin.ExecuteTaskSyncRequest.inputs:type_name -> flyteidl.core.LiteralMap + 10, // 12: flyteidl.admin.ExecuteTaskSyncResponseHeader.resource:type_name -> flyteidl.admin.Resource + 6, // 13: flyteidl.admin.ExecuteTaskSyncResponse.header:type_name -> flyteidl.admin.ExecuteTaskSyncResponseHeader + 30, // 14: flyteidl.admin.ExecuteTaskSyncResponse.outputs:type_name -> flyteidl.core.LiteralMap + 14, // 15: flyteidl.admin.GetTaskRequest.task_category:type_name -> flyteidl.admin.TaskCategory + 10, // 16: flyteidl.admin.GetTaskResponse.resource:type_name -> flyteidl.admin.Resource + 0, // 17: flyteidl.admin.Resource.state:type_name -> flyteidl.admin.State + 30, // 18: flyteidl.admin.Resource.outputs:type_name -> flyteidl.core.LiteralMap + 32, // 19: flyteidl.admin.Resource.log_links:type_name -> flyteidl.core.TaskLog + 33, // 20: flyteidl.admin.Resource.phase:type_name -> flyteidl.core.TaskExecution.Phase + 34, // 21: flyteidl.admin.Resource.custom_info:type_name -> google.protobuf.Struct + 14, // 22: flyteidl.admin.DeleteTaskRequest.task_category:type_name -> flyteidl.admin.TaskCategory + 14, // 23: flyteidl.admin.Agent.supported_task_categories:type_name -> flyteidl.admin.TaskCategory + 13, // 24: flyteidl.admin.GetAgentResponse.agent:type_name -> flyteidl.admin.Agent + 13, // 25: flyteidl.admin.ListAgentsResponse.agents:type_name -> flyteidl.admin.Agent + 35, // 26: flyteidl.admin.GetTaskMetricsRequest.start_time:type_name -> google.protobuf.Timestamp + 35, // 27: flyteidl.admin.GetTaskMetricsRequest.end_time:type_name -> google.protobuf.Timestamp + 36, // 28: flyteidl.admin.GetTaskMetricsRequest.step:type_name -> google.protobuf.Duration + 14, // 29: flyteidl.admin.GetTaskMetricsRequest.task_category:type_name -> flyteidl.admin.TaskCategory + 37, // 30: flyteidl.admin.GetTaskMetricsResponse.results:type_name -> flyteidl.core.ExecutionMetricResult + 14, // 31: flyteidl.admin.GetTaskLogsRequest.task_category:type_name -> flyteidl.admin.TaskCategory + 22, // 32: flyteidl.admin.GetTaskLogsResponse.header:type_name -> flyteidl.admin.GetTaskLogsResponseHeader + 23, // 33: flyteidl.admin.GetTaskLogsResponse.body:type_name -> flyteidl.admin.GetTaskLogsResponseBody + 34, // [34:34] is the sub-list for method output_type + 34, // [34:34] is the sub-list for method input_type + 34, // [34:34] is the sub-list for extension type_name + 34, // [34:34] is the sub-list for extension extendee + 0, // [0:34] is the sub-list for field type_name } func init() { file_flyteidl_admin_agent_proto_init() } @@ -1469,7 +2134,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*GetTaskRequest); i { + switch v := v.(*CreateRequestHeader); i { case 0: return &v.state case 1: @@ -1481,7 +2146,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*GetTaskResponse); i { + switch v := v.(*ExecuteTaskSyncRequest); i { case 0: return &v.state case 1: @@ -1493,7 +2158,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*Resource); i { + switch v := v.(*ExecuteTaskSyncResponseHeader); i { case 0: return &v.state case 1: @@ -1505,7 +2170,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*DeleteTaskRequest); i { + switch v := v.(*ExecuteTaskSyncResponse); i { case 0: return &v.state case 1: @@ -1517,7 +2182,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*DeleteTaskResponse); i { + switch v := v.(*GetTaskRequest); i { case 0: return &v.state case 1: @@ -1529,7 +2194,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*Agent); i { + switch v := v.(*GetTaskResponse); i { case 0: return &v.state case 1: @@ -1541,7 +2206,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*GetAgentRequest); i { + switch v := v.(*Resource); i { case 0: return &v.state case 1: @@ -1553,7 +2218,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*GetAgentResponse); i { + switch v := v.(*DeleteTaskRequest); i { case 0: return &v.state case 1: @@ -1565,7 +2230,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[11].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*ListAgentsRequest); i { + switch v := v.(*DeleteTaskResponse); i { case 0: return &v.state case 1: @@ -1577,7 +2242,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[12].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*ListAgentsResponse); i { + switch v := v.(*Agent); i { case 0: return &v.state case 1: @@ -1589,7 +2254,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[13].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*GetTaskMetricsRequest); i { + switch v := v.(*TaskCategory); i { case 0: return &v.state case 1: @@ -1601,7 +2266,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[14].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*GetTaskMetricsResponse); i { + switch v := v.(*GetAgentRequest); i { case 0: return &v.state case 1: @@ -1613,7 +2278,7 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[15].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*GetTaskLogsRequest); i { + switch v := v.(*GetAgentResponse); i { case 0: return &v.state case 1: @@ -1625,6 +2290,90 @@ func file_flyteidl_admin_agent_proto_init() { } } file_flyteidl_admin_agent_proto_msgTypes[16].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListAgentsRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_flyteidl_admin_agent_proto_msgTypes[17].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListAgentsResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_flyteidl_admin_agent_proto_msgTypes[18].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetTaskMetricsRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_flyteidl_admin_agent_proto_msgTypes[19].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetTaskMetricsResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_flyteidl_admin_agent_proto_msgTypes[20].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetTaskLogsRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_flyteidl_admin_agent_proto_msgTypes[21].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetTaskLogsResponseHeader); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_flyteidl_admin_agent_proto_msgTypes[22].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetTaskLogsResponseBody); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_flyteidl_admin_agent_proto_msgTypes[23].Exporter = func(v interface{}, i int) interface{} { switch v := v.(*GetTaskLogsResponse); i { case 0: return &v.state @@ -1637,9 +2386,17 @@ func file_flyteidl_admin_agent_proto_init() { } } } - file_flyteidl_admin_agent_proto_msgTypes[2].OneofWrappers = []interface{}{ - (*CreateTaskResponse_ResourceMeta)(nil), - (*CreateTaskResponse_Resource)(nil), + file_flyteidl_admin_agent_proto_msgTypes[4].OneofWrappers = []interface{}{ + (*ExecuteTaskSyncRequest_Header)(nil), + (*ExecuteTaskSyncRequest_Inputs)(nil), + } + file_flyteidl_admin_agent_proto_msgTypes[6].OneofWrappers = []interface{}{ + (*ExecuteTaskSyncResponse_Header)(nil), + (*ExecuteTaskSyncResponse_Outputs)(nil), + } + file_flyteidl_admin_agent_proto_msgTypes[23].OneofWrappers = []interface{}{ + (*GetTaskLogsResponse_Header)(nil), + (*GetTaskLogsResponse_Body)(nil), } type x struct{} out := protoimpl.TypeBuilder{ @@ -1647,7 +2404,7 @@ func file_flyteidl_admin_agent_proto_init() { GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: file_flyteidl_admin_agent_proto_rawDesc, NumEnums: 1, - NumMessages: 20, + NumMessages: 27, NumExtensions: 0, NumServices: 0, }, diff --git a/flyteidl/gen/pb-go/flyteidl/admin/execution.pb.go b/flyteidl/gen/pb-go/flyteidl/admin/execution.pb.go index 205ea0f303..e288f658ad 100644 --- a/flyteidl/gen/pb-go/flyteidl/admin/execution.pb.go +++ b/flyteidl/gen/pb-go/flyteidl/admin/execution.pb.go @@ -89,6 +89,8 @@ const ( ExecutionMetadata_CHILD_WORKFLOW ExecutionMetadata_ExecutionMode = 4 // This execution was recovered from another execution. ExecutionMetadata_RECOVERED ExecutionMetadata_ExecutionMode = 5 + // Execution was kicked off by the artifact trigger system + ExecutionMetadata_TRIGGER ExecutionMetadata_ExecutionMode = 6 ) // Enum value maps for ExecutionMetadata_ExecutionMode. @@ -100,6 +102,7 @@ var ( 3: "RELAUNCH", 4: "CHILD_WORKFLOW", 5: "RECOVERED", + 6: "TRIGGER", } ExecutionMetadata_ExecutionMode_value = map[string]int32{ "MANUAL": 0, @@ -108,6 +111,7 @@ var ( "RELAUNCH": 3, "CHILD_WORKFLOW": 4, "RECOVERED": 5, + "TRIGGER": 6, } ) @@ -2113,7 +2117,7 @@ var file_flyteidl_admin_execution_proto_rawDesc = []byte{ 0x09, 0x52, 0x10, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x12, 0x1c, 0x0a, 0x09, 0x6e, 0x61, 0x6d, 0x65, 0x73, 0x70, 0x61, 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x6e, 0x61, 0x6d, 0x65, 0x73, 0x70, 0x61, 0x63, - 0x65, 0x22, 0xf8, 0x04, 0x0a, 0x11, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, + 0x65, 0x22, 0x85, 0x05, 0x0a, 0x11, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x43, 0x0a, 0x04, 0x6d, 0x6f, 0x64, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x2f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, @@ -2146,166 +2150,167 @@ var file_flyteidl_admin_execution_proto_rawDesc = []byte{ 0x74, 0x5f, 0x69, 0x64, 0x73, 0x18, 0x12, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x49, 0x44, 0x52, 0x0b, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, - 0x49, 0x64, 0x73, 0x22, 0x67, 0x0a, 0x0d, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, + 0x49, 0x64, 0x73, 0x22, 0x74, 0x0a, 0x0d, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x6f, 0x64, 0x65, 0x12, 0x0a, 0x0a, 0x06, 0x4d, 0x41, 0x4e, 0x55, 0x41, 0x4c, 0x10, 0x00, 0x12, 0x0d, 0x0a, 0x09, 0x53, 0x43, 0x48, 0x45, 0x44, 0x55, 0x4c, 0x45, 0x44, 0x10, 0x01, 0x12, 0x0a, 0x0a, 0x06, 0x53, 0x59, 0x53, 0x54, 0x45, 0x4d, 0x10, 0x02, 0x12, 0x0c, 0x0a, 0x08, 0x52, 0x45, 0x4c, 0x41, 0x55, 0x4e, 0x43, 0x48, 0x10, 0x03, 0x12, 0x12, 0x0a, 0x0e, 0x43, 0x48, 0x49, 0x4c, 0x44, 0x5f, 0x57, 0x4f, 0x52, 0x4b, 0x46, 0x4c, 0x4f, 0x57, 0x10, 0x04, 0x12, 0x0d, 0x0a, - 0x09, 0x52, 0x45, 0x43, 0x4f, 0x56, 0x45, 0x52, 0x45, 0x44, 0x10, 0x05, 0x22, 0x56, 0x0a, 0x10, - 0x4e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4c, 0x69, 0x73, 0x74, - 0x12, 0x42, 0x0a, 0x0d, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, - 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, - 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x4e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, - 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x0d, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, - 0x69, 0x6f, 0x6e, 0x73, 0x22, 0x90, 0x08, 0x0a, 0x0d, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, - 0x6f, 0x6e, 0x53, 0x70, 0x65, 0x63, 0x12, 0x3a, 0x0a, 0x0b, 0x6c, 0x61, 0x75, 0x6e, 0x63, 0x68, - 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, - 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x49, 0x64, 0x65, 0x6e, - 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, 0x0a, 0x6c, 0x61, 0x75, 0x6e, 0x63, 0x68, 0x50, 0x6c, - 0x61, 0x6e, 0x12, 0x35, 0x0a, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, - 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x42, 0x02, 0x18, - 0x01, 0x52, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x12, 0x3d, 0x0a, 0x08, 0x6d, 0x65, 0x74, - 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x66, 0x6c, - 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, - 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x08, - 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x48, 0x0a, 0x0d, 0x6e, 0x6f, 0x74, 0x69, - 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, - 0x20, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, - 0x2e, 0x4e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4c, 0x69, 0x73, - 0x74, 0x48, 0x00, 0x52, 0x0d, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, - 0x6e, 0x73, 0x12, 0x21, 0x0a, 0x0b, 0x64, 0x69, 0x73, 0x61, 0x62, 0x6c, 0x65, 0x5f, 0x61, 0x6c, - 0x6c, 0x18, 0x06, 0x20, 0x01, 0x28, 0x08, 0x48, 0x00, 0x52, 0x0a, 0x64, 0x69, 0x73, 0x61, 0x62, - 0x6c, 0x65, 0x41, 0x6c, 0x6c, 0x12, 0x2e, 0x0a, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x18, - 0x07, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, - 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x52, 0x06, 0x6c, - 0x61, 0x62, 0x65, 0x6c, 0x73, 0x12, 0x3d, 0x0a, 0x0b, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, - 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1b, 0x2e, 0x66, 0x6c, 0x79, - 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x41, 0x6e, 0x6e, 0x6f, - 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x52, 0x0b, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, - 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x49, 0x0a, 0x10, 0x73, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, - 0x5f, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x78, 0x74, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1e, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, - 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x78, 0x74, 0x52, 0x0f, - 0x73, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x78, 0x74, 0x12, - 0x39, 0x0a, 0x09, 0x61, 0x75, 0x74, 0x68, 0x5f, 0x72, 0x6f, 0x6c, 0x65, 0x18, 0x10, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, - 0x6d, 0x69, 0x6e, 0x2e, 0x41, 0x75, 0x74, 0x68, 0x52, 0x6f, 0x6c, 0x65, 0x42, 0x02, 0x18, 0x01, - 0x52, 0x08, 0x61, 0x75, 0x74, 0x68, 0x52, 0x6f, 0x6c, 0x65, 0x12, 0x4d, 0x0a, 0x12, 0x71, 0x75, - 0x61, 0x6c, 0x69, 0x74, 0x79, 0x5f, 0x6f, 0x66, 0x5f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, - 0x18, 0x11, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x51, 0x75, 0x61, 0x6c, 0x69, 0x74, 0x79, 0x4f, 0x66, - 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x52, 0x10, 0x71, 0x75, 0x61, 0x6c, 0x69, 0x74, 0x79, - 0x4f, 0x66, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x27, 0x0a, 0x0f, 0x6d, 0x61, 0x78, - 0x5f, 0x70, 0x61, 0x72, 0x61, 0x6c, 0x6c, 0x65, 0x6c, 0x69, 0x73, 0x6d, 0x18, 0x12, 0x20, 0x01, - 0x28, 0x05, 0x52, 0x0e, 0x6d, 0x61, 0x78, 0x50, 0x61, 0x72, 0x61, 0x6c, 0x6c, 0x65, 0x6c, 0x69, - 0x73, 0x6d, 0x12, 0x58, 0x0a, 0x16, 0x72, 0x61, 0x77, 0x5f, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, - 0x5f, 0x64, 0x61, 0x74, 0x61, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x18, 0x13, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, - 0x6d, 0x69, 0x6e, 0x2e, 0x52, 0x61, 0x77, 0x4f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x44, 0x61, 0x74, - 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x52, 0x13, 0x72, 0x61, 0x77, 0x4f, 0x75, 0x74, 0x70, - 0x75, 0x74, 0x44, 0x61, 0x74, 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x50, 0x0a, 0x12, - 0x63, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x5f, 0x61, 0x73, 0x73, 0x69, 0x67, 0x6e, 0x6d, 0x65, - 0x6e, 0x74, 0x18, 0x14, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, - 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, - 0x72, 0x41, 0x73, 0x73, 0x69, 0x67, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x52, 0x11, 0x63, 0x6c, 0x75, - 0x73, 0x74, 0x65, 0x72, 0x41, 0x73, 0x73, 0x69, 0x67, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x12, 0x40, - 0x0a, 0x0d, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x72, 0x75, 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, 0x18, - 0x15, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x42, 0x6f, 0x6f, 0x6c, 0x56, 0x61, 0x6c, 0x75, - 0x65, 0x52, 0x0d, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x72, 0x75, 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, - 0x12, 0x27, 0x0a, 0x0f, 0x6f, 0x76, 0x65, 0x72, 0x77, 0x72, 0x69, 0x74, 0x65, 0x5f, 0x63, 0x61, - 0x63, 0x68, 0x65, 0x18, 0x16, 0x20, 0x01, 0x28, 0x08, 0x52, 0x0e, 0x6f, 0x76, 0x65, 0x72, 0x77, - 0x72, 0x69, 0x74, 0x65, 0x43, 0x61, 0x63, 0x68, 0x65, 0x12, 0x28, 0x0a, 0x04, 0x65, 0x6e, 0x76, - 0x73, 0x18, 0x17, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x14, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, - 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x6e, 0x76, 0x73, 0x52, 0x04, 0x65, - 0x6e, 0x76, 0x73, 0x12, 0x12, 0x0a, 0x04, 0x74, 0x61, 0x67, 0x73, 0x18, 0x18, 0x20, 0x03, 0x28, - 0x09, 0x52, 0x04, 0x74, 0x61, 0x67, 0x73, 0x42, 0x18, 0x0a, 0x16, 0x6e, 0x6f, 0x74, 0x69, 0x66, - 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6f, 0x76, 0x65, 0x72, 0x72, 0x69, 0x64, 0x65, - 0x73, 0x4a, 0x04, 0x08, 0x04, 0x10, 0x05, 0x22, 0x6d, 0x0a, 0x19, 0x45, 0x78, 0x65, 0x63, 0x75, - 0x74, 0x69, 0x6f, 0x6e, 0x54, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x65, 0x52, 0x65, 0x71, + 0x09, 0x52, 0x45, 0x43, 0x4f, 0x56, 0x45, 0x52, 0x45, 0x44, 0x10, 0x05, 0x12, 0x0b, 0x0a, 0x07, + 0x54, 0x52, 0x49, 0x47, 0x47, 0x45, 0x52, 0x10, 0x06, 0x22, 0x56, 0x0a, 0x10, 0x4e, 0x6f, 0x74, + 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4c, 0x69, 0x73, 0x74, 0x12, 0x42, 0x0a, + 0x0d, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x01, + 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, + 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x4e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, + 0x6f, 0x6e, 0x52, 0x0d, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x73, 0x22, 0x90, 0x08, 0x0a, 0x0d, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, + 0x70, 0x65, 0x63, 0x12, 0x3a, 0x0a, 0x0b, 0x6c, 0x61, 0x75, 0x6e, 0x63, 0x68, 0x5f, 0x70, 0x6c, + 0x61, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, + 0x69, 0x65, 0x72, 0x52, 0x0a, 0x6c, 0x61, 0x75, 0x6e, 0x63, 0x68, 0x50, 0x6c, 0x61, 0x6e, 0x12, + 0x35, 0x0a, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x42, 0x02, 0x18, 0x01, 0x52, 0x06, + 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x12, 0x3d, 0x0a, 0x08, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, + 0x74, 0x61, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, + 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x08, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x48, 0x0a, 0x0d, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, + 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x20, 0x2e, 0x66, + 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x4e, 0x6f, + 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4c, 0x69, 0x73, 0x74, 0x48, 0x00, + 0x52, 0x0d, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, + 0x21, 0x0a, 0x0b, 0x64, 0x69, 0x73, 0x61, 0x62, 0x6c, 0x65, 0x5f, 0x61, 0x6c, 0x6c, 0x18, 0x06, + 0x20, 0x01, 0x28, 0x08, 0x48, 0x00, 0x52, 0x0a, 0x64, 0x69, 0x73, 0x61, 0x62, 0x6c, 0x65, 0x41, + 0x6c, 0x6c, 0x12, 0x2e, 0x0a, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x18, 0x07, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, + 0x6d, 0x69, 0x6e, 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x52, 0x06, 0x6c, 0x61, 0x62, 0x65, + 0x6c, 0x73, 0x12, 0x3d, 0x0a, 0x0b, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x73, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1b, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, + 0x69, 0x6f, 0x6e, 0x73, 0x52, 0x0b, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x73, 0x12, 0x49, 0x0a, 0x10, 0x73, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x5f, 0x63, 0x6f, + 0x6e, 0x74, 0x65, 0x78, 0x74, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1e, 0x2e, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x65, 0x63, 0x75, + 0x72, 0x69, 0x74, 0x79, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x78, 0x74, 0x52, 0x0f, 0x73, 0x65, 0x63, + 0x75, 0x72, 0x69, 0x74, 0x79, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x78, 0x74, 0x12, 0x39, 0x0a, 0x09, + 0x61, 0x75, 0x74, 0x68, 0x5f, 0x72, 0x6f, 0x6c, 0x65, 0x18, 0x10, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x18, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, + 0x2e, 0x41, 0x75, 0x74, 0x68, 0x52, 0x6f, 0x6c, 0x65, 0x42, 0x02, 0x18, 0x01, 0x52, 0x08, 0x61, + 0x75, 0x74, 0x68, 0x52, 0x6f, 0x6c, 0x65, 0x12, 0x4d, 0x0a, 0x12, 0x71, 0x75, 0x61, 0x6c, 0x69, + 0x74, 0x79, 0x5f, 0x6f, 0x66, 0x5f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x18, 0x11, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x51, 0x75, 0x61, 0x6c, 0x69, 0x74, 0x79, 0x4f, 0x66, 0x53, 0x65, 0x72, + 0x76, 0x69, 0x63, 0x65, 0x52, 0x10, 0x71, 0x75, 0x61, 0x6c, 0x69, 0x74, 0x79, 0x4f, 0x66, 0x53, + 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x27, 0x0a, 0x0f, 0x6d, 0x61, 0x78, 0x5f, 0x70, 0x61, + 0x72, 0x61, 0x6c, 0x6c, 0x65, 0x6c, 0x69, 0x73, 0x6d, 0x18, 0x12, 0x20, 0x01, 0x28, 0x05, 0x52, + 0x0e, 0x6d, 0x61, 0x78, 0x50, 0x61, 0x72, 0x61, 0x6c, 0x6c, 0x65, 0x6c, 0x69, 0x73, 0x6d, 0x12, + 0x58, 0x0a, 0x16, 0x72, 0x61, 0x77, 0x5f, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x5f, 0x64, 0x61, + 0x74, 0x61, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x18, 0x13, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x23, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, + 0x2e, 0x52, 0x61, 0x77, 0x4f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x44, 0x61, 0x74, 0x61, 0x43, 0x6f, + 0x6e, 0x66, 0x69, 0x67, 0x52, 0x13, 0x72, 0x61, 0x77, 0x4f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x44, + 0x61, 0x74, 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x50, 0x0a, 0x12, 0x63, 0x6c, 0x75, + 0x73, 0x74, 0x65, 0x72, 0x5f, 0x61, 0x73, 0x73, 0x69, 0x67, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x18, + 0x14, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, + 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x41, 0x73, + 0x73, 0x69, 0x67, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x52, 0x11, 0x63, 0x6c, 0x75, 0x73, 0x74, 0x65, + 0x72, 0x41, 0x73, 0x73, 0x69, 0x67, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x12, 0x40, 0x0a, 0x0d, 0x69, + 0x6e, 0x74, 0x65, 0x72, 0x72, 0x75, 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, 0x18, 0x15, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, + 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x42, 0x6f, 0x6f, 0x6c, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x0d, + 0x69, 0x6e, 0x74, 0x65, 0x72, 0x72, 0x75, 0x70, 0x74, 0x69, 0x62, 0x6c, 0x65, 0x12, 0x27, 0x0a, + 0x0f, 0x6f, 0x76, 0x65, 0x72, 0x77, 0x72, 0x69, 0x74, 0x65, 0x5f, 0x63, 0x61, 0x63, 0x68, 0x65, + 0x18, 0x16, 0x20, 0x01, 0x28, 0x08, 0x52, 0x0e, 0x6f, 0x76, 0x65, 0x72, 0x77, 0x72, 0x69, 0x74, + 0x65, 0x43, 0x61, 0x63, 0x68, 0x65, 0x12, 0x28, 0x0a, 0x04, 0x65, 0x6e, 0x76, 0x73, 0x18, 0x17, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x14, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, + 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x6e, 0x76, 0x73, 0x52, 0x04, 0x65, 0x6e, 0x76, 0x73, + 0x12, 0x12, 0x0a, 0x04, 0x74, 0x61, 0x67, 0x73, 0x18, 0x18, 0x20, 0x03, 0x28, 0x09, 0x52, 0x04, + 0x74, 0x61, 0x67, 0x73, 0x42, 0x18, 0x0a, 0x16, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, + 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6f, 0x76, 0x65, 0x72, 0x72, 0x69, 0x64, 0x65, 0x73, 0x4a, 0x04, + 0x08, 0x04, 0x10, 0x05, 0x22, 0x6d, 0x0a, 0x19, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, + 0x6e, 0x54, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, + 0x74, 0x12, 0x3a, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2a, 0x2e, + 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x57, 0x6f, + 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, + 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, 0x02, 0x69, 0x64, 0x12, 0x14, 0x0a, + 0x05, 0x63, 0x61, 0x75, 0x73, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x63, 0x61, + 0x75, 0x73, 0x65, 0x22, 0x1c, 0x0a, 0x1a, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, + 0x54, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, + 0x65, 0x22, 0x5d, 0x0a, 0x1f, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, + 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x65, 0x74, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3a, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, 0x02, 0x69, 0x64, - 0x12, 0x14, 0x0a, 0x05, 0x63, 0x61, 0x75, 0x73, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x05, 0x63, 0x61, 0x75, 0x73, 0x65, 0x22, 0x1c, 0x0a, 0x1a, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, - 0x69, 0x6f, 0x6e, 0x54, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x65, 0x52, 0x65, 0x73, 0x70, - 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x5d, 0x0a, 0x1f, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, - 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x65, 0x74, 0x44, 0x61, 0x74, 0x61, - 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3a, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x0b, 0x32, 0x2a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, - 0x6f, 0x72, 0x65, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, - 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, - 0x02, 0x69, 0x64, 0x22, 0x88, 0x02, 0x0a, 0x20, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, - 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x65, 0x74, 0x44, 0x61, 0x74, 0x61, - 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x35, 0x0a, 0x07, 0x6f, 0x75, 0x74, 0x70, - 0x75, 0x74, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, 0x6c, 0x79, 0x74, - 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x55, 0x72, 0x6c, 0x42, 0x6c, - 0x6f, 0x62, 0x42, 0x02, 0x18, 0x01, 0x52, 0x07, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x12, - 0x33, 0x0a, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, - 0x17, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, - 0x2e, 0x55, 0x72, 0x6c, 0x42, 0x6c, 0x6f, 0x62, 0x42, 0x02, 0x18, 0x01, 0x52, 0x06, 0x69, 0x6e, - 0x70, 0x75, 0x74, 0x73, 0x12, 0x3a, 0x0a, 0x0b, 0x66, 0x75, 0x6c, 0x6c, 0x5f, 0x69, 0x6e, 0x70, - 0x75, 0x74, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, - 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, - 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x0a, 0x66, 0x75, 0x6c, 0x6c, 0x49, 0x6e, 0x70, 0x75, 0x74, 0x73, - 0x12, 0x3c, 0x0a, 0x0c, 0x66, 0x75, 0x6c, 0x6c, 0x5f, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, - 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x22, 0x88, 0x02, 0x0a, 0x20, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, + 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x65, 0x74, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x35, 0x0a, 0x07, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x55, 0x72, 0x6c, 0x42, 0x6c, 0x6f, 0x62, 0x42, + 0x02, 0x18, 0x01, 0x52, 0x07, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x12, 0x33, 0x0a, 0x06, + 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, + 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x55, 0x72, + 0x6c, 0x42, 0x6c, 0x6f, 0x62, 0x42, 0x02, 0x18, 0x01, 0x52, 0x06, 0x69, 0x6e, 0x70, 0x75, 0x74, + 0x73, 0x12, 0x3a, 0x0a, 0x0b, 0x66, 0x75, 0x6c, 0x6c, 0x5f, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, + 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, - 0x70, 0x52, 0x0b, 0x66, 0x75, 0x6c, 0x6c, 0x4f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x22, 0x8a, - 0x01, 0x0a, 0x16, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x55, 0x70, 0x64, 0x61, - 0x74, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3a, 0x0a, 0x02, 0x69, 0x64, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, - 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, - 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, - 0x72, 0x52, 0x02, 0x69, 0x64, 0x12, 0x34, 0x0a, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x18, 0x02, - 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1e, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, - 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, - 0x74, 0x61, 0x74, 0x65, 0x52, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x22, 0xae, 0x01, 0x0a, 0x1b, - 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, 0x74, 0x61, 0x74, 0x65, 0x43, 0x68, - 0x61, 0x6e, 0x67, 0x65, 0x44, 0x65, 0x74, 0x61, 0x69, 0x6c, 0x73, 0x12, 0x34, 0x0a, 0x05, 0x73, - 0x74, 0x61, 0x74, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1e, 0x2e, 0x66, 0x6c, 0x79, - 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, 0x63, - 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, 0x74, 0x61, 0x74, 0x65, 0x52, 0x05, 0x73, 0x74, 0x61, 0x74, - 0x65, 0x12, 0x3b, 0x0a, 0x0b, 0x6f, 0x63, 0x63, 0x75, 0x72, 0x72, 0x65, 0x64, 0x5f, 0x61, 0x74, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, - 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, - 0x6d, 0x70, 0x52, 0x0a, 0x6f, 0x63, 0x63, 0x75, 0x72, 0x72, 0x65, 0x64, 0x41, 0x74, 0x12, 0x1c, - 0x0a, 0x09, 0x70, 0x72, 0x69, 0x6e, 0x63, 0x69, 0x70, 0x61, 0x6c, 0x18, 0x03, 0x20, 0x01, 0x28, - 0x09, 0x52, 0x09, 0x70, 0x72, 0x69, 0x6e, 0x63, 0x69, 0x70, 0x61, 0x6c, 0x22, 0x19, 0x0a, 0x17, + 0x70, 0x52, 0x0a, 0x66, 0x75, 0x6c, 0x6c, 0x49, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x12, 0x3c, 0x0a, + 0x0c, 0x66, 0x75, 0x6c, 0x6c, 0x5f, 0x6f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x18, 0x04, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x0b, + 0x66, 0x75, 0x6c, 0x6c, 0x4f, 0x75, 0x74, 0x70, 0x75, 0x74, 0x73, 0x22, 0x8a, 0x01, 0x0a, 0x16, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x52, - 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x76, 0x0a, 0x22, 0x57, 0x6f, 0x72, 0x6b, 0x66, - 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x65, 0x74, 0x4d, - 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3a, 0x0a, - 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2a, 0x2e, 0x66, 0x6c, 0x79, 0x74, - 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, - 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, - 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, 0x02, 0x69, 0x64, 0x12, 0x14, 0x0a, 0x05, 0x64, 0x65, 0x70, - 0x74, 0x68, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x64, 0x65, 0x70, 0x74, 0x68, 0x22, - 0x4e, 0x0a, 0x23, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, 0x75, - 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, - 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x27, 0x0a, 0x04, 0x73, 0x70, 0x61, 0x6e, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x0b, 0x32, 0x13, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, - 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x70, 0x61, 0x6e, 0x52, 0x04, 0x73, 0x70, 0x61, 0x6e, 0x2a, - 0x3e, 0x0a, 0x0e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, 0x74, 0x61, 0x74, - 0x65, 0x12, 0x14, 0x0a, 0x10, 0x45, 0x58, 0x45, 0x43, 0x55, 0x54, 0x49, 0x4f, 0x4e, 0x5f, 0x41, - 0x43, 0x54, 0x49, 0x56, 0x45, 0x10, 0x00, 0x12, 0x16, 0x0a, 0x12, 0x45, 0x58, 0x45, 0x43, 0x55, - 0x54, 0x49, 0x4f, 0x4e, 0x5f, 0x41, 0x52, 0x43, 0x48, 0x49, 0x56, 0x45, 0x44, 0x10, 0x01, 0x42, - 0xba, 0x01, 0x0a, 0x12, 0x63, 0x6f, 0x6d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, - 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x42, 0x0e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, - 0x6e, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3b, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, - 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, - 0x79, 0x74, 0x65, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, - 0x2f, 0x70, 0x62, 0x2d, 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, - 0x61, 0x64, 0x6d, 0x69, 0x6e, 0xa2, 0x02, 0x03, 0x46, 0x41, 0x58, 0xaa, 0x02, 0x0e, 0x46, 0x6c, - 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0xca, 0x02, 0x0e, 0x46, - 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0xe2, 0x02, 0x1a, - 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0x5c, 0x47, - 0x50, 0x42, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, 0x02, 0x0f, 0x46, 0x6c, 0x79, - 0x74, 0x65, 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0x62, 0x06, 0x70, 0x72, - 0x6f, 0x74, 0x6f, 0x33, + 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3a, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x2a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, 0x75, + 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, 0x02, + 0x69, 0x64, 0x12, 0x34, 0x0a, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, + 0x0e, 0x32, 0x1e, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, + 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, 0x74, 0x61, 0x74, + 0x65, 0x52, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x22, 0xae, 0x01, 0x0a, 0x1b, 0x45, 0x78, 0x65, + 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, 0x74, 0x61, 0x74, 0x65, 0x43, 0x68, 0x61, 0x6e, 0x67, + 0x65, 0x44, 0x65, 0x74, 0x61, 0x69, 0x6c, 0x73, 0x12, 0x34, 0x0a, 0x05, 0x73, 0x74, 0x61, 0x74, + 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1e, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, + 0x6f, 0x6e, 0x53, 0x74, 0x61, 0x74, 0x65, 0x52, 0x05, 0x73, 0x74, 0x61, 0x74, 0x65, 0x12, 0x3b, + 0x0a, 0x0b, 0x6f, 0x63, 0x63, 0x75, 0x72, 0x72, 0x65, 0x64, 0x5f, 0x61, 0x74, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, + 0x0a, 0x6f, 0x63, 0x63, 0x75, 0x72, 0x72, 0x65, 0x64, 0x41, 0x74, 0x12, 0x1c, 0x0a, 0x09, 0x70, + 0x72, 0x69, 0x6e, 0x63, 0x69, 0x70, 0x61, 0x6c, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, + 0x70, 0x72, 0x69, 0x6e, 0x63, 0x69, 0x70, 0x61, 0x6c, 0x22, 0x19, 0x0a, 0x17, 0x45, 0x78, 0x65, + 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x52, 0x65, 0x73, 0x70, + 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x76, 0x0a, 0x22, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, + 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x72, + 0x69, 0x63, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3a, 0x0a, 0x02, 0x69, 0x64, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, + 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, + 0x65, 0x72, 0x52, 0x02, 0x69, 0x64, 0x12, 0x14, 0x0a, 0x05, 0x64, 0x65, 0x70, 0x74, 0x68, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x64, 0x65, 0x70, 0x74, 0x68, 0x22, 0x4e, 0x0a, 0x23, + 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, + 0x6e, 0x47, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x12, 0x27, 0x0a, 0x04, 0x73, 0x70, 0x61, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x13, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x53, 0x70, 0x61, 0x6e, 0x52, 0x04, 0x73, 0x70, 0x61, 0x6e, 0x2a, 0x3e, 0x0a, 0x0e, + 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x53, 0x74, 0x61, 0x74, 0x65, 0x12, 0x14, + 0x0a, 0x10, 0x45, 0x58, 0x45, 0x43, 0x55, 0x54, 0x49, 0x4f, 0x4e, 0x5f, 0x41, 0x43, 0x54, 0x49, + 0x56, 0x45, 0x10, 0x00, 0x12, 0x16, 0x0a, 0x12, 0x45, 0x58, 0x45, 0x43, 0x55, 0x54, 0x49, 0x4f, + 0x4e, 0x5f, 0x41, 0x52, 0x43, 0x48, 0x49, 0x56, 0x45, 0x44, 0x10, 0x01, 0x42, 0xba, 0x01, 0x0a, + 0x12, 0x63, 0x6f, 0x6d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, + 0x6d, 0x69, 0x6e, 0x42, 0x0e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x50, 0x72, + 0x6f, 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3b, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, + 0x6d, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, 0x62, + 0x2d, 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x61, 0x64, 0x6d, + 0x69, 0x6e, 0xa2, 0x02, 0x03, 0x46, 0x41, 0x58, 0xaa, 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x2e, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0xca, 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, + 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0xe2, 0x02, 0x1a, 0x46, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0x5c, 0x47, 0x50, 0x42, 0x4d, + 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, 0x02, 0x0f, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x3a, 0x3a, 0x41, 0x64, 0x6d, 0x69, 0x6e, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, + 0x33, } var ( diff --git a/flyteidl/gen/pb-go/flyteidl/core/artifact_id.pb.go b/flyteidl/gen/pb-go/flyteidl/core/artifact_id.pb.go index 86cc4025eb..7d692916a7 100644 --- a/flyteidl/gen/pb-go/flyteidl/core/artifact_id.pb.go +++ b/flyteidl/gen/pb-go/flyteidl/core/artifact_id.pb.go @@ -99,7 +99,6 @@ type ArtifactBindingData struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - Index uint32 `protobuf:"varint,1,opt,name=index,proto3" json:"index,omitempty"` // These two fields are only relevant in the partition value case // // Types that are assignable to PartitionData: @@ -143,13 +142,6 @@ func (*ArtifactBindingData) Descriptor() ([]byte, []int) { return file_flyteidl_core_artifact_id_proto_rawDescGZIP(), []int{1} } -func (x *ArtifactBindingData) GetIndex() uint32 { - if x != nil { - return x.Index - } - return 0 -} - func (m *ArtifactBindingData) GetPartitionData() isArtifactBindingData_PartitionData { if m != nil { return m.PartitionData @@ -708,102 +700,101 @@ var file_flyteidl_core_artifact_id_proto_rawDesc = []byte{ 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x10, 0x0a, 0x03, 0x6f, 0x72, 0x67, 0x18, 0x04, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x03, 0x6f, 0x72, 0x67, 0x22, 0xb9, 0x01, 0x0a, 0x13, 0x41, 0x72, 0x74, + 0x01, 0x28, 0x09, 0x52, 0x03, 0x6f, 0x72, 0x67, 0x22, 0xa3, 0x01, 0x0a, 0x13, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x44, 0x61, 0x74, 0x61, - 0x12, 0x14, 0x0a, 0x05, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0d, 0x52, - 0x05, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x12, 0x25, 0x0a, 0x0d, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, - 0x69, 0x6f, 0x6e, 0x5f, 0x6b, 0x65, 0x79, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, - 0x0c, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x4b, 0x65, 0x79, 0x12, 0x35, 0x0a, - 0x16, 0x62, 0x69, 0x6e, 0x64, 0x5f, 0x74, 0x6f, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x70, 0x61, - 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x48, 0x00, 0x52, - 0x13, 0x62, 0x69, 0x6e, 0x64, 0x54, 0x6f, 0x54, 0x69, 0x6d, 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, - 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x1c, 0x0a, 0x09, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, 0x72, - 0x6d, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, - 0x72, 0x6d, 0x42, 0x10, 0x0a, 0x0e, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x5f, - 0x64, 0x61, 0x74, 0x61, 0x22, 0x24, 0x0a, 0x10, 0x49, 0x6e, 0x70, 0x75, 0x74, 0x42, 0x69, 0x6e, - 0x64, 0x69, 0x6e, 0x67, 0x44, 0x61, 0x74, 0x61, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x72, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x76, 0x61, 0x72, 0x22, 0x92, 0x02, 0x0a, 0x0a, 0x4c, - 0x61, 0x62, 0x65, 0x6c, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x23, 0x0a, 0x0c, 0x73, 0x74, 0x61, - 0x74, 0x69, 0x63, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x48, - 0x00, 0x52, 0x0b, 0x73, 0x74, 0x61, 0x74, 0x69, 0x63, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x3b, - 0x0a, 0x0a, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, - 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x48, 0x00, - 0x52, 0x09, 0x74, 0x69, 0x6d, 0x65, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x51, 0x0a, 0x11, 0x74, - 0x72, 0x69, 0x67, 0x67, 0x65, 0x72, 0x65, 0x64, 0x5f, 0x62, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, - 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x42, - 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x44, 0x61, 0x74, 0x61, 0x48, 0x00, 0x52, 0x10, 0x74, 0x72, - 0x69, 0x67, 0x67, 0x65, 0x72, 0x65, 0x64, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x12, 0x46, - 0x0a, 0x0d, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x5f, 0x62, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x18, - 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, - 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x49, 0x6e, 0x70, 0x75, 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, - 0x6e, 0x67, 0x44, 0x61, 0x74, 0x61, 0x48, 0x00, 0x52, 0x0c, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x42, - 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x42, 0x07, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, - 0x9d, 0x01, 0x0a, 0x0a, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x3a, - 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x24, 0x2e, - 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x50, 0x61, - 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x45, 0x6e, - 0x74, 0x72, 0x79, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x1a, 0x53, 0x0a, 0x0a, 0x56, 0x61, - 0x6c, 0x75, 0x65, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x2f, 0x0a, 0x05, 0x76, 0x61, - 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, + 0x12, 0x25, 0x0a, 0x0d, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6b, 0x65, + 0x79, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x0c, 0x70, 0x61, 0x72, 0x74, 0x69, + 0x74, 0x69, 0x6f, 0x6e, 0x4b, 0x65, 0x79, 0x12, 0x35, 0x0a, 0x16, 0x62, 0x69, 0x6e, 0x64, 0x5f, + 0x74, 0x6f, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, + 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x48, 0x00, 0x52, 0x13, 0x62, 0x69, 0x6e, 0x64, 0x54, + 0x6f, 0x54, 0x69, 0x6d, 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x1c, + 0x0a, 0x09, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, 0x72, 0x6d, 0x18, 0x04, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x09, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, 0x72, 0x6d, 0x42, 0x10, 0x0a, 0x0e, + 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x64, 0x61, 0x74, 0x61, 0x22, 0x24, + 0x0a, 0x10, 0x49, 0x6e, 0x70, 0x75, 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x44, 0x61, + 0x74, 0x61, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x03, 0x76, 0x61, 0x72, 0x22, 0x92, 0x02, 0x0a, 0x0a, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x56, 0x61, + 0x6c, 0x75, 0x65, 0x12, 0x23, 0x0a, 0x0c, 0x73, 0x74, 0x61, 0x74, 0x69, 0x63, 0x5f, 0x76, 0x61, + 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x0b, 0x73, 0x74, 0x61, + 0x74, 0x69, 0x63, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x3b, 0x0a, 0x0a, 0x74, 0x69, 0x6d, 0x65, + 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, + 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, + 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x48, 0x00, 0x52, 0x09, 0x74, 0x69, 0x6d, 0x65, + 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x51, 0x0a, 0x11, 0x74, 0x72, 0x69, 0x67, 0x67, 0x65, 0x72, + 0x65, 0x64, 0x5f, 0x62, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, + 0x44, 0x61, 0x74, 0x61, 0x48, 0x00, 0x52, 0x10, 0x74, 0x72, 0x69, 0x67, 0x67, 0x65, 0x72, 0x65, + 0x64, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x12, 0x46, 0x0a, 0x0d, 0x69, 0x6e, 0x70, 0x75, + 0x74, 0x5f, 0x62, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x1f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x49, 0x6e, 0x70, 0x75, 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x44, 0x61, 0x74, 0x61, + 0x48, 0x00, 0x52, 0x0c, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, + 0x42, 0x07, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0x9d, 0x01, 0x0a, 0x0a, 0x50, 0x61, + 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x3a, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, + 0x65, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, + 0x6e, 0x73, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x05, 0x76, + 0x61, 0x6c, 0x75, 0x65, 0x1a, 0x53, 0x0a, 0x0a, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x45, 0x6e, 0x74, + 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x03, 0x6b, 0x65, 0x79, 0x12, 0x2f, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, + 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x40, 0x0a, 0x0d, 0x54, 0x69, 0x6d, + 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x2f, 0x0a, 0x05, 0x76, 0x61, + 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x56, - 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, - 0x40, 0x0a, 0x0d, 0x54, 0x69, 0x6d, 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, - 0x12, 0x2f, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, - 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, - 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, - 0x65, 0x22, 0xe5, 0x01, 0x0a, 0x0a, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x49, 0x44, - 0x12, 0x3d, 0x0a, 0x0c, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x5f, 0x6b, 0x65, 0x79, - 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x4b, - 0x65, 0x79, 0x52, 0x0b, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x4b, 0x65, 0x79, 0x12, - 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, - 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x39, 0x0a, 0x0a, 0x70, 0x61, 0x72, - 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, - 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x50, 0x61, - 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x52, 0x0a, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, - 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x43, 0x0a, 0x0e, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x70, 0x61, 0x72, - 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, - 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x69, 0x6d, - 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x0d, 0x74, 0x69, 0x6d, 0x65, - 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x22, 0x7d, 0x0a, 0x0b, 0x41, 0x72, 0x74, - 0x69, 0x66, 0x61, 0x63, 0x74, 0x54, 0x61, 0x67, 0x12, 0x3d, 0x0a, 0x0c, 0x61, 0x72, 0x74, 0x69, - 0x66, 0x61, 0x63, 0x74, 0x5f, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, - 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x4b, 0x65, 0x79, 0x52, 0x0b, 0x61, 0x72, 0x74, 0x69, - 0x66, 0x61, 0x63, 0x74, 0x4b, 0x65, 0x79, 0x12, 0x2f, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x56, 0x61, 0x6c, 0x75, - 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0xf0, 0x01, 0x0a, 0x0d, 0x41, 0x72, 0x74, - 0x69, 0x66, 0x61, 0x63, 0x74, 0x51, 0x75, 0x65, 0x72, 0x79, 0x12, 0x3c, 0x0a, 0x0b, 0x61, 0x72, - 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, - 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, - 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x49, 0x44, 0x48, 0x00, 0x52, 0x0a, 0x61, 0x72, - 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x49, 0x64, 0x12, 0x3f, 0x0a, 0x0c, 0x61, 0x72, 0x74, 0x69, - 0x66, 0x61, 0x63, 0x74, 0x5f, 0x74, 0x61, 0x67, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, - 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x54, 0x61, 0x67, 0x48, 0x00, 0x52, 0x0b, 0x61, 0x72, - 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x54, 0x61, 0x67, 0x12, 0x12, 0x0a, 0x03, 0x75, 0x72, 0x69, - 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x03, 0x75, 0x72, 0x69, 0x12, 0x3e, 0x0a, - 0x07, 0x62, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, - 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x44, 0x61, - 0x74, 0x61, 0x48, 0x00, 0x52, 0x07, 0x62, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x42, 0x0c, 0x0a, - 0x0a, 0x69, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x42, 0xb5, 0x01, 0x0a, 0x11, - 0x63, 0x6f, 0x6d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, - 0x65, 0x42, 0x0f, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x49, 0x64, 0x50, 0x72, 0x6f, - 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3a, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, - 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x2f, - 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, 0x62, 0x2d, - 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, 0x72, 0x65, - 0xa2, 0x02, 0x03, 0x46, 0x43, 0x58, 0xaa, 0x02, 0x0d, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x2e, 0x43, 0x6f, 0x72, 0x65, 0xca, 0x02, 0x0d, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x5c, 0x43, 0x6f, 0x72, 0x65, 0xe2, 0x02, 0x19, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x5c, 0x43, 0x6f, 0x72, 0x65, 0x5c, 0x47, 0x50, 0x42, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, - 0x74, 0x61, 0xea, 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x43, - 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0xe5, 0x01, 0x0a, 0x0a, + 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x49, 0x44, 0x12, 0x3d, 0x0a, 0x0c, 0x61, 0x72, + 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x5f, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x1a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x4b, 0x65, 0x79, 0x52, 0x0b, 0x61, 0x72, + 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x4b, 0x65, 0x79, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, + 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, + 0x69, 0x6f, 0x6e, 0x12, 0x39, 0x0a, 0x0a, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, + 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, + 0x6e, 0x73, 0x52, 0x0a, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x43, + 0x0a, 0x0e, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, + 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, + 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x0d, 0x74, 0x69, 0x6d, 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, + 0x69, 0x6f, 0x6e, 0x22, 0x7d, 0x0a, 0x0b, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x54, + 0x61, 0x67, 0x12, 0x3d, 0x0a, 0x0c, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x5f, 0x6b, + 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, + 0x74, 0x4b, 0x65, 0x79, 0x52, 0x0b, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x4b, 0x65, + 0x79, 0x12, 0x2f, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, + 0x75, 0x65, 0x22, 0xf0, 0x01, 0x0a, 0x0d, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x51, + 0x75, 0x65, 0x72, 0x79, 0x12, 0x3c, 0x0a, 0x0b, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, + 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, + 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, + 0x63, 0x74, 0x49, 0x44, 0x48, 0x00, 0x52, 0x0a, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, + 0x49, 0x64, 0x12, 0x3f, 0x0a, 0x0c, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, 0x5f, 0x74, + 0x61, 0x67, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, + 0x74, 0x54, 0x61, 0x67, 0x48, 0x00, 0x52, 0x0b, 0x61, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, 0x74, + 0x54, 0x61, 0x67, 0x12, 0x12, 0x0a, 0x03, 0x75, 0x72, 0x69, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, + 0x48, 0x00, 0x52, 0x03, 0x75, 0x72, 0x69, 0x12, 0x3e, 0x0a, 0x07, 0x62, 0x69, 0x6e, 0x64, 0x69, + 0x6e, 0x67, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x74, 0x69, 0x66, 0x61, 0x63, + 0x74, 0x42, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x44, 0x61, 0x74, 0x61, 0x48, 0x00, 0x52, 0x07, + 0x62, 0x69, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x42, 0x0c, 0x0a, 0x0a, 0x69, 0x64, 0x65, 0x6e, 0x74, + 0x69, 0x66, 0x69, 0x65, 0x72, 0x42, 0xb5, 0x01, 0x0a, 0x11, 0x63, 0x6f, 0x6d, 0x2e, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0f, 0x41, 0x72, 0x74, + 0x69, 0x66, 0x61, 0x63, 0x74, 0x49, 0x64, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3a, + 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, 0x62, 0x2d, 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0xa2, 0x02, 0x03, 0x46, 0x43, 0x58, + 0xaa, 0x02, 0x0d, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x43, 0x6f, 0x72, 0x65, + 0xca, 0x02, 0x0d, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x43, 0x6f, 0x72, 0x65, + 0xe2, 0x02, 0x19, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x43, 0x6f, 0x72, 0x65, + 0x5c, 0x47, 0x50, 0x42, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, 0x02, 0x0e, 0x46, + 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x43, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( diff --git a/flyteidl/gen/pb-go/flyteidl/core/workflow.pb.go b/flyteidl/gen/pb-go/flyteidl/core/workflow.pb.go index f9318f539f..8b6cc6ab1d 100644 --- a/flyteidl/gen/pb-go/flyteidl/core/workflow.pb.go +++ b/flyteidl/gen/pb-go/flyteidl/core/workflow.pb.go @@ -1498,6 +1498,8 @@ type TaskNodeOverrides struct { // Overrides for all non-standard resources, not captured by // v1.ResourceRequirements, to allocate to a task. ExtendedResources *ExtendedResources `protobuf:"bytes,2,opt,name=extended_resources,json=extendedResources,proto3" json:"extended_resources,omitempty"` + // Override for the image used by task pods. + ContainerImage string `protobuf:"bytes,3,opt,name=container_image,json=containerImage,proto3" json:"container_image,omitempty"` } func (x *TaskNodeOverrides) Reset() { @@ -1546,6 +1548,13 @@ func (x *TaskNodeOverrides) GetExtendedResources() *ExtendedResources { return nil } +func (x *TaskNodeOverrides) GetContainerImage() string { + if x != nil { + return x.ContainerImage + } + return "" +} + // A structure that uniquely identifies a launch plan in the system. type LaunchPlanTemplate struct { state protoimpl.MessageState @@ -1839,7 +1848,7 @@ var file_flyteidl_core_workflow_proto_rawDesc = []byte{ 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x44, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x73, 0x52, 0x10, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x44, 0x65, 0x66, - 0x61, 0x75, 0x6c, 0x74, 0x73, 0x22, 0x9c, 0x01, 0x0a, 0x11, 0x54, 0x61, 0x73, 0x6b, 0x4e, 0x6f, + 0x61, 0x75, 0x6c, 0x74, 0x73, 0x22, 0xc5, 0x01, 0x0a, 0x11, 0x54, 0x61, 0x73, 0x6b, 0x4e, 0x6f, 0x64, 0x65, 0x4f, 0x76, 0x65, 0x72, 0x72, 0x69, 0x64, 0x65, 0x73, 0x12, 0x36, 0x0a, 0x09, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x52, @@ -1849,30 +1858,33 @@ var file_flyteidl_core_workflow_proto_rawDesc = []byte{ 0x20, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x45, 0x78, 0x74, 0x65, 0x6e, 0x64, 0x65, 0x64, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x73, 0x52, 0x11, 0x65, 0x78, 0x74, 0x65, 0x6e, 0x64, 0x65, 0x64, 0x52, 0x65, 0x73, 0x6f, 0x75, - 0x72, 0x63, 0x65, 0x73, 0x22, 0xba, 0x01, 0x0a, 0x12, 0x4c, 0x61, 0x75, 0x6e, 0x63, 0x68, 0x50, - 0x6c, 0x61, 0x6e, 0x54, 0x65, 0x6d, 0x70, 0x6c, 0x61, 0x74, 0x65, 0x12, 0x29, 0x0a, 0x02, 0x69, - 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, - 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, - 0x65, 0x72, 0x52, 0x02, 0x69, 0x64, 0x12, 0x3b, 0x0a, 0x09, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x66, - 0x61, 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x66, 0x6c, 0x79, 0x74, - 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x54, 0x79, 0x70, 0x65, 0x64, 0x49, - 0x6e, 0x74, 0x65, 0x72, 0x66, 0x61, 0x63, 0x65, 0x52, 0x09, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x66, - 0x61, 0x63, 0x65, 0x12, 0x3c, 0x0a, 0x0c, 0x66, 0x69, 0x78, 0x65, 0x64, 0x5f, 0x69, 0x6e, 0x70, - 0x75, 0x74, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, - 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, - 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x0b, 0x66, 0x69, 0x78, 0x65, 0x64, 0x49, 0x6e, 0x70, 0x75, 0x74, - 0x73, 0x42, 0xb3, 0x01, 0x0a, 0x11, 0x63, 0x6f, 0x6d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, - 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0d, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, - 0x77, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3a, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, - 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, - 0x79, 0x74, 0x65, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, - 0x2f, 0x70, 0x62, 0x2d, 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, - 0x63, 0x6f, 0x72, 0x65, 0xa2, 0x02, 0x03, 0x46, 0x43, 0x58, 0xaa, 0x02, 0x0d, 0x46, 0x6c, 0x79, - 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x43, 0x6f, 0x72, 0x65, 0xca, 0x02, 0x0d, 0x46, 0x6c, 0x79, - 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x43, 0x6f, 0x72, 0x65, 0xe2, 0x02, 0x19, 0x46, 0x6c, 0x79, - 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x43, 0x6f, 0x72, 0x65, 0x5c, 0x47, 0x50, 0x42, 0x4d, 0x65, - 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x3a, 0x3a, 0x43, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x72, 0x63, 0x65, 0x73, 0x12, 0x27, 0x0a, 0x0f, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, + 0x72, 0x5f, 0x69, 0x6d, 0x61, 0x67, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e, 0x63, + 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x49, 0x6d, 0x61, 0x67, 0x65, 0x22, 0xba, 0x01, + 0x0a, 0x12, 0x4c, 0x61, 0x75, 0x6e, 0x63, 0x68, 0x50, 0x6c, 0x61, 0x6e, 0x54, 0x65, 0x6d, 0x70, + 0x6c, 0x61, 0x74, 0x65, 0x12, 0x29, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x49, 0x64, 0x65, 0x6e, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x52, 0x02, 0x69, 0x64, 0x12, + 0x3b, 0x0a, 0x09, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x66, 0x61, 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x2e, 0x54, 0x79, 0x70, 0x65, 0x64, 0x49, 0x6e, 0x74, 0x65, 0x72, 0x66, 0x61, 0x63, + 0x65, 0x52, 0x09, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x66, 0x61, 0x63, 0x65, 0x12, 0x3c, 0x0a, 0x0c, + 0x66, 0x69, 0x78, 0x65, 0x64, 0x5f, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x18, 0x03, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x74, 0x65, 0x72, 0x61, 0x6c, 0x4d, 0x61, 0x70, 0x52, 0x0b, 0x66, + 0x69, 0x78, 0x65, 0x64, 0x49, 0x6e, 0x70, 0x75, 0x74, 0x73, 0x42, 0xb3, 0x01, 0x0a, 0x11, 0x63, + 0x6f, 0x6d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x42, 0x0d, 0x57, 0x6f, 0x72, 0x6b, 0x66, 0x6c, 0x6f, 0x77, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x50, + 0x01, 0x5a, 0x3a, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x2f, 0x66, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, 0x62, 0x2d, 0x67, 0x6f, 0x2f, + 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0xa2, 0x02, 0x03, + 0x46, 0x43, 0x58, 0xaa, 0x02, 0x0d, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x43, + 0x6f, 0x72, 0x65, 0xca, 0x02, 0x0d, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x43, + 0x6f, 0x72, 0x65, 0xe2, 0x02, 0x19, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x43, + 0x6f, 0x72, 0x65, 0x5c, 0x47, 0x50, 0x42, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, + 0x02, 0x0e, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x43, 0x6f, 0x72, 0x65, + 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( diff --git a/flyteidl/gen/pb-go/flyteidl/service/agent.pb.go b/flyteidl/gen/pb-go/flyteidl/service/agent.pb.go index c9ffc4fa68..078dc9dd8a 100644 --- a/flyteidl/gen/pb-go/flyteidl/service/agent.pb.go +++ b/flyteidl/gen/pb-go/flyteidl/service/agent.pb.go @@ -30,99 +30,139 @@ var file_flyteidl_service_agent_proto_rawDesc = []byte{ 0x1a, 0x1c, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1a, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2f, 0x61, - 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x32, 0xcc, 0x03, 0x0a, 0x11, 0x41, - 0x73, 0x79, 0x6e, 0x63, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, - 0x12, 0x55, 0x0a, 0x0a, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x12, 0x21, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, - 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, - 0x69, 0x6e, 0x2e, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x73, - 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x4c, 0x0a, 0x07, 0x47, 0x65, 0x74, 0x54, 0x61, - 0x73, 0x6b, 0x12, 0x1e, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, - 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, - 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x73, 0x70, 0x6f, - 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x55, 0x0a, 0x0a, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, - 0x61, 0x73, 0x6b, 0x12, 0x21, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, - 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, - 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, - 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, - 0x73, 0x6b, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x61, 0x0a, 0x0e, - 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x12, 0x25, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, - 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, - 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x26, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, - 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, - 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, - 0x58, 0x0a, 0x0b, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x12, 0x22, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, - 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x1a, 0x23, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, - 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, - 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x32, 0xf0, 0x01, 0x0a, 0x14, 0x41, 0x67, - 0x65, 0x6e, 0x74, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x53, 0x65, 0x72, 0x76, 0x69, - 0x63, 0x65, 0x12, 0x6b, 0x0a, 0x08, 0x47, 0x65, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x12, 0x1f, - 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, - 0x47, 0x65, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, - 0x20, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, - 0x2e, 0x47, 0x65, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, - 0x65, 0x22, 0x1c, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x16, 0x12, 0x14, 0x2f, 0x61, 0x70, 0x69, 0x2f, - 0x76, 0x31, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2f, 0x7b, 0x6e, 0x61, 0x6d, 0x65, 0x7d, 0x12, - 0x6b, 0x0a, 0x0a, 0x4c, 0x69, 0x73, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x12, 0x21, 0x2e, - 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x4c, - 0x69, 0x73, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, + 0x67, 0x65, 0x6e, 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x32, 0xa1, 0x01, 0x0a, 0x10, 0x53, + 0x79, 0x6e, 0x63, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, + 0x8c, 0x01, 0x0a, 0x0f, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x53, + 0x79, 0x6e, 0x63, 0x12, 0x26, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, + 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, + 0x53, 0x79, 0x6e, 0x63, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x45, 0x78, 0x65, + 0x63, 0x75, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x53, 0x79, 0x6e, 0x63, 0x52, 0x65, 0x73, 0x70, + 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x24, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x1e, 0x3a, 0x01, 0x2a, 0x22, + 0x19, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2f, 0x74, + 0x61, 0x73, 0x6b, 0x2f, 0x73, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x28, 0x01, 0x30, 0x01, 0x32, 0xe3, + 0x06, 0x0a, 0x11, 0x41, 0x73, 0x79, 0x6e, 0x63, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x53, 0x65, 0x72, + 0x76, 0x69, 0x63, 0x65, 0x12, 0x72, 0x0a, 0x0a, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, + 0x73, 0x6b, 0x12, 0x21, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, + 0x6d, 0x69, 0x6e, 0x2e, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, + 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, + 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x54, 0x61, 0x73, + 0x6b, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x1d, 0x82, 0xd3, 0xe4, 0x93, 0x02, + 0x17, 0x3a, 0x01, 0x2a, 0x22, 0x12, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, + 0x65, 0x6e, 0x74, 0x2f, 0x74, 0x61, 0x73, 0x6b, 0x12, 0xa3, 0x01, 0x0a, 0x07, 0x47, 0x65, 0x74, + 0x54, 0x61, 0x73, 0x6b, 0x12, 0x1e, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, + 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, + 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, + 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x57, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x51, 0x12, 0x4f, 0x2f, + 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2f, 0x74, 0x61, 0x73, + 0x6b, 0x2f, 0x7b, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, + 0x2e, 0x6e, 0x61, 0x6d, 0x65, 0x7d, 0x2f, 0x7b, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, + 0x65, 0x67, 0x6f, 0x72, 0x79, 0x2e, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x7d, 0x2f, 0x7b, + 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x7d, 0x12, 0xb7, + 0x01, 0x0a, 0x0a, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x12, 0x21, 0x2e, + 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x44, + 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, - 0x6e, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, - 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x16, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x10, 0x12, 0x0e, 0x2f, 0x61, - 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x42, 0xc2, 0x01, 0x0a, - 0x14, 0x63, 0x6f, 0x6d, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x73, 0x65, - 0x72, 0x76, 0x69, 0x63, 0x65, 0x42, 0x0a, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x50, 0x72, 0x6f, 0x74, - 0x6f, 0x50, 0x01, 0x5a, 0x3d, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, - 0x66, 0x6c, 0x79, 0x74, 0x65, 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x2f, 0x66, - 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, 0x62, 0x2d, 0x67, - 0x6f, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, - 0x63, 0x65, 0xa2, 0x02, 0x03, 0x46, 0x53, 0x58, 0xaa, 0x02, 0x10, 0x46, 0x6c, 0x79, 0x74, 0x65, - 0x69, 0x64, 0x6c, 0x2e, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0xca, 0x02, 0x10, 0x46, 0x6c, - 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0xe2, 0x02, - 0x1c, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, - 0x65, 0x5c, 0x47, 0x50, 0x42, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, 0x02, 0x11, - 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, - 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x6e, 0x2e, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x54, 0x61, 0x73, 0x6b, 0x52, 0x65, 0x73, 0x70, + 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x62, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x5c, 0x2a, 0x5a, 0x2f, 0x61, + 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2f, 0x74, 0x61, 0x73, 0x6b, + 0x5f, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x2f, 0x7b, 0x74, 0x61, 0x73, + 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x2e, 0x6e, 0x61, 0x6d, 0x65, 0x7d, + 0x2f, 0x7b, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x2e, + 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x7d, 0x2f, 0x7b, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, + 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x7d, 0x12, 0xc0, 0x01, 0x0a, 0x0e, 0x47, 0x65, 0x74, + 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x12, 0x25, 0x2e, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, + 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x1a, 0x26, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, + 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65, 0x74, 0x72, 0x69, + 0x63, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x5f, 0x82, 0xd3, 0xe4, 0x93, + 0x02, 0x59, 0x12, 0x57, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, 0x65, 0x6e, + 0x74, 0x2f, 0x74, 0x61, 0x73, 0x6b, 0x2f, 0x6d, 0x65, 0x74, 0x72, 0x69, 0x63, 0x73, 0x2f, 0x7b, + 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x2e, 0x6e, 0x61, + 0x6d, 0x65, 0x7d, 0x2f, 0x7b, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, + 0x72, 0x79, 0x2e, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x7d, 0x2f, 0x7b, 0x72, 0x65, 0x73, + 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x7d, 0x12, 0xb6, 0x01, 0x0a, 0x0b, + 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x12, 0x22, 0x2e, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, + 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, + 0x23, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, + 0x2e, 0x47, 0x65, 0x74, 0x54, 0x61, 0x73, 0x6b, 0x4c, 0x6f, 0x67, 0x73, 0x52, 0x65, 0x73, 0x70, + 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x5c, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x56, 0x12, 0x54, 0x2f, 0x61, + 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x2f, 0x74, 0x61, 0x73, 0x6b, + 0x2f, 0x6c, 0x6f, 0x67, 0x73, 0x2f, 0x7b, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x63, 0x61, 0x74, 0x65, + 0x67, 0x6f, 0x72, 0x79, 0x2e, 0x6e, 0x61, 0x6d, 0x65, 0x7d, 0x2f, 0x7b, 0x74, 0x61, 0x73, 0x6b, + 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x79, 0x2e, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, + 0x6e, 0x7d, 0x2f, 0x7b, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6d, 0x65, 0x74, + 0x61, 0x7d, 0x30, 0x01, 0x32, 0xf0, 0x01, 0x0a, 0x14, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x4d, 0x65, + 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x6b, 0x0a, + 0x08, 0x47, 0x65, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x12, 0x1f, 0x2e, 0x66, 0x6c, 0x79, 0x74, + 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x41, 0x67, + 0x65, 0x6e, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x20, 0x2e, 0x66, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x47, 0x65, 0x74, 0x41, + 0x67, 0x65, 0x6e, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x1c, 0x82, 0xd3, + 0xe4, 0x93, 0x02, 0x16, 0x12, 0x14, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, 0x2f, 0x61, 0x67, + 0x65, 0x6e, 0x74, 0x2f, 0x7b, 0x6e, 0x61, 0x6d, 0x65, 0x7d, 0x12, 0x6b, 0x0a, 0x0a, 0x4c, 0x69, + 0x73, 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x12, 0x21, 0x2e, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x41, 0x67, + 0x65, 0x6e, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x6c, + 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x2e, 0x4c, 0x69, 0x73, + 0x74, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, + 0x16, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x10, 0x12, 0x0e, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x31, + 0x2f, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x73, 0x42, 0xc2, 0x01, 0x0a, 0x14, 0x63, 0x6f, 0x6d, 0x2e, + 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, + 0x42, 0x0a, 0x41, 0x67, 0x65, 0x6e, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x50, 0x01, 0x5a, 0x3d, + 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, + 0x6f, 0x72, 0x67, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x2f, 0x66, 0x6c, 0x79, 0x74, 0x65, 0x69, + 0x64, 0x6c, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x70, 0x62, 0x2d, 0x67, 0x6f, 0x2f, 0x66, 0x6c, 0x79, + 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0xa2, 0x02, 0x03, + 0x46, 0x53, 0x58, 0xaa, 0x02, 0x10, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, 0x6c, 0x2e, 0x53, + 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0xca, 0x02, 0x10, 0x46, 0x6c, 0x79, 0x74, 0x65, 0x69, 0x64, + 0x6c, 0x5c, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0xe2, 0x02, 0x1c, 0x46, 0x6c, 0x79, 0x74, + 0x65, 0x69, 0x64, 0x6c, 0x5c, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x5c, 0x47, 0x50, 0x42, + 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0xea, 0x02, 0x11, 0x46, 0x6c, 0x79, 0x74, 0x65, + 0x69, 0x64, 0x6c, 0x3a, 0x3a, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x62, 0x06, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x33, } var file_flyteidl_service_agent_proto_goTypes = []interface{}{ - (*admin.CreateTaskRequest)(nil), // 0: flyteidl.admin.CreateTaskRequest - (*admin.GetTaskRequest)(nil), // 1: flyteidl.admin.GetTaskRequest - (*admin.DeleteTaskRequest)(nil), // 2: flyteidl.admin.DeleteTaskRequest - (*admin.GetTaskMetricsRequest)(nil), // 3: flyteidl.admin.GetTaskMetricsRequest - (*admin.GetTaskLogsRequest)(nil), // 4: flyteidl.admin.GetTaskLogsRequest - (*admin.GetAgentRequest)(nil), // 5: flyteidl.admin.GetAgentRequest - (*admin.ListAgentsRequest)(nil), // 6: flyteidl.admin.ListAgentsRequest - (*admin.CreateTaskResponse)(nil), // 7: flyteidl.admin.CreateTaskResponse - (*admin.GetTaskResponse)(nil), // 8: flyteidl.admin.GetTaskResponse - (*admin.DeleteTaskResponse)(nil), // 9: flyteidl.admin.DeleteTaskResponse - (*admin.GetTaskMetricsResponse)(nil), // 10: flyteidl.admin.GetTaskMetricsResponse - (*admin.GetTaskLogsResponse)(nil), // 11: flyteidl.admin.GetTaskLogsResponse - (*admin.GetAgentResponse)(nil), // 12: flyteidl.admin.GetAgentResponse - (*admin.ListAgentsResponse)(nil), // 13: flyteidl.admin.ListAgentsResponse + (*admin.ExecuteTaskSyncRequest)(nil), // 0: flyteidl.admin.ExecuteTaskSyncRequest + (*admin.CreateTaskRequest)(nil), // 1: flyteidl.admin.CreateTaskRequest + (*admin.GetTaskRequest)(nil), // 2: flyteidl.admin.GetTaskRequest + (*admin.DeleteTaskRequest)(nil), // 3: flyteidl.admin.DeleteTaskRequest + (*admin.GetTaskMetricsRequest)(nil), // 4: flyteidl.admin.GetTaskMetricsRequest + (*admin.GetTaskLogsRequest)(nil), // 5: flyteidl.admin.GetTaskLogsRequest + (*admin.GetAgentRequest)(nil), // 6: flyteidl.admin.GetAgentRequest + (*admin.ListAgentsRequest)(nil), // 7: flyteidl.admin.ListAgentsRequest + (*admin.ExecuteTaskSyncResponse)(nil), // 8: flyteidl.admin.ExecuteTaskSyncResponse + (*admin.CreateTaskResponse)(nil), // 9: flyteidl.admin.CreateTaskResponse + (*admin.GetTaskResponse)(nil), // 10: flyteidl.admin.GetTaskResponse + (*admin.DeleteTaskResponse)(nil), // 11: flyteidl.admin.DeleteTaskResponse + (*admin.GetTaskMetricsResponse)(nil), // 12: flyteidl.admin.GetTaskMetricsResponse + (*admin.GetTaskLogsResponse)(nil), // 13: flyteidl.admin.GetTaskLogsResponse + (*admin.GetAgentResponse)(nil), // 14: flyteidl.admin.GetAgentResponse + (*admin.ListAgentsResponse)(nil), // 15: flyteidl.admin.ListAgentsResponse } var file_flyteidl_service_agent_proto_depIdxs = []int32{ - 0, // 0: flyteidl.service.AsyncAgentService.CreateTask:input_type -> flyteidl.admin.CreateTaskRequest - 1, // 1: flyteidl.service.AsyncAgentService.GetTask:input_type -> flyteidl.admin.GetTaskRequest - 2, // 2: flyteidl.service.AsyncAgentService.DeleteTask:input_type -> flyteidl.admin.DeleteTaskRequest - 3, // 3: flyteidl.service.AsyncAgentService.GetTaskMetrics:input_type -> flyteidl.admin.GetTaskMetricsRequest - 4, // 4: flyteidl.service.AsyncAgentService.GetTaskLogs:input_type -> flyteidl.admin.GetTaskLogsRequest - 5, // 5: flyteidl.service.AgentMetadataService.GetAgent:input_type -> flyteidl.admin.GetAgentRequest - 6, // 6: flyteidl.service.AgentMetadataService.ListAgents:input_type -> flyteidl.admin.ListAgentsRequest - 7, // 7: flyteidl.service.AsyncAgentService.CreateTask:output_type -> flyteidl.admin.CreateTaskResponse - 8, // 8: flyteidl.service.AsyncAgentService.GetTask:output_type -> flyteidl.admin.GetTaskResponse - 9, // 9: flyteidl.service.AsyncAgentService.DeleteTask:output_type -> flyteidl.admin.DeleteTaskResponse - 10, // 10: flyteidl.service.AsyncAgentService.GetTaskMetrics:output_type -> flyteidl.admin.GetTaskMetricsResponse - 11, // 11: flyteidl.service.AsyncAgentService.GetTaskLogs:output_type -> flyteidl.admin.GetTaskLogsResponse - 12, // 12: flyteidl.service.AgentMetadataService.GetAgent:output_type -> flyteidl.admin.GetAgentResponse - 13, // 13: flyteidl.service.AgentMetadataService.ListAgents:output_type -> flyteidl.admin.ListAgentsResponse - 7, // [7:14] is the sub-list for method output_type - 0, // [0:7] is the sub-list for method input_type + 0, // 0: flyteidl.service.SyncAgentService.ExecuteTaskSync:input_type -> flyteidl.admin.ExecuteTaskSyncRequest + 1, // 1: flyteidl.service.AsyncAgentService.CreateTask:input_type -> flyteidl.admin.CreateTaskRequest + 2, // 2: flyteidl.service.AsyncAgentService.GetTask:input_type -> flyteidl.admin.GetTaskRequest + 3, // 3: flyteidl.service.AsyncAgentService.DeleteTask:input_type -> flyteidl.admin.DeleteTaskRequest + 4, // 4: flyteidl.service.AsyncAgentService.GetTaskMetrics:input_type -> flyteidl.admin.GetTaskMetricsRequest + 5, // 5: flyteidl.service.AsyncAgentService.GetTaskLogs:input_type -> flyteidl.admin.GetTaskLogsRequest + 6, // 6: flyteidl.service.AgentMetadataService.GetAgent:input_type -> flyteidl.admin.GetAgentRequest + 7, // 7: flyteidl.service.AgentMetadataService.ListAgents:input_type -> flyteidl.admin.ListAgentsRequest + 8, // 8: flyteidl.service.SyncAgentService.ExecuteTaskSync:output_type -> flyteidl.admin.ExecuteTaskSyncResponse + 9, // 9: flyteidl.service.AsyncAgentService.CreateTask:output_type -> flyteidl.admin.CreateTaskResponse + 10, // 10: flyteidl.service.AsyncAgentService.GetTask:output_type -> flyteidl.admin.GetTaskResponse + 11, // 11: flyteidl.service.AsyncAgentService.DeleteTask:output_type -> flyteidl.admin.DeleteTaskResponse + 12, // 12: flyteidl.service.AsyncAgentService.GetTaskMetrics:output_type -> flyteidl.admin.GetTaskMetricsResponse + 13, // 13: flyteidl.service.AsyncAgentService.GetTaskLogs:output_type -> flyteidl.admin.GetTaskLogsResponse + 14, // 14: flyteidl.service.AgentMetadataService.GetAgent:output_type -> flyteidl.admin.GetAgentResponse + 15, // 15: flyteidl.service.AgentMetadataService.ListAgents:output_type -> flyteidl.admin.ListAgentsResponse + 8, // [8:16] is the sub-list for method output_type + 0, // [0:8] is the sub-list for method input_type 0, // [0:0] is the sub-list for extension type_name 0, // [0:0] is the sub-list for extension extendee 0, // [0:0] is the sub-list for field type_name @@ -141,7 +181,7 @@ func file_flyteidl_service_agent_proto_init() { NumEnums: 0, NumMessages: 0, NumExtensions: 0, - NumServices: 2, + NumServices: 3, }, GoTypes: file_flyteidl_service_agent_proto_goTypes, DependencyIndexes: file_flyteidl_service_agent_proto_depIdxs, diff --git a/flyteidl/gen/pb-go/flyteidl/service/agent_grpc.pb.go b/flyteidl/gen/pb-go/flyteidl/service/agent_grpc.pb.go index 67ebe7b012..98f057da12 100644 --- a/flyteidl/gen/pb-go/flyteidl/service/agent_grpc.pb.go +++ b/flyteidl/gen/pb-go/flyteidl/service/agent_grpc.pb.go @@ -19,6 +19,128 @@ import ( // Requires gRPC-Go v1.32.0 or later. const _ = grpc.SupportPackageIsVersion7 +const ( + SyncAgentService_ExecuteTaskSync_FullMethodName = "/flyteidl.service.SyncAgentService/ExecuteTaskSync" +) + +// SyncAgentServiceClient is the client API for SyncAgentService service. +// +// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. +type SyncAgentServiceClient interface { + // ExecuteTaskSync streams the create request and inputs to the agent service and streams the outputs back. + ExecuteTaskSync(ctx context.Context, opts ...grpc.CallOption) (SyncAgentService_ExecuteTaskSyncClient, error) +} + +type syncAgentServiceClient struct { + cc grpc.ClientConnInterface +} + +func NewSyncAgentServiceClient(cc grpc.ClientConnInterface) SyncAgentServiceClient { + return &syncAgentServiceClient{cc} +} + +func (c *syncAgentServiceClient) ExecuteTaskSync(ctx context.Context, opts ...grpc.CallOption) (SyncAgentService_ExecuteTaskSyncClient, error) { + stream, err := c.cc.NewStream(ctx, &SyncAgentService_ServiceDesc.Streams[0], SyncAgentService_ExecuteTaskSync_FullMethodName, opts...) + if err != nil { + return nil, err + } + x := &syncAgentServiceExecuteTaskSyncClient{stream} + return x, nil +} + +type SyncAgentService_ExecuteTaskSyncClient interface { + Send(*admin.ExecuteTaskSyncRequest) error + Recv() (*admin.ExecuteTaskSyncResponse, error) + grpc.ClientStream +} + +type syncAgentServiceExecuteTaskSyncClient struct { + grpc.ClientStream +} + +func (x *syncAgentServiceExecuteTaskSyncClient) Send(m *admin.ExecuteTaskSyncRequest) error { + return x.ClientStream.SendMsg(m) +} + +func (x *syncAgentServiceExecuteTaskSyncClient) Recv() (*admin.ExecuteTaskSyncResponse, error) { + m := new(admin.ExecuteTaskSyncResponse) + if err := x.ClientStream.RecvMsg(m); err != nil { + return nil, err + } + return m, nil +} + +// SyncAgentServiceServer is the server API for SyncAgentService service. +// All implementations should embed UnimplementedSyncAgentServiceServer +// for forward compatibility +type SyncAgentServiceServer interface { + // ExecuteTaskSync streams the create request and inputs to the agent service and streams the outputs back. + ExecuteTaskSync(SyncAgentService_ExecuteTaskSyncServer) error +} + +// UnimplementedSyncAgentServiceServer should be embedded to have forward compatible implementations. +type UnimplementedSyncAgentServiceServer struct { +} + +func (UnimplementedSyncAgentServiceServer) ExecuteTaskSync(SyncAgentService_ExecuteTaskSyncServer) error { + return status.Errorf(codes.Unimplemented, "method ExecuteTaskSync not implemented") +} + +// UnsafeSyncAgentServiceServer may be embedded to opt out of forward compatibility for this service. +// Use of this interface is not recommended, as added methods to SyncAgentServiceServer will +// result in compilation errors. +type UnsafeSyncAgentServiceServer interface { + mustEmbedUnimplementedSyncAgentServiceServer() +} + +func RegisterSyncAgentServiceServer(s grpc.ServiceRegistrar, srv SyncAgentServiceServer) { + s.RegisterService(&SyncAgentService_ServiceDesc, srv) +} + +func _SyncAgentService_ExecuteTaskSync_Handler(srv interface{}, stream grpc.ServerStream) error { + return srv.(SyncAgentServiceServer).ExecuteTaskSync(&syncAgentServiceExecuteTaskSyncServer{stream}) +} + +type SyncAgentService_ExecuteTaskSyncServer interface { + Send(*admin.ExecuteTaskSyncResponse) error + Recv() (*admin.ExecuteTaskSyncRequest, error) + grpc.ServerStream +} + +type syncAgentServiceExecuteTaskSyncServer struct { + grpc.ServerStream +} + +func (x *syncAgentServiceExecuteTaskSyncServer) Send(m *admin.ExecuteTaskSyncResponse) error { + return x.ServerStream.SendMsg(m) +} + +func (x *syncAgentServiceExecuteTaskSyncServer) Recv() (*admin.ExecuteTaskSyncRequest, error) { + m := new(admin.ExecuteTaskSyncRequest) + if err := x.ServerStream.RecvMsg(m); err != nil { + return nil, err + } + return m, nil +} + +// SyncAgentService_ServiceDesc is the grpc.ServiceDesc for SyncAgentService service. +// It's only intended for direct use with grpc.RegisterService, +// and not to be introspected or modified (even as a copy) +var SyncAgentService_ServiceDesc = grpc.ServiceDesc{ + ServiceName: "flyteidl.service.SyncAgentService", + HandlerType: (*SyncAgentServiceServer)(nil), + Methods: []grpc.MethodDesc{}, + Streams: []grpc.StreamDesc{ + { + StreamName: "ExecuteTaskSync", + Handler: _SyncAgentService_ExecuteTaskSync_Handler, + ServerStreams: true, + ClientStreams: true, + }, + }, + Metadata: "flyteidl/service/agent.proto", +} + const ( AsyncAgentService_CreateTask_FullMethodName = "/flyteidl.service.AsyncAgentService/CreateTask" AsyncAgentService_GetTask_FullMethodName = "/flyteidl.service.AsyncAgentService/GetTask" @@ -31,7 +153,7 @@ const ( // // For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. type AsyncAgentServiceClient interface { - // Send a task create request to the agent server. + // CreateTask sends a task create request to the agent service. CreateTask(ctx context.Context, in *admin.CreateTaskRequest, opts ...grpc.CallOption) (*admin.CreateTaskResponse, error) // Get job status. GetTask(ctx context.Context, in *admin.GetTaskRequest, opts ...grpc.CallOption) (*admin.GetTaskResponse, error) @@ -44,7 +166,7 @@ type AsyncAgentServiceClient interface { // - various other errors GetTaskMetrics(ctx context.Context, in *admin.GetTaskMetricsRequest, opts ...grpc.CallOption) (*admin.GetTaskMetricsResponse, error) // GetTaskLogs returns task execution logs, if available. - GetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) (*admin.GetTaskLogsResponse, error) + GetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) (AsyncAgentService_GetTaskLogsClient, error) } type asyncAgentServiceClient struct { @@ -91,20 +213,43 @@ func (c *asyncAgentServiceClient) GetTaskMetrics(ctx context.Context, in *admin. return out, nil } -func (c *asyncAgentServiceClient) GetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) (*admin.GetTaskLogsResponse, error) { - out := new(admin.GetTaskLogsResponse) - err := c.cc.Invoke(ctx, AsyncAgentService_GetTaskLogs_FullMethodName, in, out, opts...) +func (c *asyncAgentServiceClient) GetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) (AsyncAgentService_GetTaskLogsClient, error) { + stream, err := c.cc.NewStream(ctx, &AsyncAgentService_ServiceDesc.Streams[0], AsyncAgentService_GetTaskLogs_FullMethodName, opts...) if err != nil { return nil, err } - return out, nil + x := &asyncAgentServiceGetTaskLogsClient{stream} + if err := x.ClientStream.SendMsg(in); err != nil { + return nil, err + } + if err := x.ClientStream.CloseSend(); err != nil { + return nil, err + } + return x, nil +} + +type AsyncAgentService_GetTaskLogsClient interface { + Recv() (*admin.GetTaskLogsResponse, error) + grpc.ClientStream +} + +type asyncAgentServiceGetTaskLogsClient struct { + grpc.ClientStream +} + +func (x *asyncAgentServiceGetTaskLogsClient) Recv() (*admin.GetTaskLogsResponse, error) { + m := new(admin.GetTaskLogsResponse) + if err := x.ClientStream.RecvMsg(m); err != nil { + return nil, err + } + return m, nil } // AsyncAgentServiceServer is the server API for AsyncAgentService service. // All implementations should embed UnimplementedAsyncAgentServiceServer // for forward compatibility type AsyncAgentServiceServer interface { - // Send a task create request to the agent server. + // CreateTask sends a task create request to the agent service. CreateTask(context.Context, *admin.CreateTaskRequest) (*admin.CreateTaskResponse, error) // Get job status. GetTask(context.Context, *admin.GetTaskRequest) (*admin.GetTaskResponse, error) @@ -117,7 +262,7 @@ type AsyncAgentServiceServer interface { // - various other errors GetTaskMetrics(context.Context, *admin.GetTaskMetricsRequest) (*admin.GetTaskMetricsResponse, error) // GetTaskLogs returns task execution logs, if available. - GetTaskLogs(context.Context, *admin.GetTaskLogsRequest) (*admin.GetTaskLogsResponse, error) + GetTaskLogs(*admin.GetTaskLogsRequest, AsyncAgentService_GetTaskLogsServer) error } // UnimplementedAsyncAgentServiceServer should be embedded to have forward compatible implementations. @@ -136,8 +281,8 @@ func (UnimplementedAsyncAgentServiceServer) DeleteTask(context.Context, *admin.D func (UnimplementedAsyncAgentServiceServer) GetTaskMetrics(context.Context, *admin.GetTaskMetricsRequest) (*admin.GetTaskMetricsResponse, error) { return nil, status.Errorf(codes.Unimplemented, "method GetTaskMetrics not implemented") } -func (UnimplementedAsyncAgentServiceServer) GetTaskLogs(context.Context, *admin.GetTaskLogsRequest) (*admin.GetTaskLogsResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method GetTaskLogs not implemented") +func (UnimplementedAsyncAgentServiceServer) GetTaskLogs(*admin.GetTaskLogsRequest, AsyncAgentService_GetTaskLogsServer) error { + return status.Errorf(codes.Unimplemented, "method GetTaskLogs not implemented") } // UnsafeAsyncAgentServiceServer may be embedded to opt out of forward compatibility for this service. @@ -223,22 +368,25 @@ func _AsyncAgentService_GetTaskMetrics_Handler(srv interface{}, ctx context.Cont return interceptor(ctx, in, info, handler) } -func _AsyncAgentService_GetTaskLogs_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(admin.GetTaskLogsRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(AsyncAgentServiceServer).GetTaskLogs(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: AsyncAgentService_GetTaskLogs_FullMethodName, +func _AsyncAgentService_GetTaskLogs_Handler(srv interface{}, stream grpc.ServerStream) error { + m := new(admin.GetTaskLogsRequest) + if err := stream.RecvMsg(m); err != nil { + return err } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AsyncAgentServiceServer).GetTaskLogs(ctx, req.(*admin.GetTaskLogsRequest)) - } - return interceptor(ctx, in, info, handler) + return srv.(AsyncAgentServiceServer).GetTaskLogs(m, &asyncAgentServiceGetTaskLogsServer{stream}) +} + +type AsyncAgentService_GetTaskLogsServer interface { + Send(*admin.GetTaskLogsResponse) error + grpc.ServerStream +} + +type asyncAgentServiceGetTaskLogsServer struct { + grpc.ServerStream +} + +func (x *asyncAgentServiceGetTaskLogsServer) Send(m *admin.GetTaskLogsResponse) error { + return x.ServerStream.SendMsg(m) } // AsyncAgentService_ServiceDesc is the grpc.ServiceDesc for AsyncAgentService service. @@ -264,12 +412,14 @@ var AsyncAgentService_ServiceDesc = grpc.ServiceDesc{ MethodName: "GetTaskMetrics", Handler: _AsyncAgentService_GetTaskMetrics_Handler, }, + }, + Streams: []grpc.StreamDesc{ { - MethodName: "GetTaskLogs", - Handler: _AsyncAgentService_GetTaskLogs_Handler, + StreamName: "GetTaskLogs", + Handler: _AsyncAgentService_GetTaskLogs_Handler, + ServerStreams: true, }, }, - Streams: []grpc.StreamDesc{}, Metadata: "flyteidl/service/agent.proto", } diff --git a/flyteidl/gen/pb-go/gateway/flyteidl/service/admin.swagger.json b/flyteidl/gen/pb-go/gateway/flyteidl/service/admin.swagger.json index 73dbd3b8dc..191143dd89 100644 --- a/flyteidl/gen/pb-go/gateway/flyteidl/service/admin.swagger.json +++ b/flyteidl/gen/pb-go/gateway/flyteidl/service/admin.swagger.json @@ -4273,10 +4273,11 @@ "SYSTEM", "RELAUNCH", "CHILD_WORKFLOW", - "RECOVERED" + "RECOVERED", + "TRIGGER" ], "default": "MANUAL", - "description": "The method by which this execution was launched.\n\n - MANUAL: The default execution mode, MANUAL implies that an execution was launched by an individual.\n - SCHEDULED: A schedule triggered this execution launch.\n - SYSTEM: A system process was responsible for launching this execution rather an individual.\n - RELAUNCH: This execution was launched with identical inputs as a previous execution.\n - CHILD_WORKFLOW: This execution was triggered by another execution.\n - RECOVERED: This execution was recovered from another execution." + "description": "The method by which this execution was launched.\n\n - MANUAL: The default execution mode, MANUAL implies that an execution was launched by an individual.\n - SCHEDULED: A schedule triggered this execution launch.\n - SYSTEM: A system process was responsible for launching this execution rather an individual.\n - RELAUNCH: This execution was launched with identical inputs as a previous execution.\n - CHILD_WORKFLOW: This execution was triggered by another execution.\n - RECOVERED: This execution was recovered from another execution.\n - TRIGGER: Execution was kicked off by the artifact trigger system" }, "IOStrategyDownloadMode": { "type": "string", @@ -6474,10 +6475,6 @@ "coreArtifactBindingData": { "type": "object", "properties": { - "index": { - "type": "integer", - "format": "int64" - }, "partition_key": { "type": "string" }, @@ -8080,6 +8077,10 @@ "extended_resources": { "$ref": "#/definitions/coreExtendedResources", "description": "Overrides for all non-standard resources, not captured by\nv1.ResourceRequirements, to allocate to a task." + }, + "container_image": { + "type": "string", + "description": "Override for the image used by task pods." } }, "description": "Optional task node overrides that will be applied at task execution time." diff --git a/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.pb.gw.go b/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.pb.gw.go index a5fec7e0f7..06ae8800ab 100644 --- a/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.pb.gw.go +++ b/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.pb.gw.go @@ -33,6 +33,478 @@ var _ = runtime.String var _ = utilities.NewDoubleArray var _ = metadata.Join +func request_SyncAgentService_ExecuteTaskSync_0(ctx context.Context, marshaler runtime.Marshaler, client extService.SyncAgentServiceClient, req *http.Request, pathParams map[string]string) (extService.SyncAgentService_ExecuteTaskSyncClient, runtime.ServerMetadata, error) { + var metadata runtime.ServerMetadata + stream, err := client.ExecuteTaskSync(ctx) + if err != nil { + grpclog.Infof("Failed to start streaming: %v", err) + return nil, metadata, err + } + dec := marshaler.NewDecoder(req.Body) + handleSend := func() error { + var protoReq extAdmin.ExecuteTaskSyncRequest + err := dec.Decode(&protoReq) + if err == io.EOF { + return err + } + if err != nil { + grpclog.Infof("Failed to decode request: %v", err) + return err + } + if err := stream.Send(&protoReq); err != nil { + grpclog.Infof("Failed to send request: %v", err) + return err + } + return nil + } + go func() { + for { + if err := handleSend(); err != nil { + break + } + } + if err := stream.CloseSend(); err != nil { + grpclog.Infof("Failed to terminate client stream: %v", err) + } + }() + header, err := stream.Header() + if err != nil { + grpclog.Infof("Failed to get header from client: %v", err) + return nil, metadata, err + } + metadata.HeaderMD = header + return stream, metadata, nil +} + +func request_AsyncAgentService_CreateTask_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AsyncAgentServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.CreateTaskRequest + var metadata runtime.ServerMetadata + + newReader, berr := utilities.IOReaderFactory(req.Body) + if berr != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", berr) + } + if err := marshaler.NewDecoder(newReader()).Decode(&protoReq); err != nil && err != io.EOF { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := client.CreateTask(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err + +} + +func local_request_AsyncAgentService_CreateTask_0(ctx context.Context, marshaler runtime.Marshaler, server extService.AsyncAgentServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.CreateTaskRequest + var metadata runtime.ServerMetadata + + newReader, berr := utilities.IOReaderFactory(req.Body) + if berr != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", berr) + } + if err := marshaler.NewDecoder(newReader()).Decode(&protoReq); err != nil && err != io.EOF { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := server.CreateTask(ctx, &protoReq) + return msg, metadata, err + +} + +var ( + filter_AsyncAgentService_GetTask_0 = &utilities.DoubleArray{Encoding: map[string]int{"task_category": 0, "name": 1, "version": 2, "resource_meta": 3, "resourceMeta": 4}, Base: []int{1, 6, 5, 6, 7, 8, 2, 0, 4, 0, 0, 0, 0, 0}, Check: []int{0, 1, 1, 1, 1, 1, 2, 7, 2, 9, 3, 4, 5, 6}} +) + +func request_AsyncAgentService_GetTask_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AsyncAgentServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.GetTaskRequest + var metadata runtime.ServerMetadata + + var ( + val string + ok bool + err error + _ = err + ) + + val, ok = pathParams["task_category.name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.name") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.name", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.name", err) + } + + val, ok = pathParams["task_category.version"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.version") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.version", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.version", err) + } + + val, ok = pathParams["resource_meta"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "resource_meta") + } + + protoReq.ResourceMeta, err = runtime.Bytes(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "resource_meta", err) + } + + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AsyncAgentService_GetTask_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := client.GetTask(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err + +} + +func local_request_AsyncAgentService_GetTask_0(ctx context.Context, marshaler runtime.Marshaler, server extService.AsyncAgentServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.GetTaskRequest + var metadata runtime.ServerMetadata + + var ( + val string + ok bool + err error + _ = err + ) + + val, ok = pathParams["task_category.name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.name") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.name", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.name", err) + } + + val, ok = pathParams["task_category.version"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.version") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.version", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.version", err) + } + + val, ok = pathParams["resource_meta"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "resource_meta") + } + + protoReq.ResourceMeta, err = runtime.Bytes(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "resource_meta", err) + } + + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AsyncAgentService_GetTask_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := server.GetTask(ctx, &protoReq) + return msg, metadata, err + +} + +var ( + filter_AsyncAgentService_DeleteTask_0 = &utilities.DoubleArray{Encoding: map[string]int{"task_category": 0, "name": 1, "version": 2, "resource_meta": 3, "resourceMeta": 4}, Base: []int{1, 6, 5, 6, 7, 8, 2, 0, 4, 0, 0, 0, 0, 0}, Check: []int{0, 1, 1, 1, 1, 1, 2, 7, 2, 9, 3, 4, 5, 6}} +) + +func request_AsyncAgentService_DeleteTask_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AsyncAgentServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.DeleteTaskRequest + var metadata runtime.ServerMetadata + + var ( + val string + ok bool + err error + _ = err + ) + + val, ok = pathParams["task_category.name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.name") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.name", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.name", err) + } + + val, ok = pathParams["task_category.version"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.version") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.version", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.version", err) + } + + val, ok = pathParams["resource_meta"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "resource_meta") + } + + protoReq.ResourceMeta, err = runtime.Bytes(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "resource_meta", err) + } + + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AsyncAgentService_DeleteTask_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := client.DeleteTask(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err + +} + +func local_request_AsyncAgentService_DeleteTask_0(ctx context.Context, marshaler runtime.Marshaler, server extService.AsyncAgentServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.DeleteTaskRequest + var metadata runtime.ServerMetadata + + var ( + val string + ok bool + err error + _ = err + ) + + val, ok = pathParams["task_category.name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.name") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.name", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.name", err) + } + + val, ok = pathParams["task_category.version"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.version") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.version", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.version", err) + } + + val, ok = pathParams["resource_meta"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "resource_meta") + } + + protoReq.ResourceMeta, err = runtime.Bytes(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "resource_meta", err) + } + + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AsyncAgentService_DeleteTask_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := server.DeleteTask(ctx, &protoReq) + return msg, metadata, err + +} + +var ( + filter_AsyncAgentService_GetTaskMetrics_0 = &utilities.DoubleArray{Encoding: map[string]int{"task_category": 0, "name": 1, "version": 2, "resource_meta": 3, "resourceMeta": 4}, Base: []int{1, 6, 5, 6, 7, 8, 2, 0, 4, 0, 0, 0, 0, 0}, Check: []int{0, 1, 1, 1, 1, 1, 2, 7, 2, 9, 3, 4, 5, 6}} +) + +func request_AsyncAgentService_GetTaskMetrics_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AsyncAgentServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.GetTaskMetricsRequest + var metadata runtime.ServerMetadata + + var ( + val string + ok bool + err error + _ = err + ) + + val, ok = pathParams["task_category.name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.name") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.name", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.name", err) + } + + val, ok = pathParams["task_category.version"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.version") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.version", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.version", err) + } + + val, ok = pathParams["resource_meta"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "resource_meta") + } + + protoReq.ResourceMeta, err = runtime.Bytes(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "resource_meta", err) + } + + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AsyncAgentService_GetTaskMetrics_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := client.GetTaskMetrics(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err + +} + +func local_request_AsyncAgentService_GetTaskMetrics_0(ctx context.Context, marshaler runtime.Marshaler, server extService.AsyncAgentServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.GetTaskMetricsRequest + var metadata runtime.ServerMetadata + + var ( + val string + ok bool + err error + _ = err + ) + + val, ok = pathParams["task_category.name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.name") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.name", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.name", err) + } + + val, ok = pathParams["task_category.version"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.version") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.version", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.version", err) + } + + val, ok = pathParams["resource_meta"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "resource_meta") + } + + protoReq.ResourceMeta, err = runtime.Bytes(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "resource_meta", err) + } + + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AsyncAgentService_GetTaskMetrics_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + msg, err := server.GetTaskMetrics(ctx, &protoReq) + return msg, metadata, err + +} + +var ( + filter_AsyncAgentService_GetTaskLogs_0 = &utilities.DoubleArray{Encoding: map[string]int{"task_category": 0, "name": 1, "version": 2, "resource_meta": 3, "resourceMeta": 4}, Base: []int{1, 6, 5, 6, 7, 8, 2, 0, 4, 0, 0, 0, 0, 0}, Check: []int{0, 1, 1, 1, 1, 1, 2, 7, 2, 9, 3, 4, 5, 6}} +) + +func request_AsyncAgentService_GetTaskLogs_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AsyncAgentServiceClient, req *http.Request, pathParams map[string]string) (extService.AsyncAgentService_GetTaskLogsClient, runtime.ServerMetadata, error) { + var protoReq extAdmin.GetTaskLogsRequest + var metadata runtime.ServerMetadata + + var ( + val string + ok bool + err error + _ = err + ) + + val, ok = pathParams["task_category.name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.name") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.name", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.name", err) + } + + val, ok = pathParams["task_category.version"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "task_category.version") + } + + err = runtime.PopulateFieldFromPath(&protoReq, "task_category.version", val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "task_category.version", err) + } + + val, ok = pathParams["resource_meta"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "resource_meta") + } + + protoReq.ResourceMeta, err = runtime.Bytes(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "resource_meta", err) + } + + if err := req.ParseForm(); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_AsyncAgentService_GetTaskLogs_0); err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err) + } + + stream, err := client.GetTaskLogs(ctx, &protoReq) + if err != nil { + return nil, metadata, err + } + header, err := stream.Header() + if err != nil { + return nil, metadata, err + } + metadata.HeaderMD = header + return stream, metadata, nil + +} + func request_AgentMetadataService_GetAgent_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AgentMetadataServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { var protoReq extAdmin.GetAgentRequest var metadata runtime.ServerMetadata @@ -70,37 +542,169 @@ func local_request_AgentMetadataService_GetAgent_0(ctx context.Context, marshale _ = err ) - val, ok = pathParams["name"] - if !ok { - return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "name") - } + val, ok = pathParams["name"] + if !ok { + return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "name") + } + + protoReq.Name, err = runtime.String(val) + if err != nil { + return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "name", err) + } + + msg, err := server.GetAgent(ctx, &protoReq) + return msg, metadata, err + +} + +func request_AgentMetadataService_ListAgents_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AgentMetadataServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.ListAgentsRequest + var metadata runtime.ServerMetadata + + msg, err := client.ListAgents(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) + return msg, metadata, err + +} + +func local_request_AgentMetadataService_ListAgents_0(ctx context.Context, marshaler runtime.Marshaler, server extService.AgentMetadataServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { + var protoReq extAdmin.ListAgentsRequest + var metadata runtime.ServerMetadata + + msg, err := server.ListAgents(ctx, &protoReq) + return msg, metadata, err + +} + +// RegisterSyncAgentServiceHandlerServer registers the http handlers for service SyncAgentService to "mux". +// UnaryRPC :call SyncAgentServiceServer directly. +// StreamingRPC :currently unsupported pending https://github.com/grpc/grpc-go/issues/906. +// Note that using this registration option will cause many gRPC library features to stop working. Consider using RegisterSyncAgentServiceHandlerFromEndpoint instead. +func RegisterSyncAgentServiceHandlerServer(ctx context.Context, mux *runtime.ServeMux, server extService.SyncAgentServiceServer) error { + + mux.Handle("POST", pattern_SyncAgentService_ExecuteTaskSync_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + err := status.Error(codes.Unimplemented, "streaming calls are not yet supported in the in-process transport") + _, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + }) + + return nil +} + +// RegisterAsyncAgentServiceHandlerServer registers the http handlers for service AsyncAgentService to "mux". +// UnaryRPC :call AsyncAgentServiceServer directly. +// StreamingRPC :currently unsupported pending https://github.com/grpc/grpc-go/issues/906. +// Note that using this registration option will cause many gRPC library features to stop working. Consider using RegisterAsyncAgentServiceHandlerFromEndpoint instead. +func RegisterAsyncAgentServiceHandlerServer(ctx context.Context, mux *runtime.ServeMux, server extService.AsyncAgentServiceServer) error { + + mux.Handle("POST", pattern_AsyncAgentService_CreateTask_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateIncomingContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/CreateTask", runtime.WithHTTPPathPattern("/api/v1/agent/task")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AsyncAgentService_CreateTask_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_AsyncAgentService_CreateTask_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + + }) + + mux.Handle("GET", pattern_AsyncAgentService_GetTask_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateIncomingContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/GetTask", runtime.WithHTTPPathPattern("/api/v1/agent/task/{task_category.name}/{task_category.version}/{resource_meta}")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AsyncAgentService_GetTask_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_AsyncAgentService_GetTask_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) - protoReq.Name, err = runtime.String(val) - if err != nil { - return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "name", err) - } + }) - msg, err := server.GetAgent(ctx, &protoReq) - return msg, metadata, err + mux.Handle("DELETE", pattern_AsyncAgentService_DeleteTask_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateIncomingContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/DeleteTask", runtime.WithHTTPPathPattern("/api/v1/agent/task_executions/{task_category.name}/{task_category.version}/{resource_meta}")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AsyncAgentService_DeleteTask_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } -} + forward_AsyncAgentService_DeleteTask_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) -func request_AgentMetadataService_ListAgents_0(ctx context.Context, marshaler runtime.Marshaler, client extService.AgentMetadataServiceClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { - var protoReq extAdmin.ListAgentsRequest - var metadata runtime.ServerMetadata + }) - msg, err := client.ListAgents(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD)) - return msg, metadata, err + mux.Handle("GET", pattern_AsyncAgentService_GetTaskMetrics_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + var stream runtime.ServerTransportStream + ctx = grpc.NewContextWithServerTransportStream(ctx, &stream) + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateIncomingContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/GetTaskMetrics", runtime.WithHTTPPathPattern("/api/v1/agent/task/metrics/{task_category.name}/{task_category.version}/{resource_meta}")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := local_request_AsyncAgentService_GetTaskMetrics_0(annotatedContext, inboundMarshaler, server, req, pathParams) + md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer()) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } -} + forward_AsyncAgentService_GetTaskMetrics_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) -func local_request_AgentMetadataService_ListAgents_0(ctx context.Context, marshaler runtime.Marshaler, server extService.AgentMetadataServiceServer, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) { - var protoReq extAdmin.ListAgentsRequest - var metadata runtime.ServerMetadata + }) - msg, err := server.ListAgents(ctx, &protoReq) - return msg, metadata, err + mux.Handle("GET", pattern_AsyncAgentService_GetTaskLogs_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + err := status.Error(codes.Unimplemented, "streaming calls are not yet supported in the in-process transport") + _, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + }) + return nil } // RegisterAgentMetadataServiceHandlerServer registers the http handlers for service AgentMetadataService to "mux". @@ -162,6 +766,252 @@ func RegisterAgentMetadataServiceHandlerServer(ctx context.Context, mux *runtime return nil } +// RegisterSyncAgentServiceHandlerFromEndpoint is same as RegisterSyncAgentServiceHandler but +// automatically dials to "endpoint" and closes the connection when "ctx" gets done. +func RegisterSyncAgentServiceHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error) { + conn, err := grpc.DialContext(ctx, endpoint, opts...) + if err != nil { + return err + } + defer func() { + if err != nil { + if cerr := conn.Close(); cerr != nil { + grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr) + } + return + } + go func() { + <-ctx.Done() + if cerr := conn.Close(); cerr != nil { + grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr) + } + }() + }() + + return RegisterSyncAgentServiceHandler(ctx, mux, conn) +} + +// RegisterSyncAgentServiceHandler registers the http handlers for service SyncAgentService to "mux". +// The handlers forward requests to the grpc endpoint over "conn". +func RegisterSyncAgentServiceHandler(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error { + return RegisterSyncAgentServiceHandlerClient(ctx, mux, extService.NewSyncAgentServiceClient(conn)) +} + +// RegisterSyncAgentServiceHandlerClient registers the http handlers for service SyncAgentService +// to "mux". The handlers forward requests to the grpc endpoint over the given implementation of "extService.SyncAgentServiceClient". +// Note: the gRPC framework executes interceptors within the gRPC handler. If the passed in "extService.SyncAgentServiceClient" +// doesn't go through the normal gRPC flow (creating a gRPC client etc.) then it will be up to the passed in +// "extService.SyncAgentServiceClient" to call the correct interceptors. +func RegisterSyncAgentServiceHandlerClient(ctx context.Context, mux *runtime.ServeMux, client extService.SyncAgentServiceClient) error { + + mux.Handle("POST", pattern_SyncAgentService_ExecuteTaskSync_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateContext(ctx, mux, req, "/flyteidl.service.SyncAgentService/ExecuteTaskSync", runtime.WithHTTPPathPattern("/api/v1/agent/task/stream")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_SyncAgentService_ExecuteTaskSync_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_SyncAgentService_ExecuteTaskSync_0(annotatedContext, mux, outboundMarshaler, w, req, func() (proto.Message, error) { return resp.Recv() }, mux.GetForwardResponseOptions()...) + + }) + + return nil +} + +var ( + pattern_SyncAgentService_ExecuteTaskSync_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 2, 3, 2, 4}, []string{"api", "v1", "agent", "task", "stream"}, "")) +) + +var ( + forward_SyncAgentService_ExecuteTaskSync_0 = runtime.ForwardResponseStream +) + +// RegisterAsyncAgentServiceHandlerFromEndpoint is same as RegisterAsyncAgentServiceHandler but +// automatically dials to "endpoint" and closes the connection when "ctx" gets done. +func RegisterAsyncAgentServiceHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error) { + conn, err := grpc.DialContext(ctx, endpoint, opts...) + if err != nil { + return err + } + defer func() { + if err != nil { + if cerr := conn.Close(); cerr != nil { + grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr) + } + return + } + go func() { + <-ctx.Done() + if cerr := conn.Close(); cerr != nil { + grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr) + } + }() + }() + + return RegisterAsyncAgentServiceHandler(ctx, mux, conn) +} + +// RegisterAsyncAgentServiceHandler registers the http handlers for service AsyncAgentService to "mux". +// The handlers forward requests to the grpc endpoint over "conn". +func RegisterAsyncAgentServiceHandler(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error { + return RegisterAsyncAgentServiceHandlerClient(ctx, mux, extService.NewAsyncAgentServiceClient(conn)) +} + +// RegisterAsyncAgentServiceHandlerClient registers the http handlers for service AsyncAgentService +// to "mux". The handlers forward requests to the grpc endpoint over the given implementation of "extService.AsyncAgentServiceClient". +// Note: the gRPC framework executes interceptors within the gRPC handler. If the passed in "extService.AsyncAgentServiceClient" +// doesn't go through the normal gRPC flow (creating a gRPC client etc.) then it will be up to the passed in +// "extService.AsyncAgentServiceClient" to call the correct interceptors. +func RegisterAsyncAgentServiceHandlerClient(ctx context.Context, mux *runtime.ServeMux, client extService.AsyncAgentServiceClient) error { + + mux.Handle("POST", pattern_AsyncAgentService_CreateTask_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/CreateTask", runtime.WithHTTPPathPattern("/api/v1/agent/task")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AsyncAgentService_CreateTask_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_AsyncAgentService_CreateTask_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + + }) + + mux.Handle("GET", pattern_AsyncAgentService_GetTask_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/GetTask", runtime.WithHTTPPathPattern("/api/v1/agent/task/{task_category.name}/{task_category.version}/{resource_meta}")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AsyncAgentService_GetTask_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_AsyncAgentService_GetTask_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + + }) + + mux.Handle("DELETE", pattern_AsyncAgentService_DeleteTask_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/DeleteTask", runtime.WithHTTPPathPattern("/api/v1/agent/task_executions/{task_category.name}/{task_category.version}/{resource_meta}")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AsyncAgentService_DeleteTask_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_AsyncAgentService_DeleteTask_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + + }) + + mux.Handle("GET", pattern_AsyncAgentService_GetTaskMetrics_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/GetTaskMetrics", runtime.WithHTTPPathPattern("/api/v1/agent/task/metrics/{task_category.name}/{task_category.version}/{resource_meta}")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AsyncAgentService_GetTaskMetrics_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_AsyncAgentService_GetTaskMetrics_0(annotatedContext, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...) + + }) + + mux.Handle("GET", pattern_AsyncAgentService_GetTaskLogs_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) { + ctx, cancel := context.WithCancel(req.Context()) + defer cancel() + inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req) + var err error + var annotatedContext context.Context + annotatedContext, err = runtime.AnnotateContext(ctx, mux, req, "/flyteidl.service.AsyncAgentService/GetTaskLogs", runtime.WithHTTPPathPattern("/api/v1/agent/task/logs/{task_category.name}/{task_category.version}/{resource_meta}")) + if err != nil { + runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err) + return + } + resp, md, err := request_AsyncAgentService_GetTaskLogs_0(annotatedContext, inboundMarshaler, client, req, pathParams) + annotatedContext = runtime.NewServerMetadataContext(annotatedContext, md) + if err != nil { + runtime.HTTPError(annotatedContext, mux, outboundMarshaler, w, req, err) + return + } + + forward_AsyncAgentService_GetTaskLogs_0(annotatedContext, mux, outboundMarshaler, w, req, func() (proto.Message, error) { return resp.Recv() }, mux.GetForwardResponseOptions()...) + + }) + + return nil +} + +var ( + pattern_AsyncAgentService_CreateTask_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 2, 3}, []string{"api", "v1", "agent", "task"}, "")) + + pattern_AsyncAgentService_GetTask_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 2, 3, 1, 0, 4, 1, 5, 4, 1, 0, 4, 1, 5, 5, 1, 0, 4, 1, 5, 6}, []string{"api", "v1", "agent", "task", "task_category.name", "task_category.version", "resource_meta"}, "")) + + pattern_AsyncAgentService_DeleteTask_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 2, 3, 1, 0, 4, 1, 5, 4, 1, 0, 4, 1, 5, 5, 1, 0, 4, 1, 5, 6}, []string{"api", "v1", "agent", "task_executions", "task_category.name", "task_category.version", "resource_meta"}, "")) + + pattern_AsyncAgentService_GetTaskMetrics_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 2, 3, 2, 4, 1, 0, 4, 1, 5, 5, 1, 0, 4, 1, 5, 6, 1, 0, 4, 1, 5, 7}, []string{"api", "v1", "agent", "task", "metrics", "task_category.name", "task_category.version", "resource_meta"}, "")) + + pattern_AsyncAgentService_GetTaskLogs_0 = runtime.MustPattern(runtime.NewPattern(1, []int{2, 0, 2, 1, 2, 2, 2, 3, 2, 4, 1, 0, 4, 1, 5, 5, 1, 0, 4, 1, 5, 6, 1, 0, 4, 1, 5, 7}, []string{"api", "v1", "agent", "task", "logs", "task_category.name", "task_category.version", "resource_meta"}, "")) +) + +var ( + forward_AsyncAgentService_CreateTask_0 = runtime.ForwardResponseMessage + + forward_AsyncAgentService_GetTask_0 = runtime.ForwardResponseMessage + + forward_AsyncAgentService_DeleteTask_0 = runtime.ForwardResponseMessage + + forward_AsyncAgentService_GetTaskMetrics_0 = runtime.ForwardResponseMessage + + forward_AsyncAgentService_GetTaskLogs_0 = runtime.ForwardResponseStream +) + // RegisterAgentMetadataServiceHandlerFromEndpoint is same as RegisterAgentMetadataServiceHandler but // automatically dials to "endpoint" and closes the connection when "ctx" gets done. func RegisterAgentMetadataServiceHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error) { diff --git a/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.swagger.json b/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.swagger.json index 5ce68f3e61..ff115cad21 100644 --- a/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.swagger.json +++ b/flyteidl/gen/pb-go/gateway/flyteidl/service/agent.swagger.json @@ -5,6 +5,9 @@ "version": "version not set" }, "tags": [ + { + "name": "SyncAgentService" + }, { "name": "AsyncAgentService" }, @@ -19,6 +22,362 @@ "application/json" ], "paths": { + "/api/v1/agent/task": { + "post": { + "summary": "CreateTask sends a task create request to the agent service.", + "operationId": "AsyncAgentService_CreateTask", + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "$ref": "#/definitions/adminCreateTaskResponse" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "$ref": "#/definitions/googlerpcStatus" + } + } + }, + "parameters": [ + { + "name": "body", + "description": "Represents a request structure to create task.", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/adminCreateTaskRequest" + } + } + ], + "tags": [ + "AsyncAgentService" + ] + } + }, + "/api/v1/agent/task/logs/{task_category.name}/{task_category.version}/{resource_meta}": { + "get": { + "summary": "GetTaskLogs returns task execution logs, if available.", + "operationId": "AsyncAgentService_GetTaskLogs", + "responses": { + "200": { + "description": "A successful response.(streaming responses)", + "schema": { + "type": "object", + "properties": { + "result": { + "$ref": "#/definitions/adminGetTaskLogsResponse" + }, + "error": { + "$ref": "#/definitions/googlerpcStatus" + } + }, + "title": "Stream result of adminGetTaskLogsResponse" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "$ref": "#/definitions/googlerpcStatus" + } + } + }, + "parameters": [ + { + "name": "task_category.name", + "description": "The name of the task type.", + "in": "path", + "required": true, + "type": "string" + }, + { + "name": "task_category.version", + "description": "The version of the task type.", + "in": "path", + "required": true, + "type": "integer", + "format": "int32" + }, + { + "name": "resource_meta", + "description": "Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata).", + "in": "path", + "required": true, + "type": "string", + "format": "byte" + }, + { + "name": "task_type", + "description": "A predefined yet extensible Task type identifier.", + "in": "query", + "required": false, + "type": "string" + }, + { + "name": "lines", + "description": "Number of lines to return.", + "in": "query", + "required": false, + "type": "string", + "format": "uint64" + }, + { + "name": "token", + "description": "In the case of multiple pages of results, the server-provided token can be used to fetch the next page\nin a query. If there are no more results, this value will be empty.", + "in": "query", + "required": false, + "type": "string" + } + ], + "tags": [ + "AsyncAgentService" + ] + } + }, + "/api/v1/agent/task/metrics/{task_category.name}/{task_category.version}/{resource_meta}": { + "get": { + "summary": "GetTaskMetrics returns one or more task execution metrics, if available.", + "description": "Errors include\n * OutOfRange if metrics are not available for the specified task time range\n * various other errors", + "operationId": "AsyncAgentService_GetTaskMetrics", + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "$ref": "#/definitions/adminGetTaskMetricsResponse" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "$ref": "#/definitions/googlerpcStatus" + } + } + }, + "parameters": [ + { + "name": "task_category.name", + "description": "The name of the task type.", + "in": "path", + "required": true, + "type": "string" + }, + { + "name": "task_category.version", + "description": "The version of the task type.", + "in": "path", + "required": true, + "type": "integer", + "format": "int32" + }, + { + "name": "resource_meta", + "description": "Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata).", + "in": "path", + "required": true, + "type": "string", + "format": "byte" + }, + { + "name": "task_type", + "description": "A predefined yet extensible Task type identifier.", + "in": "query", + "required": false, + "type": "string" + }, + { + "name": "queries", + "description": "The metrics to query. If empty, will return a default set of metrics.\ne.g. EXECUTION_METRIC_USED_CPU_AVG or EXECUTION_METRIC_USED_MEMORY_BYTES_AVG", + "in": "query", + "required": false, + "type": "array", + "items": { + "type": "string" + }, + "collectionFormat": "multi" + }, + { + "name": "start_time", + "description": "Start timestamp, inclusive.", + "in": "query", + "required": false, + "type": "string", + "format": "date-time" + }, + { + "name": "end_time", + "description": "End timestamp, inclusive..", + "in": "query", + "required": false, + "type": "string", + "format": "date-time" + }, + { + "name": "step", + "description": "Query resolution step width in duration format or float number of seconds.", + "in": "query", + "required": false, + "type": "string" + } + ], + "tags": [ + "AsyncAgentService" + ] + } + }, + "/api/v1/agent/task/stream": { + "post": { + "summary": "ExecuteTaskSync streams the create request and inputs to the agent service and streams the outputs back.", + "operationId": "SyncAgentService_ExecuteTaskSync", + "responses": { + "200": { + "description": "A successful response.(streaming responses)", + "schema": { + "type": "object", + "properties": { + "result": { + "$ref": "#/definitions/adminExecuteTaskSyncResponse" + }, + "error": { + "$ref": "#/definitions/googlerpcStatus" + } + }, + "title": "Stream result of adminExecuteTaskSyncResponse" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "$ref": "#/definitions/googlerpcStatus" + } + } + }, + "parameters": [ + { + "name": "body", + "description": " (streaming inputs)", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/adminExecuteTaskSyncRequest" + } + } + ], + "tags": [ + "SyncAgentService" + ] + } + }, + "/api/v1/agent/task/{task_category.name}/{task_category.version}/{resource_meta}": { + "get": { + "summary": "Get job status.", + "operationId": "AsyncAgentService_GetTask", + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "$ref": "#/definitions/adminGetTaskResponse" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "$ref": "#/definitions/googlerpcStatus" + } + } + }, + "parameters": [ + { + "name": "task_category.name", + "description": "The name of the task type.", + "in": "path", + "required": true, + "type": "string" + }, + { + "name": "task_category.version", + "description": "The version of the task type.", + "in": "path", + "required": true, + "type": "integer", + "format": "int32" + }, + { + "name": "resource_meta", + "description": "Metadata about the resource to be pass to the agent.", + "in": "path", + "required": true, + "type": "string", + "format": "byte" + }, + { + "name": "task_type", + "description": "A predefined yet extensible Task type identifier.", + "in": "query", + "required": false, + "type": "string" + } + ], + "tags": [ + "AsyncAgentService" + ] + } + }, + "/api/v1/agent/task_executions/{task_category.name}/{task_category.version}/{resource_meta}": { + "delete": { + "summary": "Delete the task resource.", + "operationId": "AsyncAgentService_DeleteTask", + "responses": { + "200": { + "description": "A successful response.", + "schema": { + "$ref": "#/definitions/adminDeleteTaskResponse" + } + }, + "default": { + "description": "An unexpected error response.", + "schema": { + "$ref": "#/definitions/googlerpcStatus" + } + } + }, + "parameters": [ + { + "name": "task_category.name", + "description": "The name of the task type.", + "in": "path", + "required": true, + "type": "string" + }, + { + "name": "task_category.version", + "description": "The version of the task type.", + "in": "path", + "required": true, + "type": "integer", + "format": "int32" + }, + { + "name": "resource_meta", + "description": "Metadata about the resource to be pass to the agent.", + "in": "path", + "required": true, + "type": "string", + "format": "byte" + }, + { + "name": "task_type", + "description": "A predefined yet extensible Task type identifier.", + "in": "query", + "required": false, + "type": "string" + } + ], + "tags": [ + "AsyncAgentService" + ] + } + }, "/api/v1/agent/{name}": { "get": { "summary": "Fetch a :ref:`ref_flyteidl.admin.Agent` definition.", @@ -245,19 +604,73 @@ "type": "string" }, "description": "SupportedTaskTypes are the types of the tasks that the agent can handle." + }, + "is_sync": { + "type": "boolean", + "description": "IsSync indicates whether this agent is a sync agent. Sync agents are expected to return their\nresults synchronously when called by propeller. Given that sync agents can affect the performance\nof the system, it's important to enforce strict timeout policies.\nAn Async agent, on the other hand, is required to be able to identify jobs by an\nidentifier and query for job statuses as jobs progress." + }, + "supported_task_categories": { + "type": "array", + "items": { + "type": "object", + "$ref": "#/definitions/adminTaskCategory" + }, + "description": "Supported_task_categories are the categories of the tasks that the agent can handle." } }, "description": "A message containing the agent metadata." }, + "adminCreateRequestHeader": { + "type": "object", + "properties": { + "template": { + "$ref": "#/definitions/coreTaskTemplate", + "description": "Template of the task that encapsulates all the metadata of the task." + }, + "output_prefix": { + "type": "string", + "title": "Prefix for where task output data will be written. (e.g. s3://my-bucket/randomstring)" + }, + "task_execution_metadata": { + "$ref": "#/definitions/flyteidladminTaskExecutionMetadata", + "description": "subset of runtime task execution metadata." + }, + "max_dataset_size_bytes": { + "type": "string", + "format": "int64", + "description": "MaxDatasetSizeBytes is the maximum size of the dataset that can be generated by the task." + } + } + }, + "adminCreateTaskRequest": { + "type": "object", + "properties": { + "inputs": { + "$ref": "#/definitions/coreLiteralMap", + "title": "The inputs required to start the execution. All required inputs must be\nincluded in this map. If not required and not provided, defaults apply.\n+optional" + }, + "template": { + "$ref": "#/definitions/coreTaskTemplate", + "description": "Template of the task that encapsulates all the metadata of the task." + }, + "output_prefix": { + "type": "string", + "title": "Prefix for where task output data will be written. (e.g. s3://my-bucket/randomstring)" + }, + "task_execution_metadata": { + "$ref": "#/definitions/flyteidladminTaskExecutionMetadata", + "description": "subset of runtime task execution metadata." + } + }, + "description": "Represents a request structure to create task." + }, "adminCreateTaskResponse": { "type": "object", "properties": { "resource_meta": { "type": "string", - "format": "byte" - }, - "resource": { - "$ref": "#/definitions/adminResource" + "format": "byte", + "description": "ResourceMeta is created by the agent. It could be a string (jobId) or a dict (more complex metadata)." } }, "description": "Represents a create response structure." @@ -266,6 +679,36 @@ "type": "object", "description": "Response to delete a task." }, + "adminExecuteTaskSyncRequest": { + "type": "object", + "properties": { + "header": { + "$ref": "#/definitions/adminCreateRequestHeader" + }, + "inputs": { + "$ref": "#/definitions/coreLiteralMap" + } + } + }, + "adminExecuteTaskSyncResponse": { + "type": "object", + "properties": { + "header": { + "$ref": "#/definitions/adminExecuteTaskSyncResponseHeader" + }, + "outputs": { + "$ref": "#/definitions/coreLiteralMap" + } + } + }, + "adminExecuteTaskSyncResponseHeader": { + "type": "object", + "properties": { + "resource": { + "$ref": "#/definitions/adminResource" + } + } + }, "adminGetAgentResponse": { "type": "object", "properties": { @@ -276,6 +719,18 @@ "description": "A response containing an agent." }, "adminGetTaskLogsResponse": { + "type": "object", + "properties": { + "header": { + "$ref": "#/definitions/adminGetTaskLogsResponseHeader" + }, + "body": { + "$ref": "#/definitions/adminGetTaskLogsResponseBody" + } + }, + "description": "A response containing the logs for a task execution." + }, + "adminGetTaskLogsResponseBody": { "type": "object", "properties": { "results": { @@ -284,13 +739,17 @@ "type": "string" }, "description": "The execution log results." - }, + } + } + }, + "adminGetTaskLogsResponseHeader": { + "type": "object", + "properties": { "token": { "type": "string", "description": "In the case of multiple pages of results, the server-provided token can be used to fetch the next page\nin a query. If there are no more results, this value will be empty." } - }, - "description": "A response containing the logs for a task execution." + } }, "adminGetTaskMetricsResponse": { "type": "object", @@ -311,14 +770,6 @@ "properties": { "resource": { "$ref": "#/definitions/adminResource" - }, - "log_links": { - "type": "array", - "items": { - "type": "object", - "$ref": "#/definitions/coreTaskLog" - }, - "title": "log information for the task execution" } }, "description": "Response to get an individual task resource." @@ -362,16 +813,30 @@ "phase": { "$ref": "#/definitions/coreTaskExecutionPhase", "description": "The phase of the execution is used to determine the phase of the plugin's execution." + }, + "custom_info": { + "type": "object", + "description": "Custom data specific to the agent." } } }, - "coreArtifactBindingData": { + "adminTaskCategory": { "type": "object", "properties": { - "index": { - "type": "integer", - "format": "int64" + "name": { + "type": "string", + "description": "The name of the task type." }, + "version": { + "type": "integer", + "format": "int32", + "description": "The version of the task type." + } + } + }, + "coreArtifactBindingData": { + "type": "object", + "properties": { "partition_key": { "type": "string" }, @@ -1304,6 +1769,24 @@ }, "title": "Task Metadata" }, + "coreTaskNodeOverrides": { + "type": "object", + "properties": { + "resources": { + "$ref": "#/definitions/coreResources", + "description": "A customizable interface to convey resources requested for a task container." + }, + "extended_resources": { + "$ref": "#/definitions/coreExtendedResources", + "description": "Overrides for all non-standard resources, not captured by\nv1.ResourceRequirements, to allocate to a task." + }, + "container_image": { + "type": "string", + "description": "Override for the image used by task pods." + } + }, + "description": "Optional task node overrides that will be applied at task execution time." + }, "coreTaskTemplate": { "type": "object", "properties": { @@ -1539,6 +2022,24 @@ "type": "string" }, "title": "Environment variables attached to the task execution" + }, + "max_attempts": { + "type": "integer", + "format": "int32", + "description": "Represents the maximum number of attempts allowed for a task.\nIf a task fails, it can be retried up to this maximum number of attempts." + }, + "interruptible": { + "type": "boolean", + "description": "Indicates whether the task execution can be interrupted.\nIf set to true, the task can be stopped before completion." + }, + "interruptible_failure_threshold": { + "type": "integer", + "format": "int32", + "description": "Specifies the threshold for failure count at which the interruptible property\nwill take effect. If the number of consecutive task failures exceeds this threshold,\ninterruptible behavior will be activated." + }, + "overrides": { + "$ref": "#/definitions/coreTaskNodeOverrides", + "description": "Overrides for specific properties of the task node.\nThese overrides can be used to customize the behavior of the task node." } }, "description": "Represents a subset of runtime task execution metadata that are relevant to external plugins." diff --git a/flyteidl/gen/pb-go/gateway/flyteidl/service/external_plugin_service.swagger.json b/flyteidl/gen/pb-go/gateway/flyteidl/service/external_plugin_service.swagger.json index 5dd386c39c..eda2434bcd 100644 --- a/flyteidl/gen/pb-go/gateway/flyteidl/service/external_plugin_service.swagger.json +++ b/flyteidl/gen/pb-go/gateway/flyteidl/service/external_plugin_service.swagger.json @@ -167,10 +167,6 @@ "coreArtifactBindingData": { "type": "object", "properties": { - "index": { - "type": "integer", - "format": "int64" - }, "partition_key": { "type": "string" }, diff --git a/flyteidl/gen/pb-js/flyteidl.d.ts b/flyteidl/gen/pb-js/flyteidl.d.ts index 87de89064b..c876340a61 100644 --- a/flyteidl/gen/pb-js/flyteidl.d.ts +++ b/flyteidl/gen/pb-js/flyteidl.d.ts @@ -78,9 +78,6 @@ export namespace flyteidl { /** Properties of an ArtifactBindingData. */ interface IArtifactBindingData { - /** ArtifactBindingData index */ - index?: (number|null); - /** ArtifactBindingData partitionKey */ partitionKey?: (string|null); @@ -100,9 +97,6 @@ export namespace flyteidl { */ constructor(properties?: flyteidl.core.IArtifactBindingData); - /** ArtifactBindingData index. */ - public index: number; - /** ArtifactBindingData partitionKey. */ public partitionKey: string; @@ -4935,6 +4929,9 @@ export namespace flyteidl { /** TaskNodeOverrides extendedResources */ extendedResources?: (flyteidl.core.IExtendedResources|null); + + /** TaskNodeOverrides containerImage */ + containerImage?: (string|null); } /** Represents a TaskNodeOverrides. */ @@ -4952,6 +4949,9 @@ export namespace flyteidl { /** TaskNodeOverrides extendedResources. */ public extendedResources?: (flyteidl.core.IExtendedResources|null); + /** TaskNodeOverrides containerImage. */ + public containerImage: string; + /** * Creates a new TaskNodeOverrides instance using the specified properties. * @param [properties] Properties to set @@ -9012,6 +9012,18 @@ export namespace flyteidl { /** TaskExecutionMetadata environmentVariables */ environmentVariables?: ({ [k: string]: string }|null); + + /** TaskExecutionMetadata maxAttempts */ + maxAttempts?: (number|null); + + /** TaskExecutionMetadata interruptible */ + interruptible?: (boolean|null); + + /** TaskExecutionMetadata interruptibleFailureThreshold */ + interruptibleFailureThreshold?: (number|null); + + /** TaskExecutionMetadata overrides */ + overrides?: (flyteidl.core.ITaskNodeOverrides|null); } /** Represents a TaskExecutionMetadata. */ @@ -9041,6 +9053,18 @@ export namespace flyteidl { /** TaskExecutionMetadata environmentVariables. */ public environmentVariables: { [k: string]: string }; + /** TaskExecutionMetadata maxAttempts. */ + public maxAttempts: number; + + /** TaskExecutionMetadata interruptible. */ + public interruptible: boolean; + + /** TaskExecutionMetadata interruptibleFailureThreshold. */ + public interruptibleFailureThreshold: number; + + /** TaskExecutionMetadata overrides. */ + public overrides?: (flyteidl.core.ITaskNodeOverrides|null); + /** * Creates a new TaskExecutionMetadata instance using the specified properties. * @param [properties] Properties to set @@ -9149,9 +9173,6 @@ export namespace flyteidl { /** CreateTaskResponse resourceMeta */ resourceMeta?: (Uint8Array|null); - - /** CreateTaskResponse resource */ - resource?: (flyteidl.admin.IResource|null); } /** Represents a CreateTaskResponse. */ @@ -9166,12 +9187,6 @@ export namespace flyteidl { /** CreateTaskResponse resourceMeta. */ public resourceMeta: Uint8Array; - /** CreateTaskResponse resource. */ - public resource?: (flyteidl.admin.IResource|null); - - /** CreateTaskResponse res. */ - public res?: ("resourceMeta"|"resource"); - /** * Creates a new CreateTaskResponse instance using the specified properties. * @param [properties] Properties to set @@ -9205,6 +9220,250 @@ export namespace flyteidl { public static verify(message: { [k: string]: any }): (string|null); } + /** Properties of a CreateRequestHeader. */ + interface ICreateRequestHeader { + + /** CreateRequestHeader template */ + template?: (flyteidl.core.ITaskTemplate|null); + + /** CreateRequestHeader outputPrefix */ + outputPrefix?: (string|null); + + /** CreateRequestHeader taskExecutionMetadata */ + taskExecutionMetadata?: (flyteidl.admin.ITaskExecutionMetadata|null); + + /** CreateRequestHeader maxDatasetSizeBytes */ + maxDatasetSizeBytes?: (Long|null); + } + + /** Represents a CreateRequestHeader. */ + class CreateRequestHeader implements ICreateRequestHeader { + + /** + * Constructs a new CreateRequestHeader. + * @param [properties] Properties to set + */ + constructor(properties?: flyteidl.admin.ICreateRequestHeader); + + /** CreateRequestHeader template. */ + public template?: (flyteidl.core.ITaskTemplate|null); + + /** CreateRequestHeader outputPrefix. */ + public outputPrefix: string; + + /** CreateRequestHeader taskExecutionMetadata. */ + public taskExecutionMetadata?: (flyteidl.admin.ITaskExecutionMetadata|null); + + /** CreateRequestHeader maxDatasetSizeBytes. */ + public maxDatasetSizeBytes: Long; + + /** + * Creates a new CreateRequestHeader instance using the specified properties. + * @param [properties] Properties to set + * @returns CreateRequestHeader instance + */ + public static create(properties?: flyteidl.admin.ICreateRequestHeader): flyteidl.admin.CreateRequestHeader; + + /** + * Encodes the specified CreateRequestHeader message. Does not implicitly {@link flyteidl.admin.CreateRequestHeader.verify|verify} messages. + * @param message CreateRequestHeader message or plain object to encode + * @param [writer] Writer to encode to + * @returns Writer + */ + public static encode(message: flyteidl.admin.ICreateRequestHeader, writer?: $protobuf.Writer): $protobuf.Writer; + + /** + * Decodes a CreateRequestHeader message from the specified reader or buffer. + * @param reader Reader or buffer to decode from + * @param [length] Message length if known beforehand + * @returns CreateRequestHeader + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + public static decode(reader: ($protobuf.Reader|Uint8Array), length?: number): flyteidl.admin.CreateRequestHeader; + + /** + * Verifies a CreateRequestHeader message. + * @param message Plain object to verify + * @returns `null` if valid, otherwise the reason why it is not + */ + public static verify(message: { [k: string]: any }): (string|null); + } + + /** Properties of an ExecuteTaskSyncRequest. */ + interface IExecuteTaskSyncRequest { + + /** ExecuteTaskSyncRequest header */ + header?: (flyteidl.admin.ICreateRequestHeader|null); + + /** ExecuteTaskSyncRequest inputs */ + inputs?: (flyteidl.core.ILiteralMap|null); + } + + /** Represents an ExecuteTaskSyncRequest. */ + class ExecuteTaskSyncRequest implements IExecuteTaskSyncRequest { + + /** + * Constructs a new ExecuteTaskSyncRequest. + * @param [properties] Properties to set + */ + constructor(properties?: flyteidl.admin.IExecuteTaskSyncRequest); + + /** ExecuteTaskSyncRequest header. */ + public header?: (flyteidl.admin.ICreateRequestHeader|null); + + /** ExecuteTaskSyncRequest inputs. */ + public inputs?: (flyteidl.core.ILiteralMap|null); + + /** ExecuteTaskSyncRequest part. */ + public part?: ("header"|"inputs"); + + /** + * Creates a new ExecuteTaskSyncRequest instance using the specified properties. + * @param [properties] Properties to set + * @returns ExecuteTaskSyncRequest instance + */ + public static create(properties?: flyteidl.admin.IExecuteTaskSyncRequest): flyteidl.admin.ExecuteTaskSyncRequest; + + /** + * Encodes the specified ExecuteTaskSyncRequest message. Does not implicitly {@link flyteidl.admin.ExecuteTaskSyncRequest.verify|verify} messages. + * @param message ExecuteTaskSyncRequest message or plain object to encode + * @param [writer] Writer to encode to + * @returns Writer + */ + public static encode(message: flyteidl.admin.IExecuteTaskSyncRequest, writer?: $protobuf.Writer): $protobuf.Writer; + + /** + * Decodes an ExecuteTaskSyncRequest message from the specified reader or buffer. + * @param reader Reader or buffer to decode from + * @param [length] Message length if known beforehand + * @returns ExecuteTaskSyncRequest + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + public static decode(reader: ($protobuf.Reader|Uint8Array), length?: number): flyteidl.admin.ExecuteTaskSyncRequest; + + /** + * Verifies an ExecuteTaskSyncRequest message. + * @param message Plain object to verify + * @returns `null` if valid, otherwise the reason why it is not + */ + public static verify(message: { [k: string]: any }): (string|null); + } + + /** Properties of an ExecuteTaskSyncResponseHeader. */ + interface IExecuteTaskSyncResponseHeader { + + /** ExecuteTaskSyncResponseHeader resource */ + resource?: (flyteidl.admin.IResource|null); + } + + /** Represents an ExecuteTaskSyncResponseHeader. */ + class ExecuteTaskSyncResponseHeader implements IExecuteTaskSyncResponseHeader { + + /** + * Constructs a new ExecuteTaskSyncResponseHeader. + * @param [properties] Properties to set + */ + constructor(properties?: flyteidl.admin.IExecuteTaskSyncResponseHeader); + + /** ExecuteTaskSyncResponseHeader resource. */ + public resource?: (flyteidl.admin.IResource|null); + + /** + * Creates a new ExecuteTaskSyncResponseHeader instance using the specified properties. + * @param [properties] Properties to set + * @returns ExecuteTaskSyncResponseHeader instance + */ + public static create(properties?: flyteidl.admin.IExecuteTaskSyncResponseHeader): flyteidl.admin.ExecuteTaskSyncResponseHeader; + + /** + * Encodes the specified ExecuteTaskSyncResponseHeader message. Does not implicitly {@link flyteidl.admin.ExecuteTaskSyncResponseHeader.verify|verify} messages. + * @param message ExecuteTaskSyncResponseHeader message or plain object to encode + * @param [writer] Writer to encode to + * @returns Writer + */ + public static encode(message: flyteidl.admin.IExecuteTaskSyncResponseHeader, writer?: $protobuf.Writer): $protobuf.Writer; + + /** + * Decodes an ExecuteTaskSyncResponseHeader message from the specified reader or buffer. + * @param reader Reader or buffer to decode from + * @param [length] Message length if known beforehand + * @returns ExecuteTaskSyncResponseHeader + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + public static decode(reader: ($protobuf.Reader|Uint8Array), length?: number): flyteidl.admin.ExecuteTaskSyncResponseHeader; + + /** + * Verifies an ExecuteTaskSyncResponseHeader message. + * @param message Plain object to verify + * @returns `null` if valid, otherwise the reason why it is not + */ + public static verify(message: { [k: string]: any }): (string|null); + } + + /** Properties of an ExecuteTaskSyncResponse. */ + interface IExecuteTaskSyncResponse { + + /** ExecuteTaskSyncResponse header */ + header?: (flyteidl.admin.IExecuteTaskSyncResponseHeader|null); + + /** ExecuteTaskSyncResponse outputs */ + outputs?: (flyteidl.core.ILiteralMap|null); + } + + /** Represents an ExecuteTaskSyncResponse. */ + class ExecuteTaskSyncResponse implements IExecuteTaskSyncResponse { + + /** + * Constructs a new ExecuteTaskSyncResponse. + * @param [properties] Properties to set + */ + constructor(properties?: flyteidl.admin.IExecuteTaskSyncResponse); + + /** ExecuteTaskSyncResponse header. */ + public header?: (flyteidl.admin.IExecuteTaskSyncResponseHeader|null); + + /** ExecuteTaskSyncResponse outputs. */ + public outputs?: (flyteidl.core.ILiteralMap|null); + + /** ExecuteTaskSyncResponse res. */ + public res?: ("header"|"outputs"); + + /** + * Creates a new ExecuteTaskSyncResponse instance using the specified properties. + * @param [properties] Properties to set + * @returns ExecuteTaskSyncResponse instance + */ + public static create(properties?: flyteidl.admin.IExecuteTaskSyncResponse): flyteidl.admin.ExecuteTaskSyncResponse; + + /** + * Encodes the specified ExecuteTaskSyncResponse message. Does not implicitly {@link flyteidl.admin.ExecuteTaskSyncResponse.verify|verify} messages. + * @param message ExecuteTaskSyncResponse message or plain object to encode + * @param [writer] Writer to encode to + * @returns Writer + */ + public static encode(message: flyteidl.admin.IExecuteTaskSyncResponse, writer?: $protobuf.Writer): $protobuf.Writer; + + /** + * Decodes an ExecuteTaskSyncResponse message from the specified reader or buffer. + * @param reader Reader or buffer to decode from + * @param [length] Message length if known beforehand + * @returns ExecuteTaskSyncResponse + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + public static decode(reader: ($protobuf.Reader|Uint8Array), length?: number): flyteidl.admin.ExecuteTaskSyncResponse; + + /** + * Verifies an ExecuteTaskSyncResponse message. + * @param message Plain object to verify + * @returns `null` if valid, otherwise the reason why it is not + */ + public static verify(message: { [k: string]: any }): (string|null); + } + /** Properties of a GetTaskRequest. */ interface IGetTaskRequest { @@ -9213,6 +9472,9 @@ export namespace flyteidl { /** GetTaskRequest resourceMeta */ resourceMeta?: (Uint8Array|null); + + /** GetTaskRequest taskCategory */ + taskCategory?: (flyteidl.admin.ITaskCategory|null); } /** Represents a GetTaskRequest. */ @@ -9230,6 +9492,9 @@ export namespace flyteidl { /** GetTaskRequest resourceMeta. */ public resourceMeta: Uint8Array; + /** GetTaskRequest taskCategory. */ + public taskCategory?: (flyteidl.admin.ITaskCategory|null); + /** * Creates a new GetTaskRequest instance using the specified properties. * @param [properties] Properties to set @@ -9268,9 +9533,6 @@ export namespace flyteidl { /** GetTaskResponse resource */ resource?: (flyteidl.admin.IResource|null); - - /** GetTaskResponse logLinks */ - logLinks?: (flyteidl.core.ITaskLog[]|null); } /** Represents a GetTaskResponse. */ @@ -9285,9 +9547,6 @@ export namespace flyteidl { /** GetTaskResponse resource. */ public resource?: (flyteidl.admin.IResource|null); - /** GetTaskResponse logLinks. */ - public logLinks: flyteidl.core.ITaskLog[]; - /** * Creates a new GetTaskResponse instance using the specified properties. * @param [properties] Properties to set @@ -9338,6 +9597,9 @@ export namespace flyteidl { /** Resource phase */ phase?: (flyteidl.core.TaskExecution.Phase|null); + + /** Resource customInfo */ + customInfo?: (google.protobuf.IStruct|null); } /** Represents a Resource. */ @@ -9364,6 +9626,9 @@ export namespace flyteidl { /** Resource phase. */ public phase: flyteidl.core.TaskExecution.Phase; + /** Resource customInfo. */ + public customInfo?: (google.protobuf.IStruct|null); + /** * Creates a new Resource instance using the specified properties. * @param [properties] Properties to set @@ -9405,6 +9670,9 @@ export namespace flyteidl { /** DeleteTaskRequest resourceMeta */ resourceMeta?: (Uint8Array|null); + + /** DeleteTaskRequest taskCategory */ + taskCategory?: (flyteidl.admin.ITaskCategory|null); } /** Represents a DeleteTaskRequest. */ @@ -9422,6 +9690,9 @@ export namespace flyteidl { /** DeleteTaskRequest resourceMeta. */ public resourceMeta: Uint8Array; + /** DeleteTaskRequest taskCategory. */ + public taskCategory?: (flyteidl.admin.ITaskCategory|null); + /** * Creates a new DeleteTaskRequest instance using the specified properties. * @param [properties] Properties to set @@ -9509,6 +9780,12 @@ export namespace flyteidl { /** Agent supportedTaskTypes */ supportedTaskTypes?: (string[]|null); + + /** Agent isSync */ + isSync?: (boolean|null); + + /** Agent supportedTaskCategories */ + supportedTaskCategories?: (flyteidl.admin.ITaskCategory[]|null); } /** Represents an Agent. */ @@ -9526,6 +9803,12 @@ export namespace flyteidl { /** Agent supportedTaskTypes. */ public supportedTaskTypes: string[]; + /** Agent isSync. */ + public isSync: boolean; + + /** Agent supportedTaskCategories. */ + public supportedTaskCategories: flyteidl.admin.ITaskCategory[]; + /** * Creates a new Agent instance using the specified properties. * @param [properties] Properties to set @@ -9559,6 +9842,64 @@ export namespace flyteidl { public static verify(message: { [k: string]: any }): (string|null); } + /** Properties of a TaskCategory. */ + interface ITaskCategory { + + /** TaskCategory name */ + name?: (string|null); + + /** TaskCategory version */ + version?: (number|null); + } + + /** Represents a TaskCategory. */ + class TaskCategory implements ITaskCategory { + + /** + * Constructs a new TaskCategory. + * @param [properties] Properties to set + */ + constructor(properties?: flyteidl.admin.ITaskCategory); + + /** TaskCategory name. */ + public name: string; + + /** TaskCategory version. */ + public version: number; + + /** + * Creates a new TaskCategory instance using the specified properties. + * @param [properties] Properties to set + * @returns TaskCategory instance + */ + public static create(properties?: flyteidl.admin.ITaskCategory): flyteidl.admin.TaskCategory; + + /** + * Encodes the specified TaskCategory message. Does not implicitly {@link flyteidl.admin.TaskCategory.verify|verify} messages. + * @param message TaskCategory message or plain object to encode + * @param [writer] Writer to encode to + * @returns Writer + */ + public static encode(message: flyteidl.admin.ITaskCategory, writer?: $protobuf.Writer): $protobuf.Writer; + + /** + * Decodes a TaskCategory message from the specified reader or buffer. + * @param reader Reader or buffer to decode from + * @param [length] Message length if known beforehand + * @returns TaskCategory + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + public static decode(reader: ($protobuf.Reader|Uint8Array), length?: number): flyteidl.admin.TaskCategory; + + /** + * Verifies a TaskCategory message. + * @param message Plain object to verify + * @returns `null` if valid, otherwise the reason why it is not + */ + public static verify(message: { [k: string]: any }): (string|null); + } + /** Properties of a GetAgentRequest. */ interface IGetAgentRequest { @@ -9781,6 +10122,9 @@ export namespace flyteidl { /** GetTaskMetricsRequest step */ step?: (google.protobuf.IDuration|null); + + /** GetTaskMetricsRequest taskCategory */ + taskCategory?: (flyteidl.admin.ITaskCategory|null); } /** Represents a GetTaskMetricsRequest. */ @@ -9810,6 +10154,9 @@ export namespace flyteidl { /** GetTaskMetricsRequest step. */ public step?: (google.protobuf.IDuration|null); + /** GetTaskMetricsRequest taskCategory. */ + public taskCategory?: (flyteidl.admin.ITaskCategory|null); + /** * Creates a new GetTaskMetricsRequest instance using the specified properties. * @param [properties] Properties to set @@ -9909,6 +10256,9 @@ export namespace flyteidl { /** GetTaskLogsRequest token */ token?: (string|null); + + /** GetTaskLogsRequest taskCategory */ + taskCategory?: (flyteidl.admin.ITaskCategory|null); } /** Represents a GetTaskLogsRequest. */ @@ -9932,6 +10282,9 @@ export namespace flyteidl { /** GetTaskLogsRequest token. */ public token: string; + /** GetTaskLogsRequest taskCategory. */ + public taskCategory?: (flyteidl.admin.ITaskCategory|null); + /** * Creates a new GetTaskLogsRequest instance using the specified properties. * @param [properties] Properties to set @@ -9965,14 +10318,118 @@ export namespace flyteidl { public static verify(message: { [k: string]: any }): (string|null); } + /** Properties of a GetTaskLogsResponseHeader. */ + interface IGetTaskLogsResponseHeader { + + /** GetTaskLogsResponseHeader token */ + token?: (string|null); + } + + /** Represents a GetTaskLogsResponseHeader. */ + class GetTaskLogsResponseHeader implements IGetTaskLogsResponseHeader { + + /** + * Constructs a new GetTaskLogsResponseHeader. + * @param [properties] Properties to set + */ + constructor(properties?: flyteidl.admin.IGetTaskLogsResponseHeader); + + /** GetTaskLogsResponseHeader token. */ + public token: string; + + /** + * Creates a new GetTaskLogsResponseHeader instance using the specified properties. + * @param [properties] Properties to set + * @returns GetTaskLogsResponseHeader instance + */ + public static create(properties?: flyteidl.admin.IGetTaskLogsResponseHeader): flyteidl.admin.GetTaskLogsResponseHeader; + + /** + * Encodes the specified GetTaskLogsResponseHeader message. Does not implicitly {@link flyteidl.admin.GetTaskLogsResponseHeader.verify|verify} messages. + * @param message GetTaskLogsResponseHeader message or plain object to encode + * @param [writer] Writer to encode to + * @returns Writer + */ + public static encode(message: flyteidl.admin.IGetTaskLogsResponseHeader, writer?: $protobuf.Writer): $protobuf.Writer; + + /** + * Decodes a GetTaskLogsResponseHeader message from the specified reader or buffer. + * @param reader Reader or buffer to decode from + * @param [length] Message length if known beforehand + * @returns GetTaskLogsResponseHeader + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + public static decode(reader: ($protobuf.Reader|Uint8Array), length?: number): flyteidl.admin.GetTaskLogsResponseHeader; + + /** + * Verifies a GetTaskLogsResponseHeader message. + * @param message Plain object to verify + * @returns `null` if valid, otherwise the reason why it is not + */ + public static verify(message: { [k: string]: any }): (string|null); + } + + /** Properties of a GetTaskLogsResponseBody. */ + interface IGetTaskLogsResponseBody { + + /** GetTaskLogsResponseBody results */ + results?: (string[]|null); + } + + /** Represents a GetTaskLogsResponseBody. */ + class GetTaskLogsResponseBody implements IGetTaskLogsResponseBody { + + /** + * Constructs a new GetTaskLogsResponseBody. + * @param [properties] Properties to set + */ + constructor(properties?: flyteidl.admin.IGetTaskLogsResponseBody); + + /** GetTaskLogsResponseBody results. */ + public results: string[]; + + /** + * Creates a new GetTaskLogsResponseBody instance using the specified properties. + * @param [properties] Properties to set + * @returns GetTaskLogsResponseBody instance + */ + public static create(properties?: flyteidl.admin.IGetTaskLogsResponseBody): flyteidl.admin.GetTaskLogsResponseBody; + + /** + * Encodes the specified GetTaskLogsResponseBody message. Does not implicitly {@link flyteidl.admin.GetTaskLogsResponseBody.verify|verify} messages. + * @param message GetTaskLogsResponseBody message or plain object to encode + * @param [writer] Writer to encode to + * @returns Writer + */ + public static encode(message: flyteidl.admin.IGetTaskLogsResponseBody, writer?: $protobuf.Writer): $protobuf.Writer; + + /** + * Decodes a GetTaskLogsResponseBody message from the specified reader or buffer. + * @param reader Reader or buffer to decode from + * @param [length] Message length if known beforehand + * @returns GetTaskLogsResponseBody + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + public static decode(reader: ($protobuf.Reader|Uint8Array), length?: number): flyteidl.admin.GetTaskLogsResponseBody; + + /** + * Verifies a GetTaskLogsResponseBody message. + * @param message Plain object to verify + * @returns `null` if valid, otherwise the reason why it is not + */ + public static verify(message: { [k: string]: any }): (string|null); + } + /** Properties of a GetTaskLogsResponse. */ interface IGetTaskLogsResponse { - /** GetTaskLogsResponse results */ - results?: (string[]|null); + /** GetTaskLogsResponse header */ + header?: (flyteidl.admin.IGetTaskLogsResponseHeader|null); - /** GetTaskLogsResponse token */ - token?: (string|null); + /** GetTaskLogsResponse body */ + body?: (flyteidl.admin.IGetTaskLogsResponseBody|null); } /** Represents a GetTaskLogsResponse. */ @@ -9984,11 +10441,14 @@ export namespace flyteidl { */ constructor(properties?: flyteidl.admin.IGetTaskLogsResponse); - /** GetTaskLogsResponse results. */ - public results: string[]; + /** GetTaskLogsResponse header. */ + public header?: (flyteidl.admin.IGetTaskLogsResponseHeader|null); - /** GetTaskLogsResponse token. */ - public token: string; + /** GetTaskLogsResponse body. */ + public body?: (flyteidl.admin.IGetTaskLogsResponseBody|null); + + /** GetTaskLogsResponse part. */ + public part?: ("header"|"body"); /** * Creates a new GetTaskLogsResponse instance using the specified properties. @@ -13227,7 +13687,8 @@ export namespace flyteidl { SYSTEM = 2, RELAUNCH = 3, CHILD_WORKFLOW = 4, - RECOVERED = 5 + RECOVERED = 5, + TRIGGER = 6 } } @@ -21558,6 +22019,51 @@ export namespace flyteidl { type GetExecutionMetricsCallback = (error: (Error|null), response?: flyteidl.admin.WorkflowExecutionGetMetricsResponse) => void; } + /** Represents a SyncAgentService */ + class SyncAgentService extends $protobuf.rpc.Service { + + /** + * Constructs a new SyncAgentService service. + * @param rpcImpl RPC implementation + * @param [requestDelimited=false] Whether requests are length-delimited + * @param [responseDelimited=false] Whether responses are length-delimited + */ + constructor(rpcImpl: $protobuf.RPCImpl, requestDelimited?: boolean, responseDelimited?: boolean); + + /** + * Creates new SyncAgentService service using the specified rpc implementation. + * @param rpcImpl RPC implementation + * @param [requestDelimited=false] Whether requests are length-delimited + * @param [responseDelimited=false] Whether responses are length-delimited + * @returns RPC service. Useful where requests and/or responses are streamed. + */ + public static create(rpcImpl: $protobuf.RPCImpl, requestDelimited?: boolean, responseDelimited?: boolean): SyncAgentService; + + /** + * Calls ExecuteTaskSync. + * @param request ExecuteTaskSyncRequest message or plain object + * @param callback Node-style callback called with the error, if any, and ExecuteTaskSyncResponse + */ + public executeTaskSync(request: flyteidl.admin.IExecuteTaskSyncRequest, callback: flyteidl.service.SyncAgentService.ExecuteTaskSyncCallback): void; + + /** + * Calls ExecuteTaskSync. + * @param request ExecuteTaskSyncRequest message or plain object + * @returns Promise + */ + public executeTaskSync(request: flyteidl.admin.IExecuteTaskSyncRequest): Promise; + } + + namespace SyncAgentService { + + /** + * Callback as used by {@link flyteidl.service.SyncAgentService#executeTaskSync}. + * @param error Error, if any + * @param [response] ExecuteTaskSyncResponse + */ + type ExecuteTaskSyncCallback = (error: (Error|null), response?: flyteidl.admin.ExecuteTaskSyncResponse) => void; + } + /** Represents an AsyncAgentService */ class AsyncAgentService extends $protobuf.rpc.Service { diff --git a/flyteidl/gen/pb-js/flyteidl.js b/flyteidl/gen/pb-js/flyteidl.js index c7fac48ad5..161d05475b 100644 --- a/flyteidl/gen/pb-js/flyteidl.js +++ b/flyteidl/gen/pb-js/flyteidl.js @@ -201,7 +201,6 @@ * Properties of an ArtifactBindingData. * @memberof flyteidl.core * @interface IArtifactBindingData - * @property {number|null} [index] ArtifactBindingData index * @property {string|null} [partitionKey] ArtifactBindingData partitionKey * @property {boolean|null} [bindToTimePartition] ArtifactBindingData bindToTimePartition * @property {string|null} [transform] ArtifactBindingData transform @@ -222,14 +221,6 @@ this[keys[i]] = properties[keys[i]]; } - /** - * ArtifactBindingData index. - * @member {number} index - * @memberof flyteidl.core.ArtifactBindingData - * @instance - */ - ArtifactBindingData.prototype.index = 0; - /** * ArtifactBindingData partitionKey. * @member {string} partitionKey @@ -292,8 +283,6 @@ ArtifactBindingData.encode = function encode(message, writer) { if (!writer) writer = $Writer.create(); - if (message.index != null && message.hasOwnProperty("index")) - writer.uint32(/* id 1, wireType 0 =*/8).uint32(message.index); if (message.partitionKey != null && message.hasOwnProperty("partitionKey")) writer.uint32(/* id 2, wireType 2 =*/18).string(message.partitionKey); if (message.bindToTimePartition != null && message.hasOwnProperty("bindToTimePartition")) @@ -321,9 +310,6 @@ while (reader.pos < end) { var tag = reader.uint32(); switch (tag >>> 3) { - case 1: - message.index = reader.uint32(); - break; case 2: message.partitionKey = reader.string(); break; @@ -353,9 +339,6 @@ if (typeof message !== "object" || message === null) return "object expected"; var properties = {}; - if (message.index != null && message.hasOwnProperty("index")) - if (!$util.isInteger(message.index)) - return "index: integer expected"; if (message.partitionKey != null && message.hasOwnProperty("partitionKey")) { properties.partitionData = 1; if (!$util.isString(message.partitionKey)) @@ -11969,6 +11952,7 @@ * @interface ITaskNodeOverrides * @property {flyteidl.core.IResources|null} [resources] TaskNodeOverrides resources * @property {flyteidl.core.IExtendedResources|null} [extendedResources] TaskNodeOverrides extendedResources + * @property {string|null} [containerImage] TaskNodeOverrides containerImage */ /** @@ -12002,6 +11986,14 @@ */ TaskNodeOverrides.prototype.extendedResources = null; + /** + * TaskNodeOverrides containerImage. + * @member {string} containerImage + * @memberof flyteidl.core.TaskNodeOverrides + * @instance + */ + TaskNodeOverrides.prototype.containerImage = ""; + /** * Creates a new TaskNodeOverrides instance using the specified properties. * @function create @@ -12030,6 +12022,8 @@ $root.flyteidl.core.Resources.encode(message.resources, writer.uint32(/* id 1, wireType 2 =*/10).fork()).ldelim(); if (message.extendedResources != null && message.hasOwnProperty("extendedResources")) $root.flyteidl.core.ExtendedResources.encode(message.extendedResources, writer.uint32(/* id 2, wireType 2 =*/18).fork()).ldelim(); + if (message.containerImage != null && message.hasOwnProperty("containerImage")) + writer.uint32(/* id 3, wireType 2 =*/26).string(message.containerImage); return writer; }; @@ -12057,6 +12051,9 @@ case 2: message.extendedResources = $root.flyteidl.core.ExtendedResources.decode(reader, reader.uint32()); break; + case 3: + message.containerImage = reader.string(); + break; default: reader.skipType(tag & 7); break; @@ -12086,6 +12083,9 @@ if (error) return "extendedResources." + error; } + if (message.containerImage != null && message.hasOwnProperty("containerImage")) + if (!$util.isString(message.containerImage)) + return "containerImage: string expected"; return null; }; @@ -22073,6 +22073,10 @@ * @property {Object.|null} [annotations] TaskExecutionMetadata annotations * @property {string|null} [k8sServiceAccount] TaskExecutionMetadata k8sServiceAccount * @property {Object.|null} [environmentVariables] TaskExecutionMetadata environmentVariables + * @property {number|null} [maxAttempts] TaskExecutionMetadata maxAttempts + * @property {boolean|null} [interruptible] TaskExecutionMetadata interruptible + * @property {number|null} [interruptibleFailureThreshold] TaskExecutionMetadata interruptibleFailureThreshold + * @property {flyteidl.core.ITaskNodeOverrides|null} [overrides] TaskExecutionMetadata overrides */ /** @@ -22141,6 +22145,38 @@ */ TaskExecutionMetadata.prototype.environmentVariables = $util.emptyObject; + /** + * TaskExecutionMetadata maxAttempts. + * @member {number} maxAttempts + * @memberof flyteidl.admin.TaskExecutionMetadata + * @instance + */ + TaskExecutionMetadata.prototype.maxAttempts = 0; + + /** + * TaskExecutionMetadata interruptible. + * @member {boolean} interruptible + * @memberof flyteidl.admin.TaskExecutionMetadata + * @instance + */ + TaskExecutionMetadata.prototype.interruptible = false; + + /** + * TaskExecutionMetadata interruptibleFailureThreshold. + * @member {number} interruptibleFailureThreshold + * @memberof flyteidl.admin.TaskExecutionMetadata + * @instance + */ + TaskExecutionMetadata.prototype.interruptibleFailureThreshold = 0; + + /** + * TaskExecutionMetadata overrides. + * @member {flyteidl.core.ITaskNodeOverrides|null|undefined} overrides + * @memberof flyteidl.admin.TaskExecutionMetadata + * @instance + */ + TaskExecutionMetadata.prototype.overrides = null; + /** * Creates a new TaskExecutionMetadata instance using the specified properties. * @function create @@ -22180,6 +22216,14 @@ if (message.environmentVariables != null && message.hasOwnProperty("environmentVariables")) for (var keys = Object.keys(message.environmentVariables), i = 0; i < keys.length; ++i) writer.uint32(/* id 6, wireType 2 =*/50).fork().uint32(/* id 1, wireType 2 =*/10).string(keys[i]).uint32(/* id 2, wireType 2 =*/18).string(message.environmentVariables[keys[i]]).ldelim(); + if (message.maxAttempts != null && message.hasOwnProperty("maxAttempts")) + writer.uint32(/* id 7, wireType 0 =*/56).int32(message.maxAttempts); + if (message.interruptible != null && message.hasOwnProperty("interruptible")) + writer.uint32(/* id 8, wireType 0 =*/64).bool(message.interruptible); + if (message.interruptibleFailureThreshold != null && message.hasOwnProperty("interruptibleFailureThreshold")) + writer.uint32(/* id 9, wireType 0 =*/72).int32(message.interruptibleFailureThreshold); + if (message.overrides != null && message.hasOwnProperty("overrides")) + $root.flyteidl.core.TaskNodeOverrides.encode(message.overrides, writer.uint32(/* id 10, wireType 2 =*/82).fork()).ldelim(); return writer; }; @@ -22234,6 +22278,18 @@ reader.pos++; message.environmentVariables[key] = reader.string(); break; + case 7: + message.maxAttempts = reader.int32(); + break; + case 8: + message.interruptible = reader.bool(); + break; + case 9: + message.interruptibleFailureThreshold = reader.int32(); + break; + case 10: + message.overrides = $root.flyteidl.core.TaskNodeOverrides.decode(reader, reader.uint32()); + break; default: reader.skipType(tag & 7); break; @@ -22288,6 +22344,20 @@ if (!$util.isString(message.environmentVariables[key[i]])) return "environmentVariables: string{k:string} expected"; } + if (message.maxAttempts != null && message.hasOwnProperty("maxAttempts")) + if (!$util.isInteger(message.maxAttempts)) + return "maxAttempts: integer expected"; + if (message.interruptible != null && message.hasOwnProperty("interruptible")) + if (typeof message.interruptible !== "boolean") + return "interruptible: boolean expected"; + if (message.interruptibleFailureThreshold != null && message.hasOwnProperty("interruptibleFailureThreshold")) + if (!$util.isInteger(message.interruptibleFailureThreshold)) + return "interruptibleFailureThreshold: integer expected"; + if (message.overrides != null && message.hasOwnProperty("overrides")) { + var error = $root.flyteidl.core.TaskNodeOverrides.verify(message.overrides); + if (error) + return "overrides." + error; + } return null; }; @@ -22468,7 +22538,6 @@ * @memberof flyteidl.admin * @interface ICreateTaskResponse * @property {Uint8Array|null} [resourceMeta] CreateTaskResponse resourceMeta - * @property {flyteidl.admin.IResource|null} [resource] CreateTaskResponse resource */ /** @@ -22494,28 +22563,6 @@ */ CreateTaskResponse.prototype.resourceMeta = $util.newBuffer([]); - /** - * CreateTaskResponse resource. - * @member {flyteidl.admin.IResource|null|undefined} resource - * @memberof flyteidl.admin.CreateTaskResponse - * @instance - */ - CreateTaskResponse.prototype.resource = null; - - // OneOf field names bound to virtual getters and setters - var $oneOfFields; - - /** - * CreateTaskResponse res. - * @member {"resourceMeta"|"resource"|undefined} res - * @memberof flyteidl.admin.CreateTaskResponse - * @instance - */ - Object.defineProperty(CreateTaskResponse.prototype, "res", { - get: $util.oneOfGetter($oneOfFields = ["resourceMeta", "resource"]), - set: $util.oneOfSetter($oneOfFields) - }); - /** * Creates a new CreateTaskResponse instance using the specified properties. * @function create @@ -22542,8 +22589,6 @@ writer = $Writer.create(); if (message.resourceMeta != null && message.hasOwnProperty("resourceMeta")) writer.uint32(/* id 1, wireType 2 =*/10).bytes(message.resourceMeta); - if (message.resource != null && message.hasOwnProperty("resource")) - $root.flyteidl.admin.Resource.encode(message.resource, writer.uint32(/* id 2, wireType 2 =*/18).fork()).ldelim(); return writer; }; @@ -22568,9 +22613,6 @@ case 1: message.resourceMeta = reader.bytes(); break; - case 2: - message.resource = $root.flyteidl.admin.Resource.decode(reader, reader.uint32()); - break; default: reader.skipType(tag & 7); break; @@ -22590,26 +22632,598 @@ CreateTaskResponse.verify = function verify(message) { if (typeof message !== "object" || message === null) return "object expected"; - var properties = {}; - if (message.resourceMeta != null && message.hasOwnProperty("resourceMeta")) { - properties.res = 1; + if (message.resourceMeta != null && message.hasOwnProperty("resourceMeta")) if (!(message.resourceMeta && typeof message.resourceMeta.length === "number" || $util.isString(message.resourceMeta))) return "resourceMeta: buffer expected"; + return null; + }; + + return CreateTaskResponse; + })(); + + admin.CreateRequestHeader = (function() { + + /** + * Properties of a CreateRequestHeader. + * @memberof flyteidl.admin + * @interface ICreateRequestHeader + * @property {flyteidl.core.ITaskTemplate|null} [template] CreateRequestHeader template + * @property {string|null} [outputPrefix] CreateRequestHeader outputPrefix + * @property {flyteidl.admin.ITaskExecutionMetadata|null} [taskExecutionMetadata] CreateRequestHeader taskExecutionMetadata + * @property {Long|null} [maxDatasetSizeBytes] CreateRequestHeader maxDatasetSizeBytes + */ + + /** + * Constructs a new CreateRequestHeader. + * @memberof flyteidl.admin + * @classdesc Represents a CreateRequestHeader. + * @implements ICreateRequestHeader + * @constructor + * @param {flyteidl.admin.ICreateRequestHeader=} [properties] Properties to set + */ + function CreateRequestHeader(properties) { + if (properties) + for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) + if (properties[keys[i]] != null) + this[keys[i]] = properties[keys[i]]; + } + + /** + * CreateRequestHeader template. + * @member {flyteidl.core.ITaskTemplate|null|undefined} template + * @memberof flyteidl.admin.CreateRequestHeader + * @instance + */ + CreateRequestHeader.prototype.template = null; + + /** + * CreateRequestHeader outputPrefix. + * @member {string} outputPrefix + * @memberof flyteidl.admin.CreateRequestHeader + * @instance + */ + CreateRequestHeader.prototype.outputPrefix = ""; + + /** + * CreateRequestHeader taskExecutionMetadata. + * @member {flyteidl.admin.ITaskExecutionMetadata|null|undefined} taskExecutionMetadata + * @memberof flyteidl.admin.CreateRequestHeader + * @instance + */ + CreateRequestHeader.prototype.taskExecutionMetadata = null; + + /** + * CreateRequestHeader maxDatasetSizeBytes. + * @member {Long} maxDatasetSizeBytes + * @memberof flyteidl.admin.CreateRequestHeader + * @instance + */ + CreateRequestHeader.prototype.maxDatasetSizeBytes = $util.Long ? $util.Long.fromBits(0,0,false) : 0; + + /** + * Creates a new CreateRequestHeader instance using the specified properties. + * @function create + * @memberof flyteidl.admin.CreateRequestHeader + * @static + * @param {flyteidl.admin.ICreateRequestHeader=} [properties] Properties to set + * @returns {flyteidl.admin.CreateRequestHeader} CreateRequestHeader instance + */ + CreateRequestHeader.create = function create(properties) { + return new CreateRequestHeader(properties); + }; + + /** + * Encodes the specified CreateRequestHeader message. Does not implicitly {@link flyteidl.admin.CreateRequestHeader.verify|verify} messages. + * @function encode + * @memberof flyteidl.admin.CreateRequestHeader + * @static + * @param {flyteidl.admin.ICreateRequestHeader} message CreateRequestHeader message or plain object to encode + * @param {$protobuf.Writer} [writer] Writer to encode to + * @returns {$protobuf.Writer} Writer + */ + CreateRequestHeader.encode = function encode(message, writer) { + if (!writer) + writer = $Writer.create(); + if (message.template != null && message.hasOwnProperty("template")) + $root.flyteidl.core.TaskTemplate.encode(message.template, writer.uint32(/* id 1, wireType 2 =*/10).fork()).ldelim(); + if (message.outputPrefix != null && message.hasOwnProperty("outputPrefix")) + writer.uint32(/* id 2, wireType 2 =*/18).string(message.outputPrefix); + if (message.taskExecutionMetadata != null && message.hasOwnProperty("taskExecutionMetadata")) + $root.flyteidl.admin.TaskExecutionMetadata.encode(message.taskExecutionMetadata, writer.uint32(/* id 3, wireType 2 =*/26).fork()).ldelim(); + if (message.maxDatasetSizeBytes != null && message.hasOwnProperty("maxDatasetSizeBytes")) + writer.uint32(/* id 4, wireType 0 =*/32).int64(message.maxDatasetSizeBytes); + return writer; + }; + + /** + * Decodes a CreateRequestHeader message from the specified reader or buffer. + * @function decode + * @memberof flyteidl.admin.CreateRequestHeader + * @static + * @param {$protobuf.Reader|Uint8Array} reader Reader or buffer to decode from + * @param {number} [length] Message length if known beforehand + * @returns {flyteidl.admin.CreateRequestHeader} CreateRequestHeader + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + CreateRequestHeader.decode = function decode(reader, length) { + if (!(reader instanceof $Reader)) + reader = $Reader.create(reader); + var end = length === undefined ? reader.len : reader.pos + length, message = new $root.flyteidl.admin.CreateRequestHeader(); + while (reader.pos < end) { + var tag = reader.uint32(); + switch (tag >>> 3) { + case 1: + message.template = $root.flyteidl.core.TaskTemplate.decode(reader, reader.uint32()); + break; + case 2: + message.outputPrefix = reader.string(); + break; + case 3: + message.taskExecutionMetadata = $root.flyteidl.admin.TaskExecutionMetadata.decode(reader, reader.uint32()); + break; + case 4: + message.maxDatasetSizeBytes = reader.int64(); + break; + default: + reader.skipType(tag & 7); + break; + } + } + return message; + }; + + /** + * Verifies a CreateRequestHeader message. + * @function verify + * @memberof flyteidl.admin.CreateRequestHeader + * @static + * @param {Object.} message Plain object to verify + * @returns {string|null} `null` if valid, otherwise the reason why it is not + */ + CreateRequestHeader.verify = function verify(message) { + if (typeof message !== "object" || message === null) + return "object expected"; + if (message.template != null && message.hasOwnProperty("template")) { + var error = $root.flyteidl.core.TaskTemplate.verify(message.template); + if (error) + return "template." + error; + } + if (message.outputPrefix != null && message.hasOwnProperty("outputPrefix")) + if (!$util.isString(message.outputPrefix)) + return "outputPrefix: string expected"; + if (message.taskExecutionMetadata != null && message.hasOwnProperty("taskExecutionMetadata")) { + var error = $root.flyteidl.admin.TaskExecutionMetadata.verify(message.taskExecutionMetadata); + if (error) + return "taskExecutionMetadata." + error; + } + if (message.maxDatasetSizeBytes != null && message.hasOwnProperty("maxDatasetSizeBytes")) + if (!$util.isInteger(message.maxDatasetSizeBytes) && !(message.maxDatasetSizeBytes && $util.isInteger(message.maxDatasetSizeBytes.low) && $util.isInteger(message.maxDatasetSizeBytes.high))) + return "maxDatasetSizeBytes: integer|Long expected"; + return null; + }; + + return CreateRequestHeader; + })(); + + admin.ExecuteTaskSyncRequest = (function() { + + /** + * Properties of an ExecuteTaskSyncRequest. + * @memberof flyteidl.admin + * @interface IExecuteTaskSyncRequest + * @property {flyteidl.admin.ICreateRequestHeader|null} [header] ExecuteTaskSyncRequest header + * @property {flyteidl.core.ILiteralMap|null} [inputs] ExecuteTaskSyncRequest inputs + */ + + /** + * Constructs a new ExecuteTaskSyncRequest. + * @memberof flyteidl.admin + * @classdesc Represents an ExecuteTaskSyncRequest. + * @implements IExecuteTaskSyncRequest + * @constructor + * @param {flyteidl.admin.IExecuteTaskSyncRequest=} [properties] Properties to set + */ + function ExecuteTaskSyncRequest(properties) { + if (properties) + for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) + if (properties[keys[i]] != null) + this[keys[i]] = properties[keys[i]]; + } + + /** + * ExecuteTaskSyncRequest header. + * @member {flyteidl.admin.ICreateRequestHeader|null|undefined} header + * @memberof flyteidl.admin.ExecuteTaskSyncRequest + * @instance + */ + ExecuteTaskSyncRequest.prototype.header = null; + + /** + * ExecuteTaskSyncRequest inputs. + * @member {flyteidl.core.ILiteralMap|null|undefined} inputs + * @memberof flyteidl.admin.ExecuteTaskSyncRequest + * @instance + */ + ExecuteTaskSyncRequest.prototype.inputs = null; + + // OneOf field names bound to virtual getters and setters + var $oneOfFields; + + /** + * ExecuteTaskSyncRequest part. + * @member {"header"|"inputs"|undefined} part + * @memberof flyteidl.admin.ExecuteTaskSyncRequest + * @instance + */ + Object.defineProperty(ExecuteTaskSyncRequest.prototype, "part", { + get: $util.oneOfGetter($oneOfFields = ["header", "inputs"]), + set: $util.oneOfSetter($oneOfFields) + }); + + /** + * Creates a new ExecuteTaskSyncRequest instance using the specified properties. + * @function create + * @memberof flyteidl.admin.ExecuteTaskSyncRequest + * @static + * @param {flyteidl.admin.IExecuteTaskSyncRequest=} [properties] Properties to set + * @returns {flyteidl.admin.ExecuteTaskSyncRequest} ExecuteTaskSyncRequest instance + */ + ExecuteTaskSyncRequest.create = function create(properties) { + return new ExecuteTaskSyncRequest(properties); + }; + + /** + * Encodes the specified ExecuteTaskSyncRequest message. Does not implicitly {@link flyteidl.admin.ExecuteTaskSyncRequest.verify|verify} messages. + * @function encode + * @memberof flyteidl.admin.ExecuteTaskSyncRequest + * @static + * @param {flyteidl.admin.IExecuteTaskSyncRequest} message ExecuteTaskSyncRequest message or plain object to encode + * @param {$protobuf.Writer} [writer] Writer to encode to + * @returns {$protobuf.Writer} Writer + */ + ExecuteTaskSyncRequest.encode = function encode(message, writer) { + if (!writer) + writer = $Writer.create(); + if (message.header != null && message.hasOwnProperty("header")) + $root.flyteidl.admin.CreateRequestHeader.encode(message.header, writer.uint32(/* id 1, wireType 2 =*/10).fork()).ldelim(); + if (message.inputs != null && message.hasOwnProperty("inputs")) + $root.flyteidl.core.LiteralMap.encode(message.inputs, writer.uint32(/* id 2, wireType 2 =*/18).fork()).ldelim(); + return writer; + }; + + /** + * Decodes an ExecuteTaskSyncRequest message from the specified reader or buffer. + * @function decode + * @memberof flyteidl.admin.ExecuteTaskSyncRequest + * @static + * @param {$protobuf.Reader|Uint8Array} reader Reader or buffer to decode from + * @param {number} [length] Message length if known beforehand + * @returns {flyteidl.admin.ExecuteTaskSyncRequest} ExecuteTaskSyncRequest + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + ExecuteTaskSyncRequest.decode = function decode(reader, length) { + if (!(reader instanceof $Reader)) + reader = $Reader.create(reader); + var end = length === undefined ? reader.len : reader.pos + length, message = new $root.flyteidl.admin.ExecuteTaskSyncRequest(); + while (reader.pos < end) { + var tag = reader.uint32(); + switch (tag >>> 3) { + case 1: + message.header = $root.flyteidl.admin.CreateRequestHeader.decode(reader, reader.uint32()); + break; + case 2: + message.inputs = $root.flyteidl.core.LiteralMap.decode(reader, reader.uint32()); + break; + default: + reader.skipType(tag & 7); + break; + } + } + return message; + }; + + /** + * Verifies an ExecuteTaskSyncRequest message. + * @function verify + * @memberof flyteidl.admin.ExecuteTaskSyncRequest + * @static + * @param {Object.} message Plain object to verify + * @returns {string|null} `null` if valid, otherwise the reason why it is not + */ + ExecuteTaskSyncRequest.verify = function verify(message) { + if (typeof message !== "object" || message === null) + return "object expected"; + var properties = {}; + if (message.header != null && message.hasOwnProperty("header")) { + properties.part = 1; + { + var error = $root.flyteidl.admin.CreateRequestHeader.verify(message.header); + if (error) + return "header." + error; + } + } + if (message.inputs != null && message.hasOwnProperty("inputs")) { + if (properties.part === 1) + return "part: multiple values"; + properties.part = 1; + { + var error = $root.flyteidl.core.LiteralMap.verify(message.inputs); + if (error) + return "inputs." + error; + } + } + return null; + }; + + return ExecuteTaskSyncRequest; + })(); + + admin.ExecuteTaskSyncResponseHeader = (function() { + + /** + * Properties of an ExecuteTaskSyncResponseHeader. + * @memberof flyteidl.admin + * @interface IExecuteTaskSyncResponseHeader + * @property {flyteidl.admin.IResource|null} [resource] ExecuteTaskSyncResponseHeader resource + */ + + /** + * Constructs a new ExecuteTaskSyncResponseHeader. + * @memberof flyteidl.admin + * @classdesc Represents an ExecuteTaskSyncResponseHeader. + * @implements IExecuteTaskSyncResponseHeader + * @constructor + * @param {flyteidl.admin.IExecuteTaskSyncResponseHeader=} [properties] Properties to set + */ + function ExecuteTaskSyncResponseHeader(properties) { + if (properties) + for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) + if (properties[keys[i]] != null) + this[keys[i]] = properties[keys[i]]; + } + + /** + * ExecuteTaskSyncResponseHeader resource. + * @member {flyteidl.admin.IResource|null|undefined} resource + * @memberof flyteidl.admin.ExecuteTaskSyncResponseHeader + * @instance + */ + ExecuteTaskSyncResponseHeader.prototype.resource = null; + + /** + * Creates a new ExecuteTaskSyncResponseHeader instance using the specified properties. + * @function create + * @memberof flyteidl.admin.ExecuteTaskSyncResponseHeader + * @static + * @param {flyteidl.admin.IExecuteTaskSyncResponseHeader=} [properties] Properties to set + * @returns {flyteidl.admin.ExecuteTaskSyncResponseHeader} ExecuteTaskSyncResponseHeader instance + */ + ExecuteTaskSyncResponseHeader.create = function create(properties) { + return new ExecuteTaskSyncResponseHeader(properties); + }; + + /** + * Encodes the specified ExecuteTaskSyncResponseHeader message. Does not implicitly {@link flyteidl.admin.ExecuteTaskSyncResponseHeader.verify|verify} messages. + * @function encode + * @memberof flyteidl.admin.ExecuteTaskSyncResponseHeader + * @static + * @param {flyteidl.admin.IExecuteTaskSyncResponseHeader} message ExecuteTaskSyncResponseHeader message or plain object to encode + * @param {$protobuf.Writer} [writer] Writer to encode to + * @returns {$protobuf.Writer} Writer + */ + ExecuteTaskSyncResponseHeader.encode = function encode(message, writer) { + if (!writer) + writer = $Writer.create(); + if (message.resource != null && message.hasOwnProperty("resource")) + $root.flyteidl.admin.Resource.encode(message.resource, writer.uint32(/* id 1, wireType 2 =*/10).fork()).ldelim(); + return writer; + }; + + /** + * Decodes an ExecuteTaskSyncResponseHeader message from the specified reader or buffer. + * @function decode + * @memberof flyteidl.admin.ExecuteTaskSyncResponseHeader + * @static + * @param {$protobuf.Reader|Uint8Array} reader Reader or buffer to decode from + * @param {number} [length] Message length if known beforehand + * @returns {flyteidl.admin.ExecuteTaskSyncResponseHeader} ExecuteTaskSyncResponseHeader + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + ExecuteTaskSyncResponseHeader.decode = function decode(reader, length) { + if (!(reader instanceof $Reader)) + reader = $Reader.create(reader); + var end = length === undefined ? reader.len : reader.pos + length, message = new $root.flyteidl.admin.ExecuteTaskSyncResponseHeader(); + while (reader.pos < end) { + var tag = reader.uint32(); + switch (tag >>> 3) { + case 1: + message.resource = $root.flyteidl.admin.Resource.decode(reader, reader.uint32()); + break; + default: + reader.skipType(tag & 7); + break; + } } + return message; + }; + + /** + * Verifies an ExecuteTaskSyncResponseHeader message. + * @function verify + * @memberof flyteidl.admin.ExecuteTaskSyncResponseHeader + * @static + * @param {Object.} message Plain object to verify + * @returns {string|null} `null` if valid, otherwise the reason why it is not + */ + ExecuteTaskSyncResponseHeader.verify = function verify(message) { + if (typeof message !== "object" || message === null) + return "object expected"; if (message.resource != null && message.hasOwnProperty("resource")) { + var error = $root.flyteidl.admin.Resource.verify(message.resource); + if (error) + return "resource." + error; + } + return null; + }; + + return ExecuteTaskSyncResponseHeader; + })(); + + admin.ExecuteTaskSyncResponse = (function() { + + /** + * Properties of an ExecuteTaskSyncResponse. + * @memberof flyteidl.admin + * @interface IExecuteTaskSyncResponse + * @property {flyteidl.admin.IExecuteTaskSyncResponseHeader|null} [header] ExecuteTaskSyncResponse header + * @property {flyteidl.core.ILiteralMap|null} [outputs] ExecuteTaskSyncResponse outputs + */ + + /** + * Constructs a new ExecuteTaskSyncResponse. + * @memberof flyteidl.admin + * @classdesc Represents an ExecuteTaskSyncResponse. + * @implements IExecuteTaskSyncResponse + * @constructor + * @param {flyteidl.admin.IExecuteTaskSyncResponse=} [properties] Properties to set + */ + function ExecuteTaskSyncResponse(properties) { + if (properties) + for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) + if (properties[keys[i]] != null) + this[keys[i]] = properties[keys[i]]; + } + + /** + * ExecuteTaskSyncResponse header. + * @member {flyteidl.admin.IExecuteTaskSyncResponseHeader|null|undefined} header + * @memberof flyteidl.admin.ExecuteTaskSyncResponse + * @instance + */ + ExecuteTaskSyncResponse.prototype.header = null; + + /** + * ExecuteTaskSyncResponse outputs. + * @member {flyteidl.core.ILiteralMap|null|undefined} outputs + * @memberof flyteidl.admin.ExecuteTaskSyncResponse + * @instance + */ + ExecuteTaskSyncResponse.prototype.outputs = null; + + // OneOf field names bound to virtual getters and setters + var $oneOfFields; + + /** + * ExecuteTaskSyncResponse res. + * @member {"header"|"outputs"|undefined} res + * @memberof flyteidl.admin.ExecuteTaskSyncResponse + * @instance + */ + Object.defineProperty(ExecuteTaskSyncResponse.prototype, "res", { + get: $util.oneOfGetter($oneOfFields = ["header", "outputs"]), + set: $util.oneOfSetter($oneOfFields) + }); + + /** + * Creates a new ExecuteTaskSyncResponse instance using the specified properties. + * @function create + * @memberof flyteidl.admin.ExecuteTaskSyncResponse + * @static + * @param {flyteidl.admin.IExecuteTaskSyncResponse=} [properties] Properties to set + * @returns {flyteidl.admin.ExecuteTaskSyncResponse} ExecuteTaskSyncResponse instance + */ + ExecuteTaskSyncResponse.create = function create(properties) { + return new ExecuteTaskSyncResponse(properties); + }; + + /** + * Encodes the specified ExecuteTaskSyncResponse message. Does not implicitly {@link flyteidl.admin.ExecuteTaskSyncResponse.verify|verify} messages. + * @function encode + * @memberof flyteidl.admin.ExecuteTaskSyncResponse + * @static + * @param {flyteidl.admin.IExecuteTaskSyncResponse} message ExecuteTaskSyncResponse message or plain object to encode + * @param {$protobuf.Writer} [writer] Writer to encode to + * @returns {$protobuf.Writer} Writer + */ + ExecuteTaskSyncResponse.encode = function encode(message, writer) { + if (!writer) + writer = $Writer.create(); + if (message.header != null && message.hasOwnProperty("header")) + $root.flyteidl.admin.ExecuteTaskSyncResponseHeader.encode(message.header, writer.uint32(/* id 1, wireType 2 =*/10).fork()).ldelim(); + if (message.outputs != null && message.hasOwnProperty("outputs")) + $root.flyteidl.core.LiteralMap.encode(message.outputs, writer.uint32(/* id 2, wireType 2 =*/18).fork()).ldelim(); + return writer; + }; + + /** + * Decodes an ExecuteTaskSyncResponse message from the specified reader or buffer. + * @function decode + * @memberof flyteidl.admin.ExecuteTaskSyncResponse + * @static + * @param {$protobuf.Reader|Uint8Array} reader Reader or buffer to decode from + * @param {number} [length] Message length if known beforehand + * @returns {flyteidl.admin.ExecuteTaskSyncResponse} ExecuteTaskSyncResponse + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + ExecuteTaskSyncResponse.decode = function decode(reader, length) { + if (!(reader instanceof $Reader)) + reader = $Reader.create(reader); + var end = length === undefined ? reader.len : reader.pos + length, message = new $root.flyteidl.admin.ExecuteTaskSyncResponse(); + while (reader.pos < end) { + var tag = reader.uint32(); + switch (tag >>> 3) { + case 1: + message.header = $root.flyteidl.admin.ExecuteTaskSyncResponseHeader.decode(reader, reader.uint32()); + break; + case 2: + message.outputs = $root.flyteidl.core.LiteralMap.decode(reader, reader.uint32()); + break; + default: + reader.skipType(tag & 7); + break; + } + } + return message; + }; + + /** + * Verifies an ExecuteTaskSyncResponse message. + * @function verify + * @memberof flyteidl.admin.ExecuteTaskSyncResponse + * @static + * @param {Object.} message Plain object to verify + * @returns {string|null} `null` if valid, otherwise the reason why it is not + */ + ExecuteTaskSyncResponse.verify = function verify(message) { + if (typeof message !== "object" || message === null) + return "object expected"; + var properties = {}; + if (message.header != null && message.hasOwnProperty("header")) { + properties.res = 1; + { + var error = $root.flyteidl.admin.ExecuteTaskSyncResponseHeader.verify(message.header); + if (error) + return "header." + error; + } + } + if (message.outputs != null && message.hasOwnProperty("outputs")) { if (properties.res === 1) return "res: multiple values"; properties.res = 1; { - var error = $root.flyteidl.admin.Resource.verify(message.resource); + var error = $root.flyteidl.core.LiteralMap.verify(message.outputs); if (error) - return "resource." + error; + return "outputs." + error; } } return null; }; - return CreateTaskResponse; + return ExecuteTaskSyncResponse; })(); admin.GetTaskRequest = (function() { @@ -22620,6 +23234,7 @@ * @interface IGetTaskRequest * @property {string|null} [taskType] GetTaskRequest taskType * @property {Uint8Array|null} [resourceMeta] GetTaskRequest resourceMeta + * @property {flyteidl.admin.ITaskCategory|null} [taskCategory] GetTaskRequest taskCategory */ /** @@ -22653,6 +23268,14 @@ */ GetTaskRequest.prototype.resourceMeta = $util.newBuffer([]); + /** + * GetTaskRequest taskCategory. + * @member {flyteidl.admin.ITaskCategory|null|undefined} taskCategory + * @memberof flyteidl.admin.GetTaskRequest + * @instance + */ + GetTaskRequest.prototype.taskCategory = null; + /** * Creates a new GetTaskRequest instance using the specified properties. * @function create @@ -22681,6 +23304,8 @@ writer.uint32(/* id 1, wireType 2 =*/10).string(message.taskType); if (message.resourceMeta != null && message.hasOwnProperty("resourceMeta")) writer.uint32(/* id 2, wireType 2 =*/18).bytes(message.resourceMeta); + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) + $root.flyteidl.admin.TaskCategory.encode(message.taskCategory, writer.uint32(/* id 3, wireType 2 =*/26).fork()).ldelim(); return writer; }; @@ -22708,6 +23333,9 @@ case 2: message.resourceMeta = reader.bytes(); break; + case 3: + message.taskCategory = $root.flyteidl.admin.TaskCategory.decode(reader, reader.uint32()); + break; default: reader.skipType(tag & 7); break; @@ -22733,6 +23361,11 @@ if (message.resourceMeta != null && message.hasOwnProperty("resourceMeta")) if (!(message.resourceMeta && typeof message.resourceMeta.length === "number" || $util.isString(message.resourceMeta))) return "resourceMeta: buffer expected"; + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) { + var error = $root.flyteidl.admin.TaskCategory.verify(message.taskCategory); + if (error) + return "taskCategory." + error; + } return null; }; @@ -22746,7 +23379,6 @@ * @memberof flyteidl.admin * @interface IGetTaskResponse * @property {flyteidl.admin.IResource|null} [resource] GetTaskResponse resource - * @property {Array.|null} [logLinks] GetTaskResponse logLinks */ /** @@ -22758,7 +23390,6 @@ * @param {flyteidl.admin.IGetTaskResponse=} [properties] Properties to set */ function GetTaskResponse(properties) { - this.logLinks = []; if (properties) for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) if (properties[keys[i]] != null) @@ -22773,14 +23404,6 @@ */ GetTaskResponse.prototype.resource = null; - /** - * GetTaskResponse logLinks. - * @member {Array.} logLinks - * @memberof flyteidl.admin.GetTaskResponse - * @instance - */ - GetTaskResponse.prototype.logLinks = $util.emptyArray; - /** * Creates a new GetTaskResponse instance using the specified properties. * @function create @@ -22807,9 +23430,6 @@ writer = $Writer.create(); if (message.resource != null && message.hasOwnProperty("resource")) $root.flyteidl.admin.Resource.encode(message.resource, writer.uint32(/* id 1, wireType 2 =*/10).fork()).ldelim(); - if (message.logLinks != null && message.logLinks.length) - for (var i = 0; i < message.logLinks.length; ++i) - $root.flyteidl.core.TaskLog.encode(message.logLinks[i], writer.uint32(/* id 2, wireType 2 =*/18).fork()).ldelim(); return writer; }; @@ -22834,11 +23454,6 @@ case 1: message.resource = $root.flyteidl.admin.Resource.decode(reader, reader.uint32()); break; - case 2: - if (!(message.logLinks && message.logLinks.length)) - message.logLinks = []; - message.logLinks.push($root.flyteidl.core.TaskLog.decode(reader, reader.uint32())); - break; default: reader.skipType(tag & 7); break; @@ -22863,15 +23478,6 @@ if (error) return "resource." + error; } - if (message.logLinks != null && message.hasOwnProperty("logLinks")) { - if (!Array.isArray(message.logLinks)) - return "logLinks: array expected"; - for (var i = 0; i < message.logLinks.length; ++i) { - var error = $root.flyteidl.core.TaskLog.verify(message.logLinks[i]); - if (error) - return "logLinks." + error; - } - } return null; }; @@ -22889,6 +23495,7 @@ * @property {string|null} [message] Resource message * @property {Array.|null} [logLinks] Resource logLinks * @property {flyteidl.core.TaskExecution.Phase|null} [phase] Resource phase + * @property {google.protobuf.IStruct|null} [customInfo] Resource customInfo */ /** @@ -22947,6 +23554,14 @@ */ Resource.prototype.phase = 0; + /** + * Resource customInfo. + * @member {google.protobuf.IStruct|null|undefined} customInfo + * @memberof flyteidl.admin.Resource + * @instance + */ + Resource.prototype.customInfo = null; + /** * Creates a new Resource instance using the specified properties. * @function create @@ -22982,6 +23597,8 @@ $root.flyteidl.core.TaskLog.encode(message.logLinks[i], writer.uint32(/* id 4, wireType 2 =*/34).fork()).ldelim(); if (message.phase != null && message.hasOwnProperty("phase")) writer.uint32(/* id 5, wireType 0 =*/40).int32(message.phase); + if (message.customInfo != null && message.hasOwnProperty("customInfo")) + $root.google.protobuf.Struct.encode(message.customInfo, writer.uint32(/* id 6, wireType 2 =*/50).fork()).ldelim(); return writer; }; @@ -23020,6 +23637,9 @@ case 5: message.phase = reader.int32(); break; + case 6: + message.customInfo = $root.google.protobuf.Struct.decode(reader, reader.uint32()); + break; default: reader.skipType(tag & 7); break; @@ -23081,6 +23701,11 @@ case 7: break; } + if (message.customInfo != null && message.hasOwnProperty("customInfo")) { + var error = $root.google.protobuf.Struct.verify(message.customInfo); + if (error) + return "customInfo." + error; + } return null; }; @@ -23095,6 +23720,7 @@ * @interface IDeleteTaskRequest * @property {string|null} [taskType] DeleteTaskRequest taskType * @property {Uint8Array|null} [resourceMeta] DeleteTaskRequest resourceMeta + * @property {flyteidl.admin.ITaskCategory|null} [taskCategory] DeleteTaskRequest taskCategory */ /** @@ -23128,6 +23754,14 @@ */ DeleteTaskRequest.prototype.resourceMeta = $util.newBuffer([]); + /** + * DeleteTaskRequest taskCategory. + * @member {flyteidl.admin.ITaskCategory|null|undefined} taskCategory + * @memberof flyteidl.admin.DeleteTaskRequest + * @instance + */ + DeleteTaskRequest.prototype.taskCategory = null; + /** * Creates a new DeleteTaskRequest instance using the specified properties. * @function create @@ -23156,6 +23790,8 @@ writer.uint32(/* id 1, wireType 2 =*/10).string(message.taskType); if (message.resourceMeta != null && message.hasOwnProperty("resourceMeta")) writer.uint32(/* id 2, wireType 2 =*/18).bytes(message.resourceMeta); + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) + $root.flyteidl.admin.TaskCategory.encode(message.taskCategory, writer.uint32(/* id 3, wireType 2 =*/26).fork()).ldelim(); return writer; }; @@ -23183,6 +23819,9 @@ case 2: message.resourceMeta = reader.bytes(); break; + case 3: + message.taskCategory = $root.flyteidl.admin.TaskCategory.decode(reader, reader.uint32()); + break; default: reader.skipType(tag & 7); break; @@ -23208,6 +23847,11 @@ if (message.resourceMeta != null && message.hasOwnProperty("resourceMeta")) if (!(message.resourceMeta && typeof message.resourceMeta.length === "number" || $util.isString(message.resourceMeta))) return "resourceMeta: buffer expected"; + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) { + var error = $root.flyteidl.admin.TaskCategory.verify(message.taskCategory); + if (error) + return "taskCategory." + error; + } return null; }; @@ -23315,6 +23959,8 @@ * @interface IAgent * @property {string|null} [name] Agent name * @property {Array.|null} [supportedTaskTypes] Agent supportedTaskTypes + * @property {boolean|null} [isSync] Agent isSync + * @property {Array.|null} [supportedTaskCategories] Agent supportedTaskCategories */ /** @@ -23327,6 +23973,7 @@ */ function Agent(properties) { this.supportedTaskTypes = []; + this.supportedTaskCategories = []; if (properties) for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) if (properties[keys[i]] != null) @@ -23349,6 +23996,22 @@ */ Agent.prototype.supportedTaskTypes = $util.emptyArray; + /** + * Agent isSync. + * @member {boolean} isSync + * @memberof flyteidl.admin.Agent + * @instance + */ + Agent.prototype.isSync = false; + + /** + * Agent supportedTaskCategories. + * @member {Array.} supportedTaskCategories + * @memberof flyteidl.admin.Agent + * @instance + */ + Agent.prototype.supportedTaskCategories = $util.emptyArray; + /** * Creates a new Agent instance using the specified properties. * @function create @@ -23378,6 +24041,11 @@ if (message.supportedTaskTypes != null && message.supportedTaskTypes.length) for (var i = 0; i < message.supportedTaskTypes.length; ++i) writer.uint32(/* id 2, wireType 2 =*/18).string(message.supportedTaskTypes[i]); + if (message.isSync != null && message.hasOwnProperty("isSync")) + writer.uint32(/* id 3, wireType 0 =*/24).bool(message.isSync); + if (message.supportedTaskCategories != null && message.supportedTaskCategories.length) + for (var i = 0; i < message.supportedTaskCategories.length; ++i) + $root.flyteidl.admin.TaskCategory.encode(message.supportedTaskCategories[i], writer.uint32(/* id 4, wireType 2 =*/34).fork()).ldelim(); return writer; }; @@ -23407,6 +24075,14 @@ message.supportedTaskTypes = []; message.supportedTaskTypes.push(reader.string()); break; + case 3: + message.isSync = reader.bool(); + break; + case 4: + if (!(message.supportedTaskCategories && message.supportedTaskCategories.length)) + message.supportedTaskCategories = []; + message.supportedTaskCategories.push($root.flyteidl.admin.TaskCategory.decode(reader, reader.uint32())); + break; default: reader.skipType(tag & 7); break; @@ -23436,12 +24112,151 @@ if (!$util.isString(message.supportedTaskTypes[i])) return "supportedTaskTypes: string[] expected"; } + if (message.isSync != null && message.hasOwnProperty("isSync")) + if (typeof message.isSync !== "boolean") + return "isSync: boolean expected"; + if (message.supportedTaskCategories != null && message.hasOwnProperty("supportedTaskCategories")) { + if (!Array.isArray(message.supportedTaskCategories)) + return "supportedTaskCategories: array expected"; + for (var i = 0; i < message.supportedTaskCategories.length; ++i) { + var error = $root.flyteidl.admin.TaskCategory.verify(message.supportedTaskCategories[i]); + if (error) + return "supportedTaskCategories." + error; + } + } return null; }; return Agent; })(); + admin.TaskCategory = (function() { + + /** + * Properties of a TaskCategory. + * @memberof flyteidl.admin + * @interface ITaskCategory + * @property {string|null} [name] TaskCategory name + * @property {number|null} [version] TaskCategory version + */ + + /** + * Constructs a new TaskCategory. + * @memberof flyteidl.admin + * @classdesc Represents a TaskCategory. + * @implements ITaskCategory + * @constructor + * @param {flyteidl.admin.ITaskCategory=} [properties] Properties to set + */ + function TaskCategory(properties) { + if (properties) + for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) + if (properties[keys[i]] != null) + this[keys[i]] = properties[keys[i]]; + } + + /** + * TaskCategory name. + * @member {string} name + * @memberof flyteidl.admin.TaskCategory + * @instance + */ + TaskCategory.prototype.name = ""; + + /** + * TaskCategory version. + * @member {number} version + * @memberof flyteidl.admin.TaskCategory + * @instance + */ + TaskCategory.prototype.version = 0; + + /** + * Creates a new TaskCategory instance using the specified properties. + * @function create + * @memberof flyteidl.admin.TaskCategory + * @static + * @param {flyteidl.admin.ITaskCategory=} [properties] Properties to set + * @returns {flyteidl.admin.TaskCategory} TaskCategory instance + */ + TaskCategory.create = function create(properties) { + return new TaskCategory(properties); + }; + + /** + * Encodes the specified TaskCategory message. Does not implicitly {@link flyteidl.admin.TaskCategory.verify|verify} messages. + * @function encode + * @memberof flyteidl.admin.TaskCategory + * @static + * @param {flyteidl.admin.ITaskCategory} message TaskCategory message or plain object to encode + * @param {$protobuf.Writer} [writer] Writer to encode to + * @returns {$protobuf.Writer} Writer + */ + TaskCategory.encode = function encode(message, writer) { + if (!writer) + writer = $Writer.create(); + if (message.name != null && message.hasOwnProperty("name")) + writer.uint32(/* id 1, wireType 2 =*/10).string(message.name); + if (message.version != null && message.hasOwnProperty("version")) + writer.uint32(/* id 2, wireType 0 =*/16).int32(message.version); + return writer; + }; + + /** + * Decodes a TaskCategory message from the specified reader or buffer. + * @function decode + * @memberof flyteidl.admin.TaskCategory + * @static + * @param {$protobuf.Reader|Uint8Array} reader Reader or buffer to decode from + * @param {number} [length] Message length if known beforehand + * @returns {flyteidl.admin.TaskCategory} TaskCategory + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + TaskCategory.decode = function decode(reader, length) { + if (!(reader instanceof $Reader)) + reader = $Reader.create(reader); + var end = length === undefined ? reader.len : reader.pos + length, message = new $root.flyteidl.admin.TaskCategory(); + while (reader.pos < end) { + var tag = reader.uint32(); + switch (tag >>> 3) { + case 1: + message.name = reader.string(); + break; + case 2: + message.version = reader.int32(); + break; + default: + reader.skipType(tag & 7); + break; + } + } + return message; + }; + + /** + * Verifies a TaskCategory message. + * @function verify + * @memberof flyteidl.admin.TaskCategory + * @static + * @param {Object.} message Plain object to verify + * @returns {string|null} `null` if valid, otherwise the reason why it is not + */ + TaskCategory.verify = function verify(message) { + if (typeof message !== "object" || message === null) + return "object expected"; + if (message.name != null && message.hasOwnProperty("name")) + if (!$util.isString(message.name)) + return "name: string expected"; + if (message.version != null && message.hasOwnProperty("version")) + if (!$util.isInteger(message.version)) + return "version: integer expected"; + return null; + }; + + return TaskCategory; + })(); + admin.GetAgentRequest = (function() { /** @@ -23889,6 +24704,7 @@ * @property {google.protobuf.ITimestamp|null} [startTime] GetTaskMetricsRequest startTime * @property {google.protobuf.ITimestamp|null} [endTime] GetTaskMetricsRequest endTime * @property {google.protobuf.IDuration|null} [step] GetTaskMetricsRequest step + * @property {flyteidl.admin.ITaskCategory|null} [taskCategory] GetTaskMetricsRequest taskCategory */ /** @@ -23955,6 +24771,14 @@ */ GetTaskMetricsRequest.prototype.step = null; + /** + * GetTaskMetricsRequest taskCategory. + * @member {flyteidl.admin.ITaskCategory|null|undefined} taskCategory + * @memberof flyteidl.admin.GetTaskMetricsRequest + * @instance + */ + GetTaskMetricsRequest.prototype.taskCategory = null; + /** * Creates a new GetTaskMetricsRequest instance using the specified properties. * @function create @@ -23992,6 +24816,8 @@ $root.google.protobuf.Timestamp.encode(message.endTime, writer.uint32(/* id 5, wireType 2 =*/42).fork()).ldelim(); if (message.step != null && message.hasOwnProperty("step")) $root.google.protobuf.Duration.encode(message.step, writer.uint32(/* id 6, wireType 2 =*/50).fork()).ldelim(); + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) + $root.flyteidl.admin.TaskCategory.encode(message.taskCategory, writer.uint32(/* id 7, wireType 2 =*/58).fork()).ldelim(); return writer; }; @@ -24033,6 +24859,9 @@ case 6: message.step = $root.google.protobuf.Duration.decode(reader, reader.uint32()); break; + case 7: + message.taskCategory = $root.flyteidl.admin.TaskCategory.decode(reader, reader.uint32()); + break; default: reader.skipType(tag & 7); break; @@ -24080,6 +24909,11 @@ if (error) return "step." + error; } + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) { + var error = $root.flyteidl.admin.TaskCategory.verify(message.taskCategory); + if (error) + return "taskCategory." + error; + } return null; }; @@ -24216,6 +25050,7 @@ * @property {Uint8Array|null} [resourceMeta] GetTaskLogsRequest resourceMeta * @property {Long|null} [lines] GetTaskLogsRequest lines * @property {string|null} [token] GetTaskLogsRequest token + * @property {flyteidl.admin.ITaskCategory|null} [taskCategory] GetTaskLogsRequest taskCategory */ /** @@ -24265,6 +25100,14 @@ */ GetTaskLogsRequest.prototype.token = ""; + /** + * GetTaskLogsRequest taskCategory. + * @member {flyteidl.admin.ITaskCategory|null|undefined} taskCategory + * @memberof flyteidl.admin.GetTaskLogsRequest + * @instance + */ + GetTaskLogsRequest.prototype.taskCategory = null; + /** * Creates a new GetTaskLogsRequest instance using the specified properties. * @function create @@ -24297,6 +25140,8 @@ writer.uint32(/* id 3, wireType 0 =*/24).uint64(message.lines); if (message.token != null && message.hasOwnProperty("token")) writer.uint32(/* id 4, wireType 2 =*/34).string(message.token); + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) + $root.flyteidl.admin.TaskCategory.encode(message.taskCategory, writer.uint32(/* id 5, wireType 2 =*/42).fork()).ldelim(); return writer; }; @@ -24330,6 +25175,9 @@ case 4: message.token = reader.string(); break; + case 5: + message.taskCategory = $root.flyteidl.admin.TaskCategory.decode(reader, reader.uint32()); + break; default: reader.skipType(tag & 7); break; @@ -24361,20 +25209,253 @@ if (message.token != null && message.hasOwnProperty("token")) if (!$util.isString(message.token)) return "token: string expected"; + if (message.taskCategory != null && message.hasOwnProperty("taskCategory")) { + var error = $root.flyteidl.admin.TaskCategory.verify(message.taskCategory); + if (error) + return "taskCategory." + error; + } return null; }; return GetTaskLogsRequest; })(); + admin.GetTaskLogsResponseHeader = (function() { + + /** + * Properties of a GetTaskLogsResponseHeader. + * @memberof flyteidl.admin + * @interface IGetTaskLogsResponseHeader + * @property {string|null} [token] GetTaskLogsResponseHeader token + */ + + /** + * Constructs a new GetTaskLogsResponseHeader. + * @memberof flyteidl.admin + * @classdesc Represents a GetTaskLogsResponseHeader. + * @implements IGetTaskLogsResponseHeader + * @constructor + * @param {flyteidl.admin.IGetTaskLogsResponseHeader=} [properties] Properties to set + */ + function GetTaskLogsResponseHeader(properties) { + if (properties) + for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) + if (properties[keys[i]] != null) + this[keys[i]] = properties[keys[i]]; + } + + /** + * GetTaskLogsResponseHeader token. + * @member {string} token + * @memberof flyteidl.admin.GetTaskLogsResponseHeader + * @instance + */ + GetTaskLogsResponseHeader.prototype.token = ""; + + /** + * Creates a new GetTaskLogsResponseHeader instance using the specified properties. + * @function create + * @memberof flyteidl.admin.GetTaskLogsResponseHeader + * @static + * @param {flyteidl.admin.IGetTaskLogsResponseHeader=} [properties] Properties to set + * @returns {flyteidl.admin.GetTaskLogsResponseHeader} GetTaskLogsResponseHeader instance + */ + GetTaskLogsResponseHeader.create = function create(properties) { + return new GetTaskLogsResponseHeader(properties); + }; + + /** + * Encodes the specified GetTaskLogsResponseHeader message. Does not implicitly {@link flyteidl.admin.GetTaskLogsResponseHeader.verify|verify} messages. + * @function encode + * @memberof flyteidl.admin.GetTaskLogsResponseHeader + * @static + * @param {flyteidl.admin.IGetTaskLogsResponseHeader} message GetTaskLogsResponseHeader message or plain object to encode + * @param {$protobuf.Writer} [writer] Writer to encode to + * @returns {$protobuf.Writer} Writer + */ + GetTaskLogsResponseHeader.encode = function encode(message, writer) { + if (!writer) + writer = $Writer.create(); + if (message.token != null && message.hasOwnProperty("token")) + writer.uint32(/* id 1, wireType 2 =*/10).string(message.token); + return writer; + }; + + /** + * Decodes a GetTaskLogsResponseHeader message from the specified reader or buffer. + * @function decode + * @memberof flyteidl.admin.GetTaskLogsResponseHeader + * @static + * @param {$protobuf.Reader|Uint8Array} reader Reader or buffer to decode from + * @param {number} [length] Message length if known beforehand + * @returns {flyteidl.admin.GetTaskLogsResponseHeader} GetTaskLogsResponseHeader + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + GetTaskLogsResponseHeader.decode = function decode(reader, length) { + if (!(reader instanceof $Reader)) + reader = $Reader.create(reader); + var end = length === undefined ? reader.len : reader.pos + length, message = new $root.flyteidl.admin.GetTaskLogsResponseHeader(); + while (reader.pos < end) { + var tag = reader.uint32(); + switch (tag >>> 3) { + case 1: + message.token = reader.string(); + break; + default: + reader.skipType(tag & 7); + break; + } + } + return message; + }; + + /** + * Verifies a GetTaskLogsResponseHeader message. + * @function verify + * @memberof flyteidl.admin.GetTaskLogsResponseHeader + * @static + * @param {Object.} message Plain object to verify + * @returns {string|null} `null` if valid, otherwise the reason why it is not + */ + GetTaskLogsResponseHeader.verify = function verify(message) { + if (typeof message !== "object" || message === null) + return "object expected"; + if (message.token != null && message.hasOwnProperty("token")) + if (!$util.isString(message.token)) + return "token: string expected"; + return null; + }; + + return GetTaskLogsResponseHeader; + })(); + + admin.GetTaskLogsResponseBody = (function() { + + /** + * Properties of a GetTaskLogsResponseBody. + * @memberof flyteidl.admin + * @interface IGetTaskLogsResponseBody + * @property {Array.|null} [results] GetTaskLogsResponseBody results + */ + + /** + * Constructs a new GetTaskLogsResponseBody. + * @memberof flyteidl.admin + * @classdesc Represents a GetTaskLogsResponseBody. + * @implements IGetTaskLogsResponseBody + * @constructor + * @param {flyteidl.admin.IGetTaskLogsResponseBody=} [properties] Properties to set + */ + function GetTaskLogsResponseBody(properties) { + this.results = []; + if (properties) + for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) + if (properties[keys[i]] != null) + this[keys[i]] = properties[keys[i]]; + } + + /** + * GetTaskLogsResponseBody results. + * @member {Array.} results + * @memberof flyteidl.admin.GetTaskLogsResponseBody + * @instance + */ + GetTaskLogsResponseBody.prototype.results = $util.emptyArray; + + /** + * Creates a new GetTaskLogsResponseBody instance using the specified properties. + * @function create + * @memberof flyteidl.admin.GetTaskLogsResponseBody + * @static + * @param {flyteidl.admin.IGetTaskLogsResponseBody=} [properties] Properties to set + * @returns {flyteidl.admin.GetTaskLogsResponseBody} GetTaskLogsResponseBody instance + */ + GetTaskLogsResponseBody.create = function create(properties) { + return new GetTaskLogsResponseBody(properties); + }; + + /** + * Encodes the specified GetTaskLogsResponseBody message. Does not implicitly {@link flyteidl.admin.GetTaskLogsResponseBody.verify|verify} messages. + * @function encode + * @memberof flyteidl.admin.GetTaskLogsResponseBody + * @static + * @param {flyteidl.admin.IGetTaskLogsResponseBody} message GetTaskLogsResponseBody message or plain object to encode + * @param {$protobuf.Writer} [writer] Writer to encode to + * @returns {$protobuf.Writer} Writer + */ + GetTaskLogsResponseBody.encode = function encode(message, writer) { + if (!writer) + writer = $Writer.create(); + if (message.results != null && message.results.length) + for (var i = 0; i < message.results.length; ++i) + writer.uint32(/* id 1, wireType 2 =*/10).string(message.results[i]); + return writer; + }; + + /** + * Decodes a GetTaskLogsResponseBody message from the specified reader or buffer. + * @function decode + * @memberof flyteidl.admin.GetTaskLogsResponseBody + * @static + * @param {$protobuf.Reader|Uint8Array} reader Reader or buffer to decode from + * @param {number} [length] Message length if known beforehand + * @returns {flyteidl.admin.GetTaskLogsResponseBody} GetTaskLogsResponseBody + * @throws {Error} If the payload is not a reader or valid buffer + * @throws {$protobuf.util.ProtocolError} If required fields are missing + */ + GetTaskLogsResponseBody.decode = function decode(reader, length) { + if (!(reader instanceof $Reader)) + reader = $Reader.create(reader); + var end = length === undefined ? reader.len : reader.pos + length, message = new $root.flyteidl.admin.GetTaskLogsResponseBody(); + while (reader.pos < end) { + var tag = reader.uint32(); + switch (tag >>> 3) { + case 1: + if (!(message.results && message.results.length)) + message.results = []; + message.results.push(reader.string()); + break; + default: + reader.skipType(tag & 7); + break; + } + } + return message; + }; + + /** + * Verifies a GetTaskLogsResponseBody message. + * @function verify + * @memberof flyteidl.admin.GetTaskLogsResponseBody + * @static + * @param {Object.} message Plain object to verify + * @returns {string|null} `null` if valid, otherwise the reason why it is not + */ + GetTaskLogsResponseBody.verify = function verify(message) { + if (typeof message !== "object" || message === null) + return "object expected"; + if (message.results != null && message.hasOwnProperty("results")) { + if (!Array.isArray(message.results)) + return "results: array expected"; + for (var i = 0; i < message.results.length; ++i) + if (!$util.isString(message.results[i])) + return "results: string[] expected"; + } + return null; + }; + + return GetTaskLogsResponseBody; + })(); + admin.GetTaskLogsResponse = (function() { /** * Properties of a GetTaskLogsResponse. * @memberof flyteidl.admin * @interface IGetTaskLogsResponse - * @property {Array.|null} [results] GetTaskLogsResponse results - * @property {string|null} [token] GetTaskLogsResponse token + * @property {flyteidl.admin.IGetTaskLogsResponseHeader|null} [header] GetTaskLogsResponse header + * @property {flyteidl.admin.IGetTaskLogsResponseBody|null} [body] GetTaskLogsResponse body */ /** @@ -24386,7 +25467,6 @@ * @param {flyteidl.admin.IGetTaskLogsResponse=} [properties] Properties to set */ function GetTaskLogsResponse(properties) { - this.results = []; if (properties) for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i) if (properties[keys[i]] != null) @@ -24394,20 +25474,34 @@ } /** - * GetTaskLogsResponse results. - * @member {Array.} results + * GetTaskLogsResponse header. + * @member {flyteidl.admin.IGetTaskLogsResponseHeader|null|undefined} header * @memberof flyteidl.admin.GetTaskLogsResponse * @instance */ - GetTaskLogsResponse.prototype.results = $util.emptyArray; + GetTaskLogsResponse.prototype.header = null; /** - * GetTaskLogsResponse token. - * @member {string} token + * GetTaskLogsResponse body. + * @member {flyteidl.admin.IGetTaskLogsResponseBody|null|undefined} body + * @memberof flyteidl.admin.GetTaskLogsResponse + * @instance + */ + GetTaskLogsResponse.prototype.body = null; + + // OneOf field names bound to virtual getters and setters + var $oneOfFields; + + /** + * GetTaskLogsResponse part. + * @member {"header"|"body"|undefined} part * @memberof flyteidl.admin.GetTaskLogsResponse * @instance */ - GetTaskLogsResponse.prototype.token = ""; + Object.defineProperty(GetTaskLogsResponse.prototype, "part", { + get: $util.oneOfGetter($oneOfFields = ["header", "body"]), + set: $util.oneOfSetter($oneOfFields) + }); /** * Creates a new GetTaskLogsResponse instance using the specified properties. @@ -24433,11 +25527,10 @@ GetTaskLogsResponse.encode = function encode(message, writer) { if (!writer) writer = $Writer.create(); - if (message.results != null && message.results.length) - for (var i = 0; i < message.results.length; ++i) - writer.uint32(/* id 1, wireType 2 =*/10).string(message.results[i]); - if (message.token != null && message.hasOwnProperty("token")) - writer.uint32(/* id 2, wireType 2 =*/18).string(message.token); + if (message.header != null && message.hasOwnProperty("header")) + $root.flyteidl.admin.GetTaskLogsResponseHeader.encode(message.header, writer.uint32(/* id 1, wireType 2 =*/10).fork()).ldelim(); + if (message.body != null && message.hasOwnProperty("body")) + $root.flyteidl.admin.GetTaskLogsResponseBody.encode(message.body, writer.uint32(/* id 2, wireType 2 =*/18).fork()).ldelim(); return writer; }; @@ -24460,12 +25553,10 @@ var tag = reader.uint32(); switch (tag >>> 3) { case 1: - if (!(message.results && message.results.length)) - message.results = []; - message.results.push(reader.string()); + message.header = $root.flyteidl.admin.GetTaskLogsResponseHeader.decode(reader, reader.uint32()); break; case 2: - message.token = reader.string(); + message.body = $root.flyteidl.admin.GetTaskLogsResponseBody.decode(reader, reader.uint32()); break; default: reader.skipType(tag & 7); @@ -24486,16 +25577,25 @@ GetTaskLogsResponse.verify = function verify(message) { if (typeof message !== "object" || message === null) return "object expected"; - if (message.results != null && message.hasOwnProperty("results")) { - if (!Array.isArray(message.results)) - return "results: array expected"; - for (var i = 0; i < message.results.length; ++i) - if (!$util.isString(message.results[i])) - return "results: string[] expected"; + var properties = {}; + if (message.header != null && message.hasOwnProperty("header")) { + properties.part = 1; + { + var error = $root.flyteidl.admin.GetTaskLogsResponseHeader.verify(message.header); + if (error) + return "header." + error; + } + } + if (message.body != null && message.hasOwnProperty("body")) { + if (properties.part === 1) + return "part: multiple values"; + properties.part = 1; + { + var error = $root.flyteidl.admin.GetTaskLogsResponseBody.verify(message.body); + if (error) + return "body." + error; + } } - if (message.token != null && message.hasOwnProperty("token")) - if (!$util.isString(message.token)) - return "token: string expected"; return null; }; @@ -32009,6 +33109,7 @@ case 3: case 4: case 5: + case 6: break; } if (message.principal != null && message.hasOwnProperty("principal")) @@ -32059,6 +33160,7 @@ * @property {number} RELAUNCH=3 RELAUNCH value * @property {number} CHILD_WORKFLOW=4 CHILD_WORKFLOW value * @property {number} RECOVERED=5 RECOVERED value + * @property {number} TRIGGER=6 TRIGGER value */ ExecutionMetadata.ExecutionMode = (function() { var valuesById = {}, values = Object.create(valuesById); @@ -32068,6 +33170,7 @@ values[valuesById[3] = "RELAUNCH"] = 3; values[valuesById[4] = "CHILD_WORKFLOW"] = 4; values[valuesById[5] = "RECOVERED"] = 5; + values[valuesById[6] = "TRIGGER"] = 6; return values; })(); @@ -50746,6 +51849,74 @@ return AdminService; })(); + service.SyncAgentService = (function() { + + /** + * Constructs a new SyncAgentService service. + * @memberof flyteidl.service + * @classdesc Represents a SyncAgentService + * @extends $protobuf.rpc.Service + * @constructor + * @param {$protobuf.RPCImpl} rpcImpl RPC implementation + * @param {boolean} [requestDelimited=false] Whether requests are length-delimited + * @param {boolean} [responseDelimited=false] Whether responses are length-delimited + */ + function SyncAgentService(rpcImpl, requestDelimited, responseDelimited) { + $protobuf.rpc.Service.call(this, rpcImpl, requestDelimited, responseDelimited); + } + + (SyncAgentService.prototype = Object.create($protobuf.rpc.Service.prototype)).constructor = SyncAgentService; + + /** + * Creates new SyncAgentService service using the specified rpc implementation. + * @function create + * @memberof flyteidl.service.SyncAgentService + * @static + * @param {$protobuf.RPCImpl} rpcImpl RPC implementation + * @param {boolean} [requestDelimited=false] Whether requests are length-delimited + * @param {boolean} [responseDelimited=false] Whether responses are length-delimited + * @returns {SyncAgentService} RPC service. Useful where requests and/or responses are streamed. + */ + SyncAgentService.create = function create(rpcImpl, requestDelimited, responseDelimited) { + return new this(rpcImpl, requestDelimited, responseDelimited); + }; + + /** + * Callback as used by {@link flyteidl.service.SyncAgentService#executeTaskSync}. + * @memberof flyteidl.service.SyncAgentService + * @typedef ExecuteTaskSyncCallback + * @type {function} + * @param {Error|null} error Error, if any + * @param {flyteidl.admin.ExecuteTaskSyncResponse} [response] ExecuteTaskSyncResponse + */ + + /** + * Calls ExecuteTaskSync. + * @function executeTaskSync + * @memberof flyteidl.service.SyncAgentService + * @instance + * @param {flyteidl.admin.IExecuteTaskSyncRequest} request ExecuteTaskSyncRequest message or plain object + * @param {flyteidl.service.SyncAgentService.ExecuteTaskSyncCallback} callback Node-style callback called with the error, if any, and ExecuteTaskSyncResponse + * @returns {undefined} + * @variation 1 + */ + Object.defineProperty(SyncAgentService.prototype.executeTaskSync = function executeTaskSync(request, callback) { + return this.rpcCall(executeTaskSync, $root.flyteidl.admin.ExecuteTaskSyncRequest, $root.flyteidl.admin.ExecuteTaskSyncResponse, request, callback); + }, "name", { value: "ExecuteTaskSync" }); + + /** + * Calls ExecuteTaskSync. + * @function executeTaskSync + * @memberof flyteidl.service.SyncAgentService + * @instance + * @param {flyteidl.admin.IExecuteTaskSyncRequest} request ExecuteTaskSyncRequest message or plain object + * @returns {Promise} Promise + * @variation 2 + */ + + return SyncAgentService; + })(); + service.AsyncAgentService = (function() { /** diff --git a/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.py b/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.py index f23ad50819..950a916662 100644 --- a/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.py +++ b/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.py @@ -13,14 +13,16 @@ from flyteidl.core import literals_pb2 as flyteidl_dot_core_dot_literals__pb2 from flyteidl.core import tasks_pb2 as flyteidl_dot_core_dot_tasks__pb2 +from flyteidl.core import workflow_pb2 as flyteidl_dot_core_dot_workflow__pb2 from flyteidl.core import identifier_pb2 as flyteidl_dot_core_dot_identifier__pb2 from flyteidl.core import execution_pb2 as flyteidl_dot_core_dot_execution__pb2 from flyteidl.core import metrics_pb2 as flyteidl_dot_core_dot_metrics__pb2 from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2 from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2 +from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1a\x66lyteidl/admin/agent.proto\x12\x0e\x66lyteidl.admin\x1a\x1c\x66lyteidl/core/literals.proto\x1a\x19\x66lyteidl/core/tasks.proto\x1a\x1e\x66lyteidl/core/identifier.proto\x1a\x1d\x66lyteidl/core/execution.proto\x1a\x1b\x66lyteidl/core/metrics.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"\x98\x05\n\x15TaskExecutionMetadata\x12R\n\x11task_execution_id\x18\x01 \x01(\x0b\x32&.flyteidl.core.TaskExecutionIdentifierR\x0ftaskExecutionId\x12\x1c\n\tnamespace\x18\x02 \x01(\tR\tnamespace\x12I\n\x06labels\x18\x03 \x03(\x0b\x32\x31.flyteidl.admin.TaskExecutionMetadata.LabelsEntryR\x06labels\x12X\n\x0b\x61nnotations\x18\x04 \x03(\x0b\x32\x36.flyteidl.admin.TaskExecutionMetadata.AnnotationsEntryR\x0b\x61nnotations\x12.\n\x13k8s_service_account\x18\x05 \x01(\tR\x11k8sServiceAccount\x12t\n\x15\x65nvironment_variables\x18\x06 \x03(\x0b\x32?.flyteidl.admin.TaskExecutionMetadata.EnvironmentVariablesEntryR\x14\x65nvironmentVariables\x1a\x39\n\x0bLabelsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1a>\n\x10\x41nnotationsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1aG\n\x19\x45nvironmentVariablesEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\"\x83\x02\n\x11\x43reateTaskRequest\x12\x31\n\x06inputs\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x06inputs\x12\x37\n\x08template\x18\x02 \x01(\x0b\x32\x1b.flyteidl.core.TaskTemplateR\x08template\x12#\n\routput_prefix\x18\x03 \x01(\tR\x0coutputPrefix\x12]\n\x17task_execution_metadata\x18\x04 \x01(\x0b\x32%.flyteidl.admin.TaskExecutionMetadataR\x15taskExecutionMetadata\"z\n\x12\x43reateTaskResponse\x12%\n\rresource_meta\x18\x01 \x01(\x0cH\x00R\x0cresourceMeta\x12\x36\n\x08resource\x18\x02 \x01(\x0b\x32\x18.flyteidl.admin.ResourceH\x00R\x08resourceB\x05\n\x03res\"R\n\x0eGetTaskRequest\x12\x1b\n\ttask_type\x18\x01 \x01(\tR\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\"|\n\x0fGetTaskResponse\x12\x34\n\x08resource\x18\x01 \x01(\x0b\x32\x18.flyteidl.admin.ResourceR\x08resource\x12\x33\n\tlog_links\x18\x02 \x03(\x0b\x32\x16.flyteidl.core.TaskLogR\x08logLinks\"\xf9\x01\n\x08Resource\x12/\n\x05state\x18\x01 \x01(\x0e\x32\x15.flyteidl.admin.StateB\x02\x18\x01R\x05state\x12\x33\n\x07outputs\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x07outputs\x12\x18\n\x07message\x18\x03 \x01(\tR\x07message\x12\x33\n\tlog_links\x18\x04 \x03(\x0b\x32\x16.flyteidl.core.TaskLogR\x08logLinks\x12\x38\n\x05phase\x18\x05 \x01(\x0e\x32\".flyteidl.core.TaskExecution.PhaseR\x05phase\"U\n\x11\x44\x65leteTaskRequest\x12\x1b\n\ttask_type\x18\x01 \x01(\tR\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\"\x14\n\x12\x44\x65leteTaskResponse\"M\n\x05\x41gent\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x30\n\x14supported_task_types\x18\x02 \x03(\tR\x12supportedTaskTypes\"%\n\x0fGetAgentRequest\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\"?\n\x10GetAgentResponse\x12+\n\x05\x61gent\x18\x01 \x01(\x0b\x32\x15.flyteidl.admin.AgentR\x05\x61gent\"\x13\n\x11ListAgentsRequest\"C\n\x12ListAgentsResponse\x12-\n\x06\x61gents\x18\x01 \x03(\x0b\x32\x15.flyteidl.admin.AgentR\x06\x61gents\"\x94\x02\n\x15GetTaskMetricsRequest\x12\x1b\n\ttask_type\x18\x01 \x01(\tR\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\x12\x18\n\x07queries\x18\x03 \x03(\tR\x07queries\x12\x39\n\nstart_time\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tstartTime\x12\x35\n\x08\x65nd_time\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\x07\x65ndTime\x12-\n\x04step\x18\x06 \x01(\x0b\x32\x19.google.protobuf.DurationR\x04step\"X\n\x16GetTaskMetricsResponse\x12>\n\x07results\x18\x01 \x03(\x0b\x32$.flyteidl.core.ExecutionMetricResultR\x07results\"\x82\x01\n\x12GetTaskLogsRequest\x12\x1b\n\ttask_type\x18\x01 \x01(\tR\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\x12\x14\n\x05lines\x18\x03 \x01(\x04R\x05lines\x12\x14\n\x05token\x18\x04 \x01(\tR\x05token\"E\n\x13GetTaskLogsResponse\x12\x18\n\x07results\x18\x01 \x03(\tR\x07results\x12\x14\n\x05token\x18\x02 \x01(\tR\x05token*^\n\x05State\x12\x15\n\x11RETRYABLE_FAILURE\x10\x00\x12\x15\n\x11PERMANENT_FAILURE\x10\x01\x12\x0b\n\x07PENDING\x10\x02\x12\x0b\n\x07RUNNING\x10\x03\x12\r\n\tSUCCEEDED\x10\x04\x42\xb6\x01\n\x12\x63om.flyteidl.adminB\nAgentProtoP\x01Z;github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin\xa2\x02\x03\x46\x41X\xaa\x02\x0e\x46lyteidl.Admin\xca\x02\x0e\x46lyteidl\\Admin\xe2\x02\x1a\x46lyteidl\\Admin\\GPBMetadata\xea\x02\x0f\x46lyteidl::Adminb\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1a\x66lyteidl/admin/agent.proto\x12\x0e\x66lyteidl.admin\x1a\x1c\x66lyteidl/core/literals.proto\x1a\x19\x66lyteidl/core/tasks.proto\x1a\x1c\x66lyteidl/core/workflow.proto\x1a\x1e\x66lyteidl/core/identifier.proto\x1a\x1d\x66lyteidl/core/execution.proto\x1a\x1b\x66lyteidl/core/metrics.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\"\xe9\x06\n\x15TaskExecutionMetadata\x12R\n\x11task_execution_id\x18\x01 \x01(\x0b\x32&.flyteidl.core.TaskExecutionIdentifierR\x0ftaskExecutionId\x12\x1c\n\tnamespace\x18\x02 \x01(\tR\tnamespace\x12I\n\x06labels\x18\x03 \x03(\x0b\x32\x31.flyteidl.admin.TaskExecutionMetadata.LabelsEntryR\x06labels\x12X\n\x0b\x61nnotations\x18\x04 \x03(\x0b\x32\x36.flyteidl.admin.TaskExecutionMetadata.AnnotationsEntryR\x0b\x61nnotations\x12.\n\x13k8s_service_account\x18\x05 \x01(\tR\x11k8sServiceAccount\x12t\n\x15\x65nvironment_variables\x18\x06 \x03(\x0b\x32?.flyteidl.admin.TaskExecutionMetadata.EnvironmentVariablesEntryR\x14\x65nvironmentVariables\x12!\n\x0cmax_attempts\x18\x07 \x01(\x05R\x0bmaxAttempts\x12$\n\rinterruptible\x18\x08 \x01(\x08R\rinterruptible\x12\x46\n\x1finterruptible_failure_threshold\x18\t \x01(\x05R\x1dinterruptibleFailureThreshold\x12>\n\toverrides\x18\n \x01(\x0b\x32 .flyteidl.core.TaskNodeOverridesR\toverrides\x1a\x39\n\x0bLabelsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1a>\n\x10\x41nnotationsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\x1aG\n\x19\x45nvironmentVariablesEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\"\x83\x02\n\x11\x43reateTaskRequest\x12\x31\n\x06inputs\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x06inputs\x12\x37\n\x08template\x18\x02 \x01(\x0b\x32\x1b.flyteidl.core.TaskTemplateR\x08template\x12#\n\routput_prefix\x18\x03 \x01(\tR\x0coutputPrefix\x12]\n\x17task_execution_metadata\x18\x04 \x01(\x0b\x32%.flyteidl.admin.TaskExecutionMetadataR\x15taskExecutionMetadata\"9\n\x12\x43reateTaskResponse\x12#\n\rresource_meta\x18\x01 \x01(\x0cR\x0cresourceMeta\"\x87\x02\n\x13\x43reateRequestHeader\x12\x37\n\x08template\x18\x01 \x01(\x0b\x32\x1b.flyteidl.core.TaskTemplateR\x08template\x12#\n\routput_prefix\x18\x02 \x01(\tR\x0coutputPrefix\x12]\n\x17task_execution_metadata\x18\x03 \x01(\x0b\x32%.flyteidl.admin.TaskExecutionMetadataR\x15taskExecutionMetadata\x12\x33\n\x16max_dataset_size_bytes\x18\x04 \x01(\x03R\x13maxDatasetSizeBytes\"\x94\x01\n\x16\x45xecuteTaskSyncRequest\x12=\n\x06header\x18\x01 \x01(\x0b\x32#.flyteidl.admin.CreateRequestHeaderH\x00R\x06header\x12\x33\n\x06inputs\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapH\x00R\x06inputsB\x06\n\x04part\"U\n\x1d\x45xecuteTaskSyncResponseHeader\x12\x34\n\x08resource\x18\x01 \x01(\x0b\x32\x18.flyteidl.admin.ResourceR\x08resource\"\xa0\x01\n\x17\x45xecuteTaskSyncResponse\x12G\n\x06header\x18\x01 \x01(\x0b\x32-.flyteidl.admin.ExecuteTaskSyncResponseHeaderH\x00R\x06header\x12\x35\n\x07outputs\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapH\x00R\x07outputsB\x05\n\x03res\"\x99\x01\n\x0eGetTaskRequest\x12\x1f\n\ttask_type\x18\x01 \x01(\tB\x02\x18\x01R\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\x12\x41\n\rtask_category\x18\x03 \x01(\x0b\x32\x1c.flyteidl.admin.TaskCategoryR\x0ctaskCategory\"G\n\x0fGetTaskResponse\x12\x34\n\x08resource\x18\x01 \x01(\x0b\x32\x18.flyteidl.admin.ResourceR\x08resource\"\xb3\x02\n\x08Resource\x12/\n\x05state\x18\x01 \x01(\x0e\x32\x15.flyteidl.admin.StateB\x02\x18\x01R\x05state\x12\x33\n\x07outputs\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x07outputs\x12\x18\n\x07message\x18\x03 \x01(\tR\x07message\x12\x33\n\tlog_links\x18\x04 \x03(\x0b\x32\x16.flyteidl.core.TaskLogR\x08logLinks\x12\x38\n\x05phase\x18\x05 \x01(\x0e\x32\".flyteidl.core.TaskExecution.PhaseR\x05phase\x12\x38\n\x0b\x63ustom_info\x18\x06 \x01(\x0b\x32\x17.google.protobuf.StructR\ncustomInfo\"\x9c\x01\n\x11\x44\x65leteTaskRequest\x12\x1f\n\ttask_type\x18\x01 \x01(\tB\x02\x18\x01R\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\x12\x41\n\rtask_category\x18\x03 \x01(\x0b\x32\x1c.flyteidl.admin.TaskCategoryR\x0ctaskCategory\"\x14\n\x12\x44\x65leteTaskResponse\"\xc4\x01\n\x05\x41gent\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x34\n\x14supported_task_types\x18\x02 \x03(\tB\x02\x18\x01R\x12supportedTaskTypes\x12\x17\n\x07is_sync\x18\x03 \x01(\x08R\x06isSync\x12X\n\x19supported_task_categories\x18\x04 \x03(\x0b\x32\x1c.flyteidl.admin.TaskCategoryR\x17supportedTaskCategories\"<\n\x0cTaskCategory\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x18\n\x07version\x18\x02 \x01(\x05R\x07version\"%\n\x0fGetAgentRequest\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\"?\n\x10GetAgentResponse\x12+\n\x05\x61gent\x18\x01 \x01(\x0b\x32\x15.flyteidl.admin.AgentR\x05\x61gent\"\x13\n\x11ListAgentsRequest\"C\n\x12ListAgentsResponse\x12-\n\x06\x61gents\x18\x01 \x03(\x0b\x32\x15.flyteidl.admin.AgentR\x06\x61gents\"\xdb\x02\n\x15GetTaskMetricsRequest\x12\x1f\n\ttask_type\x18\x01 \x01(\tB\x02\x18\x01R\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\x12\x18\n\x07queries\x18\x03 \x03(\tR\x07queries\x12\x39\n\nstart_time\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tstartTime\x12\x35\n\x08\x65nd_time\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\x07\x65ndTime\x12-\n\x04step\x18\x06 \x01(\x0b\x32\x19.google.protobuf.DurationR\x04step\x12\x41\n\rtask_category\x18\x07 \x01(\x0b\x32\x1c.flyteidl.admin.TaskCategoryR\x0ctaskCategory\"X\n\x16GetTaskMetricsResponse\x12>\n\x07results\x18\x01 \x03(\x0b\x32$.flyteidl.core.ExecutionMetricResultR\x07results\"\xc9\x01\n\x12GetTaskLogsRequest\x12\x1f\n\ttask_type\x18\x01 \x01(\tB\x02\x18\x01R\x08taskType\x12#\n\rresource_meta\x18\x02 \x01(\x0cR\x0cresourceMeta\x12\x14\n\x05lines\x18\x03 \x01(\x04R\x05lines\x12\x14\n\x05token\x18\x04 \x01(\tR\x05token\x12\x41\n\rtask_category\x18\x05 \x01(\x0b\x32\x1c.flyteidl.admin.TaskCategoryR\x0ctaskCategory\"1\n\x19GetTaskLogsResponseHeader\x12\x14\n\x05token\x18\x01 \x01(\tR\x05token\"3\n\x17GetTaskLogsResponseBody\x12\x18\n\x07results\x18\x01 \x03(\tR\x07results\"\xa1\x01\n\x13GetTaskLogsResponse\x12\x43\n\x06header\x18\x01 \x01(\x0b\x32).flyteidl.admin.GetTaskLogsResponseHeaderH\x00R\x06header\x12=\n\x04\x62ody\x18\x02 \x01(\x0b\x32\'.flyteidl.admin.GetTaskLogsResponseBodyH\x00R\x04\x62odyB\x06\n\x04part*b\n\x05State\x12\x15\n\x11RETRYABLE_FAILURE\x10\x00\x12\x15\n\x11PERMANENT_FAILURE\x10\x01\x12\x0b\n\x07PENDING\x10\x02\x12\x0b\n\x07RUNNING\x10\x03\x12\r\n\tSUCCEEDED\x10\x04\x1a\x02\x18\x01\x42\xb6\x01\n\x12\x63om.flyteidl.adminB\nAgentProtoP\x01Z;github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin\xa2\x02\x03\x46\x41X\xaa\x02\x0e\x46lyteidl.Admin\xca\x02\x0e\x46lyteidl\\Admin\xe2\x02\x1a\x46lyteidl\\Admin\\GPBMetadata\xea\x02\x0f\x46lyteidl::Adminb\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -29,54 +31,80 @@ DESCRIPTOR._options = None DESCRIPTOR._serialized_options = b'\n\022com.flyteidl.adminB\nAgentProtoP\001Z;github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin\242\002\003FAX\252\002\016Flyteidl.Admin\312\002\016Flyteidl\\Admin\342\002\032Flyteidl\\Admin\\GPBMetadata\352\002\017Flyteidl::Admin' + _STATE._options = None + _STATE._serialized_options = b'\030\001' _TASKEXECUTIONMETADATA_LABELSENTRY._options = None _TASKEXECUTIONMETADATA_LABELSENTRY._serialized_options = b'8\001' _TASKEXECUTIONMETADATA_ANNOTATIONSENTRY._options = None _TASKEXECUTIONMETADATA_ANNOTATIONSENTRY._serialized_options = b'8\001' _TASKEXECUTIONMETADATA_ENVIRONMENTVARIABLESENTRY._options = None _TASKEXECUTIONMETADATA_ENVIRONMENTVARIABLESENTRY._serialized_options = b'8\001' + _GETTASKREQUEST.fields_by_name['task_type']._options = None + _GETTASKREQUEST.fields_by_name['task_type']._serialized_options = b'\030\001' _RESOURCE.fields_by_name['state']._options = None _RESOURCE.fields_by_name['state']._serialized_options = b'\030\001' - _globals['_STATE']._serialized_start=2730 - _globals['_STATE']._serialized_end=2824 - _globals['_TASKEXECUTIONMETADATA']._serialized_start=261 - _globals['_TASKEXECUTIONMETADATA']._serialized_end=925 - _globals['_TASKEXECUTIONMETADATA_LABELSENTRY']._serialized_start=731 - _globals['_TASKEXECUTIONMETADATA_LABELSENTRY']._serialized_end=788 - _globals['_TASKEXECUTIONMETADATA_ANNOTATIONSENTRY']._serialized_start=790 - _globals['_TASKEXECUTIONMETADATA_ANNOTATIONSENTRY']._serialized_end=852 - _globals['_TASKEXECUTIONMETADATA_ENVIRONMENTVARIABLESENTRY']._serialized_start=854 - _globals['_TASKEXECUTIONMETADATA_ENVIRONMENTVARIABLESENTRY']._serialized_end=925 - _globals['_CREATETASKREQUEST']._serialized_start=928 - _globals['_CREATETASKREQUEST']._serialized_end=1187 - _globals['_CREATETASKRESPONSE']._serialized_start=1189 - _globals['_CREATETASKRESPONSE']._serialized_end=1311 - _globals['_GETTASKREQUEST']._serialized_start=1313 - _globals['_GETTASKREQUEST']._serialized_end=1395 - _globals['_GETTASKRESPONSE']._serialized_start=1397 - _globals['_GETTASKRESPONSE']._serialized_end=1521 - _globals['_RESOURCE']._serialized_start=1524 - _globals['_RESOURCE']._serialized_end=1773 - _globals['_DELETETASKREQUEST']._serialized_start=1775 - _globals['_DELETETASKREQUEST']._serialized_end=1860 - _globals['_DELETETASKRESPONSE']._serialized_start=1862 - _globals['_DELETETASKRESPONSE']._serialized_end=1882 - _globals['_AGENT']._serialized_start=1884 - _globals['_AGENT']._serialized_end=1961 - _globals['_GETAGENTREQUEST']._serialized_start=1963 - _globals['_GETAGENTREQUEST']._serialized_end=2000 - _globals['_GETAGENTRESPONSE']._serialized_start=2002 - _globals['_GETAGENTRESPONSE']._serialized_end=2065 - _globals['_LISTAGENTSREQUEST']._serialized_start=2067 - _globals['_LISTAGENTSREQUEST']._serialized_end=2086 - _globals['_LISTAGENTSRESPONSE']._serialized_start=2088 - _globals['_LISTAGENTSRESPONSE']._serialized_end=2155 - _globals['_GETTASKMETRICSREQUEST']._serialized_start=2158 - _globals['_GETTASKMETRICSREQUEST']._serialized_end=2434 - _globals['_GETTASKMETRICSRESPONSE']._serialized_start=2436 - _globals['_GETTASKMETRICSRESPONSE']._serialized_end=2524 - _globals['_GETTASKLOGSREQUEST']._serialized_start=2527 - _globals['_GETTASKLOGSREQUEST']._serialized_end=2657 - _globals['_GETTASKLOGSRESPONSE']._serialized_start=2659 - _globals['_GETTASKLOGSRESPONSE']._serialized_end=2728 + _DELETETASKREQUEST.fields_by_name['task_type']._options = None + _DELETETASKREQUEST.fields_by_name['task_type']._serialized_options = b'\030\001' + _AGENT.fields_by_name['supported_task_types']._options = None + _AGENT.fields_by_name['supported_task_types']._serialized_options = b'\030\001' + _GETTASKMETRICSREQUEST.fields_by_name['task_type']._options = None + _GETTASKMETRICSREQUEST.fields_by_name['task_type']._serialized_options = b'\030\001' + _GETTASKLOGSREQUEST.fields_by_name['task_type']._options = None + _GETTASKLOGSREQUEST.fields_by_name['task_type']._serialized_options = b'\030\001' + _globals['_STATE']._serialized_start=4271 + _globals['_STATE']._serialized_end=4369 + _globals['_TASKEXECUTIONMETADATA']._serialized_start=321 + _globals['_TASKEXECUTIONMETADATA']._serialized_end=1194 + _globals['_TASKEXECUTIONMETADATA_LABELSENTRY']._serialized_start=1000 + _globals['_TASKEXECUTIONMETADATA_LABELSENTRY']._serialized_end=1057 + _globals['_TASKEXECUTIONMETADATA_ANNOTATIONSENTRY']._serialized_start=1059 + _globals['_TASKEXECUTIONMETADATA_ANNOTATIONSENTRY']._serialized_end=1121 + _globals['_TASKEXECUTIONMETADATA_ENVIRONMENTVARIABLESENTRY']._serialized_start=1123 + _globals['_TASKEXECUTIONMETADATA_ENVIRONMENTVARIABLESENTRY']._serialized_end=1194 + _globals['_CREATETASKREQUEST']._serialized_start=1197 + _globals['_CREATETASKREQUEST']._serialized_end=1456 + _globals['_CREATETASKRESPONSE']._serialized_start=1458 + _globals['_CREATETASKRESPONSE']._serialized_end=1515 + _globals['_CREATEREQUESTHEADER']._serialized_start=1518 + _globals['_CREATEREQUESTHEADER']._serialized_end=1781 + _globals['_EXECUTETASKSYNCREQUEST']._serialized_start=1784 + _globals['_EXECUTETASKSYNCREQUEST']._serialized_end=1932 + _globals['_EXECUTETASKSYNCRESPONSEHEADER']._serialized_start=1934 + _globals['_EXECUTETASKSYNCRESPONSEHEADER']._serialized_end=2019 + _globals['_EXECUTETASKSYNCRESPONSE']._serialized_start=2022 + _globals['_EXECUTETASKSYNCRESPONSE']._serialized_end=2182 + _globals['_GETTASKREQUEST']._serialized_start=2185 + _globals['_GETTASKREQUEST']._serialized_end=2338 + _globals['_GETTASKRESPONSE']._serialized_start=2340 + _globals['_GETTASKRESPONSE']._serialized_end=2411 + _globals['_RESOURCE']._serialized_start=2414 + _globals['_RESOURCE']._serialized_end=2721 + _globals['_DELETETASKREQUEST']._serialized_start=2724 + _globals['_DELETETASKREQUEST']._serialized_end=2880 + _globals['_DELETETASKRESPONSE']._serialized_start=2882 + _globals['_DELETETASKRESPONSE']._serialized_end=2902 + _globals['_AGENT']._serialized_start=2905 + _globals['_AGENT']._serialized_end=3101 + _globals['_TASKCATEGORY']._serialized_start=3103 + _globals['_TASKCATEGORY']._serialized_end=3163 + _globals['_GETAGENTREQUEST']._serialized_start=3165 + _globals['_GETAGENTREQUEST']._serialized_end=3202 + _globals['_GETAGENTRESPONSE']._serialized_start=3204 + _globals['_GETAGENTRESPONSE']._serialized_end=3267 + _globals['_LISTAGENTSREQUEST']._serialized_start=3269 + _globals['_LISTAGENTSREQUEST']._serialized_end=3288 + _globals['_LISTAGENTSRESPONSE']._serialized_start=3290 + _globals['_LISTAGENTSRESPONSE']._serialized_end=3357 + _globals['_GETTASKMETRICSREQUEST']._serialized_start=3360 + _globals['_GETTASKMETRICSREQUEST']._serialized_end=3707 + _globals['_GETTASKMETRICSRESPONSE']._serialized_start=3709 + _globals['_GETTASKMETRICSRESPONSE']._serialized_end=3797 + _globals['_GETTASKLOGSREQUEST']._serialized_start=3800 + _globals['_GETTASKLOGSREQUEST']._serialized_end=4001 + _globals['_GETTASKLOGSRESPONSEHEADER']._serialized_start=4003 + _globals['_GETTASKLOGSRESPONSEHEADER']._serialized_end=4052 + _globals['_GETTASKLOGSRESPONSEBODY']._serialized_start=4054 + _globals['_GETTASKLOGSRESPONSEBODY']._serialized_end=4105 + _globals['_GETTASKLOGSRESPONSE']._serialized_start=4108 + _globals['_GETTASKLOGSRESPONSE']._serialized_end=4269 # @@protoc_insertion_point(module_scope) diff --git a/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.pyi b/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.pyi index 7b7c6beb07..988985e5ae 100644 --- a/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.pyi +++ b/flyteidl/gen/pb_python/flyteidl/admin/agent_pb2.pyi @@ -1,10 +1,12 @@ from flyteidl.core import literals_pb2 as _literals_pb2 from flyteidl.core import tasks_pb2 as _tasks_pb2 +from flyteidl.core import workflow_pb2 as _workflow_pb2 from flyteidl.core import identifier_pb2 as _identifier_pb2 from flyteidl.core import execution_pb2 as _execution_pb2 from flyteidl.core import metrics_pb2 as _metrics_pb2 from google.protobuf import duration_pb2 as _duration_pb2 from google.protobuf import timestamp_pb2 as _timestamp_pb2 +from google.protobuf import struct_pb2 as _struct_pb2 from google.protobuf.internal import containers as _containers from google.protobuf.internal import enum_type_wrapper as _enum_type_wrapper from google.protobuf import descriptor as _descriptor @@ -27,7 +29,7 @@ RUNNING: State SUCCEEDED: State class TaskExecutionMetadata(_message.Message): - __slots__ = ["task_execution_id", "namespace", "labels", "annotations", "k8s_service_account", "environment_variables"] + __slots__ = ["task_execution_id", "namespace", "labels", "annotations", "k8s_service_account", "environment_variables", "max_attempts", "interruptible", "interruptible_failure_threshold", "overrides"] class LabelsEntry(_message.Message): __slots__ = ["key", "value"] KEY_FIELD_NUMBER: _ClassVar[int] @@ -55,13 +57,21 @@ class TaskExecutionMetadata(_message.Message): ANNOTATIONS_FIELD_NUMBER: _ClassVar[int] K8S_SERVICE_ACCOUNT_FIELD_NUMBER: _ClassVar[int] ENVIRONMENT_VARIABLES_FIELD_NUMBER: _ClassVar[int] + MAX_ATTEMPTS_FIELD_NUMBER: _ClassVar[int] + INTERRUPTIBLE_FIELD_NUMBER: _ClassVar[int] + INTERRUPTIBLE_FAILURE_THRESHOLD_FIELD_NUMBER: _ClassVar[int] + OVERRIDES_FIELD_NUMBER: _ClassVar[int] task_execution_id: _identifier_pb2.TaskExecutionIdentifier namespace: str labels: _containers.ScalarMap[str, str] annotations: _containers.ScalarMap[str, str] k8s_service_account: str environment_variables: _containers.ScalarMap[str, str] - def __init__(self, task_execution_id: _Optional[_Union[_identifier_pb2.TaskExecutionIdentifier, _Mapping]] = ..., namespace: _Optional[str] = ..., labels: _Optional[_Mapping[str, str]] = ..., annotations: _Optional[_Mapping[str, str]] = ..., k8s_service_account: _Optional[str] = ..., environment_variables: _Optional[_Mapping[str, str]] = ...) -> None: ... + max_attempts: int + interruptible: bool + interruptible_failure_threshold: int + overrides: _workflow_pb2.TaskNodeOverrides + def __init__(self, task_execution_id: _Optional[_Union[_identifier_pb2.TaskExecutionIdentifier, _Mapping]] = ..., namespace: _Optional[str] = ..., labels: _Optional[_Mapping[str, str]] = ..., annotations: _Optional[_Mapping[str, str]] = ..., k8s_service_account: _Optional[str] = ..., environment_variables: _Optional[_Mapping[str, str]] = ..., max_attempts: _Optional[int] = ..., interruptible: bool = ..., interruptible_failure_threshold: _Optional[int] = ..., overrides: _Optional[_Union[_workflow_pb2.TaskNodeOverrides, _Mapping]] = ...) -> None: ... class CreateTaskRequest(_message.Message): __slots__ = ["inputs", "template", "output_prefix", "task_execution_metadata"] @@ -76,62 +86,110 @@ class CreateTaskRequest(_message.Message): def __init__(self, inputs: _Optional[_Union[_literals_pb2.LiteralMap, _Mapping]] = ..., template: _Optional[_Union[_tasks_pb2.TaskTemplate, _Mapping]] = ..., output_prefix: _Optional[str] = ..., task_execution_metadata: _Optional[_Union[TaskExecutionMetadata, _Mapping]] = ...) -> None: ... class CreateTaskResponse(_message.Message): - __slots__ = ["resource_meta", "resource"] + __slots__ = ["resource_meta"] RESOURCE_META_FIELD_NUMBER: _ClassVar[int] - RESOURCE_FIELD_NUMBER: _ClassVar[int] resource_meta: bytes + def __init__(self, resource_meta: _Optional[bytes] = ...) -> None: ... + +class CreateRequestHeader(_message.Message): + __slots__ = ["template", "output_prefix", "task_execution_metadata", "max_dataset_size_bytes"] + TEMPLATE_FIELD_NUMBER: _ClassVar[int] + OUTPUT_PREFIX_FIELD_NUMBER: _ClassVar[int] + TASK_EXECUTION_METADATA_FIELD_NUMBER: _ClassVar[int] + MAX_DATASET_SIZE_BYTES_FIELD_NUMBER: _ClassVar[int] + template: _tasks_pb2.TaskTemplate + output_prefix: str + task_execution_metadata: TaskExecutionMetadata + max_dataset_size_bytes: int + def __init__(self, template: _Optional[_Union[_tasks_pb2.TaskTemplate, _Mapping]] = ..., output_prefix: _Optional[str] = ..., task_execution_metadata: _Optional[_Union[TaskExecutionMetadata, _Mapping]] = ..., max_dataset_size_bytes: _Optional[int] = ...) -> None: ... + +class ExecuteTaskSyncRequest(_message.Message): + __slots__ = ["header", "inputs"] + HEADER_FIELD_NUMBER: _ClassVar[int] + INPUTS_FIELD_NUMBER: _ClassVar[int] + header: CreateRequestHeader + inputs: _literals_pb2.LiteralMap + def __init__(self, header: _Optional[_Union[CreateRequestHeader, _Mapping]] = ..., inputs: _Optional[_Union[_literals_pb2.LiteralMap, _Mapping]] = ...) -> None: ... + +class ExecuteTaskSyncResponseHeader(_message.Message): + __slots__ = ["resource"] + RESOURCE_FIELD_NUMBER: _ClassVar[int] resource: Resource - def __init__(self, resource_meta: _Optional[bytes] = ..., resource: _Optional[_Union[Resource, _Mapping]] = ...) -> None: ... + def __init__(self, resource: _Optional[_Union[Resource, _Mapping]] = ...) -> None: ... + +class ExecuteTaskSyncResponse(_message.Message): + __slots__ = ["header", "outputs"] + HEADER_FIELD_NUMBER: _ClassVar[int] + OUTPUTS_FIELD_NUMBER: _ClassVar[int] + header: ExecuteTaskSyncResponseHeader + outputs: _literals_pb2.LiteralMap + def __init__(self, header: _Optional[_Union[ExecuteTaskSyncResponseHeader, _Mapping]] = ..., outputs: _Optional[_Union[_literals_pb2.LiteralMap, _Mapping]] = ...) -> None: ... class GetTaskRequest(_message.Message): - __slots__ = ["task_type", "resource_meta"] + __slots__ = ["task_type", "resource_meta", "task_category"] TASK_TYPE_FIELD_NUMBER: _ClassVar[int] RESOURCE_META_FIELD_NUMBER: _ClassVar[int] + TASK_CATEGORY_FIELD_NUMBER: _ClassVar[int] task_type: str resource_meta: bytes - def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ...) -> None: ... + task_category: TaskCategory + def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ..., task_category: _Optional[_Union[TaskCategory, _Mapping]] = ...) -> None: ... class GetTaskResponse(_message.Message): - __slots__ = ["resource", "log_links"] + __slots__ = ["resource"] RESOURCE_FIELD_NUMBER: _ClassVar[int] - LOG_LINKS_FIELD_NUMBER: _ClassVar[int] resource: Resource - log_links: _containers.RepeatedCompositeFieldContainer[_execution_pb2.TaskLog] - def __init__(self, resource: _Optional[_Union[Resource, _Mapping]] = ..., log_links: _Optional[_Iterable[_Union[_execution_pb2.TaskLog, _Mapping]]] = ...) -> None: ... + def __init__(self, resource: _Optional[_Union[Resource, _Mapping]] = ...) -> None: ... class Resource(_message.Message): - __slots__ = ["state", "outputs", "message", "log_links", "phase"] + __slots__ = ["state", "outputs", "message", "log_links", "phase", "custom_info"] STATE_FIELD_NUMBER: _ClassVar[int] OUTPUTS_FIELD_NUMBER: _ClassVar[int] MESSAGE_FIELD_NUMBER: _ClassVar[int] LOG_LINKS_FIELD_NUMBER: _ClassVar[int] PHASE_FIELD_NUMBER: _ClassVar[int] + CUSTOM_INFO_FIELD_NUMBER: _ClassVar[int] state: State outputs: _literals_pb2.LiteralMap message: str log_links: _containers.RepeatedCompositeFieldContainer[_execution_pb2.TaskLog] phase: _execution_pb2.TaskExecution.Phase - def __init__(self, state: _Optional[_Union[State, str]] = ..., outputs: _Optional[_Union[_literals_pb2.LiteralMap, _Mapping]] = ..., message: _Optional[str] = ..., log_links: _Optional[_Iterable[_Union[_execution_pb2.TaskLog, _Mapping]]] = ..., phase: _Optional[_Union[_execution_pb2.TaskExecution.Phase, str]] = ...) -> None: ... + custom_info: _struct_pb2.Struct + def __init__(self, state: _Optional[_Union[State, str]] = ..., outputs: _Optional[_Union[_literals_pb2.LiteralMap, _Mapping]] = ..., message: _Optional[str] = ..., log_links: _Optional[_Iterable[_Union[_execution_pb2.TaskLog, _Mapping]]] = ..., phase: _Optional[_Union[_execution_pb2.TaskExecution.Phase, str]] = ..., custom_info: _Optional[_Union[_struct_pb2.Struct, _Mapping]] = ...) -> None: ... class DeleteTaskRequest(_message.Message): - __slots__ = ["task_type", "resource_meta"] + __slots__ = ["task_type", "resource_meta", "task_category"] TASK_TYPE_FIELD_NUMBER: _ClassVar[int] RESOURCE_META_FIELD_NUMBER: _ClassVar[int] + TASK_CATEGORY_FIELD_NUMBER: _ClassVar[int] task_type: str resource_meta: bytes - def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ...) -> None: ... + task_category: TaskCategory + def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ..., task_category: _Optional[_Union[TaskCategory, _Mapping]] = ...) -> None: ... class DeleteTaskResponse(_message.Message): __slots__ = [] def __init__(self) -> None: ... class Agent(_message.Message): - __slots__ = ["name", "supported_task_types"] + __slots__ = ["name", "supported_task_types", "is_sync", "supported_task_categories"] NAME_FIELD_NUMBER: _ClassVar[int] SUPPORTED_TASK_TYPES_FIELD_NUMBER: _ClassVar[int] + IS_SYNC_FIELD_NUMBER: _ClassVar[int] + SUPPORTED_TASK_CATEGORIES_FIELD_NUMBER: _ClassVar[int] name: str supported_task_types: _containers.RepeatedScalarFieldContainer[str] - def __init__(self, name: _Optional[str] = ..., supported_task_types: _Optional[_Iterable[str]] = ...) -> None: ... + is_sync: bool + supported_task_categories: _containers.RepeatedCompositeFieldContainer[TaskCategory] + def __init__(self, name: _Optional[str] = ..., supported_task_types: _Optional[_Iterable[str]] = ..., is_sync: bool = ..., supported_task_categories: _Optional[_Iterable[_Union[TaskCategory, _Mapping]]] = ...) -> None: ... + +class TaskCategory(_message.Message): + __slots__ = ["name", "version"] + NAME_FIELD_NUMBER: _ClassVar[int] + VERSION_FIELD_NUMBER: _ClassVar[int] + name: str + version: int + def __init__(self, name: _Optional[str] = ..., version: _Optional[int] = ...) -> None: ... class GetAgentRequest(_message.Message): __slots__ = ["name"] @@ -156,20 +214,22 @@ class ListAgentsResponse(_message.Message): def __init__(self, agents: _Optional[_Iterable[_Union[Agent, _Mapping]]] = ...) -> None: ... class GetTaskMetricsRequest(_message.Message): - __slots__ = ["task_type", "resource_meta", "queries", "start_time", "end_time", "step"] + __slots__ = ["task_type", "resource_meta", "queries", "start_time", "end_time", "step", "task_category"] TASK_TYPE_FIELD_NUMBER: _ClassVar[int] RESOURCE_META_FIELD_NUMBER: _ClassVar[int] QUERIES_FIELD_NUMBER: _ClassVar[int] START_TIME_FIELD_NUMBER: _ClassVar[int] END_TIME_FIELD_NUMBER: _ClassVar[int] STEP_FIELD_NUMBER: _ClassVar[int] + TASK_CATEGORY_FIELD_NUMBER: _ClassVar[int] task_type: str resource_meta: bytes queries: _containers.RepeatedScalarFieldContainer[str] start_time: _timestamp_pb2.Timestamp end_time: _timestamp_pb2.Timestamp step: _duration_pb2.Duration - def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ..., queries: _Optional[_Iterable[str]] = ..., start_time: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., end_time: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., step: _Optional[_Union[_duration_pb2.Duration, _Mapping]] = ...) -> None: ... + task_category: TaskCategory + def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ..., queries: _Optional[_Iterable[str]] = ..., start_time: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., end_time: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., step: _Optional[_Union[_duration_pb2.Duration, _Mapping]] = ..., task_category: _Optional[_Union[TaskCategory, _Mapping]] = ...) -> None: ... class GetTaskMetricsResponse(_message.Message): __slots__ = ["results"] @@ -178,21 +238,35 @@ class GetTaskMetricsResponse(_message.Message): def __init__(self, results: _Optional[_Iterable[_Union[_metrics_pb2.ExecutionMetricResult, _Mapping]]] = ...) -> None: ... class GetTaskLogsRequest(_message.Message): - __slots__ = ["task_type", "resource_meta", "lines", "token"] + __slots__ = ["task_type", "resource_meta", "lines", "token", "task_category"] TASK_TYPE_FIELD_NUMBER: _ClassVar[int] RESOURCE_META_FIELD_NUMBER: _ClassVar[int] LINES_FIELD_NUMBER: _ClassVar[int] TOKEN_FIELD_NUMBER: _ClassVar[int] + TASK_CATEGORY_FIELD_NUMBER: _ClassVar[int] task_type: str resource_meta: bytes lines: int token: str - def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ..., lines: _Optional[int] = ..., token: _Optional[str] = ...) -> None: ... + task_category: TaskCategory + def __init__(self, task_type: _Optional[str] = ..., resource_meta: _Optional[bytes] = ..., lines: _Optional[int] = ..., token: _Optional[str] = ..., task_category: _Optional[_Union[TaskCategory, _Mapping]] = ...) -> None: ... -class GetTaskLogsResponse(_message.Message): - __slots__ = ["results", "token"] - RESULTS_FIELD_NUMBER: _ClassVar[int] +class GetTaskLogsResponseHeader(_message.Message): + __slots__ = ["token"] TOKEN_FIELD_NUMBER: _ClassVar[int] - results: _containers.RepeatedScalarFieldContainer[str] token: str - def __init__(self, results: _Optional[_Iterable[str]] = ..., token: _Optional[str] = ...) -> None: ... + def __init__(self, token: _Optional[str] = ...) -> None: ... + +class GetTaskLogsResponseBody(_message.Message): + __slots__ = ["results"] + RESULTS_FIELD_NUMBER: _ClassVar[int] + results: _containers.RepeatedScalarFieldContainer[str] + def __init__(self, results: _Optional[_Iterable[str]] = ...) -> None: ... + +class GetTaskLogsResponse(_message.Message): + __slots__ = ["header", "body"] + HEADER_FIELD_NUMBER: _ClassVar[int] + BODY_FIELD_NUMBER: _ClassVar[int] + header: GetTaskLogsResponseHeader + body: GetTaskLogsResponseBody + def __init__(self, header: _Optional[_Union[GetTaskLogsResponseHeader, _Mapping]] = ..., body: _Optional[_Union[GetTaskLogsResponseBody, _Mapping]] = ...) -> None: ... diff --git a/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.py b/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.py index 918db78e56..4c2a628407 100644 --- a/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.py +++ b/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.py @@ -24,7 +24,7 @@ from google.protobuf import wrappers_pb2 as google_dot_protobuf_dot_wrappers__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1e\x66lyteidl/admin/execution.proto\x12\x0e\x66lyteidl.admin\x1a\'flyteidl/admin/cluster_assignment.proto\x1a\x1b\x66lyteidl/admin/common.proto\x1a\x1c\x66lyteidl/core/literals.proto\x1a\x1d\x66lyteidl/core/execution.proto\x1a\x1f\x66lyteidl/core/artifact_id.proto\x1a\x1e\x66lyteidl/core/identifier.proto\x1a\x1b\x66lyteidl/core/metrics.proto\x1a\x1c\x66lyteidl/core/security.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/wrappers.proto\"\xd6\x01\n\x16\x45xecutionCreateRequest\x12\x18\n\x07project\x18\x01 \x01(\tR\x07project\x12\x16\n\x06\x64omain\x18\x02 \x01(\tR\x06\x64omain\x12\x12\n\x04name\x18\x03 \x01(\tR\x04name\x12\x31\n\x04spec\x18\x04 \x01(\x0b\x32\x1d.flyteidl.admin.ExecutionSpecR\x04spec\x12\x31\n\x06inputs\x18\x05 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x06inputs\x12\x10\n\x03org\x18\x06 \x01(\tR\x03org\"\x99\x01\n\x18\x45xecutionRelaunchRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x12\n\x04name\x18\x03 \x01(\tR\x04name\x12\'\n\x0foverwrite_cache\x18\x04 \x01(\x08R\x0eoverwriteCacheJ\x04\x08\x02\x10\x03\"\xa8\x01\n\x17\x45xecutionRecoverRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x12\n\x04name\x18\x02 \x01(\tR\x04name\x12=\n\x08metadata\x18\x03 \x01(\x0b\x32!.flyteidl.admin.ExecutionMetadataR\x08metadata\"U\n\x17\x45xecutionCreateResponse\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\"Y\n\x1bWorkflowExecutionGetRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\"\xb6\x01\n\tExecution\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x31\n\x04spec\x18\x02 \x01(\x0b\x32\x1d.flyteidl.admin.ExecutionSpecR\x04spec\x12:\n\x07\x63losure\x18\x03 \x01(\x0b\x32 .flyteidl.admin.ExecutionClosureR\x07\x63losure\"`\n\rExecutionList\x12\x39\n\nexecutions\x18\x01 \x03(\x0b\x32\x19.flyteidl.admin.ExecutionR\nexecutions\x12\x14\n\x05token\x18\x02 \x01(\tR\x05token\"e\n\x0eLiteralMapBlob\x12\x37\n\x06values\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01H\x00R\x06values\x12\x12\n\x03uri\x18\x02 \x01(\tH\x00R\x03uriB\x06\n\x04\x64\x61ta\"C\n\rAbortMetadata\x12\x14\n\x05\x63\x61use\x18\x01 \x01(\tR\x05\x63\x61use\x12\x1c\n\tprincipal\x18\x02 \x01(\tR\tprincipal\"\x98\x07\n\x10\x45xecutionClosure\x12>\n\x07outputs\x18\x01 \x01(\x0b\x32\x1e.flyteidl.admin.LiteralMapBlobB\x02\x18\x01H\x00R\x07outputs\x12\x35\n\x05\x65rror\x18\x02 \x01(\x0b\x32\x1d.flyteidl.core.ExecutionErrorH\x00R\x05\x65rror\x12%\n\x0b\x61\x62ort_cause\x18\n \x01(\tB\x02\x18\x01H\x00R\nabortCause\x12\x46\n\x0e\x61\x62ort_metadata\x18\x0c \x01(\x0b\x32\x1d.flyteidl.admin.AbortMetadataH\x00R\rabortMetadata\x12@\n\x0boutput_data\x18\r \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01H\x00R\noutputData\x12\x46\n\x0f\x63omputed_inputs\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01R\x0e\x63omputedInputs\x12<\n\x05phase\x18\x04 \x01(\x0e\x32&.flyteidl.core.WorkflowExecution.PhaseR\x05phase\x12\x39\n\nstarted_at\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tstartedAt\x12\x35\n\x08\x64uration\x18\x06 \x01(\x0b\x32\x19.google.protobuf.DurationR\x08\x64uration\x12\x39\n\ncreated_at\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tcreatedAt\x12\x39\n\nupdated_at\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tupdatedAt\x12\x42\n\rnotifications\x18\t \x03(\x0b\x32\x1c.flyteidl.admin.NotificationR\rnotifications\x12:\n\x0bworkflow_id\x18\x0b \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\nworkflowId\x12]\n\x14state_change_details\x18\x0e \x01(\x0b\x32+.flyteidl.admin.ExecutionStateChangeDetailsR\x12stateChangeDetailsB\x0f\n\routput_result\"[\n\x0eSystemMetadata\x12+\n\x11\x65xecution_cluster\x18\x01 \x01(\tR\x10\x65xecutionCluster\x12\x1c\n\tnamespace\x18\x02 \x01(\tR\tnamespace\"\xf8\x04\n\x11\x45xecutionMetadata\x12\x43\n\x04mode\x18\x01 \x01(\x0e\x32/.flyteidl.admin.ExecutionMetadata.ExecutionModeR\x04mode\x12\x1c\n\tprincipal\x18\x02 \x01(\tR\tprincipal\x12\x18\n\x07nesting\x18\x03 \x01(\rR\x07nesting\x12=\n\x0cscheduled_at\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\x0bscheduledAt\x12Z\n\x15parent_node_execution\x18\x05 \x01(\x0b\x32&.flyteidl.core.NodeExecutionIdentifierR\x13parentNodeExecution\x12[\n\x13reference_execution\x18\x10 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x12referenceExecution\x12G\n\x0fsystem_metadata\x18\x11 \x01(\x0b\x32\x1e.flyteidl.admin.SystemMetadataR\x0esystemMetadata\x12<\n\x0c\x61rtifact_ids\x18\x12 \x03(\x0b\x32\x19.flyteidl.core.ArtifactIDR\x0b\x61rtifactIds\"g\n\rExecutionMode\x12\n\n\x06MANUAL\x10\x00\x12\r\n\tSCHEDULED\x10\x01\x12\n\n\x06SYSTEM\x10\x02\x12\x0c\n\x08RELAUNCH\x10\x03\x12\x12\n\x0e\x43HILD_WORKFLOW\x10\x04\x12\r\n\tRECOVERED\x10\x05\"V\n\x10NotificationList\x12\x42\n\rnotifications\x18\x01 \x03(\x0b\x32\x1c.flyteidl.admin.NotificationR\rnotifications\"\x90\x08\n\rExecutionSpec\x12:\n\x0blaunch_plan\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\nlaunchPlan\x12\x35\n\x06inputs\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01R\x06inputs\x12=\n\x08metadata\x18\x03 \x01(\x0b\x32!.flyteidl.admin.ExecutionMetadataR\x08metadata\x12H\n\rnotifications\x18\x05 \x01(\x0b\x32 .flyteidl.admin.NotificationListH\x00R\rnotifications\x12!\n\x0b\x64isable_all\x18\x06 \x01(\x08H\x00R\ndisableAll\x12.\n\x06labels\x18\x07 \x01(\x0b\x32\x16.flyteidl.admin.LabelsR\x06labels\x12=\n\x0b\x61nnotations\x18\x08 \x01(\x0b\x32\x1b.flyteidl.admin.AnnotationsR\x0b\x61nnotations\x12I\n\x10security_context\x18\n \x01(\x0b\x32\x1e.flyteidl.core.SecurityContextR\x0fsecurityContext\x12\x39\n\tauth_role\x18\x10 \x01(\x0b\x32\x18.flyteidl.admin.AuthRoleB\x02\x18\x01R\x08\x61uthRole\x12M\n\x12quality_of_service\x18\x11 \x01(\x0b\x32\x1f.flyteidl.core.QualityOfServiceR\x10qualityOfService\x12\'\n\x0fmax_parallelism\x18\x12 \x01(\x05R\x0emaxParallelism\x12X\n\x16raw_output_data_config\x18\x13 \x01(\x0b\x32#.flyteidl.admin.RawOutputDataConfigR\x13rawOutputDataConfig\x12P\n\x12\x63luster_assignment\x18\x14 \x01(\x0b\x32!.flyteidl.admin.ClusterAssignmentR\x11\x63lusterAssignment\x12@\n\rinterruptible\x18\x15 \x01(\x0b\x32\x1a.google.protobuf.BoolValueR\rinterruptible\x12\'\n\x0foverwrite_cache\x18\x16 \x01(\x08R\x0eoverwriteCache\x12(\n\x04\x65nvs\x18\x17 \x01(\x0b\x32\x14.flyteidl.admin.EnvsR\x04\x65nvs\x12\x12\n\x04tags\x18\x18 \x03(\tR\x04tagsB\x18\n\x16notification_overridesJ\x04\x08\x04\x10\x05\"m\n\x19\x45xecutionTerminateRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x14\n\x05\x63\x61use\x18\x02 \x01(\tR\x05\x63\x61use\"\x1c\n\x1a\x45xecutionTerminateResponse\"]\n\x1fWorkflowExecutionGetDataRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\"\x88\x02\n WorkflowExecutionGetDataResponse\x12\x35\n\x07outputs\x18\x01 \x01(\x0b\x32\x17.flyteidl.admin.UrlBlobB\x02\x18\x01R\x07outputs\x12\x33\n\x06inputs\x18\x02 \x01(\x0b\x32\x17.flyteidl.admin.UrlBlobB\x02\x18\x01R\x06inputs\x12:\n\x0b\x66ull_inputs\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\nfullInputs\x12<\n\x0c\x66ull_outputs\x18\x04 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x0b\x66ullOutputs\"\x8a\x01\n\x16\x45xecutionUpdateRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x34\n\x05state\x18\x02 \x01(\x0e\x32\x1e.flyteidl.admin.ExecutionStateR\x05state\"\xae\x01\n\x1b\x45xecutionStateChangeDetails\x12\x34\n\x05state\x18\x01 \x01(\x0e\x32\x1e.flyteidl.admin.ExecutionStateR\x05state\x12;\n\x0boccurred_at\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\noccurredAt\x12\x1c\n\tprincipal\x18\x03 \x01(\tR\tprincipal\"\x19\n\x17\x45xecutionUpdateResponse\"v\n\"WorkflowExecutionGetMetricsRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x14\n\x05\x64\x65pth\x18\x02 \x01(\x05R\x05\x64\x65pth\"N\n#WorkflowExecutionGetMetricsResponse\x12\'\n\x04span\x18\x01 \x01(\x0b\x32\x13.flyteidl.core.SpanR\x04span*>\n\x0e\x45xecutionState\x12\x14\n\x10\x45XECUTION_ACTIVE\x10\x00\x12\x16\n\x12\x45XECUTION_ARCHIVED\x10\x01\x42\xba\x01\n\x12\x63om.flyteidl.adminB\x0e\x45xecutionProtoP\x01Z;github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin\xa2\x02\x03\x46\x41X\xaa\x02\x0e\x46lyteidl.Admin\xca\x02\x0e\x46lyteidl\\Admin\xe2\x02\x1a\x46lyteidl\\Admin\\GPBMetadata\xea\x02\x0f\x46lyteidl::Adminb\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1e\x66lyteidl/admin/execution.proto\x12\x0e\x66lyteidl.admin\x1a\'flyteidl/admin/cluster_assignment.proto\x1a\x1b\x66lyteidl/admin/common.proto\x1a\x1c\x66lyteidl/core/literals.proto\x1a\x1d\x66lyteidl/core/execution.proto\x1a\x1f\x66lyteidl/core/artifact_id.proto\x1a\x1e\x66lyteidl/core/identifier.proto\x1a\x1b\x66lyteidl/core/metrics.proto\x1a\x1c\x66lyteidl/core/security.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/wrappers.proto\"\xd6\x01\n\x16\x45xecutionCreateRequest\x12\x18\n\x07project\x18\x01 \x01(\tR\x07project\x12\x16\n\x06\x64omain\x18\x02 \x01(\tR\x06\x64omain\x12\x12\n\x04name\x18\x03 \x01(\tR\x04name\x12\x31\n\x04spec\x18\x04 \x01(\x0b\x32\x1d.flyteidl.admin.ExecutionSpecR\x04spec\x12\x31\n\x06inputs\x18\x05 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x06inputs\x12\x10\n\x03org\x18\x06 \x01(\tR\x03org\"\x99\x01\n\x18\x45xecutionRelaunchRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x12\n\x04name\x18\x03 \x01(\tR\x04name\x12\'\n\x0foverwrite_cache\x18\x04 \x01(\x08R\x0eoverwriteCacheJ\x04\x08\x02\x10\x03\"\xa8\x01\n\x17\x45xecutionRecoverRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x12\n\x04name\x18\x02 \x01(\tR\x04name\x12=\n\x08metadata\x18\x03 \x01(\x0b\x32!.flyteidl.admin.ExecutionMetadataR\x08metadata\"U\n\x17\x45xecutionCreateResponse\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\"Y\n\x1bWorkflowExecutionGetRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\"\xb6\x01\n\tExecution\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x31\n\x04spec\x18\x02 \x01(\x0b\x32\x1d.flyteidl.admin.ExecutionSpecR\x04spec\x12:\n\x07\x63losure\x18\x03 \x01(\x0b\x32 .flyteidl.admin.ExecutionClosureR\x07\x63losure\"`\n\rExecutionList\x12\x39\n\nexecutions\x18\x01 \x03(\x0b\x32\x19.flyteidl.admin.ExecutionR\nexecutions\x12\x14\n\x05token\x18\x02 \x01(\tR\x05token\"e\n\x0eLiteralMapBlob\x12\x37\n\x06values\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01H\x00R\x06values\x12\x12\n\x03uri\x18\x02 \x01(\tH\x00R\x03uriB\x06\n\x04\x64\x61ta\"C\n\rAbortMetadata\x12\x14\n\x05\x63\x61use\x18\x01 \x01(\tR\x05\x63\x61use\x12\x1c\n\tprincipal\x18\x02 \x01(\tR\tprincipal\"\x98\x07\n\x10\x45xecutionClosure\x12>\n\x07outputs\x18\x01 \x01(\x0b\x32\x1e.flyteidl.admin.LiteralMapBlobB\x02\x18\x01H\x00R\x07outputs\x12\x35\n\x05\x65rror\x18\x02 \x01(\x0b\x32\x1d.flyteidl.core.ExecutionErrorH\x00R\x05\x65rror\x12%\n\x0b\x61\x62ort_cause\x18\n \x01(\tB\x02\x18\x01H\x00R\nabortCause\x12\x46\n\x0e\x61\x62ort_metadata\x18\x0c \x01(\x0b\x32\x1d.flyteidl.admin.AbortMetadataH\x00R\rabortMetadata\x12@\n\x0boutput_data\x18\r \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01H\x00R\noutputData\x12\x46\n\x0f\x63omputed_inputs\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01R\x0e\x63omputedInputs\x12<\n\x05phase\x18\x04 \x01(\x0e\x32&.flyteidl.core.WorkflowExecution.PhaseR\x05phase\x12\x39\n\nstarted_at\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tstartedAt\x12\x35\n\x08\x64uration\x18\x06 \x01(\x0b\x32\x19.google.protobuf.DurationR\x08\x64uration\x12\x39\n\ncreated_at\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tcreatedAt\x12\x39\n\nupdated_at\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\tupdatedAt\x12\x42\n\rnotifications\x18\t \x03(\x0b\x32\x1c.flyteidl.admin.NotificationR\rnotifications\x12:\n\x0bworkflow_id\x18\x0b \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\nworkflowId\x12]\n\x14state_change_details\x18\x0e \x01(\x0b\x32+.flyteidl.admin.ExecutionStateChangeDetailsR\x12stateChangeDetailsB\x0f\n\routput_result\"[\n\x0eSystemMetadata\x12+\n\x11\x65xecution_cluster\x18\x01 \x01(\tR\x10\x65xecutionCluster\x12\x1c\n\tnamespace\x18\x02 \x01(\tR\tnamespace\"\x85\x05\n\x11\x45xecutionMetadata\x12\x43\n\x04mode\x18\x01 \x01(\x0e\x32/.flyteidl.admin.ExecutionMetadata.ExecutionModeR\x04mode\x12\x1c\n\tprincipal\x18\x02 \x01(\tR\tprincipal\x12\x18\n\x07nesting\x18\x03 \x01(\rR\x07nesting\x12=\n\x0cscheduled_at\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\x0bscheduledAt\x12Z\n\x15parent_node_execution\x18\x05 \x01(\x0b\x32&.flyteidl.core.NodeExecutionIdentifierR\x13parentNodeExecution\x12[\n\x13reference_execution\x18\x10 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x12referenceExecution\x12G\n\x0fsystem_metadata\x18\x11 \x01(\x0b\x32\x1e.flyteidl.admin.SystemMetadataR\x0esystemMetadata\x12<\n\x0c\x61rtifact_ids\x18\x12 \x03(\x0b\x32\x19.flyteidl.core.ArtifactIDR\x0b\x61rtifactIds\"t\n\rExecutionMode\x12\n\n\x06MANUAL\x10\x00\x12\r\n\tSCHEDULED\x10\x01\x12\n\n\x06SYSTEM\x10\x02\x12\x0c\n\x08RELAUNCH\x10\x03\x12\x12\n\x0e\x43HILD_WORKFLOW\x10\x04\x12\r\n\tRECOVERED\x10\x05\x12\x0b\n\x07TRIGGER\x10\x06\"V\n\x10NotificationList\x12\x42\n\rnotifications\x18\x01 \x03(\x0b\x32\x1c.flyteidl.admin.NotificationR\rnotifications\"\x90\x08\n\rExecutionSpec\x12:\n\x0blaunch_plan\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\nlaunchPlan\x12\x35\n\x06inputs\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapB\x02\x18\x01R\x06inputs\x12=\n\x08metadata\x18\x03 \x01(\x0b\x32!.flyteidl.admin.ExecutionMetadataR\x08metadata\x12H\n\rnotifications\x18\x05 \x01(\x0b\x32 .flyteidl.admin.NotificationListH\x00R\rnotifications\x12!\n\x0b\x64isable_all\x18\x06 \x01(\x08H\x00R\ndisableAll\x12.\n\x06labels\x18\x07 \x01(\x0b\x32\x16.flyteidl.admin.LabelsR\x06labels\x12=\n\x0b\x61nnotations\x18\x08 \x01(\x0b\x32\x1b.flyteidl.admin.AnnotationsR\x0b\x61nnotations\x12I\n\x10security_context\x18\n \x01(\x0b\x32\x1e.flyteidl.core.SecurityContextR\x0fsecurityContext\x12\x39\n\tauth_role\x18\x10 \x01(\x0b\x32\x18.flyteidl.admin.AuthRoleB\x02\x18\x01R\x08\x61uthRole\x12M\n\x12quality_of_service\x18\x11 \x01(\x0b\x32\x1f.flyteidl.core.QualityOfServiceR\x10qualityOfService\x12\'\n\x0fmax_parallelism\x18\x12 \x01(\x05R\x0emaxParallelism\x12X\n\x16raw_output_data_config\x18\x13 \x01(\x0b\x32#.flyteidl.admin.RawOutputDataConfigR\x13rawOutputDataConfig\x12P\n\x12\x63luster_assignment\x18\x14 \x01(\x0b\x32!.flyteidl.admin.ClusterAssignmentR\x11\x63lusterAssignment\x12@\n\rinterruptible\x18\x15 \x01(\x0b\x32\x1a.google.protobuf.BoolValueR\rinterruptible\x12\'\n\x0foverwrite_cache\x18\x16 \x01(\x08R\x0eoverwriteCache\x12(\n\x04\x65nvs\x18\x17 \x01(\x0b\x32\x14.flyteidl.admin.EnvsR\x04\x65nvs\x12\x12\n\x04tags\x18\x18 \x03(\tR\x04tagsB\x18\n\x16notification_overridesJ\x04\x08\x04\x10\x05\"m\n\x19\x45xecutionTerminateRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x14\n\x05\x63\x61use\x18\x02 \x01(\tR\x05\x63\x61use\"\x1c\n\x1a\x45xecutionTerminateResponse\"]\n\x1fWorkflowExecutionGetDataRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\"\x88\x02\n WorkflowExecutionGetDataResponse\x12\x35\n\x07outputs\x18\x01 \x01(\x0b\x32\x17.flyteidl.admin.UrlBlobB\x02\x18\x01R\x07outputs\x12\x33\n\x06inputs\x18\x02 \x01(\x0b\x32\x17.flyteidl.admin.UrlBlobB\x02\x18\x01R\x06inputs\x12:\n\x0b\x66ull_inputs\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\nfullInputs\x12<\n\x0c\x66ull_outputs\x18\x04 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x0b\x66ullOutputs\"\x8a\x01\n\x16\x45xecutionUpdateRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x34\n\x05state\x18\x02 \x01(\x0e\x32\x1e.flyteidl.admin.ExecutionStateR\x05state\"\xae\x01\n\x1b\x45xecutionStateChangeDetails\x12\x34\n\x05state\x18\x01 \x01(\x0e\x32\x1e.flyteidl.admin.ExecutionStateR\x05state\x12;\n\x0boccurred_at\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.TimestampR\noccurredAt\x12\x1c\n\tprincipal\x18\x03 \x01(\tR\tprincipal\"\x19\n\x17\x45xecutionUpdateResponse\"v\n\"WorkflowExecutionGetMetricsRequest\x12:\n\x02id\x18\x01 \x01(\x0b\x32*.flyteidl.core.WorkflowExecutionIdentifierR\x02id\x12\x14\n\x05\x64\x65pth\x18\x02 \x01(\x05R\x05\x64\x65pth\"N\n#WorkflowExecutionGetMetricsResponse\x12\'\n\x04span\x18\x01 \x01(\x0b\x32\x13.flyteidl.core.SpanR\x04span*>\n\x0e\x45xecutionState\x12\x14\n\x10\x45XECUTION_ACTIVE\x10\x00\x12\x16\n\x12\x45XECUTION_ARCHIVED\x10\x01\x42\xba\x01\n\x12\x63om.flyteidl.adminB\x0e\x45xecutionProtoP\x01Z;github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin\xa2\x02\x03\x46\x41X\xaa\x02\x0e\x46lyteidl.Admin\xca\x02\x0e\x46lyteidl\\Admin\xe2\x02\x1a\x46lyteidl\\Admin\\GPBMetadata\xea\x02\x0f\x46lyteidl::Adminb\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -51,8 +51,8 @@ _WORKFLOWEXECUTIONGETDATARESPONSE.fields_by_name['outputs']._serialized_options = b'\030\001' _WORKFLOWEXECUTIONGETDATARESPONSE.fields_by_name['inputs']._options = None _WORKFLOWEXECUTIONGETDATARESPONSE.fields_by_name['inputs']._serialized_options = b'\030\001' - _globals['_EXECUTIONSTATE']._serialized_start=5409 - _globals['_EXECUTIONSTATE']._serialized_end=5471 + _globals['_EXECUTIONSTATE']._serialized_start=5422 + _globals['_EXECUTIONSTATE']._serialized_end=5484 _globals['_EXECUTIONCREATEREQUEST']._serialized_start=403 _globals['_EXECUTIONCREATEREQUEST']._serialized_end=617 _globals['_EXECUTIONRELAUNCHREQUEST']._serialized_start=620 @@ -76,29 +76,29 @@ _globals['_SYSTEMMETADATA']._serialized_start=2502 _globals['_SYSTEMMETADATA']._serialized_end=2593 _globals['_EXECUTIONMETADATA']._serialized_start=2596 - _globals['_EXECUTIONMETADATA']._serialized_end=3228 + _globals['_EXECUTIONMETADATA']._serialized_end=3241 _globals['_EXECUTIONMETADATA_EXECUTIONMODE']._serialized_start=3125 - _globals['_EXECUTIONMETADATA_EXECUTIONMODE']._serialized_end=3228 - _globals['_NOTIFICATIONLIST']._serialized_start=3230 - _globals['_NOTIFICATIONLIST']._serialized_end=3316 - _globals['_EXECUTIONSPEC']._serialized_start=3319 - _globals['_EXECUTIONSPEC']._serialized_end=4359 - _globals['_EXECUTIONTERMINATEREQUEST']._serialized_start=4361 - _globals['_EXECUTIONTERMINATEREQUEST']._serialized_end=4470 - _globals['_EXECUTIONTERMINATERESPONSE']._serialized_start=4472 - _globals['_EXECUTIONTERMINATERESPONSE']._serialized_end=4500 - _globals['_WORKFLOWEXECUTIONGETDATAREQUEST']._serialized_start=4502 - _globals['_WORKFLOWEXECUTIONGETDATAREQUEST']._serialized_end=4595 - _globals['_WORKFLOWEXECUTIONGETDATARESPONSE']._serialized_start=4598 - _globals['_WORKFLOWEXECUTIONGETDATARESPONSE']._serialized_end=4862 - _globals['_EXECUTIONUPDATEREQUEST']._serialized_start=4865 - _globals['_EXECUTIONUPDATEREQUEST']._serialized_end=5003 - _globals['_EXECUTIONSTATECHANGEDETAILS']._serialized_start=5006 - _globals['_EXECUTIONSTATECHANGEDETAILS']._serialized_end=5180 - _globals['_EXECUTIONUPDATERESPONSE']._serialized_start=5182 - _globals['_EXECUTIONUPDATERESPONSE']._serialized_end=5207 - _globals['_WORKFLOWEXECUTIONGETMETRICSREQUEST']._serialized_start=5209 - _globals['_WORKFLOWEXECUTIONGETMETRICSREQUEST']._serialized_end=5327 - _globals['_WORKFLOWEXECUTIONGETMETRICSRESPONSE']._serialized_start=5329 - _globals['_WORKFLOWEXECUTIONGETMETRICSRESPONSE']._serialized_end=5407 + _globals['_EXECUTIONMETADATA_EXECUTIONMODE']._serialized_end=3241 + _globals['_NOTIFICATIONLIST']._serialized_start=3243 + _globals['_NOTIFICATIONLIST']._serialized_end=3329 + _globals['_EXECUTIONSPEC']._serialized_start=3332 + _globals['_EXECUTIONSPEC']._serialized_end=4372 + _globals['_EXECUTIONTERMINATEREQUEST']._serialized_start=4374 + _globals['_EXECUTIONTERMINATEREQUEST']._serialized_end=4483 + _globals['_EXECUTIONTERMINATERESPONSE']._serialized_start=4485 + _globals['_EXECUTIONTERMINATERESPONSE']._serialized_end=4513 + _globals['_WORKFLOWEXECUTIONGETDATAREQUEST']._serialized_start=4515 + _globals['_WORKFLOWEXECUTIONGETDATAREQUEST']._serialized_end=4608 + _globals['_WORKFLOWEXECUTIONGETDATARESPONSE']._serialized_start=4611 + _globals['_WORKFLOWEXECUTIONGETDATARESPONSE']._serialized_end=4875 + _globals['_EXECUTIONUPDATEREQUEST']._serialized_start=4878 + _globals['_EXECUTIONUPDATEREQUEST']._serialized_end=5016 + _globals['_EXECUTIONSTATECHANGEDETAILS']._serialized_start=5019 + _globals['_EXECUTIONSTATECHANGEDETAILS']._serialized_end=5193 + _globals['_EXECUTIONUPDATERESPONSE']._serialized_start=5195 + _globals['_EXECUTIONUPDATERESPONSE']._serialized_end=5220 + _globals['_WORKFLOWEXECUTIONGETMETRICSREQUEST']._serialized_start=5222 + _globals['_WORKFLOWEXECUTIONGETMETRICSREQUEST']._serialized_end=5340 + _globals['_WORKFLOWEXECUTIONGETMETRICSRESPONSE']._serialized_start=5342 + _globals['_WORKFLOWEXECUTIONGETMETRICSRESPONSE']._serialized_end=5420 # @@protoc_insertion_point(module_scope) diff --git a/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.pyi b/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.pyi index bee241d74d..c832b3a429 100644 --- a/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.pyi +++ b/flyteidl/gen/pb_python/flyteidl/admin/execution_pb2.pyi @@ -156,12 +156,14 @@ class ExecutionMetadata(_message.Message): RELAUNCH: _ClassVar[ExecutionMetadata.ExecutionMode] CHILD_WORKFLOW: _ClassVar[ExecutionMetadata.ExecutionMode] RECOVERED: _ClassVar[ExecutionMetadata.ExecutionMode] + TRIGGER: _ClassVar[ExecutionMetadata.ExecutionMode] MANUAL: ExecutionMetadata.ExecutionMode SCHEDULED: ExecutionMetadata.ExecutionMode SYSTEM: ExecutionMetadata.ExecutionMode RELAUNCH: ExecutionMetadata.ExecutionMode CHILD_WORKFLOW: ExecutionMetadata.ExecutionMode RECOVERED: ExecutionMetadata.ExecutionMode + TRIGGER: ExecutionMetadata.ExecutionMode MODE_FIELD_NUMBER: _ClassVar[int] PRINCIPAL_FIELD_NUMBER: _ClassVar[int] NESTING_FIELD_NUMBER: _ClassVar[int] diff --git a/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.py b/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.py index 06e8ff7f81..cffe1484c7 100644 --- a/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.py +++ b/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.py @@ -15,7 +15,7 @@ from flyteidl.core import identifier_pb2 as flyteidl_dot_core_dot_identifier__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1f\x66lyteidl/core/artifact_id.proto\x12\rflyteidl.core\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1e\x66lyteidl/core/identifier.proto\"e\n\x0b\x41rtifactKey\x12\x18\n\x07project\x18\x01 \x01(\tR\x07project\x12\x16\n\x06\x64omain\x18\x02 \x01(\tR\x06\x64omain\x12\x12\n\x04name\x18\x03 \x01(\tR\x04name\x12\x10\n\x03org\x18\x04 \x01(\tR\x03org\"\xb9\x01\n\x13\x41rtifactBindingData\x12\x14\n\x05index\x18\x01 \x01(\rR\x05index\x12%\n\rpartition_key\x18\x02 \x01(\tH\x00R\x0cpartitionKey\x12\x35\n\x16\x62ind_to_time_partition\x18\x03 \x01(\x08H\x00R\x13\x62indToTimePartition\x12\x1c\n\ttransform\x18\x04 \x01(\tR\ttransformB\x10\n\x0epartition_data\"$\n\x10InputBindingData\x12\x10\n\x03var\x18\x01 \x01(\tR\x03var\"\x92\x02\n\nLabelValue\x12#\n\x0cstatic_value\x18\x01 \x01(\tH\x00R\x0bstaticValue\x12;\n\ntime_value\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.TimestampH\x00R\ttimeValue\x12Q\n\x11triggered_binding\x18\x03 \x01(\x0b\x32\".flyteidl.core.ArtifactBindingDataH\x00R\x10triggeredBinding\x12\x46\n\rinput_binding\x18\x04 \x01(\x0b\x32\x1f.flyteidl.core.InputBindingDataH\x00R\x0cinputBindingB\x07\n\x05value\"\x9d\x01\n\nPartitions\x12:\n\x05value\x18\x01 \x03(\x0b\x32$.flyteidl.core.Partitions.ValueEntryR\x05value\x1aS\n\nValueEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12/\n\x05value\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LabelValueR\x05value:\x02\x38\x01\"@\n\rTimePartition\x12/\n\x05value\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.LabelValueR\x05value\"\xe5\x01\n\nArtifactID\x12=\n\x0c\x61rtifact_key\x18\x01 \x01(\x0b\x32\x1a.flyteidl.core.ArtifactKeyR\x0b\x61rtifactKey\x12\x18\n\x07version\x18\x02 \x01(\tR\x07version\x12\x39\n\npartitions\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.PartitionsR\npartitions\x12\x43\n\x0etime_partition\x18\x04 \x01(\x0b\x32\x1c.flyteidl.core.TimePartitionR\rtimePartition\"}\n\x0b\x41rtifactTag\x12=\n\x0c\x61rtifact_key\x18\x01 \x01(\x0b\x32\x1a.flyteidl.core.ArtifactKeyR\x0b\x61rtifactKey\x12/\n\x05value\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LabelValueR\x05value\"\xf0\x01\n\rArtifactQuery\x12<\n\x0b\x61rtifact_id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.ArtifactIDH\x00R\nartifactId\x12?\n\x0c\x61rtifact_tag\x18\x02 \x01(\x0b\x32\x1a.flyteidl.core.ArtifactTagH\x00R\x0b\x61rtifactTag\x12\x12\n\x03uri\x18\x03 \x01(\tH\x00R\x03uri\x12>\n\x07\x62inding\x18\x04 \x01(\x0b\x32\".flyteidl.core.ArtifactBindingDataH\x00R\x07\x62indingB\x0c\n\nidentifierB\xb5\x01\n\x11\x63om.flyteidl.coreB\x0f\x41rtifactIdProtoP\x01Z:github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core\xa2\x02\x03\x46\x43X\xaa\x02\rFlyteidl.Core\xca\x02\rFlyteidl\\Core\xe2\x02\x19\x46lyteidl\\Core\\GPBMetadata\xea\x02\x0e\x46lyteidl::Coreb\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1f\x66lyteidl/core/artifact_id.proto\x12\rflyteidl.core\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1e\x66lyteidl/core/identifier.proto\"e\n\x0b\x41rtifactKey\x12\x18\n\x07project\x18\x01 \x01(\tR\x07project\x12\x16\n\x06\x64omain\x18\x02 \x01(\tR\x06\x64omain\x12\x12\n\x04name\x18\x03 \x01(\tR\x04name\x12\x10\n\x03org\x18\x04 \x01(\tR\x03org\"\xa3\x01\n\x13\x41rtifactBindingData\x12%\n\rpartition_key\x18\x02 \x01(\tH\x00R\x0cpartitionKey\x12\x35\n\x16\x62ind_to_time_partition\x18\x03 \x01(\x08H\x00R\x13\x62indToTimePartition\x12\x1c\n\ttransform\x18\x04 \x01(\tR\ttransformB\x10\n\x0epartition_data\"$\n\x10InputBindingData\x12\x10\n\x03var\x18\x01 \x01(\tR\x03var\"\x92\x02\n\nLabelValue\x12#\n\x0cstatic_value\x18\x01 \x01(\tH\x00R\x0bstaticValue\x12;\n\ntime_value\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.TimestampH\x00R\ttimeValue\x12Q\n\x11triggered_binding\x18\x03 \x01(\x0b\x32\".flyteidl.core.ArtifactBindingDataH\x00R\x10triggeredBinding\x12\x46\n\rinput_binding\x18\x04 \x01(\x0b\x32\x1f.flyteidl.core.InputBindingDataH\x00R\x0cinputBindingB\x07\n\x05value\"\x9d\x01\n\nPartitions\x12:\n\x05value\x18\x01 \x03(\x0b\x32$.flyteidl.core.Partitions.ValueEntryR\x05value\x1aS\n\nValueEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12/\n\x05value\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LabelValueR\x05value:\x02\x38\x01\"@\n\rTimePartition\x12/\n\x05value\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.LabelValueR\x05value\"\xe5\x01\n\nArtifactID\x12=\n\x0c\x61rtifact_key\x18\x01 \x01(\x0b\x32\x1a.flyteidl.core.ArtifactKeyR\x0b\x61rtifactKey\x12\x18\n\x07version\x18\x02 \x01(\tR\x07version\x12\x39\n\npartitions\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.PartitionsR\npartitions\x12\x43\n\x0etime_partition\x18\x04 \x01(\x0b\x32\x1c.flyteidl.core.TimePartitionR\rtimePartition\"}\n\x0b\x41rtifactTag\x12=\n\x0c\x61rtifact_key\x18\x01 \x01(\x0b\x32\x1a.flyteidl.core.ArtifactKeyR\x0b\x61rtifactKey\x12/\n\x05value\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.LabelValueR\x05value\"\xf0\x01\n\rArtifactQuery\x12<\n\x0b\x61rtifact_id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.ArtifactIDH\x00R\nartifactId\x12?\n\x0c\x61rtifact_tag\x18\x02 \x01(\x0b\x32\x1a.flyteidl.core.ArtifactTagH\x00R\x0b\x61rtifactTag\x12\x12\n\x03uri\x18\x03 \x01(\tH\x00R\x03uri\x12>\n\x07\x62inding\x18\x04 \x01(\x0b\x32\".flyteidl.core.ArtifactBindingDataH\x00R\x07\x62indingB\x0c\n\nidentifierB\xb5\x01\n\x11\x63om.flyteidl.coreB\x0f\x41rtifactIdProtoP\x01Z:github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core\xa2\x02\x03\x46\x43X\xaa\x02\rFlyteidl.Core\xca\x02\rFlyteidl\\Core\xe2\x02\x19\x46lyteidl\\Core\\GPBMetadata\xea\x02\x0e\x46lyteidl::Coreb\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -29,21 +29,21 @@ _globals['_ARTIFACTKEY']._serialized_start=115 _globals['_ARTIFACTKEY']._serialized_end=216 _globals['_ARTIFACTBINDINGDATA']._serialized_start=219 - _globals['_ARTIFACTBINDINGDATA']._serialized_end=404 - _globals['_INPUTBINDINGDATA']._serialized_start=406 - _globals['_INPUTBINDINGDATA']._serialized_end=442 - _globals['_LABELVALUE']._serialized_start=445 - _globals['_LABELVALUE']._serialized_end=719 - _globals['_PARTITIONS']._serialized_start=722 - _globals['_PARTITIONS']._serialized_end=879 - _globals['_PARTITIONS_VALUEENTRY']._serialized_start=796 - _globals['_PARTITIONS_VALUEENTRY']._serialized_end=879 - _globals['_TIMEPARTITION']._serialized_start=881 - _globals['_TIMEPARTITION']._serialized_end=945 - _globals['_ARTIFACTID']._serialized_start=948 - _globals['_ARTIFACTID']._serialized_end=1177 - _globals['_ARTIFACTTAG']._serialized_start=1179 - _globals['_ARTIFACTTAG']._serialized_end=1304 - _globals['_ARTIFACTQUERY']._serialized_start=1307 - _globals['_ARTIFACTQUERY']._serialized_end=1547 + _globals['_ARTIFACTBINDINGDATA']._serialized_end=382 + _globals['_INPUTBINDINGDATA']._serialized_start=384 + _globals['_INPUTBINDINGDATA']._serialized_end=420 + _globals['_LABELVALUE']._serialized_start=423 + _globals['_LABELVALUE']._serialized_end=697 + _globals['_PARTITIONS']._serialized_start=700 + _globals['_PARTITIONS']._serialized_end=857 + _globals['_PARTITIONS_VALUEENTRY']._serialized_start=774 + _globals['_PARTITIONS_VALUEENTRY']._serialized_end=857 + _globals['_TIMEPARTITION']._serialized_start=859 + _globals['_TIMEPARTITION']._serialized_end=923 + _globals['_ARTIFACTID']._serialized_start=926 + _globals['_ARTIFACTID']._serialized_end=1155 + _globals['_ARTIFACTTAG']._serialized_start=1157 + _globals['_ARTIFACTTAG']._serialized_end=1282 + _globals['_ARTIFACTQUERY']._serialized_start=1285 + _globals['_ARTIFACTQUERY']._serialized_end=1525 # @@protoc_insertion_point(module_scope) diff --git a/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.pyi b/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.pyi index 5eacdbb52f..e1e688c8e4 100644 --- a/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.pyi +++ b/flyteidl/gen/pb_python/flyteidl/core/artifact_id_pb2.pyi @@ -20,16 +20,14 @@ class ArtifactKey(_message.Message): def __init__(self, project: _Optional[str] = ..., domain: _Optional[str] = ..., name: _Optional[str] = ..., org: _Optional[str] = ...) -> None: ... class ArtifactBindingData(_message.Message): - __slots__ = ["index", "partition_key", "bind_to_time_partition", "transform"] - INDEX_FIELD_NUMBER: _ClassVar[int] + __slots__ = ["partition_key", "bind_to_time_partition", "transform"] PARTITION_KEY_FIELD_NUMBER: _ClassVar[int] BIND_TO_TIME_PARTITION_FIELD_NUMBER: _ClassVar[int] TRANSFORM_FIELD_NUMBER: _ClassVar[int] - index: int partition_key: str bind_to_time_partition: bool transform: str - def __init__(self, index: _Optional[int] = ..., partition_key: _Optional[str] = ..., bind_to_time_partition: bool = ..., transform: _Optional[str] = ...) -> None: ... + def __init__(self, partition_key: _Optional[str] = ..., bind_to_time_partition: bool = ..., transform: _Optional[str] = ...) -> None: ... class InputBindingData(_message.Message): __slots__ = ["var"] diff --git a/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.py b/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.py index ab629c2b9f..4c07024320 100644 --- a/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.py +++ b/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.py @@ -22,7 +22,7 @@ from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1c\x66lyteidl/core/workflow.proto\x12\rflyteidl.core\x1a\x1d\x66lyteidl/core/condition.proto\x1a\x1d\x66lyteidl/core/execution.proto\x1a\x1e\x66lyteidl/core/identifier.proto\x1a\x1d\x66lyteidl/core/interface.proto\x1a\x1c\x66lyteidl/core/literals.proto\x1a\x19\x66lyteidl/core/tasks.proto\x1a\x19\x66lyteidl/core/types.proto\x1a\x1c\x66lyteidl/core/security.proto\x1a\x1egoogle/protobuf/duration.proto\"{\n\x07IfBlock\x12>\n\tcondition\x18\x01 \x01(\x0b\x32 .flyteidl.core.BooleanExpressionR\tcondition\x12\x30\n\tthen_node\x18\x02 \x01(\x0b\x32\x13.flyteidl.core.NodeR\x08thenNode\"\xd4\x01\n\x0bIfElseBlock\x12*\n\x04\x63\x61se\x18\x01 \x01(\x0b\x32\x16.flyteidl.core.IfBlockR\x04\x63\x61se\x12,\n\x05other\x18\x02 \x03(\x0b\x32\x16.flyteidl.core.IfBlockR\x05other\x12\x32\n\telse_node\x18\x03 \x01(\x0b\x32\x13.flyteidl.core.NodeH\x00R\x08\x65lseNode\x12,\n\x05\x65rror\x18\x04 \x01(\x0b\x32\x14.flyteidl.core.ErrorH\x00R\x05\x65rrorB\t\n\x07\x64\x65\x66\x61ult\"A\n\nBranchNode\x12\x33\n\x07if_else\x18\x01 \x01(\x0b\x32\x1a.flyteidl.core.IfElseBlockR\x06ifElse\"\x97\x01\n\x08TaskNode\x12>\n\x0creference_id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierH\x00R\x0breferenceId\x12>\n\toverrides\x18\x02 \x01(\x0b\x32 .flyteidl.core.TaskNodeOverridesR\toverridesB\x0b\n\treference\"\xa6\x01\n\x0cWorkflowNode\x12\x42\n\x0elaunchplan_ref\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierH\x00R\rlaunchplanRef\x12\x45\n\x10sub_workflow_ref\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.IdentifierH\x00R\x0esubWorkflowRefB\x0b\n\treference\"/\n\x10\x41pproveCondition\x12\x1b\n\tsignal_id\x18\x01 \x01(\tR\x08signalId\"\x90\x01\n\x0fSignalCondition\x12\x1b\n\tsignal_id\x18\x01 \x01(\tR\x08signalId\x12.\n\x04type\x18\x02 \x01(\x0b\x32\x1a.flyteidl.core.LiteralTypeR\x04type\x12\x30\n\x14output_variable_name\x18\x03 \x01(\tR\x12outputVariableName\"G\n\x0eSleepCondition\x12\x35\n\x08\x64uration\x18\x01 \x01(\x0b\x32\x19.google.protobuf.DurationR\x08\x64uration\"\xc5\x01\n\x08GateNode\x12;\n\x07\x61pprove\x18\x01 \x01(\x0b\x32\x1f.flyteidl.core.ApproveConditionH\x00R\x07\x61pprove\x12\x38\n\x06signal\x18\x02 \x01(\x0b\x32\x1e.flyteidl.core.SignalConditionH\x00R\x06signal\x12\x35\n\x05sleep\x18\x03 \x01(\x0b\x32\x1d.flyteidl.core.SleepConditionH\x00R\x05sleepB\x0b\n\tcondition\"\xbf\x01\n\tArrayNode\x12\'\n\x04node\x18\x01 \x01(\x0b\x32\x13.flyteidl.core.NodeR\x04node\x12 \n\x0bparallelism\x18\x02 \x01(\rR\x0bparallelism\x12%\n\rmin_successes\x18\x03 \x01(\rH\x00R\x0cminSuccesses\x12,\n\x11min_success_ratio\x18\x04 \x01(\x02H\x00R\x0fminSuccessRatioB\x12\n\x10success_criteria\"\x8c\x03\n\x0cNodeMetadata\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x33\n\x07timeout\x18\x04 \x01(\x0b\x32\x19.google.protobuf.DurationR\x07timeout\x12\x36\n\x07retries\x18\x05 \x01(\x0b\x32\x1c.flyteidl.core.RetryStrategyR\x07retries\x12&\n\rinterruptible\x18\x06 \x01(\x08H\x00R\rinterruptible\x12\x1e\n\tcacheable\x18\x07 \x01(\x08H\x01R\tcacheable\x12%\n\rcache_version\x18\x08 \x01(\tH\x02R\x0c\x63\x61\x63heVersion\x12/\n\x12\x63\x61\x63he_serializable\x18\t \x01(\x08H\x03R\x11\x63\x61\x63heSerializableB\x15\n\x13interruptible_valueB\x11\n\x0f\x63\x61\x63heable_valueB\x15\n\x13\x63\x61\x63he_version_valueB\x1a\n\x18\x63\x61\x63he_serializable_value\"/\n\x05\x41lias\x12\x10\n\x03var\x18\x01 \x01(\tR\x03var\x12\x14\n\x05\x61lias\x18\x02 \x01(\tR\x05\x61lias\"\x9f\x04\n\x04Node\x12\x0e\n\x02id\x18\x01 \x01(\tR\x02id\x12\x37\n\x08metadata\x18\x02 \x01(\x0b\x32\x1b.flyteidl.core.NodeMetadataR\x08metadata\x12.\n\x06inputs\x18\x03 \x03(\x0b\x32\x16.flyteidl.core.BindingR\x06inputs\x12*\n\x11upstream_node_ids\x18\x04 \x03(\tR\x0fupstreamNodeIds\x12;\n\x0eoutput_aliases\x18\x05 \x03(\x0b\x32\x14.flyteidl.core.AliasR\routputAliases\x12\x36\n\ttask_node\x18\x06 \x01(\x0b\x32\x17.flyteidl.core.TaskNodeH\x00R\x08taskNode\x12\x42\n\rworkflow_node\x18\x07 \x01(\x0b\x32\x1b.flyteidl.core.WorkflowNodeH\x00R\x0cworkflowNode\x12<\n\x0b\x62ranch_node\x18\x08 \x01(\x0b\x32\x19.flyteidl.core.BranchNodeH\x00R\nbranchNode\x12\x36\n\tgate_node\x18\t \x01(\x0b\x32\x17.flyteidl.core.GateNodeH\x00R\x08gateNode\x12\x39\n\narray_node\x18\n \x01(\x0b\x32\x18.flyteidl.core.ArrayNodeH\x00R\tarrayNodeB\x08\n\x06target\"\xfc\x02\n\x10WorkflowMetadata\x12M\n\x12quality_of_service\x18\x01 \x01(\x0b\x32\x1f.flyteidl.core.QualityOfServiceR\x10qualityOfService\x12N\n\non_failure\x18\x02 \x01(\x0e\x32/.flyteidl.core.WorkflowMetadata.OnFailurePolicyR\tonFailure\x12=\n\x04tags\x18\x03 \x03(\x0b\x32).flyteidl.core.WorkflowMetadata.TagsEntryR\x04tags\x1a\x37\n\tTagsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\"Q\n\x0fOnFailurePolicy\x12\x14\n\x10\x46\x41IL_IMMEDIATELY\x10\x00\x12(\n$FAIL_AFTER_EXECUTABLE_NODES_COMPLETE\x10\x01\"@\n\x18WorkflowMetadataDefaults\x12$\n\rinterruptible\x18\x01 \x01(\x08R\rinterruptible\"\xa2\x03\n\x10WorkflowTemplate\x12)\n\x02id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\x02id\x12;\n\x08metadata\x18\x02 \x01(\x0b\x32\x1f.flyteidl.core.WorkflowMetadataR\x08metadata\x12;\n\tinterface\x18\x03 \x01(\x0b\x32\x1d.flyteidl.core.TypedInterfaceR\tinterface\x12)\n\x05nodes\x18\x04 \x03(\x0b\x32\x13.flyteidl.core.NodeR\x05nodes\x12\x30\n\x07outputs\x18\x05 \x03(\x0b\x32\x16.flyteidl.core.BindingR\x07outputs\x12\x36\n\x0c\x66\x61ilure_node\x18\x06 \x01(\x0b\x32\x13.flyteidl.core.NodeR\x0b\x66\x61ilureNode\x12T\n\x11metadata_defaults\x18\x07 \x01(\x0b\x32\'.flyteidl.core.WorkflowMetadataDefaultsR\x10metadataDefaults\"\x9c\x01\n\x11TaskNodeOverrides\x12\x36\n\tresources\x18\x01 \x01(\x0b\x32\x18.flyteidl.core.ResourcesR\tresources\x12O\n\x12\x65xtended_resources\x18\x02 \x01(\x0b\x32 .flyteidl.core.ExtendedResourcesR\x11\x65xtendedResources\"\xba\x01\n\x12LaunchPlanTemplate\x12)\n\x02id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\x02id\x12;\n\tinterface\x18\x02 \x01(\x0b\x32\x1d.flyteidl.core.TypedInterfaceR\tinterface\x12<\n\x0c\x66ixed_inputs\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x0b\x66ixedInputsB\xb3\x01\n\x11\x63om.flyteidl.coreB\rWorkflowProtoP\x01Z:github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core\xa2\x02\x03\x46\x43X\xaa\x02\rFlyteidl.Core\xca\x02\rFlyteidl\\Core\xe2\x02\x19\x46lyteidl\\Core\\GPBMetadata\xea\x02\x0e\x46lyteidl::Coreb\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1c\x66lyteidl/core/workflow.proto\x12\rflyteidl.core\x1a\x1d\x66lyteidl/core/condition.proto\x1a\x1d\x66lyteidl/core/execution.proto\x1a\x1e\x66lyteidl/core/identifier.proto\x1a\x1d\x66lyteidl/core/interface.proto\x1a\x1c\x66lyteidl/core/literals.proto\x1a\x19\x66lyteidl/core/tasks.proto\x1a\x19\x66lyteidl/core/types.proto\x1a\x1c\x66lyteidl/core/security.proto\x1a\x1egoogle/protobuf/duration.proto\"{\n\x07IfBlock\x12>\n\tcondition\x18\x01 \x01(\x0b\x32 .flyteidl.core.BooleanExpressionR\tcondition\x12\x30\n\tthen_node\x18\x02 \x01(\x0b\x32\x13.flyteidl.core.NodeR\x08thenNode\"\xd4\x01\n\x0bIfElseBlock\x12*\n\x04\x63\x61se\x18\x01 \x01(\x0b\x32\x16.flyteidl.core.IfBlockR\x04\x63\x61se\x12,\n\x05other\x18\x02 \x03(\x0b\x32\x16.flyteidl.core.IfBlockR\x05other\x12\x32\n\telse_node\x18\x03 \x01(\x0b\x32\x13.flyteidl.core.NodeH\x00R\x08\x65lseNode\x12,\n\x05\x65rror\x18\x04 \x01(\x0b\x32\x14.flyteidl.core.ErrorH\x00R\x05\x65rrorB\t\n\x07\x64\x65\x66\x61ult\"A\n\nBranchNode\x12\x33\n\x07if_else\x18\x01 \x01(\x0b\x32\x1a.flyteidl.core.IfElseBlockR\x06ifElse\"\x97\x01\n\x08TaskNode\x12>\n\x0creference_id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierH\x00R\x0breferenceId\x12>\n\toverrides\x18\x02 \x01(\x0b\x32 .flyteidl.core.TaskNodeOverridesR\toverridesB\x0b\n\treference\"\xa6\x01\n\x0cWorkflowNode\x12\x42\n\x0elaunchplan_ref\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierH\x00R\rlaunchplanRef\x12\x45\n\x10sub_workflow_ref\x18\x02 \x01(\x0b\x32\x19.flyteidl.core.IdentifierH\x00R\x0esubWorkflowRefB\x0b\n\treference\"/\n\x10\x41pproveCondition\x12\x1b\n\tsignal_id\x18\x01 \x01(\tR\x08signalId\"\x90\x01\n\x0fSignalCondition\x12\x1b\n\tsignal_id\x18\x01 \x01(\tR\x08signalId\x12.\n\x04type\x18\x02 \x01(\x0b\x32\x1a.flyteidl.core.LiteralTypeR\x04type\x12\x30\n\x14output_variable_name\x18\x03 \x01(\tR\x12outputVariableName\"G\n\x0eSleepCondition\x12\x35\n\x08\x64uration\x18\x01 \x01(\x0b\x32\x19.google.protobuf.DurationR\x08\x64uration\"\xc5\x01\n\x08GateNode\x12;\n\x07\x61pprove\x18\x01 \x01(\x0b\x32\x1f.flyteidl.core.ApproveConditionH\x00R\x07\x61pprove\x12\x38\n\x06signal\x18\x02 \x01(\x0b\x32\x1e.flyteidl.core.SignalConditionH\x00R\x06signal\x12\x35\n\x05sleep\x18\x03 \x01(\x0b\x32\x1d.flyteidl.core.SleepConditionH\x00R\x05sleepB\x0b\n\tcondition\"\xbf\x01\n\tArrayNode\x12\'\n\x04node\x18\x01 \x01(\x0b\x32\x13.flyteidl.core.NodeR\x04node\x12 \n\x0bparallelism\x18\x02 \x01(\rR\x0bparallelism\x12%\n\rmin_successes\x18\x03 \x01(\rH\x00R\x0cminSuccesses\x12,\n\x11min_success_ratio\x18\x04 \x01(\x02H\x00R\x0fminSuccessRatioB\x12\n\x10success_criteria\"\x8c\x03\n\x0cNodeMetadata\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x33\n\x07timeout\x18\x04 \x01(\x0b\x32\x19.google.protobuf.DurationR\x07timeout\x12\x36\n\x07retries\x18\x05 \x01(\x0b\x32\x1c.flyteidl.core.RetryStrategyR\x07retries\x12&\n\rinterruptible\x18\x06 \x01(\x08H\x00R\rinterruptible\x12\x1e\n\tcacheable\x18\x07 \x01(\x08H\x01R\tcacheable\x12%\n\rcache_version\x18\x08 \x01(\tH\x02R\x0c\x63\x61\x63heVersion\x12/\n\x12\x63\x61\x63he_serializable\x18\t \x01(\x08H\x03R\x11\x63\x61\x63heSerializableB\x15\n\x13interruptible_valueB\x11\n\x0f\x63\x61\x63heable_valueB\x15\n\x13\x63\x61\x63he_version_valueB\x1a\n\x18\x63\x61\x63he_serializable_value\"/\n\x05\x41lias\x12\x10\n\x03var\x18\x01 \x01(\tR\x03var\x12\x14\n\x05\x61lias\x18\x02 \x01(\tR\x05\x61lias\"\x9f\x04\n\x04Node\x12\x0e\n\x02id\x18\x01 \x01(\tR\x02id\x12\x37\n\x08metadata\x18\x02 \x01(\x0b\x32\x1b.flyteidl.core.NodeMetadataR\x08metadata\x12.\n\x06inputs\x18\x03 \x03(\x0b\x32\x16.flyteidl.core.BindingR\x06inputs\x12*\n\x11upstream_node_ids\x18\x04 \x03(\tR\x0fupstreamNodeIds\x12;\n\x0eoutput_aliases\x18\x05 \x03(\x0b\x32\x14.flyteidl.core.AliasR\routputAliases\x12\x36\n\ttask_node\x18\x06 \x01(\x0b\x32\x17.flyteidl.core.TaskNodeH\x00R\x08taskNode\x12\x42\n\rworkflow_node\x18\x07 \x01(\x0b\x32\x1b.flyteidl.core.WorkflowNodeH\x00R\x0cworkflowNode\x12<\n\x0b\x62ranch_node\x18\x08 \x01(\x0b\x32\x19.flyteidl.core.BranchNodeH\x00R\nbranchNode\x12\x36\n\tgate_node\x18\t \x01(\x0b\x32\x17.flyteidl.core.GateNodeH\x00R\x08gateNode\x12\x39\n\narray_node\x18\n \x01(\x0b\x32\x18.flyteidl.core.ArrayNodeH\x00R\tarrayNodeB\x08\n\x06target\"\xfc\x02\n\x10WorkflowMetadata\x12M\n\x12quality_of_service\x18\x01 \x01(\x0b\x32\x1f.flyteidl.core.QualityOfServiceR\x10qualityOfService\x12N\n\non_failure\x18\x02 \x01(\x0e\x32/.flyteidl.core.WorkflowMetadata.OnFailurePolicyR\tonFailure\x12=\n\x04tags\x18\x03 \x03(\x0b\x32).flyteidl.core.WorkflowMetadata.TagsEntryR\x04tags\x1a\x37\n\tTagsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\tR\x05value:\x02\x38\x01\"Q\n\x0fOnFailurePolicy\x12\x14\n\x10\x46\x41IL_IMMEDIATELY\x10\x00\x12(\n$FAIL_AFTER_EXECUTABLE_NODES_COMPLETE\x10\x01\"@\n\x18WorkflowMetadataDefaults\x12$\n\rinterruptible\x18\x01 \x01(\x08R\rinterruptible\"\xa2\x03\n\x10WorkflowTemplate\x12)\n\x02id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\x02id\x12;\n\x08metadata\x18\x02 \x01(\x0b\x32\x1f.flyteidl.core.WorkflowMetadataR\x08metadata\x12;\n\tinterface\x18\x03 \x01(\x0b\x32\x1d.flyteidl.core.TypedInterfaceR\tinterface\x12)\n\x05nodes\x18\x04 \x03(\x0b\x32\x13.flyteidl.core.NodeR\x05nodes\x12\x30\n\x07outputs\x18\x05 \x03(\x0b\x32\x16.flyteidl.core.BindingR\x07outputs\x12\x36\n\x0c\x66\x61ilure_node\x18\x06 \x01(\x0b\x32\x13.flyteidl.core.NodeR\x0b\x66\x61ilureNode\x12T\n\x11metadata_defaults\x18\x07 \x01(\x0b\x32\'.flyteidl.core.WorkflowMetadataDefaultsR\x10metadataDefaults\"\xc5\x01\n\x11TaskNodeOverrides\x12\x36\n\tresources\x18\x01 \x01(\x0b\x32\x18.flyteidl.core.ResourcesR\tresources\x12O\n\x12\x65xtended_resources\x18\x02 \x01(\x0b\x32 .flyteidl.core.ExtendedResourcesR\x11\x65xtendedResources\x12\'\n\x0f\x63ontainer_image\x18\x03 \x01(\tR\x0e\x63ontainerImage\"\xba\x01\n\x12LaunchPlanTemplate\x12)\n\x02id\x18\x01 \x01(\x0b\x32\x19.flyteidl.core.IdentifierR\x02id\x12;\n\tinterface\x18\x02 \x01(\x0b\x32\x1d.flyteidl.core.TypedInterfaceR\tinterface\x12<\n\x0c\x66ixed_inputs\x18\x03 \x01(\x0b\x32\x19.flyteidl.core.LiteralMapR\x0b\x66ixedInputsB\xb3\x01\n\x11\x63om.flyteidl.coreB\rWorkflowProtoP\x01Z:github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core\xa2\x02\x03\x46\x43X\xaa\x02\rFlyteidl.Core\xca\x02\rFlyteidl\\Core\xe2\x02\x19\x46lyteidl\\Core\\GPBMetadata\xea\x02\x0e\x46lyteidl::Coreb\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -70,7 +70,7 @@ _globals['_WORKFLOWTEMPLATE']._serialized_start=3155 _globals['_WORKFLOWTEMPLATE']._serialized_end=3573 _globals['_TASKNODEOVERRIDES']._serialized_start=3576 - _globals['_TASKNODEOVERRIDES']._serialized_end=3732 - _globals['_LAUNCHPLANTEMPLATE']._serialized_start=3735 - _globals['_LAUNCHPLANTEMPLATE']._serialized_end=3921 + _globals['_TASKNODEOVERRIDES']._serialized_end=3773 + _globals['_LAUNCHPLANTEMPLATE']._serialized_start=3776 + _globals['_LAUNCHPLANTEMPLATE']._serialized_end=3962 # @@protoc_insertion_point(module_scope) diff --git a/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.pyi b/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.pyi index efee1e9ec2..11efca2392 100644 --- a/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.pyi +++ b/flyteidl/gen/pb_python/flyteidl/core/workflow_pb2.pyi @@ -199,12 +199,14 @@ class WorkflowTemplate(_message.Message): def __init__(self, id: _Optional[_Union[_identifier_pb2.Identifier, _Mapping]] = ..., metadata: _Optional[_Union[WorkflowMetadata, _Mapping]] = ..., interface: _Optional[_Union[_interface_pb2.TypedInterface, _Mapping]] = ..., nodes: _Optional[_Iterable[_Union[Node, _Mapping]]] = ..., outputs: _Optional[_Iterable[_Union[_literals_pb2.Binding, _Mapping]]] = ..., failure_node: _Optional[_Union[Node, _Mapping]] = ..., metadata_defaults: _Optional[_Union[WorkflowMetadataDefaults, _Mapping]] = ...) -> None: ... class TaskNodeOverrides(_message.Message): - __slots__ = ["resources", "extended_resources"] + __slots__ = ["resources", "extended_resources", "container_image"] RESOURCES_FIELD_NUMBER: _ClassVar[int] EXTENDED_RESOURCES_FIELD_NUMBER: _ClassVar[int] + CONTAINER_IMAGE_FIELD_NUMBER: _ClassVar[int] resources: _tasks_pb2.Resources extended_resources: _tasks_pb2.ExtendedResources - def __init__(self, resources: _Optional[_Union[_tasks_pb2.Resources, _Mapping]] = ..., extended_resources: _Optional[_Union[_tasks_pb2.ExtendedResources, _Mapping]] = ...) -> None: ... + container_image: str + def __init__(self, resources: _Optional[_Union[_tasks_pb2.Resources, _Mapping]] = ..., extended_resources: _Optional[_Union[_tasks_pb2.ExtendedResources, _Mapping]] = ..., container_image: _Optional[str] = ...) -> None: ... class LaunchPlanTemplate(_message.Message): __slots__ = ["id", "interface", "fixed_inputs"] diff --git a/flyteidl/gen/pb_python/flyteidl/service/agent_pb2.py b/flyteidl/gen/pb_python/flyteidl/service/agent_pb2.py index 4a1a3d8f70..2c4f728fba 100644 --- a/flyteidl/gen/pb_python/flyteidl/service/agent_pb2.py +++ b/flyteidl/gen/pb_python/flyteidl/service/agent_pb2.py @@ -15,7 +15,7 @@ from flyteidl.admin import agent_pb2 as flyteidl_dot_admin_dot_agent__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1c\x66lyteidl/service/agent.proto\x12\x10\x66lyteidl.service\x1a\x1cgoogle/api/annotations.proto\x1a\x1a\x66lyteidl/admin/agent.proto2\xcc\x03\n\x11\x41syncAgentService\x12U\n\nCreateTask\x12!.flyteidl.admin.CreateTaskRequest\x1a\".flyteidl.admin.CreateTaskResponse\"\x00\x12L\n\x07GetTask\x12\x1e.flyteidl.admin.GetTaskRequest\x1a\x1f.flyteidl.admin.GetTaskResponse\"\x00\x12U\n\nDeleteTask\x12!.flyteidl.admin.DeleteTaskRequest\x1a\".flyteidl.admin.DeleteTaskResponse\"\x00\x12\x61\n\x0eGetTaskMetrics\x12%.flyteidl.admin.GetTaskMetricsRequest\x1a&.flyteidl.admin.GetTaskMetricsResponse\"\x00\x12X\n\x0bGetTaskLogs\x12\".flyteidl.admin.GetTaskLogsRequest\x1a#.flyteidl.admin.GetTaskLogsResponse\"\x00\x32\xf0\x01\n\x14\x41gentMetadataService\x12k\n\x08GetAgent\x12\x1f.flyteidl.admin.GetAgentRequest\x1a .flyteidl.admin.GetAgentResponse\"\x1c\x82\xd3\xe4\x93\x02\x16\x12\x14/api/v1/agent/{name}\x12k\n\nListAgents\x12!.flyteidl.admin.ListAgentsRequest\x1a\".flyteidl.admin.ListAgentsResponse\"\x16\x82\xd3\xe4\x93\x02\x10\x12\x0e/api/v1/agentsB\xc2\x01\n\x14\x63om.flyteidl.serviceB\nAgentProtoP\x01Z=github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service\xa2\x02\x03\x46SX\xaa\x02\x10\x46lyteidl.Service\xca\x02\x10\x46lyteidl\\Service\xe2\x02\x1c\x46lyteidl\\Service\\GPBMetadata\xea\x02\x11\x46lyteidl::Serviceb\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1c\x66lyteidl/service/agent.proto\x12\x10\x66lyteidl.service\x1a\x1cgoogle/api/annotations.proto\x1a\x1a\x66lyteidl/admin/agent.proto2\xa1\x01\n\x10SyncAgentService\x12\x8c\x01\n\x0f\x45xecuteTaskSync\x12&.flyteidl.admin.ExecuteTaskSyncRequest\x1a\'.flyteidl.admin.ExecuteTaskSyncResponse\"$\x82\xd3\xe4\x93\x02\x1e:\x01*\"\x19/api/v1/agent/task/stream(\x01\x30\x01\x32\xe3\x06\n\x11\x41syncAgentService\x12r\n\nCreateTask\x12!.flyteidl.admin.CreateTaskRequest\x1a\".flyteidl.admin.CreateTaskResponse\"\x1d\x82\xd3\xe4\x93\x02\x17:\x01*\"\x12/api/v1/agent/task\x12\xa3\x01\n\x07GetTask\x12\x1e.flyteidl.admin.GetTaskRequest\x1a\x1f.flyteidl.admin.GetTaskResponse\"W\x82\xd3\xe4\x93\x02Q\x12O/api/v1/agent/task/{task_category.name}/{task_category.version}/{resource_meta}\x12\xb7\x01\n\nDeleteTask\x12!.flyteidl.admin.DeleteTaskRequest\x1a\".flyteidl.admin.DeleteTaskResponse\"b\x82\xd3\xe4\x93\x02\\*Z/api/v1/agent/task_executions/{task_category.name}/{task_category.version}/{resource_meta}\x12\xc0\x01\n\x0eGetTaskMetrics\x12%.flyteidl.admin.GetTaskMetricsRequest\x1a&.flyteidl.admin.GetTaskMetricsResponse\"_\x82\xd3\xe4\x93\x02Y\x12W/api/v1/agent/task/metrics/{task_category.name}/{task_category.version}/{resource_meta}\x12\xb6\x01\n\x0bGetTaskLogs\x12\".flyteidl.admin.GetTaskLogsRequest\x1a#.flyteidl.admin.GetTaskLogsResponse\"\\\x82\xd3\xe4\x93\x02V\x12T/api/v1/agent/task/logs/{task_category.name}/{task_category.version}/{resource_meta}0\x01\x32\xf0\x01\n\x14\x41gentMetadataService\x12k\n\x08GetAgent\x12\x1f.flyteidl.admin.GetAgentRequest\x1a .flyteidl.admin.GetAgentResponse\"\x1c\x82\xd3\xe4\x93\x02\x16\x12\x14/api/v1/agent/{name}\x12k\n\nListAgents\x12!.flyteidl.admin.ListAgentsRequest\x1a\".flyteidl.admin.ListAgentsResponse\"\x16\x82\xd3\xe4\x93\x02\x10\x12\x0e/api/v1/agentsB\xc2\x01\n\x14\x63om.flyteidl.serviceB\nAgentProtoP\x01Z=github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service\xa2\x02\x03\x46SX\xaa\x02\x10\x46lyteidl.Service\xca\x02\x10\x46lyteidl\\Service\xe2\x02\x1c\x46lyteidl\\Service\\GPBMetadata\xea\x02\x11\x46lyteidl::Serviceb\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -24,12 +24,26 @@ DESCRIPTOR._options = None DESCRIPTOR._serialized_options = b'\n\024com.flyteidl.serviceB\nAgentProtoP\001Z=github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service\242\002\003FSX\252\002\020Flyteidl.Service\312\002\020Flyteidl\\Service\342\002\034Flyteidl\\Service\\GPBMetadata\352\002\021Flyteidl::Service' + _SYNCAGENTSERVICE.methods_by_name['ExecuteTaskSync']._options = None + _SYNCAGENTSERVICE.methods_by_name['ExecuteTaskSync']._serialized_options = b'\202\323\344\223\002\036:\001*\"\031/api/v1/agent/task/stream' + _ASYNCAGENTSERVICE.methods_by_name['CreateTask']._options = None + _ASYNCAGENTSERVICE.methods_by_name['CreateTask']._serialized_options = b'\202\323\344\223\002\027:\001*\"\022/api/v1/agent/task' + _ASYNCAGENTSERVICE.methods_by_name['GetTask']._options = None + _ASYNCAGENTSERVICE.methods_by_name['GetTask']._serialized_options = b'\202\323\344\223\002Q\022O/api/v1/agent/task/{task_category.name}/{task_category.version}/{resource_meta}' + _ASYNCAGENTSERVICE.methods_by_name['DeleteTask']._options = None + _ASYNCAGENTSERVICE.methods_by_name['DeleteTask']._serialized_options = b'\202\323\344\223\002\\*Z/api/v1/agent/task_executions/{task_category.name}/{task_category.version}/{resource_meta}' + _ASYNCAGENTSERVICE.methods_by_name['GetTaskMetrics']._options = None + _ASYNCAGENTSERVICE.methods_by_name['GetTaskMetrics']._serialized_options = b'\202\323\344\223\002Y\022W/api/v1/agent/task/metrics/{task_category.name}/{task_category.version}/{resource_meta}' + _ASYNCAGENTSERVICE.methods_by_name['GetTaskLogs']._options = None + _ASYNCAGENTSERVICE.methods_by_name['GetTaskLogs']._serialized_options = b'\202\323\344\223\002V\022T/api/v1/agent/task/logs/{task_category.name}/{task_category.version}/{resource_meta}' _AGENTMETADATASERVICE.methods_by_name['GetAgent']._options = None _AGENTMETADATASERVICE.methods_by_name['GetAgent']._serialized_options = b'\202\323\344\223\002\026\022\024/api/v1/agent/{name}' _AGENTMETADATASERVICE.methods_by_name['ListAgents']._options = None _AGENTMETADATASERVICE.methods_by_name['ListAgents']._serialized_options = b'\202\323\344\223\002\020\022\016/api/v1/agents' - _globals['_ASYNCAGENTSERVICE']._serialized_start=109 - _globals['_ASYNCAGENTSERVICE']._serialized_end=569 - _globals['_AGENTMETADATASERVICE']._serialized_start=572 - _globals['_AGENTMETADATASERVICE']._serialized_end=812 + _globals['_SYNCAGENTSERVICE']._serialized_start=109 + _globals['_SYNCAGENTSERVICE']._serialized_end=270 + _globals['_ASYNCAGENTSERVICE']._serialized_start=273 + _globals['_ASYNCAGENTSERVICE']._serialized_end=1140 + _globals['_AGENTMETADATASERVICE']._serialized_start=1143 + _globals['_AGENTMETADATASERVICE']._serialized_end=1383 # @@protoc_insertion_point(module_scope) diff --git a/flyteidl/gen/pb_python/flyteidl/service/agent_pb2_grpc.py b/flyteidl/gen/pb_python/flyteidl/service/agent_pb2_grpc.py index 7d04b0cd33..8436a9d17f 100644 --- a/flyteidl/gen/pb_python/flyteidl/service/agent_pb2_grpc.py +++ b/flyteidl/gen/pb_python/flyteidl/service/agent_pb2_grpc.py @@ -5,8 +5,73 @@ from flyteidl.admin import agent_pb2 as flyteidl_dot_admin_dot_agent__pb2 +class SyncAgentServiceStub(object): + """SyncAgentService defines an RPC Service that allows propeller to send the request to the agent server synchronously. + """ + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.ExecuteTaskSync = channel.stream_stream( + '/flyteidl.service.SyncAgentService/ExecuteTaskSync', + request_serializer=flyteidl_dot_admin_dot_agent__pb2.ExecuteTaskSyncRequest.SerializeToString, + response_deserializer=flyteidl_dot_admin_dot_agent__pb2.ExecuteTaskSyncResponse.FromString, + ) + + +class SyncAgentServiceServicer(object): + """SyncAgentService defines an RPC Service that allows propeller to send the request to the agent server synchronously. + """ + + def ExecuteTaskSync(self, request_iterator, context): + """ExecuteTaskSync streams the create request and inputs to the agent service and streams the outputs back. + """ + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_SyncAgentServiceServicer_to_server(servicer, server): + rpc_method_handlers = { + 'ExecuteTaskSync': grpc.stream_stream_rpc_method_handler( + servicer.ExecuteTaskSync, + request_deserializer=flyteidl_dot_admin_dot_agent__pb2.ExecuteTaskSyncRequest.FromString, + response_serializer=flyteidl_dot_admin_dot_agent__pb2.ExecuteTaskSyncResponse.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'flyteidl.service.SyncAgentService', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + + # This class is part of an EXPERIMENTAL API. +class SyncAgentService(object): + """SyncAgentService defines an RPC Service that allows propeller to send the request to the agent server synchronously. + """ + + @staticmethod + def ExecuteTaskSync(request_iterator, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.stream_stream(request_iterator, target, '/flyteidl.service.SyncAgentService/ExecuteTaskSync', + flyteidl_dot_admin_dot_agent__pb2.ExecuteTaskSyncRequest.SerializeToString, + flyteidl_dot_admin_dot_agent__pb2.ExecuteTaskSyncResponse.FromString, + options, channel_credentials, + insecure, call_credentials, compression, wait_for_ready, timeout, metadata) + + class AsyncAgentServiceStub(object): - """AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server. + """AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server asynchronously. """ def __init__(self, channel): @@ -35,7 +100,7 @@ def __init__(self, channel): request_serializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskMetricsRequest.SerializeToString, response_deserializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskMetricsResponse.FromString, ) - self.GetTaskLogs = channel.unary_unary( + self.GetTaskLogs = channel.unary_stream( '/flyteidl.service.AsyncAgentService/GetTaskLogs', request_serializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskLogsRequest.SerializeToString, response_deserializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskLogsResponse.FromString, @@ -43,11 +108,11 @@ def __init__(self, channel): class AsyncAgentServiceServicer(object): - """AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server. + """AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server asynchronously. """ def CreateTask(self, request, context): - """Send a task create request to the agent server. + """CreateTask sends a task create request to the agent service. """ context.set_code(grpc.StatusCode.UNIMPLEMENTED) context.set_details('Method not implemented!') @@ -108,7 +173,7 @@ def add_AsyncAgentServiceServicer_to_server(servicer, server): request_deserializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskMetricsRequest.FromString, response_serializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskMetricsResponse.SerializeToString, ), - 'GetTaskLogs': grpc.unary_unary_rpc_method_handler( + 'GetTaskLogs': grpc.unary_stream_rpc_method_handler( servicer.GetTaskLogs, request_deserializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskLogsRequest.FromString, response_serializer=flyteidl_dot_admin_dot_agent__pb2.GetTaskLogsResponse.SerializeToString, @@ -121,7 +186,7 @@ def add_AsyncAgentServiceServicer_to_server(servicer, server): # This class is part of an EXPERIMENTAL API. class AsyncAgentService(object): - """AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server. + """AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server asynchronously. """ @staticmethod @@ -203,7 +268,7 @@ def GetTaskLogs(request, wait_for_ready=None, timeout=None, metadata=None): - return grpc.experimental.unary_unary(request, target, '/flyteidl.service.AsyncAgentService/GetTaskLogs', + return grpc.experimental.unary_stream(request, target, '/flyteidl.service.AsyncAgentService/GetTaskLogs', flyteidl_dot_admin_dot_agent__pb2.GetTaskLogsRequest.SerializeToString, flyteidl_dot_admin_dot_agent__pb2.GetTaskLogsResponse.FromString, options, channel_credentials, diff --git a/flyteidl/gen/pb_rust/flyteidl.admin.rs b/flyteidl/gen/pb_rust/flyteidl.admin.rs index 4f61389a42..7205966c77 100644 --- a/flyteidl/gen/pb_rust/flyteidl.admin.rs +++ b/flyteidl/gen/pb_rust/flyteidl.admin.rs @@ -21,6 +21,23 @@ pub struct TaskExecutionMetadata { /// Environment variables attached to the task execution #[prost(map="string, string", tag="6")] pub environment_variables: ::std::collections::HashMap<::prost::alloc::string::String, ::prost::alloc::string::String>, + /// Represents the maximum number of attempts allowed for a task. + /// If a task fails, it can be retried up to this maximum number of attempts. + #[prost(int32, tag="7")] + pub max_attempts: i32, + /// Indicates whether the task execution can be interrupted. + /// If set to true, the task can be stopped before completion. + #[prost(bool, tag="8")] + pub interruptible: bool, + /// Specifies the threshold for failure count at which the interruptible property + /// will take effect. If the number of consecutive task failures exceeds this threshold, + /// interruptible behavior will be activated. + #[prost(int32, tag="9")] + pub interruptible_failure_threshold: i32, + /// Overrides for specific properties of the task node. + /// These overrides can be used to customize the behavior of the task node. + #[prost(message, optional, tag="10")] + pub overrides: ::core::option::Option, } /// Represents a request structure to create task. #[allow(clippy::derive_partial_eq_without_eq)] @@ -45,22 +62,68 @@ pub struct CreateTaskRequest { #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Message)] pub struct CreateTaskResponse { + /// ResourceMeta is created by the agent. It could be a string (jobId) or a dict (more complex metadata). + #[prost(bytes="vec", tag="1")] + pub resource_meta: ::prost::alloc::vec::Vec, +} +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct CreateRequestHeader { + /// Template of the task that encapsulates all the metadata of the task. + #[prost(message, optional, tag="1")] + pub template: ::core::option::Option, + /// Prefix for where task output data will be written. (e.g. s3://my-bucket/randomstring) + #[prost(string, tag="2")] + pub output_prefix: ::prost::alloc::string::String, + /// subset of runtime task execution metadata. + #[prost(message, optional, tag="3")] + pub task_execution_metadata: ::core::option::Option, + /// MaxDatasetSizeBytes is the maximum size of the dataset that can be generated by the task. + #[prost(int64, tag="4")] + pub max_dataset_size_bytes: i64, +} +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct ExecuteTaskSyncRequest { + #[prost(oneof="execute_task_sync_request::Part", tags="1, 2")] + pub part: ::core::option::Option, +} +/// Nested message and enum types in `ExecuteTaskSyncRequest`. +pub mod execute_task_sync_request { + #[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Oneof)] + pub enum Part { + #[prost(message, tag="1")] + Header(super::CreateRequestHeader), + #[prost(message, tag="2")] + Inputs(super::super::core::LiteralMap), + } +} +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct ExecuteTaskSyncResponseHeader { + #[prost(message, optional, tag="1")] + pub resource: ::core::option::Option, +} +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct ExecuteTaskSyncResponse { /// Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). /// Resource is for synchronous task execution. - #[prost(oneof="create_task_response::Res", tags="1, 2")] - pub res: ::core::option::Option, + #[prost(oneof="execute_task_sync_response::Res", tags="1, 2")] + pub res: ::core::option::Option, } -/// Nested message and enum types in `CreateTaskResponse`. -pub mod create_task_response { +/// Nested message and enum types in `ExecuteTaskSyncResponse`. +pub mod execute_task_sync_response { /// Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). /// Resource is for synchronous task execution. #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Oneof)] pub enum Res { - #[prost(bytes, tag="1")] - ResourceMeta(::prost::alloc::vec::Vec), + #[prost(message, tag="1")] + Header(super::ExecuteTaskSyncResponseHeader), #[prost(message, tag="2")] - Resource(super::Resource), + Outputs(super::super::core::LiteralMap), } } /// A message used to fetch a job resource from flyte agent server. @@ -68,11 +131,15 @@ pub mod create_task_response { #[derive(Clone, PartialEq, ::prost::Message)] pub struct GetTaskRequest { /// A predefined yet extensible Task type identifier. + #[deprecated] #[prost(string, tag="1")] pub task_type: ::prost::alloc::string::String, /// Metadata about the resource to be pass to the agent. #[prost(bytes="vec", tag="2")] pub resource_meta: ::prost::alloc::vec::Vec, + /// A predefined yet extensible Task type identifier. + #[prost(message, optional, tag="3")] + pub task_category: ::core::option::Option, } /// Response to get an individual task resource. #[allow(clippy::derive_partial_eq_without_eq)] @@ -80,9 +147,6 @@ pub struct GetTaskRequest { pub struct GetTaskResponse { #[prost(message, optional, tag="1")] pub resource: ::core::option::Option, - /// log information for the task execution - #[prost(message, repeated, tag="2")] - pub log_links: ::prost::alloc::vec::Vec, } #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Message)] @@ -105,17 +169,24 @@ pub struct Resource { /// The phase of the execution is used to determine the phase of the plugin's execution. #[prost(enumeration="super::core::task_execution::Phase", tag="5")] pub phase: i32, + /// Custom data specific to the agent. + #[prost(message, optional, tag="6")] + pub custom_info: ::core::option::Option<::prost_types::Struct>, } /// A message used to delete a task. #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Message)] pub struct DeleteTaskRequest { /// A predefined yet extensible Task type identifier. + #[deprecated] #[prost(string, tag="1")] pub task_type: ::prost::alloc::string::String, /// Metadata about the resource to be pass to the agent. #[prost(bytes="vec", tag="2")] pub resource_meta: ::prost::alloc::vec::Vec, + /// A predefined yet extensible Task type identifier. + #[prost(message, optional, tag="3")] + pub task_category: ::core::option::Option, } /// Response to delete a task. #[allow(clippy::derive_partial_eq_without_eq)] @@ -130,8 +201,29 @@ pub struct Agent { #[prost(string, tag="1")] pub name: ::prost::alloc::string::String, /// SupportedTaskTypes are the types of the tasks that the agent can handle. + #[deprecated] #[prost(string, repeated, tag="2")] pub supported_task_types: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, + /// IsSync indicates whether this agent is a sync agent. Sync agents are expected to return their + /// results synchronously when called by propeller. Given that sync agents can affect the performance + /// of the system, it's important to enforce strict timeout policies. + /// An Async agent, on the other hand, is required to be able to identify jobs by an + /// identifier and query for job statuses as jobs progress. + #[prost(bool, tag="3")] + pub is_sync: bool, + /// Supported_task_categories are the categories of the tasks that the agent can handle. + #[prost(message, repeated, tag="4")] + pub supported_task_categories: ::prost::alloc::vec::Vec, +} +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct TaskCategory { + /// The name of the task type. + #[prost(string, tag="1")] + pub name: ::prost::alloc::string::String, + /// The version of the task type. + #[prost(int32, tag="2")] + pub version: i32, } /// A request to get an agent. #[allow(clippy::derive_partial_eq_without_eq)] @@ -165,6 +257,7 @@ pub struct ListAgentsResponse { #[derive(Clone, PartialEq, ::prost::Message)] pub struct GetTaskMetricsRequest { /// A predefined yet extensible Task type identifier. + #[deprecated] #[prost(string, tag="1")] pub task_type: ::prost::alloc::string::String, /// Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). @@ -183,6 +276,9 @@ pub struct GetTaskMetricsRequest { /// Query resolution step width in duration format or float number of seconds. #[prost(message, optional, tag="6")] pub step: ::core::option::Option<::prost_types::Duration>, + /// A predefined yet extensible Task type identifier. + #[prost(message, optional, tag="7")] + pub task_category: ::core::option::Option, } /// A response containing a list of metrics for a task execution. #[allow(clippy::derive_partial_eq_without_eq)] @@ -197,6 +293,7 @@ pub struct GetTaskMetricsResponse { #[derive(Clone, PartialEq, ::prost::Message)] pub struct GetTaskLogsRequest { /// A predefined yet extensible Task type identifier. + #[deprecated] #[prost(string, tag="1")] pub task_type: ::prost::alloc::string::String, /// Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). @@ -209,19 +306,43 @@ pub struct GetTaskLogsRequest { /// in a query. If there are no more results, this value will be empty. #[prost(string, tag="4")] pub token: ::prost::alloc::string::String, + /// A predefined yet extensible Task type identifier. + #[prost(message, optional, tag="5")] + pub task_category: ::core::option::Option, } -/// A response containing the logs for a task execution. #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Message)] -pub struct GetTaskLogsResponse { - /// The execution log results. - #[prost(string, repeated, tag="1")] - pub results: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, +pub struct GetTaskLogsResponseHeader { /// In the case of multiple pages of results, the server-provided token can be used to fetch the next page /// in a query. If there are no more results, this value will be empty. - #[prost(string, tag="2")] + #[prost(string, tag="1")] pub token: ::prost::alloc::string::String, } +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct GetTaskLogsResponseBody { + /// The execution log results. + #[prost(string, repeated, tag="1")] + pub results: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, +} +/// A response containing the logs for a task execution. +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct GetTaskLogsResponse { + #[prost(oneof="get_task_logs_response::Part", tags="1, 2")] + pub part: ::core::option::Option, +} +/// Nested message and enum types in `GetTaskLogsResponse`. +pub mod get_task_logs_response { + #[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Oneof)] + pub enum Part { + #[prost(message, tag="1")] + Header(super::GetTaskLogsResponseHeader), + #[prost(message, tag="2")] + Body(super::GetTaskLogsResponseBody), + } +} /// The state of the execution is used to control its visibility in the UI/CLI. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] #[repr(i32)] @@ -1188,6 +1309,8 @@ pub mod execution_metadata { ChildWorkflow = 4, /// This execution was recovered from another execution. Recovered = 5, + /// Execution was kicked off by the artifact trigger system + Trigger = 6, } impl ExecutionMode { /// String value of the enum field names used in the ProtoBuf definition. @@ -1202,6 +1325,7 @@ pub mod execution_metadata { ExecutionMode::Relaunch => "RELAUNCH", ExecutionMode::ChildWorkflow => "CHILD_WORKFLOW", ExecutionMode::Recovered => "RECOVERED", + ExecutionMode::Trigger => "TRIGGER", } } /// Creates an enum from field names used in the ProtoBuf definition. @@ -1213,6 +1337,7 @@ pub mod execution_metadata { "RELAUNCH" => Some(Self::Relaunch), "CHILD_WORKFLOW" => Some(Self::ChildWorkflow), "RECOVERED" => Some(Self::Recovered), + "TRIGGER" => Some(Self::Trigger), _ => None, } } diff --git a/flyteidl/gen/pb_rust/flyteidl.core.rs b/flyteidl/gen/pb_rust/flyteidl.core.rs index b2cc8088f5..07973186b4 100644 --- a/flyteidl/gen/pb_rust/flyteidl.core.rs +++ b/flyteidl/gen/pb_rust/flyteidl.core.rs @@ -730,8 +730,6 @@ pub struct ArtifactKey { #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Message)] pub struct ArtifactBindingData { - #[prost(uint32, tag="1")] - pub index: u32, /// This is only relevant in the time partition case #[prost(string, tag="4")] pub transform: ::prost::alloc::string::String, @@ -1703,6 +1701,147 @@ pub mod sql { } } } +/// Defines a 2-level tree where the root is a comparison operator and Operands are primitives or known variables. +/// Each expression results in a boolean result. +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct ComparisonExpression { + #[prost(enumeration="comparison_expression::Operator", tag="1")] + pub operator: i32, + #[prost(message, optional, tag="2")] + pub left_value: ::core::option::Option, + #[prost(message, optional, tag="3")] + pub right_value: ::core::option::Option, +} +/// Nested message and enum types in `ComparisonExpression`. +pub mod comparison_expression { + /// Binary Operator for each expression + #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] + #[repr(i32)] + pub enum Operator { + Eq = 0, + Neq = 1, + /// Greater Than + Gt = 2, + Gte = 3, + /// Less Than + Lt = 4, + Lte = 5, + } + impl Operator { + /// String value of the enum field names used in the ProtoBuf definition. + /// + /// The values are not transformed in any way and thus are considered stable + /// (if the ProtoBuf definition does not change) and safe for programmatic use. + pub fn as_str_name(&self) -> &'static str { + match self { + Operator::Eq => "EQ", + Operator::Neq => "NEQ", + Operator::Gt => "GT", + Operator::Gte => "GTE", + Operator::Lt => "LT", + Operator::Lte => "LTE", + } + } + /// Creates an enum from field names used in the ProtoBuf definition. + pub fn from_str_name(value: &str) -> ::core::option::Option { + match value { + "EQ" => Some(Self::Eq), + "NEQ" => Some(Self::Neq), + "GT" => Some(Self::Gt), + "GTE" => Some(Self::Gte), + "LT" => Some(Self::Lt), + "LTE" => Some(Self::Lte), + _ => None, + } + } + } +} +/// Defines an operand to a comparison expression. +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct Operand { + #[prost(oneof="operand::Val", tags="1, 2, 3")] + pub val: ::core::option::Option, +} +/// Nested message and enum types in `Operand`. +pub mod operand { + #[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Oneof)] + pub enum Val { + /// Can be a constant + #[prost(message, tag="1")] + Primitive(super::Primitive), + /// Or one of this node's input variables + #[prost(string, tag="2")] + Var(::prost::alloc::string::String), + /// Replace the primitive field + #[prost(message, tag="3")] + Scalar(super::Scalar), + } +} +/// Defines a boolean expression tree. It can be a simple or a conjunction expression. +/// Multiple expressions can be combined using a conjunction or a disjunction to result in a final boolean result. +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct BooleanExpression { + #[prost(oneof="boolean_expression::Expr", tags="1, 2")] + pub expr: ::core::option::Option, +} +/// Nested message and enum types in `BooleanExpression`. +pub mod boolean_expression { + #[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Oneof)] + pub enum Expr { + #[prost(message, tag="1")] + Conjunction(::prost::alloc::boxed::Box), + #[prost(message, tag="2")] + Comparison(super::ComparisonExpression), + } +} +/// Defines a conjunction expression of two boolean expressions. +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct ConjunctionExpression { + #[prost(enumeration="conjunction_expression::LogicalOperator", tag="1")] + pub operator: i32, + #[prost(message, optional, boxed, tag="2")] + pub left_expression: ::core::option::Option<::prost::alloc::boxed::Box>, + #[prost(message, optional, boxed, tag="3")] + pub right_expression: ::core::option::Option<::prost::alloc::boxed::Box>, +} +/// Nested message and enum types in `ConjunctionExpression`. +pub mod conjunction_expression { + /// Nested conditions. They can be conjoined using AND / OR + /// Order of evaluation is not important as the operators are Commutative + #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] + #[repr(i32)] + pub enum LogicalOperator { + /// Conjunction + And = 0, + Or = 1, + } + impl LogicalOperator { + /// String value of the enum field names used in the ProtoBuf definition. + /// + /// The values are not transformed in any way and thus are considered stable + /// (if the ProtoBuf definition does not change) and safe for programmatic use. + pub fn as_str_name(&self) -> &'static str { + match self { + LogicalOperator::And => "AND", + LogicalOperator::Or => "OR", + } + } + /// Creates an enum from field names used in the ProtoBuf definition. + pub fn from_str_name(value: &str) -> ::core::option::Option { + match value { + "AND" => Some(Self::And), + "OR" => Some(Self::Or), + _ => None, + } + } + } +} /// Indicates various phases of Workflow Execution #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Message)] @@ -2032,199 +2171,6 @@ pub mod quality_of_service { Spec(super::QualityOfServiceSpec), } } -/// Span represents a duration trace of Flyte execution. The id field denotes a Flyte execution entity or an operation -/// which uniquely identifies the Span. The spans attribute allows this Span to be further broken down into more -/// precise definitions. -#[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct Span { - /// start_time defines the instance this span began. - #[prost(message, optional, tag="1")] - pub start_time: ::core::option::Option<::prost_types::Timestamp>, - /// end_time defines the instance this span completed. - #[prost(message, optional, tag="2")] - pub end_time: ::core::option::Option<::prost_types::Timestamp>, - /// spans defines a collection of Spans that breakdown this execution. - #[prost(message, repeated, tag="7")] - pub spans: ::prost::alloc::vec::Vec, - #[prost(oneof="span::Id", tags="3, 4, 5, 6")] - pub id: ::core::option::Option, -} -/// Nested message and enum types in `Span`. -pub mod span { - #[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Oneof)] - pub enum Id { - /// workflow_id is the id of the workflow execution this Span represents. - #[prost(message, tag="3")] - WorkflowId(super::WorkflowExecutionIdentifier), - /// node_id is the id of the node execution this Span represents. - #[prost(message, tag="4")] - NodeId(super::NodeExecutionIdentifier), - /// task_id is the id of the task execution this Span represents. - #[prost(message, tag="5")] - TaskId(super::TaskExecutionIdentifier), - /// operation_id is the id of a unique operation that this Span represents. - #[prost(string, tag="6")] - OperationId(::prost::alloc::string::String), - } -} -/// ExecutionMetrics is a collection of metrics that are collected during the execution of a Flyte task. -#[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct ExecutionMetricResult { - /// The metric this data represents. e.g. EXECUTION_METRIC_USED_CPU_AVG or EXECUTION_METRIC_USED_MEMORY_BYTES_AVG. - #[prost(string, tag="1")] - pub metric: ::prost::alloc::string::String, - /// The result data in prometheus range query result format - /// - /// This may include multiple time series, differentiated by their metric labels. - /// Start time is greater of (execution attempt start, 48h ago) - /// End time is lesser of (execution attempt end, now) - #[prost(message, optional, tag="2")] - pub data: ::core::option::Option<::prost_types::Struct>, -} -/// Defines a 2-level tree where the root is a comparison operator and Operands are primitives or known variables. -/// Each expression results in a boolean result. -#[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct ComparisonExpression { - #[prost(enumeration="comparison_expression::Operator", tag="1")] - pub operator: i32, - #[prost(message, optional, tag="2")] - pub left_value: ::core::option::Option, - #[prost(message, optional, tag="3")] - pub right_value: ::core::option::Option, -} -/// Nested message and enum types in `ComparisonExpression`. -pub mod comparison_expression { - /// Binary Operator for each expression - #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] - #[repr(i32)] - pub enum Operator { - Eq = 0, - Neq = 1, - /// Greater Than - Gt = 2, - Gte = 3, - /// Less Than - Lt = 4, - Lte = 5, - } - impl Operator { - /// String value of the enum field names used in the ProtoBuf definition. - /// - /// The values are not transformed in any way and thus are considered stable - /// (if the ProtoBuf definition does not change) and safe for programmatic use. - pub fn as_str_name(&self) -> &'static str { - match self { - Operator::Eq => "EQ", - Operator::Neq => "NEQ", - Operator::Gt => "GT", - Operator::Gte => "GTE", - Operator::Lt => "LT", - Operator::Lte => "LTE", - } - } - /// Creates an enum from field names used in the ProtoBuf definition. - pub fn from_str_name(value: &str) -> ::core::option::Option { - match value { - "EQ" => Some(Self::Eq), - "NEQ" => Some(Self::Neq), - "GT" => Some(Self::Gt), - "GTE" => Some(Self::Gte), - "LT" => Some(Self::Lt), - "LTE" => Some(Self::Lte), - _ => None, - } - } - } -} -/// Defines an operand to a comparison expression. -#[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct Operand { - #[prost(oneof="operand::Val", tags="1, 2, 3")] - pub val: ::core::option::Option, -} -/// Nested message and enum types in `Operand`. -pub mod operand { - #[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Oneof)] - pub enum Val { - /// Can be a constant - #[prost(message, tag="1")] - Primitive(super::Primitive), - /// Or one of this node's input variables - #[prost(string, tag="2")] - Var(::prost::alloc::string::String), - /// Replace the primitive field - #[prost(message, tag="3")] - Scalar(super::Scalar), - } -} -/// Defines a boolean expression tree. It can be a simple or a conjunction expression. -/// Multiple expressions can be combined using a conjunction or a disjunction to result in a final boolean result. -#[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct BooleanExpression { - #[prost(oneof="boolean_expression::Expr", tags="1, 2")] - pub expr: ::core::option::Option, -} -/// Nested message and enum types in `BooleanExpression`. -pub mod boolean_expression { - #[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Oneof)] - pub enum Expr { - #[prost(message, tag="1")] - Conjunction(::prost::alloc::boxed::Box), - #[prost(message, tag="2")] - Comparison(super::ComparisonExpression), - } -} -/// Defines a conjunction expression of two boolean expressions. -#[allow(clippy::derive_partial_eq_without_eq)] -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct ConjunctionExpression { - #[prost(enumeration="conjunction_expression::LogicalOperator", tag="1")] - pub operator: i32, - #[prost(message, optional, boxed, tag="2")] - pub left_expression: ::core::option::Option<::prost::alloc::boxed::Box>, - #[prost(message, optional, boxed, tag="3")] - pub right_expression: ::core::option::Option<::prost::alloc::boxed::Box>, -} -/// Nested message and enum types in `ConjunctionExpression`. -pub mod conjunction_expression { - /// Nested conditions. They can be conjoined using AND / OR - /// Order of evaluation is not important as the operators are Commutative - #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] - #[repr(i32)] - pub enum LogicalOperator { - /// Conjunction - And = 0, - Or = 1, - } - impl LogicalOperator { - /// String value of the enum field names used in the ProtoBuf definition. - /// - /// The values are not transformed in any way and thus are considered stable - /// (if the ProtoBuf definition does not change) and safe for programmatic use. - pub fn as_str_name(&self) -> &'static str { - match self { - LogicalOperator::And => "AND", - LogicalOperator::Or => "OR", - } - } - /// Creates an enum from field names used in the ProtoBuf definition. - pub fn from_str_name(value: &str) -> ::core::option::Option { - match value { - "AND" => Some(Self::And), - "OR" => Some(Self::Or), - _ => None, - } - } - } -} /// Defines a condition and the execution unit that should be executed if the condition is satisfied. #[allow(clippy::derive_partial_eq_without_eq)] #[derive(Clone, PartialEq, ::prost::Message)] @@ -2634,6 +2580,9 @@ pub struct TaskNodeOverrides { /// v1.ResourceRequirements, to allocate to a task. #[prost(message, optional, tag="2")] pub extended_resources: ::core::option::Option, + /// Override for the image used by task pods. + #[prost(string, tag="3")] + pub container_image: ::prost::alloc::string::String, } /// A structure that uniquely identifies a launch plan in the system. #[allow(clippy::derive_partial_eq_without_eq)] @@ -2649,6 +2598,58 @@ pub struct LaunchPlanTemplate { #[prost(message, optional, tag="3")] pub fixed_inputs: ::core::option::Option, } +/// Span represents a duration trace of Flyte execution. The id field denotes a Flyte execution entity or an operation +/// which uniquely identifies the Span. The spans attribute allows this Span to be further broken down into more +/// precise definitions. +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct Span { + /// start_time defines the instance this span began. + #[prost(message, optional, tag="1")] + pub start_time: ::core::option::Option<::prost_types::Timestamp>, + /// end_time defines the instance this span completed. + #[prost(message, optional, tag="2")] + pub end_time: ::core::option::Option<::prost_types::Timestamp>, + /// spans defines a collection of Spans that breakdown this execution. + #[prost(message, repeated, tag="7")] + pub spans: ::prost::alloc::vec::Vec, + #[prost(oneof="span::Id", tags="3, 4, 5, 6")] + pub id: ::core::option::Option, +} +/// Nested message and enum types in `Span`. +pub mod span { + #[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Oneof)] + pub enum Id { + /// workflow_id is the id of the workflow execution this Span represents. + #[prost(message, tag="3")] + WorkflowId(super::WorkflowExecutionIdentifier), + /// node_id is the id of the node execution this Span represents. + #[prost(message, tag="4")] + NodeId(super::NodeExecutionIdentifier), + /// task_id is the id of the task execution this Span represents. + #[prost(message, tag="5")] + TaskId(super::TaskExecutionIdentifier), + /// operation_id is the id of a unique operation that this Span represents. + #[prost(string, tag="6")] + OperationId(::prost::alloc::string::String), + } +} +/// ExecutionMetrics is a collection of metrics that are collected during the execution of a Flyte task. +#[allow(clippy::derive_partial_eq_without_eq)] +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct ExecutionMetricResult { + /// The metric this data represents. e.g. EXECUTION_METRIC_USED_CPU_AVG or EXECUTION_METRIC_USED_MEMORY_BYTES_AVG. + #[prost(string, tag="1")] + pub metric: ::prost::alloc::string::String, + /// The result data in prometheus range query result format + /// + /// This may include multiple time series, differentiated by their metric labels. + /// Start time is greater of (execution attempt start, 48h ago) + /// End time is lesser of (execution attempt end, now) + #[prost(message, optional, tag="2")] + pub data: ::core::option::Option<::prost_types::Struct>, +} /// Adjacency list for the workflow. This is created as part of the compilation process. Every process after the compilation /// step uses this created ConnectionSet #[allow(clippy::derive_partial_eq_without_eq)] diff --git a/flyteidl/protos/flyteidl/admin/agent.proto b/flyteidl/protos/flyteidl/admin/agent.proto index 5b7043e86f..0256483d8f 100644 --- a/flyteidl/protos/flyteidl/admin/agent.proto +++ b/flyteidl/protos/flyteidl/admin/agent.proto @@ -5,15 +5,17 @@ option go_package = "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin import "flyteidl/core/literals.proto"; import "flyteidl/core/tasks.proto"; +import "flyteidl/core/workflow.proto"; import "flyteidl/core/identifier.proto"; import "flyteidl/core/execution.proto"; import "flyteidl/core/metrics.proto"; import "google/protobuf/duration.proto"; import "google/protobuf/timestamp.proto"; - +import "google/protobuf/struct.proto"; // The state of the execution is used to control its visibility in the UI/CLI. enum State { + option deprecated = true; RETRYABLE_FAILURE = 0; PERMANENT_FAILURE = 1; PENDING = 2; @@ -35,6 +37,19 @@ message TaskExecutionMetadata { string k8s_service_account = 5; // Environment variables attached to the task execution map environment_variables = 6; + // Represents the maximum number of attempts allowed for a task. + // If a task fails, it can be retried up to this maximum number of attempts. + int32 max_attempts = 7; + // Indicates whether the task execution can be interrupted. + // If set to true, the task can be stopped before completion. + bool interruptible = 8; + // Specifies the threshold for failure count at which the interruptible property + // will take effect. If the number of consecutive task failures exceeds this threshold, + // interruptible behavior will be activated. + int32 interruptible_failure_threshold = 9; + // Overrides for specific properties of the task node. + // These overrides can be used to customize the behavior of the task node. + core.TaskNodeOverrides overrides = 10; } // Represents a request structure to create task. @@ -53,28 +68,55 @@ message CreateTaskRequest { // Represents a create response structure. message CreateTaskResponse { + // ResourceMeta is created by the agent. It could be a string (jobId) or a dict (more complex metadata). + bytes resource_meta = 1; +} + +message CreateRequestHeader { + // Template of the task that encapsulates all the metadata of the task. + core.TaskTemplate template = 1; + // Prefix for where task output data will be written. (e.g. s3://my-bucket/randomstring) + string output_prefix = 2; + // subset of runtime task execution metadata. + TaskExecutionMetadata task_execution_metadata = 3; + // MaxDatasetSizeBytes is the maximum size of the dataset that can be generated by the task. + int64 max_dataset_size_bytes = 4; +} + + +message ExecuteTaskSyncRequest { + oneof part { + CreateRequestHeader header = 1; + core.LiteralMap inputs = 2; + } +} + +message ExecuteTaskSyncResponseHeader { + Resource resource = 1; +} + +message ExecuteTaskSyncResponse { // Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). // Resource is for synchronous task execution. oneof res { - bytes resource_meta = 1; - Resource resource = 2; + ExecuteTaskSyncResponseHeader header = 1; + core.LiteralMap outputs = 2; } } // A message used to fetch a job resource from flyte agent server. message GetTaskRequest { // A predefined yet extensible Task type identifier. - string task_type = 1; + string task_type = 1 [deprecated = true]; // Metadata about the resource to be pass to the agent. bytes resource_meta = 2; + // A predefined yet extensible Task type identifier. + TaskCategory task_category = 3; } // Response to get an individual task resource. message GetTaskResponse { Resource resource = 1; - - // log information for the task execution - repeated core.TaskLog log_links = 2; } message Resource { @@ -90,14 +132,18 @@ message Resource { repeated core.TaskLog log_links = 4; // The phase of the execution is used to determine the phase of the plugin's execution. core.TaskExecution.Phase phase = 5; + // Custom data specific to the agent. + google.protobuf.Struct custom_info = 6; } // A message used to delete a task. message DeleteTaskRequest { // A predefined yet extensible Task type identifier. - string task_type = 1; + string task_type = 1 [deprecated = true]; // Metadata about the resource to be pass to the agent. bytes resource_meta = 2; + // A predefined yet extensible Task type identifier. + TaskCategory task_category = 3; } // Response to delete a task. @@ -109,7 +155,24 @@ message Agent { string name = 1; // SupportedTaskTypes are the types of the tasks that the agent can handle. - repeated string supported_task_types = 2; + repeated string supported_task_types = 2 [deprecated = true]; + + // IsSync indicates whether this agent is a sync agent. Sync agents are expected to return their + // results synchronously when called by propeller. Given that sync agents can affect the performance + // of the system, it's important to enforce strict timeout policies. + // An Async agent, on the other hand, is required to be able to identify jobs by an + // identifier and query for job statuses as jobs progress. + bool is_sync = 3; + + // Supported_task_categories are the categories of the tasks that the agent can handle. + repeated TaskCategory supported_task_categories = 4; +} + +message TaskCategory { + // The name of the task type. + string name = 1; + // The version of the task type. + int32 version = 2; } // A request to get an agent. @@ -134,7 +197,7 @@ message ListAgentsResponse { // A request to get the metrics from a task execution. message GetTaskMetricsRequest { // A predefined yet extensible Task type identifier. - string task_type = 1; + string task_type = 1 [deprecated = true]; // Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). bytes resource_meta = 2; // The metrics to query. If empty, will return a default set of metrics. @@ -146,6 +209,8 @@ message GetTaskMetricsRequest { google.protobuf.Timestamp end_time = 5; // Query resolution step width in duration format or float number of seconds. google.protobuf.Duration step = 6; + // A predefined yet extensible Task type identifier. + TaskCategory task_category = 7; } // A response containing a list of metrics for a task execution. @@ -157,7 +222,7 @@ message GetTaskMetricsResponse { // A request to get the log from a task execution. message GetTaskLogsRequest { // A predefined yet extensible Task type identifier. - string task_type = 1; + string task_type = 1 [deprecated = true]; // Metadata is created by the agent. It could be a string (jobId) or a dict (more complex metadata). bytes resource_meta = 2; // Number of lines to return. @@ -165,13 +230,25 @@ message GetTaskLogsRequest { // In the case of multiple pages of results, the server-provided token can be used to fetch the next page // in a query. If there are no more results, this value will be empty. string token = 4; + // A predefined yet extensible Task type identifier. + TaskCategory task_category = 5; } -// A response containing the logs for a task execution. -message GetTaskLogsResponse { - // The execution log results. - repeated string results = 1; +message GetTaskLogsResponseHeader { // In the case of multiple pages of results, the server-provided token can be used to fetch the next page // in a query. If there are no more results, this value will be empty. - string token = 2; -} \ No newline at end of file + string token = 1; +} + +message GetTaskLogsResponseBody { + // The execution log results. + repeated string results = 1; +} + +// A response containing the logs for a task execution. +message GetTaskLogsResponse { + oneof part { + GetTaskLogsResponseHeader header = 1; + GetTaskLogsResponseBody body = 2; + } +} diff --git a/flyteidl/protos/flyteidl/admin/execution.proto b/flyteidl/protos/flyteidl/admin/execution.proto index f3dbad3fb3..a46d6efdd3 100644 --- a/flyteidl/protos/flyteidl/admin/execution.proto +++ b/flyteidl/protos/flyteidl/admin/execution.proto @@ -221,6 +221,9 @@ message ExecutionMetadata { // This execution was recovered from another execution. RECOVERED = 5; + + // Execution was kicked off by the artifact trigger system + TRIGGER = 6; } ExecutionMode mode = 1; diff --git a/flyteidl/protos/flyteidl/core/artifact_id.proto b/flyteidl/protos/flyteidl/core/artifact_id.proto index 11efd83160..fe887e336a 100644 --- a/flyteidl/protos/flyteidl/core/artifact_id.proto +++ b/flyteidl/protos/flyteidl/core/artifact_id.proto @@ -18,8 +18,6 @@ message ArtifactKey { // Only valid for triggers message ArtifactBindingData { - uint32 index = 1; - // These two fields are only relevant in the partition value case oneof partition_data { string partition_key = 2; diff --git a/flyteidl/protos/flyteidl/core/workflow.proto b/flyteidl/protos/flyteidl/core/workflow.proto index 8433fc9e31..dcbe9367f9 100644 --- a/flyteidl/protos/flyteidl/core/workflow.proto +++ b/flyteidl/protos/flyteidl/core/workflow.proto @@ -297,6 +297,9 @@ message TaskNodeOverrides { // Overrides for all non-standard resources, not captured by // v1.ResourceRequirements, to allocate to a task. ExtendedResources extended_resources = 2; + + // Override for the image used by task pods. + string container_image = 3; } // A structure that uniquely identifies a launch plan in the system. diff --git a/flyteidl/protos/flyteidl/service/agent.proto b/flyteidl/protos/flyteidl/service/agent.proto index 7538c0b819..cd6b93a972 100644 --- a/flyteidl/protos/flyteidl/service/agent.proto +++ b/flyteidl/protos/flyteidl/service/agent.proto @@ -6,24 +6,58 @@ option go_package = "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/servi import "google/api/annotations.proto"; import "flyteidl/admin/agent.proto"; -// AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server. +// SyncAgentService defines an RPC Service that allows propeller to send the request to the agent server synchronously. +service SyncAgentService { + // ExecuteTaskSync streams the create request and inputs to the agent service and streams the outputs back. + rpc ExecuteTaskSync (stream flyteidl.admin.ExecuteTaskSyncRequest) returns (stream flyteidl.admin.ExecuteTaskSyncResponse){ + option (google.api.http) = { + post: "/api/v1/agent/task/stream" + body: "*" + }; + }; +} + +// AsyncAgentService defines an RPC Service that allows propeller to send the request to the agent server asynchronously. service AsyncAgentService { - // Send a task create request to the agent server. - rpc CreateTask (flyteidl.admin.CreateTaskRequest) returns (flyteidl.admin.CreateTaskResponse){}; + // CreateTask sends a task create request to the agent service. + rpc CreateTask (flyteidl.admin.CreateTaskRequest) returns (flyteidl.admin.CreateTaskResponse){ + option (google.api.http) = { + post: "/api/v1/agent/task" + body: "*" + }; + }; + // Get job status. - rpc GetTask (flyteidl.admin.GetTaskRequest) returns (flyteidl.admin.GetTaskResponse){}; + rpc GetTask (flyteidl.admin.GetTaskRequest) returns (flyteidl.admin.GetTaskResponse){ + option (google.api.http) = { + get: "/api/v1/agent/task/{task_category.name}/{task_category.version}/{resource_meta}" + }; + }; + // Delete the task resource. - rpc DeleteTask (flyteidl.admin.DeleteTaskRequest) returns (flyteidl.admin.DeleteTaskResponse){}; + rpc DeleteTask (flyteidl.admin.DeleteTaskRequest) returns (flyteidl.admin.DeleteTaskResponse){ + option (google.api.http) = { + delete: "/api/v1/agent/task_executions/{task_category.name}/{task_category.version}/{resource_meta}" + }; + }; // GetTaskMetrics returns one or more task execution metrics, if available. // // Errors include // * OutOfRange if metrics are not available for the specified task time range // * various other errors - rpc GetTaskMetrics(flyteidl.admin.GetTaskMetricsRequest) returns (flyteidl.admin.GetTaskMetricsResponse){}; + rpc GetTaskMetrics(flyteidl.admin.GetTaskMetricsRequest) returns (flyteidl.admin.GetTaskMetricsResponse){ + option (google.api.http) = { + get: "/api/v1/agent/task/metrics/{task_category.name}/{task_category.version}/{resource_meta}" + }; + }; // GetTaskLogs returns task execution logs, if available. - rpc GetTaskLogs(flyteidl.admin.GetTaskLogsRequest) returns (flyteidl.admin.GetTaskLogsResponse){}; + rpc GetTaskLogs(flyteidl.admin.GetTaskLogsRequest) returns (stream flyteidl.admin.GetTaskLogsResponse){ + option (google.api.http) = { + get: "/api/v1/agent/task/logs/{task_category.name}/{task_category.version}/{resource_meta}" + }; + }; } // AgentMetadataService defines an RPC service that is also served over HTTP via grpc-gateway. @@ -32,7 +66,7 @@ service AgentMetadataService { // Fetch a :ref:`ref_flyteidl.admin.Agent` definition. rpc GetAgent (flyteidl.admin.GetAgentRequest) returns (flyteidl.admin.GetAgentResponse){ option (google.api.http) = { - get: "/api/v1/agent/{name}" + get: "/api/v1/agent/{name}" }; }; diff --git a/flyteplugins/go/tasks/pluginmachinery/catalog/client.go b/flyteplugins/go/tasks/pluginmachinery/catalog/client.go index 5d0e981e08..56cb4be202 100644 --- a/flyteplugins/go/tasks/pluginmachinery/catalog/client.go +++ b/flyteplugins/go/tasks/pluginmachinery/catalog/client.go @@ -49,6 +49,13 @@ func (s Status) GetMetadata() *core.CatalogMetadata { return s.metadata } +func NewPutFailureStatus(key *Key) Status { + md := &core.CatalogMetadata{ + DatasetId: &key.Identifier, + } + return Status{cacheStatus: core.CatalogCacheStatus_CACHE_PUT_FAILURE, metadata: md} +} + func NewStatus(cacheStatus core.CatalogCacheStatus, md *core.CatalogMetadata) Status { return Status{cacheStatus: cacheStatus, metadata: md} } diff --git a/flyteplugins/go/tasks/pluginmachinery/catalog/client_test.go b/flyteplugins/go/tasks/pluginmachinery/catalog/client_test.go index 196b5374de..15a4347351 100644 --- a/flyteplugins/go/tasks/pluginmachinery/catalog/client_test.go +++ b/flyteplugins/go/tasks/pluginmachinery/catalog/client_test.go @@ -23,8 +23,28 @@ var ( Name: "artifactName", }, } + key = &Key{ + Identifier: core.Identifier{ + Project: "project", + Domain: "domain", + Name: "name", + Version: "1.0.0", + }, + CacheVersion: "1.0.0", + TypedInterface: core.TypedInterface{ + Inputs: nil, + Outputs: nil, + }, + } ) +func TestNewPutFailureStatus(t *testing.T) { + status := NewPutFailureStatus(key) + + assert.Equal(t, status.GetCacheStatus(), core.CatalogCacheStatus_CACHE_PUT_FAILURE) + assert.EqualValues(t, status.GetMetadata().GetDatasetId(), &key.Identifier) +} + func TestStatus(t *testing.T) { status := NewStatus(cacheStatus, &catalogMetadata) diff --git a/flyteplugins/go/tasks/pluginmachinery/core/exec_metadata.go b/flyteplugins/go/tasks/pluginmachinery/core/exec_metadata.go index 9ac650baaa..83d7dbcf12 100644 --- a/flyteplugins/go/tasks/pluginmachinery/core/exec_metadata.go +++ b/flyteplugins/go/tasks/pluginmachinery/core/exec_metadata.go @@ -12,6 +12,7 @@ import ( type TaskOverrides interface { GetResources() *v1.ResourceRequirements GetExtendedResources() *core.ExtendedResources + GetContainerImage() string GetConfig() *v1.ConfigMap } diff --git a/flyteplugins/go/tasks/pluginmachinery/core/mocks/task_overrides.go b/flyteplugins/go/tasks/pluginmachinery/core/mocks/task_overrides.go index d00973c114..cc3c528007 100644 --- a/flyteplugins/go/tasks/pluginmachinery/core/mocks/task_overrides.go +++ b/flyteplugins/go/tasks/pluginmachinery/core/mocks/task_overrides.go @@ -82,6 +82,41 @@ func (_m *TaskOverrides) GetExtendedResources() *flyteidlcore.ExtendedResources return r0 } +type TaskOverrides_GetContainerImage struct { + *mock.Call +} + +func (_m TaskOverrides_GetContainerImage) Return(_a0 string) *TaskOverrides_GetContainerImage { + return &TaskOverrides_GetContainerImage{Call: _m.Call.Return(_a0)} +} + +func (_m *TaskOverrides) OnGetContainerImage() *TaskOverrides_GetContainerImage { + c_call := _m.On("GetContainerImage") + return &TaskOverrides_GetContainerImage{Call: c_call} +} + +func (_m *TaskOverrides) OnGetContainerImageMatch(matchers ...interface{}) *TaskOverrides_GetContainerImage { + c_call := _m.On("GetContainerImage", matchers...) + return &TaskOverrides_GetContainerImage{Call: c_call} +} + +// GetContainerImage provides a mock function with given fields: +func (_m *TaskOverrides) GetContainerImage() string { + ret := _m.Called() + + var r0 string + if rf, ok := ret.Get(0).(func() string); ok { + r0 = rf() + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(string) + } + } + + return r0 +} + + type TaskOverrides_GetResources struct { *mock.Call } diff --git a/flyteplugins/go/tasks/pluginmachinery/core/template/template.go b/flyteplugins/go/tasks/pluginmachinery/core/template/template.go index 385abf09f4..82b306ffbe 100644 --- a/flyteplugins/go/tasks/pluginmachinery/core/template/template.go +++ b/flyteplugins/go/tasks/pluginmachinery/core/template/template.go @@ -47,6 +47,7 @@ var perRetryUniqueKey = regexp.MustCompile(`(?i){{\s*[\.$]PerRetryUniqueKey\s*}} var taskTemplateRegex = regexp.MustCompile(`(?i){{\s*[\.$]TaskTemplatePath\s*}}`) var prevCheckpointPrefixRegex = regexp.MustCompile(`(?i){{\s*[\.$]PrevCheckpointPrefix\s*}}`) var currCheckpointPrefixRegex = regexp.MustCompile(`(?i){{\s*[\.$]CheckpointOutputPrefix\s*}}`) +var namespaceRegex = regexp.MustCompile(`(?i){{\s*[\.$]Namespace\s*}}`) type ErrorCollection struct { Errors []error @@ -127,6 +128,9 @@ func render(ctx context.Context, inputTemplate string, params Parameters, perRet val = taskTemplateRegex.ReplaceAllString(val, p.String()) } + // Replace namespace last, in case it was embedded in other templates + val = namespaceRegex.ReplaceAllString(val, params.TaskExecMetadata.GetNamespace()) + inputs, err := params.Inputs.Get(ctx) if err != nil { return val, errors.Wrapf(err, "unable to read inputs") diff --git a/flyteplugins/go/tasks/pluginmachinery/core/template/template_test.go b/flyteplugins/go/tasks/pluginmachinery/core/template/template_test.go index ba2f5f6ae9..956ec33cfd 100644 --- a/flyteplugins/go/tasks/pluginmachinery/core/template/template_test.go +++ b/flyteplugins/go/tasks/pluginmachinery/core/template/template_test.go @@ -77,6 +77,7 @@ func TestReplaceTemplateCommandArgs(t *testing.T) { taskExecutionID.On("GetGeneratedName").Return("per_retry_unique_key") taskMetadata := &pluginsCoreMocks.TaskExecutionMetadata{} taskMetadata.On("GetTaskExecutionID").Return(taskExecutionID) + taskMetadata.On("GetNamespace").Return("test-namespace") t.Run("empty cmd", func(t *testing.T) { actual, err := Render(context.TODO(), []string{}, Parameters{}) @@ -572,6 +573,38 @@ func TestReplaceTemplateCommandArgs(t *testing.T) { "--checkpoint=s3://new-checkpoint/prefix", }, actual) }) + + t.Run("namespace embedded replacement", func(t *testing.T) { + params := Parameters{ + TaskExecMetadata: taskMetadata, + Inputs: in, + OutputPath: dummyOutputPaths{ + outputPath: out.outputPath, + rawOutputDataPrefix: "s3://raw-data/prefix/{{ .Namespace }}", + prevCheckpointPath: "s3://prev-checkpoint/prefix/{{ .Namespace}}", + checkpointPath: "s3://new-checkpoint/prefix/{{.namespace}}", + }, + } + actual, err := Render(context.TODO(), []string{ + "hello", + "world", + "{{ .Input }}", + "{{ .OutputPrefix }}", + "--prev={{ .prevcheckpointprefix }}", + "--checkpoint={{ .checkpointoutputprefix }}", + "--raw-data-output={{ .rawoutputdataprefix }}", + }, params) + assert.NoError(t, err) + assert.Equal(t, []string{ + "hello", + "world", + "input/blah", + "output/blah", + "--prev=s3://prev-checkpoint/prefix/test-namespace", + "--checkpoint=s3://new-checkpoint/prefix/test-namespace", + "--raw-data-output=s3://raw-data/prefix/test-namespace", + }, actual) + }) } func TestReplaceTemplateCommandArgsSpecialChars(t *testing.T) { @@ -588,6 +621,7 @@ func TestReplaceTemplateCommandArgsSpecialChars(t *testing.T) { taskExecutionID.On("GetGeneratedName").Return("per-retry-unique-key") taskMetadata := &pluginsCoreMocks.TaskExecutionMetadata{} taskMetadata.On("GetTaskExecutionID").Return(taskExecutionID) + taskMetadata.On("GetNamespace").Return("my-namespace") params.TaskExecMetadata = taskMetadata actual, err := Render(context.TODO(), []string{ @@ -611,6 +645,7 @@ func TestReplaceTemplateCommandArgsSpecialChars(t *testing.T) { taskExecutionID.On("GetGeneratedName").Return("33 per retry-unique-key") taskMetadata := &pluginsCoreMocks.TaskExecutionMetadata{} taskMetadata.On("GetTaskExecutionID").Return(taskExecutionID) + taskMetadata.On("GetNamespace").Return("my-namespace") params.TaskExecMetadata = taskMetadata testString := "doesn't start with a number" diff --git a/flyteplugins/go/tasks/pluginmachinery/flytek8s/container_helper_test.go b/flyteplugins/go/tasks/pluginmachinery/flytek8s/container_helper_test.go index f07db9417d..d48b99ab39 100644 --- a/flyteplugins/go/tasks/pluginmachinery/flytek8s/container_helper_test.go +++ b/flyteplugins/go/tasks/pluginmachinery/flytek8s/container_helper_test.go @@ -385,6 +385,7 @@ func TestToK8sContainer(t *testing.T) { mockTaskExecMetadata.OnGetEnvironmentVariables().Return(map[string]string{ "foo": "bar", }) + mockTaskExecMetadata.OnGetNamespace().Return("my-namespace") tCtx := &mocks.TaskExecutionContext{} tCtx.OnTaskExecutionMetadata().Return(&mockTaskExecMetadata) @@ -457,6 +458,7 @@ func getTemplateParametersForTest(resourceRequirements, platformResources *v1.Re mockTaskExecMetadata.OnGetOverrides().Return(&mockOverrides) mockTaskExecMetadata.OnGetPlatformResources().Return(platformResources) mockTaskExecMetadata.OnGetEnvironmentVariables().Return(nil) + mockTaskExecMetadata.OnGetNamespace().Return("my-namespace") mockInputReader := mocks2.InputReader{} mockInputPath := storage.DataReference("s3://input/path") diff --git a/flyteplugins/go/tasks/pluginmachinery/flytek8s/plugin_exec_context.go b/flyteplugins/go/tasks/pluginmachinery/flytek8s/plugin_exec_context.go index d09bcdb363..ff9ae2ba9f 100644 --- a/flyteplugins/go/tasks/pluginmachinery/flytek8s/plugin_exec_context.go +++ b/flyteplugins/go/tasks/pluginmachinery/flytek8s/plugin_exec_context.go @@ -27,6 +27,10 @@ func (to *pluginTaskOverrides) GetExtendedResources() *core.ExtendedResources { return to.TaskOverrides.GetExtendedResources() } +func (to *pluginTaskOverrides) GetContainerImage() string { + return to.TaskOverrides.GetContainerImage() +} + type pluginTaskExecutionMetadata struct { pluginsCore.TaskExecutionMetadata interruptible *bool diff --git a/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper.go b/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper.go index 69111dc030..7f010f6b7c 100644 --- a/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper.go +++ b/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper.go @@ -317,7 +317,7 @@ func BuildRawPod(ctx context.Context, tCtx pluginsCore.TaskExecutionContext) (*v } // ApplyFlytePodConfiguration updates the PodSpec and ObjectMeta with various Flyte configuration. This includes -// applying default k8s configuration, resource requests, injecting copilot containers, and merging with the +// applying default k8s configuration, applying overrides (resources etc.), injecting copilot containers, and merging with the // configuration PodTemplate (if exists). func ApplyFlytePodConfiguration(ctx context.Context, tCtx pluginsCore.TaskExecutionContext, podSpec *v1.PodSpec, objectMeta *metav1.ObjectMeta, primaryContainerName string) (*v1.PodSpec, *metav1.ObjectMeta, error) { taskTemplate, err := tCtx.TaskReader().Read(ctx) @@ -400,9 +400,23 @@ func ApplyFlytePodConfiguration(ctx context.Context, tCtx pluginsCore.TaskExecut ApplyGPUNodeSelectors(podSpec, extendedResources.GetGpuAccelerator()) } + // Override container image if necessary + if len(tCtx.TaskExecutionMetadata().GetOverrides().GetContainerImage()) > 0 { + ApplyContainerImageOverride(podSpec, tCtx.TaskExecutionMetadata().GetOverrides().GetContainerImage(), primaryContainerName) + } + return podSpec, objectMeta, nil } +func ApplyContainerImageOverride(podSpec *v1.PodSpec, containerImage string, primaryContainerName string) { + for i, c := range podSpec.Containers { + if c.Name == primaryContainerName { + podSpec.Containers[i].Image = containerImage + return + } + } +} + // ToK8sPodSpec builds a PodSpec and ObjectMeta based on the definition passed by the TaskExecutionContext. This // involves parsing the raw PodSpec definition and applying all Flyte configuration options. func ToK8sPodSpec(ctx context.Context, tCtx pluginsCore.TaskExecutionContext) (*v1.PodSpec, *metav1.ObjectMeta, string, error) { diff --git a/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper_test.go b/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper_test.go index b12813b387..08b6c5b9d0 100644 --- a/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper_test.go +++ b/flyteplugins/go/tasks/pluginmachinery/flytek8s/pod_helper_test.go @@ -26,7 +26,7 @@ import ( "github.com/flyteorg/flyte/flytestdlib/storage" ) -func dummyTaskExecutionMetadata(resources *v1.ResourceRequirements, extendedResources *core.ExtendedResources) pluginsCore.TaskExecutionMetadata { +func dummyTaskExecutionMetadata(resources *v1.ResourceRequirements, extendedResources *core.ExtendedResources, containerImage string) pluginsCore.TaskExecutionMetadata { taskExecutionMetadata := &pluginsCoreMock.TaskExecutionMetadata{} taskExecutionMetadata.On("GetNamespace").Return("test-namespace") taskExecutionMetadata.On("GetAnnotations").Return(map[string]string{"annotation-1": "val1"}) @@ -52,6 +52,7 @@ func dummyTaskExecutionMetadata(resources *v1.ResourceRequirements, extendedReso to := &pluginsCoreMock.TaskOverrides{} to.On("GetResources").Return(resources) to.On("GetExtendedResources").Return(extendedResources) + to.On("GetContainerImage").Return(containerImage) taskExecutionMetadata.On("GetOverrides").Return(to) taskExecutionMetadata.On("IsInterruptible").Return(true) taskExecutionMetadata.OnGetPlatformResources().Return(&v1.ResourceRequirements{}) @@ -79,7 +80,7 @@ func dummyInputReader() io.InputReader { return inputReader } -func dummyExecContext(taskTemplate *core.TaskTemplate, r *v1.ResourceRequirements, rm *core.ExtendedResources) pluginsCore.TaskExecutionContext { +func dummyExecContext(taskTemplate *core.TaskTemplate, r *v1.ResourceRequirements, rm *core.ExtendedResources, containerImage string) pluginsCore.TaskExecutionContext { ow := &pluginsIOMock.OutputWriter{} ow.OnGetOutputPrefixPath().Return("") ow.OnGetRawOutputPrefix().Return("") @@ -87,7 +88,7 @@ func dummyExecContext(taskTemplate *core.TaskTemplate, r *v1.ResourceRequirement ow.OnGetPreviousCheckpointsPrefix().Return("/prev") tCtx := &pluginsCoreMock.TaskExecutionContext{} - tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(r, rm)) + tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(r, rm, containerImage)) tCtx.OnInputReader().Return(dummyInputReader()) tCtx.OnOutputWriter().Return(ow) @@ -700,7 +701,7 @@ func updatePod(t *testing.T) { v1.ResourceCPU: resource.MustParse("1024m"), v1.ResourceEphemeralStorage: resource.MustParse("100M"), }, - }, nil) + }, nil, "") pod := &v1.Pod{ Spec: v1.PodSpec{ @@ -753,7 +754,7 @@ func updatePod(t *testing.T) { } func TestUpdatePodWithDefaultAffinityAndInterruptibleNodeSelectorRequirement(t *testing.T) { - taskExecutionMetadata := dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil) + taskExecutionMetadata := dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil, "") assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{ DefaultAffinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ @@ -817,7 +818,7 @@ func toK8sPodInterruptible(t *testing.T) { v1.ResourceCPU: resource.MustParse("1024m"), v1.ResourceEphemeralStorage: resource.MustParse("100M"), }, - }, nil) + }, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) @@ -884,7 +885,7 @@ func TestToK8sPod(t *testing.T) { v1.ResourceCPU: resource.MustParse("1024m"), v1.ResourceEphemeralStorage: resource.MustParse("100M"), }, - }, nil) + }, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) @@ -901,7 +902,7 @@ func TestToK8sPod(t *testing.T) { v1.ResourceCPU: resource.MustParse("1024m"), v1.ResourceEphemeralStorage: resource.MustParse("100M"), }, - }, nil) + }, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) @@ -919,7 +920,7 @@ func TestToK8sPod(t *testing.T) { v1.ResourceCPU: resource.MustParse("1024m"), v1.ResourceEphemeralStorage: resource.MustParse("100M"), }, - }, nil) + }, nil, "") assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{ DefaultNodeSelector: map[string]string{ @@ -946,7 +947,7 @@ func TestToK8sPod(t *testing.T) { }, })) - x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil) + x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) assert.NotNil(t, p.SecurityContext) @@ -958,7 +959,7 @@ func TestToK8sPod(t *testing.T) { assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{ EnableHostNetworkingPod: &enabled, })) - x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil) + x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) assert.True(t, p.HostNetwork) @@ -969,7 +970,7 @@ func TestToK8sPod(t *testing.T) { assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{ EnableHostNetworkingPod: &enabled, })) - x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil) + x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) assert.False(t, p.HostNetwork) @@ -977,7 +978,7 @@ func TestToK8sPod(t *testing.T) { t.Run("skipSettingHostNetwork", func(t *testing.T) { assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{})) - x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil) + x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) assert.False(t, p.HostNetwork) @@ -1011,7 +1012,7 @@ func TestToK8sPod(t *testing.T) { }, })) - x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil) + x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) assert.NotNil(t, p.DNSConfig) @@ -1032,7 +1033,7 @@ func TestToK8sPod(t *testing.T) { "foo": "bar", }, })) - x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil) + x := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{}, nil, "") p, _, _, err := ToK8sPodSpec(ctx, x) assert.NoError(t, err) for _, c := range p.Containers { @@ -1047,6 +1048,18 @@ func TestToK8sPod(t *testing.T) { }) } +func TestToK8sPodContainerImage(t *testing.T) { + t.Run("Override container image", func(t *testing.T) { + taskContext := dummyExecContext(dummyTaskTemplate(), &v1.ResourceRequirements{ + Requests: v1.ResourceList{ + v1.ResourceCPU: resource.MustParse("1024m"), + }}, nil, "foo:latest") + p, _, _, err := ToK8sPodSpec(context.TODO(), taskContext) + assert.NoError(t, err) + assert.Equal(t, "foo:latest", p.Containers[0].Image) + }) +} + func TestToK8sPodExtendedResources(t *testing.T) { assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{ GpuDeviceNodeLabel: "gpu-node-label", @@ -1152,7 +1165,7 @@ func TestToK8sPodExtendedResources(t *testing.T) { t.Run(f.name, func(t *testing.T) { taskTemplate := dummyTaskTemplate() taskTemplate.ExtendedResources = f.extendedResourcesBase - taskContext := dummyExecContext(taskTemplate, f.resources, f.extendedResourcesOverride) + taskContext := dummyExecContext(taskTemplate, f.resources, f.extendedResourcesOverride, "") p, _, _, err := ToK8sPodSpec(context.TODO(), taskContext) assert.NoError(t, err) @@ -1790,7 +1803,7 @@ func TestGetPodTemplate(t *testing.T) { taskReader.On("Read", mock.Anything).Return(task, nil) tCtx := &pluginsCoreMock.TaskExecutionContext{} - tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil)) + tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil, "")) tCtx.OnTaskReader().Return(taskReader) // initialize PodTemplateStore @@ -1816,7 +1829,7 @@ func TestGetPodTemplate(t *testing.T) { taskReader.On("Read", mock.Anything).Return(task, nil) tCtx := &pluginsCoreMock.TaskExecutionContext{} - tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil)) + tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil, "")) tCtx.OnTaskReader().Return(taskReader) // initialize PodTemplateStore @@ -1843,7 +1856,7 @@ func TestGetPodTemplate(t *testing.T) { taskReader.On("Read", mock.Anything).Return(task, nil) tCtx := &pluginsCoreMock.TaskExecutionContext{} - tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil)) + tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil, "")) tCtx.OnTaskReader().Return(taskReader) // initialize PodTemplateStore @@ -1871,7 +1884,7 @@ func TestGetPodTemplate(t *testing.T) { taskReader.On("Read", mock.Anything).Return(task, nil) tCtx := &pluginsCoreMock.TaskExecutionContext{} - tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil)) + tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil, "")) tCtx.OnTaskReader().Return(taskReader) // initialize PodTemplateStore @@ -1913,7 +1926,7 @@ func TestMergeWithBasePodTemplate(t *testing.T) { taskReader.On("Read", mock.Anything).Return(task, nil) tCtx := &pluginsCoreMock.TaskExecutionContext{} - tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil)) + tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil, "")) tCtx.OnTaskReader().Return(taskReader) resultPodSpec, resultObjectMeta, err := MergeWithBasePodTemplate(context.TODO(), tCtx, &podSpec, &objectMeta, "foo") @@ -1967,7 +1980,7 @@ func TestMergeWithBasePodTemplate(t *testing.T) { taskReader.On("Read", mock.Anything).Return(task, nil) tCtx := &pluginsCoreMock.TaskExecutionContext{} - tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil)) + tCtx.OnTaskExecutionMetadata().Return(dummyTaskExecutionMetadata(&v1.ResourceRequirements{}, nil, "")) tCtx.OnTaskReader().Return(taskReader) resultPodSpec, resultObjectMeta, err := MergeWithBasePodTemplate(context.TODO(), tCtx, &podSpec, &objectMeta, "foo") diff --git a/flyteplugins/go/tasks/plugins/array/awsbatch/launcher_test.go b/flyteplugins/go/tasks/plugins/array/awsbatch/launcher_test.go index a1493f741b..7a2d88cabb 100644 --- a/flyteplugins/go/tasks/plugins/array/awsbatch/launcher_test.go +++ b/flyteplugins/go/tasks/plugins/array/awsbatch/launcher_test.go @@ -68,6 +68,7 @@ func TestLaunchSubTasks(t *testing.T) { tMeta.OnGetTaskExecutionID().Return(tID) tMeta.OnGetOverrides().Return(overrides) tMeta.OnGetPlatformResources().Return(&v1.ResourceRequirements{}) + tMeta.OnGetNamespace().Return("my-namespace") ow := &mocks3.OutputWriter{} ow.OnGetOutputPrefixPath().Return("/prefix/") diff --git a/flyteplugins/go/tasks/plugins/array/k8s/management_test.go b/flyteplugins/go/tasks/plugins/array/k8s/management_test.go index 2bd1d5eefe..0eaed65467 100644 --- a/flyteplugins/go/tasks/plugins/array/k8s/management_test.go +++ b/flyteplugins/go/tasks/plugins/array/k8s/management_test.go @@ -94,6 +94,7 @@ func getMockTaskExecutionContext(ctx context.Context, parallelism int) *mocks.Ta }, }) overrides.OnGetExtendedResources().Return(nil) + overrides.OnGetContainerImage().Return("") tMeta := &mocks.TaskExecutionMetadata{} tMeta.OnGetTaskExecutionID().Return(tID) diff --git a/flyteplugins/go/tasks/plugins/k8s/dask/dask_test.go b/flyteplugins/go/tasks/plugins/k8s/dask/dask_test.go index e7d43a2256..86ca034e7b 100644 --- a/flyteplugins/go/tasks/plugins/k8s/dask/dask_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/dask/dask_test.go @@ -194,6 +194,7 @@ func dummyDaskTaskContext(taskTemplate *core.TaskTemplate, resources *v1.Resourc overrides := &mocks.TaskOverrides{} overrides.OnGetResources().Return(resources) overrides.OnGetExtendedResources().Return(extendedResources) + overrides.OnGetContainerImage().Return("") taskExecutionMetadata.OnGetOverrides().Return(overrides) taskCtx.On("TaskExecutionMetadata").Return(taskExecutionMetadata) return taskCtx diff --git a/flyteplugins/go/tasks/plugins/k8s/kfoperators/mpi/mpi_test.go b/flyteplugins/go/tasks/plugins/k8s/kfoperators/mpi/mpi_test.go index 29fe47a446..3c823510ac 100644 --- a/flyteplugins/go/tasks/plugins/k8s/kfoperators/mpi/mpi_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/kfoperators/mpi/mpi_test.go @@ -153,6 +153,7 @@ func dummyMPITaskContext(taskTemplate *core.TaskTemplate, resources *corev1.Reso overrides := &mocks.TaskOverrides{} overrides.OnGetResources().Return(resources) overrides.OnGetExtendedResources().Return(extendedResources) + overrides.OnGetContainerImage().Return("") taskExecutionMetadata := &mocks.TaskExecutionMetadata{} taskExecutionMetadata.OnGetTaskExecutionID().Return(tID) diff --git a/flyteplugins/go/tasks/plugins/k8s/kfoperators/pytorch/pytorch_test.go b/flyteplugins/go/tasks/plugins/k8s/kfoperators/pytorch/pytorch_test.go index 0700644578..aea096e4ba 100644 --- a/flyteplugins/go/tasks/plugins/k8s/kfoperators/pytorch/pytorch_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/kfoperators/pytorch/pytorch_test.go @@ -123,7 +123,7 @@ func dummyPytorchTaskTemplate(id string, args ...interface{}) *core.TaskTemplate } } -func dummyPytorchTaskContext(taskTemplate *core.TaskTemplate, resources *corev1.ResourceRequirements, extendedResources *core.ExtendedResources) pluginsCore.TaskExecutionContext { +func dummyPytorchTaskContext(taskTemplate *core.TaskTemplate, resources *corev1.ResourceRequirements, extendedResources *core.ExtendedResources, containerImage string) pluginsCore.TaskExecutionContext { taskCtx := &mocks.TaskExecutionContext{} inputReader := &pluginIOMocks.InputReader{} inputReader.OnGetInputPrefixPath().Return("/input/prefix") @@ -159,6 +159,7 @@ func dummyPytorchTaskContext(taskTemplate *core.TaskTemplate, resources *corev1. overrides := &mocks.TaskOverrides{} overrides.OnGetResources().Return(resources) overrides.OnGetExtendedResources().Return(extendedResources) + overrides.OnGetContainerImage().Return(containerImage) taskExecutionMetadata := &mocks.TaskExecutionMetadata{} taskExecutionMetadata.OnGetTaskExecutionID().Return(tID) @@ -279,7 +280,7 @@ func dummyPytorchJobResource(pytorchResourceHandler pytorchOperatorResourceHandl ptObj := dummyPytorchCustomObj(workers) taskTemplate := dummyPytorchTaskTemplate("job1", ptObj) - resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) if err != nil { panic(err) } @@ -307,7 +308,7 @@ func TestBuildResourcePytorchElastic(t *testing.T) { ptObj := dummyElasticPytorchCustomObj(2, plugins.ElasticConfig{MinReplicas: 1, MaxReplicas: 2, NprocPerNode: 4, RdzvBackend: "c10d"}) taskTemplate := dummyPytorchTaskTemplate("job2", ptObj) - resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, resource) @@ -350,7 +351,7 @@ func TestBuildResourcePytorch(t *testing.T) { ptObj := dummyPytorchCustomObj(100) taskTemplate := dummyPytorchTaskTemplate("job3", ptObj) - res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, res) @@ -386,6 +387,74 @@ func TestBuildResourcePytorch(t *testing.T) { } } +func TestBuildResourcePytorchContainerImage(t *testing.T) { + assert.NoError(t, flytek8sConfig.SetK8sPluginConfig(&flytek8sConfig.K8sPluginConfig{})) + + fixtures := []struct { + name string + resources *corev1.ResourceRequirements + containerImageOverride string + }{ + { + "without overrides", + &corev1.ResourceRequirements{ + Limits: corev1.ResourceList{ + flytek8s.ResourceNvidiaGPU: resource.MustParse("1"), + }, + }, + "", + }, + { + "with overrides", + &corev1.ResourceRequirements{ + Limits: corev1.ResourceList{ + flytek8s.ResourceNvidiaGPU: resource.MustParse("1"), + }, + }, + "container-image-override", + }, + } + + testConfigs := []struct { + name string + plugin *plugins.DistributedPyTorchTrainingTask + }{ + { + "pytorch", + dummyPytorchCustomObj(100), + }, + { + "elastic pytorch", + dummyElasticPytorchCustomObj(2, plugins.ElasticConfig{MinReplicas: 1, MaxReplicas: 2, NprocPerNode: 4, RdzvBackend: "c10d"}), + }, + } + + for _, tCfg := range testConfigs { + for _, f := range fixtures { + t.Run(tCfg.name+" "+f.name, func(t *testing.T) { + taskTemplate := dummyPytorchTaskTemplate("job", tCfg.plugin) + taskContext := dummyPytorchTaskContext(taskTemplate, f.resources, nil, f.containerImageOverride) + pytorchResourceHandler := pytorchOperatorResourceHandler{} + r, err := pytorchResourceHandler.BuildResource(context.TODO(), taskContext) + assert.NoError(t, err) + assert.NotNil(t, r) + pytorchJob, ok := r.(*kubeflowv1.PyTorchJob) + assert.True(t, ok) + + for _, replicaSpec := range pytorchJob.Spec.PyTorchReplicaSpecs { + var expectedContainerImage string + if len(f.containerImageOverride) > 0 { + expectedContainerImage = f.containerImageOverride + } else { + expectedContainerImage = testImage + } + assert.Equal(t, expectedContainerImage, replicaSpec.Template.Spec.Containers[0].Image) + } + }) + } + } +} + func TestBuildResourcePytorchExtendedResources(t *testing.T) { assert.NoError(t, flytek8sConfig.SetK8sPluginConfig(&flytek8sConfig.K8sPluginConfig{ GpuDeviceNodeLabel: "gpu-node-label", @@ -506,7 +575,7 @@ func TestBuildResourcePytorchExtendedResources(t *testing.T) { t.Run(tCfg.name+" "+f.name, func(t *testing.T) { taskTemplate := dummyPytorchTaskTemplate("job", tCfg.plugin) taskTemplate.ExtendedResources = f.extendedResourcesBase - taskContext := dummyPytorchTaskContext(taskTemplate, f.resources, f.extendedResourcesOverride) + taskContext := dummyPytorchTaskContext(taskTemplate, f.resources, f.extendedResourcesOverride, "") pytorchResourceHandler := pytorchOperatorResourceHandler{} r, err := pytorchResourceHandler.BuildResource(context.TODO(), taskContext) assert.NoError(t, err) @@ -539,7 +608,7 @@ func TestGetTaskPhase(t *testing.T) { return dummyPytorchJobResource(pytorchResourceHandler, 2, conditionType) } - taskCtx := dummyPytorchTaskContext(dummyPytorchTaskTemplate("", dummyPytorchCustomObj(2)), resourceRequirements, nil) + taskCtx := dummyPytorchTaskContext(dummyPytorchTaskTemplate("", dummyPytorchCustomObj(2)), resourceRequirements, nil, "") taskPhase, err := pytorchResourceHandler.GetTaskPhase(ctx, taskCtx, dummyPytorchJobResourceCreator(commonOp.JobCreated)) assert.NoError(t, err) assert.Equal(t, pluginsCore.PhaseQueued, taskPhase.Phase()) @@ -582,7 +651,7 @@ func TestGetLogs(t *testing.T) { pytorchResourceHandler := pytorchOperatorResourceHandler{} pytorchJob := dummyPytorchJobResource(pytorchResourceHandler, workers, commonOp.JobRunning) - taskCtx := dummyPytorchTaskContext(dummyPytorchTaskTemplate("", dummyPytorchCustomObj(workers)), resourceRequirements, nil) + taskCtx := dummyPytorchTaskContext(dummyPytorchTaskTemplate("", dummyPytorchCustomObj(workers)), resourceRequirements, nil, "") jobLogs, err := common.GetLogs(taskCtx, common.PytorchTaskType, pytorchJob.ObjectMeta, hasMaster, workers, 0, 0, 0) assert.NoError(t, err) assert.Equal(t, 3, len(jobLogs)) @@ -602,7 +671,7 @@ func TestGetLogsElastic(t *testing.T) { pytorchResourceHandler := pytorchOperatorResourceHandler{} pytorchJob := dummyPytorchJobResource(pytorchResourceHandler, workers, commonOp.JobRunning) - taskCtx := dummyPytorchTaskContext(dummyPytorchTaskTemplate("", dummyPytorchCustomObj(workers)), resourceRequirements, nil) + taskCtx := dummyPytorchTaskContext(dummyPytorchTaskTemplate("", dummyPytorchCustomObj(workers)), resourceRequirements, nil, "") jobLogs, err := common.GetLogs(taskCtx, common.PytorchTaskType, pytorchJob.ObjectMeta, hasMaster, workers, 0, 0, 0) assert.NoError(t, err) assert.Equal(t, 2, len(jobLogs)) @@ -633,7 +702,7 @@ func TestReplicaCounts(t *testing.T) { ptObj := dummyPytorchCustomObj(test.workerReplicaCount) taskTemplate := dummyPytorchTaskTemplate("the job", ptObj) - res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) if test.expectError { assert.Error(t, err) assert.Nil(t, res) @@ -715,7 +784,7 @@ func TestBuildResourcePytorchV1(t *testing.T) { taskTemplate := dummyPytorchTaskTemplate("job4", taskConfig) taskTemplate.TaskTypeVersion = 1 - res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, res) @@ -759,7 +828,7 @@ func TestBuildResourcePytorchV1WithRunPolicy(t *testing.T) { taskTemplate := dummyPytorchTaskTemplate("job5", taskConfig) taskTemplate.TaskTypeVersion = 1 - res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, res) @@ -819,7 +888,7 @@ func TestBuildResourcePytorchV1WithOnlyWorkerSpec(t *testing.T) { taskTemplate := dummyPytorchTaskTemplate("job5", taskConfig) taskTemplate.TaskTypeVersion = 1 - res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, res) @@ -890,7 +959,7 @@ func TestBuildResourcePytorchV1ResourceTolerations(t *testing.T) { taskTemplate := dummyPytorchTaskTemplate("job4", taskConfig) taskTemplate.TaskTypeVersion = 1 - res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + res, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, res) @@ -912,7 +981,7 @@ func TestBuildResourcePytorchV1WithElastic(t *testing.T) { taskTemplate.TaskTypeVersion = 1 pytorchResourceHandler := pytorchOperatorResourceHandler{} - resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, resource) @@ -948,7 +1017,7 @@ func TestBuildResourcePytorchV1WithZeroWorker(t *testing.T) { pytorchResourceHandler := pytorchOperatorResourceHandler{} taskTemplate := dummyPytorchTaskTemplate("job5", taskConfig) taskTemplate.TaskTypeVersion = 1 - _, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + _, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.Error(t, err) } @@ -965,7 +1034,7 @@ func TestGetReplicaCount(t *testing.T) { pytorchResourceHandler := pytorchOperatorResourceHandler{} tfObj := dummyPytorchCustomObj(1) taskTemplate := dummyPytorchTaskTemplate("the job", tfObj) - resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil)) + resource, err := pytorchResourceHandler.BuildResource(context.TODO(), dummyPytorchTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.NoError(t, err) assert.NotNil(t, resource) PytorchJob, ok := resource.(*kubeflowv1.PyTorchJob) diff --git a/flyteplugins/go/tasks/plugins/k8s/kfoperators/tensorflow/tensorflow_test.go b/flyteplugins/go/tasks/plugins/k8s/kfoperators/tensorflow/tensorflow_test.go index 2402bd4c5a..80e95871d1 100644 --- a/flyteplugins/go/tasks/plugins/k8s/kfoperators/tensorflow/tensorflow_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/kfoperators/tensorflow/tensorflow_test.go @@ -154,6 +154,7 @@ func dummyTensorFlowTaskContext(taskTemplate *core.TaskTemplate, resources *core overrides := &mocks.TaskOverrides{} overrides.OnGetResources().Return(resources) overrides.OnGetExtendedResources().Return(extendedResources) + overrides.OnGetContainerImage().Return("") taskExecutionMetadata := &mocks.TaskExecutionMetadata{} taskExecutionMetadata.OnGetTaskExecutionID().Return(tID) diff --git a/flyteplugins/go/tasks/plugins/k8s/pod/container_test.go b/flyteplugins/go/tasks/plugins/k8s/pod/container_test.go index e69f764549..9c098bd708 100644 --- a/flyteplugins/go/tasks/plugins/k8s/pod/container_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/pod/container_test.go @@ -79,7 +79,7 @@ func dummyContainerTaskTemplateWithPodSpec(command []string, args []string) *cor return taskTemplate } -func dummyContainerTaskMetadata(resources *v1.ResourceRequirements, extendedResources *core.ExtendedResources, returnsServiceAccount bool) pluginsCore.TaskExecutionMetadata { +func dummyContainerTaskMetadata(resources *v1.ResourceRequirements, extendedResources *core.ExtendedResources, returnsServiceAccount bool, containerImage string) pluginsCore.TaskExecutionMetadata { taskMetadata := &pluginsCoreMock.TaskExecutionMetadata{} taskMetadata.On("GetNamespace").Return("test-namespace") taskMetadata.On("GetAnnotations").Return(map[string]string{"annotation-1": "val1"}) @@ -118,6 +118,7 @@ func dummyContainerTaskMetadata(resources *v1.ResourceRequirements, extendedReso to := &pluginsCoreMock.TaskOverrides{} to.On("GetResources").Return(resources) to.On("GetExtendedResources").Return(extendedResources) + to.OnGetContainerImage().Return(containerImage) taskMetadata.On("GetOverrides").Return(to) taskMetadata.On("IsInterruptible").Return(true) taskMetadata.On("GetEnvironmentVariables").Return(nil) @@ -176,19 +177,19 @@ func TestContainerTaskExecutor_BuildResource(t *testing.T) { { name: "BuildResource", taskTemplate: dummyContainerTaskTemplate(command, args), - taskMetadata: dummyContainerTaskMetadata(containerResourceRequirements, nil, true), + taskMetadata: dummyContainerTaskMetadata(containerResourceRequirements, nil, true, ""), expectServiceAccount: serviceAccount, }, { name: "BuildResource_PodTemplate", taskTemplate: dummyContainerTaskTemplateWithPodSpec(command, args), - taskMetadata: dummyContainerTaskMetadata(containerResourceRequirements, nil, true), + taskMetadata: dummyContainerTaskMetadata(containerResourceRequirements, nil, true, ""), expectServiceAccount: podTemplateServiceAccount, }, { name: "BuildResource_SecurityContext", taskTemplate: dummyContainerTaskTemplate(command, args), - taskMetadata: dummyContainerTaskMetadata(containerResourceRequirements, nil, false), + taskMetadata: dummyContainerTaskMetadata(containerResourceRequirements, nil, false, ""), expectServiceAccount: securityContextServiceAccount, }, } @@ -321,7 +322,7 @@ func TestContainerTaskExecutor_BuildResource_ExtendedResources(t *testing.T) { t.Run(f.name, func(t *testing.T) { taskTemplate := dummyContainerTaskTemplate([]string{"command"}, []string{"{{.Input}}"}) taskTemplate.ExtendedResources = f.extendedResourcesBase - taskMetadata := dummyContainerTaskMetadata(f.resources, f.extendedResourcesOverride, true) + taskMetadata := dummyContainerTaskMetadata(f.resources, f.extendedResourcesOverride, true, "") taskContext := dummyContainerTaskContext(taskTemplate, taskMetadata) r, err := DefaultPodPlugin.BuildResource(context.TODO(), taskContext) assert.Nil(t, err) @@ -343,11 +344,54 @@ func TestContainerTaskExecutor_BuildResource_ExtendedResources(t *testing.T) { } } +func TestContainerTaskExecutor_BuildResource_ContainerImage(t *testing.T) { + assert.NoError(t, flytek8sConfig.SetK8sPluginConfig(&flytek8sConfig.K8sPluginConfig{})) + + fixtures := []struct { + name string + resources *v1.ResourceRequirements + containerImageOverride string + }{ + { + "without overrides", + &v1.ResourceRequirements{ + Limits: v1.ResourceList{ + flytek8s.ResourceNvidiaGPU: resource.MustParse("1"), + }, + }, + "", + }, + { + "with overrides", + &v1.ResourceRequirements{ + Limits: v1.ResourceList{ + flytek8s.ResourceNvidiaGPU: resource.MustParse("1"), + }, + }, + "test-image", + }, + } + + for _, f := range fixtures { + t.Run(f.name, func(t *testing.T) { + taskTemplate := dummyContainerTaskTemplate([]string{"command"}, []string{"{{.Input}}"}) + taskMetadata := dummyContainerTaskMetadata(f.resources, nil, true, f.containerImageOverride) + taskContext := dummyContainerTaskContext(taskTemplate, taskMetadata) + r, err := DefaultPodPlugin.BuildResource(context.TODO(), taskContext) + assert.Nil(t, err) + assert.NotNil(t, r) + _, ok := r.(*v1.Pod) + assert.True(t, ok) + assert.Equal(t, f.containerImageOverride, r.(*v1.Pod).Spec.Containers[0].Image) + }) + } +} + func TestContainerTaskExecutor_GetTaskStatus(t *testing.T) { command := []string{"command"} args := []string{"{{.Input}}"} taskTemplate := dummyContainerTaskTemplate(command, args) - taskMetadata := dummyContainerTaskMetadata(containerResourceRequirements, nil, true) + taskMetadata := dummyContainerTaskMetadata(containerResourceRequirements, nil, true, "") taskCtx := dummyContainerTaskContext(taskTemplate, taskMetadata) j := &v1.Pod{ @@ -437,7 +481,7 @@ func TestContainerTaskExecutor_GetTaskStatus_InvalidImageName(t *testing.T) { command := []string{"command"} args := []string{"{{.Input}}"} taskTemplate := dummyContainerTaskTemplate(command, args) - taskMetadata := dummyContainerTaskMetadata(containerResourceRequirements, nil, true) + taskMetadata := dummyContainerTaskMetadata(containerResourceRequirements, nil, true, "") taskCtx := dummyContainerTaskContext(taskTemplate, taskMetadata) ctx := context.TODO() diff --git a/flyteplugins/go/tasks/plugins/k8s/pod/sidecar_test.go b/flyteplugins/go/tasks/plugins/k8s/pod/sidecar_test.go index a9848f3276..06166d15fb 100644 --- a/flyteplugins/go/tasks/plugins/k8s/pod/sidecar_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/pod/sidecar_test.go @@ -90,6 +90,7 @@ func dummySidecarTaskMetadata(resources *v1.ResourceRequirements, extendedResour to := &pluginsCoreMock.TaskOverrides{} to.On("GetResources").Return(resources) to.On("GetExtendedResources").Return(extendedResources) + to.On("GetContainerImage").Return("") taskMetadata.On("GetOverrides").Return(to) taskMetadata.On("GetEnvironmentVariables").Return(nil) diff --git a/flyteplugins/go/tasks/plugins/k8s/ray/config.go b/flyteplugins/go/tasks/plugins/k8s/ray/config.go index e73fc4dc7d..045f8a09a1 100644 --- a/flyteplugins/go/tasks/plugins/k8s/ray/config.go +++ b/flyteplugins/go/tasks/plugins/k8s/ray/config.go @@ -23,6 +23,7 @@ var ( DashboardHost: "0.0.0.0", EnableUsageStats: false, KubeRayCrdVersion: "v1alpha1", + ServiceAccount: "default", Defaults: DefaultConfig{ HeadNode: NodeConfig{ StartParameters: map[string]string{ @@ -87,6 +88,7 @@ type Config struct { Defaults DefaultConfig `json:"defaults" pflag:"-,Default configuration for ray jobs"` EnableUsageStats bool `json:"enableUsageStats" pflag:",Enable usage stats for ray jobs. These stats are submitted to usage-stats.ray.io per https://docs.ray.io/en/latest/cluster/usage-stats.html"` KubeRayCrdVersion string `json:"kubeRayCrdVersion" pflag:",Version of the Ray CRD to use when creating RayClusters or RayJobs."` + ServiceAccount string `json:"serviceAccount" pflag:",The k8s service account to run as"` } type DefaultConfig struct { diff --git a/flyteplugins/go/tasks/plugins/k8s/ray/config_test.go b/flyteplugins/go/tasks/plugins/k8s/ray/config_test.go index 08c44ce9e1..d19c047ca6 100644 --- a/flyteplugins/go/tasks/plugins/k8s/ray/config_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/ray/config_test.go @@ -25,3 +25,13 @@ func TestLoadConfig(t *testing.T) { assert.DeepEqual(t, config.RemoteClusterConfig, remoteConfig) }) } + +func TestLoadDefaultServiceAccountConfig(t *testing.T) { + rayConfig := GetConfig() + assert.Assert(t, rayConfig != nil) + + t.Run("serviceAccount", func(t *testing.T) { + config := GetConfig() + assert.Equal(t, config.ServiceAccount, "default") + }) +} diff --git a/flyteplugins/go/tasks/plugins/k8s/ray/ray.go b/flyteplugins/go/tasks/plugins/k8s/ray/ray.go index 25291b2066..bc34753425 100644 --- a/flyteplugins/go/tasks/plugins/k8s/ray/ray.go +++ b/flyteplugins/go/tasks/plugins/k8s/ray/ray.go @@ -114,6 +114,8 @@ func (rayJobResourceHandler) BuildResource(ctx context.Context, taskCtx pluginsC headNodeRayStartParams[DisableUsageStatsStartParameter] = DisableUsageStatsStartParameterVal } + podSpec.ServiceAccountName = cfg.ServiceAccount + headPodSpec := podSpec.DeepCopy() if cfg.KubeRayCrdVersion == "v1" { @@ -300,12 +302,15 @@ func constructV1Job(taskCtx pluginsCore.TaskExecutionContext, rayJob plugins.Ray ttlSecondsAfterFinished = &rayJob.TtlSecondsAfterFinished } + submitterPodTemplate := buildSubmitterPodTemplate(headPodSpec, objectMeta, taskCtx) + jobSpec := rayv1.RayJobSpec{ RayClusterSpec: &rayClusterSpec, Entrypoint: strings.Join(primaryContainer.Args, " "), ShutdownAfterJobFinishes: shutdownAfterJobFinishes, TTLSecondsAfterFinished: ttlSecondsAfterFinished, RuntimeEnv: rayJob.RuntimeEnv, + SubmitterPodTemplate: &submitterPodTemplate, } return &rayv1.RayJob{ @@ -425,6 +430,17 @@ func buildHeadPodTemplate(primaryContainer *v1.Container, podSpec *v1.PodSpec, o return podTemplateSpec } +func buildSubmitterPodTemplate(podSpec *v1.PodSpec, objectMeta *metav1.ObjectMeta, taskCtx pluginsCore.TaskExecutionContext) v1.PodTemplateSpec { + podTemplateSpec := v1.PodTemplateSpec{ + Spec: *podSpec, + ObjectMeta: *objectMeta, + } + cfg := config.GetK8sPluginConfig() + podTemplateSpec.SetLabels(utils.UnionMaps(cfg.DefaultLabels, podTemplateSpec.GetLabels(), utils.CopyMap(taskCtx.TaskExecutionMetadata().GetLabels()))) + podTemplateSpec.SetAnnotations(utils.UnionMaps(cfg.DefaultAnnotations, podTemplateSpec.GetAnnotations(), utils.CopyMap(taskCtx.TaskExecutionMetadata().GetAnnotations()))) + return podTemplateSpec +} + func buildWorkerPodTemplate(primaryContainer *v1.Container, podSpec *v1.PodSpec, objectMetadata *metav1.ObjectMeta, taskCtx pluginsCore.TaskExecutionContext) v1.PodTemplateSpec { // Some configs are copy from https://github.com/ray-project/kuberay/blob/b72e6bdcd9b8c77a9dc6b5da8560910f3a0c3ffd/apiserver/pkg/util/cluster.go#L185 // They should always be the same, so we could hard code here. diff --git a/flyteplugins/go/tasks/plugins/k8s/ray/ray_test.go b/flyteplugins/go/tasks/plugins/k8s/ray/ray_test.go index 02ed83db14..69f65c19ef 100644 --- a/flyteplugins/go/tasks/plugins/k8s/ray/ray_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/ray/ray_test.go @@ -106,7 +106,7 @@ func dummyRayTaskTemplate(id string, rayJob *plugins.RayJob) *core.TaskTemplate } } -func dummyRayTaskContext(taskTemplate *core.TaskTemplate, resources *corev1.ResourceRequirements, extendedResources *core.ExtendedResources) pluginsCore.TaskExecutionContext { +func dummyRayTaskContext(taskTemplate *core.TaskTemplate, resources *corev1.ResourceRequirements, extendedResources *core.ExtendedResources, containerImage string) pluginsCore.TaskExecutionContext { taskCtx := &mocks.TaskExecutionContext{} inputReader := &pluginIOMocks.InputReader{} inputReader.OnGetInputPrefixPath().Return("/input/prefix") @@ -141,6 +141,7 @@ func dummyRayTaskContext(taskTemplate *core.TaskTemplate, resources *corev1.Reso overrides := &mocks.TaskOverrides{} overrides.OnGetResources().Return(resources) overrides.OnGetExtendedResources().Return(extendedResources) + overrides.OnGetContainerImage().Return(containerImage) taskExecutionMetadata := &mocks.TaskExecutionMetadata{} taskExecutionMetadata.OnGetTaskExecutionID().Return(tID) @@ -175,7 +176,7 @@ func TestBuildResourceRay(t *testing.T) { err := config.SetK8sPluginConfig(&config.K8sPluginConfig{DefaultTolerations: toleration}) assert.Nil(t, err) - RayResource, err := rayJobResourceHandler.BuildResource(context.TODO(), dummyRayTaskContext(taskTemplate, resourceRequirements, nil)) + RayResource, err := rayJobResourceHandler.BuildResource(context.TODO(), dummyRayTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.Nil(t, err) assert.NotNil(t, RayResource) @@ -210,6 +211,63 @@ func TestBuildResourceRay(t *testing.T) { assert.Equal(t, ray.Spec.RayClusterSpec.WorkerGroupSpecs[0].Template.Spec.Tolerations, toleration) } +func TestBuildResourceRayContainerImage(t *testing.T) { + assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{})) + + fixtures := []struct { + name string + resources *corev1.ResourceRequirements + containerImageOverride string + }{ + { + "without overrides", + &corev1.ResourceRequirements{ + Limits: corev1.ResourceList{ + flytek8s.ResourceNvidiaGPU: resource.MustParse("1"), + }, + }, + "", + }, + { + "with overrides", + &corev1.ResourceRequirements{ + Limits: corev1.ResourceList{ + flytek8s.ResourceNvidiaGPU: resource.MustParse("1"), + }, + }, + "container-image-override", + }, + } + + for _, f := range fixtures { + t.Run(f.name, func(t *testing.T) { + taskTemplate := dummyRayTaskTemplate("id", dummyRayCustomObj()) + taskContext := dummyRayTaskContext(taskTemplate, f.resources, nil, f.containerImageOverride) + rayJobResourceHandler := rayJobResourceHandler{} + r, err := rayJobResourceHandler.BuildResource(context.TODO(), taskContext) + assert.Nil(t, err) + assert.NotNil(t, r) + rayJob, ok := r.(*rayv1alpha1.RayJob) + assert.True(t, ok) + + var expectedContainerImage string + if len(f.containerImageOverride) > 0 { + expectedContainerImage = f.containerImageOverride + } else { + expectedContainerImage = testImage + } + + // Head node + headNodeSpec := rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.Spec + assert.Equal(t, expectedContainerImage, headNodeSpec.Containers[0].Image) + + // Worker node + workerNodeSpec := rayJob.Spec.RayClusterSpec.WorkerGroupSpecs[0].Template.Spec + assert.Equal(t, expectedContainerImage, workerNodeSpec.Containers[0].Image) + }) + } +} + func TestBuildResourceRayExtendedResources(t *testing.T) { assert.NoError(t, config.SetK8sPluginConfig(&config.K8sPluginConfig{ GpuDeviceNodeLabel: "gpu-node-label", @@ -315,7 +373,7 @@ func TestBuildResourceRayExtendedResources(t *testing.T) { t.Run(p.name, func(t *testing.T) { taskTemplate := dummyRayTaskTemplate("ray-id", dummyRayCustomObj()) taskTemplate.ExtendedResources = p.extendedResourcesBase - taskContext := dummyRayTaskContext(taskTemplate, p.resources, p.extendedResourcesOverride) + taskContext := dummyRayTaskContext(taskTemplate, p.resources, p.extendedResourcesOverride, "") rayJobResourceHandler := rayJobResourceHandler{} r, err := rayJobResourceHandler.BuildResource(context.TODO(), taskContext) assert.Nil(t, err) @@ -374,7 +432,7 @@ func TestDefaultStartParameters(t *testing.T) { err := config.SetK8sPluginConfig(&config.K8sPluginConfig{DefaultTolerations: toleration}) assert.Nil(t, err) - RayResource, err := rayJobResourceHandler.BuildResource(context.TODO(), dummyRayTaskContext(taskTemplate, resourceRequirements, nil)) + RayResource, err := rayJobResourceHandler.BuildResource(context.TODO(), dummyRayTaskContext(taskTemplate, resourceRequirements, nil, "")) assert.Nil(t, err) assert.NotNil(t, RayResource) @@ -582,7 +640,7 @@ func TestInjectLogsSidecar(t *testing.T) { assert.NoError(t, SetConfig(&Config{ LogsSidecar: p.logsSidecarCfg, })) - taskContext := dummyRayTaskContext(&p.taskTemplate, resourceRequirements, nil) + taskContext := dummyRayTaskContext(&p.taskTemplate, resourceRequirements, nil, "") rayJobResourceHandler := rayJobResourceHandler{} r, err := rayJobResourceHandler.BuildResource(context.TODO(), taskContext) assert.Nil(t, err) diff --git a/flyteplugins/go/tasks/plugins/k8s/spark/spark_test.go b/flyteplugins/go/tasks/plugins/k8s/spark/spark_test.go index 264f9514e9..561901226a 100644 --- a/flyteplugins/go/tasks/plugins/k8s/spark/spark_test.go +++ b/flyteplugins/go/tasks/plugins/k8s/spark/spark_test.go @@ -385,6 +385,7 @@ func dummySparkTaskContext(taskTemplate *core.TaskTemplate, interruptible bool) overrides.On("GetResources").Return(&corev1.ResourceRequirements{}) // No support for GPUs, and consequently, ExtendedResources on Spark plugin. overrides.On("GetExtendedResources").Return(nil) + overrides.OnGetContainerImage().Return("") taskExecutionMetadata := &mocks.TaskExecutionMetadata{} taskExecutionMetadata.On("GetTaskExecutionID").Return(tID) diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/client.go b/flyteplugins/go/tasks/plugins/webapi/agent/client.go index b118f64596..b525acc5c3 100644 --- a/flyteplugins/go/tasks/plugins/webapi/agent/client.go +++ b/flyteplugins/go/tasks/plugins/webapi/agent/client.go @@ -19,13 +19,27 @@ import ( "github.com/flyteorg/flyte/flytestdlib/logger" ) +const defaultTaskTypeVersion = 0 + +type Agent struct { + // IsSync indicates whether this agent is a sync agent. Sync agents are expected to return their + // results synchronously when called by propeller. Given that sync agents can affect the performance + // of the system, it's important to enforce strict timeout policies. + // An Async agent, on the other hand, is required to be able to identify jobs by an + // identifier and query for job statuses as jobs progress. + IsSync bool + // AgentDeployment is the agent deployment where this agent is running. + AgentDeployment *Deployment +} + // ClientSet contains the clients exposed to communicate with various agent services. type ClientSet struct { - agentClients map[string]service.AsyncAgentServiceClient // map[endpoint] => client - agentMetadataClients map[string]service.AgentMetadataServiceClient // map[endpoint] => client + asyncAgentClients map[string]service.AsyncAgentServiceClient // map[endpoint] => AsyncAgentServiceClient + syncAgentClients map[string]service.SyncAgentServiceClient // map[endpoint] => SyncAgentServiceClient + agentMetadataClients map[string]service.AgentMetadataServiceClient // map[endpoint] => AgentMetadataServiceClient } -func getGrpcConnection(ctx context.Context, agent *Agent) (*grpc.ClientConn, error) { +func getGrpcConnection(ctx context.Context, agent *Deployment) (*grpc.ClientConn, error) { var opts []grpc.DialOption if agent.Insecure { @@ -67,7 +81,7 @@ func getGrpcConnection(ctx context.Context, agent *Agent) (*grpc.ClientConn, err return conn, nil } -func getFinalTimeout(operation string, agent *Agent) config.Duration { +func getFinalTimeout(operation string, agent *Deployment) config.Duration { if t, exists := agent.Timeouts[operation]; exists { return t } @@ -75,7 +89,7 @@ func getFinalTimeout(operation string, agent *Agent) config.Duration { return agent.DefaultTimeout } -func getFinalContext(ctx context.Context, operation string, agent *Agent) (context.Context, context.CancelFunc) { +func getFinalContext(ctx context.Context, operation string, agent *Deployment) (context.Context, context.CancelFunc) { timeout := getFinalTimeout(operation, agent).Duration if timeout == 0 { return ctx, func() {} @@ -84,20 +98,21 @@ func getFinalContext(ctx context.Context, operation string, agent *Agent) (conte return context.WithTimeout(ctx, timeout) } -func initializeAgentRegistry(cs *ClientSet) (map[string]*Agent, error) { - agentRegistry := make(map[string]*Agent) +func initializeAgentRegistry(cs *ClientSet) (Registry, error) { + agentRegistry := make(Registry) cfg := GetConfig() - var agentDeployments []*Agent + var agentDeployments []*Deployment // Ensure that the old configuration is backward compatible - for taskType, agentID := range cfg.AgentForTaskTypes { - agentRegistry[taskType] = cfg.Agents[agentID] + for taskType, agentDeploymentID := range cfg.AgentForTaskTypes { + agent := Agent{AgentDeployment: cfg.AgentDeployments[agentDeploymentID], IsSync: false} + agentRegistry[taskType] = map[int32]*Agent{defaultTaskTypeVersion: &agent} } if len(cfg.DefaultAgent.Endpoint) != 0 { agentDeployments = append(agentDeployments, &cfg.DefaultAgent) } - agentDeployments = append(agentDeployments, maps.Values(cfg.Agents)...) + agentDeployments = append(agentDeployments, maps.Values(cfg.AgentDeployments)...) for _, agentDeployment := range agentDeployments { client := cs.agentMetadataClients[agentDeployment.Endpoint] @@ -120,12 +135,20 @@ func initializeAgentRegistry(cs *ClientSet) (map[string]*Agent, error) { return nil, fmt.Errorf("failed to list agent: [%v] with error: [%v]", agentDeployment, err) } - agents := res.GetAgents() - for _, agent := range agents { - supportedTaskTypes := agent.SupportedTaskTypes - for _, supportedTaskType := range supportedTaskTypes { - agentRegistry[supportedTaskType] = agentDeployment + for _, agent := range res.GetAgents() { + deprecatedSupportedTaskTypes := agent.SupportedTaskTypes + for _, supportedTaskType := range deprecatedSupportedTaskTypes { + agent := &Agent{AgentDeployment: agentDeployment, IsSync: agent.IsSync} + agentRegistry[supportedTaskType] = map[int32]*Agent{defaultTaskTypeVersion: agent} } + + supportedTaskCategories := agent.SupportedTaskCategories + for _, supportedCategory := range supportedTaskCategories { + agent := &Agent{AgentDeployment: agentDeployment, IsSync: agent.IsSync} + agentRegistry[supportedCategory.GetName()] = map[int32]*Agent{supportedCategory.GetVersion(): agent} + } + logger.Infof(context.Background(), "[%v] is a sync agent: [%v]", agent.Name, agent.IsSync) + logger.Infof(context.Background(), "[%v] supports task category: [%v]", agent.Name, supportedTaskCategories) } } @@ -133,27 +156,30 @@ func initializeAgentRegistry(cs *ClientSet) (map[string]*Agent, error) { } func initializeClients(ctx context.Context) (*ClientSet, error) { - agentClients := make(map[string]service.AsyncAgentServiceClient) + asyncAgentClients := make(map[string]service.AsyncAgentServiceClient) + syncAgentClients := make(map[string]service.SyncAgentServiceClient) agentMetadataClients := make(map[string]service.AgentMetadataServiceClient) - var agentDeployments []*Agent + var agentDeployments []*Deployment cfg := GetConfig() if len(cfg.DefaultAgent.Endpoint) != 0 { agentDeployments = append(agentDeployments, &cfg.DefaultAgent) } - agentDeployments = append(agentDeployments, maps.Values(cfg.Agents)...) - for _, agentDeployment := range agentDeployments { - conn, err := getGrpcConnection(ctx, agentDeployment) + agentDeployments = append(agentDeployments, maps.Values(cfg.AgentDeployments)...) + for _, agentService := range agentDeployments { + conn, err := getGrpcConnection(ctx, agentService) if err != nil { return nil, err } - agentClients[agentDeployment.Endpoint] = service.NewAsyncAgentServiceClient(conn) - agentMetadataClients[agentDeployment.Endpoint] = service.NewAgentMetadataServiceClient(conn) + syncAgentClients[agentService.Endpoint] = service.NewSyncAgentServiceClient(conn) + asyncAgentClients[agentService.Endpoint] = service.NewAsyncAgentServiceClient(conn) + agentMetadataClients[agentService.Endpoint] = service.NewAgentMetadataServiceClient(conn) } return &ClientSet{ - agentClients: agentClients, + syncAgentClients: syncAgentClients, + asyncAgentClients: asyncAgentClients, agentMetadataClients: agentMetadataClients, }, nil } diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/client_test.go b/flyteplugins/go/tasks/plugins/webapi/agent/client_test.go index d68811d037..4ad7f8cbaa 100644 --- a/flyteplugins/go/tasks/plugins/webapi/agent/client_test.go +++ b/flyteplugins/go/tasks/plugins/webapi/agent/client_test.go @@ -9,10 +9,22 @@ import ( func TestInitializeClients(t *testing.T) { cfg := defaultConfig + cfg.AgentDeployments = map[string]*Deployment{ + "x": { + Endpoint: "x", + }, + "y": { + Endpoint: "y", + }, + } ctx := context.Background() err := SetConfig(&cfg) assert.NoError(t, err) cs, err := initializeClients(ctx) assert.NoError(t, err) assert.NotNil(t, cs) + _, ok := cs.syncAgentClients["y"] + assert.True(t, ok) + _, ok = cs.asyncAgentClients["x"] + assert.True(t, ok) } diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/config.go b/flyteplugins/go/tasks/plugins/webapi/agent/config.go index cb0bd3089f..3f9fd354b6 100644 --- a/flyteplugins/go/tasks/plugins/webapi/agent/config.go +++ b/flyteplugins/go/tasks/plugins/webapi/agent/config.go @@ -39,7 +39,7 @@ var ( Value: 50, }, }, - DefaultAgent: Agent{ + DefaultAgent: Deployment{ Endpoint: "", Insecure: true, DefaultTimeout: config.Duration{Duration: 10 * time.Second}, @@ -61,19 +61,19 @@ type Config struct { ResourceConstraints core.ResourceConstraintsSpec `json:"resourceConstraints" pflag:"-,Defines resource constraints on how many executions to be created per project/overall at any given time."` // The default agent if there does not exist a more specific matching against task types - DefaultAgent Agent `json:"defaultAgent" pflag:",The default agent."` + DefaultAgent Deployment `json:"defaultAgent" pflag:",The default agent."` - // The agents used to match against specific task types. {AgentId: Agent} - Agents map[string]*Agent `json:"agents" pflag:",The agents."` + // The agents used to match against specific task types. {agentDeploymentID: AgentDeployment} + AgentDeployments map[string]*Deployment `json:"agents" pflag:",The agents."` - // Maps task types to their agents. {TaskType: AgentId} + // Maps task types to their agents. {TaskType: agentDeploymentID} AgentForTaskTypes map[string]string `json:"agentForTaskTypes" pflag:"-,"` // SupportedTaskTypes is a list of task types that are supported by this plugin. SupportedTaskTypes []string `json:"supportedTaskTypes" pflag:"-,Defines a list of task types that are supported by this plugin."` } -type Agent struct { +type Deployment struct { // Endpoint points to an agent gRPC endpoint Endpoint string `json:"endpoint"` diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/config_test.go b/flyteplugins/go/tasks/plugins/webapi/agent/config_test.go index a3e6d69d66..09abffb83c 100644 --- a/flyteplugins/go/tasks/plugins/webapi/agent/config_test.go +++ b/flyteplugins/go/tasks/plugins/webapi/agent/config_test.go @@ -27,7 +27,7 @@ func TestGetAndSetConfig(t *testing.T) { }, } cfg.DefaultAgent.DefaultTimeout = config.Duration{Duration: 10 * time.Second} - cfg.Agents = map[string]*Agent{ + cfg.AgentDeployments = map[string]*Deployment{ "agent_1": { Insecure: cfg.DefaultAgent.Insecure, DefaultServiceConfig: cfg.DefaultAgent.DefaultServiceConfig, diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/integration_test.go b/flyteplugins/go/tasks/plugins/webapi/agent/integration_test.go index fe3b45b881..a4ddc5e303 100644 --- a/flyteplugins/go/tasks/plugins/webapi/agent/integration_test.go +++ b/flyteplugins/go/tasks/plugins/webapi/agent/integration_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "fmt" + "io" "sync/atomic" "testing" "time" @@ -13,6 +14,7 @@ import ( "k8s.io/apimachinery/pkg/util/rand" "k8s.io/utils/strings/slices" + agentMocks "github.com/flyteorg/flyte/flyteidl/clients/go/admin/mocks" "github.com/flyteorg/flyte/flyteidl/clients/go/coreutils" "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" flyteIdlCore "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core" @@ -24,7 +26,6 @@ import ( pluginCoreMocks "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/core/mocks" ioMocks "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/io/mocks" "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/webapi" - agentMocks "github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent/mocks" "github.com/flyteorg/flyte/flyteplugins/tests" "github.com/flyteorg/flyte/flytestdlib/contextutils" "github.com/flyteorg/flyte/flytestdlib/promutils" @@ -72,9 +73,9 @@ func TestEndToEnd(t *testing.T) { } basePrefix := storage.DataReference("fake://bucket/prefix/") - t.Run("run a job", func(t *testing.T) { - pluginEntry := pluginmachinery.CreateRemotePlugin(newMockAgentPlugin()) - plugin, err := pluginEntry.LoadPlugin(context.TODO(), newFakeSetupContext("test1")) + t.Run("run an async task", func(t *testing.T) { + pluginEntry := pluginmachinery.CreateRemotePlugin(newMockAsyncAgentPlugin()) + plugin, err := pluginEntry.LoadPlugin(context.TODO(), newFakeSetupContext("async task")) assert.NoError(t, err) phase := tests.RunPluginEndToEndTest(t, plugin, &template, inputs, nil, nil, iter) @@ -85,14 +86,38 @@ func TestEndToEnd(t *testing.T) { assert.Equal(t, true, phase.Phase().IsSuccess()) }) + t.Run("run a sync task", func(t *testing.T) { + pluginEntry := pluginmachinery.CreateRemotePlugin(newMockSyncAgentPlugin()) + plugin, err := pluginEntry.LoadPlugin(context.TODO(), newFakeSetupContext("sync task")) + assert.NoError(t, err) + + template.Type = "openai" + template.Interface = &flyteIdlCore.TypedInterface{ + Outputs: &flyteIdlCore.VariableMap{ + Variables: map[string]*flyteIdlCore.Variable{ + "x": {Type: &flyteIdlCore.LiteralType{ + Type: &flyteIdlCore.LiteralType_Simple{ + Simple: flyteIdlCore.SimpleType_INTEGER, + }, + }, + }, + }, + }, + } + expectedOutputs, err := coreutils.MakeLiteralMap(map[string]interface{}{"x": 1}) + assert.NoError(t, err) + phase := tests.RunPluginEndToEndTest(t, plugin, &template, inputs, expectedOutputs, nil, iter) + assert.Equal(t, true, phase.Phase().IsSuccess()) + }) + t.Run("failed to create a job", func(t *testing.T) { - agentPlugin := newMockAgentPlugin() + agentPlugin := newMockAsyncAgentPlugin() agentPlugin.PluginLoader = func(ctx context.Context, iCtx webapi.PluginSetupContext) (webapi.AsyncPlugin, error) { return Plugin{ metricScope: iCtx.MetricsScope(), cfg: GetConfig(), cs: &ClientSet{ - agentClients: map[string]service.AsyncAgentServiceClient{}, + asyncAgentClients: map[string]service.AsyncAgentServiceClient{}, agentMetadataClients: map[string]service.AgentMetadataServiceClient{}, }, }, nil @@ -124,7 +149,7 @@ func TestEndToEnd(t *testing.T) { tr.OnRead(context.Background()).Return(nil, fmt.Errorf("read fail")) tCtx.OnTaskReader().Return(tr) - agentPlugin := newMockAgentPlugin() + agentPlugin := newMockAsyncAgentPlugin() pluginEntry := pluginmachinery.CreateRemotePlugin(agentPlugin) plugin, err := pluginEntry.LoadPlugin(context.TODO(), newFakeSetupContext("test3")) assert.NoError(t, err) @@ -145,7 +170,7 @@ func TestEndToEnd(t *testing.T) { inputReader.OnGetMatch(mock.Anything).Return(nil, fmt.Errorf("read fail")) tCtx.OnInputReader().Return(inputReader) - agentPlugin := newMockAgentPlugin() + agentPlugin := newMockAsyncAgentPlugin() pluginEntry := pluginmachinery.CreateRemotePlugin(agentPlugin) plugin, err := pluginEntry.LoadPlugin(context.TODO(), newFakeSetupContext("test4")) assert.NoError(t, err) @@ -224,26 +249,23 @@ func getTaskContext(t *testing.T) *pluginCoreMocks.TaskExecutionContext { return tCtx } -func newMockAgentPlugin() webapi.PluginEntry { - - agentClient := new(agentMocks.AsyncAgentServiceClient) +func newMockAsyncAgentPlugin() webapi.PluginEntry { + asyncAgentClient := new(agentMocks.AsyncAgentServiceClient) mockCreateRequestMatcher := mock.MatchedBy(func(request *admin.CreateTaskRequest) bool { expectedArgs := []string{"pyflyte-fast-execute", "--output-prefix", "/tmp/123"} return slices.Equal(request.Template.GetContainer().Args, expectedArgs) }) - agentClient.On("CreateTask", mock.Anything, mockCreateRequestMatcher).Return(&admin.CreateTaskResponse{ - Res: &admin.CreateTaskResponse_ResourceMeta{ - ResourceMeta: []byte{1, 2, 3, 4}, - }}, nil) + asyncAgentClient.On("CreateTask", mock.Anything, mockCreateRequestMatcher).Return(&admin.CreateTaskResponse{ + ResourceMeta: []byte{1, 2, 3, 4}}, nil) mockGetRequestMatcher := mock.MatchedBy(func(request *admin.GetTaskRequest) bool { - return request.GetTaskType() == "spark" + return request.GetTaskCategory().GetName() == "spark" }) - agentClient.On("GetTask", mock.Anything, mockGetRequestMatcher).Return( - &admin.GetTaskResponse{Resource: &admin.Resource{State: admin.State_SUCCEEDED}}, nil) + asyncAgentClient.On("GetTask", mock.Anything, mockGetRequestMatcher).Return( + &admin.GetTaskResponse{Resource: &admin.Resource{Phase: flyteIdlCore.TaskExecution_SUCCEEDED}}, nil) - agentClient.On("DeleteTask", mock.Anything, mock.Anything).Return( + asyncAgentClient.On("DeleteTask", mock.Anything, mock.Anything).Return( &admin.DeleteTaskResponse{}, nil) cfg := defaultConfig @@ -251,16 +273,57 @@ func newMockAgentPlugin() webapi.PluginEntry { return webapi.PluginEntry{ ID: "agent-service", - SupportedTaskTypes: []core.TaskType{"bigquery_query_job_task", "spark", "api_task"}, + SupportedTaskTypes: []core.TaskType{"bigquery_query_job_task", "spark"}, + PluginLoader: func(ctx context.Context, iCtx webapi.PluginSetupContext) (webapi.AsyncPlugin, error) { + return Plugin{ + metricScope: iCtx.MetricsScope(), + cfg: &cfg, + cs: &ClientSet{ + asyncAgentClients: map[string]service.AsyncAgentServiceClient{ + defaultAgentEndpoint: asyncAgentClient, + }, + }, + }, nil + }, + } +} + +func newMockSyncAgentPlugin() webapi.PluginEntry { + syncAgentClient := new(agentMocks.SyncAgentServiceClient) + output, _ := coreutils.MakeLiteralMap(map[string]interface{}{"x": 1}) + resource := &admin.Resource{Phase: flyteIdlCore.TaskExecution_SUCCEEDED, Outputs: output} + + stream := new(agentMocks.SyncAgentService_ExecuteTaskSyncClient) + stream.OnRecv().Return(&admin.ExecuteTaskSyncResponse{ + Res: &admin.ExecuteTaskSyncResponse_Header{ + Header: &admin.ExecuteTaskSyncResponseHeader{ + Resource: resource, + }, + }, + }, nil).Once() + + stream.OnRecv().Return(nil, io.EOF).Once() + stream.OnSendMatch(mock.Anything).Return(nil) + stream.OnCloseSendMatch(mock.Anything).Return(nil) + + syncAgentClient.OnExecuteTaskSyncMatch(mock.Anything).Return(stream, nil) + + cfg := defaultConfig + cfg.DefaultAgent.Endpoint = defaultAgentEndpoint + + return webapi.PluginEntry{ + ID: "agent-service", + SupportedTaskTypes: []core.TaskType{"openai"}, PluginLoader: func(ctx context.Context, iCtx webapi.PluginSetupContext) (webapi.AsyncPlugin, error) { return Plugin{ metricScope: iCtx.MetricsScope(), cfg: &cfg, cs: &ClientSet{ - agentClients: map[string]service.AsyncAgentServiceClient{ - "localhost:8000": agentClient, + syncAgentClients: map[string]service.SyncAgentServiceClient{ + defaultAgentEndpoint: syncAgentClient, }, }, + agentRegistry: Registry{"openai": {defaultTaskTypeVersion: {AgentDeployment: &Deployment{Endpoint: defaultAgentEndpoint}, IsSync: true}}}, }, nil }, } diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/mocks/AgentMetadataServiceClient.go b/flyteplugins/go/tasks/plugins/webapi/agent/mocks/AgentMetadataServiceClient.go deleted file mode 100644 index d7f40932b7..0000000000 --- a/flyteplugins/go/tasks/plugins/webapi/agent/mocks/AgentMetadataServiceClient.go +++ /dev/null @@ -1,114 +0,0 @@ -// Code generated by mockery v1.0.1. DO NOT EDIT. - -package mocks - -import ( - context "context" - - admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" - - grpc "google.golang.org/grpc" - - mock "github.com/stretchr/testify/mock" -) - -// AgentMetadataServiceClient is an autogenerated mock type for the AgentMetadataServiceClient type -type AgentMetadataServiceClient struct { - mock.Mock -} - -type AgentMetadataServiceClient_GetAgent struct { - *mock.Call -} - -func (_m AgentMetadataServiceClient_GetAgent) Return(_a0 *admin.GetAgentResponse, _a1 error) *AgentMetadataServiceClient_GetAgent { - return &AgentMetadataServiceClient_GetAgent{Call: _m.Call.Return(_a0, _a1)} -} - -func (_m *AgentMetadataServiceClient) OnGetAgent(ctx context.Context, in *admin.GetAgentRequest, opts ...grpc.CallOption) *AgentMetadataServiceClient_GetAgent { - c_call := _m.On("GetAgent", ctx, in, opts) - return &AgentMetadataServiceClient_GetAgent{Call: c_call} -} - -func (_m *AgentMetadataServiceClient) OnGetAgentMatch(matchers ...interface{}) *AgentMetadataServiceClient_GetAgent { - c_call := _m.On("GetAgent", matchers...) - return &AgentMetadataServiceClient_GetAgent{Call: c_call} -} - -// GetAgent provides a mock function with given fields: ctx, in, opts -func (_m *AgentMetadataServiceClient) GetAgent(ctx context.Context, in *admin.GetAgentRequest, opts ...grpc.CallOption) (*admin.GetAgentResponse, error) { - _va := make([]interface{}, len(opts)) - for _i := range opts { - _va[_i] = opts[_i] - } - var _ca []interface{} - _ca = append(_ca, ctx, in) - _ca = append(_ca, _va...) - ret := _m.Called(_ca...) - - var r0 *admin.GetAgentResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.GetAgentRequest, ...grpc.CallOption) *admin.GetAgentResponse); ok { - r0 = rf(ctx, in, opts...) - } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.GetAgentResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.GetAgentRequest, ...grpc.CallOption) error); ok { - r1 = rf(ctx, in, opts...) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} - -type AgentMetadataServiceClient_ListAgents struct { - *mock.Call -} - -func (_m AgentMetadataServiceClient_ListAgents) Return(_a0 *admin.ListAgentsResponse, _a1 error) *AgentMetadataServiceClient_ListAgents { - return &AgentMetadataServiceClient_ListAgents{Call: _m.Call.Return(_a0, _a1)} -} - -func (_m *AgentMetadataServiceClient) OnListAgents(ctx context.Context, in *admin.ListAgentsRequest, opts ...grpc.CallOption) *AgentMetadataServiceClient_ListAgents { - c_call := _m.On("ListAgents", ctx, in, opts) - return &AgentMetadataServiceClient_ListAgents{Call: c_call} -} - -func (_m *AgentMetadataServiceClient) OnListAgentsMatch(matchers ...interface{}) *AgentMetadataServiceClient_ListAgents { - c_call := _m.On("ListAgents", matchers...) - return &AgentMetadataServiceClient_ListAgents{Call: c_call} -} - -// ListAgents provides a mock function with given fields: ctx, in, opts -func (_m *AgentMetadataServiceClient) ListAgents(ctx context.Context, in *admin.ListAgentsRequest, opts ...grpc.CallOption) (*admin.ListAgentsResponse, error) { - _va := make([]interface{}, len(opts)) - for _i := range opts { - _va[_i] = opts[_i] - } - var _ca []interface{} - _ca = append(_ca, ctx, in) - _ca = append(_ca, _va...) - ret := _m.Called(_ca...) - - var r0 *admin.ListAgentsResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.ListAgentsRequest, ...grpc.CallOption) *admin.ListAgentsResponse); ok { - r0 = rf(ctx, in, opts...) - } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.ListAgentsResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.ListAgentsRequest, ...grpc.CallOption) error); ok { - r1 = rf(ctx, in, opts...) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/mocks/AsyncAgentServiceClient.go b/flyteplugins/go/tasks/plugins/webapi/agent/mocks/AsyncAgentServiceClient.go deleted file mode 100644 index f11ef1adfe..0000000000 --- a/flyteplugins/go/tasks/plugins/webapi/agent/mocks/AsyncAgentServiceClient.go +++ /dev/null @@ -1,258 +0,0 @@ -// Code generated by mockery v1.0.1. DO NOT EDIT. - -package mocks - -import ( - context "context" - - admin "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" - - grpc "google.golang.org/grpc" - - mock "github.com/stretchr/testify/mock" -) - -// AsyncAgentServiceClient is an autogenerated mock type for the AsyncAgentServiceClient type -type AsyncAgentServiceClient struct { - mock.Mock -} - -type AsyncAgentServiceClient_CreateTask struct { - *mock.Call -} - -func (_m AsyncAgentServiceClient_CreateTask) Return(_a0 *admin.CreateTaskResponse, _a1 error) *AsyncAgentServiceClient_CreateTask { - return &AsyncAgentServiceClient_CreateTask{Call: _m.Call.Return(_a0, _a1)} -} - -func (_m *AsyncAgentServiceClient) OnCreateTask(ctx context.Context, in *admin.CreateTaskRequest, opts ...grpc.CallOption) *AsyncAgentServiceClient_CreateTask { - c_call := _m.On("CreateTask", ctx, in, opts) - return &AsyncAgentServiceClient_CreateTask{Call: c_call} -} - -func (_m *AsyncAgentServiceClient) OnCreateTaskMatch(matchers ...interface{}) *AsyncAgentServiceClient_CreateTask { - c_call := _m.On("CreateTask", matchers...) - return &AsyncAgentServiceClient_CreateTask{Call: c_call} -} - -// CreateTask provides a mock function with given fields: ctx, in, opts -func (_m *AsyncAgentServiceClient) CreateTask(ctx context.Context, in *admin.CreateTaskRequest, opts ...grpc.CallOption) (*admin.CreateTaskResponse, error) { - _va := make([]interface{}, len(opts)) - for _i := range opts { - _va[_i] = opts[_i] - } - var _ca []interface{} - _ca = append(_ca, ctx, in) - _ca = append(_ca, _va...) - ret := _m.Called(_ca...) - - var r0 *admin.CreateTaskResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.CreateTaskRequest, ...grpc.CallOption) *admin.CreateTaskResponse); ok { - r0 = rf(ctx, in, opts...) - } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.CreateTaskResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.CreateTaskRequest, ...grpc.CallOption) error); ok { - r1 = rf(ctx, in, opts...) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} - -type AsyncAgentServiceClient_DeleteTask struct { - *mock.Call -} - -func (_m AsyncAgentServiceClient_DeleteTask) Return(_a0 *admin.DeleteTaskResponse, _a1 error) *AsyncAgentServiceClient_DeleteTask { - return &AsyncAgentServiceClient_DeleteTask{Call: _m.Call.Return(_a0, _a1)} -} - -func (_m *AsyncAgentServiceClient) OnDeleteTask(ctx context.Context, in *admin.DeleteTaskRequest, opts ...grpc.CallOption) *AsyncAgentServiceClient_DeleteTask { - c_call := _m.On("DeleteTask", ctx, in, opts) - return &AsyncAgentServiceClient_DeleteTask{Call: c_call} -} - -func (_m *AsyncAgentServiceClient) OnDeleteTaskMatch(matchers ...interface{}) *AsyncAgentServiceClient_DeleteTask { - c_call := _m.On("DeleteTask", matchers...) - return &AsyncAgentServiceClient_DeleteTask{Call: c_call} -} - -// DeleteTask provides a mock function with given fields: ctx, in, opts -func (_m *AsyncAgentServiceClient) DeleteTask(ctx context.Context, in *admin.DeleteTaskRequest, opts ...grpc.CallOption) (*admin.DeleteTaskResponse, error) { - _va := make([]interface{}, len(opts)) - for _i := range opts { - _va[_i] = opts[_i] - } - var _ca []interface{} - _ca = append(_ca, ctx, in) - _ca = append(_ca, _va...) - ret := _m.Called(_ca...) - - var r0 *admin.DeleteTaskResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.DeleteTaskRequest, ...grpc.CallOption) *admin.DeleteTaskResponse); ok { - r0 = rf(ctx, in, opts...) - } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.DeleteTaskResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.DeleteTaskRequest, ...grpc.CallOption) error); ok { - r1 = rf(ctx, in, opts...) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} - -type AsyncAgentServiceClient_GetTask struct { - *mock.Call -} - -func (_m AsyncAgentServiceClient_GetTask) Return(_a0 *admin.GetTaskResponse, _a1 error) *AsyncAgentServiceClient_GetTask { - return &AsyncAgentServiceClient_GetTask{Call: _m.Call.Return(_a0, _a1)} -} - -func (_m *AsyncAgentServiceClient) OnGetTask(ctx context.Context, in *admin.GetTaskRequest, opts ...grpc.CallOption) *AsyncAgentServiceClient_GetTask { - c_call := _m.On("GetTask", ctx, in, opts) - return &AsyncAgentServiceClient_GetTask{Call: c_call} -} - -func (_m *AsyncAgentServiceClient) OnGetTaskMatch(matchers ...interface{}) *AsyncAgentServiceClient_GetTask { - c_call := _m.On("GetTask", matchers...) - return &AsyncAgentServiceClient_GetTask{Call: c_call} -} - -// GetTask provides a mock function with given fields: ctx, in, opts -func (_m *AsyncAgentServiceClient) GetTask(ctx context.Context, in *admin.GetTaskRequest, opts ...grpc.CallOption) (*admin.GetTaskResponse, error) { - _va := make([]interface{}, len(opts)) - for _i := range opts { - _va[_i] = opts[_i] - } - var _ca []interface{} - _ca = append(_ca, ctx, in) - _ca = append(_ca, _va...) - ret := _m.Called(_ca...) - - var r0 *admin.GetTaskResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.GetTaskRequest, ...grpc.CallOption) *admin.GetTaskResponse); ok { - r0 = rf(ctx, in, opts...) - } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.GetTaskResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.GetTaskRequest, ...grpc.CallOption) error); ok { - r1 = rf(ctx, in, opts...) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} - -type AsyncAgentServiceClient_GetTaskLogs struct { - *mock.Call -} - -func (_m AsyncAgentServiceClient_GetTaskLogs) Return(_a0 *admin.GetTaskLogsResponse, _a1 error) *AsyncAgentServiceClient_GetTaskLogs { - return &AsyncAgentServiceClient_GetTaskLogs{Call: _m.Call.Return(_a0, _a1)} -} - -func (_m *AsyncAgentServiceClient) OnGetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) *AsyncAgentServiceClient_GetTaskLogs { - c_call := _m.On("GetTaskLogs", ctx, in, opts) - return &AsyncAgentServiceClient_GetTaskLogs{Call: c_call} -} - -func (_m *AsyncAgentServiceClient) OnGetTaskLogsMatch(matchers ...interface{}) *AsyncAgentServiceClient_GetTaskLogs { - c_call := _m.On("GetTaskLogs", matchers...) - return &AsyncAgentServiceClient_GetTaskLogs{Call: c_call} -} - -// GetTaskLogs provides a mock function with given fields: ctx, in, opts -func (_m *AsyncAgentServiceClient) GetTaskLogs(ctx context.Context, in *admin.GetTaskLogsRequest, opts ...grpc.CallOption) (*admin.GetTaskLogsResponse, error) { - _va := make([]interface{}, len(opts)) - for _i := range opts { - _va[_i] = opts[_i] - } - var _ca []interface{} - _ca = append(_ca, ctx, in) - _ca = append(_ca, _va...) - ret := _m.Called(_ca...) - - var r0 *admin.GetTaskLogsResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.GetTaskLogsRequest, ...grpc.CallOption) *admin.GetTaskLogsResponse); ok { - r0 = rf(ctx, in, opts...) - } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.GetTaskLogsResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.GetTaskLogsRequest, ...grpc.CallOption) error); ok { - r1 = rf(ctx, in, opts...) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} - -type AsyncAgentServiceClient_GetTaskMetrics struct { - *mock.Call -} - -func (_m AsyncAgentServiceClient_GetTaskMetrics) Return(_a0 *admin.GetTaskMetricsResponse, _a1 error) *AsyncAgentServiceClient_GetTaskMetrics { - return &AsyncAgentServiceClient_GetTaskMetrics{Call: _m.Call.Return(_a0, _a1)} -} - -func (_m *AsyncAgentServiceClient) OnGetTaskMetrics(ctx context.Context, in *admin.GetTaskMetricsRequest, opts ...grpc.CallOption) *AsyncAgentServiceClient_GetTaskMetrics { - c_call := _m.On("GetTaskMetrics", ctx, in, opts) - return &AsyncAgentServiceClient_GetTaskMetrics{Call: c_call} -} - -func (_m *AsyncAgentServiceClient) OnGetTaskMetricsMatch(matchers ...interface{}) *AsyncAgentServiceClient_GetTaskMetrics { - c_call := _m.On("GetTaskMetrics", matchers...) - return &AsyncAgentServiceClient_GetTaskMetrics{Call: c_call} -} - -// GetTaskMetrics provides a mock function with given fields: ctx, in, opts -func (_m *AsyncAgentServiceClient) GetTaskMetrics(ctx context.Context, in *admin.GetTaskMetricsRequest, opts ...grpc.CallOption) (*admin.GetTaskMetricsResponse, error) { - _va := make([]interface{}, len(opts)) - for _i := range opts { - _va[_i] = opts[_i] - } - var _ca []interface{} - _ca = append(_ca, ctx, in) - _ca = append(_ca, _va...) - ret := _m.Called(_ca...) - - var r0 *admin.GetTaskMetricsResponse - if rf, ok := ret.Get(0).(func(context.Context, *admin.GetTaskMetricsRequest, ...grpc.CallOption) *admin.GetTaskMetricsResponse); ok { - r0 = rf(ctx, in, opts...) - } else { - if ret.Get(0) != nil { - r0 = ret.Get(0).(*admin.GetTaskMetricsResponse) - } - } - - var r1 error - if rf, ok := ret.Get(1).(func(context.Context, *admin.GetTaskMetricsRequest, ...grpc.CallOption) error); ok { - r1 = rf(ctx, in, opts...) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go b/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go index 13115f89b4..11ef7871b3 100644 --- a/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go +++ b/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go @@ -4,6 +4,7 @@ import ( "context" "encoding/gob" "fmt" + "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service" "time" "golang.org/x/exp/maps" @@ -14,23 +15,26 @@ import ( "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery" "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/core" "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/core/template" - "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/io" + flyteIO "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/io" "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/ioutils" "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/webapi" "github.com/flyteorg/flyte/flytestdlib/logger" "github.com/flyteorg/flyte/flytestdlib/promutils" ) +type Registry map[string]map[int32]*Agent // map[taskTypeName][taskTypeVersion] => Agent + type Plugin struct { metricScope promutils.Scope cfg *Config cs *ClientSet - agentRegistry map[string]*Agent // map[taskType] => Agent + agentRegistry Registry } type ResourceWrapper struct { - Phase flyteIdl.TaskExecution_Phase - State admin.State // This is deprecated. + Phase flyteIdl.TaskExecution_Phase + // Deprecated: Please Use Phase instead. + State admin.State Outputs *flyteIdl.LiteralMap Message string LogLinks []*flyteIdl.TaskLog @@ -38,9 +42,8 @@ type ResourceWrapper struct { type ResourceMetaWrapper struct { OutputPrefix string - Token string AgentResourceMeta []byte - TaskType string + TaskCategory admin.TaskCategory } func (p Plugin) GetConfig() webapi.PluginConfig { @@ -79,62 +82,125 @@ func (p Plugin) Create(ctx context.Context, taskCtx webapi.TaskExecutionContextR return nil, nil, err } taskTemplate.GetContainer().Args = modifiedArgs + defer func() { + // Restore unrendered template for subsequent renders. + taskTemplate.GetContainer().Args = argTemplate + }() } outputPrefix := taskCtx.OutputWriter().GetOutputPrefixPath().String() - agent := getFinalAgent(taskTemplate.Type, p.cfg, p.agentRegistry) - - client := p.cs.agentClients[agent.Endpoint] - if client == nil { - return nil, nil, fmt.Errorf("default agent is not connected, please check if endpoint:[%v] is up and running", agent.Endpoint) - } + taskCategory := admin.TaskCategory{Name: taskTemplate.Type, Version: taskTemplate.TaskTypeVersion} + agent, isSync := getFinalAgent(&taskCategory, p.cfg, p.agentRegistry) finalCtx, cancel := getFinalContext(ctx, "CreateTask", agent) defer cancel() taskExecutionMetadata := buildTaskExecutionMetadata(taskCtx.TaskExecutionMetadata()) - res, err := client.CreateTask(finalCtx, &admin.CreateTaskRequest{Inputs: inputs, Template: taskTemplate, OutputPrefix: outputPrefix, TaskExecutionMetadata: &taskExecutionMetadata}) + + if isSync { + client, err := p.getSyncAgentClient(ctx, agent) + if err != nil { + return nil, nil, err + } + header := &admin.CreateRequestHeader{Template: taskTemplate, OutputPrefix: outputPrefix, TaskExecutionMetadata: &taskExecutionMetadata} + return p.ExecuteTaskSync(finalCtx, client, header, inputs) + } + + // Use async agent client + client, err := p.getAsyncAgentClient(ctx, agent) if err != nil { return nil, nil, err } - - // Restore unrendered template for subsequent renders. - if taskTemplate.GetContainer() != nil { - taskTemplate.GetContainer().Args = argTemplate - } - - // If the agent returned a resource, we assume this is a synchronous task. - // The state should be a terminal state, for example, SUCCEEDED, PERMANENT_FAILURE, or RETRYABLE_FAILURE. - if res.GetResource() != nil { - logger.Infof(ctx, "Agent is executing a synchronous task.") - return nil, - ResourceWrapper{ - Phase: res.GetResource().Phase, - State: res.GetResource().State, - Outputs: res.GetResource().Outputs, - Message: res.GetResource().Message, - LogLinks: res.GetResource().LogLinks, - }, nil + request := &admin.CreateTaskRequest{Inputs: inputs, Template: taskTemplate, OutputPrefix: outputPrefix, TaskExecutionMetadata: &taskExecutionMetadata} + res, err := client.CreateTask(finalCtx, request) + if err != nil { + return nil, nil, err } - logger.Infof(ctx, "Agent is executing an asynchronous task.") return ResourceMetaWrapper{ OutputPrefix: outputPrefix, AgentResourceMeta: res.GetResourceMeta(), - Token: "", - TaskType: taskTemplate.Type, + TaskCategory: taskCategory, }, nil, nil } +func (p Plugin) ExecuteTaskSync( + ctx context.Context, + client service.SyncAgentServiceClient, + header *admin.CreateRequestHeader, + inputs *flyteIdl.LiteralMap, +) (webapi.ResourceMeta, webapi.Resource, error) { + stream, err := client.ExecuteTaskSync(ctx) + if err != nil { + return nil, nil, err + } + + headerProto := &admin.ExecuteTaskSyncRequest{ + Part: &admin.ExecuteTaskSyncRequest_Header{ + Header: header, + }, + } + + err = stream.Send(headerProto) + if err != nil { + return nil, nil, fmt.Errorf("failed to send headerProto with error: %w", err) + } + inputsProto := &admin.ExecuteTaskSyncRequest{ + Part: &admin.ExecuteTaskSyncRequest_Inputs{ + Inputs: inputs, + }, + } + err = stream.Send(inputsProto) + + if err != nil { + return nil, nil, fmt.Errorf("failed to send inputsProto with error: %w", err) + } + + in, err := stream.Recv() + if err != nil { + return nil, nil, err + } + if in.GetHeader() == nil { + return nil, nil, fmt.Errorf("expected header in the response, but got none") + } + // TODO: Read the streaming output from the agent, and merge it into the final output. + // For now, Propeller assumes that the output is always in the header. + resource := in.GetHeader().GetResource() + + if err := stream.CloseSend(); err != nil { + return nil, nil, err + } + + if err != nil { + logger.Errorf(ctx, "Failed to write output with err %s", err.Error()) + return nil, nil, err + } + + return nil, ResourceWrapper{ + Phase: resource.Phase, + Outputs: resource.Outputs, + Message: resource.Message, + LogLinks: resource.LogLinks, + }, err +} + func (p Plugin) Get(ctx context.Context, taskCtx webapi.GetContext) (latest webapi.Resource, err error) { metadata := taskCtx.ResourceMeta().(ResourceMetaWrapper) - agent := getFinalAgent(metadata.TaskType, p.cfg, p.agentRegistry) + agent, _ := getFinalAgent(&metadata.TaskCategory, p.cfg, p.agentRegistry) - client := p.cs.agentClients[agent.Endpoint] + client, err := p.getAsyncAgentClient(ctx, agent) + if err != nil { + return nil, err + } finalCtx, cancel := getFinalContext(ctx, "GetTask", agent) defer cancel() - res, err := client.GetTask(finalCtx, &admin.GetTaskRequest{TaskType: metadata.TaskType, ResourceMeta: metadata.AgentResourceMeta}) + request := &admin.GetTaskRequest{ + TaskType: metadata.TaskCategory.Name, + TaskCategory: &metadata.TaskCategory, + ResourceMeta: metadata.AgentResourceMeta, + } + res, err := client.GetTask(finalCtx, request) if err != nil { return nil, err } @@ -144,7 +210,7 @@ func (p Plugin) Get(ctx context.Context, taskCtx webapi.GetContext) (latest weba State: res.Resource.State, Outputs: res.Resource.Outputs, Message: res.Resource.Message, - LogLinks: res.LogLinks, + LogLinks: res.Resource.LogLinks, }, nil } @@ -153,13 +219,21 @@ func (p Plugin) Delete(ctx context.Context, taskCtx webapi.DeleteContext) error return nil } metadata := taskCtx.ResourceMeta().(ResourceMetaWrapper) - agent := getFinalAgent(metadata.TaskType, p.cfg, p.agentRegistry) + agent, _ := getFinalAgent(&metadata.TaskCategory, p.cfg, p.agentRegistry) - client := p.cs.agentClients[agent.Endpoint] + client, err := p.getAsyncAgentClient(ctx, agent) + if err != nil { + return err + } finalCtx, cancel := getFinalContext(ctx, "DeleteTask", agent) defer cancel() - _, err := client.DeleteTask(finalCtx, &admin.DeleteTaskRequest{TaskType: metadata.TaskType, ResourceMeta: metadata.AgentResourceMeta}) + request := &admin.DeleteTaskRequest{ + TaskType: metadata.TaskCategory.Name, + TaskCategory: &metadata.TaskCategory, + ResourceMeta: metadata.AgentResourceMeta, + } + _, err = client.DeleteTask(finalCtx, request) return err } @@ -177,16 +251,16 @@ func (p Plugin) Status(ctx context.Context, taskCtx webapi.StatusContext) (phase case flyteIdl.TaskExecution_RUNNING: return core.PhaseInfoRunning(core.DefaultPhaseVersion, taskInfo), nil case flyteIdl.TaskExecution_SUCCEEDED: - err = writeOutput(ctx, taskCtx, resource) + err = writeOutput(ctx, taskCtx, resource.Outputs) if err != nil { logger.Errorf(ctx, "Failed to write output with err %s", err.Error()) return core.PhaseInfoUndefined, err } return core.PhaseInfoSuccess(taskInfo), nil case flyteIdl.TaskExecution_ABORTED: - return core.PhaseInfoFailure(pluginErrors.TaskFailedWithError, "failed to run the job with aborted phase", taskInfo), nil + return core.PhaseInfoFailure(pluginErrors.TaskFailedWithError, "failed to run the job with aborted phase.\n"+resource.Message, taskInfo), nil case flyteIdl.TaskExecution_FAILED: - return core.PhaseInfoFailure(pluginErrors.TaskFailedWithError, "failed to run the job", taskInfo), nil + return core.PhaseInfoFailure(pluginErrors.TaskFailedWithError, "failed to run the job.\n"+resource.Message, taskInfo), nil } // The default phase is undefined. @@ -201,11 +275,11 @@ func (p Plugin) Status(ctx context.Context, taskCtx webapi.StatusContext) (phase case admin.State_RUNNING: return core.PhaseInfoRunning(core.DefaultPhaseVersion, taskInfo), nil case admin.State_PERMANENT_FAILURE: - return core.PhaseInfoFailure(pluginErrors.TaskFailedWithError, "failed to run the job", taskInfo), nil + return core.PhaseInfoFailure(pluginErrors.TaskFailedWithError, "failed to run the job.\n"+resource.Message, taskInfo), nil case admin.State_RETRYABLE_FAILURE: - return core.PhaseInfoRetryableFailure(pluginErrors.TaskFailedWithError, "failed to run the job", taskInfo), nil + return core.PhaseInfoRetryableFailure(pluginErrors.TaskFailedWithError, "failed to run the job.\n"+resource.Message, taskInfo), nil case admin.State_SUCCEEDED: - err = writeOutput(ctx, taskCtx, resource) + err = writeOutput(ctx, taskCtx, resource.Outputs) if err != nil { logger.Errorf(ctx, "Failed to write output with err %s", err.Error()) return core.PhaseInfoUndefined, err @@ -215,7 +289,33 @@ func (p Plugin) Status(ctx context.Context, taskCtx webapi.StatusContext) (phase return core.PhaseInfoUndefined, pluginErrors.Errorf(core.SystemErrorCode, "unknown execution state [%v].", resource.State) } -func writeOutput(ctx context.Context, taskCtx webapi.StatusContext, resource ResourceWrapper) error { +func (p Plugin) getSyncAgentClient(ctx context.Context, agent *Deployment) (service.SyncAgentServiceClient, error) { + client, ok := p.cs.syncAgentClients[agent.Endpoint] + if !ok { + conn, err := getGrpcConnection(ctx, agent) + if err != nil { + return nil, fmt.Errorf("failed to get grpc connection with error: %v", err) + } + client = service.NewSyncAgentServiceClient(conn) + p.cs.syncAgentClients[agent.Endpoint] = client + } + return client, nil +} + +func (p Plugin) getAsyncAgentClient(ctx context.Context, agent *Deployment) (service.AsyncAgentServiceClient, error) { + client, ok := p.cs.asyncAgentClients[agent.Endpoint] + if !ok { + conn, err := getGrpcConnection(ctx, agent) + if err != nil { + return nil, fmt.Errorf("failed to get grpc connection with error: %v", err) + } + client = service.NewAsyncAgentServiceClient(conn) + p.cs.asyncAgentClients[agent.Endpoint] = client + } + return client, nil +} + +func writeOutput(ctx context.Context, taskCtx webapi.StatusContext, outputs *flyteIdl.LiteralMap) error { taskTemplate, err := taskCtx.TaskReader().Read(ctx) if err != nil { return err @@ -226,23 +326,23 @@ func writeOutput(ctx context.Context, taskCtx webapi.StatusContext, resource Res return nil } - var opReader io.OutputReader - if resource.Outputs != nil { - logger.Debugf(ctx, "Agent returned an output.") - opReader = ioutils.NewInMemoryOutputReader(resource.Outputs, nil, nil) + var opReader flyteIO.OutputReader + if outputs != nil { + logger.Debugf(ctx, "AgentDeployment returned an output.") + opReader = ioutils.NewInMemoryOutputReader(outputs, nil, nil) } else { - logger.Debugf(ctx, "Agent didn't return any output, assuming file based outputs.") + logger.Debugf(ctx, "AgentDeployment didn't return any output, assuming file based outputs.") opReader = ioutils.NewRemoteFileOutputReader(ctx, taskCtx.DataStore(), taskCtx.OutputWriter(), taskCtx.MaxDatasetSizeBytes()) } return taskCtx.OutputWriter().Put(ctx, opReader) } -func getFinalAgent(taskType string, cfg *Config, agentRegistry map[string]*Agent) *Agent { - if agent, exists := agentRegistry[taskType]; exists { - return agent +func getFinalAgent(taskCategory *admin.TaskCategory, cfg *Config, agentRegistry Registry) (*Deployment, bool) { + if agent, exists := agentRegistry[taskCategory.Name][taskCategory.Version]; exists { + return agent.AgentDeployment, agent.IsSync } - return &cfg.DefaultAgent + return &cfg.DefaultAgent, false } func buildTaskExecutionMetadata(taskExecutionMetadata core.TaskExecutionMetadata) admin.TaskExecutionMetadata { @@ -271,7 +371,7 @@ func newAgentPlugin() webapi.PluginEntry { cfg := GetConfig() supportedTaskTypes := append(maps.Keys(agentRegistry), cfg.SupportedTaskTypes...) - logger.Infof(context.Background(), "Agent supports task types: %v", supportedTaskTypes) + logger.Infof(context.Background(), "AgentDeployment service supports task types: %v", supportedTaskTypes) return webapi.PluginEntry{ ID: "agent-service", diff --git a/flyteplugins/go/tasks/plugins/webapi/agent/plugin_test.go b/flyteplugins/go/tasks/plugins/webapi/agent/plugin_test.go index bddea6869e..9fa36c5c42 100644 --- a/flyteplugins/go/tasks/plugins/webapi/agent/plugin_test.go +++ b/flyteplugins/go/tasks/plugins/webapi/agent/plugin_test.go @@ -10,6 +10,7 @@ import ( "github.com/stretchr/testify/mock" "golang.org/x/exp/maps" + agentMocks "github.com/flyteorg/flyte/flyteidl/clients/go/admin/mocks" "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" flyteIdl "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core" flyteIdlCore "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core" @@ -17,7 +18,6 @@ import ( pluginsCore "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/core" pluginCoreMocks "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/core/mocks" webapiPlugin "github.com/flyteorg/flyte/flyteplugins/go/tasks/pluginmachinery/webapi/mocks" - agentMocks "github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent/mocks" "github.com/flyteorg/flyte/flytestdlib/config" "github.com/flyteorg/flyte/flytestdlib/promutils" ) @@ -31,8 +31,8 @@ func TestPlugin(t *testing.T) { cfg := defaultConfig cfg.WebAPI.Caching.Workers = 1 cfg.WebAPI.Caching.ResyncInterval.Duration = 5 * time.Second - cfg.DefaultAgent = Agent{Endpoint: "test-agent.flyte.svc.cluster.local:80"} - cfg.Agents = map[string]*Agent{"spark_agent": {Endpoint: "localhost:80"}} + cfg.DefaultAgent = Deployment{Endpoint: "test-agent.flyte.svc.cluster.local:80"} + cfg.AgentDeployments = map[string]*Deployment{"spark_agent": {Endpoint: "localhost:80"}} cfg.AgentForTaskTypes = map[string]string{"spark": "spark_agent", "bar": "bar_agent"} plugin := Plugin{ @@ -52,34 +52,38 @@ func TestPlugin(t *testing.T) { }) t.Run("test newAgentPlugin", func(t *testing.T) { - p := newMockAgentPlugin() + p := newMockAsyncAgentPlugin() assert.NotNil(t, p) assert.Equal(t, "agent-service", p.ID) assert.NotNil(t, p.PluginLoader) }) t.Run("test getFinalAgent", func(t *testing.T) { - agentRegistry := map[string]*Agent{"spark": {Endpoint: "localhost:80"}} - agent := getFinalAgent("spark", &cfg, agentRegistry) - assert.Equal(t, agent.Endpoint, "localhost:80") - agent = getFinalAgent("foo", &cfg, agentRegistry) - assert.Equal(t, agent.Endpoint, cfg.DefaultAgent.Endpoint) - agent = getFinalAgent("bar", &cfg, agentRegistry) - assert.Equal(t, agent.Endpoint, cfg.DefaultAgent.Endpoint) + agent := &Agent{AgentDeployment: &Deployment{Endpoint: "localhost:80"}} + agentRegistry := Registry{"spark": {defaultTaskTypeVersion: agent}} + spark := &admin.TaskCategory{Name: "spark", Version: defaultTaskTypeVersion} + foo := &admin.TaskCategory{Name: "foo", Version: defaultTaskTypeVersion} + bar := &admin.TaskCategory{Name: "bar", Version: defaultTaskTypeVersion} + agentDeployment, _ := getFinalAgent(spark, &cfg, agentRegistry) + assert.Equal(t, agentDeployment.Endpoint, "localhost:80") + agentDeployment, _ = getFinalAgent(foo, &cfg, agentRegistry) + assert.Equal(t, agentDeployment.Endpoint, cfg.DefaultAgent.Endpoint) + agentDeployment, _ = getFinalAgent(bar, &cfg, agentRegistry) + assert.Equal(t, agentDeployment.Endpoint, cfg.DefaultAgent.Endpoint) }) t.Run("test getFinalTimeout", func(t *testing.T) { - timeout := getFinalTimeout("CreateTask", &Agent{Endpoint: "localhost:8080", Timeouts: map[string]config.Duration{"CreateTask": {Duration: 1 * time.Millisecond}}}) + timeout := getFinalTimeout("CreateTask", &Deployment{Endpoint: "localhost:8080", Timeouts: map[string]config.Duration{"CreateTask": {Duration: 1 * time.Millisecond}}}) assert.Equal(t, 1*time.Millisecond, timeout.Duration) - timeout = getFinalTimeout("DeleteTask", &Agent{Endpoint: "localhost:8080", DefaultTimeout: config.Duration{Duration: 10 * time.Second}}) + timeout = getFinalTimeout("DeleteTask", &Deployment{Endpoint: "localhost:8080", DefaultTimeout: config.Duration{Duration: 10 * time.Second}}) assert.Equal(t, 10*time.Second, timeout.Duration) }) t.Run("test getFinalContext", func(t *testing.T) { - ctx, _ := getFinalContext(context.TODO(), "DeleteTask", &Agent{}) + ctx, _ := getFinalContext(context.TODO(), "DeleteTask", &Deployment{}) assert.Equal(t, context.TODO(), ctx) - ctx, _ = getFinalContext(context.TODO(), "CreateTask", &Agent{Endpoint: "localhost:8080", Timeouts: map[string]config.Duration{"CreateTask": {Duration: 1 * time.Millisecond}}}) + ctx, _ = getFinalContext(context.TODO(), "CreateTask", &Deployment{Endpoint: "localhost:8080", Timeouts: map[string]config.Duration{"CreateTask": {Duration: 1 * time.Millisecond}}}) assert.NotEqual(t, context.TODO(), ctx) }) @@ -270,11 +274,15 @@ func TestPlugin(t *testing.T) { func getMockMetadataServiceClient() *agentMocks.AgentMetadataServiceClient { mockMetadataServiceClient := new(agentMocks.AgentMetadataServiceClient) mockRequest := &admin.ListAgentsRequest{} + supportedTaskCategories := make([]*admin.TaskCategory, 3) + supportedTaskCategories[0] = &admin.TaskCategory{Name: "task1", Version: defaultTaskTypeVersion} + supportedTaskCategories[1] = &admin.TaskCategory{Name: "task2", Version: defaultTaskTypeVersion} + supportedTaskCategories[2] = &admin.TaskCategory{Name: "task3", Version: defaultTaskTypeVersion} mockResponse := &admin.ListAgentsResponse{ Agents: []*admin.Agent{ { - Name: "test-agent", - SupportedTaskTypes: []string{"task1", "task2", "task3"}, + Name: "test-agent", + SupportedTaskCategories: supportedTaskCategories, }, }, } @@ -290,12 +298,12 @@ func TestInitializeAgentRegistry(t *testing.T) { agentMetadataClients[defaultAgentEndpoint] = getMockMetadataServiceClient() cs := &ClientSet{ - agentClients: agentClients, + asyncAgentClients: agentClients, agentMetadataClients: agentMetadataClients, } cfg := defaultConfig - cfg.Agents = map[string]*Agent{"custom_agent": {Endpoint: defaultAgentEndpoint}} + cfg.AgentDeployments = map[string]*Deployment{"custom_agent": {Endpoint: defaultAgentEndpoint}} cfg.AgentForTaskTypes = map[string]string{"task1": "agent-deployment-1", "task2": "agent-deployment-2"} err := SetConfig(&cfg) assert.NoError(t, err) diff --git a/flyteplugins/go/tasks/plugins/webapi/athena/utils_test.go b/flyteplugins/go/tasks/plugins/webapi/athena/utils_test.go index da490838fe..eabbfaf1fa 100644 --- a/flyteplugins/go/tasks/plugins/webapi/athena/utils_test.go +++ b/flyteplugins/go/tasks/plugins/webapi/athena/utils_test.go @@ -177,6 +177,7 @@ func Test_ExtractQueryInfo(t *testing.T) { tID := &mocks2.TaskExecutionID{} tMeta.OnGetTaskExecutionID().Return(tID) + tMeta.OnGetNamespace().Return("my-namespace") tID.OnGetGeneratedName().Return("generated-name") diff --git a/flyteplugins/tests/end_to_end.go b/flyteplugins/tests/end_to_end.go index 037ae877d9..1a55b52c0c 100644 --- a/flyteplugins/tests/end_to_end.go +++ b/flyteplugins/tests/end_to_end.go @@ -147,6 +147,7 @@ func RunPluginEndToEndTest(t *testing.T, executor pluginCore.Plugin, template *i Limits: map[v1.ResourceName]resource.Quantity{}, }) overrides.OnGetExtendedResources().Return(&idlCore.ExtendedResources{}) + overrides.OnGetContainerImage().Return("") tMeta := &coreMocks.TaskExecutionMetadata{} tMeta.OnGetTaskExecutionID().Return(tID) diff --git a/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/iface.go b/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/iface.go index 6590aaa04a..664ebb6767 100644 --- a/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/iface.go +++ b/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/iface.go @@ -442,6 +442,7 @@ type ExecutableNode interface { GetActiveDeadline() *time.Duration IsInterruptible() *bool GetName() string + GetContainerImage() string } // ExecutableWorkflowStatus is an interface for the Workflow p. This is the mutable portion for a Workflow diff --git a/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/mocks/ExecutableNode.go b/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/mocks/ExecutableNode.go index ae1663bba1..e6a03b7deb 100644 --- a/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/mocks/ExecutableNode.go +++ b/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/mocks/ExecutableNode.go @@ -154,6 +154,38 @@ func (_m *ExecutableNode) GetConfig() *v1.ConfigMap { return r0 } +type ExecutableNode_GetContainerImage struct { + *mock.Call +} + +func (_m ExecutableNode_GetContainerImage) Return(_a0 string) *ExecutableNode_GetContainerImage { + return &ExecutableNode_GetContainerImage{Call: _m.Call.Return(_a0)} +} + +func (_m *ExecutableNode) OnGetContainerImage() *ExecutableNode_GetContainerImage { + c_call := _m.On("GetContainerImage") + return &ExecutableNode_GetContainerImage{Call: c_call} +} + +func (_m *ExecutableNode) OnGetContainerImageMatch(matchers ...interface{}) *ExecutableNode_GetContainerImage { + c_call := _m.On("GetContainerImage", matchers...) + return &ExecutableNode_GetContainerImage{Call: c_call} +} + +// GetContainerImage provides a mock function with given fields: +func (_m *ExecutableNode) GetContainerImage() string { + ret := _m.Called() + + var r0 string + if rf, ok := ret.Get(0).(func() string); ok { + r0 = rf() + } else { + r0 = ret.Get(0).(string) + } + + return r0 +} + type ExecutableNode_GetExecutionDeadline struct { *mock.Call } diff --git a/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/nodes.go b/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/nodes.go index 8a22cae974..6554357031 100644 --- a/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/nodes.go +++ b/flytepropeller/pkg/apis/flyteworkflow/v1alpha1/nodes.go @@ -175,6 +175,8 @@ type NodeSpec struct { // The value set to True means task is OK with getting interrupted // +optional Interruptible *bool `json:"interruptible,omitempty"` + + ContainerImage string `json:"containerImage,omitempty"` } func (in *NodeSpec) GetName() string { @@ -273,3 +275,7 @@ func (in *NodeSpec) IsEndNode() bool { func (in *NodeSpec) GetInputBindings() []*Binding { return in.InputBindings } + +func (in *NodeSpec) GetContainerImage() string { + return in.ContainerImage +} diff --git a/flytepropeller/pkg/compiler/transformers/k8s/node.go b/flytepropeller/pkg/compiler/transformers/k8s/node.go index 7120f3ad90..7b5df9b3b5 100644 --- a/flytepropeller/pkg/compiler/transformers/k8s/node.go +++ b/flytepropeller/pkg/compiler/transformers/k8s/node.go @@ -30,6 +30,7 @@ func buildNodeSpec(n *core.Node, tasks []*core.CompiledTask, errs errors.Compile var task *core.TaskTemplate var resources *core.Resources var extendedResources *v1alpha1.ExtendedResources + var containerImage string if n.GetTaskNode() != nil { taskID := n.GetTaskNode().GetReferenceId().String() // TODO: Use task index for quick lookup @@ -55,6 +56,10 @@ func buildNodeSpec(n *core.Node, tasks []*core.CompiledTask, errs errors.Compile ExtendedResources: overrides.GetExtendedResources(), } } + + if len(overrides.GetContainerImage()) > 0 { + containerImage = overrides.GetContainerImage() + } } } @@ -96,6 +101,7 @@ func buildNodeSpec(n *core.Node, tasks []*core.CompiledTask, errs errors.Compile InputBindings: toBindingValueArray(n.GetInputs()), ActiveDeadline: activeDeadline, Interruptible: interruptible, + ContainerImage: containerImage, } switch v := n.GetTarget().(type) { diff --git a/flytepropeller/pkg/compiler/transformers/k8s/node_test.go b/flytepropeller/pkg/compiler/transformers/k8s/node_test.go index f2d6c79494..ff1f263b6e 100644 --- a/flytepropeller/pkg/compiler/transformers/k8s/node_test.go +++ b/flytepropeller/pkg/compiler/transformers/k8s/node_test.go @@ -140,6 +140,24 @@ func TestBuildNodeSpec(t *testing.T) { assert.Equal(t, expectedGpuDevice, spec.GetExtendedResources().GetGpuAccelerator().GetDevice()) }) + t.Run("node with container image override", func(t *testing.T) { + expectedContainerImage := "test-image:latest" + n.Node.Target = &core.Node_TaskNode{ + TaskNode: &core.TaskNode{ + Reference: &core.TaskNode_ReferenceId{ + ReferenceId: &core.Identifier{Name: "ref_2"}, + }, + Overrides: &core.TaskNodeOverrides{ + ContainerImage: expectedContainerImage, + }, + }, + } + + spec := mustBuild(t, n, 1, errs.NewScope()) + assert.NotNil(t, spec.GetContainerImage()) + assert.Equal(t, expectedContainerImage, spec.GetContainerImage()) + }) + t.Run("LaunchPlanRef", func(t *testing.T) { n.Node.Target = &core.Node_WorkflowNode{ WorkflowNode: &core.WorkflowNode{ diff --git a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go index b3db72d129..6a029255a8 100644 --- a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go +++ b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go @@ -147,8 +147,8 @@ func (m *CatalogClient) Get(ctx context.Context, key catalog.Key) (catalog.Entry return catalog.NewCatalogEntry(ioutils.NewInMemoryOutputReader(outputs, nil, nil), catalog.NewStatus(core.CatalogCacheStatus_CACHE_HIT, md)), nil } -// CreateDataset creates a Dataset in datacatalog including the associated metadata. -func (m *CatalogClient) CreateDataset(ctx context.Context, key catalog.Key, metadata *datacatalog.Metadata) (*datacatalog.DatasetID, error) { +// createDataset creates a Dataset in datacatalog including the associated metadata. +func (m *CatalogClient) createDataset(ctx context.Context, key catalog.Key, metadata *datacatalog.Metadata) (*datacatalog.DatasetID, error) { datasetID, err := GenerateDatasetIDForTask(ctx, key) if err != nil { logger.Errorf(ctx, "DataCatalog failed to generate dataset for ID: %s, err: %s", key.Identifier, err) @@ -205,9 +205,9 @@ func (m *CatalogClient) prepareInputsAndOutputs(ctx context.Context, key catalog return inputs, outputs, nil } -// CreateArtifact creates an Artifact in datacatalog including its associated ArtifactData and tags it with a hash of +// createArtifact creates an Artifact in datacatalog including its associated ArtifactData and tags it with a hash of // the provided input values for retrieval. -func (m *CatalogClient) CreateArtifact(ctx context.Context, key catalog.Key, datasetID *datacatalog.DatasetID, inputs *core.LiteralMap, outputs *core.LiteralMap, metadata catalog.Metadata) (catalog.Status, error) { +func (m *CatalogClient) createArtifact(ctx context.Context, key catalog.Key, datasetID *datacatalog.DatasetID, inputs *core.LiteralMap, outputs *core.LiteralMap, metadata catalog.Metadata) (catalog.Status, error) { logger.Debugf(ctx, "Creating artifact for key %+v, dataset %+v and execution %+v", key, datasetID, metadata) // Create the artifact for the execution that belongs in the task @@ -263,8 +263,8 @@ func (m *CatalogClient) CreateArtifact(ctx context.Context, key catalog.Key, dat return catalog.NewStatus(core.CatalogCacheStatus_CACHE_POPULATED, EventCatalogMetadata(datasetID, tag, nil)), nil } -// UpdateArtifact overwrites the ArtifactData of an existing artifact with the provided data in datacatalog. -func (m *CatalogClient) UpdateArtifact(ctx context.Context, key catalog.Key, datasetID *datacatalog.DatasetID, inputs *core.LiteralMap, outputs *core.LiteralMap, metadata catalog.Metadata) (catalog.Status, error) { +// updateArtifact overwrites the ArtifactData of an existing artifact with the provided data in datacatalog. +func (m *CatalogClient) updateArtifact(ctx context.Context, key catalog.Key, datasetID *datacatalog.DatasetID, inputs *core.LiteralMap, outputs *core.LiteralMap, metadata catalog.Metadata) (catalog.Status, error) { logger.Debugf(ctx, "Updating artifact for key %+v, dataset %+v and execution %+v", key, datasetID, metadata) artifactDataList := make([]*datacatalog.ArtifactData, 0, len(outputs.Literals)) @@ -316,17 +316,21 @@ func (m *CatalogClient) UpdateArtifact(ctx context.Context, key catalog.Key, dat // Lastly, CatalogClient will create an Artifact tagged with the input value hash and store the provided execution data. func (m *CatalogClient) Put(ctx context.Context, key catalog.Key, reader io.OutputReader, metadata catalog.Metadata) (catalog.Status, error) { // Ensure dataset exists, idempotent operations. Populate Metadata for later recovery - datasetID, err := m.CreateDataset(ctx, key, GetDatasetMetadataForSource(metadata.TaskExecutionIdentifier)) + datasetID, err := m.createDataset(ctx, key, GetDatasetMetadataForSource(metadata.TaskExecutionIdentifier)) if err != nil { - return catalog.Status{}, err + return catalog.NewPutFailureStatus(&key), err } inputs, outputs, err := m.prepareInputsAndOutputs(ctx, key, reader) if err != nil { - return catalog.Status{}, err + return catalog.NewPutFailureStatus(&key), err } - return m.CreateArtifact(ctx, key, datasetID, inputs, outputs, metadata) + createArtifactStatus, err := m.createArtifact(ctx, key, datasetID, inputs, outputs, metadata) + if err != nil { + return catalog.NewPutFailureStatus(&key), err + } + return createArtifactStatus, err } // Update stores the result of a task execution as a cached Artifact, overwriting any already stored data from a previous @@ -337,27 +341,31 @@ func (m *CatalogClient) Put(ctx context.Context, key catalog.Key, reader io.Outp // has of the input values will exist. func (m *CatalogClient) Update(ctx context.Context, key catalog.Key, reader io.OutputReader, metadata catalog.Metadata) (catalog.Status, error) { // Ensure dataset exists, idempotent operations. Populate Metadata for later recovery - datasetID, err := m.CreateDataset(ctx, key, GetDatasetMetadataForSource(metadata.TaskExecutionIdentifier)) + datasetID, err := m.createDataset(ctx, key, GetDatasetMetadataForSource(metadata.TaskExecutionIdentifier)) if err != nil { - return catalog.Status{}, err + return catalog.NewPutFailureStatus(&key), err } inputs, outputs, err := m.prepareInputsAndOutputs(ctx, key, reader) if err != nil { - return catalog.Status{}, err + return catalog.NewPutFailureStatus(&key), err } - catalogStatus, err := m.UpdateArtifact(ctx, key, datasetID, inputs, outputs, metadata) + catalogStatus, err := m.updateArtifact(ctx, key, datasetID, inputs, outputs, metadata) if err != nil { if status.Code(err) == codes.NotFound { // No existing artifact found (e.g. initial execution of task with overwrite flag already set), // silently ignore error and create artifact instead to make overwriting an idempotent operation. logger.Debugf(ctx, "Artifact %+v for dataset %+v does not exist while updating, creating instead", key, datasetID) - return m.CreateArtifact(ctx, key, datasetID, inputs, outputs, metadata) + createArtifactStatus, err := m.createArtifact(ctx, key, datasetID, inputs, outputs, metadata) + if err != nil { + return catalog.NewPutFailureStatus(&key), err + } + return createArtifactStatus, nil } logger.Errorf(ctx, "Failed to update artifact %+v for dataset %+v: %v", key, datasetID, err) - return catalog.Status{}, err + return catalog.NewPutFailureStatus(&key), err } logger.Debugf(ctx, "Successfully updated artifact %+v for dataset %+v", key, datasetID) diff --git a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog_test.go b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog_test.go index d9787ceeaa..ce8f6f4069 100644 --- a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog_test.go +++ b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog_test.go @@ -443,6 +443,73 @@ func TestCatalog_Put(t *testing.T) { assert.Equal(t, "flyte_cached-BE6CZsMk6N3ExR_4X9EuwBgj2Jh2UwasXK3a_pM9xlY", s.GetMetadata().ArtifactTag.Name) }) + t.Run("Create dataset fails", func(t *testing.T) { + ir := &mocks2.InputReader{} + ir.On("Get", mock.Anything).Return(sampleParameters, nil, nil) + + mockClient := &mocks.DataCatalogClient{} + discovery := &CatalogClient{ + client: mockClient, + } + + mockClient.On("CreateDataset", + ctx, + mock.MatchedBy(func(o *datacatalog.CreateDatasetRequest) bool { + return true + }), + ).Return(&datacatalog.CreateDatasetResponse{}, errors.New("generic error")) + + newKey := sampleKey + newKey.InputReader = ir + or := ioutils.NewInMemoryOutputReader(sampleParameters, nil, nil) + s, err := discovery.Put(ctx, newKey, or, catalog.Metadata{ + WorkflowExecutionIdentifier: &core.WorkflowExecutionIdentifier{ + Name: "test", + }, + TaskExecutionIdentifier: nil, + }) + assert.Error(t, err) + assert.Equal(t, core.CatalogCacheStatus_CACHE_PUT_FAILURE, s.GetCacheStatus()) + assert.NotNil(t, s.GetMetadata()) + }) + + t.Run("Create artifact fails", func(t *testing.T) { + ir := &mocks2.InputReader{} + ir.On("Get", mock.Anything).Return(sampleParameters, nil, nil) + + mockClient := &mocks.DataCatalogClient{} + discovery := &CatalogClient{ + client: mockClient, + } + + mockClient.On("CreateDataset", + ctx, + mock.MatchedBy(func(o *datacatalog.CreateDatasetRequest) bool { + return true + }), + ).Return(&datacatalog.CreateDatasetResponse{}, nil) + + mockClient.On("CreateArtifact", + ctx, + mock.MatchedBy(func(o *datacatalog.CreateArtifactRequest) bool { + return true + }), + ).Return(&datacatalog.CreateArtifactResponse{}, errors.New("generic error")) + + newKey := sampleKey + newKey.InputReader = ir + or := ioutils.NewInMemoryOutputReader(sampleParameters, nil, nil) + s, err := discovery.Put(ctx, newKey, or, catalog.Metadata{ + WorkflowExecutionIdentifier: &core.WorkflowExecutionIdentifier{ + Name: "test", + }, + TaskExecutionIdentifier: nil, + }) + assert.Error(t, err) + assert.Equal(t, core.CatalogCacheStatus_CACHE_PUT_FAILURE, s.GetCacheStatus()) + assert.NotNil(t, s.GetMetadata()) + }) + t.Run("Create new cached execution with no inputs/outputs", func(t *testing.T) { mockClient := &mocks.DataCatalogClient{} catalogClient := &CatalogClient{ @@ -611,6 +678,70 @@ func TestCatalog_Update(t *testing.T) { assert.Equal(t, taskID.NodeExecutionId.String(), sourceTID.NodeExecutionId.String()) }) + t.Run("Overwrite non-existing execution", func(t *testing.T) { + ir := &mocks2.InputReader{} + ir.On("Get", mock.Anything).Return(sampleParameters, nil, nil) + + mockClient := &mocks.DataCatalogClient{} + discovery := &CatalogClient{ + client: mockClient, + } + + mockClient.On("CreateDataset", + ctx, + mock.MatchedBy(func(o *datacatalog.CreateDatasetRequest) bool { + return true + }), + ).Return(&datacatalog.CreateDatasetResponse{}, nil) + + mockClient.On("UpdateArtifact", ctx, mock.Anything).Return(nil, status.New(codes.NotFound, "missing entity of type Artifact with identifier id").Err()) + + mockClient.On("CreateArtifact", + ctx, + mock.MatchedBy(func(o *datacatalog.CreateArtifactRequest) bool { + return true + }), + ).Return(&datacatalog.CreateArtifactResponse{}, errors.New("generic error")) + + taskID := &core.TaskExecutionIdentifier{ + TaskId: &core.Identifier{ + ResourceType: core.ResourceType_TASK, + Name: sampleKey.Identifier.Name, + Project: sampleKey.Identifier.Project, + Domain: sampleKey.Identifier.Domain, + Version: "version", + }, + NodeExecutionId: &core.NodeExecutionIdentifier{ + ExecutionId: &core.WorkflowExecutionIdentifier{ + Name: "wf", + Project: "p1", + Domain: "d1", + }, + NodeId: "unknown", // not set in Put request below --> defaults to "unknown" + }, + RetryAttempt: 0, + } + + newKey := sampleKey + newKey.InputReader = ir + or := ioutils.NewInMemoryOutputReader(sampleParameters, nil, nil) + s, err := discovery.Update(ctx, newKey, or, catalog.Metadata{ + WorkflowExecutionIdentifier: &core.WorkflowExecutionIdentifier{ + Name: taskID.NodeExecutionId.ExecutionId.Name, + Domain: taskID.NodeExecutionId.ExecutionId.Domain, + Project: taskID.NodeExecutionId.ExecutionId.Project, + }, + TaskExecutionIdentifier: &core.TaskExecutionIdentifier{ + TaskId: &sampleKey.Identifier, + NodeExecutionId: taskID.NodeExecutionId, + RetryAttempt: 0, + }, + }) + assert.Error(t, err) + assert.Equal(t, core.CatalogCacheStatus_CACHE_PUT_FAILURE, s.GetCacheStatus()) + assert.NotNil(t, s.GetMetadata()) + }) + t.Run("Overwrite non-existing execution", func(t *testing.T) { ir := &mocks2.InputReader{} ir.On("Get", mock.Anything).Return(sampleParameters, nil, nil) @@ -702,6 +833,36 @@ func TestCatalog_Update(t *testing.T) { assert.True(t, addTagCalled) }) + t.Run("Error while creating dataset", func(t *testing.T) { + ir := &mocks2.InputReader{} + ir.On("Get", mock.Anything).Return(sampleParameters, nil, nil) + + mockClient := &mocks.DataCatalogClient{} + discovery := &CatalogClient{ + client: mockClient, + } + + mockClient.On("CreateDataset", + ctx, + mock.MatchedBy(func(o *datacatalog.CreateDatasetRequest) bool { + return true + }), + ).Return(&datacatalog.CreateDatasetResponse{}, errors.New("generic error")) + + newKey := sampleKey + newKey.InputReader = ir + or := ioutils.NewInMemoryOutputReader(sampleParameters, nil, nil) + s, err := discovery.Update(ctx, newKey, or, catalog.Metadata{ + WorkflowExecutionIdentifier: &core.WorkflowExecutionIdentifier{ + Name: "test", + }, + TaskExecutionIdentifier: nil, + }) + assert.Error(t, err) + assert.Equal(t, core.CatalogCacheStatus_CACHE_PUT_FAILURE, s.GetCacheStatus()) + assert.NotNil(t, s.GetMetadata()) + }) + t.Run("Error while overwriting execution", func(t *testing.T) { ir := &mocks2.InputReader{} ir.On("Get", mock.Anything).Return(sampleParameters, nil, nil) @@ -714,7 +875,6 @@ func TestCatalog_Update(t *testing.T) { mockClient.On("CreateDataset", ctx, mock.MatchedBy(func(o *datacatalog.CreateDatasetRequest) bool { - assert.True(t, proto.Equal(o.Dataset.Id, datasetID)) return true }), ).Return(&datacatalog.CreateDatasetResponse{}, nil) @@ -733,8 +893,8 @@ func TestCatalog_Update(t *testing.T) { }) assert.Error(t, err) assert.Equal(t, genericErr, err) - assert.Equal(t, core.CatalogCacheStatus_CACHE_DISABLED, s.GetCacheStatus()) - assert.Nil(t, s.GetMetadata()) + assert.Equal(t, core.CatalogCacheStatus_CACHE_PUT_FAILURE, s.GetCacheStatus()) + assert.NotNil(t, s.GetMetadata()) }) } diff --git a/flytepropeller/pkg/controller/nodes/transformers.go b/flytepropeller/pkg/controller/nodes/transformers.go index eb96a38d9c..b034c5b90f 100644 --- a/flytepropeller/pkg/controller/nodes/transformers.go +++ b/flytepropeller/pkg/controller/nodes/transformers.go @@ -178,8 +178,10 @@ func ToNodeExecutionEvent(nodeExecID *core.NodeExecutionIdentifier, if node.GetKind() == v1alpha1.NodeKindWorkflow && node.GetWorkflowNode() != nil && node.GetWorkflowNode().GetSubWorkflowRef() != nil { nev.IsParent = true } else if node.GetKind() == v1alpha1.NodeKindArray { - nev.IsParent = true nev.IsArray = true + if config.GetConfig().ArrayNodeEventVersion == 1 { + nev.IsParent = true + } } else if dynamicNodePhase != v1alpha1.DynamicNodePhaseNone { nev.IsDynamic = true if nev.GetTaskNodeMetadata() != nil && nev.GetTaskNodeMetadata().DynamicWorkflow != nil { diff --git a/kustomize/overlays/eks/kustomization.yaml b/kustomize/overlays/eks/kustomization.yaml index f116ed2b78..d50b42c64f 100644 --- a/kustomize/overlays/eks/kustomization.yaml +++ b/kustomize/overlays/eks/kustomization.yaml @@ -21,23 +21,23 @@ bases: images: # FlyteAdmin - name: flyteadmin # match images with this name - newTag: v1.10.7 # FLYTEADMIN_TAG override the tag + newTag: v1.11.0-b0 # FLYTEADMIN_TAG override the tag newName: cr.flyte.org/flyteorg/flyteadmin # override the name # FlyteConsole - name: flyteconsole # match images with this name - newTag: v1.10.2 # FLYTECONSOLE_TAG the tag + newTag: v1.10.3 # FLYTECONSOLE_TAG the tag newName: cr.flyte.org/flyteorg/flyteconsole # override the namep # Flyte DataCatalog - name: datacatalog # match images with this name - newTag: v1.10.7 # DATACATALOG_TAG override the tag + newTag: v1.11.0-b0 # DATACATALOG_TAG override the tag newName: cr.flyte.org/flyteorg/datacatalog # override the name # FlytePropeller - name: flytepropeller # match images with this name - newTag: v1.10.7 # FLYTEPROPELLER_TAG override the tag + newTag: v1.11.0-b0 # FLYTEPROPELLER_TAG override the tag newName: cr.flyte.org/flyteorg/flytepropeller # override the name # Webhook - name: webhook # match images with this name - newTag: v1.10.7 # FLYTEPROPELLER_TAG override the tag + newTag: v1.11.0-b0 # FLYTEPROPELLER_TAG override the tag newName: cr.flyte.org/flyteorg/flytepropeller # override the name # Override postgres image to use alpine based (rather smaller) docker image - name: postgres diff --git a/rfc/system/1893-caching-of-offloaded-objects.md b/rfc/system/1893-caching-of-offloaded-objects.md index 806b6cfd36..9a748df02d 100644 --- a/rfc/system/1893-caching-of-offloaded-objects.md +++ b/rfc/system/1893-caching-of-offloaded-objects.md @@ -6,7 +6,7 @@ ## 1 Executive Summary -We propose a way to override the default behavior of [caching task executions](https://docs.flyte.org/en/latest/flytesnacks/examples/development_lifecycle/task_cache.html), enabling cache-by-value semantics for certain categories of objects. +We propose a way to override the default behavior of [caching task executions](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/caching.html), enabling cache-by-value semantics for certain categories of objects. ## 2 Motivation diff --git a/rfc/system/2633-eviction-of-cached-task-outputs.md b/rfc/system/2633-eviction-of-cached-task-outputs.md index 45c3cd8d73..52b4adebf6 100644 --- a/rfc/system/2633-eviction-of-cached-task-outputs.md +++ b/rfc/system/2633-eviction-of-cached-task-outputs.md @@ -158,7 +158,7 @@ The potential for malicious exploitation is deemed non-existent as no access to 3. Which Flyte tools (`flyteconsole`/`flytectl`) should support the proposed `AdminService` API extension for `flyteadmin`, if any? - **RESOLVED**: `flytectl`, `flytekit.remote`, `flyteconsole` 4. Should we support automatic eviction of cached results on workflow archival (opt-out via `flyteconsole`)? -5. Should we evict [Intratask Checkpoints](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/checkpoint.html#intratask-checkpoints) from the cache as well since they might return cached results? If so, should we evict them from the backend side or pass the `cache_override` flag along to `flytekit`/its `Checkpointer` to skip any available entries? +5. Should we evict [Intratask Checkpoints](https://docs.flyte.org/en/latest/user_guide/advanced_composition/intratask_checkpoints.html) from the cache as well since they might return cached results? If so, should we evict them from the backend side or pass the `cache_override` flag along to `flytekit`/its `Checkpointer` to skip any available entries? - **RESOLVED**: not for the initial implementation. Intratask checkpoints are only relevant for consecutive retries of a task - their results would not be considered when launching another execution with a `cache_override` flag set. ## 9 Conclusion