diff --git a/.github/workflows/single-binary.yml b/.github/workflows/single-binary.yml index 27ed4fabbc..4f3fdea994 100644 --- a/.github/workflows/single-binary.yml +++ b/.github/workflows/single-binary.yml @@ -175,7 +175,7 @@ jobs: run: | python -m pip install --upgrade pip pip install uv - uv pip install --system flytekit flytekitplugins-deck-standard "numpy<2.0.0" pyarrow + uv pip install --system flytekit flytekitplugins-deck-standard "numpy<2.0.0" pyarrow pandas uv pip freeze - name: Checkout flytesnacks uses: actions/checkout@v4 @@ -188,7 +188,7 @@ jobs: flytekit_version=$(pip show flytekit | grep -i version | awk '{ print $2 }') while read -r line; do - pyflyte --config ./boilerplate/flyte/end2end/functional-test-config.yaml \ + pyflyte -vv --config ./boilerplate/flyte/end2end/functional-test-config.yaml \ register \ --project flytesnacks \ --domain development \ diff --git a/CHANGELOG/CHANGELOG-v1.13.2.md b/CHANGELOG/CHANGELOG-v1.13.2.md new file mode 100644 index 0000000000..ec81e91526 --- /dev/null +++ b/CHANGELOG/CHANGELOG-v1.13.2.md @@ -0,0 +1,53 @@ +# Flyte 1.13.2 Release Notes + +## What's Changed +* Enable echo plugin by default by @pingsutw in https://github.com/flyteorg/flyte/pull/5679 +* Do not emit execution id label by default in single binary by @eapolinario in https://github.com/flyteorg/flyte/pull/5704 +* Using new offloaded metadata literal message for literal offloading by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/5705 +* Improve error message for MismatchingTypes by @pingsutw in https://github.com/flyteorg/flyte/pull/5639 +* [Docs] Echo Task by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5707 +* Improve execution name readability by @wayner0628 in https://github.com/flyteorg/flyte/pull/5637 +* Configure imagePullPolicy to be Always pull on flyte sandbox environment by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/5709 +* Set IsDefault to False for echo plugin by @pingsutw in https://github.com/flyteorg/flyte/pull/5713 +* Move default execution name generation to flyteadmin by @wayner0628 in https://github.com/flyteorg/flyte/pull/5714 +* Update helm/docs per changes in supported task discovery by @Sovietaced in https://github.com/flyteorg/flyte/pull/5694 +* [flyteagent] Add Logging for Agent Supported Task Types by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5718 +* extend pod customization to include init containers by @samhita-alla in https://github.com/flyteorg/flyte/pull/5685 +* Update "Try Serverless" language in Quickstart guide by @neverett in https://github.com/flyteorg/flyte/pull/5698 +* Refactor flyteadmin to pass proto structs as pointers by @Sovietaced in https://github.com/flyteorg/flyte/pull/5717 +* fix: Use deterministic execution names in scheduler by @pingsutw in https://github.com/flyteorg/flyte/pull/5724 +* [flyteagent] Enable `agent-service` in every configuration by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5730 +* [Docs][flyteagent] Remove `agent-service` config in flyte agent documentation by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5731 +* Fix flytectl returning the oldest workflow when using --latest flag by @RRap0so in https://github.com/flyteorg/flyte/pull/5716 +* Remove explicit go toolchain versions by @ddl-ebrown in https://github.com/flyteorg/flyte/pull/5723 +* Add listing api to stow storage by @bgedik in https://github.com/flyteorg/flyte/pull/5741 +* Use latest upload/download-artifact action version by @Sovietaced in https://github.com/flyteorg/flyte/pull/5743 +* Introduced SMTP notification by @robert-ulbrich-mercedes-benz in https://github.com/flyteorg/flyte/pull/5535 +* Added literal offloading for array node map tasks by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/5697 +* pytorch object.inv moved by @wild-endeavor in https://github.com/flyteorg/flyte/pull/5755 +* [RFC] Binary IDL With MessagePack Bytes by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5742 +* Rename literal offloading flags and enable offloading in local config by @eapolinario in https://github.com/flyteorg/flyte/pull/5754 +* Add semver check for dev and beta version by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/5757 +* Add comments for semver regex by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/5758 +* Enable caching in flyte-core helmchart by @cpaulik in https://github.com/flyteorg/flyte/pull/5739 +* [Flyte][1][IDL] Binary IDL With MessagePack by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5751 +* [Flyte][2][Literal Type For Scalar] Binary IDL With MessagePack by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5761 +* Backoff on etcd errors by @EngHabu in https://github.com/flyteorg/flyte/pull/5710 +* [Docs][flyteagent] Update Databricks Agent Setup to V2 by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5766 +* [Docs][flyteagent] Improve Agent Secret Setup by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5765 +* [RFC] Offloaded Raw Literals by @wild-endeavor in https://github.com/flyteorg/flyte/pull/5103 +* Use pluggable clock for auto refresh cache and make unit tests fast by @Sovietaced in https://github.com/flyteorg/flyte/pull/5767 +* [Docs] Flyte Deck example by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5777 +* Update ImageSpec documentation by @pingsutw in https://github.com/flyteorg/flyte/pull/5748 +* Revert "Improve execution name readability" by @pingsutw in https://github.com/flyteorg/flyte/pull/5740 +* [Docs] Flyte Deck example V2 by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5781 +* [flytepropeller][flyteadmin] Compiler unknown literal type error handling by @Future-Outlier in https://github.com/flyteorg/flyte/pull/5651 +* [Temporary fix] Pin flytekit version for single binary builds by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/5798 +* Fix propeller crash when inferring literal type for an offloaded literal by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/5771 + +## New Contributors +* @wayner0628 made their first contribution in https://github.com/flyteorg/flyte/pull/5637 +* @robert-ulbrich-mercedes-benz made their first contribution in https://github.com/flyteorg/flyte/pull/5535 +* @cpaulik made their first contribution in https://github.com/flyteorg/flyte/pull/5739 + +**Full Changelog**: https://github.com/flyteorg/flyte/compare/v1.13.1...v1.13.2 diff --git a/charts/flyte-binary/README.md b/charts/flyte-binary/README.md index c5daebc219..dbbdb3e732 100644 --- a/charts/flyte-binary/README.md +++ b/charts/flyte-binary/README.md @@ -41,7 +41,7 @@ Chart for basic single Flyte executable deployment | configuration.auth.oidc.clientId | string | `""` | | | configuration.auth.oidc.clientSecret | string | `""` | | | configuration.co-pilot.image.repository | string | `"cr.flyte.org/flyteorg/flytecopilot"` | | -| configuration.co-pilot.image.tag | string | `"v1.13.1"` | | +| configuration.co-pilot.image.tag | string | `"v1.13.2"` | | | configuration.database.dbname | string | `"flyte"` | | | configuration.database.host | string | `"127.0.0.1"` | | | configuration.database.options | string | `"sslmode=disable"` | | diff --git a/charts/flyte-binary/values.yaml b/charts/flyte-binary/values.yaml index c6c0e32fca..a35102f0c6 100644 --- a/charts/flyte-binary/values.yaml +++ b/charts/flyte-binary/values.yaml @@ -160,7 +160,7 @@ configuration: # repository CoPilot sidecar image repository repository: cr.flyte.org/flyteorg/flytecopilot # FLYTECOPILOT_IMAGE # tag CoPilot sidecar image tag - tag: v1.13.1 # FLYTECOPILOT_TAG + tag: v1.13.2 # FLYTECOPILOT_TAG # agentService Flyte Agent configuration agentService: defaultAgent: diff --git a/charts/flyte-core/README.md b/charts/flyte-core/README.md index 58a1f5a7c2..9a33582c4c 100644 --- a/charts/flyte-core/README.md +++ b/charts/flyte-core/README.md @@ -95,8 +95,8 @@ helm install gateway bitnami/contour -n flyte | configmap.clusters.clusterConfigs | list | `[]` | | | configmap.clusters.labelClusterMap | object | `{}` | | | configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI | -| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.1","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | -| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.1","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | +| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.2","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | +| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.2","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | configmap.core | object | `{"manager":{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"},"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration | | configmap.core.manager | object | `{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config). | | configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). | @@ -131,7 +131,7 @@ helm install gateway bitnami/contour -n flyte | datacatalog.extraArgs | object | `{}` | Appends extra command line arguments to the main command | | datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment | -| datacatalog.image.tag | string | `"v1.13.1"` | Docker image tag | +| datacatalog.image.tag | string | `"v1.13.2"` | Docker image tag | | datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | | datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | | datacatalog.podEnv | object | `{}` | Additional Datacatalog container environment variables | @@ -167,7 +167,7 @@ helm install gateway bitnami/contour -n flyte | flyteadmin.extraArgs | object | `{}` | Appends extra command line arguments to the serve command | | flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | | | flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment | -| flyteadmin.image.tag | string | `"v1.13.1"` | | +| flyteadmin.image.tag | string | `"v1.13.2"` | | | flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | | flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | | flyteadmin.podLabels | object | `{}` | Labels for Flyteadmin pods | @@ -238,7 +238,7 @@ helm install gateway bitnami/contour -n flyte | flytepropeller.extraArgs | object | `{}` | Appends extra command line arguments to the main command | | flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | | | flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment | -| flytepropeller.image.tag | string | `"v1.13.1"` | | +| flytepropeller.image.tag | string | `"v1.13.2"` | | | flytepropeller.manager | bool | `false` | | | flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment | | flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods | @@ -270,7 +270,7 @@ helm install gateway bitnami/contour -n flyte | flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment | -| flytescheduler.image.tag | string | `"v1.13.1"` | Docker image tag | +| flytescheduler.image.tag | string | `"v1.13.2"` | Docker image tag | | flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment | | flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods | | flytescheduler.podEnv | object | `{}` | Additional Flytescheduler container environment variables | diff --git a/charts/flyte-core/values.yaml b/charts/flyte-core/values.yaml index a1d83eb522..cf0f6a5986 100755 --- a/charts/flyte-core/values.yaml +++ b/charts/flyte-core/values.yaml @@ -16,7 +16,7 @@ flyteadmin: image: # -- Docker image for Flyteadmin deployment repository: cr.flyte.org/flyteorg/flyteadmin # FLYTEADMIN_IMAGE - tag: v1.13.1 # FLYTEADMIN_TAG + tag: v1.13.2 # FLYTEADMIN_TAG pullPolicy: IfNotPresent # -- Additional flyteadmin container environment variables # @@ -139,7 +139,7 @@ flytescheduler: # -- Docker image for Flytescheduler deployment repository: cr.flyte.org/flyteorg/flytescheduler # FLYTESCHEDULER_IMAGE # -- Docker image tag - tag: v1.13.1 # FLYTESCHEDULER_TAG + tag: v1.13.2 # FLYTESCHEDULER_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytescheduler deployment @@ -205,7 +205,7 @@ datacatalog: # -- Docker image for Datacatalog deployment repository: cr.flyte.org/flyteorg/datacatalog # DATACATALOG_IMAGE # -- Docker image tag - tag: v1.13.1 # DATACATALOG_TAG + tag: v1.13.2 # DATACATALOG_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Datacatalog deployment @@ -302,7 +302,7 @@ flytepropeller: image: # -- Docker image for Flytepropeller deployment repository: cr.flyte.org/flyteorg/flytepropeller # FLYTEPROPELLER_IMAGE - tag: v1.13.1 # FLYTEPROPELLER_TAG + tag: v1.13.2 # FLYTEPROPELLER_TAG pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytepropeller deployment resources: @@ -803,7 +803,7 @@ configmap: # -- Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) co-pilot: name: flyte-copilot- - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 # FLYTECOPILOT_IMAGE + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 # FLYTECOPILOT_IMAGE start-timeout: 30s # -- Core propeller configuration diff --git a/charts/flyte/README.md b/charts/flyte/README.md index 1c0d00661c..679923d8cd 100644 --- a/charts/flyte/README.md +++ b/charts/flyte/README.md @@ -71,7 +71,7 @@ helm upgrade -f values-sandbox.yaml flyte . | contour.tolerations | list | `[]` | tolerations for Contour deployment | | daskoperator | object | `{"enabled":false}` | Optional: Dask Plugin using the Dask Operator | | daskoperator.enabled | bool | `false` | - enable or disable the dask operator deployment installation | -| flyte | object | `{"cluster_resource_manager":{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"service_account_name":"flyteadmin","templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]},"common":{"databaseSecret":{"name":"","secretManifest":{}},"flyteNamespaceTemplate":{"enabled":false},"ingress":{"albSSLRedirect":false,"annotations":{"nginx.ingress.kubernetes.io/app-root":"/console"},"enabled":true,"host":"","separateGrpcIngress":false,"separateGrpcIngressAnnotations":{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC"},"tls":{"enabled":false},"webpackHMR":true}},"configmap":{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpc":{"port":8089},"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.1","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sensor":"agent-service","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service","echo"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-from-configmaps":[],"default-env-from-secrets":[],"default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}},"datacatalog":{"affinity":{},"configPath":"/etc/datacatalog/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/datacatalog","tag":"v1.13.1"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"db":{"admin":{"database":{"dbname":"flyteadmin","host":"postgres","port":5432,"username":"postgres"}},"datacatalog":{"database":{"dbname":"datacatalog","host":"postgres","port":5432,"username":"postgres"}}},"deployRedoc":true,"flyteadmin":{"additionalVolumeMounts":[],"additionalVolumes":[],"affinity":{},"configPath":"/etc/flyte/config/*.yaml","env":[],"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteadmin","tag":"v1.13.1"},"initialProjects":["flytesnacks","flytetester","flyteexamples"],"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"loadBalancerSourceRanges":[],"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flyteconsole":{"affinity":{},"ga":{"enabled":true,"tracking_id":"G-0QW4DJWJ20"},"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteconsole","tag":"v1.17.1"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","memory":"275Mi"},"requests":{"cpu":"10m","memory":"250Mi"}},"service":{"annotations":{},"type":"ClusterIP"},"tolerations":[]},"flytepropeller":{"affinity":{},"cacheSizeMbs":0,"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytepropeller","tag":"v1.13.1"},"manager":false,"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flytescheduler":{"affinity":{},"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytescheduler","tag":"v1.13.1"},"nodeSelector":{},"podAnnotations":{},"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"storage":{"bucketName":"my-s3-bucket","custom":{},"gcs":null,"s3":{"region":"us-east-1"},"type":"sandbox"},"webhook":{"enabled":true,"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]}},"workflow_notifications":{"config":{},"enabled":false},"workflow_scheduler":{"enabled":true,"type":"native"}}` | ------------------------------------------------------------------- Core System settings This section consists of Core components of Flyte and their deployment settings. This includes FlyteAdmin service, Datacatalog, FlytePropeller and Flyteconsole | +| flyte | object | `{"cluster_resource_manager":{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"service_account_name":"flyteadmin","templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]},"common":{"databaseSecret":{"name":"","secretManifest":{}},"flyteNamespaceTemplate":{"enabled":false},"ingress":{"albSSLRedirect":false,"annotations":{"nginx.ingress.kubernetes.io/app-root":"/console"},"enabled":true,"host":"","separateGrpcIngress":false,"separateGrpcIngressAnnotations":{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC"},"tls":{"enabled":false},"webpackHMR":true}},"configmap":{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpc":{"port":8089},"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.2","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sensor":"agent-service","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service","echo"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-from-configmaps":[],"default-env-from-secrets":[],"default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}},"datacatalog":{"affinity":{},"configPath":"/etc/datacatalog/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/datacatalog","tag":"v1.13.2"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"db":{"admin":{"database":{"dbname":"flyteadmin","host":"postgres","port":5432,"username":"postgres"}},"datacatalog":{"database":{"dbname":"datacatalog","host":"postgres","port":5432,"username":"postgres"}}},"deployRedoc":true,"flyteadmin":{"additionalVolumeMounts":[],"additionalVolumes":[],"affinity":{},"configPath":"/etc/flyte/config/*.yaml","env":[],"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteadmin","tag":"v1.13.2"},"initialProjects":["flytesnacks","flytetester","flyteexamples"],"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"loadBalancerSourceRanges":[],"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flyteconsole":{"affinity":{},"ga":{"enabled":true,"tracking_id":"G-0QW4DJWJ20"},"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteconsole","tag":"v1.17.1"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","memory":"275Mi"},"requests":{"cpu":"10m","memory":"250Mi"}},"service":{"annotations":{},"type":"ClusterIP"},"tolerations":[]},"flytepropeller":{"affinity":{},"cacheSizeMbs":0,"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytepropeller","tag":"v1.13.2"},"manager":false,"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"flytescheduler":{"affinity":{},"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytescheduler","tag":"v1.13.2"},"nodeSelector":{},"podAnnotations":{},"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]},"tolerations":[]},"storage":{"bucketName":"my-s3-bucket","custom":{},"gcs":null,"s3":{"region":"us-east-1"},"type":"sandbox"},"webhook":{"enabled":true,"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":[]}},"workflow_notifications":{"config":{},"enabled":false},"workflow_scheduler":{"enabled":true,"type":"native"}}` | ------------------------------------------------------------------- Core System settings This section consists of Core components of Flyte and their deployment settings. This includes FlyteAdmin service, Datacatalog, FlytePropeller and Flyteconsole | | flyte.cluster_resource_manager | object | `{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"service_account_name":"flyteadmin","templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]}` | Configuration for the Cluster resource manager component. This is an optional component, that enables automatic cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain | | flyte.cluster_resource_manager.config.cluster_resources | object | `{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refresh":"5m","refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}` | ClusterResource parameters Refer to the [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#ClusterResourceConfig) to customize. | | flyte.cluster_resource_manager.config.cluster_resources.standaloneDeployment | bool | `false` | Starts the cluster resource manager in standalone mode with requisite auth credentials to call flyteadmin service endpoints | @@ -91,15 +91,15 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.common.ingress.separateGrpcIngressAnnotations | object | `{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC"}` | - Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled. | | flyte.common.ingress.tls | object | `{"enabled":false}` | - TLS Settings | | flyte.common.ingress.webpackHMR | bool | `true` | - Enable or disable HMR route to flyteconsole. This is useful only for frontend development. | -| flyte.configmap | object | `{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpc":{"port":8089},"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.1","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sensor":"agent-service","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service","echo"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-from-configmaps":[],"default-env-from-secrets":[],"default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}}` | ----------------------------------------------------------------- CONFIGMAPS SETTINGS | +| flyte.configmap | object | `{"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpc":{"port":8089},"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.2","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sensor":"agent-service","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service","echo"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-from-configmaps":[],"default-env-from-secrets":[],"default-env-vars":[{"FLYTE_AWS_ENDPOINT":"http://minio.flyte:9000"},{"FLYTE_AWS_ACCESS_KEY_ID":"minio"},{"FLYTE_AWS_SECRET_ACCESS_KEY":"miniostorage"}],"default-memory":"200Mi"}}},"logger":{"logger":{"level":5,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":null,"type":"noop"}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":true,"kubernetes-template-uri":"http://localhost:30082/#/log/{{ \"{{\" }} .namespace {{ \"}}\" }}/{{ \"{{\" }} .podName {{ \"}}\" }}/pod?namespace={{ \"{{\" }} .namespace {{ \"}}\" }}"}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"200Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}}` | ----------------------------------------------------------------- CONFIGMAPS SETTINGS | | flyte.configmap.adminServer | object | `{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":2,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpc":{"port":8089},"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}}` | FlyteAdmin server configuration | | flyte.configmap.adminServer.auth | object | `{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}}` | Authentication configuration | | flyte.configmap.adminServer.server.security.secure | bool | `false` | Controls whether to serve requests over SSL/TLS. | | flyte.configmap.adminServer.server.security.useAuth | bool | `false` | Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. | | flyte.configmap.catalog | object | `{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}}` | Catalog Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/catalog#Config) Additional advanced Catalog configuration [here](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/catalog#Config) | | flyte.configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI | -| flyte.configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.1","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | -| flyte.configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.1","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | +| flyte.configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.2","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | +| flyte.configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.13.2","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | flyte.configmap.core | object | `{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration | | flyte.configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). | | flyte.configmap.datacatalogServer | object | `{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}}` | Datacatalog server config | @@ -120,7 +120,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.datacatalog.configPath | string | `"/etc/datacatalog/config/*.yaml"` | Default regex string for searching configuration files | | flyte.datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment | -| flyte.datacatalog.image.tag | string | `"v1.13.1"` | Docker image tag | +| flyte.datacatalog.image.tag | string | `"v1.13.2"` | Docker image tag | | flyte.datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | | flyte.datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | | flyte.datacatalog.replicaCount | int | `1` | Replicas count for Datacatalog deployment | @@ -136,7 +136,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.flyteadmin.env | list | `[]` | Additional flyteadmin container environment variables e.g. SendGrid's API key - name: SENDGRID_API_KEY value: "" e.g. secret environment variable (you can combine it with .additionalVolumes): - name: SENDGRID_API_KEY valueFrom: secretKeyRef: name: sendgrid-secret key: api_key | | flyte.flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment | -| flyte.flyteadmin.image.tag | string | `"v1.13.1"` | Docker image tag | +| flyte.flyteadmin.image.tag | string | `"v1.13.2"` | Docker image tag | | flyte.flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create | | flyte.flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | | flyte.flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | @@ -162,7 +162,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.flytepropeller.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | flyte.flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment | -| flyte.flytepropeller.image.tag | string | `"v1.13.1"` | Docker image tag | +| flyte.flytepropeller.image.tag | string | `"v1.13.2"` | Docker image tag | | flyte.flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment | | flyte.flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods | | flyte.flytepropeller.replicaCount | int | `1` | Replicas count for Flytepropeller deployment | @@ -176,7 +176,7 @@ helm upgrade -f values-sandbox.yaml flyte . | flyte.flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | flyte.flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | flyte.flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment | -| flyte.flytescheduler.image.tag | string | `"v1.13.1"` | Docker image tag | +| flyte.flytescheduler.image.tag | string | `"v1.13.2"` | Docker image tag | | flyte.flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment | | flyte.flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods | | flyte.flytescheduler.resources | object | `{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Flytescheduler deployment | diff --git a/charts/flyte/values.yaml b/charts/flyte/values.yaml index 63c0a34c64..cfdb31ae93 100755 --- a/charts/flyte/values.yaml +++ b/charts/flyte/values.yaml @@ -15,7 +15,7 @@ flyte: # -- Docker image for Flyteadmin deployment repository: cr.flyte.org/flyteorg/flyteadmin # FLYTEADMIN_IMAGE # -- Docker image tag - tag: v1.13.1 # FLYTEADMIN_TAG + tag: v1.13.2 # FLYTEADMIN_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Additional flyteadmin container environment variables @@ -83,7 +83,7 @@ flyte: # -- Docker image for Flytescheduler deployment repository: cr.flyte.org/flyteorg/flytescheduler # FLYTESCHEDULER_IMAGE # -- Docker image tag - tag: v1.13.1 # FLYTESCHEDULER_TAG + tag: v1.13.2 # FLYTESCHEDULER_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytescheduler deployment @@ -128,7 +128,7 @@ flyte: # -- Docker image for Datacatalog deployment repository: cr.flyte.org/flyteorg/datacatalog # DATACATALOG_IMAGE # -- Docker image tag - tag: v1.13.1 # DATACATALOG_TAG + tag: v1.13.2 # DATACATALOG_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Datacatalog deployment @@ -177,7 +177,7 @@ flyte: # -- Docker image for Flytepropeller deployment repository: cr.flyte.org/flyteorg/flytepropeller # FLYTEPROPELLER_IMAGE # -- Docker image tag - tag: v1.13.1 # FLYTEPROPELLER_TAG + tag: v1.13.2 # FLYTEPROPELLER_TAG # -- Docker image pull policy pullPolicy: IfNotPresent # -- Default resources requests and limits for Flytepropeller deployment @@ -471,7 +471,7 @@ flyte: # -- Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) co-pilot: name: flyte-copilot- - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 # FLYTECOPILOT_IMAGE + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 # FLYTECOPILOT_IMAGE start-timeout: 30s # -- Core propeller configuration diff --git a/charts/flyteagent/README.md b/charts/flyteagent/README.md index 6bd7b056c4..47ca1380b3 100644 --- a/charts/flyteagent/README.md +++ b/charts/flyteagent/README.md @@ -20,7 +20,7 @@ A Helm chart for Flyte agent | fullnameOverride | string | `""` | | | image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | image.repository | string | `"cr.flyte.org/flyteorg/flyteagent"` | Docker image for flyteagent deployment | -| image.tag | string | `"1.13.4"` | Docker image tag | +| image.tag | string | `"1.13.6"` | Docker image tag | | nameOverride | string | `""` | | | nodeSelector | object | `{}` | nodeSelector for flyteagent deployment | | podAnnotations | object | `{}` | Annotations for flyteagent pods | diff --git a/charts/flyteagent/values.yaml b/charts/flyteagent/values.yaml index 845248af90..fbc790b286 100755 --- a/charts/flyteagent/values.yaml +++ b/charts/flyteagent/values.yaml @@ -23,7 +23,7 @@ image: # -- Docker image for flyteagent deployment repository: cr.flyte.org/flyteorg/flyteagent # FLYTEAGENT_IMAGE # -- Docker image tag - tag: 1.13.4 # FLYTEAGENT_TAG + tag: 1.13.6 # FLYTEAGENT_TAG # -- Docker image pull policy pullPolicy: IfNotPresent ports: diff --git a/deployment/agent/flyte_agent_helm_generated.yaml b/deployment/agent/flyte_agent_helm_generated.yaml index 53fe9016c2..e90de9a379 100644 --- a/deployment/agent/flyte_agent_helm_generated.yaml +++ b/deployment/agent/flyte_agent_helm_generated.yaml @@ -79,7 +79,7 @@ spec: - pyflyte - serve - agent - image: "cr.flyte.org/flyteorg/flyteagent:1.13.4" + image: "cr.flyte.org/flyteorg/flyteagent:1.13.6" imagePullPolicy: "IfNotPresent" name: flyteagent volumeMounts: diff --git a/deployment/eks/flyte_aws_scheduler_helm_generated.yaml b/deployment/eks/flyte_aws_scheduler_helm_generated.yaml index 5b2e60ab85..f68a6c78e9 100644 --- a/deployment/eks/flyte_aws_scheduler_helm_generated.yaml +++ b/deployment/eks/flyte_aws_scheduler_helm_generated.yaml @@ -438,7 +438,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -886,7 +886,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -904,7 +904,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - seed-projects - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -922,7 +922,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -939,7 +939,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -966,7 +966,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -1073,7 +1073,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -1203,7 +1203,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -1221,7 +1221,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1284,7 +1284,7 @@ spec: template: metadata: annotations: - configChecksum: "16400bbf28ab823ca433a04379f00fecb11530c3c265dd1bdc3e30209ac6a6d" + configChecksum: "6c8b21f7f9e96d92cfc0932e5a4289e969380662d96d3e6728a142bf01291c1" prometheus.io/path: "/metrics" prometheus.io/port: "10254" labels: @@ -1312,7 +1312,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -1366,9 +1366,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.13.1 + app.kubernetes.io/version: v1.13.2 annotations: - configChecksum: "16400bbf28ab823ca433a04379f00fecb11530c3c265dd1bdc3e30209ac6a6d" + configChecksum: "6c8b21f7f9e96d92cfc0932e5a4289e969380662d96d3e6728a142bf01291c1" prometheus.io/path: "/metrics" prometheus.io/port: "10254" spec: @@ -1382,7 +1382,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -1409,7 +1409,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/eks/flyte_helm_controlplane_generated.yaml b/deployment/eks/flyte_helm_controlplane_generated.yaml index 9ab13c9cd8..18cc68ed57 100644 --- a/deployment/eks/flyte_helm_controlplane_generated.yaml +++ b/deployment/eks/flyte_helm_controlplane_generated.yaml @@ -589,7 +589,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -607,7 +607,8 @@ spec: - /etc/flyte/config/*.yaml - migrate - seed-projects - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -625,7 +626,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -642,7 +643,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -669,7 +670,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -776,7 +777,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -906,7 +907,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -924,7 +925,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1007,7 +1008,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1027,7 +1028,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: diff --git a/deployment/eks/flyte_helm_dataplane_generated.yaml b/deployment/eks/flyte_helm_dataplane_generated.yaml index aa649b24ea..d2071582f5 100644 --- a/deployment/eks/flyte_helm_dataplane_generated.yaml +++ b/deployment/eks/flyte_helm_dataplane_generated.yaml @@ -94,7 +94,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -430,7 +430,7 @@ spec: template: metadata: annotations: - configChecksum: "16400bbf28ab823ca433a04379f00fecb11530c3c265dd1bdc3e30209ac6a6d" + configChecksum: "6c8b21f7f9e96d92cfc0932e5a4289e969380662d96d3e6728a142bf01291c1" prometheus.io/path: "/metrics" prometheus.io/port: "10254" labels: @@ -458,7 +458,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -512,9 +512,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.13.1 + app.kubernetes.io/version: v1.13.2 annotations: - configChecksum: "16400bbf28ab823ca433a04379f00fecb11530c3c265dd1bdc3e30209ac6a6d" + configChecksum: "6c8b21f7f9e96d92cfc0932e5a4289e969380662d96d3e6728a142bf01291c1" prometheus.io/path: "/metrics" prometheus.io/port: "10254" spec: @@ -528,7 +528,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -555,7 +555,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/eks/flyte_helm_generated.yaml b/deployment/eks/flyte_helm_generated.yaml index be01c4fe75..6c48b083bd 100644 --- a/deployment/eks/flyte_helm_generated.yaml +++ b/deployment/eks/flyte_helm_generated.yaml @@ -469,7 +469,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -917,7 +917,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -935,7 +935,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - seed-projects - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -953,7 +953,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -970,7 +970,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -997,7 +997,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -1104,7 +1104,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -1234,7 +1234,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -1252,7 +1252,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1335,7 +1335,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1355,7 +1355,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: @@ -1414,7 +1414,7 @@ spec: template: metadata: annotations: - configChecksum: "16400bbf28ab823ca433a04379f00fecb11530c3c265dd1bdc3e30209ac6a6d" + configChecksum: "6c8b21f7f9e96d92cfc0932e5a4289e969380662d96d3e6728a142bf01291c1" prometheus.io/path: "/metrics" prometheus.io/port: "10254" labels: @@ -1442,7 +1442,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -1496,9 +1496,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.13.1 + app.kubernetes.io/version: v1.13.2 annotations: - configChecksum: "16400bbf28ab823ca433a04379f00fecb11530c3c265dd1bdc3e30209ac6a6d" + configChecksum: "6c8b21f7f9e96d92cfc0932e5a4289e969380662d96d3e6728a142bf01291c1" prometheus.io/path: "/metrics" prometheus.io/port: "10254" spec: @@ -1512,7 +1512,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -1539,7 +1539,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/gcp/flyte_helm_controlplane_generated.yaml b/deployment/gcp/flyte_helm_controlplane_generated.yaml index c505c33488..11074bebe9 100644 --- a/deployment/gcp/flyte_helm_controlplane_generated.yaml +++ b/deployment/gcp/flyte_helm_controlplane_generated.yaml @@ -604,7 +604,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -622,7 +622,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - seed-projects - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -640,7 +640,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -657,7 +657,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -684,7 +684,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -791,7 +791,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -921,7 +921,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -939,7 +939,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1022,7 +1022,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1042,7 +1042,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: diff --git a/deployment/gcp/flyte_helm_dataplane_generated.yaml b/deployment/gcp/flyte_helm_dataplane_generated.yaml index 40f2542640..b3f9bdc12d 100644 --- a/deployment/gcp/flyte_helm_dataplane_generated.yaml +++ b/deployment/gcp/flyte_helm_dataplane_generated.yaml @@ -94,7 +94,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -438,7 +438,7 @@ spec: template: metadata: annotations: - configChecksum: "b11d6c5dd0dd16bced82fb44fd2fb31c2f27a134543ab7edaf060da42ab27e4" + configChecksum: "0056f7638fe13bd9187bfaf4011fde4cfbecd13c152ba443bba7fe25ca4777a" prometheus.io/path: "/metrics" prometheus.io/port: "10254" labels: @@ -465,7 +465,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -519,9 +519,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.13.1 + app.kubernetes.io/version: v1.13.2 annotations: - configChecksum: "b11d6c5dd0dd16bced82fb44fd2fb31c2f27a134543ab7edaf060da42ab27e4" + configChecksum: "0056f7638fe13bd9187bfaf4011fde4cfbecd13c152ba443bba7fe25ca4777a" prometheus.io/path: "/metrics" prometheus.io/port: "10254" spec: @@ -535,7 +535,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -562,7 +562,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/gcp/flyte_helm_generated.yaml b/deployment/gcp/flyte_helm_generated.yaml index 3c029e13a8..3d5618f0a6 100644 --- a/deployment/gcp/flyte_helm_generated.yaml +++ b/deployment/gcp/flyte_helm_generated.yaml @@ -482,7 +482,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -940,7 +940,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -958,7 +958,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - seed-projects - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -976,7 +976,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -993,7 +993,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -1020,7 +1020,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -1127,7 +1127,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -1257,7 +1257,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -1275,7 +1275,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -1358,7 +1358,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -1378,7 +1378,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: @@ -1437,7 +1437,7 @@ spec: template: metadata: annotations: - configChecksum: "b11d6c5dd0dd16bced82fb44fd2fb31c2f27a134543ab7edaf060da42ab27e4" + configChecksum: "0056f7638fe13bd9187bfaf4011fde4cfbecd13c152ba443bba7fe25ca4777a" prometheus.io/path: "/metrics" prometheus.io/port: "10254" labels: @@ -1464,7 +1464,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -1518,9 +1518,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.13.1 + app.kubernetes.io/version: v1.13.2 annotations: - configChecksum: "b11d6c5dd0dd16bced82fb44fd2fb31c2f27a134543ab7edaf060da42ab27e4" + configChecksum: "0056f7638fe13bd9187bfaf4011fde4cfbecd13c152ba443bba7fe25ca4777a" prometheus.io/path: "/metrics" prometheus.io/port: "10254" spec: @@ -1534,7 +1534,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -1561,7 +1561,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml b/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml index 93c749c118..7e2bf69502 100644 --- a/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml +++ b/deployment/sandbox-binary/flyte_sandbox_binary_helm_generated.yaml @@ -118,7 +118,7 @@ data: stackdriver-enabled: false k8s: co-pilot: - image: "cr.flyte.org/flyteorg/flytecopilot:v1.13.1" + image: "cr.flyte.org/flyteorg/flytecopilot:v1.13.2" k8s-array: logs: config: @@ -361,7 +361,7 @@ spec: app.kubernetes.io/instance: flyte app.kubernetes.io/component: flyte-binary annotations: - checksum/configuration: 63b53dc1e5c5ee68d2b80d66b0a1613bf736840a2cbd5162704e6f91a09787ae + checksum/configuration: faaefbd3b3b2ddfd4e718bd77c02c632c75e7111dad0a6e25dc415dc88add73f checksum/configuration-secret: d5d93f4e67780b21593dc3799f0f6682aab0765e708e4020939975d14d44f929 checksum/cluster-resource-templates: 7dfa59f3d447e9c099b8f8ffad3af466fecbc9cf9f8c97295d9634254a55d4ae spec: diff --git a/deployment/sandbox/flyte_helm_generated.yaml b/deployment/sandbox/flyte_helm_generated.yaml index 8b0b7a8682..a21855ce99 100644 --- a/deployment/sandbox/flyte_helm_generated.yaml +++ b/deployment/sandbox/flyte_helm_generated.yaml @@ -594,7 +594,7 @@ data: plugins: k8s: co-pilot: - image: cr.flyte.org/flyteorg/flytecopilot:v1.13.1 + image: cr.flyte.org/flyteorg/flytecopilot:v1.13.2 name: flyte-copilot- start-timeout: 30s core.yaml: | @@ -6724,7 +6724,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations securityContext: @@ -6741,7 +6741,7 @@ spec: - /etc/flyte/config/*.yaml - migrate - seed-projects - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: seed-projects securityContext: @@ -6758,7 +6758,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - sync - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources securityContext: @@ -6774,7 +6774,7 @@ spec: - mountPath: /etc/secrets/ name: admin-secrets - name: generate-secrets - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" command: ["/bin/sh", "-c"] args: @@ -6801,7 +6801,7 @@ spec: - --config - /etc/flyte/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: flyteadmin ports: @@ -6898,7 +6898,7 @@ spec: - /etc/flyte/config/*.yaml - clusterresource - run - image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.1" + image: "cr.flyte.org/flyteorg/flyteadmin:v1.13.2" imagePullPolicy: "IfNotPresent" name: sync-cluster-resources volumeMounts: @@ -7023,7 +7023,7 @@ spec: - /etc/datacatalog/config/*.yaml - migrate - run - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: run-migrations volumeMounts: @@ -7040,7 +7040,7 @@ spec: - --config - /etc/datacatalog/config/*.yaml - serve - image: "cr.flyte.org/flyteorg/datacatalog:v1.13.1" + image: "cr.flyte.org/flyteorg/datacatalog:v1.13.2" imagePullPolicy: "IfNotPresent" name: datacatalog ports: @@ -7113,7 +7113,7 @@ spec: - precheck - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler-check securityContext: @@ -7132,7 +7132,7 @@ spec: - run - --config - /etc/flyte/config/*.yaml - image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.1" + image: "cr.flyte.org/flyteorg/flytescheduler:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytescheduler ports: @@ -7188,7 +7188,7 @@ spec: template: metadata: annotations: - configChecksum: "8a003fdbed4b3801328c26cb5c202e4ca113875347388366d29a302d610e7e4" + configChecksum: "eda2e0a9fc32f46471061f47df324b50fe3ded8a90c0ac18a75755c50eb80b7" prometheus.io/path: "/metrics" prometheus.io/port: "10254" labels: @@ -7215,7 +7215,7 @@ spec: valueFrom: fieldRef: fieldPath: metadata.namespace - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" name: flytepropeller ports: @@ -7262,9 +7262,9 @@ spec: labels: app: flyte-pod-webhook app.kubernetes.io/name: flyte-pod-webhook - app.kubernetes.io/version: v1.13.1 + app.kubernetes.io/version: v1.13.2 annotations: - configChecksum: "8a003fdbed4b3801328c26cb5c202e4ca113875347388366d29a302d610e7e4" + configChecksum: "eda2e0a9fc32f46471061f47df324b50fe3ded8a90c0ac18a75755c50eb80b7" prometheus.io/path: "/metrics" prometheus.io/port: "10254" spec: @@ -7278,7 +7278,7 @@ spec: serviceAccountName: flyte-pod-webhook initContainers: - name: generate-secrets - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller @@ -7305,7 +7305,7 @@ spec: mountPath: /etc/flyte/config containers: - name: webhook - image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.1" + image: "cr.flyte.org/flyteorg/flytepropeller:v1.13.2" imagePullPolicy: "IfNotPresent" command: - flytepropeller diff --git a/docker/sandbox-bundled/manifests/complete-agent.yaml b/docker/sandbox-bundled/manifests/complete-agent.yaml index c0b108c9a5..b60b04d233 100644 --- a/docker/sandbox-bundled/manifests/complete-agent.yaml +++ b/docker/sandbox-bundled/manifests/complete-agent.yaml @@ -471,7 +471,7 @@ data: stackdriver-enabled: false k8s: co-pilot: - image: "cr.flyte.org/flyteorg/flytecopilot:v1.13.1" + image: "cr.flyte.org/flyteorg/flytecopilot:v1.13.2" k8s-array: logs: config: @@ -817,7 +817,7 @@ type: Opaque --- apiVersion: v1 data: - haSharedSecret: aVh1N3lZb0F1c2l0NHVuRg== + haSharedSecret: cmRzbzQ4N3RQaWhuMk00OA== proxyPassword: "" proxyUsername: "" kind: Secret @@ -1248,7 +1248,7 @@ spec: metadata: annotations: checksum/cluster-resource-templates: 6fd9b172465e3089fcc59f738b92b8dc4d8939360c19de8ee65f68b0e7422035 - checksum/configuration: f746817691b502fb50f04992e148847f89c0a32d7df822dbda7f1f5fdf84f420 + checksum/configuration: dc6e26fec37cad413a92bf06f2840ea1e497284312275ff06e22b152dee1566b checksum/configuration-secret: 09216ffaa3d29e14f88b1f30af580d02a2a5e014de4d750b7f275cc07ed4e914 labels: app.kubernetes.io/component: flyte-binary @@ -1414,7 +1414,7 @@ spec: metadata: annotations: checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81 - checksum/secret: 042e6b21a3852a65952e0701cd9667e53bfef57590eea4d116b261472f29a882 + checksum/secret: 51528951e92c2bf712bbde990941593aae1fcf72144a1fe944c312ddad86e161 labels: app: docker-registry release: flyte-sandbox @@ -1757,7 +1757,7 @@ spec: value: minio - name: FLYTE_AWS_SECRET_ACCESS_KEY value: miniostorage - image: cr.flyte.org/flyteorg/flyteagent:1.13.4 + image: cr.flyte.org/flyteorg/flyteagent:1.13.6 imagePullPolicy: IfNotPresent name: flyteagent ports: diff --git a/docker/sandbox-bundled/manifests/complete.yaml b/docker/sandbox-bundled/manifests/complete.yaml index fbb4295f25..d5aaaadf7f 100644 --- a/docker/sandbox-bundled/manifests/complete.yaml +++ b/docker/sandbox-bundled/manifests/complete.yaml @@ -460,7 +460,7 @@ data: stackdriver-enabled: false k8s: co-pilot: - image: "cr.flyte.org/flyteorg/flytecopilot:v1.13.1" + image: "cr.flyte.org/flyteorg/flytecopilot:v1.13.2" k8s-array: logs: config: @@ -799,7 +799,7 @@ type: Opaque --- apiVersion: v1 data: - haSharedSecret: Q2EyanRtd1JjWmVKS2tHMw== + haSharedSecret: T1I2Q2tTcmREVG15MldGUQ== proxyPassword: "" proxyUsername: "" kind: Secret @@ -1197,7 +1197,7 @@ spec: metadata: annotations: checksum/cluster-resource-templates: 6fd9b172465e3089fcc59f738b92b8dc4d8939360c19de8ee65f68b0e7422035 - checksum/configuration: 967349c227efb6765bb7509d14dc7b0d62b07904a337dd70c8682d52d870590a + checksum/configuration: a6f3ea502338c626b7824453ce7dc8b6fcd441d68865c075e2e74d797bc607fa checksum/configuration-secret: 09216ffaa3d29e14f88b1f30af580d02a2a5e014de4d750b7f275cc07ed4e914 labels: app.kubernetes.io/component: flyte-binary @@ -1363,7 +1363,7 @@ spec: metadata: annotations: checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81 - checksum/secret: 01201329c98a1417f04feeef00dc21e67cf73d99ac9b99486ce5788eca0c282c + checksum/secret: d723e395edc0fd2f221b9088efffe0d1f4dfabdef9892065fdabe12233362cf5 labels: app: docker-registry release: flyte-sandbox diff --git a/docker/sandbox-bundled/manifests/dev.yaml b/docker/sandbox-bundled/manifests/dev.yaml index fe04e4c059..d87f3a1642 100644 --- a/docker/sandbox-bundled/manifests/dev.yaml +++ b/docker/sandbox-bundled/manifests/dev.yaml @@ -499,7 +499,7 @@ metadata: --- apiVersion: v1 data: - haSharedSecret: N3dIemE2TnF1b3l1SWdNTw== + haSharedSecret: ZnltNHNiZ01NRFNkb1RlMA== proxyPassword: "" proxyUsername: "" kind: Secret @@ -934,7 +934,7 @@ spec: metadata: annotations: checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81 - checksum/secret: ca7423805c2fd3a98507c790af575a6e5389f50d6baa09bd8c49cb59c4452340 + checksum/secret: eeab364c20a0e8ad5a1526ccd7ddbd1d5a442087e7267c4d761279102b81be21 labels: app: docker-registry release: flyte-sandbox diff --git a/docs/conf.py b/docs/conf.py index 2be3b0185f..7ff7c90baf 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -36,7 +36,7 @@ # The short X.Y version version = "" # The full version, including alpha/beta/rc tags -release = "1.13.1" +release = "1.13.2" # -- General configuration --------------------------------------------------- diff --git a/docs/deployment/agents/chatgpt.rst b/docs/deployment/agents/chatgpt.rst index cb0b44fa39..9483d95d35 100644 --- a/docs/deployment/agents/chatgpt.rst +++ b/docs/deployment/agents/chatgpt.rst @@ -78,45 +78,25 @@ Specify agent configuration Add the OpenAI API token ------------------------------- -1. Install flyteagent pod using helm: +1. Install the flyteagent pod using helm: -.. code-block:: +.. code-block:: bash helm repo add flyteorg https://flyteorg.github.io/flyte helm install flyteagent flyteorg/flyteagent --namespace flyte -2. Get the base64 value of your OpenAI API token: +2. Set Your OpenAI API Token as a Secret (Base64 Encoded): -.. code-block:: +.. code-block:: bash - echo -n "" | base64 + SECRET_VALUE=$(echo -n "" | base64) && \ + kubectl patch secret flyteagent -n flyte --patch "{\"data\":{\"flyte_openai_api_key\":\"$SECRET_VALUE\"}}" -3. Edit the flyteagent secret: +3. Restart development: - .. code-block:: bash - - kubectl edit secret flyteagent -n flyte - - .. code-block:: yaml - :emphasize-lines: 3 - - apiVersion: v1 - data: - flyte_openai_api_key: - kind: Secret - metadata: - annotations: - meta.helm.sh/release-name: flyteagent - meta.helm.sh/release-namespace: flyte - creationTimestamp: "2023-10-04T04:09:03Z" - labels: - app.kubernetes.io/managed-by: Helm - name: flyteagent - namespace: flyte - resourceVersion: "753" - uid: 5ac1e1b6-2a4c-4e26-9001-d4ba72c39e54 - type: Opaque +.. code-block:: bash + kubectl rollout restart deployment flyteagent -n flyte Upgrade the Flyte Helm release ------------------------------ diff --git a/docs/deployment/agents/databricks.rst b/docs/deployment/agents/databricks.rst index 0458fb3667..b419144021 100644 --- a/docs/deployment/agents/databricks.rst +++ b/docs/deployment/agents/databricks.rst @@ -147,7 +147,7 @@ Specify agent configuration container: container container_array: k8s-array sidecar: sidecar - spark: agent-service + databricks: agent-service enabled-plugins: - container - sidecar @@ -171,7 +171,7 @@ Specify agent configuration default-for-task-types: - container: container - container_array: k8s-array - - spark: agent-service + - databricks: agent-service .. group-tab:: Flyte core @@ -192,52 +192,32 @@ Specify agent configuration container: container sidecar: sidecar container_array: k8s-array - spark: agent-service + databricks: agent-service Add the Databricks access token ------------------------------- You have to set the Databricks token to the Flyte configuration. -1. Install flyteagent pod using helm +1. Install the flyteagent pod using helm .. code-block:: helm repo add flyteorg https://flyteorg.github.io/flyte helm install flyteagent flyteorg/flyteagent --namespace flyte -2. Get the base64 value of your Databricks token. +2. Set Your Databricks Token as a Secret (Base64 Encoded): -.. code-block:: +.. code-block:: bash - echo -n "" | base64 + SECRET_VALUE=$(echo -n "" | base64) && \ + kubectl patch secret flyteagent -n flyte --patch "{\"data\":{\"flyte_databricks_access_token\":\"$SECRET_VALUE\"}}" -3. Edit the flyteagent secret - - .. code-block:: bash - - kubectl edit secret flyteagent -n flyte - - .. code-block:: yaml - :emphasize-lines: 3 - - apiVersion: v1 - data: - flyte_databricks_access_token: - kind: Secret - metadata: - annotations: - meta.helm.sh/release-name: flyteagent - meta.helm.sh/release-namespace: flyte - creationTimestamp: "2023-10-04T04:09:03Z" - labels: - app.kubernetes.io/managed-by: Helm - name: flyteagent - namespace: flyte - resourceVersion: "753" - uid: 5ac1e1b6-2a4c-4e26-9001-d4ba72c39e54 - type: Opaque +3. Restart development: + +.. code-block:: bash + kubectl rollout restart deployment flyteagent -n flyte Upgrade the deployment ---------------------- diff --git a/docs/deployment/agents/openai_batch.rst b/docs/deployment/agents/openai_batch.rst index 2cfa70471a..8e1c622b73 100644 --- a/docs/deployment/agents/openai_batch.rst +++ b/docs/deployment/agents/openai_batch.rst @@ -65,38 +65,18 @@ Add the OpenAI API token helm repo add flyteorg https://flyteorg.github.io/flyte helm install flyteagent flyteorg/flyteagent --namespace flyte -2. Get the base64 value of your OpenAI API token: +2. Set Your OpenAI API Token as a Secret (Base64 Encoded): -.. code-block:: - - echo -n "" | base64 +.. code-block:: bash -3. Edit the flyteagent secret: + SECRET_VALUE=$(echo -n "" | base64) && \ + kubectl patch secret flyteagent -n flyte --patch "{\"data\":{\"flyte_openai_api_key\":\"$SECRET_VALUE\"}}" - .. code-block:: bash +3. Restart development: - kubectl edit secret flyteagent -n flyte - - .. code-block:: yaml - :emphasize-lines: 3 - - apiVersion: v1 - data: - FLYTE_OPENAI_API_KEY: - kind: Secret - metadata: - annotations: - meta.helm.sh/release-name: flyteagent - meta.helm.sh/release-namespace: flyte - creationTimestamp: "2023-10-04T04:09:03Z" - labels: - app.kubernetes.io/managed-by: Helm - name: flyteagent - namespace: flyte - resourceVersion: "753" - uid: 5ac1e1b6-2a4c-4e26-9001-d4ba72c39e54 - type: Opaque +.. code-block:: bash + kubectl rollout restart deployment flyteagent -n flyte Upgrade the Flyte Helm release ------------------------------ diff --git a/docs/deployment/configuration/generated/flyteadmin_config.rst b/docs/deployment/configuration/generated/flyteadmin_config.rst index 0912738015..c8bbe4bd32 100644 --- a/docs/deployment/configuration/generated/flyteadmin_config.rst +++ b/docs/deployment/configuration/generated/flyteadmin_config.rst @@ -2383,6 +2383,11 @@ emailer (`interfaces.NotificationsEmailerConfig`_) apiKeyEnvVar: "" apiKeyFilePath: "" serviceName: "" + smtpPasswordSecretName: "" + smtpPort: "" + smtpServer: "" + smtpSkipTLSVerify: false + smtpUsername: "" sender: "" subject: "" @@ -2420,6 +2425,11 @@ emailServerConfig (`interfaces.EmailServerConfig`_) apiKeyEnvVar: "" apiKeyFilePath: "" serviceName: "" + smtpPasswordSecretName: "" + smtpPort: "" + smtpServer: "" + smtpSkipTLSVerify: false + smtpUsername: "" subject (string) @@ -2485,6 +2495,56 @@ apiKeyFilePath (string) "" +smtpServer (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + +smtpPort (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + +smtpSkipTLSVerify (bool) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "false" + + +smtpUsername (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + +smtpPasswordSecretName (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + interfaces.NotificationsProcessorConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -3898,6 +3958,22 @@ Configuration for array nodes event-version: 0 +literal-offloading-config (`config.LiteralOffloadingConfig`_) +------------------------------------------------------------------------------------------------------------------------ + +config used for literal offloading. + +**Default Value**: + +.. code-block:: yaml + + Enabled: false + max-size-in-mb-for-offloading: 1000 + min-size-in-mb-for-offloading: 10 + supported-sdk-versions: + FLYTE_SDK: 1.13.5 + + config.ArrayNodeConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -4218,6 +4294,55 @@ Name (string) "" +config.LiteralOffloadingConfig +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Enabled (bool) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "false" + + +supported-sdk-versions (map[string]string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Maps flytekit and union SDK names to minimum supported version that can handle reading offloaded literals. + +**Default Value**: + +.. code-block:: yaml + + FLYTE_SDK: 1.13.5 + + +min-size-in-mb-for-offloading (int64) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Size of a literal at which to trigger offloading + +**Default Value**: + +.. code-block:: yaml + + "10" + + +max-size-in-mb-for-offloading (int64) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Size of a literal at which to fail fast + +**Default Value**: + +.. code-block:: yaml + + "1000" + + config.NodeConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/deployment/configuration/generated/flytepropeller_config.rst b/docs/deployment/configuration/generated/flytepropeller_config.rst index 554ebec849..fc78d202ca 100644 --- a/docs/deployment/configuration/generated/flytepropeller_config.rst +++ b/docs/deployment/configuration/generated/flytepropeller_config.rst @@ -994,6 +994,9 @@ agent-service (`agent.Config`_) Value: 50 ProjectScopeResourceConstraint: Value: 100 + supportedTaskTypes: + - task_type_1 + - task_type_2 webApi: caching: maxSystemFailures: 5 @@ -4587,6 +4590,22 @@ Configuration for array nodes event-version: 0 +literal-offloading-config (`config.LiteralOffloadingConfig`_) +------------------------------------------------------------------------------------------------------------------------ + +config used for literal offloading. + +**Default Value**: + +.. code-block:: yaml + + Enabled: false + max-size-in-mb-for-offloading: 1000 + min-size-in-mb-for-offloading: 10 + supported-sdk-versions: + FLYTE_SDK: 1.13.5 + + admin-launcher (`launchplan.AdminConfig`_) ------------------------------------------------------------------------------------------------------------------------ @@ -5053,6 +5072,55 @@ Name (string) "" +config.LiteralOffloadingConfig +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Enabled (bool) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "false" + + +supported-sdk-versions (map[string]string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Maps flytekit and union SDK names to minimum supported version that can handle reading offloaded literals. + +**Default Value**: + +.. code-block:: yaml + + FLYTE_SDK: 1.13.5 + + +min-size-in-mb-for-offloading (int64) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Size of a literal at which to trigger offloading + +**Default Value**: + +.. code-block:: yaml + + "10" + + +max-size-in-mb-for-offloading (int64) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Size of a literal at which to fail fast + +**Default Value**: + +.. code-block:: yaml + + "1000" + + config.NodeConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/deployment/configuration/generated/scheduler_config.rst b/docs/deployment/configuration/generated/scheduler_config.rst index 98ff1ee343..8904155e7c 100644 --- a/docs/deployment/configuration/generated/scheduler_config.rst +++ b/docs/deployment/configuration/generated/scheduler_config.rst @@ -2383,6 +2383,11 @@ emailer (`interfaces.NotificationsEmailerConfig`_) apiKeyEnvVar: "" apiKeyFilePath: "" serviceName: "" + smtpPasswordSecretName: "" + smtpPort: "" + smtpServer: "" + smtpSkipTLSVerify: false + smtpUsername: "" sender: "" subject: "" @@ -2420,6 +2425,11 @@ emailServerConfig (`interfaces.EmailServerConfig`_) apiKeyEnvVar: "" apiKeyFilePath: "" serviceName: "" + smtpPasswordSecretName: "" + smtpPort: "" + smtpServer: "" + smtpSkipTLSVerify: false + smtpUsername: "" subject (string) @@ -2485,6 +2495,56 @@ apiKeyFilePath (string) "" +smtpServer (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + +smtpPort (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + +smtpSkipTLSVerify (bool) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "false" + + +smtpUsername (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + +smtpPasswordSecretName (string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "" + + interfaces.NotificationsProcessorConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -3898,6 +3958,22 @@ Configuration for array nodes event-version: 0 +literal-offloading-config (`config.LiteralOffloadingConfig`_) +------------------------------------------------------------------------------------------------------------------------ + +config used for literal offloading. + +**Default Value**: + +.. code-block:: yaml + + Enabled: false + max-size-in-mb-for-offloading: 1000 + min-size-in-mb-for-offloading: 10 + supported-sdk-versions: + FLYTE_SDK: 1.13.5 + + config.ArrayNodeConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -4218,6 +4294,55 @@ Name (string) "" +config.LiteralOffloadingConfig +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Enabled (bool) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +**Default Value**: + +.. code-block:: yaml + + "false" + + +supported-sdk-versions (map[string]string) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Maps flytekit and union SDK names to minimum supported version that can handle reading offloaded literals. + +**Default Value**: + +.. code-block:: yaml + + FLYTE_SDK: 1.13.5 + + +min-size-in-mb-for-offloading (int64) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Size of a literal at which to trigger offloading + +**Default Value**: + +.. code-block:: yaml + + "10" + + +max-size-in-mb-for-offloading (int64) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Size of a literal at which to fail fast + +**Default Value**: + +.. code-block:: yaml + + "1000" + + config.NodeConfig ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/flyte_agents/deploying_agents_to_the_flyte_sandbox.md b/docs/flyte_agents/deploying_agents_to_the_flyte_sandbox.md index c4f1a2881e..2f068eb681 100644 --- a/docs/flyte_agents/deploying_agents_to_the_flyte_sandbox.md +++ b/docs/flyte_agents/deploying_agents_to_the_flyte_sandbox.md @@ -52,30 +52,8 @@ image: localhost:30000/flyteagent:example 3. Set up your secrets: Let's take Databricks agent as an example: ```bash -kubectl edit secret flyteagent -n flyte -``` -Get your `BASE64_ENCODED_DATABRICKS_TOKEN`: -```bash -echo -n "" | base64 -``` -Add your token to the `data` field: -```yaml -apiVersion: v1 -data: - flyte_databricks_access_token: -kind: Secret -metadata: - annotations: - meta.helm.sh/release-name: flyteagent - meta.helm.sh/release-namespace: flyte - creationTimestamp: "2023-10-04T04:09:03Z" - labels: - app.kubernetes.io/managed-by: Helm - name: flyteagent - namespace: flyte - resourceVersion: "753" - uid: 5ac1e1b6-2a4c-4e26-9001-d4ba72c39e54 -type: Opaque +SECRET_VALUE=$(echo -n "" | base64) && \ +kubectl patch secret flyteagent -n flyte --patch "{\"data\":{\"flyte_databricks_access_token\":\"$SECRET_VALUE\"}}" ``` :::{note} Please ensure two things: @@ -85,7 +63,7 @@ Please ensure two things: 4. Restart development: ```bash -kubectl rollout restart deployment flyte-sandbox -n flyte +kubectl rollout restart deployment flyteagent -n flyte ``` 5. Test your agent remotely in the Flyte sandbox: diff --git a/docs/user_guide/customizing_dependencies/imagespec.md b/docs/user_guide/customizing_dependencies/imagespec.md index ccdd52fe28..d9bf8f24bf 100644 --- a/docs/user_guide/customizing_dependencies/imagespec.md +++ b/docs/user_guide/customizing_dependencies/imagespec.md @@ -6,9 +6,13 @@ .. tags:: Containerization, Intermediate ``` -`ImageSpec` is a way to specify how to build a container image without a Dockerfile. The `ImageSpec` by default will be -converted to an [Envd](https://envd.tensorchord.ai/) config, and the [Envd builder](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-envd/flytekitplugins/envd/image_builder.py#L12-L34) will build the image for you. However, you can also register your own builder to build -the image using other tools. + +`ImageSpec` allows you to customize the container image for your Flyte tasks without a Dockerfile. +`ImageSpec` speeds up the build process by allowing you to reuse previously downloaded packages from the PyPI and APT caches. + +By default, the `ImageSpec` will be built using the `default` builder associated with Flytekit, but you can register your own builder. + +For example, [flytekitplugins-envd](https://github.com/flyteorg/flytekit/blob/c06ef30518dec2057e554fbed375dfa43b985c60/plugins/flytekit-envd/flytekitplugins/envd/image_builder.py#L25) is another image builder that uses envd to build the ImageSpec. For every {py:class}`flytekit.PythonFunctionTask` task or a task decorated with the `@task` decorator, you can specify rules for binding container images. By default, flytekit binds a single container image, i.e., @@ -16,58 +20,164 @@ the [default Docker image](https://ghcr.io/flyteorg/flytekit), to all tasks. To use the `container_image` parameter available in the {py:func}`flytekit.task` decorator, and pass an `ImageSpec`. -Before building the image, Flytekit checks the container registry first to see if the image already exists. By doing so, it avoids having to rebuild the image over and over again. If the image does not exist, flytekit will build the image before registering the workflow, and replace the image name in the task template with the newly built image name. - -```{note} -To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. -``` - -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py -:caption: customizing_dependencies/image_spec.py -:lines: 1-4 -``` +Before building the image, Flytekit checks the container registry to see if the image already exists. +If the image does not exist, +Flytekit will build the image before registering the workflow and replace the image name in the task template with the newly built image name. :::{admonition} Prerequisites :class: important -- Install [flytekitplugins-envd](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-envd) to build the `ImageSpec`. -- To build the image on remote machine, check this [doc](https://envd.tensorchord.ai/teams/context.html#start-remote-buildkitd-on-builder-machine). +- Make sure `docker` is running on your local machine. - When using a registry in ImageSpec, `docker login` is required to push the image ::: -You can specify python packages, apt packages, and environment variables in the `ImageSpec`. +## Install Python or APT packages +You can specify Python packages and APT packages in the `ImageSpec`. These specified packages will be added on top of the [default image](https://github.com/flyteorg/flytekit/blob/master/Dockerfile), which can be found in the Flytekit Dockerfile. More specifically, flytekit invokes [DefaultImages.default_image()](https://github.com/flyteorg/flytekit/blob/f2cfef0ec098d4ae8f042ab915b0b30d524092c6/flytekit/configuration/default_images.py#L26-L27) function. -This function determines and returns the default image based on the Python version and flytekit version. For example, if you are using python 3.8 and flytekit 0.16.0, the default image assigned will be `ghcr.io/flyteorg/flytekit:py3.8-1.6.0`. -If desired, you can also override the default image by providing a custom `base_image` parameter when using the `ImageSpec`. - -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py -:caption: customizing_dependencies/image_spec.py -:lines: 6-19 -``` +This function determines and returns the default image based on the Python version and flytekit version. +For example, if you are using Python 3.8 and flytekit 1.6.0, the default image assigned will be `ghcr.io/flyteorg/flytekit:py3.8-1.6.0`. :::{important} Replace `ghcr.io/flyteorg` with a container registry you can publish to. To upload the image to the local registry in the demo cluster, indicate the registry as `localhost:30000`. ::: -`is_container` is used to determine whether the task is utilizing the image constructed from the `ImageSpec`. -If the task is indeed using the image built from the `ImageSpec`, it will then import Tensorflow. +```python +from flytekit import ImageSpec + +sklearn_image_spec = ImageSpec( + packages=["scikit-learn", "tensorflow==2.5.0"], + apt_packages=["curl", "wget"], + registry="ghcr.io/flyteorg", +) +``` + +## Install Conda packages +Define the ImageSpec to install packages from a specific conda channel. +```python +image_spec = ImageSpec( + conda_packages=["langchain"], + conda_channels=["conda-forge"], # List of channels to pull packages from. + registry="ghcr.io/flyteorg", +) +``` + +## Use different Python versions in the image +You can specify the Python version in the `ImageSpec` to build the image with a different Python version. + +```python +image_spec = ImageSpec( + packages=["pandas"], + python_version="3.9", + registry="ghcr.io/flyteorg", +) +``` + +## Import modules only in a specify imageSpec environment + +`is_container()` is used to determine whether the task is utilizing the image constructed from the `ImageSpec`. +If the task is indeed using the image built from the `ImageSpec`, it will return true. This approach helps minimize module loading time and prevents unnecessary dependency installation within a single image. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py -:caption: customizing_dependencies/image_spec.py -:lines: 21-22 +In the following example, both `task1` and `task2` will import the `pandas` module. However, `Tensorflow` will only be imported in `task2`. + +```python +from flytekit import ImageSpec, task +import pandas as pd + +pandas_image_spec = ImageSpec( + packages=["pandas"], + registry="ghcr.io/flyteorg", +) + +tensorflow_image_spec = ImageSpec( + packages=["tensorflow", "pandas"], + registry="ghcr.io/flyteorg", +) + +# Return if and only if the task is using the image built from tensorflow_image_spec. +if tensorflow_image_spec.is_container(): + import tensorflow as tf + +@task(container_image=pandas_image_spec) +def task1() -> pd.DataFrame: + return pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [1, 22]}) + + +@task(container_image=tensorflow_image_spec) +def task2() -> int: + num_gpus = len(tf.config.list_physical_devices('GPU')) + print("Num GPUs Available: ", num_gpus) + return num_gpus +``` + +## Install CUDA in the image +There are few ways to install CUDA in the image. + +### Use Nvidia docker image +CUDA is pre-installed in the Nvidia docker image. You can specify the base image in the `ImageSpec`. +```python +image_spec = ImageSpec( + base_image="nvidia/cuda:12.6.1-cudnn-devel-ubuntu22.04", + packages=["tensorflow", "pandas"], + python_version="3.9", + registry="ghcr.io/flyteorg", +) ``` -To enable tasks to utilize the images built with `ImageSpec`, you can specify the `container_image` parameter for those tasks. +### Install packages from extra index +CUDA can be installed by specifying the `pip_extra_index_url` in the `ImageSpec`. +```python +image_spec = ImageSpec( + name="pytorch-mnist", + packages=["torch", "torchvision", "flytekitplugins-kfpytorch"], + pip_extra_index_url=["https://download.pytorch.org/whl/cu118"], + registry="ghcr.io/flyteorg", +) +``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/customizing_dependencies/customizing_dependencies/image_spec.py -:caption: customizing_dependencies/image_spec.py -:lines: 27-56 +## Build an image in different architecture +You can specify the platform in the `ImageSpec` to build the image in a different architecture, such as `linux/arm64` or `darwin/arm64`. +```python +image_spec = ImageSpec( + packages=["pandas"], + platform="linux/arm64", + registry="ghcr.io/flyteorg", +) ``` -There exists an option to override the container image by providing an Image Spec YAML file to the `pyflyte run` or `pyflyte register` command. +## Install flytekit from GitHub +When you update the flytekit, you may want to test the changes with your tasks. +You can install the flytekit from a specific commit hash in the `ImageSpec`. + +```python +new_flytekit = "git+https://github.com/flyteorg/flytekit@90a4455c2cc2b3e171dfff69f605f47d48ea1ff1" +new_spark_plugins = f"git+https://github.com/flyteorg/flytekit.git@90a4455c2cc2b3e171dfff69f605f47d48ea1ff1#subdirectory=plugins/flytekit-spark" + +image_spec = ImageSpec( + apt_packages=["git"], + packages=[new_flytekit, new_spark_plugins], + registry="ghcr.io/flyteorg", +) +``` + +## Customize the tag of the image +You can customize the tag of the image by specifying the `tag_format` in the `ImageSpec`. +In the following example, the full qualified image name will be `ghcr.io/flyteorg/my-image:-dev`. + +```python +image_spec = ImageSpec( + name="my-image", + packages=["pandas"], + tag_format="{spec_hash}-dev", + registry="ghcr.io/flyteorg", +) +``` + +## Define ImageSpec in a YAML File + +You can override the container image by providing an ImageSpec YAML file to the `pyflyte run` or `pyflyte register` command. This allows for greater flexibility in specifying a custom container image. For example: ```yaml @@ -85,19 +195,24 @@ env: pyflyte run --remote --image image.yaml image_spec.py wf ``` +## Build the image without registering the workflow + If you only want to build the image without registering the workflow, you can use the `pyflyte build` command. ``` pyflyte build --remote image_spec.py wf ``` -In some cases, you may want to force an image to rebuild, even if the image spec hasn’t changed. If you want to overwrite an existing image, you can pass the `FLYTE_FORCE_PUSH_IMAGE_SPEC=True` to `pyflyte` command or add `force_push()` to the ImageSpec. +## Force push an image + +In some cases, you may want to force an image to rebuild, even if the ImageSpec hasn’t changed. +To overwrite an existing image, pass the `FLYTE_FORCE_PUSH_IMAGE_SPEC=True` to the `pyflyte` command. ```bash FLYTE_FORCE_PUSH_IMAGE_SPEC=True pyflyte run --remote image_spec.py wf ``` -or +You can also force push an image in the Python code by calling the `force_push()` method. ```python image = ImageSpec(registry="ghcr.io/flyteorg", packages=["pandas"]).force_push() diff --git a/docs/user_guide/development_lifecycle/decks.md b/docs/user_guide/development_lifecycle/decks.md index c59cbd1945..366302d49e 100644 --- a/docs/user_guide/development_lifecycle/decks.md +++ b/docs/user_guide/development_lifecycle/decks.md @@ -39,9 +39,9 @@ We create a new deck named `pca` and render Markdown content along with a You can begin by initializing an {ref}`ImageSpec ` object to encompass all the necessary dependencies. This approach automatically triggers a Docker build, alleviating the need for you to manually create a Docker image. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py -:lines: 15-19 +:lines: 15-27 ``` :::{important} @@ -96,9 +96,9 @@ When the task connected with a deck object is executed, these objects employ ren Creates a profile report from a Pandas DataFrame. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py -:lines: 44-51 +:lines: 56-63 ``` :::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_frame_renderer.png @@ -113,9 +113,9 @@ Creates a profile report from a Pandas DataFrame. Renders DataFrame as an HTML table. This renderer doesn't necessitate plugin installation since it's accessible within the flytekit library. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py -:lines: 57-64 +:lines: 69-76 ``` :::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_top_frame_renderer.png @@ -147,9 +147,9 @@ The median (Q2) is indicated by a line within the box. Typically, the whiskers extend to the edges of the box, plus or minus 1.5 times the interquartile range (IQR: Q3-Q1). -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py -:lines: 85-91 +:lines: 97-103 ``` :::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_box_renderer.png @@ -162,9 +162,9 @@ plus or minus 1.5 times the interquartile range (IQR: Q3-Q1). Converts a {ref}`FlyteFile ` or `PIL.Image.Image` object into an HTML string, where the image data is encoded as a base64 string. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py -:lines: 97-111 +:lines: 109-123 ``` :::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_image_renderer.png @@ -176,9 +176,9 @@ where the image data is encoded as a base64 string. Converts a Pandas dataframe into an HTML table. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py -:lines: 115-123 +:lines: 127-135 ``` :::{figure} https://raw.githubusercontent.com/flyteorg/static-resources/main/flytesnacks/user_guide/flyte_decks_table_renderer.png diff --git a/flyte-single-binary-local.yaml b/flyte-single-binary-local.yaml index 7dae04df58..3251815694 100644 --- a/flyte-single-binary-local.yaml +++ b/flyte-single-binary-local.yaml @@ -90,6 +90,8 @@ storage: access_key_id: minio secret_key: miniostorage container: my-s3-bucket + limits: + maxDownloadMBs: 1000 task_resources: defaults: diff --git a/flyteadmin/go.mod b/flyteadmin/go.mod index cfc2bfa010..836bc69979 100644 --- a/flyteadmin/go.mod +++ b/flyteadmin/go.mod @@ -48,7 +48,6 @@ require ( github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.9.0 github.com/wI2L/jsondiff v0.5.0 - github.com/wolfeidau/humanhash v1.1.0 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go.opentelemetry.io/otel v1.24.0 golang.org/x/net v0.27.0 diff --git a/flyteadmin/go.sum b/flyteadmin/go.sum index 31a1714ed7..7c9c02881f 100644 --- a/flyteadmin/go.sum +++ b/flyteadmin/go.sum @@ -1301,8 +1301,6 @@ github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6Kllzaw github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= github.com/wI2L/jsondiff v0.5.0 h1:RRMTi/mH+R2aXcPe1VYyvGINJqQfC3R+KSEakuU1Ikw= github.com/wI2L/jsondiff v0.5.0/go.mod h1:qqG6hnK0Lsrz2BpIVCxWiK9ItsBCpIZQiv0izJjOZ9s= -github.com/wolfeidau/humanhash v1.1.0 h1:06KgtyyABJGBbrfMONrW7S+b5TTYVyrNB/jss5n7F3E= -github.com/wolfeidau/humanhash v1.1.0/go.mod h1:jkpynR1bfyfkmKEQudIC0osWKynFAoayRjzH9OJdVIg= github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c/go.mod h1:lB8K/P019DLNhemzwFU4jHLhdvlE6uDZjXFejJXr49I= github.com/xdg/stringprep v0.0.0-20180714160509-73f8eece6fdc/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y= github.com/xdg/stringprep v1.0.0/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y= diff --git a/flyteadmin/pkg/async/schedule/aws/workflow_executor.go b/flyteadmin/pkg/async/schedule/aws/workflow_executor.go index 523fdd077e..c4a5d75d14 100644 --- a/flyteadmin/pkg/async/schedule/aws/workflow_executor.go +++ b/flyteadmin/pkg/async/schedule/aws/workflow_executor.go @@ -15,7 +15,7 @@ import ( "github.com/flyteorg/flyte/flyteadmin/pkg/async" scheduleInterfaces "github.com/flyteorg/flyte/flyteadmin/pkg/async/schedule/interfaces" - "github.com/flyteorg/flyte/flyteadmin/pkg/common/naming" + "github.com/flyteorg/flyte/flyteadmin/pkg/common" "github.com/flyteorg/flyte/flyteadmin/pkg/errors" "github.com/flyteorg/flyte/flyteadmin/pkg/manager/interfaces" runtimeInterfaces "github.com/flyteorg/flyte/flyteadmin/pkg/runtime/interfaces" @@ -129,7 +129,7 @@ func generateExecutionName(launchPlan *admin.LaunchPlan, kickoffTime time.Time) Name: launchPlan.Id.Name, }) randomSeed := kickoffTime.UnixNano() + int64(hashedIdentifier) - return naming.GetExecutionName(randomSeed) + return common.GetExecutionName(randomSeed) } func (e *workflowExecutor) formulateExecutionCreateRequest( @@ -207,6 +207,7 @@ func (e *workflowExecutor) run() error { continue } executionRequest := e.formulateExecutionCreateRequest(launchPlan, scheduledWorkflowExecutionRequest.KickoffTime) + ctx = contextutils.WithWorkflowID(ctx, fmt.Sprintf(workflowIdentifierFmt, executionRequest.Project, executionRequest.Domain, executionRequest.Name)) err = e.resolveKickoffTimeArg(scheduledWorkflowExecutionRequest, launchPlan, executionRequest) diff --git a/flyteadmin/pkg/common/executions.go b/flyteadmin/pkg/common/executions.go index 4ac1ec7300..fbb5bdd6bd 100644 --- a/flyteadmin/pkg/common/executions.go +++ b/flyteadmin/pkg/common/executions.go @@ -1,9 +1,22 @@ package common import ( + "fmt" + + "k8s.io/apimachinery/pkg/util/rand" + "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core" ) +const ExecutionIDLength = 20 +const ExecutionStringFormat = "a%s" + +/* #nosec */ +func GetExecutionName(seed int64) string { + rand.Seed(seed) + return fmt.Sprintf(ExecutionStringFormat, rand.String(ExecutionIDLength-1)) +} + var terminalExecutionPhases = map[core.WorkflowExecution_Phase]bool{ core.WorkflowExecution_SUCCEEDED: true, core.WorkflowExecution_FAILED: true, diff --git a/flyteadmin/pkg/common/executions_test.go b/flyteadmin/pkg/common/executions_test.go new file mode 100644 index 0000000000..628abd6e9d --- /dev/null +++ b/flyteadmin/pkg/common/executions_test.go @@ -0,0 +1,23 @@ +package common + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" +) + +const AllowedExecutionIDStartCharStr = "abcdefghijklmnopqrstuvwxyz" +const AllowedExecutionIDStr = "abcdefghijklmnopqrstuvwxyz1234567890" + +var AllowedExecutionIDStartChars = []rune(AllowedExecutionIDStartCharStr) +var AllowedExecutionIDChars = []rune(AllowedExecutionIDStr) + +func TestGetExecutionName(t *testing.T) { + randString := GetExecutionName(time.Now().UnixNano()) + assert.Len(t, randString, ExecutionIDLength) + assert.Contains(t, AllowedExecutionIDStartChars, rune(randString[0])) + for i := 1; i < len(randString); i++ { + assert.Contains(t, AllowedExecutionIDChars, rune(randString[i])) + } +} diff --git a/flyteadmin/pkg/common/naming/execution_name.go b/flyteadmin/pkg/common/naming/execution_name.go deleted file mode 100644 index 01aa3fe8b6..0000000000 --- a/flyteadmin/pkg/common/naming/execution_name.go +++ /dev/null @@ -1,30 +0,0 @@ -package naming - -import ( - "fmt" - - "github.com/wolfeidau/humanhash" - "k8s.io/apimachinery/pkg/util/rand" - - "github.com/flyteorg/flyte/flyteadmin/pkg/runtime" - runtimeInterfaces "github.com/flyteorg/flyte/flyteadmin/pkg/runtime/interfaces" -) - -const ExecutionIDLength = 20 -const ExecutionIDLengthLimit = 63 -const ExecutionStringFormat = "a%s" - -var configProvider runtimeInterfaces.ApplicationConfiguration = runtime.NewApplicationConfigurationProvider() - -/* #nosec */ -func GetExecutionName(seed int64) string { - rand.Seed(seed) - config := configProvider.GetTopLevelConfig() - if config.FeatureGates.EnableFriendlyNames { - hashKey := []byte(rand.String(ExecutionIDLength)) - // Ignoring the error as it's guaranteed hash key longer than result in this context. - result, _ := humanhash.Humanize(hashKey, 4) - return result - } - return fmt.Sprintf(ExecutionStringFormat, rand.String(ExecutionIDLength-1)) -} diff --git a/flyteadmin/pkg/common/naming/execution_name_test.go b/flyteadmin/pkg/common/naming/execution_name_test.go deleted file mode 100644 index 91f04d3bf1..0000000000 --- a/flyteadmin/pkg/common/naming/execution_name_test.go +++ /dev/null @@ -1,78 +0,0 @@ -package naming - -import ( - "context" - "strings" - "testing" - "time" - - "github.com/stretchr/testify/assert" - - runtimeInterfaces "github.com/flyteorg/flyte/flyteadmin/pkg/runtime/interfaces" - runtimeMocks "github.com/flyteorg/flyte/flyteadmin/pkg/runtime/mocks" - "github.com/flyteorg/flyte/flyteadmin/scheduler/identifier" - "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core" -) - -const AllowedExecutionIDAlphabetStr = "abcdefghijklmnopqrstuvwxyz" -const AllowedExecutionIDAlphanumericStr = "abcdefghijklmnopqrstuvwxyz1234567890" -const AllowedExecutionIDFriendlyNameStr = "abcdefghijklmnopqrstuvwxyz-" - -var AllowedExecutionIDAlphabets = []rune(AllowedExecutionIDAlphabetStr) -var AllowedExecutionIDAlphanumerics = []rune(AllowedExecutionIDAlphanumericStr) -var AllowedExecutionIDFriendlyNameChars = []rune(AllowedExecutionIDFriendlyNameStr) - -func TestGetExecutionName(t *testing.T) { - originalConfigProvider := configProvider - defer func() { configProvider = originalConfigProvider }() - - mockConfigProvider := &runtimeMocks.MockApplicationProvider{} - configProvider = mockConfigProvider - - t.Run("general name", func(t *testing.T) { - appConfig := runtimeInterfaces.ApplicationConfig{ - FeatureGates: runtimeInterfaces.FeatureGates{ - EnableFriendlyNames: false, - }, - } - mockConfigProvider.SetTopLevelConfig(appConfig) - - randString := GetExecutionName(time.Now().UnixNano()) - assert.Len(t, randString, ExecutionIDLength) - assert.Contains(t, AllowedExecutionIDAlphabets, rune(randString[0])) - for i := 1; i < len(randString); i++ { - assert.Contains(t, AllowedExecutionIDAlphanumerics, rune(randString[i])) - } - }) - - t.Run("friendly name", func(t *testing.T) { - appConfig := runtimeInterfaces.ApplicationConfig{ - FeatureGates: runtimeInterfaces.FeatureGates{ - EnableFriendlyNames: true, - }, - } - mockConfigProvider.SetTopLevelConfig(appConfig) - - randString := GetExecutionName(time.Now().UnixNano()) - assert.LessOrEqual(t, len(randString), ExecutionIDLengthLimit) - for i := 0; i < len(randString); i++ { - assert.Contains(t, AllowedExecutionIDFriendlyNameChars, rune(randString[i])) - } - hyphenCount := strings.Count(randString, "-") - assert.Equal(t, 3, hyphenCount, "FriendlyName should contain exactly three hyphens") - words := strings.Split(randString, "-") - assert.Equal(t, 4, len(words), "FriendlyName should be split into exactly four words") - }) - - t.Run("deterministic name", func(t *testing.T) { - hashValue := identifier.HashScheduledTimeStamp(context.Background(), &core.Identifier{ - Project: "Project", - Domain: "Domain", - Name: "Name", - Version: "Version", - }, time.Time{}) - - name := GetExecutionName(int64(hashValue)) - assert.Equal(t, name, "carpet-juliet-kentucky-kentucky") - }) -} diff --git a/flyteadmin/pkg/errors/errors.go b/flyteadmin/pkg/errors/errors.go index 0fd9542ba7..5fc48b0b67 100644 --- a/flyteadmin/pkg/errors/errors.go +++ b/flyteadmin/pkg/errors/errors.go @@ -213,3 +213,8 @@ func NewInactiveProjectError(ctx context.Context, id string) FlyteAdminError { } return statusErr } + +func NewInvalidLiteralTypeError(name string, err error) FlyteAdminError { + return NewFlyteAdminErrorf(codes.InvalidArgument, + fmt.Sprintf("Failed to validate literal type for [%s] with err: %s", name, err)) +} diff --git a/flyteadmin/pkg/manager/impl/util/shared.go b/flyteadmin/pkg/manager/impl/util/shared.go index ba8fc41760..8402451200 100644 --- a/flyteadmin/pkg/manager/impl/util/shared.go +++ b/flyteadmin/pkg/manager/impl/util/shared.go @@ -8,7 +8,6 @@ import ( "google.golang.org/grpc/codes" "github.com/flyteorg/flyte/flyteadmin/pkg/common" - "github.com/flyteorg/flyte/flyteadmin/pkg/common/naming" "github.com/flyteorg/flyte/flyteadmin/pkg/errors" "github.com/flyteorg/flyte/flyteadmin/pkg/manager/impl/shared" "github.com/flyteorg/flyte/flyteadmin/pkg/manager/impl/validation" @@ -26,7 +25,7 @@ func GetExecutionName(request *admin.ExecutionCreateRequest) string { if request.Name != "" { return request.Name } - return naming.GetExecutionName(time.Now().UnixNano()) + return common.GetExecutionName(time.Now().UnixNano()) } func GetTask(ctx context.Context, repo repoInterfaces.Repository, identifier *core.Identifier) ( diff --git a/flyteadmin/pkg/manager/impl/util/shared_test.go b/flyteadmin/pkg/manager/impl/util/shared_test.go index 114dbebdfb..b9b296971e 100644 --- a/flyteadmin/pkg/manager/impl/util/shared_test.go +++ b/flyteadmin/pkg/manager/impl/util/shared_test.go @@ -12,8 +12,8 @@ import ( "github.com/stretchr/testify/assert" "google.golang.org/grpc/codes" + "github.com/flyteorg/flyte/flyteadmin/pkg/common" commonMocks "github.com/flyteorg/flyte/flyteadmin/pkg/common/mocks" - "github.com/flyteorg/flyte/flyteadmin/pkg/common/naming" flyteAdminErrors "github.com/flyteorg/flyte/flyteadmin/pkg/errors" "github.com/flyteorg/flyte/flyteadmin/pkg/manager/impl/testutils" managerInterfaces "github.com/flyteorg/flyte/flyteadmin/pkg/manager/interfaces" @@ -42,7 +42,7 @@ func TestPopulateExecutionID(t *testing.T) { Domain: "domain", }) assert.NotEmpty(t, name) - assert.Len(t, name, naming.ExecutionIDLength) + assert.Len(t, name, common.ExecutionIDLength) } func TestPopulateExecutionID_ExistingName(t *testing.T) { diff --git a/flyteadmin/pkg/manager/impl/validation/execution_validator.go b/flyteadmin/pkg/manager/impl/validation/execution_validator.go index f7b385b8a8..f5fd30598a 100644 --- a/flyteadmin/pkg/manager/impl/validation/execution_validator.go +++ b/flyteadmin/pkg/manager/impl/validation/execution_validator.go @@ -100,12 +100,10 @@ func CheckAndFetchInputsForExecution( } executionInputMap[name] = expectedInput.GetDefault() } else { - var inputType *core.LiteralType - switch executionInputMap[name].GetValue().(type) { - case *core.Literal_OffloadedMetadata: - inputType = executionInputMap[name].GetOffloadedMetadata().GetInferredType() - default: - inputType = validators.LiteralTypeForLiteral(executionInputMap[name]) + inputType := validators.LiteralTypeForLiteral(executionInputMap[name]) + err := validators.ValidateLiteralType(inputType) + if err != nil { + return nil, errors.NewInvalidLiteralTypeError(name, err) } if !validators.AreTypesCastable(inputType, expectedInput.GetVar().GetType()) { return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "invalid %s input wrong type. Expected %s, but got %s", name, expectedInput.GetVar().GetType(), inputType) diff --git a/flyteadmin/pkg/manager/impl/validation/execution_validator_test.go b/flyteadmin/pkg/manager/impl/validation/execution_validator_test.go index 7e5f991788..943e5006e7 100644 --- a/flyteadmin/pkg/manager/impl/validation/execution_validator_test.go +++ b/flyteadmin/pkg/manager/impl/validation/execution_validator_test.go @@ -17,6 +17,8 @@ import ( var execConfig = testutils.GetApplicationConfigWithDefaultDomains() +const failedToValidateLiteralType = "Failed to validate literal type" + func TestValidateExecEmptyProject(t *testing.T) { request := testutils.GetExecutionRequest() request.Project = "" @@ -209,6 +211,42 @@ func TestValidateExecEmptyInputs(t *testing.T) { assert.EqualValues(t, expectedMap, actualInputs) } +func TestValidateExecUnknownIDLInputs(t *testing.T) { + unsupportedLiteral := &core.Literal{ + Value: &core.Literal_Scalar{ + Scalar: &core.Scalar{}, + }, + } + defaultInputs := &core.ParameterMap{ + Parameters: map[string]*core.Parameter{ + "foo": { + Var: &core.Variable{ + // 1000 means an unsupported type + Type: &core.LiteralType{Type: &core.LiteralType_Simple{Simple: 1000}}, + }, + Behavior: &core.Parameter_Default{ + Default: unsupportedLiteral, + }, + }, + }, + } + userInputs := &core.LiteralMap{ + Literals: map[string]*core.Literal{ + "foo": unsupportedLiteral, // This will lead to a nil inputType + }, + } + + _, err := CheckAndFetchInputsForExecution( + userInputs, + nil, + defaultInputs, + ) + assert.NotNil(t, err) + + // Expected error message + assert.Contains(t, err.Error(), failedToValidateLiteralType) +} + func TestValidExecutionId(t *testing.T) { err := CheckValidExecutionID("abcde123", "a") assert.Nil(t, err) diff --git a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go index 0a5e87fe99..2a49b4da87 100644 --- a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go +++ b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator.go @@ -143,6 +143,10 @@ func checkAndFetchExpectedInputForLaunchPlan( return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "unexpected fixed_input %s", name) } inputType := validators.LiteralTypeForLiteral(fixedInput) + err := validators.ValidateLiteralType(inputType) + if err != nil { + return nil, errors.NewInvalidLiteralTypeError(name, err) + } if !validators.AreTypesCastable(inputType, value.GetType()) { return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "invalid fixed_input wrong type %s, expected %v, got %v instead", name, value.GetType(), inputType) diff --git a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go index 3bad5110c5..ab2832eeeb 100644 --- a/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go +++ b/flyteadmin/pkg/manager/impl/validation/launch_plan_validator_test.go @@ -231,6 +231,50 @@ func TestGetLpExpectedInvalidFixedInput(t *testing.T) { assert.Nil(t, actualMap) } +func TestGetLpExpectedInvalidFixedInputWithUnknownIDL(t *testing.T) { + unsupportedLiteral := &core.Literal{ + Value: &core.Literal_Scalar{ + Scalar: &core.Scalar{}, + }, + } + workflowVariableMap := &core.VariableMap{ + Variables: map[string]*core.Variable{ + "foo": { + Type: &core.LiteralType{Type: &core.LiteralType_Simple{Simple: 1000}}, + }, + }, + } + defaultInputs := &core.ParameterMap{ + Parameters: map[string]*core.Parameter{ + "foo": { + Var: &core.Variable{ + // 1000 means an unsupported type + Type: &core.LiteralType{Type: &core.LiteralType_Simple{Simple: 1000}}, + }, + Behavior: &core.Parameter_Default{ + Default: unsupportedLiteral, + }, + }, + }, + } + fixedInputs := &core.LiteralMap{ + Literals: map[string]*core.Literal{ + "foo": unsupportedLiteral, // This will lead to a nil inputType + }, + } + + _, err := checkAndFetchExpectedInputForLaunchPlan( + workflowVariableMap, + fixedInputs, + defaultInputs, + ) + + assert.NotNil(t, err) + + // Expected error message + assert.Contains(t, err.Error(), failedToValidateLiteralType) +} + func TestGetLpExpectedNoFixedInput(t *testing.T) { request := testutils.GetLaunchPlanRequest() actualMap, err := checkAndFetchExpectedInputForLaunchPlan( diff --git a/flyteadmin/pkg/manager/impl/validation/signal_validator.go b/flyteadmin/pkg/manager/impl/validation/signal_validator.go index c5e49401b0..af1d4425aa 100644 --- a/flyteadmin/pkg/manager/impl/validation/signal_validator.go +++ b/flyteadmin/pkg/manager/impl/validation/signal_validator.go @@ -76,6 +76,10 @@ func ValidateSignalSetRequest(ctx context.Context, db repositoryInterfaces.Repos if err != nil { return err } + err = propellervalidators.ValidateLiteralType(valueType) + if err != nil { + return errors.NewInvalidLiteralTypeError("", err) + } if !propellervalidators.AreTypesCastable(lookupSignal.Type, valueType) { return errors.NewFlyteAdminErrorf(codes.InvalidArgument, "requested signal value [%v] is not castable to existing signal type [%v]", diff --git a/flyteadmin/pkg/manager/impl/validation/signal_validator_test.go b/flyteadmin/pkg/manager/impl/validation/signal_validator_test.go index aa978f80d6..c78c2c366b 100644 --- a/flyteadmin/pkg/manager/impl/validation/signal_validator_test.go +++ b/flyteadmin/pkg/manager/impl/validation/signal_validator_test.go @@ -283,4 +283,52 @@ func TestValidateSignalUpdateRequest(t *testing.T) { utils.AssertEqualWithSanitizedRegex(t, "requested signal value [scalar:{ primitive:{ boolean:false } } ] is not castable to existing signal type [[8 1]]", ValidateSignalSetRequest(ctx, repo, request).Error()) }) + + t.Run("UnknownIDLType", func(t *testing.T) { + ctx := context.TODO() + + // Define an unsupported literal type with a simple type of 1000 + unsupportedLiteralType := &core.LiteralType{ + Type: &core.LiteralType_Simple{ + Simple: 1000, // Using 1000 as an unsupported type + }, + } + unsupportedLiteralTypeBytes, _ := proto.Marshal(unsupportedLiteralType) + + // Mock the repository to return a signal with this unsupported type + repo := repositoryMocks.NewMockRepository() + repo.SignalRepo().(*repositoryMocks.SignalRepoInterface). + OnGetMatch(mock.Anything, mock.Anything).Return( + models.Signal{ + Type: unsupportedLiteralTypeBytes, // Set the unsupported type + }, + nil, + ) + + // Set up the unsupported literal that will trigger the nil valueType condition + unsupportedLiteral := &core.Literal{ + Value: &core.Literal_Scalar{ + Scalar: &core.Scalar{}, + }, + } + + request := admin.SignalSetRequest{ + Id: &core.SignalIdentifier{ + ExecutionId: &core.WorkflowExecutionIdentifier{ + Project: "project", + Domain: "domain", + Name: "name", + }, + SignalId: "signal", + }, + Value: unsupportedLiteral, // This will lead to valueType being nil + } + + // Invoke the function and check for the expected error + err := ValidateSignalSetRequest(ctx, repo, &request) + assert.NotNil(t, err) + + // Expected error message + assert.Contains(t, err.Error(), failedToValidateLiteralType) + }) } diff --git a/flyteadmin/pkg/manager/impl/validation/validation.go b/flyteadmin/pkg/manager/impl/validation/validation.go index 894eaee435..de2927495c 100644 --- a/flyteadmin/pkg/manager/impl/validation/validation.go +++ b/flyteadmin/pkg/manager/impl/validation/validation.go @@ -1,7 +1,6 @@ package validation import ( - "fmt" "net/url" "strconv" "strings" @@ -283,19 +282,9 @@ func validateParameterMap(inputMap *core.ParameterMap, fieldName string) error { defaultValue := defaultInput.GetDefault() if defaultValue != nil { inputType := validators.LiteralTypeForLiteral(defaultValue) - - if inputType == nil { - return errors.NewFlyteAdminErrorf(codes.InvalidArgument, - fmt.Sprintf( - "Flyte encountered an issue while determining\n"+ - "the type of the default value for Parameter '%s' in '%s'.\n"+ - "Registered type: [%s].\n"+ - "Flyte needs to support the latest FlyteIDL to support this type.\n"+ - "Suggested solution: Please update all of your Flyte images to the latest version and "+ - "try again.", - name, fieldName, defaultInput.GetVar().GetType().String(), - ), - ) + err := validators.ValidateLiteralType(inputType) + if err != nil { + return errors.NewInvalidLiteralTypeError(name, err) } if !validators.AreTypesCastable(inputType, defaultInput.GetVar().GetType()) { diff --git a/flyteadmin/pkg/manager/impl/validation/validation_test.go b/flyteadmin/pkg/manager/impl/validation/validation_test.go index 0cc20cfd1b..265868789e 100644 --- a/flyteadmin/pkg/manager/impl/validation/validation_test.go +++ b/flyteadmin/pkg/manager/impl/validation/validation_test.go @@ -347,16 +347,7 @@ func TestValidateParameterMap(t *testing.T) { err := validateParameterMap(&exampleMap, fieldName) assert.Error(t, err) fmt.Println(err.Error()) - expectedErrMsg := fmt.Sprintf( - "Flyte encountered an issue while determining\n"+ - "the type of the default value for Parameter '%s' in '%s'.\n"+ - "Registered type: [%s].\n"+ - "Flyte needs to support the latest FlyteIDL to support this type.\n"+ - "Suggested solution: Please update all of your Flyte images to the latest version and "+ - "try again.", - name, fieldName, exampleMap.Parameters[name].GetVar().GetType().String(), - ) - assert.Equal(t, expectedErrMsg, err.Error()) + assert.Contains(t, err.Error(), failedToValidateLiteralType) }) } diff --git a/flyteadmin/pkg/runtime/interfaces/application_configuration.go b/flyteadmin/pkg/runtime/interfaces/application_configuration.go index e6fa17bff1..b77c2a2926 100644 --- a/flyteadmin/pkg/runtime/interfaces/application_configuration.go +++ b/flyteadmin/pkg/runtime/interfaces/application_configuration.go @@ -49,8 +49,7 @@ type PostgresConfig struct { } type FeatureGates struct { - EnableArtifacts bool `json:"enableArtifacts" pflag:",Enable artifacts feature."` - EnableFriendlyNames bool `json:"enableFriendlyNames" pflag:",Enable generation of friendly execution names feature."` + EnableArtifacts bool `json:"enableArtifacts" pflag:",Enable artifacts feature."` } // ApplicationConfig is the base configuration to start admin diff --git a/flyteadmin/scheduler/executor/executor_impl.go b/flyteadmin/scheduler/executor/executor_impl.go index e269d79b2a..5e4a8fcf8e 100644 --- a/flyteadmin/scheduler/executor/executor_impl.go +++ b/flyteadmin/scheduler/executor/executor_impl.go @@ -2,6 +2,7 @@ package executor import ( "context" + "strings" "time" "github.com/prometheus/client_golang/prometheus" @@ -11,7 +12,6 @@ import ( "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/util/retry" - "github.com/flyteorg/flyte/flyteadmin/pkg/common/naming" "github.com/flyteorg/flyte/flyteadmin/scheduler/identifier" "github.com/flyteorg/flyte/flyteadmin/scheduler/repositories/models" "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin" @@ -54,18 +54,22 @@ func (w *executor) Execute(ctx context.Context, scheduledTime time.Time, s model } // Making the identifier deterministic using the hash of the identifier and scheduled time - hashValue := identifier.HashScheduledTimeStamp(ctx, &core.Identifier{ + executionIdentifier, err := identifier.GetExecutionIdentifier(ctx, &core.Identifier{ Project: s.Project, Domain: s.Domain, Name: s.Name, Version: s.Version, }, scheduledTime) - executionName := naming.GetExecutionName(int64(hashValue)) + if err != nil { + logger.Errorf(ctx, "failed to generate execution identifier for schedule %+v due to %v", s, err) + return err + } + executionRequest := &admin.ExecutionCreateRequest{ Project: s.Project, Domain: s.Domain, - Name: executionName, + Name: "f" + strings.ReplaceAll(executionIdentifier.String(), "-", "")[:19], Spec: &admin.ExecutionSpec{ LaunchPlan: &core.Identifier{ ResourceType: core.ResourceType_LAUNCH_PLAN, @@ -93,7 +97,7 @@ func (w *executor) Execute(ctx context.Context, scheduledTime time.Time, s model // Do maximum of 30 retries on failures with constant backoff factor opts := wait.Backoff{Duration: 3000, Factor: 2.0, Steps: 30} - err := retry.OnError(opts, + err = retry.OnError(opts, func(err error) bool { // For idempotent behavior ignore the AlreadyExists error which happens if we try to schedule a launchplan // for execution at the same time which is already available in admin. diff --git a/flyteadmin/scheduler/identifier/identifier.go b/flyteadmin/scheduler/identifier/identifier.go index caf5c94296..5d386e8652 100644 --- a/flyteadmin/scheduler/identifier/identifier.go +++ b/flyteadmin/scheduler/identifier/identifier.go @@ -34,7 +34,7 @@ func GetScheduleName(ctx context.Context, s models.SchedulableEntity) string { // GetExecutionIdentifier returns UUID using the hashed value of the schedule identifier and the scheduledTime func GetExecutionIdentifier(ctx context.Context, identifier *core.Identifier, scheduledTime time.Time) (uuid.UUID, error) { - hashValue := HashScheduledTimeStamp(ctx, identifier, scheduledTime) + hashValue := hashScheduledTimeStamp(ctx, identifier, scheduledTime) b := make([]byte, 16) binary.LittleEndian.PutUint64(b, hashValue) return uuid.FromBytes(b) @@ -55,8 +55,8 @@ func hashIdentifier(ctx context.Context, identifier *core.Identifier) uint64 { return h.Sum64() } -// HashScheduledTimeStamp return the hash of the identifier and the scheduledTime -func HashScheduledTimeStamp(ctx context.Context, identifier *core.Identifier, scheduledTime time.Time) uint64 { +// hashScheduledTimeStamp return the hash of the identifier and the scheduledTime +func hashScheduledTimeStamp(ctx context.Context, identifier *core.Identifier, scheduledTime time.Time) uint64 { h := fnv.New64() _, err := h.Write([]byte(fmt.Sprintf(executionIDInputsFormat, identifier.Project, identifier.Domain, identifier.Name, identifier.Version, scheduledTime.Unix()))) diff --git a/flyteidl/clients/go/coreutils/extract_literal.go b/flyteidl/clients/go/coreutils/extract_literal.go index 5801296dc3..f9918dd0f8 100644 --- a/flyteidl/clients/go/coreutils/extract_literal.go +++ b/flyteidl/clients/go/coreutils/extract_literal.go @@ -96,6 +96,10 @@ func ExtractFromLiteral(literal *core.Literal) (interface{}, error) { } } return mapResult, nil + case *core.Literal_OffloadedMetadata: + // Return the URI of the offloaded metadata to be used when displaying in flytectl + return literalValue.OffloadedMetadata.Uri, nil + } return nil, fmt.Errorf("unsupported literal type %T", literal) } diff --git a/flyteidl/clients/go/coreutils/extract_literal_test.go b/flyteidl/clients/go/coreutils/extract_literal_test.go index 2ce8747fd5..760e7bee0a 100644 --- a/flyteidl/clients/go/coreutils/extract_literal_test.go +++ b/flyteidl/clients/go/coreutils/extract_literal_test.go @@ -200,6 +200,30 @@ func TestFetchLiteral(t *testing.T) { assert.Equal(t, literalVal, extractedLiteralVal) }) + t.Run("Offloaded metadata", func(t *testing.T) { + literalVal := "s3://blah/blah/blah" + var storedLiteralType = &core.LiteralType{ + Type: &core.LiteralType_CollectionType{ + CollectionType: &core.LiteralType{ + Type: &core.LiteralType_Simple{ + Simple: core.SimpleType_INTEGER, + }, + }, + }, + } + offloadedLiteral := &core.Literal{ + Value: &core.Literal_OffloadedMetadata{ + OffloadedMetadata: &core.LiteralOffloadedMetadata{ + Uri: literalVal, + InferredType: storedLiteralType, + }, + }, + } + extractedLiteralVal, err := ExtractFromLiteral(offloadedLiteral) + assert.NoError(t, err) + assert.Equal(t, literalVal, extractedLiteralVal) + }) + t.Run("Union", func(t *testing.T) { literalVal := int64(1) var literalType = &core.LiteralType{ diff --git a/flyteidl/clients/go/coreutils/literals.go b/flyteidl/clients/go/coreutils/literals.go index 3527ac246b..278fa30dfc 100644 --- a/flyteidl/clients/go/coreutils/literals.go +++ b/flyteidl/clients/go/coreutils/literals.go @@ -636,7 +636,6 @@ func MakeLiteralForType(t *core.LiteralType, v interface{}) (*core.Literal, erro if !found { return nil, fmt.Errorf("incorrect union value [%s], supported values %+v", v, newT.UnionType.Variants) } - default: return nil, fmt.Errorf("unsupported type %s", t.String()) } diff --git a/flytepropeller/pkg/compiler/errors/compiler_error_test.go b/flytepropeller/pkg/compiler/errors/compiler_error_test.go index 5a993aeff2..8394ed5bb7 100644 --- a/flytepropeller/pkg/compiler/errors/compiler_error_test.go +++ b/flytepropeller/pkg/compiler/errors/compiler_error_test.go @@ -33,6 +33,7 @@ func TestErrorCodes(t *testing.T) { UnrecognizedValue: NewUnrecognizedValueErr("", ""), WorkflowBuildError: NewWorkflowBuildError(errors.New("")), NoNodesFound: NewNoNodesFoundErr(""), + InvalidLiteralTypeError: NewInvalidLiteralTypeErr("", "", errors.New("")), } for key, value := range testCases { @@ -48,6 +49,6 @@ func TestIncludeSource(t *testing.T) { SetConfig(Config{IncludeSource: true}) e = NewCycleDetectedInWorkflowErr("", "") - assert.Equal(t, e.source, "compiler_error_test.go:50") + assert.Equal(t, e.source, "compiler_error_test.go:51") SetConfig(Config{}) } diff --git a/flytepropeller/pkg/compiler/errors/compiler_errors.go b/flytepropeller/pkg/compiler/errors/compiler_errors.go index 9b762f72ca..b2e3796edd 100755 --- a/flytepropeller/pkg/compiler/errors/compiler_errors.go +++ b/flytepropeller/pkg/compiler/errors/compiler_errors.go @@ -96,6 +96,9 @@ const ( // Field not found in the dataclass FieldNotFoundError ErrorCode = "FieldNotFound" + + // IDL not found when variable binding + InvalidLiteralTypeError ErrorCode = "InvalidLiteralType" ) func NewBranchNodeNotSpecified(branchNodeID string) *CompileError { @@ -218,6 +221,14 @@ func NewMismatchingVariablesErr(nodeID, fromVar, fromType, toVar, toType string) ) } +func NewInvalidLiteralTypeErr(nodeID, inputVar string, err error) *CompileError { + return newError( + InvalidLiteralTypeError, + fmt.Sprintf("Failed to validate literal type for [%s] with err: %s", inputVar, err), + nodeID, + ) +} + func NewMismatchingBindingsErr(nodeID, sinkParam, expectedType, receivedType string) *CompileError { return newError( MismatchingBindings, diff --git a/flytepropeller/pkg/compiler/transformers/k8s/inputs.go b/flytepropeller/pkg/compiler/transformers/k8s/inputs.go index 0976df669b..26f50d4ddd 100644 --- a/flytepropeller/pkg/compiler/transformers/k8s/inputs.go +++ b/flytepropeller/pkg/compiler/transformers/k8s/inputs.go @@ -35,12 +35,11 @@ func validateInputs(nodeID common.NodeID, iface *core.TypedInterface, inputs cor continue } - var inputType *core.LiteralType - switch inputVal.GetValue().(type) { - case *core.Literal_OffloadedMetadata: - inputType = inputVal.GetOffloadedMetadata().GetInferredType() - default: - inputType = validators.LiteralTypeForLiteral(inputVal) + inputType := validators.LiteralTypeForLiteral(inputVal) + err := validators.ValidateLiteralType(inputType) + if err != nil { + errs.Collect(errors.NewInvalidLiteralTypeErr(nodeID, inputVar, err)) + continue } if !validators.AreTypesCastable(inputType, v.Type) { errs.Collect(errors.NewMismatchingTypesErr(nodeID, inputVar, v.Type.String(), inputType.String())) diff --git a/flytepropeller/pkg/compiler/transformers/k8s/inputs_test.go b/flytepropeller/pkg/compiler/transformers/k8s/inputs_test.go index 01d667c35d..d77aafec49 100644 --- a/flytepropeller/pkg/compiler/transformers/k8s/inputs_test.go +++ b/flytepropeller/pkg/compiler/transformers/k8s/inputs_test.go @@ -1 +1,55 @@ package k8s + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core" + "github.com/flyteorg/flyte/flytepropeller/pkg/compiler/common" + "github.com/flyteorg/flyte/flytepropeller/pkg/compiler/errors" +) + +func TestValidateInputs_InvalidLiteralType(t *testing.T) { + nodeID := common.NodeID("test-node") + + iface := &core.TypedInterface{ + Inputs: &core.VariableMap{ + Variables: map[string]*core.Variable{ + "input1": { + Type: &core.LiteralType{ + Type: &core.LiteralType_Simple{ + Simple: 1000, + }, + }, + }, + }, + }, + } + + inputs := core.LiteralMap{ + Literals: map[string]*core.Literal{ + "input1": nil, // Set this to nil to trigger the nil case + }, + } + + errs := errors.NewCompileErrors() + ok := validateInputs(nodeID, iface, inputs, errs) + + assert.False(t, ok) + assert.True(t, errs.HasErrors()) + + idlNotFound := false + var errMsg string + for _, err := range errs.Errors().List() { + if err.Code() == "InvalidLiteralType" { + idlNotFound = true + errMsg = err.Error() + break + } + } + assert.True(t, idlNotFound, "Expected InvalidLiteralType error was not found in errors") + + expectedContainedErrorMsg := "Failed to validate literal type" + assert.Contains(t, errMsg, expectedContainedErrorMsg) +} diff --git a/flytepropeller/pkg/compiler/validators/utils.go b/flytepropeller/pkg/compiler/validators/utils.go index fb4ba04548..d1bffbbe09 100644 --- a/flytepropeller/pkg/compiler/validators/utils.go +++ b/flytepropeller/pkg/compiler/validators/utils.go @@ -249,6 +249,23 @@ func literalTypeForLiterals(literals []*core.Literal) *core.LiteralType { return buildMultipleTypeUnion(innerType) } +// ValidateLiteralType check if the literal type is valid, return error if the literal is invalid. +func ValidateLiteralType(lt *core.LiteralType) error { + if lt == nil { + err := fmt.Errorf("got unknown literal type: [%v].\n"+ + "Suggested solution: Please update all your Flyte deployment images to the latest version and try again", lt) + return err + } + if lt.GetCollectionType() != nil { + return ValidateLiteralType(lt.GetCollectionType()) + } + if lt.GetMapValueType() != nil { + return ValidateLiteralType(lt.GetMapValueType()) + } + + return nil +} + // LiteralTypeForLiteral gets LiteralType for literal, nil if the value of literal is unknown, or type collection/map of // type None if the literal is a non-homogeneous type. func LiteralTypeForLiteral(l *core.Literal) *core.LiteralType { @@ -267,8 +284,9 @@ func LiteralTypeForLiteral(l *core.Literal) *core.LiteralType { MapValueType: literalTypeForLiterals(maps.Values(l.GetMap().Literals)), }, } + case *core.Literal_OffloadedMetadata: + return l.GetOffloadedMetadata().GetInferredType() } - return nil } diff --git a/flytepropeller/pkg/compiler/validators/utils_test.go b/flytepropeller/pkg/compiler/validators/utils_test.go index 26e34988c3..b6737c7e62 100644 --- a/flytepropeller/pkg/compiler/validators/utils_test.go +++ b/flytepropeller/pkg/compiler/validators/utils_test.go @@ -413,6 +413,47 @@ func TestLiteralTypeForLiterals(t *testing.T) { assert.True(t, proto.Equal(expectedLt, lt)) }) + t.Run("nested Lists with different types", func(t *testing.T) { + inferredType := &core.LiteralType{ + Type: &core.LiteralType_CollectionType{ + CollectionType: &core.LiteralType{ + Type: &core.LiteralType_CollectionType{ + CollectionType: &core.LiteralType{ + Type: &core.LiteralType_UnionType{ + UnionType: &core.UnionType{ + Variants: []*core.LiteralType{ + { + Type: &core.LiteralType_Simple{ + Simple: core.SimpleType_INTEGER, + }, + }, + { + Type: &core.LiteralType_Simple{ + Simple: core.SimpleType_STRING, + }, + }, + }, + }, + }, + }, + }, + }, + }, + } + literals := &core.Literal{ + Value: &core.Literal_OffloadedMetadata{ + OffloadedMetadata: &core.LiteralOffloadedMetadata{ + Uri: "dummy/uri", + SizeBytes: 1000, + InferredType: inferredType, + }, + }, + } + expectedLt := inferredType + lt := LiteralTypeForLiteral(literals) + assert.True(t, proto.Equal(expectedLt, lt)) + }) + } func TestJoinVariableMapsUniqueKeys(t *testing.T) { diff --git a/flytepropeller/pkg/controller/nodes/array/handler.go b/flytepropeller/pkg/controller/nodes/array/handler.go index 5e9f910e14..834a016cb2 100644 --- a/flytepropeller/pkg/controller/nodes/array/handler.go +++ b/flytepropeller/pkg/controller/nodes/array/handler.go @@ -191,12 +191,29 @@ func (a *arrayNodeHandler) Handle(ctx context.Context, nCtx interfaces.NodeExecu } size := -1 - for _, variable := range literalMap.Literals { + + for key, variable := range literalMap.Literals { literalType := validators.LiteralTypeForLiteral(variable) + err := validators.ValidateLiteralType(literalType) + if err != nil { + errMsg := fmt.Sprintf("Failed to validate literal type for [%s] with err: %s", key, err) + return handler.DoTransition(handler.TransitionTypeEphemeral, + handler.PhaseInfoFailure(idlcore.ExecutionError_USER, errors.IDLNotFoundErr, errMsg, nil), + ), nil + } + if variable.GetOffloadedMetadata() != nil { + // variable will be overwritten with the contents of the offloaded data which contains the actual large literal. + // We need this for the map task to be able to create the subNodeSpec + err := common.ReadLargeLiteral(ctx, nCtx.DataStore(), variable) + if err != nil { + return handler.DoTransition(handler.TransitionTypeEphemeral, + handler.PhaseInfoFailure(idlcore.ExecutionError_SYSTEM, errors.RuntimeExecutionError, "couldn't read the offloaded literal", nil), + ), nil + } + } switch literalType.Type.(type) { case *idlcore.LiteralType_CollectionType: collectionLength := len(variable.GetCollection().Literals) - if size == -1 { size = collectionLength } else if size != collectionLength { diff --git a/flytepropeller/pkg/controller/nodes/array/handler_test.go b/flytepropeller/pkg/controller/nodes/array/handler_test.go index cb2f2898a6..08eea22e09 100644 --- a/flytepropeller/pkg/controller/nodes/array/handler_test.go +++ b/flytepropeller/pkg/controller/nodes/array/handler_test.go @@ -20,6 +20,7 @@ import ( execmocks "github.com/flyteorg/flyte/flytepropeller/pkg/controller/executors/mocks" "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes" "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/catalog" + "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/errors" gatemocks "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/gate/mocks" "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/handler" "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/interfaces" @@ -943,6 +944,68 @@ func TestHandleArrayNodePhaseExecuting(t *testing.T) { } } +func TestHandle_InvalidLiteralType(t *testing.T) { + ctx := context.Background() + scope := promutils.NewTestScope() + dataStore, err := storage.NewDataStore(&storage.Config{ + Type: storage.TypeMemory, + }, scope) + assert.NoError(t, err) + nodeHandler := &mocks.NodeHandler{} + + // Initialize ArrayNodeHandler + arrayNodeHandler, err := createArrayNodeHandler(ctx, t, nodeHandler, dataStore, scope) + assert.NoError(t, err) + + // Test cases + tests := []struct { + name string + inputLiteral *idlcore.Literal + expectedTransitionType handler.TransitionType + expectedPhase handler.EPhase + expectedErrorCode string + expectedContainedErrorMsg string + }{ + { + name: "InvalidLiteralType", + inputLiteral: &idlcore.Literal{ + Value: &idlcore.Literal_Scalar{ + Scalar: &idlcore.Scalar{}, + }, + }, + expectedTransitionType: handler.TransitionTypeEphemeral, + expectedPhase: handler.EPhaseFailed, + expectedErrorCode: errors.IDLNotFoundErr, + expectedContainedErrorMsg: "Failed to validate literal type", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + // Create NodeExecutionContext + literalMap := &idlcore.LiteralMap{ + Literals: map[string]*idlcore.Literal{ + "invalidInput": test.inputLiteral, + }, + } + arrayNodeState := &handler.ArrayNodeState{ + Phase: v1alpha1.ArrayNodePhaseNone, + } + nCtx := createNodeExecutionContext(dataStore, newBufferedEventRecorder(), nil, literalMap, &arrayNodeSpec, arrayNodeState, 0, workflowMaxParallelism) + + // Evaluate node + transition, err := arrayNodeHandler.Handle(ctx, nCtx) + assert.NoError(t, err) + + // Validate results + assert.Equal(t, test.expectedTransitionType, transition.Type()) + assert.Equal(t, test.expectedPhase, transition.Info().GetPhase()) + assert.Equal(t, test.expectedErrorCode, transition.Info().GetErr().Code) + assert.Contains(t, transition.Info().GetErr().Message, test.expectedContainedErrorMsg) + }) + } +} + func TestHandleArrayNodePhaseExecutingSubNodeFailures(t *testing.T) { ctx := context.Background() diff --git a/flytepropeller/pkg/controller/nodes/attr_path_resolver.go b/flytepropeller/pkg/controller/nodes/attr_path_resolver.go index 42150cb887..fa19d2bf5c 100644 --- a/flytepropeller/pkg/controller/nodes/attr_path_resolver.go +++ b/flytepropeller/pkg/controller/nodes/attr_path_resolver.go @@ -1,15 +1,19 @@ package nodes import ( + "context" + "google.golang.org/protobuf/types/known/structpb" "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core" + "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/common" "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/errors" + "github.com/flyteorg/flyte/flytestdlib/storage" ) // resolveAttrPathInPromise resolves the literal with attribute path // If the promise is chained with attributes (e.g. promise.a["b"][0]), then we need to resolve the promise -func resolveAttrPathInPromise(nodeID string, literal *core.Literal, bindAttrPath []*core.PromiseAttribute) (*core.Literal, error) { +func resolveAttrPathInPromise(ctx context.Context, datastore *storage.DataStore, nodeID string, literal *core.Literal, bindAttrPath []*core.PromiseAttribute) (*core.Literal, error) { var currVal *core.Literal = literal var tmpVal *core.Literal var err error @@ -17,7 +21,16 @@ func resolveAttrPathInPromise(nodeID string, literal *core.Literal, bindAttrPath count := 0 for _, attr := range bindAttrPath { + if currVal.GetOffloadedMetadata() != nil { + // currVal will be overwritten with the contents of the offloaded data which contains the actual large literal. + err := common.ReadLargeLiteral(ctx, datastore, currVal) + if err != nil { + return nil, errors.Errorf(errors.PromiseAttributeResolveError, nodeID, "failed to read offloaded metadata for promise") + } + } switch currVal.GetValue().(type) { + case *core.Literal_OffloadedMetadata: + return nil, errors.Errorf(errors.PromiseAttributeResolveError, nodeID, "unexpected offloaded metadata type") case *core.Literal_Map: tmpVal, exist = currVal.GetMap().GetLiterals()[attr.GetStringValue()] if !exist { diff --git a/flytepropeller/pkg/controller/nodes/attr_path_resolver_test.go b/flytepropeller/pkg/controller/nodes/attr_path_resolver_test.go index fb966c666e..8724f5287d 100644 --- a/flytepropeller/pkg/controller/nodes/attr_path_resolver_test.go +++ b/flytepropeller/pkg/controller/nodes/attr_path_resolver_test.go @@ -1,6 +1,7 @@ package nodes import ( + "context" "testing" "github.com/stretchr/testify/assert" @@ -319,7 +320,7 @@ func TestResolveAttrPathIn(t *testing.T) { } for i, arg := range args { - resolved, err := resolveAttrPathInPromise("", arg.literal, arg.path) + resolved, err := resolveAttrPathInPromise(context.Background(), nil, "", arg.literal, arg.path) if arg.hasError { assert.Error(t, err, i) assert.ErrorContains(t, err, errors.PromiseAttributeResolveError, i) diff --git a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go index a04623f88a..00a99d6c54 100644 --- a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go +++ b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/datacatalog.go @@ -122,7 +122,7 @@ func (m *CatalogClient) Get(ctx context.Context, key catalog.Key) (catalog.Entry logger.Debugf(ctx, "DataCatalog failed to get artifact by tag %+v, err: %+v", tag, err) return catalog.Entry{}, err } - logger.Debugf(ctx, "Artifact found %v from tag %v", artifact, tag) + logger.Debugf(ctx, "Artifact found %v from tag %v", artifact.GetId(), tag) var relevantTag *datacatalog.Tag if len(artifact.GetTags()) > 0 { @@ -230,7 +230,7 @@ func (m *CatalogClient) createArtifact(ctx context.Context, key catalog.Key, dat createArtifactRequest := &datacatalog.CreateArtifactRequest{Artifact: cachedArtifact} _, err := m.client.CreateArtifact(ctx, createArtifactRequest) if err != nil { - logger.Errorf(ctx, "Failed to create Artifact %+v, err: %v", cachedArtifact, err) + logger.Errorf(ctx, "Failed to create Artifact %+v, err: %v", cachedArtifact.Id, err) return catalog.Status{}, err } logger.Debugf(ctx, "Created artifact: %v, with %v outputs from execution %+v", cachedArtifact.Id, len(artifactDataList), metadata) @@ -259,7 +259,7 @@ func (m *CatalogClient) createArtifact(ctx context.Context, key catalog.Key, dat } } - logger.Debugf(ctx, "Successfully created artifact %+v for key %+v, dataset %+v and execution %+v", cachedArtifact, key, datasetID, metadata) + logger.Debugf(ctx, "Successfully created artifact %+v for key %+v, dataset %+v and execution %+v", cachedArtifact.Id, key, datasetID, metadata) return catalog.NewStatus(core.CatalogCacheStatus_CACHE_POPULATED, EventCatalogMetadata(datasetID, tag, nil)), nil } diff --git a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer.go b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer.go index 5c0ac0c30b..ba94bdadec 100644 --- a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer.go +++ b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer.go @@ -54,6 +54,10 @@ func GenerateTaskOutputsFromArtifact(id core.Identifier, taskInterface core.Type expectedVarType := outputVariables[artifactData.Name].GetType() inputType := validators.LiteralTypeForLiteral(artifactData.Value) + err := validators.ValidateLiteralType(inputType) + if err != nil { + return nil, fmt.Errorf("failed to validate literal type for %s with err: %s", artifactData.Name, err) + } if !validators.AreTypesCastable(inputType, expectedVarType) { return nil, fmt.Errorf("unexpected artifactData: [%v] type: [%v] does not match any task output type: [%v]", artifactData.Name, inputType, expectedVarType) } diff --git a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer_test.go b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer_test.go index 1c6b9e2e1b..92e4c82926 100644 --- a/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer_test.go +++ b/flytepropeller/pkg/controller/nodes/catalog/datacatalog/transformer_test.go @@ -331,3 +331,43 @@ func TestDatasetIDToIdentifier(t *testing.T) { assert.Equal(t, "d", id.Domain) assert.Equal(t, "v", id.Version) } + +func TestGenerateTaskOutputsFromArtifact_IDLNotFound(t *testing.T) { + taskID := core.Identifier{ + ResourceType: core.ResourceType_TASK, + Project: "project", + Domain: "domain", + Name: "name", + Version: "version", + } + + taskInterface := core.TypedInterface{ + Outputs: &core.VariableMap{ + Variables: map[string]*core.Variable{ + "output1": { + Type: &core.LiteralType{ + Type: &core.LiteralType_Simple{ + Simple: 1000, + }, + }, + }, + }, + }, + } + + artifact := &datacatalog.Artifact{ + Id: "artifact_id", + Data: []*datacatalog.ArtifactData{ + { + Name: "output1", + Value: &core.Literal{}, // This will cause LiteralTypeForLiteral to return nil + }, + }, + } + + _, err := GenerateTaskOutputsFromArtifact(taskID, taskInterface, artifact) + + expectedContainedErrorMsg := "failed to validate literal type" + assert.Error(t, err) + assert.Contains(t, err.Error(), expectedContainedErrorMsg) +} diff --git a/flytepropeller/pkg/controller/nodes/common/utils.go b/flytepropeller/pkg/controller/nodes/common/utils.go index b02d830fe9..dd16b53f3a 100644 --- a/flytepropeller/pkg/controller/nodes/common/utils.go +++ b/flytepropeller/pkg/controller/nodes/common/utils.go @@ -2,6 +2,7 @@ package common import ( "context" + "encoding/base64" "fmt" "strconv" @@ -17,6 +18,7 @@ import ( "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/handler" "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/interfaces" "github.com/flyteorg/flyte/flytestdlib/logger" + "github.com/flyteorg/flyte/flytestdlib/pbhash" "github.com/flyteorg/flyte/flytestdlib/storage" ) @@ -79,6 +81,27 @@ func GetTargetEntity(ctx context.Context, nCtx interfaces.NodeExecutionContext) return targetEntity } +// ReadLargeLiteral reads the offloaded large literal needed by array node task +func ReadLargeLiteral(ctx context.Context, datastore *storage.DataStore, + tobeRead *idlcore.Literal) error { + if tobeRead.GetOffloadedMetadata() == nil { + return fmt.Errorf("unsupported type for reading offloaded literal") + } + dataReference := tobeRead.GetOffloadedMetadata().GetUri() + if dataReference == "" { + return fmt.Errorf("uri is empty for offloaded literal") + } + // read the offloaded literal + size := tobeRead.GetOffloadedMetadata().GetSizeBytes() + if err := datastore.ReadProtobuf(ctx, storage.DataReference(dataReference), tobeRead); err != nil { + logger.Errorf(ctx, "Failed to read the offloaded literal at location [%s] with error [%s]", dataReference, err) + return err + } + + logger.Infof(ctx, "read offloaded literal at location [%s] with size [%s]", dataReference, size) + return nil +} + // OffloadLargeLiteral offloads the large literal if meets the threshold conditions func OffloadLargeLiteral(ctx context.Context, datastore *storage.DataStore, dataReference storage.DataReference, toBeOffloaded *idlcore.Literal, literalOffloadingConfig config.LiteralOffloadingConfig) error { @@ -108,6 +131,16 @@ func OffloadLargeLiteral(ctx context.Context, datastore *storage.DataStore, data return err } + if toBeOffloaded.GetHash() == "" { + // compute the hash of the literal + literalDigest, err := pbhash.ComputeHash(ctx, toBeOffloaded) + if err != nil { + logger.Errorf(ctx, "Failed to compute hash for offloaded literal with error [%s]", err) + return err + } + // Set the hash or else respect what the user set in the literal + toBeOffloaded.Hash = base64.RawURLEncoding.EncodeToString(literalDigest) + } // update the literal with the offloaded URI, size and inferred type toBeOffloaded.Value = &idlcore.Literal_OffloadedMetadata{ OffloadedMetadata: &idlcore.LiteralOffloadedMetadata{ diff --git a/flytepropeller/pkg/controller/nodes/common/utils_test.go b/flytepropeller/pkg/controller/nodes/common/utils_test.go index 7d5ce1e372..875ede858b 100644 --- a/flytepropeller/pkg/controller/nodes/common/utils_test.go +++ b/flytepropeller/pkg/controller/nodes/common/utils_test.go @@ -2,6 +2,7 @@ package common import ( "context" + "encoding/base64" "testing" "github.com/stretchr/testify/assert" @@ -14,6 +15,7 @@ import ( executorMocks "github.com/flyteorg/flyte/flytepropeller/pkg/controller/executors/mocks" nodeMocks "github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/interfaces/mocks" "github.com/flyteorg/flyte/flytestdlib/contextutils" + "github.com/flyteorg/flyte/flytestdlib/pbhash" "github.com/flyteorg/flyte/flytestdlib/promutils" "github.com/flyteorg/flyte/flytestdlib/promutils/labeled" "github.com/flyteorg/flyte/flytestdlib/storage" @@ -82,6 +84,40 @@ func init() { labeled.SetMetricKeys(contextutils.AppNameKey) } +func TestReadLargeLiteral(t *testing.T) { + t.Run("read successful", func(t *testing.T) { + ctx := context.Background() + datastore, _ := storage.NewDataStore(&storage.Config{Type: storage.TypeMemory}, promutils.NewTestScope()) + dataReference := storage.DataReference("foo/bar") + toBeRead := &idlCore.Literal{ + Value: &idlCore.Literal_Scalar{ + Scalar: &idlCore.Scalar{ + Value: &idlCore.Scalar_Primitive{ + Primitive: &idlCore.Primitive{ + Value: &idlCore.Primitive_Integer{ + Integer: 1, + }, + }, + }, + }, + }, + } + err := datastore.WriteProtobuf(ctx, dataReference, storage.Options{}, toBeRead) + assert.Nil(t, err) + + offloadedLiteral := &idlCore.Literal{ + Value: &idlCore.Literal_OffloadedMetadata{ + OffloadedMetadata: &idlCore.LiteralOffloadedMetadata{ + Uri: dataReference.String(), + }, + }, + } + err = ReadLargeLiteral(ctx, datastore, offloadedLiteral) + assert.Nil(t, err) + assert.Equal(t, int64(1), offloadedLiteral.GetScalar().GetPrimitive().GetInteger()) + }) +} + func TestOffloadLargeLiteral(t *testing.T) { t.Run("offload successful with valid size", func(t *testing.T) { ctx := context.Background() @@ -100,17 +136,46 @@ func TestOffloadLargeLiteral(t *testing.T) { }, }, } + expectedLiteralDigest, err := pbhash.ComputeHash(ctx, toBeOffloaded) + assert.Nil(t, err) literalOffloadingConfig := config.LiteralOffloadingConfig{ MinSizeInMBForOffloading: 0, MaxSizeInMBForOffloading: 1, } inferredType := validators.LiteralTypeForLiteral(toBeOffloaded) - err := OffloadLargeLiteral(ctx, datastore, dataReference, toBeOffloaded, literalOffloadingConfig) + err = OffloadLargeLiteral(ctx, datastore, dataReference, toBeOffloaded, literalOffloadingConfig) assert.NoError(t, err) assert.Equal(t, "foo/bar", toBeOffloaded.GetOffloadedMetadata().GetUri()) assert.Equal(t, uint64(6), toBeOffloaded.GetOffloadedMetadata().GetSizeBytes()) assert.Equal(t, inferredType.GetSimple(), toBeOffloaded.GetOffloadedMetadata().InferredType.GetSimple()) + assert.Equal(t, base64.RawURLEncoding.EncodeToString(expectedLiteralDigest), toBeOffloaded.Hash) + }) + t.Run("offload successful with valid size and hash passed in", func(t *testing.T) { + ctx := context.Background() + datastore, _ := storage.NewDataStore(&storage.Config{Type: storage.TypeMemory}, promutils.NewTestScope()) + dataReference := storage.DataReference("foo/bar") + toBeOffloaded := &idlCore.Literal{ + Value: &idlCore.Literal_Scalar{ + Scalar: &idlCore.Scalar{ + Value: &idlCore.Scalar_Primitive{ + Primitive: &idlCore.Primitive{ + Value: &idlCore.Primitive_Integer{ + Integer: 1, + }, + }, + }, + }, + }, + Hash: "hash", + } + literalOffloadingConfig := config.LiteralOffloadingConfig{ + MinSizeInMBForOffloading: 0, + MaxSizeInMBForOffloading: 1, + } + err := OffloadLargeLiteral(ctx, datastore, dataReference, toBeOffloaded, literalOffloadingConfig) + assert.NoError(t, err) + assert.Equal(t, "hash", toBeOffloaded.Hash) }) t.Run("offload fails with size larger than max", func(t *testing.T) { diff --git a/flytepropeller/pkg/controller/nodes/errors/codes.go b/flytepropeller/pkg/controller/nodes/errors/codes.go index 53d3bbc8d7..dafccdc5c0 100644 --- a/flytepropeller/pkg/controller/nodes/errors/codes.go +++ b/flytepropeller/pkg/controller/nodes/errors/codes.go @@ -27,4 +27,5 @@ const ( CatalogCallFailed ErrorCode = "CatalogCallFailed" InvalidArrayLength ErrorCode = "InvalidArrayLength" PromiseAttributeResolveError ErrorCode = "PromiseAttributeResolveError" + IDLNotFoundErr ErrorCode = "IDLNotFoundErr" ) diff --git a/flytepropeller/pkg/controller/nodes/output_resolver.go b/flytepropeller/pkg/controller/nodes/output_resolver.go index 85eebd4de5..df8a6dfe19 100644 --- a/flytepropeller/pkg/controller/nodes/output_resolver.go +++ b/flytepropeller/pkg/controller/nodes/output_resolver.go @@ -66,7 +66,7 @@ func (r remoteFileOutputResolver) ExtractOutput(ctx context.Context, nl executor // resolving binding attribute path if exist if len(bindAttrPath) > 0 { - output, err = resolveAttrPathInPromise(n.GetID(), output, bindAttrPath) + output, err = resolveAttrPathInPromise(ctx, r.store, n.GetID(), output, bindAttrPath) } return output, err diff --git a/flytepropeller/pkg/controller/nodes/task/backoff/handler.go b/flytepropeller/pkg/controller/nodes/task/backoff/handler.go index fc890c7a09..aadbfed514 100644 --- a/flytepropeller/pkg/controller/nodes/task/backoff/handler.go +++ b/flytepropeller/pkg/controller/nodes/task/backoff/handler.go @@ -195,8 +195,12 @@ func IsResourceQuotaExceeded(err error) bool { return apiErrors.IsForbidden(err) && strings.Contains(err.Error(), "exceeded quota") } +func IsEtcdError(err error) bool { + return apiErrors.IsForbidden(err) && strings.Contains(err.Error(), "etcdserver:") +} + func IsBackOffError(err error) bool { - return IsResourceQuotaExceeded(err) || apiErrors.IsTooManyRequests(err) || apiErrors.IsServerTimeout(err) + return IsResourceQuotaExceeded(err) || apiErrors.IsTooManyRequests(err) || apiErrors.IsServerTimeout(err) || IsEtcdError(err) } func GetComputeResourceAndQuantity(err error, resourceRegex *regexp.Regexp) v1.ResourceList { diff --git a/flytestdlib/.golangci.yml b/flytestdlib/.golangci.yml index 7f4dbc80e8..e3bff2320b 100644 --- a/flytestdlib/.golangci.yml +++ b/flytestdlib/.golangci.yml @@ -26,7 +26,6 @@ linters: - structcheck - typecheck - unconvert - - unparam - unused - varcheck diff --git a/flytestdlib/cache/auto_refresh.go b/flytestdlib/cache/auto_refresh.go index 8218e577a8..19d38bdaff 100644 --- a/flytestdlib/cache/auto_refresh.go +++ b/flytestdlib/cache/auto_refresh.go @@ -9,8 +9,8 @@ import ( lru "github.com/hashicorp/golang-lru" "github.com/prometheus/client_golang/prometheus" - "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/util/workqueue" + "k8s.io/utils/clock" "github.com/flyteorg/flyte/flytestdlib/contextutils" "github.com/flyteorg/flyte/flytestdlib/errors" @@ -125,6 +125,7 @@ type autoRefresh struct { workqueue workqueue.RateLimitingInterface parallelizm int lock sync.RWMutex + clock clock.Clock } func getEvictionFunction(counter prometheus.Counter) func(key interface{}, value interface{}) { @@ -165,17 +166,29 @@ func (w *autoRefresh) Start(ctx context.Context) error { } enqueueCtx := contextutils.WithGoroutineLabel(ctx, fmt.Sprintf("%v-enqueue", w.name)) - - go wait.Until(func() { - err := w.enqueueBatches(enqueueCtx) - if err != nil { - logger.Errorf(enqueueCtx, "Failed to enqueue. Error: %v", err) - } - }, w.syncPeriod, enqueueCtx.Done()) + go w.enqueueLoop(enqueueCtx) return nil } +func (w *autoRefresh) enqueueLoop(ctx context.Context) { + timer := w.clock.NewTimer(w.syncPeriod) + defer timer.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-timer.C(): + err := w.enqueueBatches(ctx) + if err != nil { + logger.Errorf(ctx, "Failed to enqueue. Error: %v", err) + } + timer.Reset(w.syncPeriod) + } + } +} + // Update updates the item only if it exists in the cache, return true if we updated the item. func (w *autoRefresh) Update(id ItemID, item Item) (ok bool) { w.lock.Lock() @@ -221,7 +234,7 @@ func (w *autoRefresh) GetOrCreate(id ItemID, item Item) (Item, error) { batch := make([]ItemWrapper, 0, 1) batch = append(batch, itemWrapper{id: id, item: item}) w.workqueue.AddRateLimited(&batch) - w.processing.Store(id, time.Now()) + w.processing.Store(id, w.clock.Now()) return item, nil } @@ -265,7 +278,7 @@ func (w *autoRefresh) enqueueBatches(ctx context.Context) error { b := batch w.workqueue.AddRateLimited(&b) for i := 1; i < len(b); i++ { - w.processing.Store(b[i].GetID(), time.Now()) + w.processing.Store(b[i].GetID(), w.clock.Now()) } } @@ -365,7 +378,7 @@ func (w *autoRefresh) inProcessing(key interface{}) bool { item, found := w.processing.Load(key) if found { // handle potential race conditions where the item is in processing but not in the workqueue - if timeItem, ok := item.(time.Time); ok && time.Since(timeItem) > (w.syncPeriod*5) { + if timeItem, ok := item.(time.Time); ok && w.clock.Since(timeItem) > (w.syncPeriod*5) { w.processing.Delete(key) return false } @@ -377,6 +390,11 @@ func (w *autoRefresh) inProcessing(key interface{}) bool { // Instantiates a new AutoRefresh Cache that syncs items in batches. func NewAutoRefreshBatchedCache(name string, createBatches CreateBatchesFunc, syncCb SyncFunc, syncRateLimiter workqueue.RateLimiter, resyncPeriod time.Duration, parallelizm, size int, scope promutils.Scope) (AutoRefresh, error) { + return newAutoRefreshBatchedCacheWithClock(name, createBatches, syncCb, syncRateLimiter, resyncPeriod, parallelizm, size, scope, clock.RealClock{}) +} + +func newAutoRefreshBatchedCacheWithClock(name string, createBatches CreateBatchesFunc, syncCb SyncFunc, syncRateLimiter workqueue.RateLimiter, + resyncPeriod time.Duration, parallelizm, size int, scope promutils.Scope, clock clock.WithTicker) (AutoRefresh, error) { metrics := newMetrics(scope) lruCache, err := lru.NewWithEvict(size, getEvictionFunction(metrics.Evictions)) @@ -394,7 +412,11 @@ func NewAutoRefreshBatchedCache(name string, createBatches CreateBatchesFunc, sy processing: &sync.Map{}, toDelete: newSyncSet(), syncPeriod: resyncPeriod, - workqueue: workqueue.NewNamedRateLimitingQueue(syncRateLimiter, scope.CurrentScope()), + workqueue: workqueue.NewRateLimitingQueueWithConfig(syncRateLimiter, workqueue.RateLimitingQueueConfig{ + Name: scope.CurrentScope(), + Clock: clock, + }), + clock: clock, } return cache, nil @@ -406,3 +428,8 @@ func NewAutoRefreshCache(name string, syncCb SyncFunc, syncRateLimiter workqueue return NewAutoRefreshBatchedCache(name, SingleItemBatches, syncCb, syncRateLimiter, resyncPeriod, parallelizm, size, scope) } + +func newAutoRefreshCacheWithClock(name string, syncCb SyncFunc, syncRateLimiter workqueue.RateLimiter, resyncPeriod time.Duration, + parallelizm, size int, scope promutils.Scope, clock clock.WithTicker) (AutoRefresh, error) { + return newAutoRefreshBatchedCacheWithClock(name, SingleItemBatches, syncCb, syncRateLimiter, resyncPeriod, parallelizm, size, scope, clock) +} diff --git a/flytestdlib/cache/auto_refresh_test.go b/flytestdlib/cache/auto_refresh_test.go index 5e1c49777e..e6b210bfcd 100644 --- a/flytestdlib/cache/auto_refresh_test.go +++ b/flytestdlib/cache/auto_refresh_test.go @@ -10,6 +10,7 @@ import ( "github.com/stretchr/testify/assert" "k8s.io/client-go/util/workqueue" + testingclock "k8s.io/utils/clock/testing" "github.com/flyteorg/flyte/flytestdlib/atomic" "github.com/flyteorg/flyte/flytestdlib/errors" @@ -74,12 +75,13 @@ func (p *panickingSyncer) sync(_ context.Context, _ Batch) ([]ItemSyncResponse, } func TestCacheFour(t *testing.T) { - testResyncPeriod := 10 * time.Millisecond + testResyncPeriod := 5 * time.Second rateLimiter := workqueue.DefaultControllerRateLimiter() + fakeClock := testingclock.NewFakeClock(time.Now()) t.Run("normal operation", func(t *testing.T) { // the size of the cache is at least as large as the number of items we're storing - cache, err := NewAutoRefreshCache("fake1", syncFakeItem, rateLimiter, testResyncPeriod, 10, 10, promutils.NewTestScope()) + cache, err := newAutoRefreshCacheWithClock("fake1", syncFakeItem, rateLimiter, testResyncPeriod, 10, 10, promutils.NewTestScope(), fakeClock) assert.NoError(t, err) ctx, cancel := context.WithCancel(context.Background()) @@ -94,18 +96,21 @@ func TestCacheFour(t *testing.T) { } assert.EventuallyWithT(t, func(c *assert.CollectT) { + // trigger periodic sync + fakeClock.Step(testResyncPeriod) + for i := 1; i <= 10; i++ { item, err := cache.Get(fmt.Sprintf("%d", i)) assert.NoError(c, err) assert.Equal(c, 10, item.(fakeCacheItem).val) } - }, 3*time.Second, 100*time.Millisecond) + }, 3*time.Second, time.Millisecond) cancel() }) t.Run("Not Found", func(t *testing.T) { // the size of the cache is at least as large as the number of items we're storing - cache, err := NewAutoRefreshCache("fake2", syncFakeItem, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope()) + cache, err := newAutoRefreshCacheWithClock("fake2", syncFakeItem, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope(), fakeClock) assert.NoError(t, err) ctx, cancel := context.WithCancel(context.Background()) @@ -133,7 +138,7 @@ func TestCacheFour(t *testing.T) { }) t.Run("Enqueue nothing", func(t *testing.T) { - cache, err := NewAutoRefreshCache("fake3", syncTerminalItem, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope()) + cache, err := newAutoRefreshCacheWithClock("fake3", syncTerminalItem, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope(), fakeClock) assert.NoError(t, err) ctx, cancel := context.WithCancel(context.Background()) @@ -147,15 +152,16 @@ func TestCacheFour(t *testing.T) { assert.NoError(t, err) } - // Wait half a second for all resync periods to complete + // Enqueue first batch + fakeClock.Step(testResyncPeriod) // If the cache tries to enqueue the item, a panic will be thrown. - time.Sleep(500 * time.Millisecond) + fakeClock.Step(testResyncPeriod) cancel() }) t.Run("Test update and delete cache", func(t *testing.T) { - cache, err := NewAutoRefreshCache("fake3", syncTerminalItem, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope()) + cache, err := newAutoRefreshCacheWithClock("fake3", syncTerminalItem, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope(), fakeClock) assert.NoError(t, err) ctx, cancel := context.WithCancel(context.Background()) @@ -167,24 +173,27 @@ func TestCacheFour(t *testing.T) { }) assert.NoError(t, err) - // Wait half a second for all resync periods to complete // If the cache tries to enqueue the item, a panic will be thrown. - time.Sleep(500 * time.Millisecond) + fakeClock.Step(testResyncPeriod) err = cache.DeleteDelayed(itemID) assert.NoError(t, err) - time.Sleep(500 * time.Millisecond) - item, err := cache.Get(itemID) - assert.Nil(t, item) - assert.Error(t, err) + assert.EventuallyWithT(t, func(c *assert.CollectT) { + // trigger a sync + fakeClock.Step(testResyncPeriod) + + item, err := cache.Get(itemID) + assert.Nil(c, item) + assert.Error(c, err) + }, 3*time.Second, time.Millisecond) cancel() }) t.Run("Test panic on sync and shutdown", func(t *testing.T) { syncer := &panickingSyncer{} - cache, err := NewAutoRefreshCache("fake3", syncer.sync, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope()) + cache, err := newAutoRefreshCacheWithClock("fake3", syncer.sync, rateLimiter, testResyncPeriod, 10, 2, promutils.NewTestScope(), fakeClock) assert.NoError(t, err) ctx, cancel := context.WithCancel(context.Background()) @@ -198,15 +207,12 @@ func TestCacheFour(t *testing.T) { // wait for all workers to run assert.Eventually(t, func() bool { + // trigger a sync + fakeClock.Step(testResyncPeriod) + return syncer.callCount.Load() == int32(10) }, 5*time.Second, time.Millisecond) - // wait some more time - time.Sleep(500 * time.Millisecond) - - // all workers should have shut down. - assert.Equal(t, int32(10), syncer.callCount.Load()) - cancel() }) } @@ -214,6 +220,7 @@ func TestCacheFour(t *testing.T) { func TestQueueBuildUp(t *testing.T) { testResyncPeriod := time.Hour rateLimiter := workqueue.DefaultControllerRateLimiter() + fakeClock := testingclock.NewFakeClock(time.Now()) syncCount := atomic.NewInt32(0) m := sync.Map{} @@ -231,7 +238,7 @@ func TestQueueBuildUp(t *testing.T) { } size := 100 - cache, err := NewAutoRefreshCache("fake2", alwaysFailing, rateLimiter, testResyncPeriod, 10, size, promutils.NewTestScope()) + cache, err := newAutoRefreshCacheWithClock("fake2", alwaysFailing, rateLimiter, testResyncPeriod, 10, size, promutils.NewTestScope(), fakeClock) assert.NoError(t, err) ctx := context.Background() @@ -244,16 +251,24 @@ func TestQueueBuildUp(t *testing.T) { } assert.NoError(t, cache.Start(ctx)) - time.Sleep(5 * time.Second) - assert.Equal(t, int32(size), syncCount.Load()) + + // wait for all workers to run + assert.Eventually(t, func() bool { + // trigger a sync and unlock the work queue + fakeClock.Step(time.Millisecond) + + return syncCount.Load() == int32(size) + }, 5*time.Second, time.Millisecond) } func TestInProcessing(t *testing.T) { syncPeriod := time.Millisecond + fakeClock := testingclock.NewFakeClock(time.Now()) cache := &autoRefresh{ processing: &sync.Map{}, syncPeriod: syncPeriod, + clock: fakeClock, } assert.False(t, cache.inProcessing("test")) diff --git a/go.mod b/go.mod index 8c8053def6..6c25974da0 100644 --- a/go.mod +++ b/go.mod @@ -179,7 +179,6 @@ require ( github.com/tidwall/pretty v1.2.0 // indirect github.com/tidwall/sjson v1.2.5 // indirect github.com/wI2L/jsondiff v0.5.0 // indirect - github.com/wolfeidau/humanhash v1.1.0 // indirect go.opencensus.io v0.24.0 // indirect go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1 // indirect diff --git a/go.sum b/go.sum index 63453bbd87..ac7b9f5987 100644 --- a/go.sum +++ b/go.sum @@ -1337,8 +1337,6 @@ github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6Kllzaw github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= github.com/wI2L/jsondiff v0.5.0 h1:RRMTi/mH+R2aXcPe1VYyvGINJqQfC3R+KSEakuU1Ikw= github.com/wI2L/jsondiff v0.5.0/go.mod h1:qqG6hnK0Lsrz2BpIVCxWiK9ItsBCpIZQiv0izJjOZ9s= -github.com/wolfeidau/humanhash v1.1.0 h1:06KgtyyABJGBbrfMONrW7S+b5TTYVyrNB/jss5n7F3E= -github.com/wolfeidau/humanhash v1.1.0/go.mod h1:jkpynR1bfyfkmKEQudIC0osWKynFAoayRjzH9OJdVIg= github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c/go.mod h1:lB8K/P019DLNhemzwFU4jHLhdvlE6uDZjXFejJXr49I= github.com/xdg/stringprep v0.0.0-20180714160509-73f8eece6fdc/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y= github.com/xdg/stringprep v1.0.0/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y= diff --git a/rfc/system/5103-offloaded-literal.md b/rfc/system/5103-offloaded-literal.md new file mode 100644 index 0000000000..fb835a6aab --- /dev/null +++ b/rfc/system/5103-offloaded-literal.md @@ -0,0 +1,119 @@ +# [RFC] Offloaded Raw Literals + +**Authors:** + +- @wild-endeavor +- @EngHabu +- @katrogan + +## 1 Executive Summary + +Flyte depends on a series of `inputs.pb` and `outputs.pb` files to do communication between nodes. This has typically served us well, except for the occasional map task that produces a large Literal collection output. This large collection typically exceeds default gRPC and configured storage limits. We sometimes also run into this issue for large dataclasses. This RFC proposes a mechanism that allows the offloading of any Literal, to get around size limitations for passing large Literal protobuf messages in the system. + +## 2 Motivation +A [cursory search](https://discuss.flyte.org/?threads%5Bquery%5D=LIMIT_EXCEEDED) of Slack history shows a few times that this has come up before (and I remember some other instances, I think that search term just wasn't included). This is something that we've historically addressed by just increasing the size of the grpc message that's allowed, but this is an unsustainable solution and severely reduces the utility of large-fan-out map tasks. + +## 3 Proposed Implementation +We propose configuring propeller to offload large literal collections, using the following config + +```yaml +type LiteralOffloadingConfig struct { + Enabled bool + // Maps flytekit SDK to minimum supported version that can handle reading offloaded literals. + SupportedSDKVersions map[string]string + // Default, 10Mbs. Determines the size of a literal at which to trigger offloading + MinSizeInMBForOffloading uint64 + // Fail fast threshold + MaxSizeInMBForOffloading uint64 +} + +``` + +### 3.1 Offloaded Literal IDL +Update the `Literal` [message](https://github.com/flyteorg/flyte/blob/4a7c3c0040b1995a43939407b99ca3e87b1eb752/flyteidl/protos/flyteidl/core/literals.proto#L94-L114) +like so + +```protobuf +message Literal { + oneof value { + // A simple value. + Scalar scalar = 1; + // A collection of literals to allow nesting. + LiteralCollection collection = 2; + // A map of strings to literals. + LiteralMap map = 3; + } + ... + // ** new below this line ** + // If this literal is offloaded, this field will contain metadata including the offload location. + string uri = 6; + // Includes information about the size of the literal. + uint64 size_bytes = 7; +} + +``` + +### 3.2 Flyte Propeller +Once offloading is enabled in the deployment config, flytepropeller can read from the [RuntimeMetadata](https://github.com/flyteorg/flyte/blob/f448a0358d8706a09b65b96543134f629327d755/flyteidl/protos/flyteidl/core/tasks.proto#L71-L87) in the task config to determine the SDK version. + +When writing outputs in the [remote_file_output_writer](https://github.com/flyteorg/flyte/blob/2ca31119d6b9258661a71f38e450f93b6692402c/flyteplugins/go/tasks/pluginmachinery/ioutils/remote_file_output_writer.go#L56-L84) the source code should detect whether the literal size exceeds the configured minimum and +- if the task is using a newer SDK version that supports reading offloaded literals, offload the literal to the configured storage backend and update the literal with the offload URI and size. +- if the task is using an older SDK version that doesn't support offloaded literals, fail the task with an error message indicating that the task output is too large and the user should update their SDK version. Downstream tasks will need to understand how to consume the offloaded literal and will need to be on newer version of the SDK as well. + +### 3.3 Flytekit & Copilot +Flytekit and Copilot will both need to detect that a Literal has been offloaded and know to download it. +- in Flytekit this can be done by checking the `uri` field in the Literal message when converting a literal [to_python_value](https://github.com/flyteorg/flytekit/blob/e394af0be9f904fbf8be675eaa8b8cdc24311ced/flytekit/core/type_engine.py#L1134) +- in Copilot, the data downloader [literal handling](https://github.com/flyteorg/flyte/blob/5f4199899922ca63f7690c82dfca42a783db64c3/flytecopilot/data/download.go#L219-L248) should fetch the value + +As a follow-up, we can also implement literal offloading in the SDK for conventional python tasks. Flytekit should also know how to offload the data. This should be done transparently to the user. + +We should fail fast in the SDK for too-large literals as part of the initial round of changes. + +**Open Question:** How will flytekit know to fail though if propeller hasn't been updated? + +### 3.4 Other Implications +#### Flytekit Remote +Flytekit Remote will need to be updated to handle offloaded literals. In order to fetch offloaded literals by URI, users must now authenticate with their cloud provider on their machines using a role which has read access to the _metadata bucket_. + +#### Console +Console code should show the offloaded literal URI and gracefully handle nil Literal [values](https://github.com/flyteorg/flyte/blob/4a7c3c0040b1995a43939407b99ca3e87b1eb752/flyteidl/protos/flyteidl/core/literals.proto#L96-L105). + +## 4 Metrics & Dashboards + +*What are the main metrics we should be measuring? For example, when interacting with an external system, it might be the external system latency. When adding a new table, how fast would it fill up?* + +## 5 Drawbacks + +*Are there any reasons why we should not do this? Here we aim to evaluate risk and check ourselves.* + +## 6 Alternatives + +Alternate suggestions that were proposed include + +* For map tasks, change the type of the output to a Union of the current user defined List and a new Offloaded type. We felt this would be a bit awkward since it changes the user-facing type itself (like if you were to pull up the map task definition in the API endpoint). It's also not extensible to other types of literals (maps of large dataclasses for example). + +* Build off of the input wrapper construct that's still in [PR](https://github.com/flyteorg/flyte/pull/4298). The idea was to have the wrapper contain in large cases, a reference to the data, and in small cases, the data itself. We didn't fully like this idea because the entire input set or output set needs to be offloaded. + * If the task downstream of a map task takes both the output list, along with some other input, after creating and upload the large pb file for the map task's output, Propeller would need to re-upload the entire large list or map (one time for each downstream task). If the offloading is done per literal, Propeller can just upload once and use. +* Modify the workflow CRD to include the offloading bits so that they're respected at execution time, and serialized at registration time. This is a bit heavier handed than just respecting the SDK version + +## 7 Potential Impact and Dependencies + +There's a couple edge cases that will just not work. + +* If the map task is of an older flytekit version but for some reason the downstream task is of a newer version, Propeller will fail unnecessarily. +* If the map task is a newer version, but the downstream task is an older version, the downstream task will fail correctly. +* If workflow is using an older SDK version and launches a child workflow with a newer SDK version, the parent workflow will fail to resolve the child workflow outputs + +Are there concerns about the fact that if we're offloading data once, and then sharing the pointer, we're no longer copying-by-value? Does this break any of the guarantees of Flyte and will we need to be more careful in the future around other changes to avoid issues? + +## 8 Unresolved questions + +Should we create a new oneof that's offloaded? + +Is there anything around sampled data, or automatically computed actual metadata (like number of elements in the list) that we should do? + +## 9 Conclusion + +Moving to literal offloading fixes a common and frustrating pain point around map tasks. It's a relatively simple change that should have a big impact on the usability of Flyte. + +```