Flyte v1.13.0 milestone release

Flyte 1.13.0 Release Notes

Flyte 1.13.0 brings a host of new features, optimizations, and fixes, enhancing the platform's functionality and user experience. This release also welcomes several new contributors to the Flyte community. Below are the highlights of this release.

🚀 New Features & Improvements

1. Features

   • Watch agent metadata service
   • Allow controlling in which task phases log links are shown
   • Add executionClusterLabel

2. Helm Charts & Manifests

   • Grafana dashboard updates for better monitoring and observability.
   • Core helm chart updates for propeller configuration of agent service.
   • Helm chart updates related to Prometheus, Webhook HPA, and Flyteconsole probes.

3. Documentation Improvements

   • Added examples and guides, including a guide for setting up the OpenAI batch agent backend and updated Optimization Performance docs.
   • Fix doc link to testing agent on local cluster and replace Azure AD OIDC URL.
   • Clarify networking configurations between data plane propeller and control plane data catalog in multi-cluster deployments.

4. Performance & Bug Fixes

   • Fix cache and token management issues, including addressing auto-refresh cache race conditions and improving token refresh mechanisms.
   • Fix SQL insert operations to handle NULL values appropriately.
   • Keep EnvFrom from pod template
   • Miscellaneous fixes for pod templates, Kubeflow webhook errors, and logging configurations.

5. Flytectl

   • Add prefetch functionality for paginator
   • Fix upgrade and version commands

🔧 Housekeeping & Deprecations

1. Miscellaneous
   • Add OTLP and sampling to otelutils
   • Refactor distributed job using common ReplicaSpec
   • Fix typos using codespell CI job.
   • Remove obsolete Flyte config files.
   • Upgraded various dependencies, including Golang, Docker, and others.
   • Move to upstream mockery, which simplifies the migration to mockery-v2
   • Key-value execution tags

🆕 New Contributors

• @zychen5186 for adding prefetch functionality for paginator.
• @EraYaN for replacing Azure AD OIDC URL.
• @Dlougach for making BaseURL insensitive to trailing slashes for metadata endpoint redirect.
• @flixr for honoring redoc.enabled=false in charts.
• @trevormcguire for including group in apiVersion in plugin_collector.
• @va6996 for inheriting execution cluster label from source execution.
• @mhotan for Helm chart updates related to Prometheus, Webhook HPA, and Flyteconsole probes.
• @eltociear for updating token_source.go.

Full Changelog

flytectl/v0.9.0

Changelog

• f87a049 Add API to get domain (#5443)
• e1d9c5c Add OTLP and sampling to otelutils (#5504)
• ba88e82 Add lint-fix make target, add gci to flytectl, remove goimports make target (#5484)
• 791471c Fix: Make 'flytectl compile' consider launchplans used within workflows (#5463)
• f54b74e Fix: replace with in OSS docs (#5501)
• ed40a94 Remove doc building from flytectl (#5494)
• c97a0e9 Remove github.ref from group (#5518)
• c512571 reimplement functions in go-mouff-update, use ghreposervice (#5470)

Flyte v1.13.0-rc0 milestone release

Flyte v1.13.0-rc0 Release Notes

Major Features and Improvements

• Key-value execution tags: Enhanced execution tagging capabilities. (#5453)
• Watch agent metadata service: Improved monitoring of agent metadata. (#5017)
• Distributed job refactoring: Refactored distributed job using common ReplicaSpec. (#5355)
• Execution cluster label inheritance: Executions now inherit cluster labels from source executions. (#5431)
• Domain API: Added new API to retrieve domain information. (#5443)
• Execution environment versioning: Added version to ExecutionEnv proto message. (#5506)
• OTLP and sampling in otelutils: Enhanced observability with OpenTelemetry Protocol (OTLP) support. (#5504)

Notable Changes

• Removed mmcloud plugin. (#5468)
• Updated k3s version to 1.29.0. (#5475)
• Improved auth flow to support custom base URLs in deployments. (#5192)
• Added flyteconsole URL to FlyteWorkflow CRD. (#5449)
• Enhanced 'flytectl compile' to consider launch plans within workflows. (#5463)
• Introduced control over task phases for log link display. (#4726)
• Improved Helm chart configurations for Prometheus, Webhook HPA, and Flyteconsole probes. (#5508)

Bug Fixes and Optimizations

• Fixed flaky auto_refresh_test. (#5438)
• Corrected NULL to empty string in SQL insert for migrations. (#5482)
• Resolved issues with broken mermaid diagrams in documentation. (#5498)
• Fixed Ray plugin to use default service account if not set in task metadata. (#5499)

Documentation and Usability

• Updated community page. (#5496)
• Improved documentation on logging link lifetime configuration. (#5503)
• Replaced 'uctl' with 'flytectl' in OSS docs for consistency. (#5501)

New Contributors

• @Dlougach, @flixr, @trevormcguire, @va6996, @mhotan, and @eltociear made their first contributions to the project.

For a complete list of changes, please refer to the full changelog.

flytectl/v0.8.24

Changelog

• 9ffc54f Indicate that jq is now required to install flytectl via the script (#5440)
• f08eb47 Upstream revert revert auth token fix (#5407)

Flyte v1.12.1-rc0 milestone release

Flyte 1.12.1-rc0 Release Notes

Flyte 1.12.1-rc0 is a release candidate that focuses on documentation enhancements, bug fixes, and improvements to the core infrastructure. This release also includes contributions from a new member of the Flyte community. Below are the highlights of this release.

🚀 New Features & Improvements

1. Documentation Improvements

   • Removed the source code renderer section from the Decks article.
   • Added documentation for OpenAI batch agent backend setup.
   • Updated the example Flyte agent Dockerfile.
   • Fixed documentation link to testing agent on local cluster.
   • Fixed Kubeflow webhook error in the documentation.
   • Updated Flytekit version to 1.12.1b2 in monodocs requirements.
   • Updated Flytefile.md and replaced SHA instead of master in RLI links.

2. Infrastructure and Configuration

   • Reverted "Ensure token is refreshed on Unauthenticated".
   • Updated core Helm chart for propeller configuration of agent service.
   • Fixed Flytectl install script in the monorepo.
   • Moved to upstream mockery.
   • Used a different git command to match the Flyteidl tags.

3. Bug Fixes

   • Handled auto-refresh cache race condition.
   • Fixed typos using codespell CI job.
   • Fixed build failure in monodocs.
   • Replaced Azure AD OIDC URL with the correct one.

4. Miscellaneous

   • Updated the lock file.
   • Added executionClusterLabel for better execution cluster management.

🆕 New Contributors

• @EraYaN for replacing Azure AD OIDC URL with the correct one.

Full Changelog

flytectl/v0.8.23

Changelog

• 1078e09 Add executionClusterLabel (#5394)
• ba3647f Fix typos using codespell CI job (#5418)
• 7d2f0d0 Revert "Ensure token is refreshed on Unauthenticated (#5388)" (#5404)
• 0583f77 [monorepo] Fix flytectl install script (#5405)

flytectl/v0.8.21

Changelog

• 1384b32 Ensure token is refreshed on Unauthenticated (#5388)
• 2f1f813 Skip flytectl upgrade tests (#5381)

flytectl/v0.8.20

Changelog

• 11fbeca Fix flytectl demo start (#5370)

flytectl/v0.8.19

Changelog

• b5d78aa #minor Documentation fix and refactoring in sandbox (#131)
• 8db28dd #minor Revert "Adopt flyteidl's ordered variable map change" (#180)
• 936a24a #minor Updated Sandbox config, with automated data configuration (#440)
• 2766c18 #minor feat: pagination support for get commands (#234)
• 41e0a40 #none Fix DCO make target (#139)
• 5da69f5 #none add go mod tidy in flyteidl release workflow (#151)
• 81471f1 #none fixed storage config in docs (#138)
• d75142c 3 usability improvements -Named executions, improved connection handling and better logging (#349)
• 2a25558 Absolute path for Source dir (#126)
• dd4655d Add ContentMD5 to fast register upload request (#303)
• 90e01a6 Add Slack button to README (#198)
• 2d65956 Add ability to recover executions (#149)
• 3e4323a Add additional nil check (#156)
• 3e9525f Add commands to fetch active and archived launchplans (#331)
• e2ed562 Add console endpoint config section and init arg (#361)
• 26d976d Add default input default behavior check for scheduler workflow registration (#368)
• 9deb2ec Add deployment docs (#127)
• d330473 Add extra host to start command (#369)
• ab171cd Add pflag binding and fix loading of config (#135)
• aa1f211 Add prefetch functionality for paginator (#5310)
• a64a416 Add support dev cluster (#353)
• d69f4b7 Add support for gate nodes (#399)
• 89cb374 Add upgrade command with some refactor (#152)
• d7e0624 Added Create command (#17)
• bb499a9 Added Github workflow for generating docs (#85)
• 73e4aa3 Added HOMEDIR/.flyte to search path for config.yaml (#59)
• a101cd6 Added Update Execution cli (#261)
• a144275 Added config init command (#123)
• 6453fab Added arm support (#206)
• 852d7ab Added boilerplate automation (#74)
• 89f6e6f Added capability to relaunch an execution (#52)
• 8924563 Added check before creating empty k8s config (#211)
• 34d7614 Added cluster resource attribute support (#65)
• c2c5649 Added command delete along with subcommand executions for terminating (#34)
• 712fdf8 Added customizable fields during registration (#80)
• b5f332a Added dco signoff preecommit (#89)
• 676f0cf Added different repo for brew release (#46)
• 3b49630 Added docs for demo and updated docs for workflow execution config with more details (#316)
• fc3c99e Added dryRun flag for printing docker command for demo and sandbox (#367)
• 02cf19e Added enableSchedule option on registration and linked missing docs (#324)
• e6855b4 Added endpoint suffix to service user for pkce token saved in token cache (#264)
• 39add71 Added env flag for sandbox start (#273)
• fdd1b15 Added error data and abort data in get execution output (#79)
• 8b1c844 Added exec command in sandbox (#122)
• 19ba5fd Added fast register functionality (#112)
• 2d2c145 Added filters flag (#82)
• 1a904e0 Added flag to pass in CAcerts (#245)
• fe72730 Added flags for using local images in sandbox (#216)
• 517beb9 Added flytectl config by env variable (#91)
• 696065c Added goreleaser dry run in build (#47)
• 400d4b6 Added gpg sign commit for flyte-bot (#92)
• 7a43a97 Added latest version check added (#111)
• 0a23db3 Added launchplan update command and moved namedentity (#170)
• 31ad383 Added minio port in sandbox (#203)
• 002d176 Added node execution data to show inputs and outputs (#136)
• 0f708dd Added non zero return code on error and AlreadyExists as success condition (#71)
• cfde70d Added option to update labels & description in project (#260)
• d5c3133 Added port for postgres in sandbox lite (#308)
• 07fccb1 Added reading of archive files for registration to flyte (#31)
• 6947a2a Added reuseable workflow (#276)
• c5ab11a Added sandbox command (#86)
• bd1d4e5 Added script for checking diff of generated code (#229)
• 5c71337 Added separate release workflow for releasing the binary (#132)
• 179da66 Added source flags in sandbox (#99)
• 0a2ddb2 Added support for arm64 in install.sh (#209)
• 4d43aee Added support for execution queue attribute (#66)
• 2762f97 Added support for pyflyte serialize fast register (#248)
• 26f139a Added support for tar.gz in register (#199)
• 4bb9675 Added transparent Get call when fetching latest version of workflow objects (#73)
• caa38e5 Added unit test for covarage (#83)
• b11a9e8 Added unit test for docker (#110)
• cf3ffa1 Added update for named entity (#54)
• 06458f5 Added validation for scheduled workflow during registration (#215)
• 3904146 Added version command (#41)
• 03be16b Added version flag in sandbox (#137)
• 36c1e4f Added version support to workflow command (#64)
• a94cbdd Added workflow flag for get launchplan (#287)
• 3749e3f Adding changes for project level matchable attr api (#357)
• 3da647f Adding flytectl.rst to index (#257)
• c68c5c2 Adding open-source boilerplate (#1)
• d04b1e2 Adding overriden filter for get workflow (#310)
• 9bcf5cc Adding support for ArrayNode (#406)
• 6cf6d1e Adding support for execution cluster label (#68)
• 5f9a2fb Adding support for structured dataset and also for node outputs in details flag (#390)
• 9b05df0 Adding support for workflow execution config (#144)
• 7c31c1e Adopt flyteidl's ordered variable map change (#158)
• 5af33e3 Always use sha for sandbox image (#266)
• a056102 Archive and Update project command (#24)
• 79e8fdd Auto rendering of JSON entities using path (#13)
• 3fb5080 Autodoc generation for cobra commands (#29)
• 790a965 Bind 30080 to 30081 in the case of sandbox (#313)
• dd369af Bind pflags to default variable fields && Remove short flags as they're not supported by pflags (#124)
• 3a48b4a Bug fix in sandbix start (#161)
• 50ee307 Bug fix in sandbox start when sandbox exist (#145)
• b55026e Bump github.com/containerd/containerd from 1.5.10 to 1.6.26 in /flytectl (#5307)
• 8db9901 Bump github.com/docker/distribution in /flytectl (#5313)
• 7b82397 Bump github.com/docker/docker in /flytectl (#5363)
• 8d2b83b Bump github.com/opencontainers/image-spec from 1.0.1 to 1.0.2 (#217)
• cb57beb Bump golang.org/x/net to v0.23.0 (#5333)
• f05acc7 Bump idna from 3.6 to 3.7 in /flytectl (#5315)
• b36a1f3 Bump jinja2 from 3.1.3 to 3.1.4 in /flytectl (#5329)
• c7d02df Bumped up flyteidl to 0.19.2 (#94)
• 29740e0 Bumped up idl to use insecureVerify flag for remote and fixed docs (#125)
• c673d3b Bumped up the flyteidl version to fix create execution issue (#53)
• 73d9cb7 Bumping go version to 1.19 (#384)
• fc2ac0a Change extra host to host-gateway (#380)
• e3977b1 Changes to 'use' in cmd folder (#235)
• dafa5d7 Changing the default log level which is used by sandbox (#129)
• 35e67f6 Check kubeConfig before starting sandbox (#363)
• b092146 Compile command (#321)
• ee25d67 Consistent language about 'git revision SHA1' (#414)
• 68324ce Consuming device flow IDL (#350)
• e5d11a8 Create executions in flytectl (#39)
• c751289 Created separate rst files for every resource (noun) (#230)
• 440ad31 Demo fit and finish (#317)
• 3414946 Devflow, boilerplate, improved code, formatting and bug fixes (#11)
• 6e20a25 Do not bump version on docs or boilerplate update (#200)
• ce0fd45 Do not upload codecoverage data from boilerplate (#478)
• 083cdbd Doc updates (#398)
• 62dfd32 Documentation setup for FlyteCTL (#26)
• 2cec59b Editorial review (#256)
• e2adac4 Expose buildkit port (#413)
• 931c651 Expose webhook service (#323)
• 2811f53 Feature/get workflow,Task,Domain,Project (#9)
• 22c203b Feature/goreleaser (#27)
• a9a45b3 Feature: update launchplan --archive to --deactivate (#449)
• a4702fb Filtering on activation state of workflows (#305)
• 3be778c Fix "Sandbox" typo on Demo page. (#346)
• 92104a8 Fix --clusterPool argument override (#360)
• 0e3f2a8 Fix case for No string to no (#419)
• a66e669 Fix config init command fails if no flyte dir doesn't exist (#150)
• d033369 Fix docs contribution guide (#197)
• b8770ee Fix docs for updating description using workflow-meta subcommand (#394)
• 1fe32fa Fix flytectl sandbox crash on M1 macs due to json-iterate (#342)
• 8dd8b05 Fix flytectlt tests (#5325)
• b03e86d Fix get task resource attribute comment (#469)
• 86e7647 Fix getAllExample (#320)
• caf9bbc Fix goreleaser (#40)
• 3e329ad Fix grammar test (#272)
• dcc2c09 Fix panic in rendering LP input descriptions (#174)
• d963c7a Fix pod task fast registration template substitution (#184)
• 2e49f75 Fix project status change on update for archived projects (#438)
• 664a604 Fix self-upgrade (#437)
• 09013f7 Fix small bug in rendering table (#14)
• 2657b26 Fix struct generation in execution file (#434)
• c516596 Fix testutils.TearDownAndVerify #none (#424)
• 95fccea Fix the error message for the update cmd (#411)
• 8cad4ed Fix typos
• 8362431 Fix: Check for git authentication when GITHUB_TOKEN exists (#463)
• 7a6ba14 Fixed continueonError bug (#292)
• f3f6765 Fixed create project (#277)
• b58b748 Fixed the non-zero return code issue in archive path (#194)
• 51226d5 Fixed the version bug and also added force flag (#246)
• 9cb07f9 Fixed validation logic for launch plan schedule registration (#280)
• e569acb Fixes in Flytectl (#109)
• 9af534f Fixes the project name and workflow name in the flytectl documentation. (#339)
• f59e5ae Fixup flytectl (#5309)
• 61eb8f2 Flyte Add Enable/Disable Options for Agent Services (#420)
• 3eb2ea3 Flytectl Doc fixes (#38)
• 514375f Flytectl get started updated (#58)
• 6923203 Force go 1.19.10 in goreleaser (#415)
• c17ee3b GH workflow change - publish binary when tag is pushed (#5)
• fb6eb2d Graphviz does not like hyphens #patch (#141)
• ec7c344 Hide global flags by overriding usage template of cobra. (#430)
• 24710f7 Hotfix register example (#172)
• 409c01c Hotfix/config init validation (#250)
• dda6460 Hydrate failure node (#456)
• e8bffb6 Image migration to ...

Flyte v1.12.0 milestone release

Flyte 1.12.0 Release Notes

Flyte 1.12.0 brings a host of new features, optimizations, and fixes, enhancing the platform's functionality and user experience. This release also welcomes several new contributors to the Flyte community. Below are the highlights of this release.

🚀 New Features & Improvements

1. Admin & Core Enhancements

   • Implemented the GetProject endpoint in FlyteAdmin.
   • Added tracking for active node and task execution counts in Propeller.
   • Set FlyteAdmin gRPC port correctly in config and included separate gRPC Ingress flag.

2. Helm Charts & Manifests

   • CI workflow enhancements for Helm charts and manifests.
   • Fixed syntax issues and rendering errors in Helm charts.
   • Updated Spark-on-K8s-operator address in Helm charts.

3. Documentation Improvements

   • Enhanced documentation across various modules, including a guide for running the newest Flyteconsole in Flyte sandbox and a troubleshooting guide for Docker errors.
   • Updated Propeller architecture documentation and added a guide for enabling/disabling local caching.

4. Performance & Bug Fixes

   • Improved lint error detection, fixing multiple issues across components.
   • Optimized Flyte components, including Golang, Protobuf, and GRPC versions.
   • Fixed several bugs related to execution phases, GPU resource overrides, and Databricks errors.

🔧 Housekeeping & Deprecations

1. Deprecated Configuration

   • Deprecated MaxDatasetSizeBytes propeller config in favor of GetLimitMegabytes storage config.
   • Removed obsolete Flyte config files.

2. Miscellaneous

   • Simplified boilerplate.
   • Regenerated Ray pflags.
   • Upgraded various dependencies, including cloudevents, logrus, and go-restful.

🆕 New Contributors

• @RRap0so for implementing the GetProject endpoint.
• @cjidboon94 for adding the GKE starter values file.
• @ddl-rliu for improving audience mismatch debugging.
• @pbrogan12 for fixing Helm chart rendering.
• @austin362667 for showing diff structure when re-registering tasks.
• @ssen85 for updating environment setup documentation.
• @dansola for changing retry error types.
• @noahjax for adding identity to task execution metadata.
• @ongkong for fixing ID bigint conversion issues.
• @sshardool for tracking active node and task execution counts.
• @Jeinhaus for adding a missing key in the auth guide.
• @yini7777 for fixing secret mounting issues.
• @Sovietaced for fixing grammatical errors in the documentation.
• @agiron123 for configuring the RunLLM widget.
• @mark-thm for removing the upper bound on FlyteIDL's protobuf dependency.

Full Changelog