Skip to content

Commit

Permalink
Update Flyte components (#4525)
Browse files Browse the repository at this point in the history 
* Update Flyte Components

Signed-off-by: Flyte-Bot <[email protected]>

* Add CHANGELOG-v1.10.6.md and bump version in rsts/conf.py

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix typo

Signed-off-by: Eduardo Apolinario <[email protected]>

* Avoid rate-limiting in gh workflows by using authenticated requests

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Flyte-Bot <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: eapolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
  • Loading branch information
@flyte-bot @eapolinario
3 people authored Dec 6, 2023
1 parent 350bb2b commit c049865
Show file tree
Hide file tree
Showing 25 changed files with 343 additions and 154 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/checks.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,8 @@ jobs:
with:
component: ${{ matrix.component }}
go-version: ${{ needs.unpack-envvars.outputs.go-version }}
secrets:
FLYTE_BOT_PAT: ${{ secrets.FLYTE_BOT_PAT }}

build_docker_images:
name: Build Images
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/go_generate.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ on:
go-version:
required: true
type: string
secrets:
FLYTE_BOT_PAT:
required: true
jobs:
generate:
runs-on: ubuntu-latest
Expand All @@ -19,6 +22,8 @@ jobs:
steps:
- uses: actions/checkout@v3
- uses: arduino/setup-protoc@v1
with:
repo-token: ${{ secrets.FLYTE_BOT_PAT }}
- uses: bufbuild/buf-setup-action@v1
- uses: actions/setup-go@v3
with:
Expand Down
157 changes: 157 additions & 0 deletions CHANGELOG/CHANGELOG-v1.10.6.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
# Flyte 1.10.6 Release

Due to a mishap in the move to the monorepo, we ended up generating the git tags between 1.10.1 to 1.10.5, so in order to decrease the confusion we decided to skip those patch versions and go straight to the next available version.

We've shipped a ton of stuff in this patch release, here are some of the highlights.

## GPU Accelerators
You'll be able to get more fine-grained in the use GPU Accelerators in your tasks. Here are some examples:

No preference of GPU accelerator to use:
```
@task(limits=Resources(gpu="1"))
def my_task() -> None:
...
```

Schedule on a specific GPU accelerator:
```
from flytekit.extras.accelerators import T4
@task(
limits=Resources(gpu="1"),
accelerator=T4,
)
def my_task() -> None:
...
```

Schedule on a Multi-instance GPU (MIG) accelerator with no preference of partition size:
```
from flytekit.extras.accelerators import A100
@task(
limits=Resources(gpu="1"),
accelerator=A100,
)
def my_task() -> None:
...
```

Schedule on a Multi-instance GPU (MIG) accelerator with a specific partition size:
```
from flytekit.extras.accelerators import A100
@task(
limits=Resources(gpu="1"),
accelerator=A100.partition_1g_5gb,
)
def my_task() -> None:
...
```

Schedule on an unpartitioned Multi-instance GPU (MIG) accelerator:
```
from flytekit.extras.accelerators import A100
@task(
limits=Resources(gpu="1"),
accelerator=A100.unpartitioned,
)
def my_task() -> None:
...
```

## Improved support for Ray logs
https://github.com/flyteorg/flyte/pull/4266 opens the door for RayJob logs to be persisted.

In https://github.com/flyteorg/flyte/pull/4397 we added support for a link to a Ray dashboard to show up in the task card.

## Updated grafana dashboards
We updated the official grafana dashboards in https://github.com/flyteorg/flyte/pull/4382.

## Support for Azure AD
A new version of our stow fork added support for Azure AD in https://github.com/flyteorg/stow/pull/9.

## Full changelog:
* Restructure Flyte releases by @eapolinario in https://github.com/flyteorg/flyte/pull/4304
* Use debian bookworm as single binary base image by @eapolinario in https://github.com/flyteorg/flyte/pull/4311
* Use local version in single-binary by @eapolinario in https://github.com/flyteorg/flyte/pull/4294
* Accessibility for README by @mishmanners in https://github.com/flyteorg/flyte/pull/4322
* Add tests in `flytepropeller/pkg /controller/executors` from 72.3% to 87.3% coverage by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4276
* fix: remove unused setting in deployment charts by @HeetVekariya in https://github.com/flyteorg/flyte/pull/4252
* Document simplified retry behaviour introduced in #3902 by @fg91 in https://github.com/flyteorg/flyte/pull/4022
* Ray logs persistence by @jeevb in https://github.com/flyteorg/flyte/pull/4266
* Not revisiting task nodes and correctly incrementing parallelism by @hamersaw in https://github.com/flyteorg/flyte/pull/4318
* Fix RunPluginEndToEndTest util by @andresgomezfrr in https://github.com/flyteorg/flyte/pull/4342
* Tune sandbox readiness checks to ensure that sandbox is fully accessi… by @jeevb in https://github.com/flyteorg/flyte/pull/4348
* Chore: Ensure Stalebot doesn't close issues we've not yet triaged. by @brndnblck in https://github.com/flyteorg/flyte/pull/4352
* Do not automatically close stale issues by @eapolinario in https://github.com/flyteorg/flyte/pull/4353
* Fix: Set flyteadmin gRPC port to 80 in ingress if using TLS between load balancer and backend by @fg91 in https://github.com/flyteorg/flyte/pull/3964
* Support Databricks WebAPI 2.1 version and Support `existing_cluster_id` and `new_cluster` options to create a Job by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4361
* Fixing caching on maptasks when using partials by @hamersaw in https://github.com/flyteorg/flyte/pull/4344
* Fix read raw limit by @honnix in https://github.com/flyteorg/flyte/pull/4370
* minor fix to eks-starter.yaml by @guyarad in https://github.com/flyteorg/flyte/pull/4337
* Reporting running if the primary container status is not yet reported by @hamersaw in https://github.com/flyteorg/flyte/pull/4339
* completing retries even if minSuccesses are achieved by @hamersaw in https://github.com/flyteorg/flyte/pull/4338
* Add comment to auth scope by @wild-endeavor in https://github.com/flyteorg/flyte/pull/4341
* Update tests by @eapolinario in https://github.com/flyteorg/flyte/pull/4381
* Update order of cluster resources config to work with both uctl and flytectl by @neverett in https://github.com/flyteorg/flyte/pull/4373
* Update tests in single-binary by @eapolinario in https://github.com/flyteorg/flyte/pull/4383
* Passthrough unique node ID in task execution ID for generating log te… by @jeevb in https://github.com/flyteorg/flyte/pull/4380
* Add Sections in the PR Template by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4367
* Update metadata in ArrayNode TaskExecutionEvents by @hamersaw in https://github.com/flyteorg/flyte/pull/4355
* Fixes list formatting in flytepropeller arch docs by @thomasjpfan in https://github.com/flyteorg/flyte/pull/4345
* Update boilerplate end2end tests by @hamersaw in https://github.com/flyteorg/flyte/pull/4393
* Handle all ray job statuses by @EngHabu in https://github.com/flyteorg/flyte/pull/4389
* Relocate sandbox config by @davidmirror-ops in https://github.com/flyteorg/flyte/pull/4385
* Refactor task logs framework by @jeevb in https://github.com/flyteorg/flyte/pull/4396
* Add support for displaying the Ray dashboard when a RayJob is active by @jeevb in https://github.com/flyteorg/flyte/pull/4397
* Disable path filtering for monorepo components by @eapolinario in https://github.com/flyteorg/flyte/pull/4404
* Silence NotFound when get task resource by @honnix in https://github.com/flyteorg/flyte/pull/4388
* adding consoleUrl parameterization based on partition by @lauralindy in https://github.com/flyteorg/flyte/pull/4375
* [Docs] Sensor Agent Doc by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4195
* [flytepropeller] Add Tests in v1alpha.go including `array_test.go`, `branch_test.go`, `error_test.go`, and `iface_test.go` with 0.13% Coverage Improvement by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4234
* Add more context for ray log template links by @jeevb in https://github.com/flyteorg/flyte/pull/4416
* Add ClusterRole config for Ray by @davidmirror-ops in https://github.com/flyteorg/flyte/pull/4405
* Fix and update scripts for generating grafana dashboards by @Tom-Newton in https://github.com/flyteorg/flyte/pull/4382
* Add artifacts branch to publish to buf on push by @squiishyy in https://github.com/flyteorg/flyte/pull/4450
* Add service monitor for flyte admin and propeller service by @vraiyaninv in https://github.com/flyteorg/flyte/pull/4427
* Fix Kubeflow TF Operator `GetTaskPhase` Bug by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4469
* Instrument opentelemetry by @hamersaw in https://github.com/flyteorg/flyte/pull/4357
* Delete the .github folder from each subdirectory by @pingsutw in https://github.com/flyteorg/flyte/pull/4480
* Fix the loop variable scheduler issue by @pmahindrakar-oss in https://github.com/flyteorg/flyte/pull/4468
* Databricks Plugin Setup Doc Enhancement by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4445
* Put ticker back in place in propeller gc by @eapolinario in https://github.com/flyteorg/flyte/pull/4490
* Store failed execution in flyteadmin by @iaroslav-ciupin in https://github.com/flyteorg/flyte/pull/4390
* Moving from flyteadmin - Upgrade coreos/go-oidc to v3 to pickup claims parsing fixes by @eapolinario in https://github.com/flyteorg/flyte/pull/4139
* Bump flyteorg/stow to 0.3.8 by @eapolinario in https://github.com/flyteorg/flyte/pull/4312
* Remove 'needs' from generate_flyte_manifest by @eapolinario in https://github.com/flyteorg/flyte/pull/4495
* Update Flyte components by @flyte-bot in https://github.com/flyteorg/flyte/pull/4302
* Modify how flytecopilot version is parsed from values file by @eapolinario in https://github.com/flyteorg/flyte/pull/4496
* Ignore component tags in goreleaser by @eapolinario in https://github.com/flyteorg/flyte/pull/4497
* Fix indentation of `shell: task` by @eapolinario in https://github.com/flyteorg/flyte/pull/4498
* Implemented simple echo plugin for testing by @hamersaw in https://github.com/flyteorg/flyte/pull/4489
* Correctly handle resource overrides in KF plugins by @jeevb in https://github.com/flyteorg/flyte/pull/4467
* Remove deprecated InjectDecoder by @EngHabu in https://github.com/flyteorg/flyte/pull/4507
* Fix $HOME resolution and webhook namespace by @EngHabu in https://github.com/flyteorg/flyte/pull/4509
* Add note on updating sandbox cluster configuration by @jeevb in https://github.com/flyteorg/flyte/pull/4510
* Add New PR Template by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4512
* [Docs] Databricks Agent Doc by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4008
* Bump version of goreleaser gh action to v5 by @eapolinario in https://github.com/flyteorg/flyte/pull/4519
* Kf operators use `GetReplicaFunc` (Error Handling) by @Future-Outlier in https://github.com/flyteorg/flyte/pull/4471

## New Contributors
* @HeetVekariya made their first contribution in https://github.com/flyteorg/flyte/pull/4252
* @andresgomezfrr made their first contribution in https://github.com/flyteorg/flyte/pull/4342
* @brndnblck made their first contribution in https://github.com/flyteorg/flyte/pull/4352
* @guyarad made their first contribution in https://github.com/flyteorg/flyte/pull/4337
* @neverett made their first contribution in https://github.com/flyteorg/flyte/pull/4373
* @thomasjpfan made their first contribution in https://github.com/flyteorg/flyte/pull/4345
* @lauralindy made their first contribution in https://github.com/flyteorg/flyte/pull/4375
* @Tom-Newton made their first contribution in https://github.com/flyteorg/flyte/pull/4382
* @vraiyaninv made their first contribution in https://github.com/flyteorg/flyte/pull/4427
2 changes: 1 addition & 1 deletion charts/flyte-binary/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Chart for basic single Flyte executable deployment
| configuration.auth.oidc.clientId | string | `""` | |
| configuration.auth.oidc.clientSecret | string | `""` | |
| configuration.co-pilot.image.repository | string | `"cr.flyte.org/flyteorg/flytecopilot"` | |
| configuration.co-pilot.image.tag | string | `"v1.10.6-b0"` | |
| configuration.co-pilot.image.tag | string | `"v1.10.6"` | |
| configuration.database.dbname | string | `"flyte"` | |
| configuration.database.host | string | `"127.0.0.1"` | |
| configuration.database.options | string | `"sslmode=disable"` | |
Expand Down
2 changes: 1 addition & 1 deletion charts/flyte-binary/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ configuration:
# repository CoPilot sidecar image repository
repository: cr.flyte.org/flyteorg/flytecopilot # FLYTECOPILOT_IMAGE
# tag CoPilot sidecar image tag
tag: v1.10.6-b0 # FLYTECOPILOT_TAG
tag: v1.10.6 # FLYTECOPILOT_TAG
# agentService Flyte Agent configuration
agentService:
defaultAgent:
Expand Down
12 changes: 6 additions & 6 deletions charts/flyte-core/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,8 @@ helm install gateway bitnami/contour -n flyte
| configmap.clusters.clusterConfigs | list | `[]` | |
| configmap.clusters.labelClusterMap | object | `{}` | |
| configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI |
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.6-b0","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.6-b0","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.6","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.10.6","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
| configmap.core | object | `{"manager":{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"},"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration |
| configmap.core.manager | object | `{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config). |
| configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). |
Expand Down Expand Up @@ -125,7 +125,7 @@ helm install gateway bitnami/contour -n flyte
| datacatalog.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
| datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
| datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment |
| datacatalog.image.tag | string | `"v1.10.6-b0"` | Docker image tag |
| datacatalog.image.tag | string | `"v1.10.6"` | Docker image tag |
| datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment |
| datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods |
| datacatalog.priorityClassName | string | `""` | Sets priorityClassName for datacatalog pod(s). |
Expand Down Expand Up @@ -157,7 +157,7 @@ helm install gateway bitnami/contour -n flyte
| flyteadmin.extraArgs | object | `{}` | Appends extra command line arguments to the serve command |
| flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | |
| flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment |
| flyteadmin.image.tag | string | `"v1.10.6-b0"` | |
| flyteadmin.image.tag | string | `"v1.10.6"` | |
| flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create |
| flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment |
| flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods |
Expand Down Expand Up @@ -209,7 +209,7 @@ helm install gateway bitnami/contour -n flyte
| flytepropeller.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
| flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | |
| flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment |
| flytepropeller.image.tag | string | `"v1.10.6-b0"` | |
| flytepropeller.image.tag | string | `"v1.10.6"` | |
| flytepropeller.manager | bool | `false` | |
| flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment |
| flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods |
Expand All @@ -236,7 +236,7 @@ helm install gateway bitnami/contour -n flyte
| flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files |
| flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
| flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment |
| flytescheduler.image.tag | string | `"v1.10.6-b0"` | Docker image tag |
| flytescheduler.image.tag | string | `"v1.10.6"` | Docker image tag |
| flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment |
| flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods |
| flytescheduler.priorityClassName | string | `""` | Sets priorityClassName for flyte scheduler pod(s). |
Expand Down
Loading

0 comments on commit c049865

Please sign in to comment.