Skip to content

v1.16.3
Compare
Choose a tag to compare
@gardener-robot-ci-3 gardener-robot-ci-3 released this 17 Feb 13:39
· 442 commits to master since this release
v1.16.3
d524884

[gardener]

⚠️ Breaking Changes

  • [USER] The already deprecated annotation keys confirmation.garden.sapcloud.io/deletion and shoot.garden.sapcloud.io/operation are no longer respected by Gardener components. If you are still using the deprecated annotation keys, please switch the the equivalents from the new API group - respectively confirmation.gardener.cloud/deletion and gardener.cloud/operation. (gardener/gardener#3375, @ialidzhikov)
  • [OPERATOR] The check for stale extension health checks is now switched from controllers.shootCare.staleExtensionHealthCheckThreshold to controllers.shootCare.staleExtensionHealthChecks.{enabled,threshold} in the GardenletConfiguration. It is now configurable and enabled by default. (gardener/gardener#3390, @ialidzhikov)
  • [OPERATOR] Deprecated ingress hostnames i.e., AlertManager - au.<shoot-name>.<project-name>.<seed-ingress-domain>, Grafana - gu.<shoot-name>.<project-name>.<seed-ingress-domain>, go.<shoot-name>.<project-name>.<seed-ingress-domain>, Prometheus - p.<shoot-name>.<project-name>.<seed-ingress-domain> were removed and will not be reachable anymore. Please use the hostnames introduced with Gardener v0.34.0 instead. (gardener/gardener#3380, @timuthy)
  • [OPERATOR] gardenlet's shoot-care-control is now using the gardener.cloud/role label key (until now it was garden.sapcloud.io/role) to perform health checks on controlplane components. Make sure you have first upgraded to at least Gardener v1.14 before you upgrade to this version of Gardener. (gardener/gardener#3350, @ialidzhikov)
  • [OPERATOR] With the activated cluster-autoscaler during roll-outs, following are the minimum versions required for different provider-extensions: gardener-extension-provider-aws v1.16.0, gardener-extension-provider-openstack v1.12.0, gardener-extension-provider-azure v1.14.0, gardener-extension-provider-gcp v1.12.0, gardener-extension-provider-alicloud v1.18.0, gardener-extension-provider-vsphere v0.1.0. (gardener/gardener#3332, @hardikdr)
  • [OPERATOR] If the nginx-ingress addon for a shoot used as seed is disabled then you can no longer enable it anymore. Instead, use the new managed ingress controller feature. You can find more information about it here. Existing shoots used as seeds with .spec.addons.nginxIngress.enabled=true will continue to work. (gardener/gardener#3131, @BeckerMax)
  • [DEVELOPER] A new mutating webhook for the cloudprovider secret has been added in the extensions library. With this change, the EnsurerContext of the genericmutator package has been moved to a separate context package. Please adapt your usage of genericmutator accordingly. (gardener/gardener#3348, @kon-angelo)
  • [DEPENDENCY] ⚠️ Go dependencies to kubernetes/* and kubernetes-sigs/controller-runtime were updated to v0.19.6 and v0.7.0 respectively. This imposes a lot of consequent breaking changes to go projects vendoring gardener/gardener. If your project/extension vendors gardener/gardener, please read the dedicated section in this issue carefully when upgrading your dependencies. (gardener/gardener#3393, @timebertt)
  • [DEPENDENCY] pkg/utils/secrets.BasicAuthSecretConfig does no longer allow generating bcrypt password hash. The corresponding functionality is now removed. (gardener/gardener#3365, @ialidzhikov)

✨ New Features

  • [USER] Logs from VerticalPodAutoscaler are accessible via the Vertical Pod Autoscaler dashboard in Grafana. (gardener/gardener#3456, @Kristian-ZH)
  • [USER] The shoot reconciliation flow is now waiting until all worker nodes have executed the most recent cloud-config user data. Similarly, the shoot care controller checks if the last successfully applied cloud-config user data on all nodes is outdated or not (and reports this in the EveryNodeReady condition). Please note that both features are only available for new nodes. (gardener/gardener#3396, @rfranzke)
  • [USER] The cluster-autoscaler is now activated even during rolling-update of the shoot clusters. The change in machine-controller-manager of adding the cluster-autoscaler.kubernetes.io/scale-down-disabled annotation during rolling-update is required, in order for autoscaler to not scale-down worker-pools (coming with machine-controller-manager 0.34.0). (gardener/gardener#3332, @hardikdr)
  • [OPERATOR] The specification and the configuration files of the cloud-config-downloader.service systemd service are now updated regularly with the original OperatingSystemConfig. (gardener/gardener#3449, @vpnachev)
  • [OPERATOR] It is now possible to trigger a restart of systemd services on particular shoot worker nodes by annotating the corresponding Node object with worker.gardener.cloud/restart-systemd-services=kubelet, for example. (gardener/gardener#3396, @rfranzke)
  • [OPERATOR] The .spec.revisionHistoryLimit is now set to 1 for Deployments. (gardener/gardener#3374, @rfranzke)
  • [OPERATOR] It is now possible to specify the spec.settings.loadBalancerServices.annotations field for shooted seeds via the "shoot.gardener.cloud/use-as-seed" annotation. You can do this by specifying the loadBalancerServices.annotations.* option - for example loadBalancerServices.annotations.service.beta.kubernetes.io/aws-load-balancer-type=nlb. (gardener/gardener#3344, @ialidzhikov)
  • [OPERATOR] The gardener admission controller now exposes metrics (gardener/gardener#3293, @wyb1)
  • [OPERATOR] New SeedKubeScheduler feature gate is added to gardenlet. When enabled, it deploys a custom kube-scheduler in gardener-kube-scheduler namespace of Seed clusters with Kubernetes version 1.17 or greater. The scheduler assigns Shoot control plane Pods to Nodes with higher resource utilization, resulting in better bin-packing of control planes. (gardener/gardener#3243, @mvladev)
  • [OPERATOR] Gardener now offers to manage a dedicated ingress controller for seed clusters (earlier, this was a manual operator task when registering seeds). You can find more information about it here. (gardener/gardener#3131, @BeckerMax)
  • [DEVELOPER] Nodeless local dev setups now run with etcd v3.4.14 and K8s v1.20.2. (gardener/gardener#3426, @timuthy)
  • [DEVELOPER] Gardener can now support shoot clusters with Kubernetes version 1.20. In order to allow creation/update of 1.20 clusters you will have to update the version of your provider extension(s) to a version that supports 1.20 as well. Please consult the respective releases and notes in the provider extension's repository. (gardener/gardener#3296, @rfranzke)
  • [DEPENDENCY] The ConfigMaps and Secrets used to store the config and state of terraform now have owner reference to the Infrastructure resource. (gardener/gardener#3275, @vpnachev)

🐛 Bug Fixes

  • [USER] metrics-server's version is updated from v0.4.1 to v0.4.2 to adopt upstream fix that was causing metrics-server to be unavailable for a while after rolling update of Nodes. (gardener/gardener#3516, @ialidzhikov)
  • [USER] Fixed a bug where service, pod or node CIDRs that are private network (RFC1918) or carrier-grade NAT (RFC6598) IPv4 blocks would produce an invalid allow-to-private-networks networkpolicy. (gardener/gardener#3462, @mvladev)
  • [USER] A bug has been fixed that prevented shoot clusters from coming up in case .spec.kubernetes.allowPrivilegedContainers=false. (gardener/gardener#3409, @rfranzke)
  • [USER] An race issue causing immediate wake up after hibernation to fail is now fixed. The hibernation is now waiting until the kube-apiserver Service is cleaned up. (gardener/gardener#3289, @ialidzhikov)
  • [OPERATOR] An issue has been fixed which caused Gardener to incorrectly label shoot clusters as unhealthy (shoot.gardener.cloud/status: unhealthy) for a short period of time. (gardener/gardener#3567, @timuthy)
  • [OPERATOR] The affinity section is removed from the Loki StatefulSet for the integration tests (gardener/gardener#3526, @vlvasilev)
  • [OPERATOR] An issue causing the generic Worker actuator to not wait until the finalizer of the out-of-tree machine controller provider is removed from the credentials secret is now fixed. (gardener/gardener#3498, @ialidzhikov)
  • [OPERATOR] An issue in the API validation has been fixed which prevented the managed ingress feature for seeds being enabled. (gardener/gardener#3448, @BeckerMax)
  • [OPERATOR] A bug has been fixed which prevented proper auto-scaling of components under control of HVPA. (gardener/gardener#3446, @rfranzke)
  • [OPERATOR] Fix gardener-seed-admission controller, etcd backup-restore and extension parsers time format (gardener/gardener#3429, @vlvasilev)
  • [OPERATOR] The generic Worker actuator does now wait until the machine-controller-manager finalizer is removed from the credentials secret that is referenced from the machine classes. (gardener/gardener#3425, @ialidzhikov)
  • [OPERATOR] Node Problem Detector is now matched by gardener.cloud--allow-to-dns and gardener.cloud--allow-to-apiserver networkpolicies can run with deny-all networkpolicy in kube-system namespace. (gardener/gardener#3424, @mvladev)
  • [OPERATOR] fix CRD for extension types to allow storing anything in status.state. (gardener/gardener#3422, @MartinWeindel)
  • [OPERATOR] Fixes a bug causing newly created Seeds to fail during bootstrap (gardener/gardener#3400, @BeckerMax)
  • [OPERATOR] A side-car container is added to kube-proxy that deletes the incorrect conntrack table entries which sometime occur after restart of kube-proxy and prevent the establishment of a tcp connection to the api-server. (gardener/gardener#3395, @DockToFuture)
  • [OPERATOR] Fixed a bug of the managed istio feature flag where the istio rolebinding was created in the wrong namespace. (gardener/gardener#3382, @danielfoehrKn)
  • [OPERATOR] An issue has been fixed which caused unwanted restarts for Grafana instances. (gardener/gardener#3379, @timuthy)
  • [OPERATOR] A bug has been fixed in gardener-controller-manager's Project controller that can lead to a continuous reconciliation of Project resources if they are stuck in Terminating state. (gardener/gardener#3371, @rfranzke)
  • [OPERATOR] An issue causing a NetworkPolicy to do not allow egress from prometheus Pod to alertmanager and vpa-exporter Pods is now fixed. (gardener/gardener#3370, @ialidzhikov)
  • [OPERATOR] An issue causing gardenlet to do not properly compute the .status.clusterIdentity field is now fixed. (gardener/gardener#3366, @ialidzhikov)
  • [OPERATOR] A bug that was renewing the bootstrap token secret on each reconciliation has been fixed. (gardener/gardener#3323, @vpnachev)
  • [OPERATOR] An issue has been fixed which did not enable VPA for the aggregate Prometheus Pod in new seed clusters. (gardener/gardener#3312, @timuthy)
  • [OPERATOR] By default, gardener-apiserver now invokes in-tree admission plugins before invoking the webhook plugins. (gardener/gardener#3298, @timebertt)
  • [OPERATOR] An issue has been fixed that prevented the execution of the Kube-API-Server's configured preStop hooks for >=1.19.x clusters. (gardener/gardener#3295, @timuthy)
  • [OPERATOR] Gardener health checks now take the effective Shoot specification into consideration if .spec.maintenance.confineSpecRollout is used. Earlier, EveryNodeReady or ControlPlaneHealthy conditions reported an invalid state if the specification was changed but not yet effective due to a rollout during shoot maintenance (confineSpecRollout: true). (gardener/gardener#3286, @timuthy)
  • [OPERATOR] KonnectivityTunnel's stability is improved and now handles kube-apiserver autoscaling. It properly sets --server-count of konnectivity-server on such event. (gardener/gardener#3267, @mvladev)
  • [DEVELOPER] The Seed and Shoot logging stack deletion is separated in two functions to avoid accidental deletion of cluster scoped resources. (gardener/gardener#3436, @vlvasilev)
  • [DEPENDENCY] A bug in the extension library that was preventing the deletion of TF secret and configmaps with empty state is now fixed. (gardener/gardener#3423, @vpnachev)
  • [DEPENDENCY] Go dependency kubernetes-sigs/controller-runtime was updated to v0.7.1. (gardener/gardener#3408, @timuthy)
  • [DEPENDENCY] Ensure a stable order of self-registered webhooks in extensions to avoid unnecessary rollouts of control plane components. (gardener/gardener#3320, @timebertt)

📖 Documentation

  • [USER] API reference documentation for kubernetes types now points to version v1.19. (gardener/gardener#3303, @mvladev)
  • [OPERATOR] Enhance documentation for Gardenlet's /healthz endpoint. (gardener/gardener#3359, @danielfoehrKn)
  • [OPERATOR] Gardener's scheduler documentation has been enhanced. It concisely explains the algorithm used to determine seed candidates. (gardener/gardener#3316, @timuthy)
  • [OPERATOR] AWS-specific annotations set on the istio-ingressgateway Service are now deprecated and are going to be removed in the next release. Please use the Seed's spec.settings.loadBalancerServices.annotations field to set or overwrite those annotations. For shoot.gardener.cloud/use-as-seed annotated Shoot clusters, see this PR. (gardener/gardener#3185, @mvladev)

🏃 Others

  • [USER] Component and Container fields are added in the logging dashboards for more flexible log queries. (gardener/gardener#3456, @Kristian-ZH)
  • [USER] Webhooks acting on configmaps in the kube-system namespace and webhooks with a TimeoutSeconds>15 for problematic resources are now also blocking Maintenance and Hibernation operations. Please consult this doc for more details. (gardener/gardener#3413, @timebertt)
  • [OPERATOR] An issue causing github.com/gardener/gardener/pkg/utils/imagevector.FindImages to not give a higher score on exact matched targetVersion or runtimeVersion is now fixed. (gardener/gardener#3568, @ialidzhikov)
  • [OPERATOR] An issue causing Shoots to be marked as Failed (and no longer retried) on transient not found error is now fixed. (gardener/gardener#3508, @ialidzhikov)
  • [OPERATOR] Keep VPA namespace env variable in sync in all cases. If they are out of sync between the admission controller and the updater, the updater might become inactive and stop actively scaling targets that have update mode Auto or Recreate. (gardener/gardener#3481, @amshuman-kr)
  • [OPERATOR] Gardener now considers the seed.spec.ingress.domain field when passing the value via gradener.seed.ingressDomain to ControllerRegistration charts. (gardener/gardener#3441, @timuthy)
  • [OPERATOR] Add CPU throttling to the "Kubernetes Pods" Grafana dashboard (gardener/gardener#3432, @istvanballok)
  • [OPERATOR] TestDefinitions have been added that contains disruptive tests (gardener/gardener#3411, @schrodit)
  • [OPERATOR] NumberOfBatchIDs for the fluent-bit-to-loki plugin is set to 5 numbers. (gardener/gardener#3402, @vlvasilev)
  • [OPERATOR] Enable fluent-bit privileged escalation for the integration test via "gardener.privileged" PodSecurityPolicy (gardener/gardener#3357, @vlvasilev)
  • [OPERATOR] The Loki initialDelaySeconds for the readinessProbe is reduces to 80 seconds. (gardener/gardener#3333, @vlvasilev)
  • [OPERATOR] The vpa-admission-controller and vpa-updater pods are now ensured with some minimal CPU and memory resources. (gardener/gardener#3330, @vpnachev)
  • [OPERATOR] Gardener will now check seed clusters for VPA functionality as a prerequisite. (gardener/gardener#3312, @timuthy)
  • [OPERATOR] Upgrade Prometheus to v2.23.0 (gardener/gardener#3297, @wyb1)
  • [OPERATOR] Change pod anti-affinity to preferredDuringSchedulingIgnoredDuringExecution for gardener-seed-admission-controller deployment in the garden namespaces of seed clusters. (gardener/gardener#3294, @hardikdr)
  • [OPERATOR] The pre-delivered cluster role gardener.cloud:admin now contains full access permissions for Events and ResourceQuotas. (gardener/gardener#3291, @timuthy)
  • [OPERATOR] Add panels to the Kubernetes API Server Details Dashboard for dropped requests. (gardener/gardener#3284, @wyb1)
  • [OPERATOR] Alerts are added for the custom metrics for fluent-bit GardenerLoki plugin (gardener/gardener#3283, @Kristian-ZH)
  • [OPERATOR] Required connections from Gardenlet to the Garden cluster has been reduced which will have positive effects on scalability and costs. (gardener/gardener#3277, @timuthy)
  • [DEVELOPER] The golang version is updated to 1.15.7. (gardener/gardener#3442, @ialidzhikov)
  • [DEVELOPER] Certain insensitive terms were removed from the source code and inline documentation to follow inclusive language best practices. (gardener/gardener#3368, @timuthy)
  • [DEVELOPER] Unit tests are now limited to a timeout of 2 minutes per test suite. (gardener/gardener#3363, @timebertt)
  • [DEPENDENCY] The extensions library is now registering webhooks for both seeds and shoots with a 10s timeout. (gardener/gardener#3440, @rfranzke)
  • [DEPENDENCY] Guestbook integration test dependencies are now fetched from bitnami repo instead of deprecated/shutdown helm repo. (gardener/gardener#3314, @dguendisch)
  • [DEPENDENCY] Making the implementation of the function GetMachineControllerManagerCloudCredentials in the WorkerDelegate optional. Alternatively, extensions can now use the field in the machine class spec.credentialsSecretRef so that all machine classes refer to the same secret from the Worker field spec.secretRef. See here for more details. (gardener/gardener#3308, @danielfoehrKn)

📰 Noteworthy

  • [USER] The kubectl get shoot table view was adapted and does no longer show the DOMAIN column. Instead, it shows the provider type as well as the last operation type. (gardener/gardener#3460, @rfranzke)
  • [USER] If the kube-apiserver deployment of a shoot cluster does not become ready then Gardener does now evaluate and return the most recent complete logs of the newest pod. It will be part of the .status.last{Operation,Errors}. As a consequence, start-up failures due to invalid feature gate or runtime config settings is being treated as configuration problem error. (gardener/gardener#3353, @rfranzke)
  • [USER] The version for the nginx-ingress addon for shoots has been updated to v0.41.2 ONLY for Kubernetes 1.20 shoot clusters. All shoot clusters with Kubernetes < 1.20 will remain with the current v0.22.0 version. Please be reminded that the nginx-ingress addon is not recommended for production scenarios and that you should deploy (+ customize) your own ingress controller instead. Please use it only for development/evaluation purposes. (gardener/gardener#3315, @rfranzke)
  • [OPERATOR] The ingress domain configuration for Seeds is now immutable. (gardener/gardener#3394, @BeckerMax)
  • [OPERATOR] The gardenlet does not fail the liveness probe when the seed resource configured in the Gardenlet's config has been deleted. (gardener/gardener#2925, @danielfoehrKn)
  • [DEPENDENCY] The terraformer library in extensions/pkg/terraformer does now adopt still running pods instead of deleting them and waiting for their deletion. This enables a faster and more reliable status/result propagation and prevents unhelpful error messages. Infrastructure extension developers do not need to wait for a clean Terraform environment in their Delete() function anymore as the library is now handling this case out-of-the-box. (gardener/gardener#3349, @rfranzke)
  • [DEPENDENCY] During the extension webhook registration, when a namespace is provided via the --webhook-config-namespace flag, the webhook config is enhanced with an owner reference pointing to the provided namespace. This will lead to auto-cleanup of the webhook config when the extension is uninstalled from a seed (earlier, the webhook config was orphaned even after uninstallation from a seed). (gardener/gardener#3341, @rfranzke)

[ingress-default-backend]

🏃 Others

[logging]

🏃 Others

Assets 3
Loading