Limited cluster infrastructure control to Gardener based Kyma clusters for end users [EPIC]

[varbanv]

Description

Provide a way for end users to make changes to the default shoot configuration for Gardener based Kyma clusters.
We will be constrained by Gardener capabilities but should be able to expose any configuration option that can be changed for a live cluster with minimal verification (mostly node type/size) by us.

Context

Problem

Currently, Kyma uses a predefined infrastructure configuration for Gardener based managed clusters and that cannot be changed or influenced by end users.
For example: customers cannot select or change the node types, or the composition or number of worker pools.
This means that if a customer needs nodes with more memory (for monolithic applications) or specialized nodes (for AI for example) they cannot use the managed Kyma offering.
Customers also are not able to do some basic network configuration like enabling NAT Gateways or use their own IPs.

Benefits

For customers:

• greater flexibility around infrastructure requirements
• ability to meet requirements in order to move workloads to Kyma
• reduce cost by reusing existing infrastructure (own IPs for example)

For us:

• increase adoption
• abstract and bundle infrastructure related requirements in one feature

Potential problems

• billing would become more complex if we don't introduce an as way for customers to track their costs
• we could have issues in HA setups if the infrastructure changes are not orchestrated

Tasks in Provisioner area:

• Provisioner update to add autoscaler configuration

Tasks in KEB area:

TBD

Tasks in BTP area:

TBD

---

⚠️ Funnel process - do not remove!

Name	Value
theme	Enterprise-grade
business value	4
technical value	8
user value	8
open-source value	8
effort
requested by

[pbochynski]

We need to clarify the scope. We want to expose more cluster configuration options, but not all of them can be effectively used by customers. Let's go case by case:

1. NAT Gateway. It is enabled by default in AWS and GCP. We can make it configurable for Azure (enable: true/false), but maybe it would be just better to turn it also on for Azure to have the same setup for all cloud providers.
2. Own IP. Configuring NAT Gateway with the customer's own IP is not possible without managing the Azure subscription as you need to provide IP address name and resource group (see documentation). I don't see any option to reuse IP defined outside of managed subscriptions. How would you like to enable it?
3. Node types. We already have the possibility to select node type. The list excludes more expensive types as it is not easy to convert CPU/memory consumption to usage metrics. Technically, we can change it even now.
4. Worker pools. This option we could expose.
5. CIDR (IP ranges). We can allow modifying default CIDR for clusters or for zones.

In the end, I see just worker pool configuration as an additional option. Other use cases are either not needed or not feasible and have to be addressed by Bring Your Own Cluster model

[ghost]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[ghost]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[koala7659]

Added task to configure following cluster parameters with Provisioner:

kubernetes:
    clusterAutoscaler:
      scaleDownDelayAfterAdd: 
      scaleDownUnneededTime:

https://github.tools.sap/kyma/backlog/issues/4253

[tobiscr]

@pbochynski / @varbanv : any conclusion what we should support on Infrastructure Manager side?

We have already a few things in our backlog:

• Multiple worker groups [KIM/feature] infrastructure-manager#46
• RBAC kubeconfigs for Clusters infrastructure-manager#117
• Customizable Auditlog configuration (internal issue: # 4881)

Anything else expected?

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[tobiscr]

High chance to be obsolete as we have overlapping with #18195

[varbanv]

Let's close this in favor of #18195