Multiple worker groups [KIM/feature]

[pbochynski]

Description
Enable possibility to create multiple worker groups with different machine types, volume types, node labels, annotations, taints.

See Gardener specs:

• Gardener model: https://github.com/gardener/gardener/blob/master/pkg/apis/extensions/v1alpha1/types_worker.go
• Gardener Shoot example: https://github.com/gardener/gardener/blob/master/example/90-shoot.yaml#L30-L152

Current example shoot from Provisioner:

 workers:
      - cri:
          name: containerd
        name: cpu-worker-0
        machine:
          type: m5.xlarge
          image:
            name: gardenlinux
            version: 1.2.3
          architecture: amd64
        maximum: 1
        minimum: 1
        maxSurge: 1
        maxUnavailable: 0
        volume:
          type: gp2
          size: 50Gi
        zones:
          - eu-central-1a
        systemComponents:
          allow: true
    workersSettings:
      sshAccess:
        enabled: true

Reasons
One size doesn't fit all. Many applications require specific nodes for particular services.

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[kyma-bot]

This issue or PR has been automatically closed due to the lack of activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle stale

If you think that I work incorrectly, kindly raise an issue with the problem.

/close

[kyma-bot]

@kyma-bot: Closing this issue.

In response to this:

This issue or PR has been automatically closed due to the lack of activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle stale

If you think that I work incorrectly, kindly raise an issue with the problem.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tobiscr]

@pbochynski : QQ - is this feature still relevant? If yes, I will start the alignment with KEB guys as it needs also their involvment.

[pbochynski]

The issue is part of bigger Epic: kyma-project/kyma#18195

[tobiscr]

We agreed with @varbanv and @PK85 to start with a minimal worker pool configuration, probably similar to the parameter we are currently already providing to Google.

[tobiscr]

JFYI:

It's important to set

    systemComponents:
          allow: true

to ensure the pool-nodes gets a label which indicates the related worker pool. This is important for later scheduling rules (via affinity configurations etc.)

---

It worked also without systemComponents, see #364

[tobiscr]

To cover billing requirements, we have to extend the contract of the RuntimeCR with 2 new fields: kyma-project/kyma-metrics-collector#89

[tobiscr]

Today we aligned the next steps for rolling out multiple worker pools, and it makes sense to distinguish in the RuntimeCR between the Kyma worker pool (primarily used by Kyma workloads) and customer worker pools.

It's meaningful to reflect this differentiation also in the RuntimeCR by using a dedicated field for the Kyma worker pool and another field (array) for the customer worker pools.

Proposal:

• workers field is used for describing the Kyma worker pool
• additionalWorkers (new field) is an array describing the worker pools created by the customer

[tobiscr]

Requires #396