Community modules in managed and non-managed environment

[pbochynski]

Description

Provide a way to install community modules using the Kyma Dashboard that is consistent with adding fully managed modules. UI should be simple and guide the user to use the best available option:

• add managed modules with Kyma CR modification (use Kyma Lifecycle Manager for installation and auto-update)
• install community modules directly in the cluster
• install regular (manageable) modules directly in the non-managed cluster

Other options should be also available, but in advanced mode (installing manageable module without KLM - opt-out from full support)

Reasons

Kyma comes with a set of Kubernetes building blocks (called modules). Modules are independent: can be added and removed in the runtime, and have an independent release cycle. Each module provides and a manager (Kubernetes operator) and a default configuration (custom resource). Some modules are available in the managed SAP Kyma Runtime with full support (SLA, automatic upgrades). Other modules (community modules) are offered only with community support (no SLA, manual upgrades, possible breaking changes). In the managed clusters the Kyma Lifecycle Manager (KLM) is responsible for installation and automatic upgrades of supported modules. KLM is running centrally in the Kyma Control Plane and is configured with Kyma custom resource in the namespace kyma-system located in the managed cluster. Users who want to use some managed modules have to add their names to the Kyma CR spec. The community modules cannot be managed by the KLM and should be installed with the desired version using kubectl with the provided Kubernetes manifests (manager deployment and default configuration).
The situation gets more complicated when users want to install the "manageable" modules in their own (not managed) cluster (like k3s) or if they want to opt-out from full management in the managed cluster (install module manager with kubectl without using KLM).

Having a list of available modules with clear indicators if the module is/can be fully managed or not, and a consistent convenient installation process is a must-have.

Acceptance Criteria

• Module catalogue improvements lifecycle-manager#1472
  • The current approach with ModuleTemplate CRD should be revisited, as a set of new requirements applies
  • This will provide a baseline for the implementation of the following ACs
• Delete modes for modules lifecycle-manager#1211
  • Blocking deletion mode supported by the KLM (deletion hangs until all the domain CRs are gone from the cluster)
  • Ignore deletion supported by the KLM (which translates to "unmanaging" modules in the context of this Epic)
• Separate business and module statuses lifecycle-manager#1471
  • Module state needs to be separated into two different entities
  • Manifest state will reflect the overall health of module deployment
  • Module state will represent whether the user-made configuration was applied correctly and the module is serving its purpose
• Busola's UI is adapted to the new API
  • Delete module with managed resources and operator resources busola#3345
  • Install community modules (not managed) busola#3350
  • It should be transparent in which state the modules are, this applies both for Manifest and Module states
• Stop the synchronisation of Module Templates to the SKR lifecycle-manager#1556 (after Busola is migrated)
• Module build pipeline is adapted to the new API
  • It should produce a list of modules that Busola can consume and display to the user
  • In case the decision is made to drop the ModuleTemplate CRD the adjustments to the output need to be made

Module scenarios

Installation

Manageable modules should be installed with KLM if Kyma CR is available (managed cluster). The direct installation (without KLM) should be possible for manageable modules but should be harder to find and guarded by warnings. It should not be possible to install the module directly if it is already a managed module.
In the open source scenario (not managed cluster) all modules should be installed directly with the provided Kubernetes manifests.

Note: direct installation and deletion are idempotent actions - and can be repeated several times without side effects. It should be possible to trigger installation or deletion several times (finish an interrupted/incomplete action by applying/deleting again, or upgrade the module manager by applying a new version)

Upgrade

For managed modules, it happens automatically. For not managed modules user has to apply the module manifest again. Default module configuration (default CR) should not be applied for updates. The module operator in the new version should handle previously created configurations.
For both cases (managed and manual) it is essential to check what is the current version of the module manager (operator) installed in the cluster. UI should compare the desired version with the version that is deployed in the cluster. If the version does not match it is the indicator that should be visible to the user. It means that for managed components KLM hasn't finished the upgrade yet, for not managed modules it should encourage the user to perform the upgrade (press the apply button).

Deletion

Module manager should not be undeployed before any managed resource (including module configuration) still exists. Otherwise, there is a good chance that the orphan resources will stay in the cluster forever. Therefore each module should declare the list of managed resources (group, version, kind) and the user should actively delete those resources before the module is finally deleted (manager undeployed). This step should be executed regardless if the module is managed by KLM or installed manually.

Warning: when you delete the managed module configuration before the module is removed from Kyma CR spec, the KLM can try to recreate it and the module installation starts again. In that case, module configuration should be removed by KLM or the customResourcePolicy should be changed to Ignore (from the default CreateAndDelete)

flowchart TD
    N([Start: Choose Module to Delete]) --> MO{"Do managed 
    resources exist?"}
    MO -->|No| O{"Is the Module 
    Managed?"}
    MO -->|Yes| DMO{Continue?}
    DMO -->|Yes| DM[Delete managed resources]
    DMO -->|No| E
    DM --> MO
    O -->|Yes| P["Delete via KLM
                  Remove module 
                  from Kyma CR"]
    O -->|No| T["Direct Deletion 
    of module manager 
    Kubernetes resources"]
    P --> E([End])
    T --> E

Loading

Managed modules should be deleted by KLM and the deletion process should be triggered by removing module entry from Kyma CR spec.
Not-managed modules should be removed directly by deleting Kubernetes resources included in the module manager manifest.

Status handling

With automatic and manual lifecycle management it is not so easy to determine if the module is installed and what is the status. Here are some rules that can be applied to determine module status and possible actions:

• module is managed and enabled if it exists in spec.modules in default Kyma CR in kyma-system namespace
• module is enabled if default CR exists and does not have deletion timestamp
• module is in the deleting state when default CR has a deletion timestamp
• module is removed if there is no entry in Kyma CR and default CR does not exist
• module is Ready if default CR is applied and has proper status (usually status.state=Ready, but can be different) and the manager is deployed in the version defined in the desired release channel

Diagram

flowchart TB
    A[Start: Choose Module to Install] --> B{Is the Cluster Managed?}
    B -->|Yes| C[Use Kyma Lifecycle Manager]
    B -->|No| D{Is the Module Manageable?}
    C --> E[Modify Kyma CR for Installation]
    E --> F[Automatic Updates via KLM]
    D -->|Yes| G[Install Manageable Module Directly]
    D -->|No| H[Install Community Module with kubectl]
    G --> I[Option: With KLM for Support]
    G --> J[Option: Without KLM - Opt-out from Full Support]
    H --> K[Manual Upgrades, Possible Breaking Changes]
    I --> L[Managed with SLA, Automatic Upgrades]
    J --> M[Warning: Limited Support]

Loading

flowchart TB
    U[Start: Check Module Status] --> V{Is Module in Kyma CR?}
    V -->|Yes| W[Managed and Enabled]
    V -->|No| X{Does Default CR Exist?}
    X -->|Yes| Y[Check if CR has Deletion Timestamp]
    Y -->|No Timestamp| Z[Module is Enabled]
    Y -->|Has Timestamp| AA[Module is Deleting]
    X -->|No| AB[Module is Removed]
    W --> AC[Ready if CR Applied and Manager at Desired Version]
    Z --> AC
    AA --> AC

Loading

Attachments

• PoC with manual installation of community modules

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[valentinvieriu]

From UI Perspective - we decided to refactor our UI to match the PRoposed Designs using the Existing implementation with KYMA CR (kyma-project/busola#2624 )

Once the new modules catalog will be ready, we will switch to the new API. We are waiting for the feedback when the new JSON based API will be ready and we can switch from reading Module Templates to display Module Metadata.

[tobiscr]

@tehret77 - JFYI, this feature will make the management of Kyma community modules more convenient in the future.

[varbanv]

Due date shifted to reflect additional work that is needed for UI/CLI, etc.

[janmedrek]

The proposal for all the module catalogue changes is almost finalized here: kyma-project/lifecycle-manager#1681

Right now, we are waiting for input from the Busola and CLI teams to shape the decision and proceed with the implementation tweaks.

After that, it is all about adapting the UI and pipelines to match the picture and we are good-to-go with releasing the Community Modules as a whole.

[janmedrek]

We would also need some documentation for the community-contributed modules, such as what metadata to provide and where it should be placed.

Also, a modulectl command for creating a module template would be welcome.

[janmedrek]

Extended with the documentation issue: kyma-project/lifecycle-manager#1812

[janmedrek]

Update:

• Deletion Modes and status separation is finished and supported by Busola
• We are working on changes to the API to support multiple versions and some additional info fields for Busola to display
• As soon as we have working components we will present a playground to the Hasselhoff team
• After the contract is finalized we can start working on the build pipelines (and most likely migrate them to the modulectl)