From af8da0bf7daf73d0d9890ffab3ee3914e57677ad Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Thu, 13 Jun 2024 17:11:48 +0200 Subject: [PATCH 01/11] Add SDD for Commodore Compile Pipeline --- .../SDDs/pages/0032-compile-pipeline.adoc | 153 ++++++++++++++++++ docs/modules/SDDs/pages/index.adoc | 1 + 2 files changed, 154 insertions(+) create mode 100644 docs/modules/SDDs/pages/0032-compile-pipeline.adoc diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc new file mode 100644 index 0000000..7885ae4 --- /dev/null +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -0,0 +1,153 @@ += SDD 0031 - Central Component Version tracking + +:sdd_author: Aline Abler +:sdd_owner: Project Syn IG +:sdd_date: 2024-06-13 +:sdd_status: draft + +include::partial$meta-info-table.adoc[] + +[NOTE] +.Summary +==== +This describes how we want to implement CI/CD for Project Syn using a Commodore compile pipeline, which enables automatic compilation of cluster catalogs whenever the corresponding tenant repository is modified. + +Furthermore, it explains how want to extend the functionality of Lieutenant to enable it to automatically configure the compile pipeline on tenant repositories. +==== + +== Motivation + +Having a continuous integration solution unlocks a number of benefits: +It solves the problem of configuration drift, where changes to the tenant repository might not be reflected in every cluster catalog because not all of them have since been compiled, and it lessens the burden on catalog maintainers who otherwise would need to locally compile each cluster individually. + +It is already fairly straightforward to manually set up basic auto-compilation for individual tenant repositories without special support from Project Syn itself. +At VSHN, such a solution has been in use for several years. + +However, certain features (such as automatic configuration of the compile pipeline) are hard to implement in a standalone fashion. +As such features are now desired, it makes sense to fully integrate the compile pipeline into Project Syn. + +By making Project Syn "CI-aware", we can implement more seamless management of compile pipeline configuration on the tenant repositories, including automated setup and automated token rotation. +This will go hand-in-hand with the existing repository management features in Lieutenant. + +Currently, we're looking at a solution that is specific to GitLab CI/CD, and to tenant and catalog repositories that are stored on GitLab. +Other CI solutions and other repository hosts might be supported in the future. + +=== Goals + +* Provide pipeline definitions for GitLab CI/CD to automatically compile and push cluster catalogs on a tenant repository. +* Enable Lieutenant to autonomously manage the configuration required to set up the compile pipeline for a cluster catalog. + +=== Non-Goals + +* Support for CI solutions other than GitLab CI/CD. + +== Design Proposal + +=== Gitlab CI/CD Pipeline for Commodore + +The pipeline definitions for the Commodore compile pipeline are managed in a dedicated GitLab repository (the pipeline repository). +Tenant repositories can then include these pipeline definitions in their `gitlab-ci.yml`. + +Execution of the pipeline is controlled via GitLab CI/CD Variables, which can be set on the tenant repository. +The variables control which catalog is compiled, and are used to provide access to the catalog repository via GitLab Project Access Token, as well as to the Lieutenant API via Commodore API token. + +In particular, the pipeline accepts the following CI/CD variables: + +* `ACCESS_TOKEN_CLUSTERNAME`, where `CLUSTERNAME` is the name of a specific cluster, with `-` replaced by `_`. This contains a GitLab Project Access Token, which must have read-write access the corresponding cluster's catalog repository. +* `COMMODORE_API_URL`. This contains the URL at which the Lieutenant API can be accessed. +* `COMMODORE_API_TOKEN`. This contains an access token for the Lieutenant API. +* `CLUSTERS`. This contains a space-separated list of cluster IDs which should be compiled and pushed automatically. + +=== CRD + +We add a new field to the `GitRepoTemplate` (and, by extension, the `GitRepo`) CRD, under the `.spec` key, called `compilePipeline`. + +The `compilePipeline` field contains configuration pertaining to the automatic setup of the compile pipeline on the tenant repository. +It is optional. +Absence of the field disables automatic setup and management of the compile pipeline. + +The `compilePipeline` field contains a dict with the following fields: + +* `gitlabCatalogProjectAccessSecretRef`: (for cluster catalog repos) Reference to a secret containing the GitLab Project Access Token that can be used to access this repository. (Managed by Lieutenant) +* `pipelineFiles`: Dictionary containing file paths as keys, and file contents as values. + These files will be committed to the tenant repository when the compile pipeline is configured, if they do not exist already. + +We add a new field to the `Cluster` CRD, under the `.status` key, called `compileMeta`. + + +[source,yaml] +---- +apiVersion: syn.tools/v1alpha1 +kind: Cluster +metadata: + name: my-cluster +spec: + gitRepoTemplate: + compilePipeline: + gitlabCatalogProjectAccessSecretRef: c-my-cluster-project-access + pipelineFiles: + .gitlab-ci.yml: | + include: + - project: syn/commodore-compile-pipeline + ref: master + file: /.gitlab/commodore-common.yml + +---- + +=== Operator + +The Lieutenant Operator will be extended to automatically manage the compile pipeline for repositories where this is enabled (by way of configuring the `compilePipeline` field). + +Since the compile pipeline has to interact with both the tenant repo as well as the cluster catalog repo, it must be enabled on both corresponding `gitRepo` definitions for the configuration to be functioning. +This way, it is possible to enable auto-compilation for some, but not all clusters on a tenant. +And furthermore, it is possible to easily disable auto-compilation for a whole tenant without having to touch each cluster's `gitRepo` definition. + +When `compilePipeline` is configured on a cluster repository (catalog repository), Lieutenant does the following: + +* Create a new `Project Access Token` on the catalog repository via the GitLab API, using the cluster repository's configured apiSecretRef. +* Store the new access token in a new secret. +* Store a reference to the new secret in `catalogProjectAccessSecretRef`. +* Add a new CI/CD variable to the corresponding tenant repository via the GitLab API, using the tenant repository's configured apiSecretRef. The value is named `ACCESS_TOKEN_CLUSTERNAME` (where `CLUSTERNAME` is the cluster ID with `-` replaced by `_`) and contains the GitLab project access token as its value. +* Update the CI/CD variable `CLUSTERS` on the corresponding tenant repository to ensure it includes the cluster's ID. + +When `compilePipeline` is configured on a *managed* tenant repository, Lieutenant does the following: + +* Render the `pipelineFiles` configured on the tenant repository, and commit them. +* Add a new CI/CD variable `COMMODORE_API_URL` to the tenant repository, containing the API URL for Lieutenant. +* Add a new CI/CD variable `COMMODORE_API_TOKEN` to the tenant repository, containing the API token for Lieutenant. + +When the `compilePipeline` field is removed from a repository, all the corresponding resources should be cleaned up. + +Unmanaged repositories cannot have automatically configured compile pipelines. + +==== Rotation of Project Access Tokens + +GitLab's project access tokens have a limited lifespan. +Lieutenant should automatically rotate them as required. + +To this end, a scheduled task should run in the operator that checks the token's expiration date via the GitLab API, and rotates any that are close to expiring. + +To rotate a token: + +* Use the GitLab API to rotate the existing token. +* Update the corresponding catalog project access secret. +* Update the corresponding CI/CD variable (`ACCESS_TOKEN_CLUSTERNAME`) in the tenant repository. + +=== Implementation Details/Notes/Constraints + +Existing compile pipeline configuration:: +If a setup already includes a bunch of tenant repositories with manually configured CI/CD, some care has to be taken to ensure the new implementation can "adopt" this configuration. ++ +In particular, these repositories would already have a working `.gitlab-ci.yml` that probably can be left as-is. ++ +Any existing manually created Project Access Tokens will be superseded by new auto-generated ones. +This will lead to a bunch of now-unused tokens needing to be cleaned up. + +External Catalog Repositories:: +There may be cases where the catalog repositories are not hosted on the same repository host as the tenant repository, in which case API access for the purpose of creating Project Access Tokens is unavailable. +The Commodore Compile Pipeline can still be used against such catalog repositories by specifying an SSH key to access them. ++ +This can still be configured manually, and the automated configuration would not interfere. + +== References + diff --git a/docs/modules/SDDs/pages/index.adoc b/docs/modules/SDDs/pages/index.adoc index 97f3dd0..19f3dfa 100644 --- a/docs/modules/SDDs/pages/index.adoc +++ b/docs/modules/SDDs/pages/index.adoc @@ -26,3 +26,4 @@ The all are using the xref:sdd-template.adoc[SDD Template]. * xref:0028-reusable-config-packages.adoc[0028 - Reusable Commodore Component Configuration Packages] * xref:0030-argocd-multitenancy.adoc[0030 - Project Syn ArgoCD Multi-Tenant Support] * xref:0031-component-version-tracking.adoc[0031 - Central Component Version tracking] +* xref:0032-compile-pipeline.adoc[0032 - Commodore Compile Pipeline] From 736e4d42956e4584ab1bb0bdc2ee59cb9bde0592 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 14 Jun 2024 14:55:55 +0200 Subject: [PATCH 02/11] Apply suggestions from code review Co-authored-by: Simon Gerber --- docs/modules/SDDs/pages/0032-compile-pipeline.adoc | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc index 7885ae4..189c74b 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -2,6 +2,7 @@ :sdd_author: Aline Abler :sdd_owner: Project Syn IG +:sdd_reviewers: Simon Gerber :sdd_date: 2024-06-13 :sdd_status: draft @@ -45,8 +46,10 @@ Other CI solutions and other repository hosts might be supported in the future. === Gitlab CI/CD Pipeline for Commodore -The pipeline definitions for the Commodore compile pipeline are managed in a dedicated GitLab repository (the pipeline repository). -Tenant repositories can then include these pipeline definitions in their `gitlab-ci.yml`. +The pipeline definitions for the Commodore compile pipeline are managed in a dedicated Git repository (the pipeline repository). +This repository can be hosted anywhere. +Tenant repositories can then include these pipeline definitions in their `gitlab-ci.yml` as a "remote" include. +See the https://docs.gitlab.com/ee/ci/yaml/includes.html#include-a-single-configuration-file[GitLab CI/CD documentation] for details. Execution of the pipeline is controlled via GitLab CI/CD Variables, which can be set on the tenant repository. The variables control which catalog is compiled, and are used to provide access to the catalog repository via GitLab Project Access Token, as well as to the Lieutenant API via Commodore API token. @@ -72,7 +75,6 @@ The `compilePipeline` field contains a dict with the following fields: * `pipelineFiles`: Dictionary containing file paths as keys, and file contents as values. These files will be committed to the tenant repository when the compile pipeline is configured, if they do not exist already. -We add a new field to the `Cluster` CRD, under the `.status` key, called `compileMeta`. [source,yaml] @@ -98,7 +100,7 @@ spec: The Lieutenant Operator will be extended to automatically manage the compile pipeline for repositories where this is enabled (by way of configuring the `compilePipeline` field). -Since the compile pipeline has to interact with both the tenant repo as well as the cluster catalog repo, it must be enabled on both corresponding `gitRepo` definitions for the configuration to be functioning. +Since the compile pipeline has to interact with both the tenant repository as well as the cluster catalog repositories, it must be enabled on both corresponding `gitRepo` definitions for the configuration to be functional. This way, it is possible to enable auto-compilation for some, but not all clusters on a tenant. And furthermore, it is possible to easily disable auto-compilation for a whole tenant without having to touch each cluster's `gitRepo` definition. @@ -114,7 +116,7 @@ When `compilePipeline` is configured on a *managed* tenant repository, Lieutenan * Render the `pipelineFiles` configured on the tenant repository, and commit them. * Add a new CI/CD variable `COMMODORE_API_URL` to the tenant repository, containing the API URL for Lieutenant. -* Add a new CI/CD variable `COMMODORE_API_TOKEN` to the tenant repository, containing the API token for Lieutenant. +* Add a new CI/CD variable `COMMODORE_API_TOKEN` to the tenant repository, containing the tenant's API token for Lieutenant. When the `compilePipeline` field is removed from a repository, all the corresponding resources should be cleaned up. From d337890a1a6a070e05ff3f06640f7614b3f1af1f Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 14 Jun 2024 16:03:45 +0200 Subject: [PATCH 03/11] Rework design based on provided suggestions --- .../SDDs/pages/0032-compile-pipeline.adoc | 150 +++++++++++------- 1 file changed, 93 insertions(+), 57 deletions(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc index 189c74b..5abe6e8 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -30,9 +30,6 @@ As such features are now desired, it makes sense to fully integrate the compile By making Project Syn "CI-aware", we can implement more seamless management of compile pipeline configuration on the tenant repositories, including automated setup and automated token rotation. This will go hand-in-hand with the existing repository management features in Lieutenant. -Currently, we're looking at a solution that is specific to GitLab CI/CD, and to tenant and catalog repositories that are stored on GitLab. -Other CI solutions and other repository hosts might be supported in the future. - === Goals * Provide pipeline definitions for GitLab CI/CD to automatically compile and push cluster catalogs on a tenant repository. @@ -44,26 +41,65 @@ Other CI solutions and other repository hosts might be supported in the future. == Design Proposal -=== Gitlab CI/CD Pipeline for Commodore +=== Requirements for Pipeline Configuration -The pipeline definitions for the Commodore compile pipeline are managed in a dedicated Git repository (the pipeline repository). -This repository can be hosted anywhere. -Tenant repositories can then include these pipeline definitions in their `gitlab-ci.yml` as a "remote" include. -See the https://docs.gitlab.com/ee/ci/yaml/includes.html#include-a-single-configuration-file[GitLab CI/CD documentation] for details. +Lieutenant imposes certain assumptions on the configuration of the pipeline: +Namely, the pipeline has to be set up on the tenant repository by way of adding (arbitrary) files to the repository, +and it is configured through setting CI/CD variables on the repository. -Execution of the pipeline is controlled via GitLab CI/CD Variables, which can be set on the tenant repository. -The variables control which catalog is compiled, and are used to provide access to the catalog repository via GitLab Project Access Token, as well as to the Lieutenant API via Commodore API token. +In particular, Lieutenant configures the following CI/CD variables: -In particular, the pipeline accepts the following CI/CD variables: - -* `ACCESS_TOKEN_CLUSTERNAME`, where `CLUSTERNAME` is the name of a specific cluster, with `-` replaced by `_`. This contains a GitLab Project Access Token, which must have read-write access the corresponding cluster's catalog repository. +* `ACCESS_TOKEN_CLUSTERNAME`, where `CLUSTERNAME` is the name of a specific cluster, with `-` replaced by `_`. + This contains a GitLab Project Access Token, which must have read-write access the corresponding cluster's catalog repository. * `COMMODORE_API_URL`. This contains the URL at which the Lieutenant API can be accessed. * `COMMODORE_API_TOKEN`. This contains an access token for the Lieutenant API. * `CLUSTERS`. This contains a space-separated list of cluster IDs which should be compiled and pushed automatically. -=== CRD +=== GitRepo CRD + +We add two new fields to the `GitRepoTemplate` (and, by extension, the `GitRepo`) CRD, under the `.spec` key, called `accessTokenSecretRef` and `ciVariables`. -We add a new field to the `GitRepoTemplate` (and, by extension, the `GitRepo`) CRD, under the `.spec` key, called `compilePipeline`. +The `accessTokenSecretRef` field contains a reference to a secret. +If it is set, the Lieutenant operator will store an access token into this secret, which can be used to access the git repository. +In the case of GitLab, this would be a Project Access Token with read-write access to the repository. + +The `ciVariables` field contains a dictionary describing variable names and corresponding values. +These variables are added to the git repository as CI/CD variables. + +[source,yaml] +---- +apiVersion: syn.tools/v1alpha1 +kind: GitRepo +metadata: + name: my-repo +spec: + accessTokenSecretRef: my-repo-access-token + ciVariables: + COMMODORE_API_URL: ... + COMMODORE_API_TOKEN: ... +---- + +=== Cluster CRD + +We add a new field to the `Cluster` CRD, under the `.spec` key, called `enableCompilePipeline`. + +The field contains a boolean flag, which controls whether the compile pipeline should be enabled or disabled for this cluster. + +It is optional; not specifying it is equivalent to setting it to `false`. + +[source,yaml] +---- +apiVersion: syn.tools/v1alpha1 +kind: Cluster +metadata: + name: c-my-cluster +spec: + enableCompilePipeline: true +---- + +=== Tenant CRD + +We add a new field to the `Tenant` CRD, under the `.spec` key, called `compilePipeline`. The `compilePipeline` field contains configuration pertaining to the automatic setup of the compile pipeline on the tenant repository. It is optional. @@ -71,72 +107,70 @@ Absence of the field disables automatic setup and management of the compile pipe The `compilePipeline` field contains a dict with the following fields: -* `gitlabCatalogProjectAccessSecretRef`: (for cluster catalog repos) Reference to a secret containing the GitLab Project Access Token that can be used to access this repository. (Managed by Lieutenant) +* `clusters`: List of cluster IDs of clusters for which the compile pipeline should be executed. + This field is managed by the operator. * `pipelineFiles`: Dictionary containing file paths as keys, and file contents as values. - These files will be committed to the tenant repository when the compile pipeline is configured, if they do not exist already. - - + These files will be added to the tenant's `gitRepoTemplate.templateFiles` by the Lieutenant operator. + This field is optional; if absent, no new template files are added to the `gitRepoTemplate`. [source,yaml] ---- apiVersion: syn.tools/v1alpha1 -kind: Cluster +kind: Tenant metadata: - name: my-cluster + name: t-my-tenant spec: - gitRepoTemplate: - compilePipeline: - gitlabCatalogProjectAccessSecretRef: c-my-cluster-project-access - pipelineFiles: - .gitlab-ci.yml: | - include: - - project: syn/commodore-compile-pipeline - ref: master - file: /.gitlab/commodore-common.yml + compilePipeline: + clusters: + - c-my-cluster + pipelineFiles: + .gitlab-ci.yml: | + include: + - project: syn/commodore-compile-pipeline + ref: master + file: /.gitlab/commodore-common.yml ---- === Operator -The Lieutenant Operator will be extended to automatically manage the compile pipeline for repositories where this is enabled (by way of configuring the `compilePipeline` field). +The Lieutenant Operator will be extended to automatically manage the compile pipeline for repositories where this is enabled (by way of configuring the `compilePipeline` field on the tenant and the `enableCompilePipeline` field on the cluster). -Since the compile pipeline has to interact with both the tenant repository as well as the cluster catalog repositories, it must be enabled on both corresponding `gitRepo` definitions for the configuration to be functional. +Since the compile pipeline has to interact with both the tenant repository as well as the cluster catalog repositories, it must be enabled on both corresponding resources for the configuration to be functional. This way, it is possible to enable auto-compilation for some, but not all clusters on a tenant. -And furthermore, it is possible to easily disable auto-compilation for a whole tenant without having to touch each cluster's `gitRepo` definition. - -When `compilePipeline` is configured on a cluster repository (catalog repository), Lieutenant does the following: - -* Create a new `Project Access Token` on the catalog repository via the GitLab API, using the cluster repository's configured apiSecretRef. -* Store the new access token in a new secret. -* Store a reference to the new secret in `catalogProjectAccessSecretRef`. -* Add a new CI/CD variable to the corresponding tenant repository via the GitLab API, using the tenant repository's configured apiSecretRef. The value is named `ACCESS_TOKEN_CLUSTERNAME` (where `CLUSTERNAME` is the cluster ID with `-` replaced by `_`) and contains the GitLab project access token as its value. -* Update the CI/CD variable `CLUSTERS` on the corresponding tenant repository to ensure it includes the cluster's ID. - -When `compilePipeline` is configured on a *managed* tenant repository, Lieutenant does the following: -* Render the `pipelineFiles` configured on the tenant repository, and commit them. -* Add a new CI/CD variable `COMMODORE_API_URL` to the tenant repository, containing the API URL for Lieutenant. -* Add a new CI/CD variable `COMMODORE_API_TOKEN` to the tenant repository, containing the tenant's API token for Lieutenant. +The operator will reconcile *GitRepos* as follows: -When the `compilePipeline` field is removed from a repository, all the corresponding resources should be cleaned up. +* When `spec.accessTokenSecretRef` is set, the operator generates an access token for the corresponding repository (via the repository host's API, using the API secret in `.spec.apiSecretRef`), and writes this token into a secret with the given name. + In the case of GitLab, this is a Project Access Token. + The operator also runs a scheduled job which refreshes these tokens when they are close to expiring, or when they no longer exist on the repository host. +* The content of `.spec.ciVariables` is written to the repository's configuration on the git host. + In the case of GitLab, it is written as CI/CD variables. -Unmanaged repositories cannot have automatically configured compile pipelines. +NOTE: If the GitRepo is of type `unmanaged`, none of these steps will be executed. -==== Rotation of Project Access Tokens +The operator will reconcile *Clusters* as follows: -GitLab's project access tokens have a limited lifespan. -Lieutenant should automatically rotate them as required. +* When `.spec.enableCompilePipeline` is set to `true`, the tenant's `spec.compilePipeline.clusters` is updated to contain the cluster ID. +* Similarly, when the field is set to `false` or missing, the tenant's `spec.compilePipeline.clusters` is updated to not contain the cluster ID. -To this end, a scheduled task should run in the operator that checks the token's expiration date via the GitLab API, and rotates any that are close to expiring. +The operator will reconcile *Tenants* as follows: -To rotate a token: - -* Use the GitLab API to rotate the existing token. -* Update the corresponding catalog project access secret. -* Update the corresponding CI/CD variable (`ACCESS_TOKEN_CLUSTERNAME`) in the tenant repository. +* When `.spec.compilePipeline` exists and isn't empty, the following entries are added to the tenant repository GitRepo's `.spec.ciVariables`: +** `COMMODORE_API_URL`, containing the URL at which the Lieutenant API can be accessed. +** `COMMODORE_API_TOKEN`, containing the tenant's access token for the Lieutenant API. +** `CLUSTERS`, containing a space-separated list of cluster IDs taken directly from `.spec.compilePipeline.clusters`. +* For each entry in `.spec.compilePipeline.pipelineFiles` whose value isn't `{remove}`, a new corresponding entry is added to the tenant's `.spec.gitRepoTemplate.templateFiles`. +* For each entry in `.spec.compilePipeline.pipelineFiles` whose value is `{remove}`, the corresponding entry is removed from the tenant's `.spec.gitRepoTemplate.templateFiles`. +* For each entry in `.spec.compilePipeline.clusters`, another entry is added to the tenant repository GitRepo's `spec.ciVariabes`. + The key is `ACCESS_TOKEN_CLUSTERNAME`, where `CLUSTERNAME` is the ID of a specific cluster, with `-` replaced by `_`. + The value is the access token to access that cluster's catalog repository, taken from the secret specified in the catalog GitRepo configuration under `.spec.accessTokenSecretRef`. === Implementation Details/Notes/Constraints +Currently, we're looking at a solution that is specific to GitLab CI/CD, and to tenant and catalog repositories that are stored on GitLab. +Other CI solutions and other repository hosts might be supported in the future. + Existing compile pipeline configuration:: If a setup already includes a bunch of tenant repositories with manually configured CI/CD, some care has to be taken to ensure the new implementation can "adopt" this configuration. + @@ -153,3 +187,5 @@ This can still be configured manually, and the automated configuration would not == References +* https://docs.gitlab.com/ee/ci/variables/[GitLab CI Variables] +* https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html[GitLab Project Access Tokens] From fe9f02f40e218307aa2fee887a48cc2eba105db3 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Mon, 17 Jun 2024 11:07:00 +0200 Subject: [PATCH 04/11] Apply suggestions from code review Co-authored-by: Simon Gerber --- .../SDDs/pages/0032-compile-pipeline.adoc | 50 ++++++++++++------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc index 5abe6e8..28cb81f 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -44,8 +44,7 @@ This will go hand-in-hand with the existing repository management features in Li === Requirements for Pipeline Configuration Lieutenant imposes certain assumptions on the configuration of the pipeline: -Namely, the pipeline has to be set up on the tenant repository by way of adding (arbitrary) files to the repository, -and it is configured through setting CI/CD variables on the repository. +Namely, the pipeline has to be set up on the tenant repository by way of adding (arbitrary) files to the repository, and it is configured through setting CI/CD variables on the repository. In particular, Lieutenant configures the following CI/CD variables: @@ -57,10 +56,10 @@ In particular, Lieutenant configures the following CI/CD variables: === GitRepo CRD -We add two new fields to the `GitRepoTemplate` (and, by extension, the `GitRepo`) CRD, under the `.spec` key, called `accessTokenSecretRef` and `ciVariables`. +We add two new fields to the `GitRepoTemplate` (and, by extension, the `GitRepo`) CRD, under the `.spec` key, called `accessTokenSecretName` and `ciVariables`. -The `accessTokenSecretRef` field contains a reference to a secret. -If it is set, the Lieutenant operator will store an access token into this secret, which can be used to access the git repository. +The `accessTokenSecretName` field contains a reference to a secret. +If it is set, the Lieutenant operator will store an access token into this secret, which can be used to access the Git repository. In the case of GitLab, this would be a Project Access Token with read-write access to the repository. The `ciVariables` field contains a dictionary describing variable names and corresponding values. @@ -73,10 +72,15 @@ kind: GitRepo metadata: name: my-repo spec: - accessTokenSecretRef: my-repo-access-token + accessTokenSecretName: my-repo-access-token ciVariables: - COMMODORE_API_URL: ... - COMMODORE_API_TOKEN: ... + - name: COMMODORE_API_URL + value: ... + - name: COMMODORE_API_TOKEN + valueFrom: + secretKeyRef: + name: api-token-secret + key: token ---- === Cluster CRD @@ -109,9 +113,6 @@ The `compilePipeline` field contains a dict with the following fields: * `clusters`: List of cluster IDs of clusters for which the compile pipeline should be executed. This field is managed by the operator. -* `pipelineFiles`: Dictionary containing file paths as keys, and file contents as values. - These files will be added to the tenant's `gitRepoTemplate.templateFiles` by the Lieutenant operator. - This field is optional; if absent, no new template files are added to the `gitRepoTemplate`. [source,yaml] ---- @@ -123,7 +124,23 @@ spec: compilePipeline: clusters: - c-my-cluster - pipelineFiles: +---- + +=== In-Repo CI/CD pipeline configuration + +Configuring the CI pipeline usually happens through files committed to the corresponding repository. +For a Lieutenant-managed pipeline configuration, these files should be managed by Lieutenant. +To achieve this, we can leverage the existing mechanism to commit template files to git repositories: + +[source,yaml] +---- +apiVersion: syn.tools/v1alpha1 +kind: Tenant +metadata: + name: t-my-tenant +spec: + gitRepoTemplate: + templateFiles: .gitlab-ci.yml: | include: - project: syn/commodore-compile-pipeline @@ -132,6 +149,7 @@ spec: ---- + === Operator The Lieutenant Operator will be extended to automatically manage the compile pipeline for repositories where this is enabled (by way of configuring the `compilePipeline` field on the tenant and the `enableCompilePipeline` field on the cluster). @@ -141,7 +159,7 @@ This way, it is possible to enable auto-compilation for some, but not all cluste The operator will reconcile *GitRepos* as follows: -* When `spec.accessTokenSecretRef` is set, the operator generates an access token for the corresponding repository (via the repository host's API, using the API secret in `.spec.apiSecretRef`), and writes this token into a secret with the given name. +* When `spec.accessTokenSecretName` is set, the operator generates an access token for the corresponding repository (via the repository host's API, using the API secret in `.spec.apiSecretRef`), and writes this token into a secret with the given name. In the case of GitLab, this is a Project Access Token. The operator also runs a scheduled job which refreshes these tokens when they are close to expiring, or when they no longer exist on the repository host. * The content of `.spec.ciVariables` is written to the repository's configuration on the git host. @@ -158,13 +176,11 @@ The operator will reconcile *Tenants* as follows: * When `.spec.compilePipeline` exists and isn't empty, the following entries are added to the tenant repository GitRepo's `.spec.ciVariables`: ** `COMMODORE_API_URL`, containing the URL at which the Lieutenant API can be accessed. -** `COMMODORE_API_TOKEN`, containing the tenant's access token for the Lieutenant API. +** `COMMODORE_API_TOKEN`, containing a reference to the secret which contains the tenant's access token for the Lieutenant API. ** `CLUSTERS`, containing a space-separated list of cluster IDs taken directly from `.spec.compilePipeline.clusters`. -* For each entry in `.spec.compilePipeline.pipelineFiles` whose value isn't `{remove}`, a new corresponding entry is added to the tenant's `.spec.gitRepoTemplate.templateFiles`. -* For each entry in `.spec.compilePipeline.pipelineFiles` whose value is `{remove}`, the corresponding entry is removed from the tenant's `.spec.gitRepoTemplate.templateFiles`. * For each entry in `.spec.compilePipeline.clusters`, another entry is added to the tenant repository GitRepo's `spec.ciVariabes`. The key is `ACCESS_TOKEN_CLUSTERNAME`, where `CLUSTERNAME` is the ID of a specific cluster, with `-` replaced by `_`. - The value is the access token to access that cluster's catalog repository, taken from the secret specified in the catalog GitRepo configuration under `.spec.accessTokenSecretRef`. + The value is a reference to the secret containing the access token to access that cluster's catalog repository, taken from the secret specified in the catalog GitRepo configuration under `.spec.accessTokenSecretName`. === Implementation Details/Notes/Constraints From 1a451419b2d87a07a90d6a5d0cb01114b421243a Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Tue, 18 Jun 2024 10:45:27 +0200 Subject: [PATCH 05/11] Apply suggestions from code review Co-authored-by: Simon Gerber --- .../SDDs/pages/0032-compile-pipeline.adoc | 36 +++++++------------ 1 file changed, 12 insertions(+), 24 deletions(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc index 28cb81f..6cab3db 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -2,7 +2,7 @@ :sdd_author: Aline Abler :sdd_owner: Project Syn IG -:sdd_reviewers: Simon Gerber +:sdd_reviewers: Simon Gerber, Tobias Brunner :sdd_date: 2024-06-13 :sdd_status: draft @@ -62,8 +62,10 @@ The `accessTokenSecretName` field contains a reference to a secret. If it is set, the Lieutenant operator will store an access token into this secret, which can be used to access the Git repository. In the case of GitLab, this would be a Project Access Token with read-write access to the repository. -The `ciVariables` field contains a dictionary describing variable names and corresponding values. -These variables are added to the git repository as CI/CD variables. +The `ciVariables` field contains a list of objects describing variable names and corresponding values. +Each object in the list has a type that's modeled after a Kubernetes container's `env` field. +In contrast to container environment variables, our variables only support specifying values directly (via field `value`) or by referencing a `Secret` resource (via field `valueFrom.secretKeyRef`). +These variables are added to the Git repository as CI/CD variables. [source,yaml] ---- @@ -113,6 +115,9 @@ The `compilePipeline` field contains a dict with the following fields: * `clusters`: List of cluster IDs of clusters for which the compile pipeline should be executed. This field is managed by the operator. +* `pipelineFiles`: Dictionary containing file paths as keys, and file contents as values. + These files will be added to the tenant's `gitRepoTemplate.templateFiles` by the Lieutenant operator. + This field is optional; if absent, no new template files are added to the `gitRepoTemplate`. [source,yaml] ---- @@ -124,35 +129,17 @@ spec: compilePipeline: clusters: - c-my-cluster ----- - -=== In-Repo CI/CD pipeline configuration - -Configuring the CI pipeline usually happens through files committed to the corresponding repository. -For a Lieutenant-managed pipeline configuration, these files should be managed by Lieutenant. -To achieve this, we can leverage the existing mechanism to commit template files to git repositories: - -[source,yaml] ----- -apiVersion: syn.tools/v1alpha1 -kind: Tenant -metadata: - name: t-my-tenant -spec: - gitRepoTemplate: - templateFiles: + pipelineFiles: .gitlab-ci.yml: | include: - project: syn/commodore-compile-pipeline ref: master file: /.gitlab/commodore-common.yml - ---- - === Operator -The Lieutenant Operator will be extended to automatically manage the compile pipeline for repositories where this is enabled (by way of configuring the `compilePipeline` field on the tenant and the `enableCompilePipeline` field on the cluster). +The Lieutenant Operator will be extended to automatically manage the compile pipeline for repositories where this is enabled (by way of deploying the CI config file in the tenant and the `enableCompilePipeline` field on the cluster). Since the compile pipeline has to interact with both the tenant repository as well as the cluster catalog repositories, it must be enabled on both corresponding resources for the configuration to be functional. This way, it is possible to enable auto-compilation for some, but not all clusters on a tenant. @@ -162,7 +149,7 @@ The operator will reconcile *GitRepos* as follows: * When `spec.accessTokenSecretName` is set, the operator generates an access token for the corresponding repository (via the repository host's API, using the API secret in `.spec.apiSecretRef`), and writes this token into a secret with the given name. In the case of GitLab, this is a Project Access Token. The operator also runs a scheduled job which refreshes these tokens when they are close to expiring, or when they no longer exist on the repository host. -* The content of `.spec.ciVariables` is written to the repository's configuration on the git host. +* The content of `.spec.ciVariables` is written to the repository's configuration on the Git host. In the case of GitLab, it is written as CI/CD variables. NOTE: If the GitRepo is of type `unmanaged`, none of these steps will be executed. @@ -181,6 +168,7 @@ The operator will reconcile *Tenants* as follows: * For each entry in `.spec.compilePipeline.clusters`, another entry is added to the tenant repository GitRepo's `spec.ciVariabes`. The key is `ACCESS_TOKEN_CLUSTERNAME`, where `CLUSTERNAME` is the ID of a specific cluster, with `-` replaced by `_`. The value is a reference to the secret containing the access token to access that cluster's catalog repository, taken from the secret specified in the catalog GitRepo configuration under `.spec.accessTokenSecretName`. +* For each entry in `.spec.compilePipeline.pipelineFiles`, a new corresponding entry is added to the tenant's `.spec.gitRepoTemplate.templateFiles`. === Implementation Details/Notes/Constraints From ac3caef4b9842505e30f819771158f4dd3144648 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 21 Jun 2024 10:30:55 +0200 Subject: [PATCH 06/11] Apply suggestions from Sebi's code review Co-authored-by: Sebastian Widmer --- docs/modules/SDDs/pages/0032-compile-pipeline.adoc | 8 +++++--- docs/modules/SDDs/pages/index.adoc | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc index 6cab3db..35206a0 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -1,8 +1,8 @@ -= SDD 0031 - Central Component Version tracking += SDD 0032 - Commodore Compile Pipeline for GitLab :sdd_author: Aline Abler :sdd_owner: Project Syn IG -:sdd_reviewers: Simon Gerber, Tobias Brunner +:sdd_reviewers: Simon Gerber, Tobias Brunner, Sebastian Widmer :sdd_date: 2024-06-13 :sdd_status: draft @@ -11,7 +11,7 @@ include::partial$meta-info-table.adoc[] [NOTE] .Summary ==== -This describes how we want to implement CI/CD for Project Syn using a Commodore compile pipeline, which enables automatic compilation of cluster catalogs whenever the corresponding tenant repository is modified. +This describes how we want to implement GitLab CI/CD for Project Syn using a Commodore compile pipeline, which enables automatic compilation of cluster catalogs whenever the corresponding tenant repository is modified. Furthermore, it explains how want to extend the functionality of Lieutenant to enable it to automatically configure the compile pipeline on tenant repositories. ==== @@ -174,6 +174,8 @@ The operator will reconcile *Tenants* as follows: Currently, we're looking at a solution that is specific to GitLab CI/CD, and to tenant and catalog repositories that are stored on GitLab. Other CI solutions and other repository hosts might be supported in the future. +The implementation takes care to gracefully skip the CI steps if not on GitLab. +We leave the implementation extensible enough to add further Git management tools and CI systems without needing breaking changes. Existing compile pipeline configuration:: If a setup already includes a bunch of tenant repositories with manually configured CI/CD, some care has to be taken to ensure the new implementation can "adopt" this configuration. diff --git a/docs/modules/SDDs/pages/index.adoc b/docs/modules/SDDs/pages/index.adoc index 19f3dfa..5cdcb87 100644 --- a/docs/modules/SDDs/pages/index.adoc +++ b/docs/modules/SDDs/pages/index.adoc @@ -26,4 +26,4 @@ The all are using the xref:sdd-template.adoc[SDD Template]. * xref:0028-reusable-config-packages.adoc[0028 - Reusable Commodore Component Configuration Packages] * xref:0030-argocd-multitenancy.adoc[0030 - Project Syn ArgoCD Multi-Tenant Support] * xref:0031-component-version-tracking.adoc[0031 - Central Component Version tracking] -* xref:0032-compile-pipeline.adoc[0032 - Commodore Compile Pipeline] +* xref:0032-compile-pipeline.adoc[0032 - Commodore Compile Pipeline for GitLab] From c0b09060b4f663b90043fa7b1e4c110e83dc5a61 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 21 Jun 2024 10:48:37 +0200 Subject: [PATCH 07/11] Separate operator-managed fields into `.status` --- ...adoc => 0032-compile-pipeline-gitlab.adoc} | 41 ++++++++++++------- docs/modules/SDDs/pages/index.adoc | 2 +- 2 files changed, 27 insertions(+), 16 deletions(-) rename docs/modules/SDDs/pages/{0032-compile-pipeline.adoc => 0032-compile-pipeline-gitlab.adoc} (83%) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc similarity index 83% rename from docs/modules/SDDs/pages/0032-compile-pipeline.adoc rename to docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc index 35206a0..7c0cb7a 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc @@ -14,6 +14,8 @@ include::partial$meta-info-table.adoc[] This describes how we want to implement GitLab CI/CD for Project Syn using a Commodore compile pipeline, which enables automatic compilation of cluster catalogs whenever the corresponding tenant repository is modified. Furthermore, it explains how want to extend the functionality of Lieutenant to enable it to automatically configure the compile pipeline on tenant repositories. + +While we only aim to support GitLab at the moment, the architecture should be sufficiently generic so it can be used for other Git hosts in the future. ==== == Motivation @@ -105,20 +107,25 @@ spec: === Tenant CRD -We add a new field to the `Tenant` CRD, under the `.spec` key, called `compilePipeline`. +We add new fields to the `Tenant` CRD: `spec.compilePipeline` and `status.compilePipeline` -The `compilePipeline` field contains configuration pertaining to the automatic setup of the compile pipeline on the tenant repository. +The `spec.compilePipeline` field contains configuration pertaining to the automatic setup of the compile pipeline on the tenant repository. It is optional. -Absence of the field disables automatic setup and management of the compile pipeline. -The `compilePipeline` field contains a dict with the following fields: +The `spec.compilePipeline` field contains a dict with the following fields: -* `clusters`: List of cluster IDs of clusters for which the compile pipeline should be executed. - This field is managed by the operator. +* `enabled`: Boolean field which enables or disables automatic setup of compile pipelines for this tenant (regardless of whether it is enabled on the tenant's clusters). * `pipelineFiles`: Dictionary containing file paths as keys, and file contents as values. These files will be added to the tenant's `gitRepoTemplate.templateFiles` by the Lieutenant operator. This field is optional; if absent, no new template files are added to the `gitRepoTemplate`. +`spec.compilePipeline` is optional. Its absence disables automatic setup of compile pipelines for the tenant, as does setting `spec.compilePipeline.enabled` to `false`. + +The `status.compilePipeline` field contains a dict with one field: + +* `clusters`: List of cluster IDs of clusters for which the compile pipeline should be executed. + This field is managed by the operator. + [source,yaml] ---- apiVersion: syn.tools/v1alpha1 @@ -127,14 +134,16 @@ metadata: name: t-my-tenant spec: compilePipeline: - clusters: - - c-my-cluster pipelineFiles: .gitlab-ci.yml: | include: - project: syn/commodore-compile-pipeline ref: master file: /.gitlab/commodore-common.yml +status: + compilePipeline: + clusters: + - c-my-cluster ---- === Operator @@ -151,6 +160,8 @@ The operator will reconcile *GitRepos* as follows: The operator also runs a scheduled job which refreshes these tokens when they are close to expiring, or when they no longer exist on the repository host. * The content of `.spec.ciVariables` is written to the repository's configuration on the Git host. In the case of GitLab, it is written as CI/CD variables. + If the content of `.spec.ciVariables` changes, the corresponding configuration on the Git host should be updated. + A scheduled job in the ooperator regularly checks for drift between `.spec.ciVariables` and the configuration on the Git host, and updates the latter if necessary. NOTE: If the GitRepo is of type `unmanaged`, none of these steps will be executed. @@ -159,13 +170,13 @@ The operator will reconcile *Clusters* as follows: * When `.spec.enableCompilePipeline` is set to `true`, the tenant's `spec.compilePipeline.clusters` is updated to contain the cluster ID. * Similarly, when the field is set to `false` or missing, the tenant's `spec.compilePipeline.clusters` is updated to not contain the cluster ID. -The operator will reconcile *Tenants* as follows: +The operator will reconcile *Tenants* as follows, if and only if `spec.compilePipeline.enabled` is set to `true`: -* When `.spec.compilePipeline` exists and isn't empty, the following entries are added to the tenant repository GitRepo's `.spec.ciVariables`: +* The following entries are added to the tenant repository GitRepo's `.spec.ciVariables`: ** `COMMODORE_API_URL`, containing the URL at which the Lieutenant API can be accessed. ** `COMMODORE_API_TOKEN`, containing a reference to the secret which contains the tenant's access token for the Lieutenant API. -** `CLUSTERS`, containing a space-separated list of cluster IDs taken directly from `.spec.compilePipeline.clusters`. -* For each entry in `.spec.compilePipeline.clusters`, another entry is added to the tenant repository GitRepo's `spec.ciVariabes`. +** `CLUSTERS`, containing a space-separated list of cluster IDs taken directly from `.status.compilePipeline.clusters`. +* For each entry in `.status.compilePipeline.clusters`, another entry is added to the tenant repository GitRepo's `spec.ciVariabes`. The key is `ACCESS_TOKEN_CLUSTERNAME`, where `CLUSTERNAME` is the ID of a specific cluster, with `-` replaced by `_`. The value is a reference to the secret containing the access token to access that cluster's catalog repository, taken from the secret specified in the catalog GitRepo configuration under `.spec.accessTokenSecretName`. * For each entry in `.spec.compilePipeline.pipelineFiles`, a new corresponding entry is added to the tenant's `.spec.gitRepoTemplate.templateFiles`. @@ -178,12 +189,12 @@ The implementation takes care to gracefully skip the CI steps if not on GitLab. We leave the implementation extensible enough to add further Git management tools and CI systems without needing breaking changes. Existing compile pipeline configuration:: -If a setup already includes a bunch of tenant repositories with manually configured CI/CD, some care has to be taken to ensure the new implementation can "adopt" this configuration. +If a setup already includes a bunch of tenant repositories with manually configured CI/CD, in the general case, the new implementation can "adopt" this configuration. + -In particular, these repositories would already have a working `.gitlab-ci.yml` that probably can be left as-is. +In particular, these repositories would already have a working `.gitlab-ci.yml` that probably could be left as-is, but can also be replaced by a lieutenant-managed one. + Any existing manually created Project Access Tokens will be superseded by new auto-generated ones. -This will lead to a bunch of now-unused tokens needing to be cleaned up. +This will lead to a bunch of now-unused tokens needing to be cleaned up, but should otherwise work without requiring extra effort. External Catalog Repositories:: There may be cases where the catalog repositories are not hosted on the same repository host as the tenant repository, in which case API access for the purpose of creating Project Access Tokens is unavailable. diff --git a/docs/modules/SDDs/pages/index.adoc b/docs/modules/SDDs/pages/index.adoc index 5cdcb87..1d918e1 100644 --- a/docs/modules/SDDs/pages/index.adoc +++ b/docs/modules/SDDs/pages/index.adoc @@ -26,4 +26,4 @@ The all are using the xref:sdd-template.adoc[SDD Template]. * xref:0028-reusable-config-packages.adoc[0028 - Reusable Commodore Component Configuration Packages] * xref:0030-argocd-multitenancy.adoc[0030 - Project Syn ArgoCD Multi-Tenant Support] * xref:0031-component-version-tracking.adoc[0031 - Central Component Version tracking] -* xref:0032-compile-pipeline.adoc[0032 - Commodore Compile Pipeline for GitLab] +* xref:0032-compile-pipeline-gitlab.adoc[0032 - Commodore Compile Pipeline for GitLab] From b376990e5abaf2243fe4a25028b2f7101efb1b3b Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 21 Jun 2024 11:10:04 +0200 Subject: [PATCH 08/11] Revert "Apply suggestions from Sebi's code review" This reverts commit ac3caef4b9842505e30f819771158f4dd3144648. --- docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc | 8 +++----- docs/modules/SDDs/pages/index.adoc | 2 +- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc index 7c0cb7a..29d0732 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc @@ -1,8 +1,8 @@ -= SDD 0032 - Commodore Compile Pipeline for GitLab += SDD 0031 - Central Component Version tracking :sdd_author: Aline Abler :sdd_owner: Project Syn IG -:sdd_reviewers: Simon Gerber, Tobias Brunner, Sebastian Widmer +:sdd_reviewers: Simon Gerber, Tobias Brunner :sdd_date: 2024-06-13 :sdd_status: draft @@ -11,7 +11,7 @@ include::partial$meta-info-table.adoc[] [NOTE] .Summary ==== -This describes how we want to implement GitLab CI/CD for Project Syn using a Commodore compile pipeline, which enables automatic compilation of cluster catalogs whenever the corresponding tenant repository is modified. +This describes how we want to implement CI/CD for Project Syn using a Commodore compile pipeline, which enables automatic compilation of cluster catalogs whenever the corresponding tenant repository is modified. Furthermore, it explains how want to extend the functionality of Lieutenant to enable it to automatically configure the compile pipeline on tenant repositories. @@ -185,8 +185,6 @@ The operator will reconcile *Tenants* as follows, if and only if `spec.compilePi Currently, we're looking at a solution that is specific to GitLab CI/CD, and to tenant and catalog repositories that are stored on GitLab. Other CI solutions and other repository hosts might be supported in the future. -The implementation takes care to gracefully skip the CI steps if not on GitLab. -We leave the implementation extensible enough to add further Git management tools and CI systems without needing breaking changes. Existing compile pipeline configuration:: If a setup already includes a bunch of tenant repositories with manually configured CI/CD, in the general case, the new implementation can "adopt" this configuration. diff --git a/docs/modules/SDDs/pages/index.adoc b/docs/modules/SDDs/pages/index.adoc index 1d918e1..19f3dfa 100644 --- a/docs/modules/SDDs/pages/index.adoc +++ b/docs/modules/SDDs/pages/index.adoc @@ -26,4 +26,4 @@ The all are using the xref:sdd-template.adoc[SDD Template]. * xref:0028-reusable-config-packages.adoc[0028 - Reusable Commodore Component Configuration Packages] * xref:0030-argocd-multitenancy.adoc[0030 - Project Syn ArgoCD Multi-Tenant Support] * xref:0031-component-version-tracking.adoc[0031 - Central Component Version tracking] -* xref:0032-compile-pipeline-gitlab.adoc[0032 - Commodore Compile Pipeline for GitLab] +* xref:0032-compile-pipeline.adoc[0032 - Commodore Compile Pipeline] From b8c3a3c52e603254b784b909b179c9475430f5a7 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 21 Jun 2024 11:12:18 +0200 Subject: [PATCH 09/11] Re-apply select parts of Sebi's code review --- ...le-pipeline-gitlab.adoc => 0032-compile-pipeline.adoc} | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) rename docs/modules/SDDs/pages/{0032-compile-pipeline-gitlab.adoc => 0032-compile-pipeline.adoc} (96%) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc similarity index 96% rename from docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc rename to docs/modules/SDDs/pages/0032-compile-pipeline.adoc index 29d0732..c52d95b 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline-gitlab.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -2,7 +2,7 @@ :sdd_author: Aline Abler :sdd_owner: Project Syn IG -:sdd_reviewers: Simon Gerber, Tobias Brunner +:sdd_reviewers: Simon Gerber, Tobias Brunner, Sebastian Widmer :sdd_date: 2024-06-13 :sdd_status: draft @@ -183,9 +183,13 @@ The operator will reconcile *Tenants* as follows, if and only if `spec.compilePi === Implementation Details/Notes/Constraints -Currently, we're looking at a solution that is specific to GitLab CI/CD, and to tenant and catalog repositories that are stored on GitLab. +Currently, we're looking at implementing this solution for GitLab CI/CD only, and for tenant and catalog repositories that are stored on GitLab. Other CI solutions and other repository hosts might be supported in the future. +The implementation takes care to gracefully skip the CI steps if not on GitLab. + +We leave the implementation extensible enough to add further Git management tools and CI systems without needing breaking changes. + Existing compile pipeline configuration:: If a setup already includes a bunch of tenant repositories with manually configured CI/CD, in the general case, the new implementation can "adopt" this configuration. + From 2a4eb2e294db32816f37b36b48991b67c2163db8 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 21 Jun 2024 11:42:14 +0200 Subject: [PATCH 10/11] Fix title --- docs/modules/SDDs/pages/0032-compile-pipeline.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc index c52d95b..30974b9 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -1,4 +1,4 @@ -= SDD 0031 - Central Component Version tracking += SDD 0032 - Commodore Compile Pipeline :sdd_author: Aline Abler :sdd_owner: Project Syn IG From 178836d36949577b2bb10d1da53ef6b007bd2766 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Fri, 21 Jun 2024 12:09:52 +0200 Subject: [PATCH 11/11] Apply suggestions from code review Co-authored-by: Simon Gerber --- docs/modules/SDDs/pages/0032-compile-pipeline.adoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc index 30974b9..bd0aaeb 100644 --- a/docs/modules/SDDs/pages/0032-compile-pipeline.adoc +++ b/docs/modules/SDDs/pages/0032-compile-pipeline.adoc @@ -4,7 +4,7 @@ :sdd_owner: Project Syn IG :sdd_reviewers: Simon Gerber, Tobias Brunner, Sebastian Widmer :sdd_date: 2024-06-13 -:sdd_status: draft +:sdd_status: accepted include::partial$meta-info-table.adoc[] @@ -34,12 +34,13 @@ This will go hand-in-hand with the existing repository management features in Li === Goals -* Provide pipeline definitions for GitLab CI/CD to automatically compile and push cluster catalogs on a tenant repository. +* Specify the interface that Commodore CI pipeline definitions need to implement in order to work with the managed CI/CD configuration * Enable Lieutenant to autonomously manage the configuration required to set up the compile pipeline for a cluster catalog. === Non-Goals * Support for CI solutions other than GitLab CI/CD. +* Provide pipeline definitions to automatically compile and push cluster catalogs on a tenant repository. == Design Proposal