From 4f4f15651507c66aa1645ba39d4b1b442ddb02b3 Mon Sep 17 00:00:00 2001 From: Johannes Kleinlercher Date: Wed, 29 May 2024 10:19:24 +0200 Subject: [PATCH] restructured onboarding teams and onboarding apps ADRs Signed-off-by: Johannes Kleinlercher --- ...ure.md => 0001-gitops-onboarding-teams.md} | 154 +----------- .../adr/0002-gitops-onboarding-apps.md | 236 ++++++++++++++++++ 2 files changed, 246 insertions(+), 144 deletions(-) rename backstage-resources/adr/{0001-gitops-structure.md => 0001-gitops-onboarding-teams.md} (72%) create mode 100644 backstage-resources/adr/0002-gitops-onboarding-apps.md diff --git a/backstage-resources/adr/0001-gitops-structure.md b/backstage-resources/adr/0001-gitops-onboarding-teams.md similarity index 72% rename from backstage-resources/adr/0001-gitops-structure.md rename to backstage-resources/adr/0001-gitops-onboarding-teams.md index 20fbc0c0..febad111 100644 --- a/backstage-resources/adr/0001-gitops-structure.md +++ b/backstage-resources/adr/0001-gitops-onboarding-teams.md @@ -1,4 +1,8 @@ -# gitops onboarding-process and repos +# Onboarding teams in a gitops way + +* Status: DRAFT +* Deciders: sX CNP oss Team +* Date: 2023-05-29 ## Introduction @@ -20,7 +24,8 @@ This ADR focuses more on the [wiring](https://cloudogu.com/en/blog/gitops-reposi Some of the options below effect the patterns above and some don't. Maybe we can explore that in a next step. -# Onboarding teams +This ADR depends also on ADR "002-gitops-onboarding-apps". + ## Context and Problem Statement @@ -29,10 +34,6 @@ As a I want a mechanism where I can onboard new teams and applications to a cluster with all required policies and standards so that stability/security of the platform is still guaranteed and no negative effects to other teams and applications arise. -As a -***dev-team*** -I want to onboard and maintain my app as a self-service without high friction and wait times. -Still, I want to apply all platform-requirements and restriction with as much automation as possible. Normally onboarding teams and apps includes creating: - argocd app-project with restrictions (source-Repos, destinations, clusterResourceWhitelist, namespaceResourceBlacklist, ...) and roles if needed @@ -288,147 +289,12 @@ pros: cons: - new namespaces is not a self-service (compared to app-in-any-namespace) -# articles and resources +# More information + +## articles and resources https://github.com/cloudogu/gitops-patterns https://github.com/akuity/awesome-argo https://cloudogu.com/en/blog/gitops-repository-patterns-part-1-introduction - - - - - - -# TODO: Old Content, maybe we integrate that later or delete it - - -# Considered options for different use-cases - -Which options are for these user-stories? - -As a -**dev-team with lots of knowledge in kubernetes, argocd, kustome and helm** -I want to onboard my app with high flexibility and with less governance as possible -to make use of any configuration mgmt tools and kubernetes manifests which are possible. -Still, I want a platform-team responsible for the cluster and the cluster is used by multiple dev-teams, -not just dedicated for one dev-team. - - -***option 7*** - -As a -**dev-team with lots of knowledge in kubernetes, argocd, kustome and helm** -I want to onboard my app with high flexibility and with less governance as possible -to make use of any configuration mgmt tools and kubernetes manifests which are possible. -The cluster is dedicated for my team, but I still want a platform-team which is responsible for the cluster-mgmt. - -***option 8*** - -As a -**dev-team with lots of knowledge in kubernetes, argocd, kustome and helm** -I want to onboard my app with high flexibility and with less governance as possible -to make use of any configuration mgmt tools and kubernetes manifests which are possible. -The cluster is dedicated for my team and I take full responsibility for this cluster. -I use or copy some basic configurations from a platform-team but treat them more as a good practice which I can use or not. - - - -## User Stories - -TODO: maybe those are some pros/cons for the different approaches - -These user stories a ordered by expertise of the platform consumers: - -- little skills and knowledge - high standardization and governance -- medium skills and knowledge - high standardization, governance, but more flexibility and insights -- high skills and knowledge - very flexible, still some governance where needed and lots of insights - -and the self-service capability of the platform - -- namespace / project / tenant as a service -- ticket / PRs to platform-team and reviews required per app - -Disclaimer: as always, enabling and "skill-up" dev-teams is always an advantages. -Still, some dev-teams are not able to skill them up or don't have enough resources. -Maybe, when this dev-teams get more experience their requirements towards more insights and flexibility -and less governance changes. -Also, maybe insights (which kubernetes-manifests are created, where do we have problems) are always helpful. -Then not so experienced teams can learn from the insights they get. -On the other way, also for experienced teams the platform should be easy to use, but not limiting in solutions. -That is all about golden paths instead of golden cages. - - - -# TODO: maintaining / changing apps is different in each options above, we should highlight how this works then - -# centrally controlled team-onboarding, self-service app onboarding - -team-onboarding via platform-repo, application onboarding self-service/automatically - -Seems to be like -- User Story 1, option 4 -- User Story 2, option 2 - -## environemnt - -Can be applied on the operator deployment pattern "Instance per Cluster" or "Hub and Spoke" - -With the "instance per namespace" pattern this solution is probably not so useful, -because there the team-isolation is more on per-argocd-instance then on per-argo-project. -However, the app onboarding via applicationsets can be applied also on this pattern. - -## process - -1. when onboarding the new team, the platform-team creates - 1. an argo project, so resource whitelists / blacklists / rbac can be defined by platform team - 2. applicationset with -[SCM Provider generator](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-SCM-Provider/) or [Git generator](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Git/) (depending of your repository pattern) -and using the defined argo project in step 1 - - since this is also done via gitops, the dev-team could also create a PR in a central platform-repo for this - -2. dev-teams create their gitops-repo based on the pattern defined in the applicationset in step 1.2 and the new application is automatically created - -## pros - -- new applications which match the applicationset pattern are automatically created without any review of a central platform team -- dev-team doesn't need to know the application definition - -## cons - -- platform-team needs to be fine with onboarding new applications without review. so some governance need to be in place (default policies in namespace, kyverno, netpol, resource-quotas, limit-ranges) -- since applicationsets are templates every application in this applicationset has very similar attributes or attributes need to be imported by config files per gitops-repo - -## examples - - -## further implementation variations - -- multiple sources so dev-teams only need to have some values in their repo and the chart is placed at some other place - see 'multiple sources' solution - -- not only automatically app creation, but also kargo resources, namespace with some default resources, ... - - -# team and application onboarding in central platform-repo, simple values, renders additional resources - -like ARZ solution: - -simple values in central repo -not only automatically app creation, but also kargo resources, namespace with some default resources, ... - -be aware that those value files can be really big and onboarding new value files (corresponds to onbarding new teams) also needs a good implementation.(or values-patterns if that is possible now, e.g. values-team-*) - - -# application in different namespaces - -# multiple sources - -## examples - -https://github.com/jkleinlercher/just-one-yaml-deployment/blob/main/argocd-config/just-one-yaml-applicationset.yaml - -# helmfiles - -# self-service and governance \ No newline at end of file diff --git a/backstage-resources/adr/0002-gitops-onboarding-apps.md b/backstage-resources/adr/0002-gitops-onboarding-apps.md new file mode 100644 index 00000000..87d6f7c9 --- /dev/null +++ b/backstage-resources/adr/0002-gitops-onboarding-apps.md @@ -0,0 +1,236 @@ +# Onboarding apps + +* Status: DRAFT +* Deciders: sX CNP oss Team +* Date: 2023-05-29 + +This ADR depends also on ADR "001-gitops-onboarding-teams". +In the different solution options we mention for which "onboarding teams" options this option is applicable. +Some are applicable to all team-onboarding-scenarios, some are just for specific team-onboarding scenarios. + +In the future we can probably draw this decision tree on a map in the future. + +## Context and Problem Statement + +As a +***dev-team*** +I want to onboard and maintain my app as a self-service without high friction and wait times. +Still, I want to apply all platform-requirements and restriction with as much automation as possible. + +Normally onboarding teams and apps includes creating: + - argocd app-project with restrictions (source-Repos, destinations, clusterResourceWhitelist, namespaceResourceBlacklist, ...) and roles if needed + - argocd app + - app namespaces + - default configuration inside namespaces (limit-ranges, quotas, deny-all network-policies) + - maybe some additonal platform-services (e.g. observability-tool ) also need to be configured for the new team/application/namespace (this isn't addressed in this ADR for now) + +Depending on the solution some of this parts are done by the platform-team, by the dev-team, or automatically. + +## Decision Drivers + +- high self-service and low friction for dev-teams +- security-standards / multi-tenant configurations in place +- since not all teams have the same knowledge, some low-knowledge teams and also high-knowledge teams should get their best solution +- the solution should fit also to different "operator deployment patterns" and "promotion patterns" + +## Considered Options + +* self-service app and app-namespace onboarding +* ApplicationSet with simple "deployment descriptor" +* ApplicationSet with Parent-Helm-Chart +* ApplicationSet with config.json representing a Parent-Helm-Chart +* Central gitops-Repo for app and app-namespace onboarding +* App onboarding for namespace-scoped argocd instance + +## Decision Outcome + +We should explore which option is the best for which team and provide different show cases. +Also, we should find out if we can support multiple approaches on one platform. + +Which options are best for these user-stories? + +As a +**dev-team without deep knowledge in argocd, kustomize, helm and kubernetes manifest** +I want to onboard my app with a very slim and easy API and as much sane defaults as possible +and strong governance to prevent any problems because of limited knowledge. + +As a +**dev-team with lots of knowledge in kubernetes, argocd, kustome and helm** +I want to onboard my app with high flexibility and with less governance as possible +to make use of any configuration mgmt tools and kubernetes manifests which are possible. +Still, I want a platform-team responsible for the cluster and the cluster is used by multiple dev-teams, +not just dedicated for one dev-team. + +As a +**dev-team with lots of knowledge in kubernetes, argocd, kustome and helm** +I want to onboard my app with high flexibility and with less governance as possible +to make use of any configuration mgmt tools and kubernetes manifests which are possible. +The cluster is dedicated for my team, but I still want a platform-team which is responsible for the cluster-mgmt. + +As a +**dev-team with lots of knowledge in kubernetes, argocd, kustome and helm** +I want to onboard my app with high flexibility and with less governance as possible +to make use of any configuration mgmt tools and kubernetes manifests which are possible. +The cluster is dedicated for my team and I take full responsibility for this cluster. +I use or copy some basic configurations from a platform-team but treat them more as a good practice which I can use or not. + + +### Consequences + +## Validation + +## Pros and Cons of the Options + +### self-service app and app-namespace onboarding + +Only applicable for team-onboarding option "Apps-in-any-Namespace and Multi-Tenant Kyverno-Policies"! +If you use this in a central argocd instance without "apps-in-any-namespace" there is a privilege-escalation-risk +because the child-app in the app-of-apps repo can use any argocd app-project in this argocd instance and can deploy in any namespace (see https://github.com/argoproj/argo-cd/issues/2785)! + +Dev-team creates a new argocd app-definition in their app-of-apps gitops-repo. +Therefore the platform-team needs to define a app-of-apps definition upfront which points to this gitops-repo. + +ApplicationSets-in-any-namespace are also possible with some restrictions: https://github.com/suxess-it/sx-cnp-oss/issues/181 +So Dev-Teams can create their own ApplicationSets without a platform-team. Without "Apps-in-any-Namespace" ApplicationSets were also only safe when platform-teams defined them, otherwise dev-teams could use any argocd app-project. + +Example: +https://github.com/suxess-it/sx-cnp-oss/blob/12883d99657732d145d2992afa21b554403abd37/platform-apps/charts/argocd/values-k3d.yaml#L13-L16 +and https://github.com/suxess-it/team1-apps/tree/main/k3d-apps + +pros: +- very high self-service for dev-teams without the need of a central managed applicationset +- no privilege escalation for dev-teams app-definitions possible, they need to stay in the teams app-project +- high standardization, security and soft-multi-tenancy standards for the namespaces because of kyverno generate policies for each namespace +- gitops-repo of the platform-team keeps quite manageable, because not every app-definition needs to be in this central gitops-repo + +cons: +- apps-in-any-namespace is still beta, although it seems very stable +- applicationsets then have some restrictions (also when used by the platform team), see https://github.com/suxess-it/sx-cnp-oss/issues/181 +- dev-team needs to have high knowledge of argocd app-definitions (could be enhanced with backstage scaffolder templates) + +### ApplicationSet with simple "deployment descriptor" + +Applicable for all team-onboarding options! + +Platform-team creates an application-set per team during team-onboarding with static argo project reference. +[SCM Provider generator](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-SCM-Provider/) or [Git generator](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Git/) (depending of your repository pattern) + +Dev-Team creates their gitops-repos with highly simplified "deployment descriptor", which is in fact a helm values-file. +The Helm-Chart is stored in a different place (helm repo, git repo) managed by the platform team. +The Helm-Chart is very flexible configurable via the values file and has lots of sane defaults. +If you are fine with the sane defaults, you have a small values file, if you need special things, you have a bigger values file. + +Example: [just-one-deployment-yaml](https://github.com/jkleinlercher/just-one-yaml-deployment/blob/main/argocd-config/just-one-yaml-applicationset.yaml) + +pros: +- Dev-team only need a very simple value file. With sane defaults the value files content can be pretty small +- Dev-teams don't need helm or kubernetes resource manifests knowledge +- very slim and easy (but not so flexible) API +- new Parent-Chart versions rollout very fast (but you need to take care about staging and backward-compatibility) + +cons: +- Dev-team has no option to define which K8s manifests and cannot add some additional resources. +- This option is always inflexible for bigger requirements, no matter how flexible the base-chart is +- maybe too less insights what is really going on for dev-teams (probably a matter of documentation and links to the base-chart) +- live-cycle-mgmt for the base-chart is hard, because changes need to be backward-compatible, since no version specification of the base-chart possible +- there is no source-of-truth in the dev-team gitops-repo, so deploying new versions of the app in the dev-stage can end in different results than in prod because the parent-chart is different between dev and prod + +### ApplicationSet with Parent-Helm-Chart + +Applicable for all team-onboarding options! + +Platform-team creates an application-set per team during team-onboarding with static argo project reference. +[SCM Provider generator](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-SCM-Provider/) or [Git generator](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Git/) (depending of your repository pattern) + +Dev-team creates gitops-repos containing a parent-helm-chart just with a values-File and referenced Sub-Chart, +which is maintained by a platform-team or some other base-chart maintainers. + +Namespace-Creation-Self-Service is only possible with Kyverno-Generate-Policies! + +Example: tbd + +pros: +- dev-team knows better which other manifests get created, as long it knows the sources of the sub-chart +- dev-team can specify the version of the chart (renovate helps with updates) +- dev-team can add additional resources in the parent-charts templates-Folder + +cons: +- since applicationsets are templates every application in this applicationset has very similar attributes (needs to be explored) + +### ApplicationSet with config.json representing a Parent-Helm-Chart + +Applicable for all team-onboarding options! + +Platform-team creates an application-set per team during team-onboarding with static argo project reference. +Dev-team creates a gitops-repo containing a config.json with definitions for helm repo, chart and version can be placed (todo: can this be implemented as optional with the same appset of option 1 or is it then mandatory). + +Namespace-Creation-Self-Service is only possible with Kyverno-Generate-Policies! + +Example: https://github.com/thschue/gitops-demo/tree/main/demo + +pros: +- dev-team doesn't need to know the helm structure (but needs to know our own config.json format) +- kubernetes resource best practices can be implemented centrally in this parent-chart + +cons: +- config.json is proprietary, no "helm template" works out-of-the box and no renovate update +- since applicationsets are templates every application in this applicationset has very similar attributes (needs to be explored) + + +### Central gitops-Repo for app and app-namespace onboarding + +Applicable only for team-onboarding option "Central gitops-Repo for app-projects, apps, namespaces and default configurations" + +dev-team creates app-definition in argocd namespace via PR in central gitops-repo which represents the argocd apps. +platform-team needs to review the PR for every app definition if the correct argo project is referenced. + +Example: tbd + +pros: +- tbd + +cons: +- central gitops-repo can get quite large and with lots of applications a dev-team could lose overview + + +### App onboarding for namespace-scoped argocd instance + +Applicable only for team-onboarding option "Seperate namespace-scoped argocd instance for every team" + +Dev-team creates a new argocd app-definition in their app-of-apps gitops-repo. +Therefore the platform-team needs to define a app-of-apps definition upfront which points to this gitops-repo. + +Dev-team can create their argo projects by themselves or use default project since it is limited by the scope of the namespace-scoped instance. +Additional namespaces need to be requested from the platform-team, since the namespaced-scope argocd is pinned to specific namespaces. + +Example: tbd + +pros: +- new apps is a complete self-service +- dev-team can make use of any configuration mgmt tools and kubernetes manifests which are possible + +cons: +- new namespaces is not a self-service (compared to app-in-any-namespace) + + +# More information + +These user stories are ordered by expertise of the platform consumers: + +- little skills and knowledge - high standardization and governance +- medium skills and knowledge - high standardization, governance, but more flexibility and insights +- high skills and knowledge - very flexible, still some governance where needed and lots of insights + +and the self-service capability of the platform + +- namespace / project / tenant as a service +- ticket / PRs to platform-team and reviews required per app + +Disclaimer: as always, enabling and "skill-up" dev-teams is always an advantages. +Still, some dev-teams are not able to skill them up or don't have enough resources. +Maybe, when this dev-teams get more experience their requirements towards more insights and flexibility +and less governance changes. +Also, maybe insights (which kubernetes-manifests are created, where do we have problems) are always helpful. +Then not so experienced teams can learn from the insights they get. +On the other way, also for experienced teams the platform should be easy to use, but not limiting in solutions. +That is all about golden paths instead of golden cages.