Our implementation of multi-cluster FluxCD. Need advice.

[alexandrtru]

Our implementation of multi-cluster FluxCD

Hi, fluxcd team!
Sorry if is duscuss somewhere.

In our company, we search the easy, flexibility and pull-model way to manage kuberenetes clusters (hundreds of clusters, and its count increase). We are experimented a lot with fluxcd and find the solution that covered most all of needs, but i have some doubts about this because it's looks like trick. So, if you can give some advice or warnings about our way using fluxcd, we will be grateful.

Core concept may called as "Cascade Kustomize", when we generate downstream Kustomization (kustomize.toolkit.fluxcd.io/v1) using .spec.postBuild in upstream (entrypoint) Kustomization.

So, our "entrypoint" for each cluster looks like:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg
  interval: 3m
  prune: true
  path: ./profiles/general/
  sourceRef:
    kind: GitRepository
    name: flux-system
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
        optional: false

Note: we provide cluster-vars configmap at deploy cluster, and fill it with necessary for substitution variables.

One of "downstream" kustomize looks like:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: vector
  namespace: flux-system
spec:
  interval: 6m
  sourceRef:
    kind: GitRepository
    name: flux-platform
  path: ./apps/vector
  prune: true
  wait: true
  dependsOn:
    - name: namespaces
    - name: secrets
  components:
    - ../../patches/00_envs/all/apps/vector
    - ../../patches/00_envs/${ENV}/apps/vector
    - ../../patches/01_os/${OS:=default}/apps/vector
    - ../../patches/02_kubernetes_versions/${KUBE_VERSION:=default}/apps/vector
    - ../../patches/03_clusters/${CLUSTER_NAME:=default}/apps/vector
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
        optional: false

As we can see substitution used in .spec.components and we targeting to some kind of patches (or components) for basic configuration placed in ./apps/vector in this (or remote) git-repository.

And in general terms that's all :) Main and killer benefit of this scheme - we are able applying patches to basic configuration in order what we need by using information that cluster tell to us via configmap. It is sufficient flexibility level for our swarm of clusters and teams that works with our clusters.

PROS:

1. human readable repository structure
2. high (sufficient) level of flexibility
3. cluster-provided targeting to kustomize and patches
4. possibility using remote repository with basic apps (kustomizes) and configuration
5. easy to automate add new cluster
6. it's works as we expect now :)

CONS/doubts:

1. Looks like we are using some kind of non-guaranteed and non-stable feature (we know about alpha state .spec.components)
2. Kustomize fails if some of component not exist. The more basic kustomizes (that contain .spec.components ) we use, the more we must generate empty (mock) components for prevent kustomize fail.
3. Tell us about some obvious cons and problems that we missing? Maybe this approach has many pitfalls?

QUESTIONS/suggestions:

1. We're worried about alpha/experimental state of .spec.components. Do you plan for delete this feature?
2. As i say in second "CONS" we must do much of stupid work. Create directories, kustomization.yml with resources: [], and place it in right places, and etc. We can try write new feature gate for kustomize-controller, that make kustomizes more fault tolerate in case if component not exists at filesystem. What your opinion to this?

It's all what i want to share.

Thanks for your work.