Bootstrap multiple clusters into one single shared directory

[HamzaZo]

Hello,

We have more than 200 clusters, and I wonder if Flux bootstrap supports bootstrapping multiple clusters into a single shared folder flux-system

For example, running the same command on multiple clusters:
flux bootstrap git --url=ssh://[email protected]/repository.git --path=clusters/flux-system

[stefanprodan]

If the clusters run the same apps then Yes you can reuse the path, but it should be clusters/shared, bootstrap will create the flux-system dir first time you run it, inside shared.

To set different values per cluster, let's say ingress domains, you can use Flux variable substitutions, even if in Git all are using the same manifests.

[HamzaZo]

it seems that the Flux Operator will do the same thing continually committing the overridden manifests as mentioned in https://fluxcd.control-plane.io/operator/fluxinstance/#sync-from-git-over-ssh

spec:
  sync:
    kind: GitRepository
    url: "ssh://[email protected]/my-org/my-fleet.git"
    ref: "refs/heads/main"
    path: "clusters/my-cluster"
    pullSecret: "git-ssh-auth"

[stefanprodan]

The Flux Operator doesn't write to Git...

[stefanprodan]

There are no manifests in Git at all, the Flux manifests are managed by the operator itself, directly in etcd, based on the version specified in the FluxInstance.

[HamzaZo]

I see, is there any way to add tolerations or podAntiAffinity to the generated manifests by the operator. I tried to do something like this but it does not work:

spec:
  kustomize:
    patches:
      - target:
          kind: Deployment
          name: "(kustomize-controller|helm-controller)"
        patch: |
          - op: add
            path: /spec/template/spec/tolerations/-1
            value: {"operator": "Exists"}

I'm getting build failed: add operation does not apply: doc is missing path: "/spec/template/spec/tolerations/-1": missing value
any idea please @stefanprodan ?

[stefanprodan]

See https://fluxcd.control-plane.io/operator/flux-kustomize/

[stefanprodan]

If you are deploying Flux on hundreds of clusters you may want to consider Flux Operator which will make Flux installation and upgrades easier at this scale.

[HamzaZo]

Is it ready for production use?

[stefanprodan]

It's based on the same code as bootstrap, so I would say Yes 💯

[HamzaZo]

Thanks for your reply, I'll give it a try