Base Chart: Documentation of Flux SSH Secrets & Kustomize Setup

[tmechsner]

Hi there, I want to suggest an extension in the documentation about Flux.

I found out that the base chart automatically creates a secret with ssh credentials when flux.gitRepositories.[x].url starts with ssh:// - would be cool to find that in the documentation (and maybe also what to do with it).

Now that the Source Controller can fetch my repository, I'm not sure how it handles my Kustomize Setup in that repository. You automatically create Kustomization-type resources but I cannot change the path or anything. While locally it works well when executing kubectl apply -k overlays/staging or kubectl apply -k overlays/prod in that repository I get the following errors from the Kustomization-resource after the successful configuration of the GitRepository:
kustomize build failed: accumulating resources: accumulation err='accumulating resources from './overlays/prod': read /tmp/kustomization-912580309/overlays/prod: is a │ │ directory': recursed merging from path '/tmp/kustomization-912580309/overlays/prod': may not add resource with an already registered id: PriorityClass.v1.scheduling.k8s.io/high-priority.[noNs]
(The high-priodity PriorityClass has already been created by previous manual apply calls that however never complained about this resource already existing)

Directory Structure in that repository:

my-repo/
├─ base/
│  ├─ kustomization.yaml
│  ├─ ...
├─ overlays/
│  ├─ prod/
│  │  ├─ kustomization.yaml
│  │  ├─ ...
│  ├─ staging/
│  │  ├─ kustomization.yaml
│  │  ├─ ...

Any input on this?

[cwrau]

Hi there, I want to suggest an extension in the documentation about Flux.

I found out that the base chart automatically creates a secret with ssh credentials when flux.gitRepositories.[x].url starts with ssh:// - would be cool to find that in the documentation (and maybe also what to do with it).

Yeah, you're right, it's not explicitly documented, only through the options; https://github.com/teutonet/teutonet-helm-charts/tree/main/charts/base-cluster#811131-property-base-cluster-configuration--flux--gitrepositories--additionalproperties--allof--item-0--oneof--item-2--url which is pointed at by https://github.com/teutonet/teutonet-helm-charts/tree/main/charts/base-cluster#component-flux.

What to do with it is documented tho; https://github.com/teutonet/teutonet-helm-charts/tree/main/charts/base-cluster#cluster-bootstrap, the last block;

# you can use this command to get the instructions again
# e.g. when adding users, gitRepositories, ...
helm -n flux-system get notes base-cluster

Now that the Source Controller can fetch my repository, I'm not sure how it handles my Kustomize Setup in that repository. You automatically create Kustomization-type resources but I cannot change the path or anything. While locally it works well when executing kubectl apply -k overlays/staging or kubectl apply -k overlays/prod in that repository I get the following errors from the Kustomization-resource after the successful configuration of the GitRepository:

kustomize build failed: accumulating resources: accumulation err='accumulating resources from './overlays/prod': read /tmp/kustomization-912580309/overlays/prod: is a │ │ directory': recursed merging from path '/tmp/kustomization-912580309/overlays/prod': may not add resource with an already registered id: PriorityClass.v1.scheduling.k8s.io/high-priority.[noNs] (The high-priodity PriorityClass has already been created by previous manual apply calls that however never complained about this resource already existing)

This error comes from the fact that by default, flux just applies everything in the repo (you can change the branch tho). With all your overlays having the same name, this means they clash.
You could create a kustomization.yaml in the root to configure flux, but we recommend just using the whole repo and discourage the usage of any kustomizations.
If you want to use the same repo for prod/staging/dev/... we would recommend using branches, this would also allow for using a PR/MR based approach to transfer changes from dev all the way up to prod.

Directory Structure in that repository:

my-repo/
├─ base/
│  ├─ kustomization.yaml
│  ├─ ...
├─ overlays/
│  ├─ prod/
│  │  ├─ kustomization.yaml
│  │  ├─ ...
│  ├─ staging/
│  │  ├─ kustomization.yaml
│  │  ├─ ...

Any input on this?

Yes, the current version doesn't allow for subpaths, as that's only really useful for kustomize setups, which we highly discourage and recommend to just use plain k8s files, using flux HelmRelease for components of course.

Might I ask why you don't use normal Helm Charts instead of kustomize? Kustomize has lots of disadvantages in comparison to Helm and offers basically nothing in return 😅

Additionally, lots of the disadvantages only slowly creep in later when it's getting more complex, which makes it even harder to switch to Helm

If you still want to use kustomize with this setup we're happy to review a PR, we can discuss internally if we want this (which I doubt), for which you can open a ticket in our customer GitLab, or you could always "just" create the keys, gitrepositories and kustomizations yourself.

I know this sounds mean towards kustomize, but we've worked with kustomize for a couple of years and can only strongly recommend against using it 😕
And switching is easier the earlier you do it. 😉

[cwrau]

@tmechsner Has this answer satisfied your question? 😁