ApplicationSet support in app-of-apps

[hairmare]

Is your feature request related to a problem? Please describe.

I've been hearing more and more feedback from downstreams that would like to get rid of having a folder per env in their argocd-configs repo. The questions i'm fielding ask things like "can't we have a branch per env and manage those with PRs in between?" or "can't you deploy the same thing to multiple envs from one definition". This issue want's to provide an alternate way of using our app-of-apps charts that addresses these issues..

Describe the solution you'd like

I want to be able to use our app-of-apps charts to create ApplicationSets instead of Applications. There would be one ApplicationSet for each current Application and you could configure ApplicationSets in various ways. I think this would support a bunch of interesting patterns but we should explore the space more while implementing this so we can also add a tried and tested example of a how we suggest you use ApplicationsSets.

In a best case scenario we'd want to aim at making it possible for a downstream to only use one main branch with only the minimal changeset needed for each env being in a file on it's own.

Describe alternatives you've considered

I have seen folx use multiple branches and pr between them but don't like that solution too much since it adds some easily forgotten git complexity. The original solution already didn't do that because of this exact reason, a multi branch architecture is a nice solution but esp. when working with devops that have an ops background and don't git in their sleep i've found it to be too much of a burden on my peers. This is further impacted by the fact that i'm usually working with people to get them onboarded and proficient in Kubernetes/Helm/Argo CD, adding Git to the already huge plate of things would be a rather large side-quest. Pretty much everyone understands folders which makes it easier to get started.

I also considered some kind of kustomize based solution and even a Helm+kustomize custom tool. The native kustomize solution would not support Helm and as such be a large bc-breaking change towards how the current values.yaml are structured. The custom tool route might be workable but most set ups have been quite confusing to grok. Again, very nice solutions but not the most transparent if you want to figure out how they actually work. This is issue isn't about saying no to kustomize, it's about prioritising ApplicationSet support since that can work nicely with our current set of values and can probably be done w/o breaking bc.

Affected chart

All app-of-apps charts.

Additional context

A lot of the leg-work for this would probably be in documentation. I'm also thinking that we should maybe create an open argocd-configs repo that follows this example as a demo here on github as part of this issue.

If we do ApplicationSet support well, i could see it being our new default deployment. I'm leaning towards keeping Application support for the long run because they will still work perfectly fine for smaller cases and proof of concept installation. Maybe I'm wrong and we should replace Applications with ApplicationSets before reaching v1 to have less options.