feat(argo-cd): Add support for application controller dynamic cluster distribution.

[oscrx]

Resolves: #2537

Let me know if anyone has questions or suggestions.

Relevant issues and PR's:

• feat: dynamic rebalancing of clusters across shards argo-cd#15036
• feat(argo-cd): Replace Application Controller StatefullSet with a Deployment and enabled autoscaling #1894
• feat(argo-cd): Upgrade Argo CD to 2.9.0 #2318
• Introduce support for dynamic cluster sharding #2537
• StatefulSet.apps "argocd-application-controller" is invalid: spec: Forbidden: updates to statefulset #2494

Thanks! @ishitasequeira

Removed env:

env:
  - name: ARGOCD_CONTROLLER_REPLICAS
    value: "1"

Added env's:

env:
  - name: ARGOCD_ENABLE_DYNAMIC_CLUSTER_DISTRIBUTION
    value: "true"
  - name: ARGOCD_CONTROLLER_HEARTBEAT_TIME
    value: "10"

Checklist:

• I have bumped the chart version according to versioning
• I have updated the documentation according to documentation
• I have updated the chart changelog with all the changes that come with this pull request according to changelog.
• Any new values are backwards compatible and/or have sensible default.
• I have signed off all my commits as required by DCO.
• My build is green (troubleshooting builds).

[pdrastil]

@oscrx Great work. Can you also please run scripts/helm-docs.sh in the root of directory and commit README.md change?

[oscrx]

I think I already did that, but after removing the trailing space in Chart.yaml will do it again :)

Also some helpful test/debug commands for anyone that's interested in reviewing:

helm dep update
helm template test . > before.yaml 
helm template test . --set controller.dynamicClusterDistribution=true > after.yaml
diff before.yaml after.yaml
# or in your favorite editor of course

[pdrastil]

This is exactly why I asked for second drop-in implementation so it's easy to sync changes with upstream on 2.11 :)

[oscrx]

After running the docs generator again the space seems the only difference. I love the generator btw, never used it before.

[pdrastil]

One more ask. We should probably create smoke test for this in .ci folder just to see that Argo starts up if different implementation is enabled.

[pdrastil]

LGTM. Thanks again

[oscrx]

I don't think the pipeline is detecting charts/argo-cd/ci/dynamic-sharding.yaml as a change..
Did I forget something?

[mkilchhofer]

I don't think the pipeline is detecting charts/argo-cd/ci/dynamic-sharding.yaml as a change..

Did I forget something?

Filename needs to end in ...-values.yaml :)

[oscrx]

Tried testing it again to see if it's flaky.

The error is a bit vague:

time="2024-02-21T23:44:19Z" level=fatal msg="(dymanic cluster distribution) failed to get app controller deployment: deployments.apps "argocd-application-controller" not found"

It's running fine in my cluster.

Startup logs

```text time="2024-02-22T00:44:29Z" level=info msg="ArgoCD Application Controller is starting" built="2024-02-14T17:37:43Z" commit=a79e0eaca415461dc36615470cecc25d6d38cefb namespace=argocd version=v2.10.1+a79e0ea time="2024-02-22T00:44:29Z" level=info msg="Processing all cluster shards" time="2024-02-22T00:44:29Z" level=info msg="Processing all cluster shards" time="2024-02-22T00:44:29Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s, appResyncJitter=0s" time="2024-02-22T00:44:29Z" level=info msg="Starting configmap/secret informers" time="2024-02-22T00:44:29Z" level=info msg="Configmap/secret informer synced" time="2024-02-22T00:44:29Z" level=warning msg="The cluster https://kubernetes.default.svc has no assigned shard." time="2024-02-22T00:44:29Z" level=warning msg="The cluster https://kubernetes.default.svc has no assigned shard." time="2024-02-22T00:44:29Z" level=info msg="Cluster https://kubernetes.default.svc has been assigned to shard 0" time="2024-02-22T00:44:29Z" level=info msg="Starting secretInformer forcluster" time="2024-02-22T00:44:29Z" level=info msg="Start syncing cluster" server="https://kubernetes.default.svc" time="2024-02-22T00:44:29Z" level=info msg="0xc000ca59e0 subscribed to settings updates" time="2024-02-22T00:44:32Z" level=info msg="Cluster successfully synced" server="https://kubernetes.default.svc" ```

The new configmap also gets refreshed every 10 seconds

apiVersion: v1
data:
  shardControllerMapping: >-
    [{"ShardNumber":0,"ControllerName":"argocd-application-controller-5f656dbc96-qlq2s","HeartbeatTime":"2024-02-22T00:51:27Z"}]
kind: ConfigMap
metadata:
  creationTimestamp: '2024-02-22T00:14:03Z'
  name: argocd-app-controller-shard-cm
  namespace: argocd
  resourceVersion: '133966466'
  uid: 4c1e03da-2c5a-48e3-9c56-e5f2508dd8d6

[oscrx]

Even 2 typo's in the upstream repo :)
https://github.com/argoproj/argo-cd/blob/6aa79f283cde2b1701c755261c8408c01c7b7566/controller/sharding/sharding.go#L377

[oscrx]

Oh I figured it out, it's getting late :)

It's because the name of the helm chart is dynamic during the smoketest, the code/container gets it's own name from an environment variable: ARGOCD_APPLICATION_CONTROLLER_NAME
I added it.

It might be needed for other components in this helm chart as well..
https://argo-cd.readthedocs.io/en/stable/user-guide/environment-variables/
Let me know what you guys think, I can include it in this PR.

Also checked where it is getting the number of replica's. That code is dynamic though.
https://github.com/argoproj/argo-cd/blob/6aa79f283cde2b1701c755261c8408c01c7b7566/controller/sharding/sharding.go#L373

[oscrx]

Any thoughts about the name variable?
We could also merge it as-is

[oscrx]

bumped and rebased

[oscrx]

I added the previously discussed variable names into a separate commit for easier review, let me know what you guys think and if we should configure this.

[yu-croco]

Thank you for your contribution ! LGTM