feat: dynamic rebalancing of clusters across shards

[ishitasequeira]

This PR is an implementation of the following proposal:
https://github.com/argoproj/argo-cd/blob/master/docs/proposals/rebalancing-clusters-across-shards-dynamically.md

Note: As implemented in this PR, the new dynamic rebalancing feature is enabled by migrating the Argo CD Application controller from StatefulSet to a Deployment. Migration instructions are provided as part of the docs. This is an alpha feature and in the future, we may choose to return to using a StatefulSet instead.

Note on DCO:

If the DCO action in the integration test fails, one or more of your commits are not signed off. Please click on the Details link next to the DCO action for instructions on how to resolve this.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.

Please see Contribution FAQs if you have questions about your pull-request.

[codecov]

Codecov Report

Patch coverage: 59.88% and project coverage change: +0.06% 🎉

Comparison is base (1e490d9) 50.00% compared to head (b119743) 50.06%.
Report is 7 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #15036      +/-   ##
==========================================
+ Coverage   50.00%   50.06%   +0.06%     
==========================================
  Files         266      266              
  Lines       45631    45834     +203     
==========================================
+ Hits        22818    22948     +130     
- Misses      20581    20651      +70     
- Partials     2232     2235       +3     

Files Changed	Coverage Δ
controller/metrics/metrics.go	78.60% <0.00%> (ø)
controller/appcontroller.go	54.18% <27.27%> (+0.04%)	⬆️
controller/sharding/sharding.go	75.43% <63.70%> (-17.64%)	⬇️
util/db/db.go	86.66% <100.00%> (+2.88%)	⬆️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[leoluz]

Thank you Ishita! This is going to be an awesome feature!
Apart from my comments there are few things pending in this PR:

• Documentation
• Tests (lots of them) :)

[ishitasequeira]

Thanks for the detailed review @leoluz !!

I am working on the documentation and adding the quality unit tests to the PR.

[ishitasequeira]

I was thinking of making a global variable for this but not configurable. This retryCount represents the number of times we retry in every heartbeat. IMO, it would retry for retryCount times in the next heartbeat of Readiness probe check even if it fails 3 times here. WDYT?

[ishitasequeira]

That sounds good. I will update it.

[ishitasequeira]

As I mentioned in other comment, we do not have a Deployment Informer today. I can check if I can add one.

I think we would not want to cache the deployments but instead get the latest deployment always to make sure we fetch the correct replica count.

[ishitasequeira]

@leoluz addressed your comments. PTAL.

[leoluz]

I would be great to have something in the docs describing how the dynamic shard rebalance works. Things like:

• How to enable it?
• What is the expected behaviour?
• What are the different methods of rebalancing shards implemented in Argo CD today?
• What is the guideline to switch the controller from StatefulSet to Deployment?

[ishitasequeira]

• How to enable it?
  It would be enabled by default when users upgrade to v2.9, as it will deploy application controller as a deployment instead of a statefulset.

What is the expected behaviour?
I added some detail around it.

What are the different methods of rebalancing shards implemented in Argo CD today?
I think this would be not needed as we already have sharding documented under the argocd-application-controller section here

What is the guideline to switch the controller from StatefulSet to Deployment?
IMO, it would be more useful to have this as part of upgrade notes instead of having it here. WDYT?

[leoluz]

Docs are looking awesome. Thank you Ishita!
Added few final comments in the docs.

[leoluz]

Found one small typo in the docs and asked one more question.

[leoluz]

LGTM. Thank you for this feature Ishita!

Note: There is still an open discussion to whether or not moving the Argo CD Controller to Deployment is the direction we should go. It would cause unnecessary burden in our user base by asking all Argo CD users to do migrations if not justified. If the same feature can be implemented keeping the current controller as StatefulSet it would avoid unnecessary migrations unless there are good and solid reasons to move to Deployment.