Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(webhook): make app refresh operation concurrent (#14269) #15326

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

phanama
Copy link
Contributor

@phanama phanama commented Sep 2, 2023

The current webhook handler processes all applications sequentially, with each iteration potentially calling the network-bound argo.RefreshApp() which calls k8s API to patch a refresh annotation. This makes webhook slow in scenarios involving hundreds of apps.

This PR introduces concurrent application processing to webhook HandleEvent(). We also expose a new configurable setting in ArgoCD CMwebhook.maxConcurrentAppRefresh to be able to control the degree of concurrency.

Fixes #14269

Checklist:

  • Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
  • The title of the PR states what changed and the related issues number (used for the release note).
  • The title of the PR conforms to the Toolchain Guide
  • I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
    * [ ] I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
  • Does this PR require documentation updates?
  • I've updated documentation as required by this PR.
  • Optional. My organization is added to USERS.md.
  • I have signed off all my commits as required by DCO
  • I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
  • My build is green (troubleshooting builds).
  • My new feature complies with the feature status guidelines.
  • I have added a brief description of why this PR is necessary and/or what this PR solves.

@codecov
Copy link

codecov bot commented Sep 2, 2023

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (99a4bf0) 49.67% compared to head (a753b47) 49.76%.
Report is 5 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #15326      +/-   ##
==========================================
+ Coverage   49.67%   49.76%   +0.09%     
==========================================
  Files         267      267              
  Lines       46383    46442      +59     
==========================================
+ Hits        23039    23112      +73     
+ Misses      21084    21068      -16     
- Partials     2260     2262       +2     
Files Coverage Δ
util/helm/cmd.go 35.21% <100.00%> (+1.14%) ⬆️
...cationset/controllers/applicationset_controller.go 60.39% <0.00%> (-0.11%) ⬇️
util/settings/settings.go 51.90% <76.92%> (+1.79%) ⬆️
cmd/argocd/commands/project_role.go 0.00% <0.00%> (ø)
pkg/apiclient/grpcproxy.go 0.00% <0.00%> (ø)
util/webhook/webhook.go 74.01% <87.71%> (+5.35%) ⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@phanama phanama force-pushed the webhook-concurrency branch from bfae177 to 5935fac Compare September 2, 2023 14:13
@phanama phanama force-pushed the webhook-concurrency branch from 58d24f7 to 507530c Compare September 3, 2023 15:41
@phanama phanama force-pushed the webhook-concurrency branch from b01e6a7 to 84ecbb3 Compare September 3, 2023 15:50
@phanama phanama force-pushed the webhook-concurrency branch from cb6b011 to 277c83f Compare September 10, 2023 06:16
if !appFilesHaveChanged(&app, changedFiles) {
// we cannot retrieve previous and next commit SHA for bitbucket.
// don't attempt to load/create cache for empty strings instead of SHA
if change.shaBefore == "" && change.shaAfter == "" {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@phanama phanama marked this pull request as ready for review September 10, 2023 07:51
@phanama phanama requested review from a team as code owners October 17, 2023 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Too many applications causing webhook to timeout
1 participant