Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update status even if reconciliation fails #16

Merged
merged 1 commit into from
Nov 1, 2023
Merged

Conversation

sarataha
Copy link
Member

@sarataha sarataha commented Oct 26, 2023

PR to ensure status is updated even when the reconciliation fails. This is needed because users in the UI might be clicking on the sync button to try and "fix" broken resources, so we should ensure that the lastHandledReconcileAt is updated correctly.

A test case was added for when stubProvider's method ListClusters returns an error which causes the reconciliation to fail.

@sarataha sarataha requested a review from foot October 26, 2023 13:42
}

func (s *stubProvider) ListClusters(ctx context.Context) ([]*providers.ProviderCluster, error) {
return s.response, nil
return s.response, s.responseErr
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 👍

@@ -83,6 +83,9 @@ func (r *AutomatedClusterDiscoveryReconciler) Reconcile(ctx context.Context, req
// Set the value of the reconciliation request in status.
if v, ok := meta.ReconcileAnnotationValue(clusterDiscovery.GetAnnotations()); ok {
clusterDiscovery.Status.LastHandledReconcileAt = v
if err := r.Status().Update(ctx, clusterDiscovery); err != nil {
return ctrl.Result{}, err
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks okay to my eye.. I'm not sure if there are implications in potentially writing the status twice per reconcilation. cc @bigkevmcd

Ideally we want probably only want to write the status once. Which we could do by refactoring the code a bit here. Kevin you also mentioned you might be re-working the code here too, so in that case we could keep the code change in this PR small like this.

We also have r.patchStatus() we could use below, which refetches the ACD before updating. Not completely if we should do that here too.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use patchStatus, because as you say, it will reload the ACD before updating, otherwise you can get into conflict errors.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sarataha Can you use patchStatus for this?

@foot
Copy link
Collaborator

foot commented Oct 27, 2023

Nice! Lets also add a PR description!

  • When might we encounter a failure
  • Adds a test for this case
  • Why? As users in the UI might be clicking on the sync to try and "fix" broken resources etc.

@@ -565,13 +566,98 @@ func TestReconcilingWithAnnotationChange(t *testing.T) {
assert.Equal(t, aksCluster.Status.LastHandledReconcileAt, "testing")
}

func TestReconcilingWithAnnotationChangeWithError(t *testing.T) {
ctx := context.TODO()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to run this outside the TestAutomatedClusterDiscoveryReconciler sub-tests?

Starting the test env will take several seconds to start and stop, so if possible, we should reuse it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me! I added it to one of the subtasks already created :)

@@ -149,6 +149,22 @@ func TestAutomatedClusterDiscoveryReconciler(t *testing.T) {
}
assertHasOwnerReference(t, gitopsCluster, clusterRef)
assertHasOwnerReference(t, secret, clusterRef)

Copy link
Collaborator

@bigkevmcd bigkevmcd Nov 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't quite what I meant, can you split out this into an explicit test for failing to reconcile recording the status?

Rather than tacking it on to the end of the test here, it can be a sub-test, take your previous test, and create a new sub-test.

Copy link
Collaborator

@bigkevmcd bigkevmcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thanks!

@bigkevmcd bigkevmcd merged commit b19cac2 into main Nov 1, 2023
3 checks passed
@bigkevmcd bigkevmcd deleted the sync-error-flow branch November 1, 2023 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants