Configurable timeout for argocd-update step

[razvan-agape]

Checklist

• I've searched the issue queue to verify this is not a duplicate feature request.
• I've pasted the output of kargo version, if applicable.
• I've pasted logs, if applicable.

Proposed Feature

The default timeout for argocd-update operation is 5 minutes.
Kargo version: 1.2.0.

Motivation

In some scenarios, an application sync may take longer that, which will mark the step as failed, although, eventually, the sync might succeed.

Suggested Implementation

It would be useful to be able to configure the timeout, or have a configurable number of retries.

[krancour]

https://docs.kargo.io/user-guide/reference-docs/promotion-templates#step-retries

[lknite]

@razvan-agape is this working for you? I tried the following and it doesn't seem to have any effect:

      - uses: argocd-update
        retry:
          errorThreshold: 1
          timeout: 2m0s

kargo v1.3.1

[krancour]

@lknite, the timeouts are not exact because steps do not continuously retry internally. If the timeout hasn't elapsed, a step that's still running (waiting on something external) is retried on the next reconciliation attempt.

In general, those attempts are every five minutes, but the next attempt can be sooner if a related resource has a state change that forces the Promotion back onto the queue. It can also be later depending on the depth of the queue.

In your case, setting the timeout to 2m, may have no practical effect in the average case where the next reconciliation attempt is made (roughly) five minutes later. It worked well for @razvan-agape because he was raising the limit.

We're a little bit at the mercy of the controller runtime here since we don't have precise control over the interval before the next reconciliation...

That said, we can probably get closer to the specified timeout by shortening the requeue interval when timeout is sooner than when the next reconciliation attempt would typically be.

I will write up a separate issue for this as soon as I've got a chance.

[krancour]

@lknite, I opened #3663