controller: try to make step timeout somewhat more accurate

[krancour]

Numerous discussions have taken place over the last few months re: configurability of the retry interval for step retries and timeouts, possibly not being honored.

I'll borrow my own recent explanation for the difficulty behind this from #3515:

the timeouts are not exact because steps do not continuously retry internally. If the timeout hasn't elapsed, a step that's still running (waiting on something external) is retried on the next reconciliation attempt.

In general, those attempts are every five minutes, but the next attempt can be sooner if a related resource has a state change that forces the Promotion back onto the queue. It can also be later depending on the depth of the queue.

...

We're a little bit at the mercy of the controller runtime here since we don't have precise control over the interval before the next reconciliation...

That said, we can probably get closer to the specified timeout by shortening the requeue interval when timeout is sooner than when the next reconciliation attempt would typically be.

If someone wishes to tackle this, it should only be a few lines of code right around here.

We can keep the hard coded five minutes, but shorten it by the appropriate amount if the running step's start time + timeout is sooner than now + five minutes.

Note: At this time, I'm still hesitant to apply a similar strategy for making retry intervals configurable because I think our inability to offer much precision there is more problematic than it is with timeouts. Let's, at least, consider it out of scope for this issue.