Updated workflow sweeper to use workflowOffsetTimeout for in progress tasks

[danmiller192]

Pull Request type

• Bugfix
• Feature
• Refactoring (no functional changes, no api changes)
• Build related changes
• WHOSUSING.md
• Other (please describe):

NOTE: Please remember to run ./gradlew spotlessApply to fix any format violations.

Changes in this PR

Preface: Currently when the sweeper processes a workflow and the task is in progress the sweeper picks up the task timeout and queues a message on the decider queue with this time to next process the workflow. This can have the effect of leaving workflows in limbo for long periods with their tasks not being scheduled until that timeout is reached.

This fix pr addresses this by changing unack method to just use the workflowOffsetTimeout. This means that by default the workflow will be reprocessed in 30 seconds rather than the default 60 mins.

Describe alternative implementation you have considered

I have looked at ways to push the workflow to the decider but due to the asynchronous nature this can on occasion fall out of step.

[v1r3n]

@danmiller192 the current approach ensures that the workflows are not sweeped unless they need to be. Changing this will have implications in the systems with large no. of running workflows.

[danmiller192]

@v1r3n totally agree, I think it would be better to avoid the extra load. I did test with some large workflows running about 600 workflows per min and it didn't seem to cause any performance issues. When running that volume without the fix I did see significant volume of tasks that would get stuck waiting. Overall I think I was intending this to be a workaround rather than a fix. I will try and see if I can find another way to address the issue.

[v1r3n]

@danmiller192 what would be the advantage of sweeping a workflow before the task timeout?

[danmiller192]

I have updated the proposal to use a config value MaxPostponeDurationSeconds to allow for tasks that may have a long and varied response time. Creating a new PR.