Fix the job jitter/delay around sidekiq retries

[gitstart-app]

This PR was created by GitStart to address the requirements from this ticket: TRAB-687.

---

What this PR achieves?

• Adds retryable_job.rb
• Add test suites for retryable_job_spec.rb module
• Update find_build_job.rb
• Update find_build_job_spec.rb by adding more tests and leaving the previous ones unchanged.
• Migrate all other jobs and tests with siddekiq_retry_in , sidekiq_entries_exhausted and sidekiq_retry_in_block

[gitstart-tramline]

Hi @kitallis

We’d like your guidance on testing this PR. Could you help validate the scenario provided in the attached screenshot, or are there any additional cases we should consider to ensure the fix is thoroughly tested?

Additionally, are there specific benchmarks or indicators we should monitor to confirm the changes are working as intended?

[kitallis]

Hi @kitallis

We’d like your guidance on testing this PR. Could you help validate the scenario provided in the attached screenshot, or are there any additional cases we should consider to ensure the fix is thoroughly tested?

Additionally, are there specific benchmarks or indicators we should monitor to confirm the changes are working as intended?

I think this cannot be tested reliably on a unit-level. We'll have to do some sort of real-time test. What we can do is issue 100s of retries for a particular job, perhaps StoreRollouts::AppStore::FindLiveReleaseJob since that has seen the most issues due to jitter and ensure that it runs on the correct ticks of time, and the variability error under controlled circumstances (like a local env), with the correct backoffs set, is not off by more than a 1 minute or so.

[kitallis]

Separately, I think the idea for this change is generally fine, but the implementation seems too complicated. If this is your final approach, we should stop and re-evaluate because I think there are issues with it that won't scale.