Repeaters, v.2: Sleep 0.2s if all locked

[kaapstorm]

Technical Summary

This past weekend I was (over?)thinking about how PR #35033 loops over repeaters, and I came up with two alternatives. This "version 2" is the first. (See PR #35687 for version 3.)

Comparison

	PR #35033	Version 2	Version 3
Finds new repeat records every	5 minutes	1 minute	1 minute
Queries for all repeaters every	5 minutes	After 200ms	As soon as needed
Rank in order of simplicity	1	2	3
Rank in order of throughput	3	2	1

The approach used in PR #35033 is relatively simple, and fast at sending repeat records of repeaters that are already being processed. But new repeat records of repeaters that are not being processed can take up to 5 minutes to be discovered and processed.

In comparison, this version, version 2, checks every minute for repeaters with repeat records ready to be sent (and if it is busy processing repeaters, it will find new repeat records sooner than that).

If all repeaters are locked, it will wait 200ms for a repeater to finish processing, and then query for repeaters again. This approach is simpler than the approach taken in version 3, but it can result in queries returning all repeaters locked. It is this unnecessary churn on Postgres that prompted version 3.

Here is what version 2 looks like in Datadog:

Review commits together 🏢

Feature Flag

PROCESS_REPEATERS

Safety Assurance

Safety story

• Tested on Staging
• Not yet in use on Production

Automated test coverage

Tests included

Rollback instructions

• This PR can be reverted after deploy with no further considerations

Labels & Review

• Risk label is set correctly
• The set of people pinged as reviewers is appropriate for the level of risk of the change

[kaapstorm]

I'm not sure that the increase in performance makes up for the increase in complexity. Closing.