Enable schedule rules by default in system tests

[patchwork01]

Background

Split from:

• Parallelise system tests #1794

A system test instance is currently paused by default, so the scheduled rules never invoke partition splitting. We invoke it explicitly, but this happens for the whole instance.

Description

Instead of explicitly invoking lambdas in system tests, we'd like to enable all the schedule rules by default. In the tests when we wait for an operation that is triggered this way, we can wait for the scheduled rule to run.

This should be more representative of how the system works in reality, and it should make the tests easier to parallelise.

Analysis

We can ensure that schedule rules are enabled when we connect to an instance in DeployedSleeperInstances.createInstanceIfMissing. We can avoid ever pausing the schedule rules in any tests.

There is currently code for working with schedule rules in SnapshotsDriver. We could move that to a separate driver for schedule rules. Note that the code to disable the snapshot rule there is not currently used, so it could be deleted.

Note that all instances are deployed paused in AwsSleeperInstanceDriver.deployInstanceIfNotPresent.

We can handle this separately for each type of rule, and perform the above changes gradually:

• Enable ingest task creation by default in system tests #3726
• Enable compaction task creation by default in system tests #3724
• Convert garbage collection system tests to invoke per table #3706
• Convert metrics generation system tests to invoke per table #3729
• Convert system tests to invoke compaction job creation per table #3725
• Convert system tests to invoke partition splitting per table #3723
• Enable ingest batcher job creation schedule in system tests #3727

Once all of these are finished, we can ensure we deploy an instance unpaused, and check that every scheduled rule is enabled when we connect to an instance for the first time in a test suite.

[patchwork01]

Closing as all the child issues are done.