Thread safety when connecting to instances in system tests

[patchwork01]

Background

Split from:

• Parallelise system tests #1794

In the JUnit-based system test suite, each test starts by connecting to an instance of Sleeper. The first time an instance is connected to in a test suite, it may not exist in which case it will be deployed, or if it already exists, it may be updated if necessary. Further tests connecting to the instance will not cause this.

Description

We'd like to ensure thread safety when connecting to an instance of Sleeper in a system test.

This should remove a restriction preventing us from running multiple system tests in parallel.

Analysis

We could have one thread that's responsible for interacting with the current state of which instances have been connected to and which are currently being deployed.

When a test connects to an instance, it can make an asynchronous call to that thread via a single-threaded ExecutorService. Once in the one thread, this call can check whether the state of the instance is known. If not, it can kick off a deployment and subscribe the test to the deployment. If it's deploying, it can subscribe the test to the deployment in progress. If it's already deployed, it can return immediately.

We could have a separate thread pool for deploying instances, so a number of instances can be deployed at the same time.