ZZ Spec Test Resources Support

This document describes how to add test resources support to the Micronaut Framework. Initially, the scope was to add Testcontainers support to Micronaut but conceptually, this document presents a more general approach to test resources.

Test resources support

Purpose

This document explores different options for improving support for Testcontainers in Micronaut. It compares existing solutions, both in Micronaut and other frameworks. When possible, it tries to abstract the solution so that it's not specific to testcontainers. In a nutshell, we want to reduce the friction of using a real database during development, or the cost of having to install external services in order to test the application.

Glossary

• Test resource: an external service, not directly implemented by the application, but required to execute tests. Typically, a database server, a kafka cluster, ...

Use cases

• As a Micronaut application developer, when I run tests, I want test resources to be automatically available

• As a Micronaut application developer, during application development (e.g ./gradlew -t run), I want to use test resources

Must do

• Solution must be supported by both Maven and Gradle.
• Test resources should be configurable (e.g the test database URL can be different from the production URL, test specific configuration must override production configuration, ...)
• Test resources not needed for a particular test shouldn’t be started

Nice to have

• Zero or minimal user configuration: without configuration, test resources should be inferred from the production requirements
• Non-intrusive: ideally, test shouldn’t have to be updated to use test resources: using an existing, working test suite which requires external services to be setup, it should be possible to use the same, untouched test suite with the new solution, spawning containers instead.
• Automatic test framework provisionning: if testcontainers is used, it should be done transparently, without having to edit build scripts/pom.xml
• Development mode: the behavior when running in development mode should be as close as possible to the one when executing tests
• Integration with Testcontainers Cloud
• Generalization of “test resources”: not simply test containers, but anything required for tests.

Must not do

The selected solution should not impact runtime dependencies. In particular, production (and deployment) code should fail if external resources are not available (only the "development mode" should support test resources). It's also important that the selected solution doesn't pollute the application classpath, which may be the case, for example, if testcontainers is transparently added to the user classpath.

Comparison of existing solutions

Micronaut

It’s already possible to use testcontainers in Micronaut. This requires manual setup, though, and only works in the "test" use case, not for "development mode". Every user has to implement the solution themselves. A typical solution involves:

• Adding testcontainers as a dependency in your build file
• Creating an application-test.yml file which sets up the database driver
• Add the @Testcontainers annotation to tests which require containers
• Add a static block to the tests setting up the appropriate containers

This approach has a number of drawbacks. In particular, it's fairly intrusive and requires updates to all tests (adding an annotation, adding a static init block, ...). In addition, it requires setting up the database driver (to use the test containers one instead of the "regular" JDBC driver. In addition, this solution simply doesn't work for "development mode”, meaning that it’s only available in tests. While this can be seen as fine, it's actually a strong limitation: because the goal of this feature is to streamline development and avoiding having to install potentially hard to setup or “polluting” external services, it's strange to have something which works when you run the tests, but not when you simply start the application.

Last but not least, it also requires users to know the GAV coordinates of test containers and potential dependencies it needs.

Spring Boot

Similarly to Micronaut, Spring Boot doesn’t provide built-in support for testcontainers. This article from Baeldung explains how to setup a test, making use of application context initializers. The approach differs from the one we explained in Micronaut in the sense that it uses the context initializer to expose variables which are required by production code (the DB URL, etc.). These properties are setup by reading a static test container value.

Another approach is to use the Spring Boot testcontainers library (not officially supported by Spring Boot), which basically only requires users to add a test dependency on a supported testcontainer support library. For example, add a dependency on the Spring Boot testcontainers Postgresql library. This library would take care of starting the test containers, and exposes variables which can in turn be used in a application-test.yml configuration file. The advantage of this technique is that the tests don't have to be updated, and setup is quite simple. The drawbacks are that it’s still not available in “dev mode” (when you run the application) and that it still requires a bit of configuration.

Quarkus

Quarkus provides the closest to what we want to support, the so-called "dev services", which provide test resources for test execution and development mode. Most of the dev services use testcontainers under the hood.

In terms of user experience, dev services are completely transparent to the user:

• Adding support for "quarkus-jdbc-postgresql" automatically adds support for running Postgresql in a test container
• No additional dependency is needed in build scripts/pom.xml
• No additional configuration is required: if, when running a test or the application in dev mode, the URL to a database is unset, then a test container is automatically started.

Extra configuration to test containers is allowed, if needed
Tests are supported, in addition to “development mode” (quarkus dev).

There are however a number of issues with this approach:

• it relies on test containers being added to the user classpath at runtime: this means that testcontainers and its dependencies "pollute" the classpath, which can lead to unexpected behavior, but also makes it difficult to support native image testing.
• the lifecycle of containers is bound to the lifecycle of the JVM, as this is the behavior of the testcontainers library.

Proposed solution

Micronaut test resources

The proposed solution is to add generic support for test resources. The lifecycle of test resources, however, would be driven by build tools (e.g. Gradle, Maven, etc.). The proposed solution will work independently for development (e.g using the ./gradlew -t run command), and for test (e.g. using the ./gradlew test command).

Conceptually, the proposed solution works similarly to Quarkus development mode: the trigger for starting a container (or a test resource in general) is a missing configuration property.

For example, given a Kafka configuration bean:

@ConfigurationProperties("kafka")
public class KafkaConfiguration {
    private List<String> bootstrapServers;

    public List<String> getBootstrapServers() {
        return bootstrapServers;
    }

    public void setBootstrapServers(List<String> bootstrapServers) {
        this.bootstrapServers = bootstrapServers;
    }
}

And a service using Kafka:

@Singleton
public class KafkaService {
    private final KafkaConfiguration kafkaConfiguration;

    public KafkaService(KafkaConfiguration kafkaConfiguration) {
        this.kafkaConfiguration = kafkaConfiguration;
    }

    //...
}

Then when the application is started in development mode (-t in Gradle, mn:run in Maven) and that the Kafka configuration is missing (e.g from the application.yml configuration file), when the KafkaConfiguration bean is read, we would:

• start a test container for Kafka
• inject the "boostrap servers" configuration value from the test container into the started Kafka service

One the application is stopped (or test execution is finished), the test containers would be stopped.

Implementation proposal

General idea

An experimental implementation of this design document lives here.

Conceptually, the implementation relies on 2 features of Micronaut:

• a PropertySourceLoader, which is responsible for listing properties that a test resource resolver can resolve
• a PropertyExpressionResolver (new in Micronaut 3.5), which is capable of resolving a property expression (e.g ${some.property}) into an actual value

For example, the property source loader will declare the following properties, with the corresponding values:

kafka
  bootstrap:
    servers: ${auto.test.resources.kafka.bootstrap.servers}

Then the property expression resolver is responsible for:

• spawning a test container for Kafka
• returning the value of the kafka.bootstrap.servers property

The level of indirection, auto.test.resources.kafka.boostrap.servers which delegates to the property expression resolver is here so that we can lazily resolve values and, for example, only spawn containers on demand, when a bean which needs them is created: it is the resolution of that property which, as a side effect, may spawn a test container.

Test resource resolvers

While PropertySourceLoader and PropertyExpressionResolver are the building bricks of this feature, we need a higher level abstraction to support the actual test resources:

A TestResourceResolver is responsible for providing a value for a missing property. For example, a TestResourceResolver may be able to provide a value for the kafka.bootstrap.servers property. For this, a TestResourceResolver needs to:

• describe the properties it is able to resolve
• describe the properties it needs to be able to resolve
• describe the property entries it may need to be able to list the properties it can resolve

As such, implementing support for spawning a Kafka container requires implementing a KafkaTestResourceResolver which provides all those requirements.

The simplest case is a test resource resolver which doesn't need any other property. The Kafka test resource resolver is such an example: the only thing it will do is resolving the kafka.bootstrap.servers property. Therefore, it will need to provide the following properties:

• resolvable properties: kafka.bootstrap.servers
• required properties: none
• required property entries: none

In practice, it may be a good idea to provide configuration of the test containers. So the Kafka test resource resolver may actually "require" properties to, say, override the default Docker image name:

• resolvable properties: kafka.bootstrap.servers
• required properties: micronaut.testresources.kafka.image-name
• required property entries: none

In which case, when resolving the kafka.bootstrap.servers property, Micronaut will provide the value of the micronaut.testresources.kafka.image-name property.

A more complex test resource resolver is one which would create a test database. In practice, such a resolver is able to resolve multiple properties (which can be queried by Micronaut in any order, in which case any of the properties must return the same test resource), and must know what kind of database to start. More interestingly, such a resolver must be able to resolve multiple datasources.

In the experimental implementation, we provide a JdbcTestResourceResolver for that purpose. The first thing it declares is the list of required property entries: this time it's not none, since it needs to know about the possible datasource names. Therefore, the resolver declares:

• required property entries: datasources

Then when Micronaut asks the resolver what properties it can resolve, it will pass in the list of datasource names, so that the resolver can answer:

• resolvable properties: datasources.default.url, datasources.default.username, datasources.default.password

But if the application declares multiple datasources, say for example that there's a default and a users datasource, then the resolvable properties will become:

• resolvable properties: datasources.default.url, datasources.default.username, datasources.default.password, datasources.users.url, datasources.users.username, datasources.users.password

This technique makes it particularly convenient to spawn different containers for the different datasources, without requiring any user configuration!

Then the resolver needs to declare the properties it needs to be able to determine what database to create. For this, it declares:

• required properties: datasources.default.dialect, datasources.default.driverClassName, datasources.users.dialect, datasources.users.driverClassName

So when the resolve method is called, the resolver can determine which database to use.

Build plugins

A key aspect of the implementation is to avoid having to inject test containers in the application classpath (even for tests). Similarly, we'd like to avoid having to modify existing test suites to add support for running test containers.

It is also important so that we can build a native image of the test suite, and run it against test containers like in the JVM mode.

Therefore, the implementation relies on build plugins to drive the lifecycle of test resources.

The pre-requisite is to run a build in "development mode", which is either running tests or running the application in continuous mode (Gradle) or mn:run (Maven).

The build plugins would inject a PropertyExpressionResolver into the application classpath. That PropertyExpressionResolver would be responsible for spawning the test containers. However, as we explained earlier, we don't want to pollute the application classpath, so this resolver must be limited to the bare minimum.

Test resources property expression resolver

As a consequence, the resolver injected by the build plugin would be limited to the following:

• communicate with an isolated process, triggered by the build plugins, which is in turn responsible for managing the lifecycle of test resources
• fetch information from the server to determine the value of a property expression

sequenceDiagram
    participant app as User code
    participant resolver as Property resolver
    participant proxy as Test resources proxy
    participant server as Test resources server
    participant tc as Testcontainers handler
    app ->> resolver: reads "kafka.bootstrap.servers"
    resolver ->> proxy: requests for value of "kafka.boostrap.servers"
    proxy ->> server: requests test resource
    server ->> tc: starts a test container for Kafka
    Note right of server: No server started for kafka.bootstrap.servers<br>Looks for plugins capable of feeding that property
    tc -->> server: metadata about containers
    server ->> proxy: sends back information about "kafka.bootstrap.servers"
    proxy ->> resolver: sends back information about "kafka.bootstrap.servers"
    resolver ->> app: unblocks the application

Loading

In a nutshell, the resolver is simply a proxy to an external process which is the real resolver. This makes it possible to really isolate application classpath from the test classpath, at the cost of a small, minimal library: the server itself would contain all the required dependencies needed to expose test resources (in particular, testcontainers modules). Because it would run in a separate process, the test containers classpath would be invisible to user code, and building native images which communicate with the server would be possible.

Test resources lifecycle

In the simplest case, the application is started in development mode, which means that there's a single process for the application. Therefore, the build plugins can:

1. start the test resources server
2. inject the proxy into the application classpath
3. start the application
4. wait until the application is shutdown
5. stop the test resources server (which triggers shutting down test containers)

However, we must take into account more complex setups. In particular, during execution of tests, it is possible that multiple processes want to access the same test resources. This is possible because tests are typically executed in forked VMs, every X test. It is also possible that tests are executed in parallel. Therefore, we must consider the behavior of the test resources server whenever concurrent requests happen, and when to shutdown test resources.

The simplest, for the initial implementation, is to share the containers independently of what is requesting them. This means that if 2 processes request the value of the kafka.boostrapservers property concurrently, in the example above, then we must make sure that only a single container is spawned. The 2 processes would share the same test container.

In practice, this might not be a reasonable behavior: if the container provides for example a database, having different tests writing to the same database concurrently might lead to flaky tests. Therefore, another option is to spawn test resources per client. We would need a way to identify the client in some form. Test resources lifecycle would be bound to the client lifecycle: they would be spawned on demand, but shutdown when the client goes away (or with a timeout).

Multiple property handling

In the process above, we described the process of spawning containers as a side effect of requesting the value of a property. This means, in practice, that a particular handler is capable of answering the question "what is the value of kafka.bootstrapservers?", and as part of the process, it would spawn a server before answering.

However, how do we know that when we read the value kafka.other.property, the server which should be returned is the same as when we read kafka.bootstrapservers?

One option is to be stateful, and whenever any of the values that the handler is capable of dealing with is requested, it would spawn a server, but we would return a complete map of properties, independently of what has been requested. Then when the 2d property is read, we already have the value in memory so we can return it without spawning a server.

Alternatively, we can say that only a single missing property can trigger the server to be spawned. The problem with this approach is that there may be cases where the values of the properties cannot be determined independently of each other.

Implementation considerations

This section simply lists several implementation ideas to take into consideration.

• Security: the proxy should only accept connections from localhost and accept connections from clients using a generated token at application startup
• Use without proxy: ideally, the "server" components should be loadable directly in a Micronaut app, in case folks don't want to rely on a proxy. The lifecycle can be handled by PropertyExpressionResolver via AutoCloseable.