-
Notifications
You must be signed in to change notification settings - Fork 1.1k
ZZ Spec Test Resources Support
This document describes how to add test resources support to the Micronaut Framework. Initially, the scope was to add Testcontainers support to Micronaut but conceptually, this document presents a more general approach to test resources.
This document explores different options for improving support for Testcontainers in Micronaut. It compares existing solutions, both in Micronaut and other frameworks. When possible, it tries to abstract the solution so that it's not specific to testcontainers. In a nutshell, we want to reduce the friction of using a real database during development, or the cost of having to install external services in order to test the application.
- Test resource: an external service, not directly implemented by the application, but required to execute tests. Typically, a database server, a kafka cluster, ...
-
As a Micronaut application developer, when I run tests, I want test resources to be automatically available
-
As a Micronaut application developer, during application development (e.g
./gradlew -t run), I want to use test resources
- Solution must be supported by both Maven and Gradle.
- Test resources should be configurable (e.g the test database URL can be different from the production URL, test specific configuration must override production configuration, ...)
- Test resources not needed for a particular test shouldn’t be started
- Zero or minimal user configuration: without configuration, test resources should be inferred from the production requirements
- Non-intrusive: ideally, test shouldn’t have to be updated to use test resources: using an existing, working test suite which requires external services to be setup, it should be possible to use the same, untouched test suite with the new solution, spawning containers instead.
- Automatic test framework provisionning: if testcontainers is used, it should be done transparently, without having to edit build scripts/pom.xml
- Development mode: the behavior when running in development mode should be as close as possible to the one when executing tests
- Integration with Testcontainers Cloud
- Generalization of “test resources”: not simply test containers, but anything required for tests.
The selected solution should not impact runtime dependencies. In particular, production (and deployment) code should fail if external resources are not available (only the "development mode" should support test resources). It's also important that the selected solution doesn't pollute the application classpath, which may be the case, for example, if testcontainers is transparently added to the user classpath.
It’s already possible to use testcontainers in Micronaut. This requires manual setup, though, and only works in the "test" use case, not for "development mode". Every user has to implement the solution themselves. A typical solution involves:
- Adding testcontainers as a dependency in your build file
- Creating an application-test.yml file which sets up the database driver
- Add the
@Testcontainersannotation to tests which require containers - Add a
staticblock to the tests setting up the appropriate containers
This approach has a number of drawbacks. In particular, it's fairly intrusive and requires updates to all tests (adding an annotation, adding a static init block, ...). In addition, it requires setting up the database driver (to use the test containers one instead of the "regular" JDBC driver. In addition, this solution simply doesn't work for "development mode”, meaning that it’s only available in tests. While this can be seen as fine, it's actually a strong limitation: because the goal of this feature is to streamline development and avoiding having to install potentially hard to setup or “polluting” external services, it's strange to have something which works when you run the tests, but not when you simply start the application.
Last but not least, it also requires users to know the GAV coordinates of test containers and potential dependencies it needs.
Similarly to Micronaut, Spring Boot doesn’t provide built-in support for testcontainers. This article from Baeldung explains how to setup a test, making use of application context initializers. The approach differs from the one we explained in Micronaut in the sense that it uses the context initializer to expose variables which are required by production code (the DB URL, etc.). These properties are setup by reading a static test container value.
Another approach is to use the Spring Boot testcontainers library (not officially supported by Spring Boot), which basically only requires users to add a test dependency on a supported testcontainer support library. For example, add a dependency on the Spring Boot testcontainers Postgresql library. This library would take care of starting the test containers, and exposes variables which can in turn be used in a application-test.yml configuration file. The advantage of this technique is that the tests don't have to be updated, and setup is quite simple. The drawbacks are that it’s still not available in “dev mode” (when you run the application) and that it still requires a bit of configuration.
Quarkus provides the closest to what we want to support, the so-called "dev services", which provide test resources for test execution and development mode. Most of the dev services use testcontainers under the hood.
In terms of user experience, dev services are completely transparent to the user:
- Adding support for "quarkus-jdbc-postgresql" automatically adds support for running Postgresql in a test container
- No additional dependency is needed in build scripts/pom.xml
- No additional configuration is required: if, when running a test or the application in dev mode, the URL to a database is unset, then a test container is automatically started.
Extra configuration to test containers is allowed, if needed
Tests are supported, in addition to “development mode” (quarkus dev).
There are however a number of issues with this approach:
- it relies on test containers being added to the user classpath at runtime: this means that testcontainers and its dependencies "pollute" the classpath, which can lead to unexpected behavior, but also makes it difficult to support native image testing.
- the lifecycle of containers is bound to the lifecycle of the JVM, as this is the behavior of the testcontainers library.
The proposed solution is to add generic support for test resources.
The lifecycle of test resources, however, would be driven by build tools (e.g. Gradle, Maven, etc.).
The proposed solution will work independently for development (e.g using the ./gradlew -t run command), and for test (e.g. using the ./gradlew test command).
Conceptually, the proposed solution works similarly to Quarkus development mode: the trigger for starting a container (or a test resource in general) is a missing configuration property.
For example, given a Kafka configuration bean:
@ConfigurationProperties("kafka")
public class KafkaConfiguration {
private List<String> bootstrapServers;
public List<String> getBootstrapServers() {
return bootstrapServers;
}
public void setBootstrapServers(List<String> bootstrapServers) {
this.bootstrapServers = bootstrapServers;
}
}And a service using Kafka:
@Singleton
public class KafkaService {
private final KafkaConfiguration kafkaConfiguration;
public KafkaService(KafkaConfiguration kafkaConfiguration) {
this.kafkaConfiguration = kafkaConfiguration;
}
//...
}Then when the application is started in development mode (-t in Gradle, mn:run in Maven) and that the Kafka configuration is missing (e.g from the application.yml configuration file), when the KafkaConfiguration bean is read, we would:
- start a test container for Kafka
- inject the "boostrap servers" configuration value from the test container into the started Kafka service
One the application is stopped (or test execution is finished), the test containers would be stopped.
In order to be able to trigger realization of test resources, the application needs a way to determine values of properties which are missing from configuration.
This implies the modification of micronaut-core to support "property expression resolvers". A PropertyExpressionResolvers role is to compute the value of a property expression, if no value is available.
In the example above, a PropertyExpressionResolver can be used to resolve the "bootstrap servers" property.
A key aspect of the implementation is to avoid having to inject test containers in the application classpath (even for tests). Similarly, we'd like to avoid having to modify existing test suites to add support for running test containers.
Therefore, the implementation relies on build plugins to drive the lifecycle of test resources.
The pre-requisite is to run a build in "development mode", which is either running tests or running the application in continuous mode (Gradle) or mn:run (Maven).
The build plugins would inject a PropertyExpressionResolver into the application classpath.
That PropertyExpressionResolver would be responsible for spawning the test containers.
However, as we explained earlier, we don't want to pollute the application classpath, so this resolver must be limited to the bare minimum.
As a consequence, the resolver injected by the build plugin would be limited to the following:
- communicate with an isolated process, triggered by the build plugins, which is in turn responsible for managing the lifecycle of test resources
- fetch information from the server to determine the value of a property expression
sequenceDiagram
participant app as User code
participant resolver as Property resolver
participant proxy as Test resources proxy
participant server as Test resources server
participant tc as Testcontainers handler
app ->> resolver: reads "kafka.bootstrap.servers"
resolver ->> proxy: requests for value of "kafka.boostrap.servers"
proxy ->> server: requests test resource
server ->> tc: starts a test container for Kafka
Note right of server: No server started for kafka.bootstrap.servers<br>Looks for plugins capable of feeding that property
tc -->> server: metadata about containers
server ->> proxy: sends back information about "kafka.bootstrap.servers"
proxy ->> resolver: sends back information about "kafka.bootstrap.servers"
resolver ->> app: unblocks the application
In a nutshell, the resolver is simply a proxy to an external process which is the real resolver. This makes it possible to really isolate application classpath from the test classpath, at the cost of a small, minimal library: the server itself would contain all the required dependencies needed to expose test resources (in particular, testcontainers modules). Because it would run in a separate process, the test containers classpath would be invisible to user code, and building native images which communicate with the server would be possible.
In the simplest case, the application is started in development mode, which means that there's a single process for the application. Therefore, the build plugins can:
- start the test resources server
- inject the proxy into the application classpath
- start the application
- wait until the application is shutdown
- stop the test resources server (which triggers shutting down test containers)
However, we must take into account more complex setups. In particular, during execution of tests, it is possible that multiple processes want to access the same test resources. This is possible because tests are typically executed in forked VMs, every X test. It is also possible that tests are executed in parallel. Therefore, we must consider the behavior of the test resources server whenever concurrent requests happen, and when to shutdown test resources.
The simplest, for the initial implementation, is to share the containers independently of what is requesting them.
This means that if 2 processes request the value of the kafka.boostrapservers property concurrently, in the example above, then we must make sure that only a single container is spawned.
The 2 processes would share the same test container.
In practice, this might not be a reasonable behavior: if the container provides for example a database, having different tests writing to the same database concurrently might lead to flaky tests. Therefore, another option is to spawn test resources per client. We would need a way to identify the client in some form. Test resources lifecycle would be bound to the client lifecycle: they would be spawned on demand, but shutdown when the client goes away (or with a timeout).
In the process above, we described the process of spawning containers as a side effect of requesting the value of a property.
This means, in practice, that a particular handler is capable of answering the question "what is the value of kafka.bootstrapservers?", and as part of the process, it would spawn a server before answering.
However, how do we know that when we read the value kafka.other.property, the server which should be returned is the same as when we read kafka.bootstrapservers?
One option is to be stateful, and whenever any of the values that the handler is capable of dealing with is requested, it would spawn a server, but we would return a complete map of properties, independently of what has been requested. Then when the 2d property is read, we already have the value in memory so we can return it without spawning a server.
Alternatively, we can say that only a single missing property can trigger the server to be spawned. The problem with this approach is that there may be cases where the values of the properties cannot be determined independently of each other.