How to run tests for many different configurations

[eijckron]

I'm working on a project that is multi-tenant and has a number of feature toggles, we are looking into component tests to validate our services work properly for 5 tenants (and growing) on production and preview (a specific set of feature toggles enabled).

Our current set up uses abstract base test classes with sub-classes for the various tenants with configuration for the base test and optionally additional tests for the specific tenant. Feature toggles are only tested at unit test level so there is limited coverage to see if the feature toggle would impact other parts of the system.

Preferably we want our component tests to run for all tenants with either production level or preview level feature toggles enabled however that results in 10 sub-classes of every test which becomes unmanageable.

I'm interested in others that have similar challenges and how they have handled them.

[quarkus-bot[bot]]

/cc @radcortez (config)

[radcortez]

How about having a separate surefire execution for each tenant with the required tenant configuration? You could have general tests for all tenants and specific tests that only activate based on the surefire configuration.

Surefire accepts system properties or env variable configuration, that are read by Quarkus and influence the runtime.

[eijckron]

Based on the suggestion from Radcortez I did a quick proof of concept.
The setup works and is better manageable than what we have currently.
Only drawback I see so far is that we loose the easy right click run test from our IDE.