Redefine e2e/usage test concept

[bisgaard-itis]

What's a usage test?

The purpose of a usage test is to determine whether or not a user workflow works as expected. That means a test is a "user story" and can be thought of as an integration test in a very broad sense. Their purpose is not to detect network issues or that the api of, say the catalog service, broke backwards compatibility in the latest release. This should be tested elsewhere. Usage tests should detect if a user story is broken but are not designed to diagnose issues on the server. Ideally, when a usage test fails it should point to more specific tests/monitoring tools closer to the server which can be run to diagnose what the problem is. In that sense a usage test is a monitoring tool, but not a very good monitoring tool: It doesn't point the developer to where the issue is. I.e. primarily we should rely on other monitoring tools. This suggests it only makes sense to look at the results of usage tests after checking that all other tests/monitoring tools are happy.

Challenges with the current e2e testing framework

• There is a presentation issue of how the e2e tests are presented in the daily. This is however the most minor problem from my POV.
• Too much noise. The e2e tests are designed around what the question of which server should be compatible with which client (see here). This leads to rather excessive testing and a lot of time is spent on fixing issues which are either not really important or detected and fixed elsewhere.
• They don't drive progress. Usage tests should tell the developer how the user sees our system and help drive developers to build a more usable/user-friendly platform/client.

Requirements the redesigned e2e/usage testing system should meet

• It must be possible to run some usage testsuite with an old version of the client against the latest compatible server in order to check that the user workflows for users using that client version is not broken.
• The client's documentation consists of tutorials which are example user workflows. The usage tests should run these.

Proposal

• Redesign e2e tests such that the tests within the repo are only guaranteed to be compatible with the client corresponding to that git ref. Instead of having bash scripts within the client repo which loop over different server configs, simplify the design to only run against a single server config.
• Go through current e2e tests and move/remove tests which are not proper usage tests (see What's a usage test?)
• Go through e2e tests and setup (where it makes sense) other monitoring tools (e.g. Grafana) for detecting issues. (examples: studies tests and log streaming tests)
• Only run usage tests against master and production deployments

[bisgaard-itis]

@pcrespov as discussed yesterday, here's an attempt to start a discussion about what our usage tests should be and when/where/why we should be running them. Please have a look and give me your feedback.

[bisgaard-itis]

See also ITISFoundation/osparc-simcore#6824