Setup docker dev deployment on a daily basis

[bossenti]

Body

Currently, we have GitHub workflow that builds new docker images for the current code base on every push for the dev branch and every release branch.
For release branches this should stay as it is, but for our dev branch we should switch to a less frequent deployment mode.
I'd suggest to go with a daily build executed e.g. at 00:00 UTC.

Mentoring

As this ticket is marked as good first issue: @dominikriemer, @tenthe or @bossenti are happy to provide help for getting started, just tag (one of) them if you want to start working on this issue and need some help.
This issue might be a good issue for people interested in GitHub workflows.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

[bossenti]

I've created a discussion of the above issue first.
From my point of view building the docker image for every commit is quite some overhead with our current development speed.
Therefore, I'd suggest to switch to a daily schedule for our dev branch instead.

Are there any negative side effects I don't have on my radar for now? @tenthe @dominikriemer

[bossenti]

@tenthe agree, a manual trigger would be of value

[bossenti]

@tenthe removing duplicated steps sounds great to me
From a first look, only the maven build seems to be duplicated to me
This could be removed, right?
But from my point of view this then requires some adaptions we are discussing in #882 to ensure that every commit runs once through the whole validation

[tenthe]

My suggestion would be to run the static code validations (e.g. license header checks, linting) and additionally the unit test on every commit. For PRs, we can additionally run the smoke e2e tests. During the nightly build, we can then run the integration test and all the e2e tests, and if everything was successful, we push the Docker snapshot containers to the registry.What do you think of this?
If you like the idea, we can discuss what commands need to be executed to perform the appropriate validations for the different parts (e.g. UI, backend, Python).

[dominikriemer]

I'm ok with nightly builds and running the complete build process in nightly builds.
We could add a flag in the Maven configuration to skip the integration tests, as I think the default when running an mvn package on the project should be that all tests including integration tests are run.
@RobertIndie what do you think would be a best practice approach for running the builds and tests concerning the trade-off between build time and complete building?

[bossenti]

@tenthe I really like your suggestion and think this could be great outcome of this discussion.
However, I don't really know how to achieve this except talking about Python.

Having one eye on our test coverage: With regard to the UI, we only have e2e tests in place so far, right? Might it be of value to think about unit tests here as well? Probably @smlabt has some ideas about how to provide a good setup for this.

[tenthe]

@bossenti in terms of ui we have three options. Currently we use the e2e test, but it requires the entire system to run. Also, Cypress has a new feature called component testing. This allows you to test components "unit." I have played with it a bit and it is actually very cool. Further, it is also possible to apply unit tests, those files are within each component directory.

[bossenti]

Looks like we already have some ideas, how do we want to continue here?
Are there same easy steps we can directly write issues for or is there so more research and conceptual work to be done first?

[tenthe]

I think we can start by creating issues and then defining the tasks directly in the issues. Maybe we can also use the issues to "collect" all the necessary commands.