Handling ECP mirrors

[iskra-anl]

In GitLab by @perarnau on Sep 24, 2019, 16:29

We need to figure out and document our approach to ECP mirrors and external gitlab runners.

Given the current constraints, I would recommend something using

• gitlab environments
• a set of protected staging/production branches
• a separate ecp-ci pipeline that only runs on master and those other protected branches
• a way to link external pipeline status into this gitlab instance

See this: https://docs.gitlab.com/ee/workflow/gitlab_flow.html#production-branch-with-gitlab-flow

[iskra-anl]

In GitLab by @perarnau on Oct 1, 2019, 09:33

What I'd like to try:

1. keep master as the "production" branch. This is the branch that we should point people at if they want an updated version of the library, and updated clean docs.

2. Introduce a new staging protected branch. This is the branch to target all of the merge requests to. I will also activate this branch in readthedocs, for us to be able to check how the docs are building.

3. Keep the separate .ecp-ci.yml ECP-CI pipeline, and activate it for all protected branches.

4. Use gitlab environments to provide a docs environments and a link to ECP-CI pipeline results.

5. Regularly merge staging into master, once the ECP-CI pipelines are green.

One of the difficulty is that I don't want to end up with a gitflow (like github) process were staging becomes the default branch and we don't merge into master anymore. I can't find a way to automatically trigger a merge request though.

@NicolasDenoyelle, @iskra-anl, @Kerilk: thoughts ?

[iskra-anl]

In GitLab by @NicolasDenoyelle on Oct 9, 2019, 08:08

This process of external pipeline seems to be a significant overhead...
Anyway, here is my opinion:

1. I agree,
2. It seems to me that staging branch will produce release tags. Will master only consist in releases?
   I am ok with that, though it seems to be an innovative way to use master.
3. Can't we just run ECP-CI pipeline as a step of our pipeline when making a release/merge in master?
   Also, you have to be more specific on what are the protected branches aside from staging. The constraints and implications of ECP-CI pipeline are not clear to me.
4. I don't know what are docs environment. My suggestion in 3. solves 4. The pipeline would show the result of ECP-CI.
5. Of course

I would have triggered ECP-CI pipeline on release tags creation from master branch.
I think it solves what you mention here. But as I said, I don't understand implications of ECP-CI pipeline.

[iskra-anl]

In GitLab by @perarnau on Oct 9, 2019, 08:34

2. I don't intend to change the release/tagging process. No release tags on staging, and we should merge into master on a regular basis to avoid drifts.
3. • we can't run the ecp-ci tests on our infrastructure, or from our gitlab, as the runners are only registered on the mirrors and we don't control them.
   • we can't run the ecp-ci tests on every branch, they don't want that, it costs too many node hours, and there are some security issues (any contributor on our repo becomes capable of running code on DOE systems).
   • if we limit the ECP-CI to master then it runs after merges, and master is not green all the time.
   • I only want to protect master, staging, and the releases
4. environments are documented here: https://docs.gitlab.com/ee/ci/environments.html it's just a link to an external resource modified by a pipeline

[iskra-anl]

In GitLab by @perarnau on Oct 18, 2019, 11:07

mentioned in merge request !90

[iskra-anl]

In GitLab by @perarnau on Oct 21, 2019, 09:51

Managed to make it work for readthedocs with this: https://xgitlab.cels.anl.gov/argo/aml/environments/3

[iskra-anl]

In GitLab by @perarnau on Oct 25, 2019, 15:43

mentioned in merge request !71

[iskra-anl]

In GitLab by @perarnau on Oct 25, 2019, 15:44

!71 and !72 were merged following this setup. I'm going to experiment with multi-pipelines triggers to see if it can improve something.