For step 1, python-gitlab (which we are exploring also in other projects) may be helpful.

Hej @zklaus ,
Thanks for the input! Yes, python-gitlab appears reasonable, @valentinaschueller and I have discussed it a couple of days ago and we think that the technical implementation will be kind of obvious, at least as long as we just want to provide the same feature set as for Redmine. What is not so clear at the moment is how exactly to use the EC-Earth Gitlab instance for monitoring, i.e. where to put monitoring output, if we want make use of Gitlab issues in the same way as for Redmine, etc. These are questions that need to be discussed in sync with ECE developers.
In summary, we think that we can implement this as soon as we know what the target exactly is. So we put actual work on hold a bit and try to push the discussion in the ECE working groups.

Sounds fine. Since the target likely will be set in a single line of configuration somewhere, you might as well get on with it and just use a dedicated test project under your own account. Switching (to the main EC-Earth repo with a dedicated label or perhaps to a dedicated monitoring project with no associated repo) should be trivial, right?

A first version of the Gitlab task works (with a private test repository I created) and I have created a draft PR #95 where we can discuss implementation-related questions.

Note: I have stuck with the same concept as in Redmine, i.e., I assume that we just create issues for each experiment and this is what gets updated.

Switching (to the main EC-Earth repo with a dedicated label or perhaps to a dedicated monitoring project with no associated repo) should be trivial, right?

Yes, will be easy! We just have to update the server_url and project_id

Discussion during EC-Earth GA today about moving to Gitlab:
Klaus:

Good idea, but is it possible to update figures in existing issues ?

Valentina:

two years ago, the issue was that the Gitlab API didn't let us delete images from the server, but it seems like this works nowadays

Uwe:

Yes!
Nevertheless, I would double check if issues and replacing images are still the best way to do it when moving to Gitlab. For instance, there are performance issues with the Redmine monitoring pages for long runs (due to a very long history building up).

I have thought about this a bit more and I think having one issue per experiment is still something I like as an idea, because of the discussion feature — or would you disagree? The question really is how problematic the issue revisions become/how much Gitlab wants to store old versions of the issue. I have tried with an example issue on a different Gitlab server to repeatedly change an image in the issue body (over the span of five minutes), and it shows up as a single edit that I edited the description "just now":

Then I added a comment and changed the picture again, moving the line below the comment:

I also don't see previous versions of the issue, so it might be that this history is simply not stored in the same way as on Redmine.

So I think we can just try amending the task with the file deletion feature in the API and test it for an example experiment? @uwefladrich what do you think?

Just a quick shout out from the off before I unsubscribe here: You should also talk to Aitor, who has implemented something similar for HCLIM, iiuc.

[...]
I have thought about this a bit more and I think having one issue per experiment is still something I like as an idea, because of the discussion feature — or would you disagree?

I absolutely agree, the issue approach has nice features! The discussion, but also that people usually know about issues, how to handle them, and have them generally in their workflow.

The question really is how problematic the issue revisions become/how much Gitlab wants to store old versions of the issue.

Some clarification: On Redmine, every issue page (as well as others) has a history, which says who and when things happened. This is at the bottom of the page, but usually hidden (actually by an optional plugin). You can click on "Show details" to see it. Hidden or not, the history entries are part of the HTML code that is sent to the local browser.

Now the problem is that every individual(!) delete from each of the monitoring events will add a single history entry. The adding of new files (altogether) adds one more entry. So basically the pages has (number_of_plots x years) history entries. We have ~25..30 plots at the moment, so in a 500 year run this adds to O(10000) entries.

The Redmine server does not have a problem with this. However, the local browser does. The page (i.e. the HTML code) is just huge and the rendering engine has a lot to do. This is the reason that the problem persists even when the page is loaded. It can happen even after a long time on the page that the browser freezes for some time when navigating the page.

Now this may all be very specific for Redmine. So it is worth a try to just use the Gitlab APi and do things in the same way as for Redmine. It might not be the same outcome. But we should have a careful look to make sure we are not running into the same trouble eventually. Because it shows only up for long monitored runs.

I have tried with an example issue on a different Gitlab server to repeatedly change an image in the issue body (over the span of five minutes), and it shows up as a single edit that I edited the description "just now": [...]

Then I added a comment and changed the picture again, moving the line below the comment: [...]

I also don't see previous versions of the issue, so it might be that this history is simply not stored in the same way as on Redmine.

This may be a good sign!

Just a quick shout out from the off before I unsubscribe here: You should also talk to Aitor, who has implemented something similar for HCLIM, iiuc.

I talked to Aitor. The approach is quite different, but maybe still interesting for our discussion. They do not use an API as such, but "automatic" commits and pushes to a dedicated repository. So the main monitoring page is the README of the repo and new plots and pages are pushed as new files as the experiments go. Also, they use a time-triggered approach (cron jobs) to update, not event-triggered, as we do.

So maybe this is interesting in a broader discussion about what the best approach is, but not directly affecting this issue (which is dedicated to miming the Redmine approach).

Okay, I have now looked around and it seems like it is still not possible to delete issue attachments as a regular user with the REST API in python-gitlab. From what I understand:

• This feature has only been added to the GraphQL API, not the REST API
• it only works for maintainers and owners of a project

Both are indicated here. I have verified that

• deleting an attachment with a GraphQL query works for a repository that I own (on our own Gitlab server)
• this does not work for a repository where I am only a developer (the SCM repo on the SMHI Gitlab):
  

So, one option would be:

• post monitoring output in a separate Gitlab project where everybody who runs experiments has maintainer status
• switch to the GraphQL API (seems to be the future of what Gitlab wants to use anyway)
• before updating an issue, delete all existing attachments using GraphQL queries

Note that GraphQL API support is in beta in python-gitlab. I think one of the consequences of this is that one has to manually provide these long query strings which are frankly quite terrible to work with? I don't know if this package is helpful in this context. See also the related discussion in the python-gitlab repository

As an alternative to the second bullet point above, one could restrict GraphQL usage to the task of deleting attachments to stay in the nice world of objects that python-gitlab provides for working on projects/issues/uploads.

I got here through a cross-reference in python-gitlab: just FYI file uploads can be manipulated with https://docs.gitlab.com/ee/api/project_markdown_uploads.html, just let us know if anything is missing in python-gitlab.

The GraphQL client will probably be GA soon but I'm not sure if we'll want to build a custom DSL on top of the query building at the moment.

I got here through a cross-reference in python-gitlab: just FYI file uploads can be manipulated with https://docs.gitlab.com/ee/api/project_markdown_uploads.html, just let us know if anything is missing in python-gitlab.

Thanks for commenting! Yes I know, we have been using this feature. Our use case is that we would like to either replace the existing file (which is not possible since the destination path is generated when uploading) or delete the existing one when uploading a new one (which is only possible through the GraphQL, not the Rest API, and requires maintainer or owner status on the project).

The GraphQL client will probably be GA soon but I'm not sure if we'll want to build a custom DSL on top of the query building at the moment.

Good to know, and makes sense!