[merge-kits] RFC - Implement a new tool to merge generated ebuild after testing pipeline

[geaaru]

Using autogen in all branches is not a solution. To autogen an ebuild it's a pretty cool because permit to be updated but doesn't ensure that the updated package will be yet compilable.
So, we need to create a new tool that permit to analyze the generated kits from the autogen branch and later create a PR with the update ebuild after an automatic testing pipeline.
These are the key activities to do:

1. integrate the reposcan feature over the anise-portage-converter tool in order to have an easy and fast way to generate the reposcan JSON file to use for identify the packages to update. The reposcan generates the cache data files used by metatools that are with the struct visible here. Having the JSON cache file will permits to easily analyze the trees and identify the new packages.
2. the new merge tool must create a PR with the new ebuild but to have a way to maintain the previous ebuild too in order to have a way to mask a fresh ebuild if something is broken.
3. we need a new tool that help to cleanup the distfiles files no more needed.
4. after that a PR is merged an automatic trigger must to update the meta-repo of the target branch in order to automatically point to the new GIT hash

[cuantar]

I like this idea a lot. When conducting experiments, ideally the scientist can completely describe the experimental apparatus and procedure. When doing scientific computing, ideally the scientist can completely describe the computer system and software. A tree with static ebuilds pointing to artifacts already existing in our CDN helps to satisfy this requirement.

[geaaru]

Yeah, in addition the idea is also using Git Tags to describe checkpoint and new stable tree like I already do with Macaroni Sambuca Stack now. The differences between one tag and another will contains the packages updated and their list.

[geaaru]

I'm thinking about implement the new merge tool but probably we can start with something less complex.
Using a specific cd/ci pipeline is something possible but introduces a limit about the users could run these steps in them private repository because this means reproduce the all Macaroni infra stack. Instead, I think that could be a good point to have a way to run this commands without a complex infra and to do this a possible solution could be:

1. periodically, try to identify new ebuilds from testing repo, merge them at the begin with KEYWORDS=""
2. with a separated steps, and/or task, execute testing phases that will permit to convert KEYWORDS="" in KEYWORDS="*"

In this way these steps could be integrated in a CD/CI but followed with manual scripts.

@cuantar @org-tekeli-borisp what do you think?

[geaaru]

Just an update about the status of this issue.

1. the mark-devkit kit merge command permits to create and/or update kits and ebuilds from existing kits/overlays. This permits to reorganize things to different targets, generate custom kits, permits to users to generate theirs kits from Macaroni Kits. It uses anise-portage-converter tool to generate the Kit Metadata JSON file to speedup checks.

2. When the list of kits merged is ready the next step is create and/or update the meta-repo repository used by ego to sync kits to a local node. The command available for this scope is: mark-devkit kit bump-release. The mark-devkit kit bump-release permits to pin a specific hash of a kit to pin specific packages too.

So, instead of having a single tool that autogen ebuild, update kits and update meta-repo I prefer to have splitted features and having major control and less break points. Different CD/CI Tasks could be executed to update different kits and if something fail this will interrupt only the broken kit, not the others.

The MARK mark-xl release will now use the new tool for ebuild updates.

The first phase will be maintained with direct commits updates but we will soon using PR in the near future to manually validate update to merge eventually leaving direct commits for some kits less critical.

What is yet in TODO:

• integrate PR creation for new packages and/or updates
• generate the distfiles directory from existing release
• cleanup old ebuilds.