feat: versioning modules

[xynydev]

We should discuss the approach to use for versioning modules. Of course using the module repository image hash or something like that could work, but it would be best IMO if it was possible to use SemVer for the modules, and pin each module to a major version such that non-breaking changes are automatically updated to.

[gmpinder]

In the recipe we could have a top level property where you can specify the version of the modules image you want. We could default it to the current major version or to latest and let the user decide to override that

[xynydev]

Versioning for individual modules would be better, though. Not sure how that would work, but I have some ideas I'll post later.

[xynydev]

Turns out I didn't get to posting my ideas earlier... Anyway, here's some I can think of now.

A: Unified module repository versioning

• OCI containers already have tags for versioning
• The modules container could similarly adopt a tag for versioning
• The version could be manually updated with semver
  • Whenever any module has a breaking change, the major version is changed
  • Whenever any module has a new feature, the minor version is changed
  • Versioning could be done with git tags instead of a bespoke system
• Users could set a global module version, or manually override it with the source:
• Since all versions have the same version number it would be hard to know which ones are more mature, or which ones were updated in a specific release
  • The version number wouldn't actually correspond to any individual module, but the whole repository as a whole
  • Versions would likely have to be constantly updating, and older versions of the module repository wouldn't receive bug fixes

B: Naive directory versioning

• Make each module directory have subdirectories corresponding to different versions, either semver or probably just major versions
• Make CLI always use the latest stable version unless otherwise specified
  • Maybe syntax like type: default-flatpaks@v2?
• Repos maintain the code for separate versions each module separately, making it easy to backport compatible improvements
• Might break compatibility with the legacy template or older versions of BlueBuild
  • Unless the repo is somehow cleverly structured such that the old image would remain with the same directory layout and a new version would be published

C: Directory versioning and multiple images

Same as option B, but

• Replace the current Containerfile with a dynamic system, that generates separate images for each module, tagged with the module's version
• Could keep compatibility with the legacy template or older version of BlueBuild by keeping the main modules container for the latest versions of images
  • And maybe leveraging the support for the / symbol in image names by publishing images such as ghcr.io/blue-build/modules/rpm-ostree
• Could leverage the OCI standard's tagging system for versioning module artifacts
• Might be possible to build images only for the modules that changed in a given push

D: The no breaking changes strategy

• Basically what we've been doing up until now
• From when a module is published, it and its configuration options then will be supported for the foreseeable future
• A module may be superseded by a differently named module with the same purpose
  • Users will be directed towards the newly named module through documentation notices and build-time logs, and updates to the deprecated module might stop
• Easiest to adopt: no new code needed in either CLI or this repository
• If a module requires frequent breaking changes configuration-wise, name changes might get jarring for users, or we might run out of new names
• No efficient way to see that two modules are part of the same lineage
• Example:
  • Instead of upcoming changes to the default-flatpaks module (refactor(default-flatpaks): Support multiple repos #146), we could just deprecate it and make a new module called flatpak-manager or flatpaks or something

---

That's all I can think of for now. I'm inclided to just go all in with option C, or go the lazy route with D. Please comment your thoughts below.

[fiftydinar]

@xynydev

C sound the best imo.

Everything else has some cons in some way.

[xynydev]

Aight, I wont make a branch or PR for it now, but I'll outline the implementation details I have in mind more clearly.

Github Action with two jobs:

• Job 1
  • Generate a build matrix for job 2
  • Iterate the modules directory, finding every module-version combination with changes from the github event that triggered it
  • Add the required metadata for job 2 to build the module images in the matrix
    • Should include info on whether the module version is the latest version
• Job 2
  • Take inputs from job 1 and generate build args for module.Containerfile
  • Build and push image to ghcr.io/blue-build/modules/modulename with correct tags1

GitHub Action with one job:

• Find and copy latest version of each module folder to a new temporary directory, ie. latest_modules/
• Build modules.Containerfile which does COPY latest_modules /modules and push it to ghcr.io/blue-build/modules
• Only for backwards compatibility, may run only once per week or something, with manual triggers if critical

CLI would use the individual containers by default, and allow specifying the version manually. Unsure what to do with source:.

[gmpinder]

I'm liking this idea so far. If we can keep the classic ghcr.io/blue-build/modules:latest for backwards compatibility, then I could definitely add functionality to pull the module image for each module type we have. We would just have to have a way to support creating new modules without worrying about updating the CLI for each new one.

I could add some logic to check for the presence of a certain module image like by using skopeo and then do a bind mount on that RUN instruction for that particular module. Else we will assume that the module is a user-created one. We can continue to use source as-is for 3rd-party modules that the user would want to use. As for determining the version to use on a per module basis, we could just add a version property or go with the @ syntax. We could specify that the @ syntax only works for the builtin modules if we really want to

[gmpinder]

Here's the module stage we have currently:

# Copy modules
# The default modules are inside blue-build/modules
# Custom modules overwrite defaults
FROM scratch AS stage-modules
COPY --from=ghcr.io/blue-build/modules:latest /modules /modules
{%- if self::modules_exists() %}
COPY ./modules /modules
{% endif %}

And here's a portion of the module run that I could edit to handle pulling the particular module image

        {%- if let Some(source) = module.source %}
  --mount=type=bind,from={{ source }},src=/modules,dst=/tmp/modules,rw \
        {%- else %}
  --mount=type=bind,from=stage-modules,src=/modules,dst=/tmp/modules,rw \
        {%- endif %}

[xynydev]

Just realized that if we publish modules with ghcr.io/blue-build/modules/<moduletype> we can still use the source:. Newer versions wouldn't pull the ghcr.io/blue-build/modules contaiber at all. I guess it could be specified that source: local is needed for local modules, but do that in some sort of soft way where users have a warning period during whch we check whether a module exists in modules/ locally and throw a deprecation warning if it does.

[xynydev]

With #240 done, here's the next steps:

• Figure out separating documentation and module.yml for each version
  • Also need to update the website to display versioned module docs properly
  • Bulk of changes done in the website repo, final changes here are to just move the docs into the versioned folder
• Decide on format in the recipe and implement versioning in CLI