Flatcar Release Automation

[sayanchowdhury]

Summary

This discussion is a collection of ideas that we could implement in order to improve and bringing more automation to the release process of Flatcar Container Linux.

Motivation

The process for releasing a new Flatcar Container Linux is complex and currently, most parts of it are manually driven. The manual nature of the task makes it prone to errors and is also very time-consuming.

The process followed is documented in the release document for the release manager to follow and coordinate.

The process can be divided into 3 stages which makes it easier to measure:

• Triggering the Jenkins builds
• Creating the Release Notes
• Publishing & Updating the Release Artifacts.

The aim of this discussion is to go through these steps, and try to find out ways to automate the process. In the beginning, it would be good to start with creating standalone scripts, and later utilizing these scripts to build a complete automated pipeline.

The benefits of automating would be:

• Lesser Tech-Debt, and would be easier for anyone on the Release Team to coordinate the release, or do the release without much of a hassle. A new member in the Flatcar Release Team has to go work with an existing member to completely understand the release process. This endeavour is time-consuming, and more than that repetitive. We do have good documentation in place for the existing pipeline, but that required a deeper understanding of FCL. Bringing Automation to the table would ease the process of onboarding of the new members to the Release Team.
• Reduced Time consumption. The whole process at the moment takes about 24-36 hours, and sometimes even more.
• Scalable & Sustainable process: The process at the moment is not sustainable, as it poses challenges whenever we need to add new ideas or changes to the release process.

Detailed Scoping

Triggering the Jenkins Builds

In preparation for starting the Jenkins builds, we perform a couple of steps, like creating the release branches, tagging the release, creating manifest and starting the Jenkins build.

Scope of Improvement/Tasks

It should be trivial to gather the steps into a single script, with the dropping a couple of questions to the Release Manager (RM), primarily VERSION, BRANCH, and if a new SDK is to be created. As the process is divided into steps, this information could be stored in a central file to access. Jenkins (Flatcar CI) provides an API to trigger remote builds so this can also be used to use the information asked earlier to trigger the builds.

Creating the Release Notes

Release notes creation is the biggest pain point of the whole release process and is also very time-consuming.

GH Template

The coreos-overlay-diff.py in the flatcar-build-scripts could be continued to be used for finding the list PRs that were made between the two release tags. The given script would be called from the primary script, and a call could be made to get the metadata of the PR via Github API.

The reason to make the call is to determine if the PR is to be added to the Release Notes or not. Fields would be added to the Github PR template to simplify the process.

• Release Notes Text:
• Release Notes Category:
• Does the PR update an existing package?:
• Does the PR involve security fix? (eg CVE)

If the field is empty, or not present, the PR would be skipped. During the parsing process, it would be a good idea to pull the names of the contributors involved in the release.

The primary script should have the ability to determine the last release and pick the metadata from the central file to create the whole Release Notes, and put it in front of the RM for review. This could be done in the markdown format and later can be converted to other formats required using pandoc.

Another idea could be to create a Github (GH) repo containing the Go/No-Go documents, which again in markdown and the process is triggered again once when the Release Team decides on "Go"

Automated GH PRs

A bunch of PRs are created by Github Actions, which creates noise in the coreos-overlay-diff.py. Add a flag to the script that would only show the latest PR for specific repositories.

Generating CHANGELOG Links

Over the lifecycle of FCL, the Release Team has updated a bunch of packages and in the Release Notes updates section, we link the release version URL. Usually, these are just a change of the version/tag number, so it would be good to generalize them to common ones. The initial list could be created using the already released ones.

CVE Generation

Publishing & Updating the Release Artifacts

Once the Go/No-Go meeting is done, the RM starts the process to release the artifacts. The whole process takes around an hour to complete and is mostly manual. As the process deals with release version numbers, there is a high chance of typo and given the repetitive steps, the process can surely be automated.

Scope of Improvement/Tasks

• Copy artifacts to the origin server
  As we are collecting the information from the RM, we could use a tool like Fabric2 to automate the process, with intervention in between, like before doing the actual sync the RM needs to confirm.

• Publish AMIs and GCP images
  Again use the Jenkins API to trigger build and use the information from the central metadata file.

• Publish Azure SKUs, Publish AWS Marketplace Images, Publish GCP Marketplace offer
  The process here could also be automated, the release documents suggest methods to automate it. Implementing these could be the time-consuming and could be prioritized after the other steps are automated.

• Update the SDK release metadata for new major Alpha releases with SDK builds
  This could again be quickly automated by simple script. To create the PR, the script can make use of the gh tool or API.

• Change current symlink on the origin server
  With the help of Fabric2 again, the symlink could be updated.

• Update the website and CF templates
  The final steps in the release process, could be automated with script and gh tool or API.

After the release process is done, the script could also spit out a report to the RM.

TODOS

• Add the version From -> To update so that it is easier for folks to track.
• Add a flag to the script that would only show the latest PR for specific repositories.

[dongsupark]

Mostly agreed.

For the Flatcar release team, I think the biggest pain of all above is Creating the Release Notes.
That's where >70% of difficulties come from.

Simply speaking, we should get away from so many Google docs and spreadsheets.
Those docs and spreadsheets are suitable for helping collaboration of the remotely distributed team, and have been useful for easy on-boarding.
However, in the long term, using the docs and spreadsheets, it gets almost impossible to keep the correct info in a scalable manner.
Think about how many times we got lost in a few lines of Google docs changes in the current release (say Beta) being different from changes in the previous equivalent ones. (say Alpha)
It could be somehow trackable when each doc have only several pages, but definitely not when every set of releases has ~20 pages like we have.
That issue is not only about release docs, but all over the place.
So pandoc or markdown-based tools might be a really great option from my perspective.

Downside is, however, then some newcomers who are not familar with the tools could experience more difficulties than Google docs.

[t-lo]

Release notes proposal (capturing the team's thoughts):

• Add a changelog/ folder to each of coreos-overlay, portage-stable, scripts.
  • with subfolders
    • changes/
    • bugfixes/
    • security/
    • updates/
• The root changelog folder contains a file changelog.md which covers changes up to the latest release of that branch.
• Each PR going into any of the three repos must also include at least a file to changes/ describing the change.
  • Security, bugfix, and update changes additionally include a respective file in the respective sub-directory
  • The contents of these files will be included in the corresponding section in changelog.md in the next release
• During a release:
  • all new files are merged into a new section on top of the changelog.md file
  • all files in the sub-directories changes/, bugfixes/, security/, updates/ will be deleted

[pothos]

The deletion of the sub-directory entries could also be omitted if it turns out to be a nice way to compare releases even after the release is done. Finding newly added files compared to the last release when assembling the changelog.md is still possible through comparing to the last release tag via git.