Update Workflows

[EliSauder]

(Edit: A lot of this has gotten cut out of the plan. See comments later on about what got removed)

Ello, here is a done pipeline for the main zig tooling/library. It still needs the website stuff (will be working on that next).

How things work and names and everything can be changed. I just wanted to get this out here for people to start looking at.

Implemented functionality:

• Build microzig on PR to main & Push to main. Both have file triggers to only run if a part of Microzig that would affect the build is changed (a).
• Upon push to main (and successful build), create a PR to zig-master with the changes.
  • Create new branch (b)
  • Create new dummy commit in branch (this is to make it so that the master test will be done against a different commit. (c)
  • Create PR against zig-master (d)
  • CI to build microzig against zig master upon PR to zig-master
• Using same publish method, publish changes upon tag creation (e)
  • Create GitHub draft release for commit (f)

CI Update TODO:

• Website Publish
  • Build & Test
  • Staging area with viewable staged sites.
  • Deploy on PR to main
• Run tests after building (Need more details on what is needed for testing)
• Add download URLs to draft releases
• Additionally publish regz and uf2 separately from the main microzig location.
• Update build.zig.zon version based on tag prior to publishing.
• Add binaries to draft releases (maybe)

Microzig Repo TODO:

• Create zig-master branch
• Add PR_CREATE_PAT PR_CREATE_TOKEN (name changed to be more accurate to what it is. see) secret for actions. This is to allow automated PR creation. The Fine-grained PAT tested with has: "Read access to metadata", and "Read and Write access to pull requests" as permissions for my fork of the microzig repository.
• Add DEPLOY_ROOT_DATA_PATH secret for actions. This specifies the root of the data directory when publishing to a static file site (My environment needed a different value than the old workflow, so I added it as a secret that can be configured).

Remarks

• (a): Runs if a change was made to any of:
  • .github/**
  • build.zig*
  • tools/**
  • core/**
  • bsp/**
  • examples/**
• (b): The branch is created off of main in the format of: master-patch/{sha}. Example: master-patch/1895868a2cf6783cd9be4653b0f99503d4c36f03. This was chosen because it prevents duplication of branch names allowing multiple patches to be open at any given time.
  • Cons: The major con of this is that if not maintained actively, the list of open branches can balloon (This is also a downside of (d))
• (c): Github stores CI results against the sha. This means that if you directly run the master test against the same commit on main, it failing will turn main red. That is something I wanted to ensure did not happen. Creates an empty commit with the following details:
  • author: ZEG Github Action <>
  • message: chore: commit for zig master build ci
• (d): Creates a PR from the master-patch/* branch to zig-master with the below details. See (b) for the main con.
  • title: Testing commit {sha} with zig master
  • body: Created by Github action
  • Example: Testing commit b9313ab485a826eae8fa070b5d9108f06bd381c5 with zig master EliSauder/microzig#47
• (e): For the time being the tags expected are in either of the below formats. I am imaging for the website, that it just behaves like trunk based development without versions. These would only be for microzig and all of its tools (i.e. everything gets versioned together)
  • v*.*.*
  • v*.*.*-* (beta, alpha, rc, etc.)
• (f): To cap off the whole process, upon a successful build and deploy, a draft release is created in GitHub for the tag that triggered the release. Draft is automatically marked as prerelease if the version contains: beta, alpha or rc

[EliSauder]

Realized I forgot to mention it in the main PR message. This currently also includes some cleanup effort. I've removed a lot of the LICENSE files except from the root of the project and from each of the tools. I also removed old workflows, unused files, etc.

[EliSauder]

Update: I removed the dependency of having a separate PAT. GITHUB_TOKEN has permission configurations available.
https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/controlling-permissions-for-github_token

[EliSauder]

Update: I removed the dependency of having a separate PAT. GITHUB_TOKEN has permission configurations available. https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/controlling-permissions-for-github_token

After doing some more testing I re-realized why I did the PAT method over continuing with the GITHUB_TOKEN. The reason is that with using GITHUB_TOKEN, it will not trigger workflows (i.e. the zig-master build workflow). Here is the section in the documentation about it. https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow#triggering-a-workflow-from-a-workflow

Here is the specific paragraph that is relavent to enabling this ability:

If you do want to trigger a workflow from within a workflow run, you can use a GitHub App installation access token or a personal access token instead of GITHUB_TOKEN to trigger events that require a token.

[EliSauder]

Zon version overriding added. This can be removed if desired, however, based on my idea on what the publish should do, everything in the repo should be versioned together. As such, the publish workflow will now overwrite the version specified in all of the zon files with the tag that is provided.

Notes: This overwrite will only occur if the version is a semver or a semver prefixed by v.

Example generation with a tag: v0.12.2.-beta25.

[EliSauder]

Update:

Features added:

• Boxzer output is now flattened in the release.
  • Any given release has the folder path zig-{{zig version}}/{{artifact}}-{{artifact_version}}/ e.g. zig-0.13.0/microzig-0.1.0/
  • The folder structure within this folder matches the image below
  • An example can be viewed here: https://microzig.funtimes.meme/zig-0.13.0/microzig-0.12.2-beta44/ (if the link doesn't work, please see the microzig releases for an example as I may have removed my test environment).

Microzig repository variables/secrets needed:

• Variables
  • MZ_TARGET_ZIG_VERSION - The primary version to build/publish for microzig/regz/uf2/etc.
  • WS_TARGET_ZIG_VERSION - The version to build/publish the website with.
• Secrets
  • DEPLOY_MZ_HOST - When publishing microzig, host address to publish to.
  • DEPLOY_MZ_PORT - When publishing microzig, port on the host to use.
  • DEPLOY_MZ_PRIVATE_KEY - When publishing microzig, private key to connect with.
  • DEPLOY_MZ_ROOT_DATA_PATH - When publishing microzig, location of directory to store data to within the host.
  • DEPLOY_MZ_USER - When publishing microzig, user to connect with.
  • DEPLOY_WS_HOST - When publishing website, host to publish to.
  • DEPLOY_WS_PORT - When publishing website, port to publish to.
  • DEPLOY_WS_PRIVATE_KEY - When publishing website, private key to connect with.
  • DEPLOY_WS_ROOT_DATA_PATH - When publishing website, location of directory to store data to within the host.
  • DEPLOY_WS_USER - When publishing website, user to connect with.
  • DOWNLOAD_URL - When publishing microzig, download url to use with boxzer.
  • PR_CREATE_TOKEN - Token that is used when creating the PR against the zig-master branch. This token is needed to ensure that the PR can trigger workflows itself.
• Other:
  • Creation of the zig-master branch.

[EliSauder]

I additionally added some of the commits from #229 to enable the website to build so I could validate that it worked.

[ikskuh]

Alternatively, can we cretae something like a "config file" where all variables for the workflows will be defined, and we can make the secret deduplication there?

[EliSauder]

Alternatively, can we cretae something like a "config file" where all variables for the workflows will be defined, and we can make the secret deduplication there?

Wouldn't that be insecure? Storing private keys in a text file doesn't seem wise. Though, if we made the content of the text file the value of the secret we could do that.
Though, I'm not sure what the benefit over just having the secrets would be.

[ikskuh]

Alternatively, can we cretae something like a "config file" where all variables for the workflows will be defined, and we can make the secret deduplication there?

Wouldn't that be insecure? Storing private keys in a text file doesn't seem wise. Though, if we made the content of the text file the value of the secret we could do that. Though, I'm not sure what the benefit over just having the secrets would be.

This wasn't meant as a physical file but more as a "github ci file" which can be included or similar, so we don't have to spread the actual secrets across all files.

But in the end, it would also just increase complexity for little benefit

[EliSauder]

Alternatively, can we cretae something like a "config file" where all variables for the workflows will be defined, and we can make the secret deduplication there?

Wouldn't that be insecure? Storing private keys in a text file doesn't seem wise. Though, if we made the content of the text file the value of the secret we could do that. Though, I'm not sure what the benefit over just having the secrets would be.

This wasn't meant as a physical file but more as a "github ci file" which can be included or similar, so we don't have to spread the actual secrets across all files.

But in the end, it would also just increase complexity for little benefit

We could turn it into a job/workflow that job would then have outputs which could be used by subsequent jobs. However, we run into the same issue of storing information that should be secret within an actual file that is publicly viewable.

[EliSauder]

Updated it with a simplified CI.

• Remove boxzer flattening and that will end up changed on the repository level.
• Simplify publish workflows (merged website publishing into one file)
• Remove the build automation against zig master.
• Reverted secrets to be microzig/website agnostic (DEPLOY_MZ_HOST & DEPLOY_WS_HOST -> DEPLOY_HOST)

Current list of required repository/org secrets should match the old list with a couple additions:

• DEPLOY_HOST
• DEPLOY_PORT
• DEPLOY_PRIVATE_KEY
• DEPLOY_ROOT_DATA_PATH
• BOXZER_DOWNLOADS_URL

Current list of required repository/org variables:

• MZ_TARGET_ZIG_VERSION
• WS_TARGET_ZIG_VERSION
• TARGET_ZIG_VERSION