From 492de865430f2f19e41f288a3ccc23a717a03290 Mon Sep 17 00:00:00 2001 From: Jonathan Matthews Date: Tue, 21 Nov 2023 16:44:20 +0000 Subject: [PATCH] content: driving buildkite pipelines with CUE Here's a new cue-by-example, showing how to manage a set of Buildkite pipeline files in CUE instead of YAML. Because Buildkite (in its non-dynamic-pipeline mode) still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is very trivial, and not hugely useful - it could do with improving. I wasn't able to use the upstream schema (https://github.com/buildkite/pipeline-schema) because of issues now tracked as cue-lang/cue#2698 and cue-lang/cue#2699. This is related to #21 but doesn't /close/ it, as that issue tracks a cue-by-example for Buildkite's "dynamic" pipeline mode, which this guide doesn't address. Signed-off-by: Jonathan Matthews --- README.md | 1 + XXX_buildkite_importing_pipelines/README.md | 374 ++++++++++++++++++++ 2 files changed, 375 insertions(+) create mode 100644 XXX_buildkite_importing_pipelines/README.md diff --git a/README.md b/README.md index cc1cb9e..63d5549 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ integrating with 3rd-party tools, services, and systems. - 003: [Controlling Kubernetes with CUE](003_kubernetes_tutorial/README.md) - 004: [Managing Mythic Beasts DNS zones with CUE](004_mythic_beasts_dns/README.md) - 005: [Driving GitLab CI/CD pipelines with CUE](005_gitlab_ci/README.md) +- XXX: [Driving Buildkite pipelines with CUE](XXX_buildkite_importing_pipelines/README.md) ## Contributing diff --git a/XXX_buildkite_importing_pipelines/README.md b/XXX_buildkite_importing_pipelines/README.md new file mode 100644 index 0000000..b75e67b --- /dev/null +++ b/XXX_buildkite_importing_pipelines/README.md @@ -0,0 +1,374 @@ +# Driving Buildkite Pipelines with CUE +by [Jonathan Matthews](https://jonathanmatthews.com) + +This guide explains how to convert Buildkite pipeline files from YAML to CUE, +check those pipelines are valid, and then use CUE's tooling layer to regenerate +YAML. + +This allows you to switch to CUE as a source of truth for Buildkite pipelines +and perform client-side validation, without Buildkite needing to know you're +managing your pipelines with CUE. + +## Prerequisites + +- You have + [CUE installed](https://alpha.cuelang.org/docs/introduction/installation/) + locally. This allows you to run `cue` commands. +- You have a set of Buildkite pipeline files. The examples shown in this guide + use a specific commit from the Buildkite + [dependent-pipeline-example](https://github.com/buildkite/dependent-pipeline-example/tree/a7419a24bee24068d4d79399bccd9093569c3013/.buildkite) + repository, *but you don't need to use that repository in any way.* +- You have [`git` installed](https://git-scm.com/downloads). + + +## Steps + +### Convert YAML pipelines to CUE + +#### :arrow_right: Begin with a clean git state + +Change directory into the root of the repository that contains your Buildkite +pipeline files, and ensure you start this process with a clean git state, with +no modified files. For example: + +:computer: `terminal` +```sh +cd dependent-pipeline-example # our example repository +git status # should report "working tree clean" +``` + +#### :arrow_right: Initialise a CUE module + +Initialise a CUE module named after the organisation and repository you're +working with. For example: + +:computer: `terminal` +```sh +cue mod init github.com/buildkite/dependent-pipeline-example +``` + +#### :arrow_right: Import YAML pipelines + +Use `cue` to import your YAML pipeline files: + +:computer: `terminal` +```sh +cue import ./.buildkite/*.yml --with-context -p buildkite -f -l pipelines: -l 'strings.TrimSuffix(path.Base(filename),path.Ext(filename))' +``` + +Check that a CUE file has been created for each YAML pipeline in the +`.buildkite` directory. For example: + +:computer: `terminal` +```sh +ls .buildkite/ +``` + +Your output should look similar to this, with matching pairs of YAML and CUE +files: + +```text +pipeline.cue pipeline.deploy.cue pipeline.deploy.yml pipeline.yml +``` + +Observe that each pipeline has been imported into the `pipelines` struct, at a +location derived from its original file name: + +:computer: `terminal` +```sh +head .buildkite/*.cue +``` + +The output should reflect your pipelines. In our example: + +```text +==> .buildkite/pipeline.cue <== +package buildkite + +pipelines: pipeline: steps: [{ + command: "echo 'Tests'" + label: ":hammer:" +}, + "wait", { + trigger: "dependent-pipeline-example-deploy" + label: ":rocket:" + branches: "master" + +==> .buildkite/pipeline.deploy.cue <== +package buildkite + +pipelines: "pipeline.deploy": steps: [{ + command: "echo 'Deploy'" + label: ":rocket:" + concurrency_group: "$BUILDKITE_PIPELINE_SLUG-deploy" + concurrency: 1 + branches: "master" +}] +``` + +#### :arrow_right: Store CUE pipelines in a dedicated directory + +Create a directory called `buildkite` to hold your CUE-based Buildkite pipeline +files. For example: + +:computer: `terminal` +```sh +mkdir -p internal/ci/buildkite +``` + +You may change the hierarchy and naming of `buildkite`'s **parent** directories +to suit your repository layout. If you do so, you will need to adapt some +commands and CUE code as you follow this guide. + +Move the newly-created CUE files into their dedicated directory. For example: + +:computer: `terminal` +```sh +mv ./.buildkite/*.cue internal/ci/buildkite +``` + +### Validate workflows + + + +#### :arrow_right: Create a pipeline schema + +Create a very basic CUE schema for Buildkite pipelines, adapted from +[Buildkite's documentation](https://buildkite.com/docs/pipelines/configuration-overview), +and place it in the `internal/ci/buildkite` directory: + +:floppy_disk: `internal/ci/buildkite/buildkite.pipeline.schema.cue` + +```CUE +package buildkite + +#Pipeline: { + steps!: [...] + env?: [string]: string + agents?: {[string]: string} | [ ..._#kv] + _#kv: =~".+=" + notify?: [...] +} +``` + +| :grey_exclamation: Info :grey_exclamation: | +|:------------------------------------------- | +| It would be great if we could use [Buildkite's authoritative pipeline schema](https://github.com/buildkite/pipeline-schema) here. Unfortunately, CUE's JSONSchema support can't currently import it. This is being tracked in CUE Issues [#2698](https://github.com/cue-lang/cue/issues/2698) and [#2699](https://github.com/cue-lang/cue/issues/2699), and this guide should be updated once the schema is useable. + +#### :arrow_right: Apply the schema + +We need to tell CUE to apply the schema to each pipeline. + +To do this we'll create a file at `internal/ci/buildkite/pipelines.cue` in our +example. + +However, **if the pipeline imports that you performed earlier *already* created +a file with that same path and name**, then simply select a different CUE +filename that *doesn't* already exist. Place the file in the +`internal/ci/buildkite/` directory. + +:floppy_disk: `internal/ci/buildkite/pipelines.cue` + +``` +package buildkite + +// each member of the pipelines struct must be a valid #Pipeline +pipelines: [_]: #Pipeline +``` + +### Generate YAML from CUE + +#### :arrow_right: Create a CUE tool file + +Create a CUE "tool" file at `internal/ci/buildkite/ci_tool.cue` and adapt the +element commented with `TODO`: + +:floppy_disk: `internal/ci/buildkite/ci_tool.cue` +```CUE +package buildkite + +import ( + "path" + "encoding/yaml" + "tool/file" +) + +_goos: string @tag(os,var=os) + +// Regenerate all pipeline files +command: regenerate: { + pipeline_files: { + // TODO: update _toolFile to reflect the directory hierarchy containing this file. + let _toolFile = "internal/ci/buildkite/ci_tool.cue" + let _pipelineDir = path.FromSlash(".buildkite", path.Unix) + let _donotedit = "Code generated by \(_toolFile); DO NOT EDIT." + + clean: { + glob: file.Glob & { + glob: path.Join([_pipelineDir, "*.yml"], _goos) + files: [...string] + } + for _, _filename in glob.files { + "Delete \(_filename)": file.RemoveAll & {path: _filename} + } + } + + create: { + for _pipelineName, _pipeline in pipelines + let _filename = _pipelineName + ".yml" { + "Generate \(_filename)": file.Create & { + $after: [ for v in clean {v}] + filename: path.Join([_pipelineDir, _filename], _goos) + contents: "# \(_donotedit)\n\n\(yaml.Marshal(_pipeline))" + } + } + } + } +} +``` + +Make the modification indicated by the `TODO` comment. + +This tool will export each CUE-based pipeline back into its required YAML file, +on demand. + +#### :arrow_right: Test the CUE tool file + +With the modified `ci_tool.cue` file in place, check that the `regenerate` +command is available **from a shell sitting at the repo root**. For example: + +:computer: `terminal` +```sh +cd $(git rev-parse --show-toplevel) # make sure we're sitting at the repository root +cue help cmd regenerate ./internal/ci/buildkite # the "./" prefix is required +``` + +Your output **must** begin with the following: + +```text +Regenerate all pipeline files +Usage: + cue cmd regenerate [flags] +[... output continues ...] +``` + +| :exclamation: WARNING :exclamation: | +|:--------------------------------------- | +| If you *don't* see the usage explanation for the `regenerate` command (or if you receive an error message) then your tool file isn't set up as CUE requires. Double check the contents of the `ci_tool.cue` file and the modifications you made to it, as well as its location in the repository. Ensure the filename is *exactly* `ci_tool.cue`. Make sure you've followed all the steps in this guide, and that you invoked the `cue help` command from the root of the repository. + +#### :arrow_right: Regenerate the YAML pipeline files + +Run the `regenerate` command to produce YAML pipeline files from CUE. For +example: + +:computer: `terminal` +```sh +cue cmd regenerate ./internal/ci/buildkite # the "./" prefix is required +``` + +#### :arrow_right: Audit changes to the YAML pipeline files + +Check that each YAML pipeline file has a single *material* change from the +original: + +:computer: `terminal` +```sh +git diff .buildkite/ +``` + +Your output should look similar to the following example: + +```diff +diff --git a/.buildkite/pipeline.deploy.yml b/.buildkite/pipeline.deploy.yml +index 4af2b9d..e0fc010 100644 +--- a/.buildkite/pipeline.deploy.yml ++++ b/.buildkite/pipeline.deploy.yml +@@ -1,6 +1,8 @@ ++# Code generated by internal/ci/buildkite/ci_tool.cue; DO NOT EDIT. ++ + steps: + - command: echo 'Deploy' +diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml +index 7446201..10cefad 100644 +--- a/.buildkite/pipeline.yml ++++ b/.buildkite/pipeline.yml +@@ -1,12 +1,14 @@ ++# Code generated by internal/ci/buildkite/ci_tool.cue; DO NOT EDIT. ++ + steps: + - command: echo 'Tests' + label: ':hammer:' + - wait + - trigger: dependent-pipeline-example-deploy +``` + +The only *material* change in each YAML file is the addition of a header that +warns the reader not to edit the file directly. + +#### :arrow_right: Add and commit files to git + +Add your files to git. For example: + +:computer: `terminal` +```sh +git add .buildkite/ internal/ci/buildkite/ cue.mod/module.cue +``` + +Make sure to include your slightly modified YAML pipeline files in +`.buildkite/` along with all the new files in `internal/ci/buildkite/` and +your `cue.mod/module.cue` file. + +Commit your files to git, with an appropriate commit message: + +:computer: `terminal` +```sh +git commit -m "ci: create CUE sources for Buildkite pipelines" +``` + +## Conclusion + +**Well done - your Buildkite pipeline files have been imported into CUE!** + +They can now be managed using CUE, leading to safer and more predictable +changes. The use of a schema to check your pipelines means that you +will catch and fix many types of mistake earlier than before, without waiting +for the slow "git add/commit/push; check if CI fails" cycle. + +From now on, each time you make a change to a CUE pipeline file, immediately +regenerate the YAML files required by Buildkite, and commit your changes +to all the CUE and YAML files. For example: + +:computer: `terminal` +```sh +cue cmd regenerate ./internal/ci/buildkite/ # the "./" prefix is required +git add .buildkite/ internal/ci/buildkite/ +git commit -m "ci: added new release pipeline" # example message +```