garm v0.1.5

[gabriel-samfira]

Welcome to GARM v0.1.5!

Before we dive into the highlights, as always a few notes.

Database upgrade ⚠️

This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes. Database downgrades are not supported. If you need to go back to a previous version, please also restore the DB from backup.

To safely create a backup of you DB, you can use the sqlite3 CLI:

# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.db

Breaking changes ⚠️

This version has moved a number of options from the config file, to the database. During the upgrade process from v0.1.4 to v0.1.5 please do not make any changes to your current config. GARM will read the config and migrate any of the settings present there, to the DB, automatically. After you start GARM with the new version and the migration has finished (it should be just a matter of seconds), you can then edit your config and remove the relevant fields.

Things that have been moved from the config to the database:

• The metadata_url, callback_url and webhook_url settings have now been moved to the database in the controller table. These settings can now be updated using the garm-cli command line tool.
• Github credentials - credentials are now created via the garm-cli command line tool. Details bellow.

Highlights

This version is packed with features and changes. Although GARM can handle a large amount of runners with ease, we are working towards enabling it to eventually scale out. As a result, we've started moving some of the settings that were in the config file, to the database. The goal is to eventually have a single source of truth for those settings and enable GARM to react to any changes immediately without the need to reload the app. This is just one motivation behind the changes in this release, but as you'll see, there are many other changes that enable you to do more with GARM.

GitHub endpoints

With the migration of credentials to the database, GARM now has the notion of "github endpoints". A github endpoint represents a deployment of GHES or GitHub itself. It is a way to tell GARM to which API endpoints it needs to connect in order to use the supplied credentials or to execute operations against a particular entity (repo, org or enterprise).

All github credentials and all entities are now bound to a github endpoint. This ensures consistency and correctness when dealing with these types of resources.

See the github endpoint section of the documentation.

Github App Support

We now have the ability to configure GARM to use GitHub Apps to access the github API. This gives us a much higher rate limit for API calls (15.000/hour compared to 5000/hour as is the case of PATs).

Check out the github credentials documentation section for more info on how to configure app credentials, as well as the required credentials for GitHub Apps.

All sensitive info is encrypted at rest and is never returned by the API.

Runner default labels

Starting with version v2.305.0 of the GitHub actions runner, the default lables ($ARCH, $OS and self-hosted) are no longer added if the --no-default-labels flag is used. GARM now takes advantage of this feature to create runners without default labels. The default labels can still be added explicitly if you require them.

Pool balancing strategies

You can now specify two modes of balancing for jobs that get handled by a configured entity. The balancing mode can be set on the entity itself, and this will influence how the pool manager for that entity will behave. The two modes of operation are:

• roundrobin - During the consolidation loop, if we have multiple queued jobs, the roundrobin strategy will balance each job to a different matching pool. This is useful if you want to spread the workload evenly across multiple clouds or regions. This is the default.
• pack - The pack strategy will try to fill up the first pool before moving on to the next, This is useful if you have cheap runners you want to use first before moving on to pools of more expensive runners.

To set the balancing stratedy for a repo, org or enterprise you can use the following command:

garm-cli repo update --pool-balancer-type pack <repo_id>

Additionally, pools now have a priority field. This priority field is used when sorting pools that match jobs which need to be handled. You can set the priority when you create the pool or you can update it later. The higher the number, the more likely the pool is to be at the beginning of the list.

A new events websocket endpoint

We now have a new websocket endpoint that can be used by projects which integrate with GARM. This websocket endpoint streams database events as they happen in GARM itself, allowing applications to not need to poll for updates.

Here is a demo of an earlier implementation, in action:

For details regarding the current implementation and how to use it, check out the documentation.

There are many other fixes and features added to this release, a lot of which are internal to GARM but they do have a positive impact on stability and reliability. I invite you to check out the full list of changes, bellow.

Thanks to everyone that has contributed to this release, be it via pull requests, raising issues or helping us debug weird behavior!

What's Changed

• Dockerfile: Build actually static binaries by @jepio in Dockerfile: Build actually static binaries #215
• chore: refactor metrics endpoint by @bavarianbidi in chore: refactor metrics endpoint #216
• extend metrics for github and provider executions by @bavarianbidi in extend metrics for github and provider executions #217
• Introduce golangci config and make help by @bavarianbidi in Introduce golangci config and make help #218
• Allow integration tests to run locally by @gabriel-samfira in Allow integration tests to run locally #219
• Install dependencies and set RUN_USER by @gabriel-samfira in Install dependencies and set RUN_USER #220
• Switch to apg for password gen by @gabriel-samfira in Switch to apg for password gen #221
• Add script and service template by @gabriel-samfira in Add script and service template #222
• Use user service by @gabriel-samfira in Use user service #223
• simplify integration test job by @gabriel-samfira in simplify integration test job #224
• Fix variable name by @gabriel-samfira in Fix variable name #225
• Add GitHub App support by @gabriel-samfira in Add GitHub App support #228
• Add ToC to using_garm.md by @gabriel-samfira in Add ToC to using_garm.md #229
• Allow bypassing Unauthorized error when deleting runner by @gabriel-samfira in Allow bypassing Unauthorized error when deleting runner #231
• Add job info in runner list by @gabriel-samfira in Add job info in runner list #232
• Add pool balancing strategy by @gabriel-samfira in Add pool balancing strategy #233
• Add OCI to provider list by @gabriel-samfira in Add OCI to provider list #234
• Refactor internal gh client by @gabriel-samfira in Refactor internal gh client #235
• Fix JIT config with empty runner group name by @gabriel-samfira in Fix JIT config with empty runner group name #237
• Check for nil pointer before dereferencing by @gabriel-samfira in Check for nil pointer before dereferencing #238
• Fix potential nil pointer dereference in cli by @gabriel-samfira in Fix potential nil pointer dereference in cli #239
• Deduplicate db code by @gabriel-samfira in Deduplicate db code #240
• Slightly simplify code by @gabriel-samfira in Slightly simplify code #241
• Remove some code, move some code around by @gabriel-samfira in Remove some code, move some code around #242
• Update dependencies and tests by @gabriel-samfira in Update dependencies and tests #244
• Move github credentials to the database by @gabriel-samfira in Move github credentials to the database #243
• Add TOC by @gabriel-samfira in Add TOC #247
• Remove check for duplicate pool by @gabriel-samfira in Remove check for duplicate pool #246
• New runners without default labels by @bavarianbidi in New runners without default labels #249
• Update go-swagger and run generate by @gabriel-samfira in Update go-swagger and run generate #251
• Add note about stable release documentation by @gabriel-samfira in Add note about stable release documentation #252
• fix: use the american english type of cancelled by @bavarianbidi in fix: use the american english type of cancelled #254
• fix: remove unnecessary github api call by @bavarianbidi in fix: remove unnecessary github api call #255
• Move URLs from default section of config to DB by @gabriel-samfira in Move URLs from default section of config to DB #256
• Fix typos by @gabriel-samfira in Fix typos #259
• Refactor Integration tests by @fabi200123 in Refactor Integration tests #261
• Add database watcher by @gabriel-samfira in Add database watcher #263
• Consume events before testing by @gabriel-samfira in Consume events before testing #266
• Remove duplicate of watcher.SetWatcher() by @fabi200123 in Remove duplicate of watcher.SetWatcher() #267
• Lower the log level of ignored jobs by @gabriel-samfira in Lower the log level of ignored jobs #268
• Use watcher and get rid of RefreshState() by @gabriel-samfira in Use watcher and get rid of RefreshState() #269
• Allow configuration of job backoff interval by @gabriel-samfira in Allow configuration of job backoff interval #274
• Add version to controller info response by @gabriel-samfira in Add version to controller info response #276
• Add event stream by @gabriel-samfira in Add event stream #275
• Reword the section about labels by @gabriel-samfira in Reword the section about labels #277
• Fix: Scope entities to endpoint by @gabriel-samfira in Fix: Scope entities to endpoint #279
• Generalize the websocket reader by @gabriel-samfira in Generalize the websocket reader #280
• Rename websocket URLs by @gabriel-samfira in Rename websocket URLs #281
• Update garm-provider-common by @gabriel-samfira in Update garm-provider-common #282
• Add JSON tags to the ChangePayload struct by @gabriel-samfira in Add JSON tags to the ChangePayload struct #283

New Contributors

• @jepio made their first contribution in Dockerfile: Build actually static binaries #215
• @fabi200123 made their first contribution in Refactor Integration tests #261

Full Changelog: v0.1.4...v0.1.5

---

This discussion was created from the release garm v0.1.5.