Ubuntu vs Amazon linux

[npalm]

Since we started the module we used amazon linux as base of the runners, only reason for the choice is the good amazon support and we use it for other setups. So re-using bits and peaces is much easier. But it seems that for the runner ubuntu looks a more logical choice. Some actions/setup-* actions only support ubuntu (for example python).

Therefore I have updated the example created by @sjagoe to support logging, docker (rootless docker) and a non privileges user (see #433). Just wondering what preferences you have? Should we stay on amazon linux or move by the time to ubuntu for running actions workflows?

[sjagoe]

I'm obviously in favour of ubuntu being default, for the reasons you mention.

In addition, our goal with the AWS hosted runners is to have effectively just larger instances that are 1:1 compatible with the github runners. So while ubuntu is useful as-is, it's not quite enough for how we use the runners, switching jobs between Github and our own fairly arbitrarily based on CPU and RAM needs.

What we did to address this was build a https://www.packer.io/ template to build an AMI containing a lot of the packages used by Github runners, based on the scripts and configuration provided by https://github.com/actions/virtual-environments/tree/main/images/linux

Based on that AMI, the only thing that needs to be installed on runner launch is the github actions runner itself.

This is probably outside of the scope of this project, though.

[npalm]

I'm obviously in favour of ubuntu being default, for the reasons you mention.

In addition, our goal with the AWS hosted runners is to have effectively just larger instances that are 1:1 compatible with the github runners. So while ubuntu is useful as-is, it's not quite enough for how we use the runners, switching jobs between Github and our own fairly arbitrarily based on CPU and RAM needs.

What we did to address this was build a https://www.packer.io/ template to build an AMI containing a lot of the packages used by Github runners, based on the scripts and configuration provided by https://github.com/actions/virtual-environments/tree/main/images/linux

Based on that AMI, the only thing that needs to be installed on runner launch is the github actions runner itself.

This is probably outside of the scope of this project, though.

I noticed as well that the github virtual environments are just packer based. Just wondering do you have any automation scripts available as open source to build AWS AMI's?

[sjagoe]

They're not open, but I can start an internal discussion on making them available. At the moment it's mixed into another repository, though.

Maybe I can look into more closely building from the github packer templates, instead though, and open that. My current implementation is very custom and a bit of a hack to use their installation scripts.

[sjagoe]

@npalm I've made our packer builder available here https://github.com/smartlyio/github-actions-environment

It's not great because it does our own setup of the VM, and then cherry-picks bits of the github virtual-environments scripts to install; it grew organically from our own minimal images and started importing the upstream github scripts to get closer to their builds.

If I was re-doing it now, I would try to fully replicate their build: I would build directly on the github virtual-environments repository, programmatically replacing the builders section of their packer template and just setting up the right environment to run their whole build in rather than cherry-picking bits that we need.

[sjagoe]

I have a branch in that repository that attempts to do the builds directly with provisioners from github. I'm busy running a test build with packer from that, but it will take a while to complete. Hopefully I can test the image built from this next week or so.

That would be a nice starting point because then we would be able to build/provide images very close to upstream github.

[mcaulifn]

Similarly, we cloned the virtual-environments repo, modified Packer to use EBS build, and slimmed down the image. I going to look into if it is possible to template that packer file out so it is more modular and also EBS/Azure compatible but I have not had a chance to do so.

[sjagoe]

@npalm I have just switched over to runners built with smartlyio/github-actions-environment#1 (plus a small change to our user-data script to create the runner user, with the below snippet).

So far it seems good; the runner starts and accepts jobs. We have had successful builds.

To create the user at startup, I added this to the start of my user-data script:

adduser --gecos ',,,' --uid 1001 --disabled-password runner
usermod -a -G docker runner

[npalm]

Thanks for sharing, I hope to run a POC in the next weeks as well.

[oradwell]

we tend to use ubuntu for all our servers. However, I didn't want to spend time switching the runners created by terraform-aws-github-runner unless there were actual issues. I found no issues mainly because we run everything in docker. I'm happy that the module installs docker into amazon linux

[sjagoe]

Prompted by @oradwell above, I just want to update: Our packer scripts here do still work to build an ubuntu image that is as close as I could get it to Github's own runners https://github.com/smartlyio/github-actions-environment/ (by directly using the upstream https://github.com/actions/virtual-environments/ packer builder)
It just needs changes to the variables, and an aws cli profile set up for authentication.

We use that to follow Github's runner updates so that we can seamlessly move jobs between Github runners and our own.

[mahela-aws]

@sjagoe I'm trying to get runners working with ubuntu-focal but so far I have been unsuccessful with this. we have created our own custom image based on packer template provided here https://github.com/philips-labs/terraform-aws-github-runner/blob/develop/images/ubuntu-focal/github_agent.ubuntu.pkr.hcl
but unfortunately runners aren't getting registered with github. we already have a working setup with amzn-linux-2 but we are trying to switch over to ubuntu runners for the very reason give above by @npalm we need to use setup-python action which is only supported by ubuntu. appreciate if you guys can provide some support

[oradwell]

can't you just use the docker container image for python?

although it's easier to do without it, we found that using docker we can test the workflow outside GitHub Actions

[mahela-aws]

we are trying incorporate pre-commit in our workflows setup-python is required by this