-
Notifications
You must be signed in to change notification settings - Fork 20
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* chore: reorder config alphabetically * chore: add openstack-cloud-yaml config option * chore: reorder charm state alphabetically * chore: reorder charm state alphabetically * chore: reorder requirements.txt alphabetically * feat: initialize openstack connection * ensure cloud config dir is present * test microstack installation on gh hosted ones * test microstack installation on gh hosted ones * use two-xlarge * use one step * checkin current workflow * Add integration test * Specify microk8s-addons (disable rbac) * bootstrap lxd controller * give a try with xlarge * Refactor and add unit tests * Add call to test connection * Move openstack check to __init__ * Revert "Move openstack check to __init__" This reverts commit 103123d. * Remove status in wait_for_idle * Rename config name * Cleanup * update script * update script with shellcheck directive * Fix requirements * Use clouds.yaml for integration test * Generate clouds.yaml using sunbeam * Fix openstack connect test in setup * chore: ignore testing clouds.yaml * chore: validate input type * chore: move type validation to charm state module * feat: initial openstack build image * feat: reduce image size * feat: create instance w/ cloud-init userdata script * chore: define literal types for runner application * chore: cloudimg multiarch support * chore: factor out get runner application func * chore: type hint openstack clouds yaml dict * chore: build on test mode only * chore: refactor too inject github client than instantiate from token * test: test for openstack servers * test: openstack integration test * test: use only openstack * fix: helpers import * fix:connect cloud * debug * chore: remove unattended upgrades * fix: use vm w/ machine constraints * fix: use units in mem/disk size * fix: translate to gigabytes * fix: increase mem * fix: increase timeout * chore remove tmate debug * chore: don't use lxd profile for openstack test * chore: run modeprobe br_netfilter before lxc launch * chore: revert image brnetfilter changes * feat: on_install for openstack mode * test: add additional charm config * chore: move openstack handle to on_install hook * chore: block all hooks on openstack config * fix: add back e2e test run workflow * chore: proxy configs * chore: wait for server creation * fix: create flavor * chore: test flavour to small * debug * fix: add retry * fix: use bootstrapped microstack for image buildtesting * chore: add comment to reemove cluster bootstrap * fix: sunbeam user show (user list permission denied) * fix: stringify proxy vars * test: unit tests for openstack manager * add retry * Revert "test: use only openstack" This reverts commit cd8f391. * remove tmate debug * add back license * remove extra line * lint & test fixes * refactor openstack_manager to openstack_cloud module * refactor block_on_openstack_config * update keyerror message * test: fix test for updated error message * fix: pass state to block_on_openstack_config * refactor get_runner_applications * add pipefail for build images * add cloudconfig type * refactor modules for unit testing * fix style issues * rename build image to build lxd image * check for different subprocess error * conditional /etc/environment update * use already defined constant * chore: move functions per contributing guid * chore: refactor _build_image_command * lint fix * test: mark expected failure test * update create connection * remove unauthorized exception handling * remove unauthorized exception handling * add back unauthorized handling * test: fix context manager related failures * test: remove unused var * chore: remove debug print --------- Co-authored-by: Christopher Bartz <[email protected]>
- Loading branch information
Showing
31 changed files
with
1,763 additions
and
779 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,204 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Copyright 2024 Canonical Ltd. | ||
| # See LICENSE file for licensing details. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # GitHub runner bin args | ||
| RUNNER_TAR_URL="$1" | ||
|
|
||
| # Proxy args | ||
| HTTP_PROXY="$2" | ||
| HTTPS_PROXY="$3" | ||
| NO_PROXY="$4" | ||
| DOCKER_PROXY_SERVICE_CONF="$5" | ||
| DOCKER_PROXY_CONF="$6" | ||
|
|
||
| # retry function | ||
| retry() { | ||
| local command="$1" | ||
| local wait_message="$2" | ||
| local max_try="$3" | ||
|
|
||
| local attempt=0 | ||
|
|
||
| while ! $command | ||
| do | ||
| attempt=$((attempt + 1)) | ||
| if [[ attempt -ge $max_try ]]; then | ||
| return | ||
| fi | ||
|
|
||
| echo "$wait_message" | ||
| sleep 10 | ||
| done | ||
| } | ||
|
|
||
| # cleanup any existing mounts | ||
| cleanup() { | ||
| sudo umount /mnt/ubuntu-image/dev/ || true | ||
| sudo umount /mnt/ubuntu-image/proc/ || true | ||
| sudo umount /mnt/ubuntu-image/sys/ || true | ||
| sudo umount /mnt/ubuntu-image || true | ||
| sudo qemu-nbd --disconnect /dev/nbd0 | ||
| } | ||
|
|
||
| # Check if proxy variables set, doesn't exist or is a different value then update. | ||
| if [[ -n "$HTTP_PROXY" ]]; then | ||
| if ! grep -q "HTTP_PROXY=" /etc/environment || ! grep -q "HTTP_PROXY=$HTTP_PROXY" /etc/environment; then | ||
| sed -i "/^HTTP_PROXY=/d" /etc/environment | ||
| echo "HTTP_PROXY=$HTTP_PROXY" >> /etc/environment | ||
| fi | ||
| if ! grep -q "http_proxy=" /etc/environment || ! grep -q "http_proxy=$HTTP_PROXY" /etc/environment; then | ||
| sed -i "/^http_proxy=/d" /etc/environment | ||
| echo "http_proxy=$HTTP_PROXY" >> /etc/environment | ||
| fi | ||
| if ! grep -q "Acquire::http::Proxy" /etc/apt/apt.conf || ! grep -q "Acquire::http::Proxy \"$HTTP_PROXY\";" /etc/apt/apt.conf; then | ||
| sed -i "/^Acquire::http::Proxy/d" /etc/apt/apt.conf | ||
| echo "Acquire::http::Proxy \"$HTTP_PROXY\";" >> /etc/apt/apt.conf | ||
| fi | ||
| fi | ||
|
|
||
| if [[ -n "$HTTPS_PROXY" ]]; then | ||
| if ! grep -q "HTTPS_PROXY=" /etc/environment || ! grep -q "HTTPS_PROXY=$HTTPS_PROXY" /etc/environment; then | ||
| sed -i "/^HTTPS_PROXY=/d" /etc/environment | ||
| echo "HTTPS_PROXY=$HTTPS_PROXY" >> /etc/environment | ||
| fi | ||
| if ! grep -q "https_proxy=" /etc/environment || ! grep -q "https_proxy=$HTTPS_PROXY" /etc/environment; then | ||
| sed -i "/^https_proxy=/d" /etc/environment | ||
| echo "https_proxy=$HTTPS_PROXY" >> /etc/environment | ||
| fi | ||
| if ! grep -q "Acquire::https::Proxy" /etc/apt/apt.conf || ! grep -q "Acquire::https::Proxy \"$HTTPS_PROXY\";" /etc/apt/apt.conf; then | ||
| sed -i "/^Acquire::https::Proxy/d" /etc/apt/apt.conf | ||
| echo "Acquire::https::Proxy \"$HTTPS_PROXY\";" >> /etc/apt/apt.conf | ||
| fi | ||
| fi | ||
|
|
||
| if [[ -n "$NO_PROXY" ]]; then | ||
| if ! grep -q "NO_PROXY=" /etc/environment || ! grep -q "NO_PROXY=$NO_PROXY" /etc/environment; then | ||
| sed -i "/^NO_PROXY=/d" /etc/environment | ||
| echo "NO_PROXY=$NO_PROXY" >> /etc/environment | ||
| fi | ||
| if ! grep -q "no_proxy=" /etc/environment || ! grep -q "no_proxy=$NO_PROXY" /etc/environment; then | ||
| sed -i "/^no_proxy=/d" /etc/environment | ||
| echo "no_proxy=$NO_PROXY" >> /etc/environment | ||
| fi | ||
| fi | ||
|
|
||
| # Architecture args | ||
| ARCH=$(uname -m) | ||
| if [[ $ARCH == 'aarch64' ]]; then | ||
| BIN_ARCH="arm64" | ||
| elif [[ $ARCH == 'arm64' ]]; then | ||
| BIN_ARCH="arm64" | ||
| elif [[ $ARCH == 'x86_64' ]]; then | ||
| BIN_ARCH="amd64" | ||
| else | ||
| echo "Unsupported CPU architecture: $ARCH" | ||
| return 1 | ||
| fi | ||
|
|
||
| # qemu-utils required to unpack qcow image | ||
| sudo DEBIAN_FRONTEND=noninteractive apt-get install qemu-utils libguestfs-tools -y | ||
|
|
||
| # enable network block device | ||
| sudo modprobe nbd | ||
|
|
||
| # cleanup any existing mounts | ||
| cleanup | ||
|
|
||
| retry "sudo wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-$BIN_ARCH.img \ | ||
| -O jammy-server-cloudimg-$BIN_ARCH.img" "Downloading cloud image" 3 | ||
|
|
||
| # resize image - installing dependencies requires more disk space | ||
| sudo qemu-img resize jammy-server-cloudimg-$BIN_ARCH.img +1.5G | ||
|
|
||
| # mount nbd | ||
| echo "Connecting network block device to image" | ||
| sudo qemu-nbd --connect=/dev/nbd0 jammy-server-cloudimg-$BIN_ARCH.img | ||
| sudo mkdir -p /mnt/ubuntu-image | ||
| retry "sudo mount -o rw /dev/nbd0p1 /mnt/ubuntu-image" "Mounting nbd0p1 device" 3 | ||
|
|
||
| # mount required system dirs | ||
| echo "Mounting sys dirs" | ||
| retry "sudo mount --bind /dev/ /mnt/ubuntu-image/dev/" "Mounting /dev/" 3 | ||
| retry "sudo mount --bind /proc/ /mnt/ubuntu-image/proc/" "Mounting /proc/" 3 | ||
| retry "sudo mount --bind /sys/ /mnt/ubuntu-image/sys/" "Mounting /sys/" 3 | ||
| sudo rm /mnt/ubuntu-image/etc/resolv.conf -f | ||
| sudo cp /etc/resolv.conf /mnt/ubuntu-image/etc/resolv.conf | ||
|
|
||
| # resize mount | ||
| echo "Resizing mounts" | ||
| sudo growpart /dev/nbd0 1 # grow partition size to available space | ||
| sudo resize2fs /dev/nbd0p1 # resize fs accordingly | ||
|
|
||
| # chroot and install dependencies | ||
| echo "Installing dependencies in chroot env" | ||
| sudo chroot /mnt/ubuntu-image/ <<EOF | ||
| set -e | ||
| # Commands within the chroot environment | ||
| df -h # print disk free space | ||
| DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get update -yq | ||
| DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get upgrade -yq | ||
| DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get install docker.io npm python3-pip shellcheck jq wget unzip gh -yq | ||
| ln -s /usr/bin/python3 /usr/bin/python | ||
| # Uninstall unattended-upgrades, to avoid lock errors when unattended-upgrades is active in the runner | ||
| DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get purge unattended-upgrades -yq | ||
| /usr/sbin/useradd -m ubuntu | ||
| /usr/bin/npm install --global yarn | ||
| /usr/sbin/groupadd microk8s | ||
| /usr/sbin/usermod -aG microk8s ubuntu | ||
| /usr/sbin/usermod -aG docker ubuntu | ||
| /usr/bin/chmod 777 /usr/local/bin | ||
| # Proxy configs | ||
| if [[ -n "$HTTP_PROXY" ]]; then | ||
| /usr/bin/npm config set proxy "$HTTP_PROXY" | ||
| fi | ||
| if [[ -n "$HTTPS_PROXY" ]]; then | ||
| /usr/bin/npm config set https-proxy "$HTTPS_PROXY" | ||
| fi | ||
| if [[ -n "$DOCKER_PROXY_SERVICE_CONF" ]]; then | ||
| mkdir -p /etc/systemd/system/docker.service.d | ||
| echo "$DOCKER_PROXY_SERVICE_CONF" > /etc/systemd/system/docker.service.d/http-proxy.conf | ||
| fi | ||
| if [[ -n "$DOCKER_PROXY_CONF" ]]; then | ||
| mkdir -p /root/.docker | ||
| echo "$DOCKER_PROXY_CONF" > /root/.docker/config.json | ||
| mkdir -p /home/ubuntu/.docker | ||
| echo "$DOCKER_PROXY_CONF" > /home/ubuntu/.docker/config.json | ||
| fi | ||
| # Reduce image size | ||
| /usr/bin/npm cache clean --force | ||
| DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get clean | ||
| # Download and verify checksum of yq | ||
| /usr/bin/wget "https://github.com/mikefarah/yq/releases/latest/download/yq_linux_$BIN_ARCH" -O "yq_linux_$BIN_ARCH" | ||
| /usr/bin/wget https://github.com/mikefarah/yq/releases/latest/download/checksums -O checksums | ||
| /usr/bin/wget https://github.com/mikefarah/yq/releases/latest/download/checksums_hashes_order -O checksums_hashes_order | ||
| /usr/bin/wget https://github.com/mikefarah/yq/releases/latest/download/extract-checksum.sh -O extract-checksum.sh | ||
| /usr/bin/bash extract-checksum.sh SHA-256 "yq_linux_$BIN_ARCH" | /usr/bin/awk '{print \$2,\$1}' | /usr/bin/sha256sum -c | /usr/bin/grep OK | ||
| rm checksums checksums_hashes_order extract-checksum.sh | ||
| /usr/bin/chmod 755 yq_linux_$BIN_ARCH | ||
| /usr/bin/mv yq_linux_$BIN_ARCH /usr/bin/yq | ||
| # Download runner bin and verify checksum | ||
| mkdir -p /home/ubuntu/actions-runner && cd /home/ubuntu/actions-runner | ||
| /usr/bin/curl -o /home/ubuntu/actions-runner.tar.gz -L $RUNNER_TAR_URL | ||
| /usr/bin/tar xzf /home/ubuntu/actions-runner.tar.gz | ||
| rm /home/ubuntu/actions-runner.tar.gz | ||
| chown -R ubuntu /home/ubuntu/ | ||
| EOF | ||
|
|
||
| # sync & cleanup | ||
| echo "Syncing" | ||
| sudo sync | ||
| cleanup | ||
|
|
||
| # Reduce image size by removing sparse space & compressing | ||
| sudo virt-sparsify --compress jammy-server-cloudimg-$BIN_ARCH.img jammy-server-cloudimg-$BIN_ARCH-compressed.img |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.