-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add imagefactory docker compose example #58
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
.env | ||
keys/* | ||
schematics/* |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
# Siderolabs self-hosted imagefactory example | ||
|
||
This code runs [sidero imagefactory](https://github.com/siderolabs/image-factory) in [docker compose](https://docs.docker.com/compose/). | ||
|
||
It also deploys a few companion components: | ||
* upstream `ghcr.io` [registry](https://distribution.github.io/distribution/) [mirror](https://distribution.github.io/distribution/recipes/mirror/) to avoid potential upstream rate limitings and speed up builds by caching previously pulled image layers | ||
* a script that applies prepared `talos` [image schematics](https://www.talos.dev/v1.8/learn-more/image-factory/#schematics) | ||
* a [registry](https://distribution.github.io/distribution/) used as storage and cache for the generated images | ||
|
||
# how to use | ||
|
||
tested with `talos` version `v1.8.1`, via [iPXE boot](https://www.talos.dev/v1.8/talos-guides/install/bare-metal-platforms/pxe/) and directly applying generated images to disk via `hcloud` (use it's [packer](https://developer.hashicorp.com/packer/integrations/hetznercloud/hcloud) integration and point [this](https://github.com/siderolabs/contrib/blob/9cd1e1c9d2469b77d2278eb07e7f61c09bb32d40/examples/terraform/hcloud/packer/hcloud_talosimage.pkr.hcl#L18) URL to your `imagefactory` instance). | ||
|
||
## preparation | ||
|
||
some preparation is required. | ||
|
||
### signing keys | ||
see [official docs](https://github.com/siderolabs/image-factory?tab=readme-ov-file#development). | ||
|
||
```shell | ||
mkdir -pv keys | ||
openssl ecparam -name prime256v1 -genkey -noout -out keys/cache-signing-key.key | ||
``` | ||
|
||
### schematics | ||
|
||
Refer to the [official docs](https://www.talos.dev/v1.8/learn-more/image-factory/#schematics) on how to create these. | ||
The [script](./scripts/sync-schematics.sh) will find and apply all files in `schematics/*.yaml`. | ||
|
||
Example: | ||
```yaml | ||
# schematics/example.yaml | ||
customization: | ||
extraKernelArgs: | ||
- gfxmode=1280x1024 | ||
- console=ttyS0,115200 | ||
- net.ifnames=0 | ||
- talos.platform=metal | ||
systemExtensions: | ||
officialExtensions: | ||
- siderolabs/amd-ucode | ||
- siderolabs/fuse3 | ||
- siderolabs/intel-ucode | ||
- siderolabs/iscsi-tools | ||
- siderolabs/qemu-guest-agent | ||
- siderolabs/tailscale | ||
- siderolabs/util-linux-tools | ||
meta: | ||
- key: 12 | ||
value: | | ||
machineLabels: | ||
env: prod | ||
type: controlplane | ||
``` | ||
|
||
### environment variables | ||
|
||
copy the [docker compose env file](https://docs.docker.com/compose/how-tos/environment-variables/variable-interpolation/#env-file) and adjust the example values. | ||
|
||
```shell | ||
cp env.example .env | ||
vim .env | ||
``` | ||
|
||
Adjust all domains and `EXT_IP` to where you want to expose your `imagefactory` instance. | ||
This is relevant for payloads sent to `iPXE` clients and URLs generated in the UI. | ||
|
||
## run | ||
|
||
after preparation is done, run `docker compose up -d`. | ||
|
||
# miscellaneous & troubleshooting | ||
|
||
This is a community contribution so expect no official support. | ||
Some trouble I ran into: | ||
|
||
## TLS | ||
|
||
The configuration used does *not* deploy TLS, so you should put this behind something like a reverse proxy that does. | ||
|
||
## connection timeouts | ||
|
||
Image generation can take some time, so clients might have to increase their connection timeout limits. If images are cached, [TTFB](https://en.wikipedia.org/wiki/Time_to_first_byte) is very short, if not `TTFB` can take up to several minutes. | ||
|
||
## iPXE and https | ||
|
||
If you want to [iPXE](https://ipxe.org)-boot from this via `https`, keep in mind that by default `iPXE` does *not* support `https` and you need to compile your own, enabling [this](https://ipxe.org/buildcfg/download_proto_https) flag. This is a pitfall for reverse proxies that automatically redirect plaintext `http` requests to `https`. | ||
|
||
## URLs not working | ||
|
||
There is a tiny problem in the `imagefactory` frontend: The URLs generated contain the external domain used and it is duplicated for some reason. This is particularly mean because the URL _visible_ in the UI looks correct, but the `HTML` `href` is not. | ||
Make sure to sanitize the URL before use, the resulting URL works as expected. Not sure yet as to _why_ this happens (misconfiguration or might be a bug). | ||
Querying the `API` seems to return the correct URL. | ||
|
||
## registry resource consumption | ||
|
||
* When building a large number of images, make sure to provide sufficient storage to the `registry` container and monitor `docker volumes` as it grows in size quite rapidly. | ||
* the image generation process is compute heavy and can take some time, depending on the compute power available. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
services: | ||
# generated images are pushed to this registry | ||
registry: | ||
image: registry:2 | ||
ports: | ||
- ${EXTERNAL_IP:-127.0.0.1}:5000:5000 | ||
volumes: | ||
# hint: when generating a large number of different schemas, this volume can grow quite large | ||
- registry:/var/lib/registry:rw | ||
# upstream ghcr mirror, caches previously pulled image layers and prevents rate limiting | ||
registry-ghcr: | ||
image: registry:2 | ||
environment: | ||
REGISTRY_PROXY_REMOTEURL: http://ghcr.io | ||
volumes: | ||
- registry-ghcr:/var/lib/registry:rw | ||
# triggers image builds for all schematics defined in `./schematics/*.yaml` | ||
schematics: | ||
image: alpine:3 | ||
environment: | ||
IMAGE_FACTORY_URL: http://imagefactory:6000 | ||
REGISTRY_URL: registry:5000 | ||
TALOS_VERSION: ${SCHEMATICS_TALOS_VERSION?} | ||
ARCH: ${SCHEMATICS_TALOS_ARCH?} | ||
VALIDATE: ${SCHEMATICS_VALIDATE?} | ||
SLEEP_TIME: ${SCHEMATICS_SLEEP_TIME?} | ||
command: > | ||
/scripts/sync-schematics.sh | ||
volumes: | ||
- ${PWD}/scripts:/scripts:ro | ||
- ${PWD}/schematics:/schematics:ro | ||
# container running the actual imagefactory | ||
imagefactory: | ||
image: ghcr.io/siderolabs/image-factory:${IMGFAC_VERSION?} | ||
# required for losetup | ||
privileged: true | ||
volumes: | ||
- ${PWD}/keys:/keys:ro | ||
# required for losetup | ||
- /dev:/dev | ||
ports: | ||
- ${EXTERNAL_IP:-127.0.0.1}:6000:6000 | ||
command: > | ||
-http-port=:6000 | ||
-external-url=http://${IMGFAC_EXTERNAL_URL?} | ||
-external-pxe-url=http://${IMGFAC_EXTERNAL_URL?} | ||
-cache-signing-key-path=/keys/cache-signing-key.key | ||
-cache-repository=registry:5000/cache | ||
-insecure-cache-repository=true | ||
-image-registry=registry-ghcr:5000 | ||
-insecure-image-registry=true | ||
-schematic-service-repository=registry:5000/image-factory/schematic | ||
-insecure-schematic-service-repository | ||
-installer-internal-repository=registry:5000/siderolabs | ||
-insecure-installer-internal-repository=true | ||
-installer-external-repository=${IMGFAC_EXT_REPO?}/siderolabs | ||
-secureboot=${IMGFAC_SECUREBOOT?} | ||
|
||
volumes: | ||
registry: | ||
registry-tls: | ||
registry-ghcr: |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
EXTERNAL_IP=127.0.0.1 | ||
|
||
SCHEMATICS_TALOS_VERSION=1.8.1 | ||
SCHEMATICS_TALOS_ARCH=amd64 | ||
SCHEMATICS_VALIDATE=false | ||
SCHEMATICS_SLEEP_TIME=600 | ||
|
||
IMGFAC_VERSION=v0.5.0 | ||
IMGFAC_EXTERNAL_URL=imgfac.example.com | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this should be a proper URL (when passed as an argument), this might be the reason the UI doesn't work properly, e.g. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd argue since no reverse-proxy is part of this scenario, we can assume it is always See 9d937af |
||
IMGFAC_EXT_REPO=reg.imgfac.example.com | ||
IMGFAC_SECUREBOOT=false |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,70 @@ | ||
#!/bin/ash | ||
set -eo pipefail | ||
|
||
trap "exit 0" SIGINT SIGTERM | ||
|
||
: ${IMAGE_FACTORY_URL:?} | ||
: ${REGISTRY_URL:?} | ||
: ${TALOS_VERSION:?} | ||
: ${ARCH:?} | ||
: ${VALIDATE:?} | ||
: ${SLEEP_TIME:?} | ||
|
||
apk add crane yq | ||
|
||
RESULTS_FILE="${RESULTS_FILE:-/tmp/results}" | ||
while true; do | ||
echo '' > "${RESULTS_FILE}" | ||
for SCHEMATIC in /schematics/*.yaml ; do | ||
# this triggers image generation based on the schema provided | ||
# docs: https://github.com/siderolabs/image-factory?tab=readme-ov-file#post-schematics | ||
echo "apply ${SCHEMATIC}" | ||
RESPONSE_FILE=/tmp/wget-response.json | ||
wget \ | ||
--header 'Content-Type: application/yaml' \ | ||
-O "${RESPONSE_FILE}" \ | ||
--post-file=${SCHEMATIC} \ | ||
${IMAGE_FACTORY_URL}/schematics \ | ||
|
||
# parse the image ID from the response | ||
SCHEMA_ID=$(yq .id < "${RESPONSE_FILE}") | ||
if test -z "${SCHEMA_ID}" ; then | ||
echo 'SCHEMA_ID was empty' | ||
exit 1 | ||
fi | ||
TMP_FILE="/tmp/${SCHEMA_ID}.tar" | ||
rm "${RESPONSE_FILE}" | ||
|
||
# docs: https://github.com/siderolabs/image-factory?tab=readme-ov-file#get-imageschematicversionpath | ||
echo 'download container' | ||
wget \ | ||
-O ${TMP_FILE} \ | ||
${IMAGE_FACTORY_URL}/image/${SCHEMA_ID}/${TALOS_VERSION}/installer-${ARCH}.tar | ||
|
||
# optional: this calls `crane validate <image>`, validating the generated image is well formed | ||
# docs: https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane_validate.md | ||
if [ "${VALIDATE}" == 'true' ] ; then | ||
echo 'validate container' | ||
crane validate --tarball ${TMP_FILE} | ||
fi | ||
|
||
echo 'publish container' | ||
crane push \ | ||
--insecure \ | ||
${TMP_FILE} \ | ||
${REGISTRY_URL}/installer/${SCHEMA_ID}:${TALOS_VERSION} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure why this is needed, Image Factory itself can work as a container registry (to pull installer images from)? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't really remember to be honest. I wrote this code some while ago and I wasn't aware the registry I added here is actually not needed, I thought imagefactory needs this as a companion, it made sense in my head I guess. Do you want me to remove the external registry (called There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This registry should be there, but there's no need to push anything there. All I'm asking is to remove this file completely, as it's not needed |
||
|
||
rm -v ${TMP_FILE} | ||
echo "${SCHEMATIC} ${SCHEMA_ID}" >> "${RESULTS_FILE}" | ||
done | ||
|
||
# this prints the image IDs resulting from each schema, | ||
# which can then be handed out to clients. | ||
echo "--- results ---" | ||
cat "${RESULTS_FILE}" | ||
echo "---------------" | ||
|
||
echo "all done, sleep ${SLEEP_TIME} sec." | ||
sleep ${SLEEP_TIME} & | ||
wait $! | ||
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess this should be fixed is the proper URL (with
http://
is passed)