Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEST: Feature/multi arch building pipeline #556

Open
wants to merge 52 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
d550c99
MA Operator pipeline
Maleware Apr 23, 2024
4419655
Remove harbor for testing
Maleware Apr 23, 2024
962ee49
Manuell trigger for testing
Maleware Apr 23, 2024
5ad946b
Add reminder for after testing
Maleware Apr 23, 2024
c9c80e6
fix cosign image string
Maleware Apr 23, 2024
f004bff
Fix cosign string for another part
Maleware Apr 23, 2024
0e657ca
Adding version to docker output parsing
Maleware Apr 23, 2024
2d9f6e8
Fixing another path for syft
Maleware Apr 23, 2024
6f5079d
Workaround for buildjet error
Maleware Apr 23, 2024
6039775
Remove cargo clean as we have to rework this action
Maleware Apr 24, 2024
4f10fe5
Using a different action to set up cargo install cargo-edit
Maleware Apr 24, 2024
a9375a5
Install helm for arm64 manually
Maleware Apr 24, 2024
7589e47
Rework arch selection mechanism
Maleware Apr 24, 2024
8144cd3
Rework arch selection mechanism 2
Maleware Apr 24, 2024
c0d7ced
Another try for arch selection
Maleware Apr 24, 2024
01d60b1
Still not working...
Maleware Apr 24, 2024
9ae2d83
Fixing makefile for manifest list build (maybe)
Maleware Apr 24, 2024
9868377
Fixing makefile for manifest list build (maybe 2)
Maleware Apr 24, 2024
6f7e7bc
Fix name in makefile
Maleware Apr 24, 2024
28fbb48
Not using vars rather directly use env
Maleware Apr 24, 2024
8e3281b
Cating makefile during action
Maleware Apr 25, 2024
99049f1
Fixing manifestpublish
Maleware Apr 25, 2024
878798c
Adding checkout
Maleware Apr 25, 2024
31046d7
Reverting test changes
Maleware Apr 25, 2024
870d6ff
Better vars
Maleware Apr 25, 2024
eddb14e
Edit vars
Maleware Apr 25, 2024
52cbc24
Trigger pipeline
Maleware Apr 26, 2024
ffade8b
Using 205 to see vars
Maleware Apr 26, 2024
35529c7
Different variables again..
Maleware Apr 26, 2024
a32a93c
Binding maybe
Maleware Apr 26, 2024
fcb1dbe
Remove ;
Maleware Apr 26, 2024
235a748
Removing vars..
Maleware Apr 26, 2024
b645465
Remove man list for harbor for testing
Maleware Apr 26, 2024
cdb1981
Remove vars everywhere
Maleware Apr 26, 2024
f9cd0cf
Typo
Maleware Apr 26, 2024
7c6b18b
Adding login to nexus
Maleware Apr 26, 2024
2d18610
Using correct makefile syntax
Maleware Apr 26, 2024
da12d0f
Escaping single line more often
Maleware Apr 26, 2024
b9e882f
Adding Harbor to the pipeline
Maleware Apr 26, 2024
ba1bf4a
Updating version if PR
Maleware Apr 26, 2024
62509b7
Updating arch tag for signing
Maleware Apr 26, 2024
33f4cf5
Wrong placement of {ARCH}
Maleware Apr 26, 2024
2720c99
Adding cargo-edit since needed
Maleware Apr 26, 2024
04e668a
typo
Maleware May 3, 2024
d4c78cb
Using stackable action for cargo-set-version again
Maleware May 6, 2024
07bf962
Merge branch 'main' into feature/multi-arch-building-pipeline
Maleware May 7, 2024
19bed55
Using stackable action to install cargo bins
Maleware May 10, 2024
cd85e00
Merge branch 'feature/multi-arch-building-pipeline' of https://github…
Maleware May 10, 2024
a828b56
Updating dependencies
Maleware May 10, 2024
e0016fb
Merge branch 'main' into feature/multi-arch-building-pipeline
Maleware May 10, 2024
bd9eda0
Reworking preflight tests
Maleware May 10, 2024
b650588
Merge branch 'feature/multi-arch-building-pipeline' of https://github…
Maleware May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 53 additions & 4 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
- '[0-9][0-9].[0-9]+.[0-9]+'
pull_request:
merge_group:
# TODO: Remove after testing
workflow_dispatch:

env:
CARGO_TERM_COLOR: always
Expand Down Expand Up @@ -305,7 +307,10 @@
needs:
- tests_passed
- select_helm_repo
runs-on: ubuntu-latest
strategy:
matrix:
runner: ["ubuntu-latest", "buildjet-2vcpu-ubuntu-2204-arm"]

Check failure on line 312 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / actionlint

[actionlint] .github/workflows/build.yml#L312

label "buildjet-2vcpu-ubuntu-2204-arm" is unknown. available labels are "windows-latest", "windows-2022", "windows-2019", "windows-2016", "ubuntu-latest", "ubuntu-22.04", "ubuntu-20.04", "ubuntu-18.04", "macos-latest", "macos-latest-xl", "macos-13-xl", "macos-13", "macos-13.0", "macos-12-xl", "macos-12", "macos-12.0", "macos-11", "macos-11.0", "macos-10.15", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file [runner-label]
Raw output
.github/workflows/build.yml:312:35: label "buildjet-2vcpu-ubuntu-2204-arm" is unknown. available labels are "windows-latest", "windows-2022", "windows-2019", "windows-2016", "ubuntu-latest", "ubuntu-22.04", "ubuntu-20.04", "ubuntu-18.04", "macos-latest", "macos-latest-xl", "macos-13-xl", "macos-13", "macos-13.0", "macos-12-xl", "macos-12", "macos-12.0", "macos-11", "macos-11.0", "macos-10.15", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file [runner-label]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runs-on: ${{ matrix.runner }}
permissions:
id-token: write
env:
Expand Down Expand Up @@ -336,10 +341,14 @@
# This step checks if the current run was triggered by a push to a pr (or a pr being created).
# If this is the case it changes the version of this project in all Cargo.toml files to include the suffix
# "-pr<prnumber>" so that the published artifacts can be linked to this PR.
- uses: stackabletech/cargo-install-action@main
# - uses: stackabletech/cargo-install-action@main
# with:

Check failure on line 345 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / yamllint

[yamllint] .github/workflows/build.yml#L345

[warning] comment not indented like content (comments-indentation)
Raw output
./.github/workflows/build.yml:345:9: [warning] comment not indented like content (comments-indentation)
# crate: cargo-edit

Check failure on line 346 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / yamllint

[yamllint] .github/workflows/build.yml#L346

[warning] comment not indented like content (comments-indentation)
Raw output
./.github/workflows/build.yml:346:11: [warning] comment not indented like content (comments-indentation)
# bin: cargo-set-version
- uses: actions-rs/[email protected]
with:
crate: cargo-edit
bin: cargo-set-version
version: latest
- name: Update version if PR
if: ${{ github.event_name == 'pull_request' }}
run: cargo set-version --offline --workspace 0.0.0-pr${{ github.event.pull_request.number }}
Expand All @@ -352,7 +361,16 @@
- name: Install syft
uses: anchore/sbom-action/download-syft@24b0d5238516480139aa8bc6f92eeb7b54a9eb0a # tag=v0.15.5
- name: Build Docker image and Helm chart
run: make -e build
run: |
# Installing helm on BuildJet only
if [ "$(arch)" = "aarch64" ]; then
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get -y install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get -y update
sudo apt-get -y install helm
fi
make -e build
- name: Publish Docker image and Helm chart
if: ${{ !github.event.pull_request.head.repo.fork }}
run: make -e publish
Expand All @@ -362,6 +380,37 @@
if: ${{ !github.event.pull_request.head.repo.fork }}
run: echo "IMAGE_TAG=$(make -e print-docker-tag)" >> $GITHUB_OUTPUT

create_manifest_list:
name: Build and publish manifest list
needs:
- package_and_publish
runs-on: ubuntu-latest
permissions:
id-token: write
env:
NEXUS_PASSWORD: ${{ secrets.NEXUS_PASSWORD }}
OCI_REGISTRY_SDP_PASSWORD: ${{ secrets.HARBOR_ROBOT_SDP_GITHUB_ACTION_BUILD_SECRET }}
OCI_REGISTRY_SDP_USERNAME: "robot$sdp+github-action-build"
OCI_REGISTRY_SDP_CHARTS_PASSWORD: ${{ secrets.HARBOR_ROBOT_SDP_CHARTS_GITHUB_ACTION_BUILD_SECRET }}
OCI_REGISTRY_SDP_CHARTS_USERNAME: "robot$sdp-charts+github-action-build"
steps:

Check failure on line 396 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / yamllint

[yamllint] .github/workflows/build.yml#L396

[error] trailing spaces (trailing-spaces)
Raw output
./.github/workflows/build.yml:396:11: [error] trailing spaces (trailing-spaces)
- name: Install cosign
uses: sigstore/cosign-installer@9614fae9e5c5eddabb09f90a270fcb487c9f7149 # tag=v3.3.0
- name: Checkout
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
submodules: recursive
- name: Update version if PR
if: ${{ github.event_name == 'pull_request' }}
run: cargo set-version --offline --workspace 0.0.0-pr${{ github.event.pull_request.number }}
- name: Build manifest list
run: |
cat Makefile
# Creating manifest list
make -e docker-manifest-list-build
# Pushing and signing manifest list
make -e docker-manifet-list-publish

openshift_preflight:
name: Run the OpenShift Preflight check on the published images
if: ${{ !github.event.pull_request.head.repo.fork }}
Expand Down
55 changes: 41 additions & 14 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@
TAG := $(shell git rev-parse --short HEAD)
OPERATOR_NAME := opa-operator
VERSION := $(shell cargo metadata --format-version 1 | jq -r '.packages[] | select(.name=="stackable-${OPERATOR_NAME}") | .version')
ARCH := $(shell arch | sed -e 's#x86_64#amd64#' | sed -e 's#aarch64#arm64#')

DOCKER_REPO := docker.stackable.tech
# TODO: Change to stackable after testing
ORGANIZATION := stackable
OCI_REGISTRY_HOSTNAME := oci.stackable.tech
OCI_REGISTRY_PROJECT_IMAGES := sdp
Expand All @@ -22,25 +24,24 @@ OCI_REGISTRY_PROJECT_CHARTS := sdp-charts
HELM_REPO := https://repo.stackable.tech/repository/helm-dev
HELM_CHART_NAME := ${OPERATOR_NAME}
HELM_CHART_ARTIFACT := target/helm/${OPERATOR_NAME}-${VERSION}.tgz

SHELL=/usr/bin/env bash -euo pipefail

render-readme:
scripts/render_readme.sh

## Docker related targets
docker-build:
docker build --force-rm --build-arg VERSION=${VERSION} -t "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}" -f docker/Dockerfile .
docker tag "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}" "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}"
docker build --force-rm --build-arg VERSION=${VERSION} -t "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-${ARCH}" -f docker/Dockerfile .
docker tag "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-${ARCH}" "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}-${ARCH}"

docker-publish:
# Push to Nexus
echo "${NEXUS_PASSWORD}" | docker login --username github --password-stdin "${DOCKER_REPO}"
DOCKER_OUTPUT=$$(docker push --all-tags "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}");\
# Obtain the digest of the pushed image from the output of `docker push`, because signing by tag is deprecated and will be removed from cosign in the future\
REPO_DIGEST_OF_IMAGE=$$(echo "$$DOCKER_OUTPUT" | awk '/^${VERSION}: digest: sha256:[0-9a-f]{64} size: [0-9]+$$/ { print $$3 }');\
REPO_DIGEST_OF_IMAGE=$$(echo "$$DOCKER_OUTPUT" | awk '/^${VERSION}-${ARCH}: digest: sha256:[0-9a-f]{64} size: [0-9]+$$/ { print $$3 }');\
if [ -z "$$REPO_DIGEST_OF_IMAGE" ]; then\
echo 'Could not find repo digest for container image: ${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}';\
echo 'Could not find repo digest for container image: ${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-${ARCH}';\
exit 1;\
fi;\
# This generates a signature and publishes it to the registry, next to the image\
Expand All @@ -51,8 +52,8 @@ docker-publish:
# Determine the PURL for the container image\
PURL="pkg:docker/${ORGANIZATION}/${OPERATOR_NAME}@$$REPO_DIGEST_OF_IMAGE?repository_url=${DOCKER_REPO}";\
# Get metadata from the image\
IMAGE_DESCRIPTION=$$(docker inspect --format='{{.Config.Labels.description}}' "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}");\
IMAGE_NAME=$$(docker inspect --format='{{.Config.Labels.name}}' "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}");\
IMAGE_DESCRIPTION=$$(docker inspect --format='{{.Config.Labels.description}}' "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-${ARCH}");\
IMAGE_NAME=$$(docker inspect --format='{{.Config.Labels.name}}' "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-${ARCH}");\
# Merge the SBOM with the metadata for the operator\
jq -s '{"metadata":{"component":{"description":"'"$$IMAGE_NAME. $$IMAGE_DESCRIPTION"'","supplier":{"name":"Stackable GmbH","url":["https://stackable.tech/"]},"author":"Stackable GmbH","purl":"'"$$PURL"'","publisher":"Stackable GmbH"}}} * .[0]' sbom.json > sbom.merged.json;\
# Attest the SBOM to the image\
Expand All @@ -65,24 +66,50 @@ docker-publish:
# Obtain the digest of the pushed image from the output of `docker push`, because signing by tag is deprecated and will be removed from cosign in the future\
REPO_DIGEST_OF_IMAGE=$$(echo "$$DOCKER_OUTPUT" | awk '/^${VERSION}: digest: sha256:[0-9a-f]{64} size: [0-9]+$$/ { print $$3 }');\
if [ -z "$$REPO_DIGEST_OF_IMAGE" ]; then\
echo 'Could not find repo digest for container image: ${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}';\
echo 'Could not find repo digest for container image: ${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}-${ARCH}';\
exit 1;\
fi;\
# This generates a signature and publishes it to the registry, next to the image\
# Uses the keyless signing flow with Github Actions as identity provider\
cosign sign -y "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}@$$REPO_DIGEST_OF_IMAGE";\
cosign sign -y "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}-${ARCH}@$$REPO_DIGEST_OF_IMAGE";\
# Generate the SBOM for the operator image, this leverages the already generated SBOM for the operator binary by cargo-cyclonedx\
syft scan --output cyclonedx-json=sbom.json --select-catalogers "-cargo-auditable-binary-cataloger" --scope all-layers --source-name "${OPERATOR_NAME}" --source-version "${VERSION}" "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}@$$REPO_DIGEST_OF_IMAGE";\
# Determine the PURL for the container image\
PURL="pkg:docker/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}@$$REPO_DIGEST_OF_IMAGE?repository_url=${OCI_REGISTRY_HOSTNAME}";\
# Get metadata from the image\
IMAGE_DESCRIPTION=$$(docker inspect --format='{{.Config.Labels.description}}' "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}");\
IMAGE_NAME=$$(docker inspect --format='{{.Config.Labels.name}}' "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}");\
IMAGE_DESCRIPTION=$$(docker inspect --format='{{.Config.Labels.description}}' "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}-${ARCH}");\
IMAGE_NAME=$$(docker inspect --format='{{.Config.Labels.name}}' "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}-${ARCH}");\
# Merge the SBOM with the metadata for the operator\
jq -s '{"metadata":{"component":{"description":"'"$$IMAGE_NAME. $$IMAGE_DESCRIPTION"'","supplier":{"name":"Stackable GmbH","url":["https://stackable.tech/"]},"author":"Stackable GmbH","purl":"'"$$PURL"'","publisher":"Stackable GmbH"}}} * .[0]' sbom.json > sbom.merged.json;\
# Attest the SBOM to the image\
cosign attest -y --predicate sbom.merged.json --type cyclonedx "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}@$$REPO_DIGEST_OF_IMAGE"

# This assumes "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-amd64 and "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-arm64 being build and pushed
docker-manifest-list-build:
docker manifest create "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}" --amend "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-amd64" --amend "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}-arm64"
docker manifest create "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}" --amend "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}-amd64" --amend "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}-arm64"

docker-manifet-list-publish:
# Push to Nexus
echo "${NEXUS_PASSWORD}" | docker login --username github --password-stdin "${DOCKER_REPO}"
# `docker manifest push` directly returns the digest of the manifest list
# As it is an experimental feature, this might change in the future
# Further reading: https://docs.docker.com/reference/cli/docker/manifest/push/
DIGEST_NEXUS=$$(docker manifest push "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}");\
# Refer to image via its digest (oci.stackable.tech/sdp/airflow@sha256:0a1b2c...)\
# This generates a signature and publishes it to the registry, next to the image\
# Uses the keyless signing flow with Github Actions as identity provider\
cosign sign -y "${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:${VERSION}@$$DIGEST_NEXUS"

# Push to Harbor
# We need to use "value" here to prevent the variable from being recursively expanded by make (username contains a dollar sign, since it's a Harbor bot)
docker login --username '${value OCI_REGISTRY_SDP_USERNAME}' --password '${OCI_REGISTRY_SDP_PASSWORD}' '${OCI_REGISTRY_HOSTNAME}'
DIGEST_HARBOR=$$(docker manifest push "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}");\
# Refer to image via its digest (oci.stackable.tech/sdp/airflow@sha256:0a1b2c...);\
# This generates a signature and publishes it to the registry, next to the image\
# Uses the keyless signing flow with Github Actions as identity provider\
cosign sign -y "${OCI_REGISTRY_HOSTNAME}/${OCI_REGISTRY_PROJECT_IMAGES}/${OPERATOR_NAME}:${VERSION}@$$DIGEST_HARBOR"

# TODO remove if not used/needed
docker: docker-build docker-publish

Expand Down Expand Up @@ -144,15 +171,15 @@ clean: chart-clean
regenerate-charts: chart-clean compile-chart

regenerate-nix:
nix run -f . regenerateNixLockfiles
nix run --extra-experimental-features nix-command --extra-experimental-features flakes -f . regenerateNixLockfiles
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI you can set it for your machine with

$ cat /etc/nix/nix.conf
experimental-features = nix-command


build: regenerate-charts regenerate-nix helm-package docker-build

publish: docker-publish helm-publish

run-dev:
kubectl apply -f deploy/stackable-operators-ns.yaml
nix run -f. tilt -- up --port 5430 --namespace stackable-operators
nix run --extra-experimental-features nix-command --extra-experimental-features flakes -f. tilt -- up --port 5430 --namespace stackable-operators

stop-dev:
nix run -f. tilt -- down
nix run --extra-experimental-features nix-command --extra-experimental-features flakes -f. tilt -- down
Loading