Test Fleet in Rancher on self-hosted runner #2804
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manno
reviewed
Sep 17, 2024
| echo -e "4\n" | rancher login "https://$public_hostname" --token "$token" --skip-verify | ||
|
|
||
| rancher clusters create second --import | ||
| until rancher cluster ls --format json | jq -r 'select(.Name=="second") | .ID' | grep -Eq "c-[a-z0-9]" ; do sleep 1; done | ||
| id=$( rancher cluster ls --format json | jq -r 'select(.Name=="second") | .ID' ) | ||
|
|
||
| kubectl config use-context "$cluster_downstream" | ||
| rancher cluster import "$id" | ||
| kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $user |
There was a problem hiding this comment.
Interesting, I wonder if it would help to pick an admin user:
weyfonk
force-pushed
the
self-hosted-runner-fleet-in-rancher
branch
2 times, most recently
from
2d9310e
to
683ec88
Compare
This is the basis for being able to test any given Fleet commit in Rancher, installing the latter through Helm.
This simplifies Rancher installation, preventing a costly local build and simply making use of existing configuration options to override the Fleet version to install from a custom repository and branch. The Rancher Docker image to use is hard-coded for now.
This is an attempt to use Rancher's org-wide hosted runners to test Fleet within Rancher, instead of dealing with GCP.
This may help prevent unknown authority errors when installing the Ginkgo CLI.
That image comes with CA certificates, curl and tar installed.
Images are already built when releasing Fleet charts against a test charts repository.
This makes use of a dedicated step to install remaining dependencies.
Do we really need a dedicated VM or self-hosted runner for this? Reusing the same setup as Fleet's multi-cluster tests to verify it.
Installing Rancher should directly take care of installing our test Fleet version.
This should fix permissions issues.
This prevents issues about package `helm` not being found in Ubuntu repositories.
This could be reused in CI, after a few improvements.
`rancher/fleet` is now approved to use such runners, with help from EIO.
Registering downstream clusters with Rancher requires the `rancher` CLI.
This should help troubleshoot failures with `Process completed with exit code 1`.
This enables downstream cluster registration to succeed, as specified in the official Rancher docs [1]. [1]: https://ranchermanager.docs.rancher.com/v2.0-v2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/import-existing-clusters#prerequisites
This may eliminate errors with downstream cluster registration.
This eases troubleshooting and enables testing Fleet against existing Rancher releases.
This is not needed when installing Rancher from Helm instead of building it.
Old parameter `installCRDs` is deprecated.
The Rancher CLI may output an empty command for a bit, leading `register-downstream-clusters.sh` to fail. Instead of trying to reverse engineer why that might be, we simply run `rancher cluster import` repeatedly until the returned command is non-empty.
This could help us understand why downstream cluster registration fails in CI although it works locally.
When installing Rancher through Helm: * the `CATTLE_SERVER_URL` needs to be set to the same value as `hostname` * TLS mode must be set to `system-store`, to prevent cert-related errors when running a Fleet agent in a downstream cluster
This waits for the upstream cluster to be ready, preventing an empty IP from being set.
Environment variables are not necessary, as a dedicated Helm value exists.
This moves `test-in-rancher` to the `dev` scripts directory, and briefly explains how to use it.
A newline follows the initial `-`, as in other workflows.
The script does not actually test anything, and is now named consistently with other scripts living in the `dev/` folder.
The script would not manage a cluster's lifecycle beyond its creation anyway, and doing so would be harder to automate. It is therefore left out of the script's scope.
This leaves the original workflow in place, upgrading Fleet in Rancher.
That block is not necessary, as tests are run against clusters afterwards.
This ensures that Fleet examples are validated against Fleet in Rancher.
weyfonk
force-pushed
the
self-hosted-runner-fleet-in-rancher
branch
from
683ec88
to
7b82940
Compare
That check is only relevant when using a custom charts branch.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This creates a new
Test Fleet in Rancherworkflow to install Fleet through the latest Rancher. An example run can be found here.This workflow can be called from this one to test any given Fleet commit against Rancher through multi-cluster tests.
Open points
maincommit against the latest Rancher release every day?fleetrepoci/chartsbranch, which is only useful for as long as Docker images pointed to by test charts exist. In practice, this means only an hour, after which those branches could, and probably should, be deleted. Do we want to automate that deletion to prevent cluttering offleetrepoci/chartswith obsolete test branches?Possible improvements
Refers to #1640.