feat: add steps to deploying to the RKE2 custom cluster #525

FrankYang0529 · 2024-03-12T08:39:29Z

harvesterhci-io-github-bot · 2024-03-12T09:06:26Z

✅ Deploy Preview for harvester-preview ready!

Name	Link
🔨 Latest commit	`b1b911a`
🔍 Latest deploy log	https://app.netlify.com/sites/harvester-preview/deploys/6602905c55b2e5008ad67ccc
😎 Deploy Preview	https://6602905c55b2e5008ad67ccc--harvester-preview.netlify.app

jillian-maroket

@FrankYang0529 Let me know if you have concerns about the changes.

docs/rancher/cloud-provider.md

starbops

Thanks for the doc update! It's really helpful. Just one nit, please see the comment below.

docs/rancher/cloud-provider.md

irishgordo · 2024-07-01T23:16:47Z

@FrankYang0529 , thanks for writing up this doc 😄 👍 -
I am working through testing this, but still noticing a few issues...

From the bash script of generate addon, it does seem to work appropriately:

curl -sfL https://raw.githubusercontent.com/harvester/cloud-provider-harvester/master/deploy/generate_addon.sh | bash -s harvester-cloud-provider default              
Creating target directory to hold files in ./tmp/kube...done
Creating a service account in default namespace: harvester-cloud-provider
W0701 15:58:31.127803   64700 helpers.go:663] --dry-run is deprecated and can be replaced with --dry-run=client.
serviceaccount/harvester-cloud-provider created

Creating a clusterrolebinding harvester-cloud-provider
W0701 15:58:31.496911   64736 helpers.go:663] --dry-run is deprecated and can be replaced with --dry-run=client.
clusterrolebinding.rbac.authorization.k8s.io/harvester-cloud-provider created

Getting uid of service account harvester-cloud-provider on default
Service Account uid: 8ffa7fa7-7307-40a2-bc04-37be24697c5b

Creating a user token secret in default namespace: harvester-cloud-provider-token
secret/harvester-cloud-provider-token created
Secret name: harvester-cloud-provider-token

Extracting ca.crt from secret...done
Getting user token from secret...done
Setting current context to: local
Cluster name: local
Endpoint: https://192.168.104.229/k8s/clusters/local

Preparing k8s-harvester-cloud-provider-default-conf
Setting a cluster entry in kubeconfig...Cluster "local" set.
Setting token credentials entry in kubeconfig...User "harvester-cloud-provider-default-local" set.
Setting a context entry in kubeconfig...Context "harvester-cloud-provider-default-local" created.
Setting the current-context in the kubeconfig file...Switched to context "harvester-cloud-provider-default-local".
########## cloud config ############
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFNU9EWXdNekF3TUI0WERUSTBNRGN3TVRFNE5UZ3lNRm9YRFRNME1EWXlPVEU0TlRneQpNRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE9UZzJNRE13TURCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJNeHFQM3pqVm5mRVIrOEc3bXBSeitncUhPNW1WdTdWd3dGVHphU3YKbnIvbzB3UlpIcmtNQWYvWHpaQUJhS0hoekVmN0RsOHI5WDZISGExa1J5ZFQyaktqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCVENvOEp1Y1h5TElnN3ptMWpXCmw2bmk5cDhjR3pBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBc01RWTV1REVxVWtsR1JQYURvQ0NhakJSWHh2NFkKa3UrY3RJeXBYa2lnYVFJaEFNNkFNdldiam9XQUpNeGNGeFRRS1Z4YjhlVldaTkczZ1k0K0EvMC9KTXpuCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.104.229/k8s/clusters/local
  name: local
contexts:
- context:
    cluster: local
    namespace: default
    user: harvester-cloud-provider-default-local
  name: harvester-cloud-provider-default-local
current-context: harvester-cloud-provider-default-local
kind: Config
preferences: {}
users:
- name: harvester-cloud-provider-default-local
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IktMbE8yMS1RSlMzYlNNb05TUTV5VkFEcnBVd2FKSGFsazVpUlc0YV9RaXcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImhhcnZlc3Rlci1jbG91ZC1wcm92aWRlci10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI4ZmZhN2ZhNy03MzA3LTQwYTItYmMwNC0zN2JlMjQ2OTdjNWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXIifQ.OzANLXDM9LI_O-zFMG2QPi9eBF0pqpuXp10RKlF3SqwL5TFz5PCF4KDCNTEC4udP0Y53X6LxY3G7-6l-6fR48EDYzpEsSeKXIicFQbyZcwjKMn8xWKT23Mp7vuirhKI0e6wArLaRtkX8NjGB_PNcP1PMDxvgjQ1N8YX3t9ZkniB4b9DIGPrdOgRO8tvntIDpe89Ip6TtvooeGZrHbuO9bzxxFJVcYZzZ-biTB3Oi6hcn9pXPQc4_3baUXJf19-buGceClDUSNntMhGKB4oTBN4YbcyaXFijg6VSmRurCOKFYDOl5TJOmgGm13kkU4G2mrvpB9uJqPSPWkvgz4tmLZA

########## cloud-init user data ############
write_files:
- encoding: b64
  content: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VKbFZFTkRRVklyWjBGM1NVSkJaMGxDUVVSQlMwSm5aM0ZvYTJwUFVGRlJSRUZxUVd0TlUwbDNTVUZaUkZaUlVVUkVRbXg1WVRKVmVVeFlUbXdLWTI1YWJHTnBNV3BaVlVGNFRucEZOVTlFV1hkTmVrRjNUVUkwV0VSVVNUQk5SR04zVFZSRk5FNVVaM2xOUm05WVJGUk5NRTFFV1hsUFZFVTBUbFJuZVFwTlJtOTNTa1JGYVUxRFFVZEJNVlZGUVhkM1dtTnRkR3hOYVRGNldsaEtNbHBZU1hSWk1rWkJUVlJqZUU5VVp6Sk5SRTEzVFVSQ1drMUNUVWRDZVhGSENsTk5ORGxCWjBWSFEwTnhSMU5OTkRsQmQwVklRVEJKUVVKTmVIRlFNM3BxVm01bVJWSXJPRWMzYlhCU2VpdG5jVWhQTlcxV2RUZFdkM2RHVkhwaFUzWUtibkl2YnpCM1VscEljbXROUVdZdldIcGFRVUpoUzBob2VrVm1OMFJzT0hJNVdEWklTR0V4YTFKNVpGUXlha3RxVVdwQ1FVMUJORWRCTVZWa1JIZEZRZ292ZDFGRlFYZEpRM0JFUVZCQ1owNVdTRkpOUWtGbU9FVkNWRUZFUVZGSUwwMUNNRWRCTVZWa1JHZFJWMEpDVkVOdk9FcDFZMWg1VEVsbk4zcHRNV3BYQ213MmJtazVjRGhqUjNwQlMwSm5aM0ZvYTJwUFVGRlJSRUZuVGtsQlJFSkdRV2xCYzAxUldUVjFSRVZ4Vld0c1IxSlFZVVJ2UTBOaGFrSlNXSGgyTkZrS2EzVXJZM1JKZVhCWWEybG5ZVkZKYUVGTk5rRk5kbGRpYW05WFFVcE5lR05HZUZSUlMxWjRZamhsVmxkYVRrY3paMWswSzBFdk1DOUtUWHB1Q2kwdExTMHRSVTVFSUVORlVsUkpSa2xEUVZSRkxTMHRMUzBLCiAgICBzZXJ2ZXI6IGh0dHBzOi8vMTkyLjE2OC4xMDQuMjI5L2s4cy9jbHVzdGVycy9sb2NhbAogIG5hbWU6IGxvY2FsCmNvbnRleHRzOgotIGNvbnRleHQ6CiAgICBjbHVzdGVyOiBsb2NhbAogICAgbmFtZXNwYWNlOiBkZWZhdWx0CiAgICB1c2VyOiBoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXItZGVmYXVsdC1sb2NhbAogIG5hbWU6IGhhcnZlc3Rlci1jbG91ZC1wcm92aWRlci1kZWZhdWx0LWxvY2FsCmN1cnJlbnQtY29udGV4dDogaGFydmVzdGVyLWNsb3VkLXByb3ZpZGVyLWRlZmF1bHQtbG9jYWwKa2luZDogQ29uZmlnCnByZWZlcmVuY2VzOiB7fQp1c2VyczoKLSBuYW1lOiBoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXItZGVmYXVsdC1sb2NhbAogIHVzZXI6CiAgICB0b2tlbjogZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrdE1iRTh5TVMxUlNsTXpZbE5OYjA1VFVUVjVWa0ZFY25CVmQyRktTR0ZzYXpWcFVsYzBZVjlSYVhjaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbWhoY25abGMzUmxjaTFqYkc5MVpDMXdjbTkyYVdSbGNpMTBiMnRsYmlJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExtNWhiV1VpT2lKb1lYSjJaWE4wWlhJdFkyeHZkV1F0Y0hKdmRtbGtaWElpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1MWFXUWlPaUk0Wm1aaE4yWmhOeTAzTXpBM0xUUXdZVEl0WW1Nd05DMHpOMkpsTWpRMk9UZGpOV0lpTENKemRXSWlPaUp6ZVhOMFpXMDZjMlZ5ZG1salpXRmpZMjkxYm5RNlpHVm1ZWFZzZERwb1lYSjJaWE4wWlhJdFkyeHZkV1F0Y0hKdmRtbGtaWElpZlEuT3pBTkxYRE05TElfTy16Rk1HMlFQaTllQkYwcHFwdVhwMTBSS2xGM1Nxd0w1VEZ6NVBDRjRLRENOVEVDNHVkUDBZNTNYNkx4WTNHNy02bC02ZlI0OEVEWXpwRXNTZUtYSWljRlFieVpjd2pLTW44eFdLVDIzTXA3dnVpcmhLSTBlNndBckxhUnRrWDhOakdCX1BOY1AxUE1EeHZnalExTjhZWDN0OVprbmlCNGI5RElHUHJkT2dSTzh0dm50SURwZTg5SXA2VHR2b29lR1pySGJ1Tzlienh4RkpWY1laelotYmlUQjNPaTZoY245cFhQUWM0XzNiYVVYSmYxOS1idUdjZUNsRFVTTm50TWhHS0I0b1RCTjRZYmN5YVhGaWpnNlZTbVJ1ckNPS0ZZRE9sNVRKT21nR20xM2trVTRHMm1ydnBCOXVKcVBTUFdrdmd6NHRtTFpBCg==
  owner: root:root
  path: /etc/kubernetes/cloud-config
  permissions: '0644'

Where my default KUBECONFIG was:

apiVersion: v1
kind: Config
clusters:
- name: "local"
  cluster:
    server: "https://192.168.104.229/k8s/clusters/local"
    certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJ2VENDQ\
      VdPZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQkdNUnd3R2dZRFZRUUtFeE5rZVc1aGJXbGoKY\
      kdsemRHVnVaWEl0YjNKbk1TWXdKQVlEVlFRRERCMWtlVzVoYldsamJHbHpkR1Z1WlhJdFkyRkFNV\
      GN4T1RnMgpNRE01TnpBZUZ3MHlOREEzTURFeE9EVTVOVGRhRncwek5EQTJNamt4T0RVNU5UZGFNR\
      Vl4SERBYUJnTlZCQW9UCkUyUjVibUZ0YVdOc2FYTjBaVzVsY2kxdmNtY3hKakFrQmdOVkJBTU1IV\
      1I1Ym1GdGFXTnNhWE4wWlc1bGNpMWoKWVVBeE56RTVPRFl3TXprM01Ga3dFd1lIS29aSXpqMENBU\
      VlJS29aSXpqMERBUWNEUWdBRXRpYUhWTWxGcnF6VgpmQzFjOFAxRkJaUE5BSEZsTTBSSUQ5cW9OV\
      XRuMTM5SVFSclFwRm5ZNCtHQ1M2aytSUGhiTlk1MDdObjJKZGovCmVDdEtSQy9BZktOQ01FQXdEZ\
      1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWUQKVlIwT0JCWUVGS\
      lZ5dVpHV0hmNWVzY1NiVW83eDV1UUhIVk5lTUFvR0NDcUdTTTQ5QkFNQ0EwZ0FNRVVDSUdvaApxa\
      UpJTnpoTFRqbldWV1BMdll1QnZNNEdrMXc1WXJMRW1zdFRFL2I5QWlFQXowQ1RwREFYdmg3QmU3d\
      3h4aUhjCkloeXEza3Jrc2llNWxYWUVMUDc2WTlZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t"

users:
- name: "local"
  user:
    token: "kubeconfig-user-hn5s4cnv2d:xzb54xhwsldjdkx7gmtkm8wbzhzdpfw5t78jvpv7ph4nq8zbhw8xxd"


contexts:
- name: "local"
  context:
    user: "local"
    cluster: "local"

current-context: "local"

And then each VM, their respective cloud-config User Data was:

#cloud-config
password: testtesttest
chpasswd:
  expire: false
ssh_pwauth: true
package_update: true
packages:
  - qemu-guest-agent
  - neovim
  - wget
  - net-tools
  - jq
runcmd:
  - - systemctl
    - enable
    - --now
    - qemu-guest-agent.service
  - wget wget https://dl.k8s.io/v1.27.15/kubernetes-client-linux-amd64.tar.gz -O /tmp/kubectl-client.tar.gz
  - cd /tmp && tar -xvf /tmp/kubectl-client.tar.gz && cp /tmp/kubernetes/client/bin/kubectl /usr/bin/kubectl
write_files:
- encoding: b64
  content: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VKbFZFTkRRVklyWjBGM1NVSkJaMGxDUVVSQlMwSm5aM0ZvYTJwUFVGRlJSRUZxUVd0TlUwbDNTVUZaUkZaUlVVUkVRbXg1WVRKVmVVeFlUbXdLWTI1YWJHTnBNV3BaVlVGNFRucEZOVTlFV1hkTmVrRjNUVUkwV0VSVVNUQk5SR04zVFZSRk5FNVVaM2xOUm05WVJGUk5NRTFFV1hsUFZFVTBUbFJuZVFwTlJtOTNTa1JGYVUxRFFVZEJNVlZGUVhkM1dtTnRkR3hOYVRGNldsaEtNbHBZU1hSWk1rWkJUVlJqZUU5VVp6Sk5SRTEzVFVSQ1drMUNUVWRDZVhGSENsTk5ORGxCWjBWSFEwTnhSMU5OTkRsQmQwVklRVEJKUVVKTmVIRlFNM3BxVm01bVJWSXJPRWMzYlhCU2VpdG5jVWhQTlcxV2RUZFdkM2RHVkhwaFUzWUtibkl2YnpCM1VscEljbXROUVdZdldIcGFRVUpoUzBob2VrVm1OMFJzT0hJNVdEWklTR0V4YTFKNVpGUXlha3RxVVdwQ1FVMUJORWRCTVZWa1JIZEZRZ292ZDFGRlFYZEpRM0JFUVZCQ1owNVdTRkpOUWtGbU9FVkNWRUZFUVZGSUwwMUNNRWRCTVZWa1JHZFJWMEpDVkVOdk9FcDFZMWg1VEVsbk4zcHRNV3BYQ213MmJtazVjRGhqUjNwQlMwSm5aM0ZvYTJwUFVGRlJSRUZuVGtsQlJFSkdRV2xCYzAxUldUVjFSRVZ4Vld0c1IxSlFZVVJ2UTBOaGFrSlNXSGgyTkZrS2EzVXJZM1JKZVhCWWEybG5ZVkZKYUVGTk5rRk5kbGRpYW05WFFVcE5lR05HZUZSUlMxWjRZamhsVmxkYVRrY3paMWswSzBFdk1DOUtUWHB1Q2kwdExTMHRSVTVFSUVORlVsUkpSa2xEUVZSRkxTMHRMUzBLCiAgICBzZXJ2ZXI6IGh0dHBzOi8vMTkyLjE2OC4xMDQuMjI5L2s4cy9jbHVzdGVycy9sb2NhbAogIG5hbWU6IGxvY2FsCmNvbnRleHRzOgotIGNvbnRleHQ6CiAgICBjbHVzdGVyOiBsb2NhbAogICAgbmFtZXNwYWNlOiBkZWZhdWx0CiAgICB1c2VyOiBoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXItZGVmYXVsdC1sb2NhbAogIG5hbWU6IGhhcnZlc3Rlci1jbG91ZC1wcm92aWRlci1kZWZhdWx0LWxvY2FsCmN1cnJlbnQtY29udGV4dDogaGFydmVzdGVyLWNsb3VkLXByb3ZpZGVyLWRlZmF1bHQtbG9jYWwKa2luZDogQ29uZmlnCnByZWZlcmVuY2VzOiB7fQp1c2VyczoKLSBuYW1lOiBoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXItZGVmYXVsdC1sb2NhbAogIHVzZXI6CiAgICB0b2tlbjogZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrdE1iRTh5TVMxUlNsTXpZbE5OYjA1VFVUVjVWa0ZFY25CVmQyRktTR0ZzYXpWcFVsYzBZVjlSYVhjaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbWhoY25abGMzUmxjaTFqYkc5MVpDMXdjbTkyYVdSbGNpMTBiMnRsYmlJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExtNWhiV1VpT2lKb1lYSjJaWE4wWlhJdFkyeHZkV1F0Y0hKdmRtbGtaWElpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1MWFXUWlPaUk0Wm1aaE4yWmhOeTAzTXpBM0xUUXdZVEl0WW1Nd05DMHpOMkpsTWpRMk9UZGpOV0lpTENKemRXSWlPaUp6ZVhOMFpXMDZjMlZ5ZG1salpXRmpZMjkxYm5RNlpHVm1ZWFZzZERwb1lYSjJaWE4wWlhJdFkyeHZkV1F0Y0hKdmRtbGtaWElpZlEuT3pBTkxYRE05TElfTy16Rk1HMlFQaTllQkYwcHFwdVhwMTBSS2xGM1Nxd0w1VEZ6NVBDRjRLRENOVEVDNHVkUDBZNTNYNkx4WTNHNy02bC02ZlI0OEVEWXpwRXNTZUtYSWljRlFieVpjd2pLTW44eFdLVDIzTXA3dnVpcmhLSTBlNndBckxhUnRrWDhOakdCX1BOY1AxUE1EeHZnalExTjhZWDN0OVprbmlCNGI5RElHUHJkT2dSTzh0dm50SURwZTg5SXA2VHR2b29lR1pySGJ1Tzlienh4RkpWY1laelotYmlUQjNPaTZoY245cFhQUWM0XzNiYVVYSmYxOS1idUdjZUNsRFVTTm50TWhHS0I0b1RCTjRZYmN5YVhGaWpnNlZTbVJ1ckNPS0ZZRE9sNVRKT21nR20xM2trVTRHMm1ydnBCOXVKcVBTUFdrdmd6NHRtTFpBCg==
  owner: root:root
  path: /etc/kubernetes/cloud-config
  permissions: '0644'

But still noticing that the two node custom rke2 cluster is hung for 10+ minutes on "updating" phase in Rancher v2.8.2, with Harvester v1.3.1.

The registration of the Worker VM & ETCD + CP VM looked to be successful with:

╭─mike at suse-workstation-team-harvester in ~/Projects/another-ipxe-examples-ugh/ipxe-examples/vagrant-pxe-harvester on main✘✘✘
╰─± ssh [email protected]
The authenticity of host '192.168.104.147 (192.168.104.147)' can't be established.
ED25519 key fingerprint is SHA256:IqXX4bM7p5XfXbpEsZ54OIltqB/kDwFrbtdDErj7EOA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.104.147' (ED25519) to the list of known hosts.
[email protected]'s password: 
openSUSE Leap 15.5 x86_64 (64-bit)

As "root" use the:
- zypper command for package management
- yast command for configuration management

Have a lot of fun...
opensuse@custom-rke2-cluster-node-01:~> curl --insecure -fL https://rancher.192.168.104.229.sslip.io/system-agent-install.sh | sudo  sh -s - --server https://rancher.192.168.104.229.sslip.io --label 'cattle.io/os=linux' --token ckfgfl2dhblvwbmtcwwbsxkct76qllvjbqmmdlrtjmpb7mvv76tzbf --ca-checksum acb1c1cbcd4e6c54739d1328e7f428773daa637a7b183cecda54f64471a67e31 --etcd --controlplane
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30889    0 30889    0     0  3885k      0 --:--:-- --:--:-- --:--:-- 4309k
[INFO]  Label: cattle.io/os=linux
[INFO]  Role requested: etcd
[INFO]  Role requested: controlplane
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[INFO]  Determined CA is necessary to connect to Rancher
[INFO]  Successfully downloaded CA certificate
[INFO]  Value from https://rancher.192.168.104.229.sslip.io/cacerts is an x509 certificate
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent binary from https://rancher.192.168.104.229.sslip.io/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Downloading rancher-system-agent-uninstall.sh script from https://rancher.192.168.104.229.sslip.io/assets/system-agent-uninstall.sh
[INFO]  Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO]  Generating Cattle ID
[INFO]  Successfully downloaded Rancher connection information
[INFO]  systemd: Creating service file
[INFO]  Creating environment file /etc/systemd/system/rancher-system-agent.env
[INFO]  Enabling rancher-system-agent.service
Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service.
[INFO]  Starting/restarting rancher-system-agent.service
opensuse@custom-rke2-cluster-node-01:~> exit
logout
Connection to 192.168.104.147 closed.
╭─mike at suse-workstation-team-harvester in ~/Projects/another-ipxe-examples-ugh/ipxe-examples/vagrant-pxe-harvester on main✘✘✘
╰─± ssh [email protected]
The authenticity of host '192.168.104.123 (192.168.104.123)' can't be established.
ED25519 key fingerprint is SHA256:fnk4cA3bv2/Ufsxp78QXAww+qWHn4ZkuAUR3jrSzx6s.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.104.123' (ED25519) to the list of known hosts.
[email protected]'s password: 
openSUSE Leap 15.5 x86_64 (64-bit)

As "root" use the:
- zypper command for package management
- yast command for configuration management

Have a lot of fun...
opensuse@custom-rke2-cluster-node-02:~> curl --insecure -fL https://rancher.192.168.104.229.sslip.io/system-agent-install.sh | sudo  sh -s - --server https://rancher.192.168.104.229.sslip.io --label 'cattle.io/os=linux' --token ckfgfl2dhblvwbmtcwwbsxkct76qllvjbqmmdlrtjmpb7mvv76tzbf --ca-checksum acb1c1cbcd4e6c54739d1328e7f428773daa637a7b183cecda54f64471a67e31 --worker
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30889    0 30889    0     0  3803k      0 --:--:-- --:--:-- --:--:-- 4309k
[INFO]  Label: cattle.io/os=linux
[INFO]  Role requested: worker
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[INFO]  Determined CA is necessary to connect to Rancher
[INFO]  Successfully downloaded CA certificate
[INFO]  Value from https://rancher.192.168.104.229.sslip.io/cacerts is an x509 certificate
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent binary from https://rancher.192.168.104.229.sslip.io/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Downloading rancher-system-agent-uninstall.sh script from https://rancher.192.168.104.229.sslip.io/assets/system-agent-uninstall.sh
[INFO]  Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO]  Generating Cattle ID
[INFO]  Successfully downloaded Rancher connection information
[INFO]  systemd: Creating service file
[INFO]  Creating environment file /etc/systemd/system/rancher-system-agent.env
[INFO]  Enabling rancher-system-agent.service
Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service.
[INFO]  Starting/restarting rancher-system-agent.service
opensuse@custom-rke2-cluster-node-02:~> exit
logout
Connection to 192.168.104.123 closed.

Signed-off-by: PoAn Yang <[email protected]>

Co-authored-by: Jillian <[email protected]>

Signed-off-by: PoAn Yang <[email protected]>

github-actions · 2024-07-02T03:11:51Z

Name	Link
🔨 Latest commit	`db49d78`
😎 Deploy Preview	https://66865bca378d14a5ce51984f--harvester-preview.netlify.app

FrankYang0529 · 2024-07-02T03:49:20Z

Hi @irishgordo, thanks for the testing. I think the only difference between our clusters is that your kubeconfig server endpoint is https://192.168.104.229/k8s/clusters/local, but my endpoint is https://192.168.3.131:6443. Could you change the endpoint to pure VIP and try again? Thank you.

########## cloud config ############
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFNU9ERXlOams0TUI0WERUSTBNRGN3TVRBMU5EUTFPRm9YRFRNME1EWXlPVEExTkRRMQpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE9UZ3hNalk1T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJDeUhTUXZMVk8xNEFuUEdiZFBpUC80R0w0SXFCVEdURHo3dWZhQ0gKRGhiL0YwTjdTVnJFU0d0RXBIN2M3eXkyeUd5UWZTV3BQWHFuYUs1STg2Y2YwNGFqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUVI2UFZXTzltMUhrbjlwdlRwCmRXVmdKTkdkZWpBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlFQTNtZUpVZFUwaURuUVNmTzNFRVFDSC9TRzdkV3QKdm41RlpzUkZMclpwVjI4Q0lBSElaZEdteUVZRTBEU2ZyNWM3Wk9Fa3ZWU0lLaEJzNVhraWs5SWVxZDZYCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.3.131:6443
  name: default
contexts:
- context:
    cluster: default
    namespace: default
    user: rke2-2-default-default
  name: rke2-2-default-default
current-context: rke2-2-default-default
kind: Config
preferences: {}
users:
- name: rke2-2-default-default
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjNzTGlvbURQc3NHQkRnUzVSNWlUN3l2VkFsZ2VGTjdnSXFIUEVYMy1IQTQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InJrZTItMi10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJya2UyLTIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkMmRlNWVhOC0zNmNlLTQ0YjUtYmM0Yy03NDI3YWVhMWIwMDgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpya2UyLTIifQ.iJAN8H1qNoBPMKQdqLySIFoSwlbQn-YsChbmBY8U5QtfSGdVwqG3OmIDMJaqo92cgcGUGW3iuBwhDY7ZWdw1ZypKlturXbYyKXEDYSrteB7ZtBhxlpCVpZt_I0l0bi8NFvPAVbnh6axGZsEt4JZYxOC5r3jNSUy9RM9sZcQcUBY93Co74PCvxxeCEg2xrihkhRr97azK9Rxp9OzZp_VNPoxjYNeAUZZVIGqHzJbJzmFEV_ePzjhPpk40XOGTv2sF2-GhQtQPIRi25kJqvNvlffRfVkCxerW1NnGasTgMJdwwamuQ8YmvlS2qMHX5IWWvwuA540p3CCwBCQRQSguIZA

########## cloud-init user data ############
write_files:
- encoding: b64
  content: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VKbFZFTkRRVklyWjBGM1NVSkJaMGxDUVVSQlMwSm5aM0ZvYTJwUFVGRlJSRUZxUVd0TlUwbDNTVUZaUkZaUlVVUkVRbXg1WVRKVmVVeFlUbXdLWTI1YWJHTnBNV3BaVlVGNFRucEZOVTlFUlhsT2FtczBUVUkwV0VSVVNUQk5SR04zVFZSQk1VNUVVVEZQUm05WVJGUk5NRTFFV1hsUFZFRXhUa1JSTVFwUFJtOTNTa1JGYVUxRFFVZEJNVlZGUVhkM1dtTnRkR3hOYVRGNldsaEtNbHBZU1hSWk1rWkJUVlJqZUU5VVozaE5hbGsxVDBSQ1drMUNUVWRDZVhGSENsTk5ORGxCWjBWSFEwTnhSMU5OTkRsQmQwVklRVEJKUVVKRGVVaFRVWFpNVms4eE5FRnVVRWRpWkZCcFVDODBSMHcwU1hGQ1ZFZFVSSG8zZFdaaFEwZ0tSR2hpTDBZd1RqZFRWbkpGVTBkMFJYQklOMk0zZVhreWVVZDVVV1pUVjNCUVdIRnVZVXMxU1RnMlkyWXdOR0ZxVVdwQ1FVMUJORWRCTVZWa1JIZEZRZ292ZDFGRlFYZEpRM0JFUVZCQ1owNVdTRkpOUWtGbU9FVkNWRUZFUVZGSUwwMUNNRWRCTVZWa1JHZFJWMEpDVVZJMlVGWlhUemx0TVVocmJqbHdkbFJ3Q21SWFZtZEtUa2RrWldwQlMwSm5aM0ZvYTJwUFVGRlJSRUZuVGtsQlJFSkdRV2xGUVROdFpVcFZaRlV3YVVSdVVWTm1Uek5GUlZGRFNDOVRSemRrVjNRS2RtNDFSbHB6VWtaTWNscHdWakk0UTBsQlNFbGFaRWR0ZVVWWlJUQkVVMlp5TldNM1drOUZhM1pXVTBsTGFFSnpOVmhyYVdzNVNXVnhaRFpZQ2kwdExTMHRSVTVFSUVORlVsUkpSa2xEUVZSRkxTMHRMUzBLCiAgICBzZXJ2ZXI6IGh0dHBzOi8vMTkyLjE2OC4zLjEzMTo2NDQzCiAgbmFtZTogZGVmYXVsdApjb250ZXh0czoKLSBjb250ZXh0OgogICAgY2x1c3RlcjogZGVmYXVsdAogICAgbmFtZXNwYWNlOiBkZWZhdWx0CiAgICB1c2VyOiBya2UyLTItZGVmYXVsdC1kZWZhdWx0CiAgbmFtZTogcmtlMi0yLWRlZmF1bHQtZGVmYXVsdApjdXJyZW50LWNvbnRleHQ6IHJrZTItMi1kZWZhdWx0LWRlZmF1bHQKa2luZDogQ29uZmlnCnByZWZlcmVuY2VzOiB7fQp1c2VyczoKLSBuYW1lOiBya2UyLTItZGVmYXVsdC1kZWZhdWx0CiAgdXNlcjoKICAgIHRva2VuOiBleUpoYkdjaU9pSlNVekkxTmlJc0ltdHBaQ0k2SWpOelRHbHZiVVJRYzNOSFFrUm5VelZTTldsVU4zbDJWa0ZzWjJWR1RqZG5TWEZJVUVWWU15MUlRVFFpZlEuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9pSmtaV1poZFd4MElpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WldOeVpYUXVibUZ0WlNJNkluSnJaVEl0TWkxMGIydGxiaUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG01aGJXVWlPaUp5YTJVeUxUSWlMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObGNuWnBZMlV0WVdOamIzVnVkQzUxYVdRaU9pSmtNbVJsTldWaE9DMHpObU5sTFRRMFlqVXRZbU0wWXkwM05ESTNZV1ZoTVdJd01EZ2lMQ0p6ZFdJaU9pSnplWE4wWlcwNmMyVnlkbWxqWldGalkyOTFiblE2WkdWbVlYVnNkRHB5YTJVeUxUSWlmUS5pSkFOOEgxcU5vQlBNS1FkcUx5U0lGb1N3bGJRbi1Zc0NoYm1CWThVNVF0ZlNHZFZ3cUczT21JRE1KYXFvOTJjZ2NHVUdXM2l1QndoRFk3WldkdzFaeXBLbHR1clhiWXlLWEVEWVNydGVCN1p0Qmh4bHBDVnBadF9JMGwwYmk4TkZ2UEFWYm5oNmF4R1pzRXQ0SlpZeE9DNXIzak5TVXk5Uk05c1pjUWNVQlk5M0NvNzRQQ3Z4eGVDRWcyeHJpaGtoUnI5N2F6SzlSeHA5T3pacF9WTlBveGpZTmVBVVpaVklHcUh6SmJKem1GRVZfZVB6amhQcGs0MFhPR1R2MnNGMi1HaFF0UVBJUmkyNWtKcXZOdmxmZlJmVmtDeGVyVzFObkdhc1RnTUpkd3dhbXVROFltdmxTMnFNSFg1SVdXdnd1QTU0MHAzQ0N3QkNRUlFTZ3VJWkEK
  owner: root:root
  path: /etc/kubernetes/cloud-config
  permissions: '0644'

irishgordo · 2024-07-02T06:01:20Z

@FrankYang0529 thanks for that mention.
😄 👍
This works great after the edit to just be :6443.
Using the /etc/rancher/rke2/rke2.yaml from the node itself.
Re-running the curl -sfL https://raw.githubusercontent.com/harvester/cloud-provider-harvester/master/deploy/generate_addon.sh | bash -s harvester-cloud-provider default - and then leveraging the cloud-config user-data block generated.

I'm still running into an issue where:

https://rancher.192.168.104.229.sslip.io/dashboard/c/c-m-f7ln5rrh/explorer#cluster-events

Cannot backup to S3, at:

https://192.168.104.128:9001/browser/custom-rke2-bucket

DM'ing you the credentials.

Shifting to just :6443, was successful, both nodes came up online:

╭─mike at suse-workstation-team-harvester in ~/Projects/another-ipxe-examples-ugh/ipxe-examples/vagrant-pxe-harvester on main✘✘✘
╰─± curl -sfL https://raw.githubusercontent.com/harvester/cloud-provider-harvester/master/deploy/generate_addon.sh | bash -s harvester-cloud-provider default
Creating target directory to hold files in ./tmp/kube...done
Creating a service account in default namespace: harvester-cloud-provider
W0701 21:58:44.646823  170362 helpers.go:663] --dry-run is deprecated and can be replaced with --dry-run=client.
serviceaccount/harvester-cloud-provider configured

Creating a clusterrolebinding harvester-cloud-provider
W0701 21:58:45.027831  170404 helpers.go:663] --dry-run is deprecated and can be replaced with --dry-run=client.
clusterrolebinding.rbac.authorization.k8s.io/harvester-cloud-provider configured

Getting uid of service account harvester-cloud-provider on default
Service Account uid: 8ffa7fa7-7307-40a2-bc04-37be24697c5b

Creating a user token secret in default namespace: harvester-cloud-provider-token
secret/harvester-cloud-provider-token unchanged
Secret name: harvester-cloud-provider-token

Extracting ca.crt from secret...done
Getting user token from secret...done
Setting current context to: default
Cluster name: default
Endpoint: https://192.168.104.85:6443

Preparing k8s-harvester-cloud-provider-default-conf
Setting a cluster entry in kubeconfig...Cluster "default" set.
Setting token credentials entry in kubeconfig...User "harvester-cloud-provider-default-default" set.
Setting a context entry in kubeconfig...Context "harvester-cloud-provider-default-default" created.
Setting the current-context in the kubeconfig file...Switched to context "harvester-cloud-provider-default-default".
########## cloud config ############
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFNU9EWXdNekF3TUI0WERUSTBNRGN3TVRFNE5UZ3lNRm9YRFRNME1EWXlPVEU0TlRneQpNRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE9UZzJNRE13TURCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJNeHFQM3pqVm5mRVIrOEc3bXBSeitncUhPNW1WdTdWd3dGVHphU3YKbnIvbzB3UlpIcmtNQWYvWHpaQUJhS0hoekVmN0RsOHI5WDZISGExa1J5ZFQyaktqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCVENvOEp1Y1h5TElnN3ptMWpXCmw2bmk5cDhjR3pBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBc01RWTV1REVxVWtsR1JQYURvQ0NhakJSWHh2NFkKa3UrY3RJeXBYa2lnYVFJaEFNNkFNdldiam9XQUpNeGNGeFRRS1Z4YjhlVldaTkczZ1k0K0EvMC9KTXpuCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.104.85:6443
  name: default
contexts:
- context:
    cluster: default
    namespace: default
    user: harvester-cloud-provider-default-default
  name: harvester-cloud-provider-default-default
current-context: harvester-cloud-provider-default-default
kind: Config
preferences: {}
users:
- name: harvester-cloud-provider-default-default
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IktMbE8yMS1RSlMzYlNNb05TUTV5VkFEcnBVd2FKSGFsazVpUlc0YV9RaXcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImhhcnZlc3Rlci1jbG91ZC1wcm92aWRlci10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI4ZmZhN2ZhNy03MzA3LTQwYTItYmMwNC0zN2JlMjQ2OTdjNWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXIifQ.OzANLXDM9LI_O-zFMG2QPi9eBF0pqpuXp10RKlF3SqwL5TFz5PCF4KDCNTEC4udP0Y53X6LxY3G7-6l-6fR48EDYzpEsSeKXIicFQbyZcwjKMn8xWKT23Mp7vuirhKI0e6wArLaRtkX8NjGB_PNcP1PMDxvgjQ1N8YX3t9ZkniB4b9DIGPrdOgRO8tvntIDpe89Ip6TtvooeGZrHbuO9bzxxFJVcYZzZ-biTB3Oi6hcn9pXPQc4_3baUXJf19-buGceClDUSNntMhGKB4oTBN4YbcyaXFijg6VSmRurCOKFYDOl5TJOmgGm13kkU4G2mrvpB9uJqPSPWkvgz4tmLZA

########## cloud-init user data ############
write_files:
- encoding: b64
  content: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VKbFZFTkRRVklyWjBGM1NVSkJaMGxDUVVSQlMwSm5aM0ZvYTJwUFVGRlJSRUZxUVd0TlUwbDNTVUZaUkZaUlVVUkVRbXg1WVRKVmVVeFlUbXdLWTI1YWJHTnBNV3BaVlVGNFRucEZOVTlFV1hkTmVrRjNUVUkwV0VSVVNUQk5SR04zVFZSRk5FNVVaM2xOUm05WVJGUk5NRTFFV1hsUFZFVTBUbFJuZVFwTlJtOTNTa1JGYVUxRFFVZEJNVlZGUVhkM1dtTnRkR3hOYVRGNldsaEtNbHBZU1hSWk1rWkJUVlJqZUU5VVp6Sk5SRTEzVFVSQ1drMUNUVWRDZVhGSENsTk5ORGxCWjBWSFEwTnhSMU5OTkRsQmQwVklRVEJKUVVKTmVIRlFNM3BxVm01bVJWSXJPRWMzYlhCU2VpdG5jVWhQTlcxV2RUZFdkM2RHVkhwaFUzWUtibkl2YnpCM1VscEljbXROUVdZdldIcGFRVUpoUzBob2VrVm1OMFJzT0hJNVdEWklTR0V4YTFKNVpGUXlha3RxVVdwQ1FVMUJORWRCTVZWa1JIZEZRZ292ZDFGRlFYZEpRM0JFUVZCQ1owNVdTRkpOUWtGbU9FVkNWRUZFUVZGSUwwMUNNRWRCTVZWa1JHZFJWMEpDVkVOdk9FcDFZMWg1VEVsbk4zcHRNV3BYQ213MmJtazVjRGhqUjNwQlMwSm5aM0ZvYTJwUFVGRlJSRUZuVGtsQlJFSkdRV2xCYzAxUldUVjFSRVZ4Vld0c1IxSlFZVVJ2UTBOaGFrSlNXSGgyTkZrS2EzVXJZM1JKZVhCWWEybG5ZVkZKYUVGTk5rRk5kbGRpYW05WFFVcE5lR05HZUZSUlMxWjRZamhsVmxkYVRrY3paMWswSzBFdk1DOUtUWHB1Q2kwdExTMHRSVTVFSUVORlVsUkpSa2xEUVZSRkxTMHRMUzBLCiAgICBzZXJ2ZXI6IGh0dHBzOi8vMTkyLjE2OC4xMDQuODU6NjQ0MwogIG5hbWU6IGRlZmF1bHQKY29udGV4dHM6Ci0gY29udGV4dDoKICAgIGNsdXN0ZXI6IGRlZmF1bHQKICAgIG5hbWVzcGFjZTogZGVmYXVsdAogICAgdXNlcjogaGFydmVzdGVyLWNsb3VkLXByb3ZpZGVyLWRlZmF1bHQtZGVmYXVsdAogIG5hbWU6IGhhcnZlc3Rlci1jbG91ZC1wcm92aWRlci1kZWZhdWx0LWRlZmF1bHQKY3VycmVudC1jb250ZXh0OiBoYXJ2ZXN0ZXItY2xvdWQtcHJvdmlkZXItZGVmYXVsdC1kZWZhdWx0CmtpbmQ6IENvbmZpZwpwcmVmZXJlbmNlczoge30KdXNlcnM6Ci0gbmFtZTogaGFydmVzdGVyLWNsb3VkLXByb3ZpZGVyLWRlZmF1bHQtZGVmYXVsdAogIHVzZXI6CiAgICB0b2tlbjogZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrdE1iRTh5TVMxUlNsTXpZbE5OYjA1VFVUVjVWa0ZFY25CVmQyRktTR0ZzYXpWcFVsYzBZVjlSYVhjaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbWhoY25abGMzUmxjaTFqYkc5MVpDMXdjbTkyYVdSbGNpMTBiMnRsYmlJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExtNWhiV1VpT2lKb1lYSjJaWE4wWlhJdFkyeHZkV1F0Y0hKdmRtbGtaWElpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1MWFXUWlPaUk0Wm1aaE4yWmhOeTAzTXpBM0xUUXdZVEl0WW1Nd05DMHpOMkpsTWpRMk9UZGpOV0lpTENKemRXSWlPaUp6ZVhOMFpXMDZjMlZ5ZG1salpXRmpZMjkxYm5RNlpHVm1ZWFZzZERwb1lYSjJaWE4wWlhJdFkyeHZkV1F0Y0hKdmRtbGtaWElpZlEuT3pBTkxYRE05TElfTy16Rk1HMlFQaTllQkYwcHFwdVhwMTBSS2xGM1Nxd0w1VEZ6NVBDRjRLRENOVEVDNHVkUDBZNTNYNkx4WTNHNy02bC02ZlI0OEVEWXpwRXNTZUtYSWljRlFieVpjd2pLTW44eFdLVDIzTXA3dnVpcmhLSTBlNndBckxhUnRrWDhOakdCX1BOY1AxUE1EeHZnalExTjhZWDN0OVprbmlCNGI5RElHUHJkT2dSTzh0dm50SURwZTg5SXA2VHR2b29lR1pySGJ1Tzlienh4RkpWY1laelotYmlUQjNPaTZoY245cFhQUWM0XzNiYVVYSmYxOS1idUdjZUNsRFVTTm50TWhHS0I0b1RCTjRZYmN5YVhGaWpnNlZTbVJ1ckNPS0ZZRE9sNVRKT21nR20xM2trVTRHMm1ydnBCOXVKcVBTUFdrdmd6NHRtTFpBCg==
  owner: root:root
  path: /etc/kubernetes/cloud-config
  permissions: '0644'

irishgordo · 2024-07-02T22:34:34Z

After a bit of configuration work I am able to have a:

custom RKE2 v1.27, 2 node (1 cp+etcd) (1 worker) cluster spun up, after creating a cloud-config user-data template, that held the information from curl -sfL https://raw.githubusercontent.com/harvester/cloud-provider-harvester/master/deploy/generate_addon.sh | bash -s harvester-cloud-provider default -> where the KUBECONFIG used had only the VIP & :6443, nothing else
MinIO RELEASE.2024-06-29T01-20-47Z, configured with https present, running on Harvester, to serve as the S3 compatible back-end for the custom-rke2 cluster (w/ Harvester as Cloud Provider) for etcd-snapshots via ensuring endpoint did just specify the IPv4/DNS (without https) protocol

Deployments leveraging CSI & CP, w/ nginx & pvc -> on the custom rke2 cluster v1.27 based with Rancher v2.8.2 & Harvester v1.3.1 to be a success.

There may just be a UI bug, but seems to be Rancher side with the way data is fed about the snapshots present.

Additionally, rancher-system-agent, systemd logs with journalctl -u rancher-system-agent.service --follow seem to be coming across correctly with:

opensuse@new-rke2-custom-01:~> systemctl status rancher-system-agent.service 
● rancher-system-agent.service - Rancher System Agent
     Loaded: loaded (/etc/systemd/system/rancher-system-agent.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2024-07-02 05:10:59 UTC; 16h ago
       Docs: https://www.rancher.com
   Main PID: 11855 (rancher-system-)
      Tasks: 14 (limit: 4915)
     CGroup: /system.slice/rancher-system-agent.service
             └─ 11855 /usr/local/bin/rancher-system-agent sentinel
opensuse@new-rke2-custom-01:~> sudo journaclt -u rancher-system-agent.service --follow
sudo: journaclt: command not found
opensuse@new-rke2-custom-01:~> sudo journalctl -u rancher-system-agent.service --follow
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956537     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956537     10498080 2024-07-02T21:42:17Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956537     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956537                              10498080 2024-07-02T21:42:17Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956621     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956621     10498080 2024-07-02T21:43:41Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956621     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956621                              10498080 2024-07-02T21:43:41Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956826     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956826                              10498080 2024-07-02T21:47:06Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956826     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956826     10498080 2024-07-02T21:47:06Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719957159     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719957159     10498080 2024-07-02T21:52:39Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719957159     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719957159                              10498080 2024-07-02T21:52:39Z"
Jul 02 22:02:41 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:41Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3 2>/dev/null] finished with err: <nil> and exit code: 0"
Jul 02 22:02:42 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:02:42Z" level=info msg="[K8s] updated plan secret fleet-default/custom-8e5e20291d80-machine-plan with feedback"
Jul 02 22:09:01 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:01Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee"
Jul 02 22:09:01 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:01Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.27.14-rke2r1 to directory /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0"
Jul 02 22:09:01 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:01Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Jul 02 22:09:01 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:01Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.27.14-rke2r1"
Jul 02 22:09:02 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:02Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0/installer.sh"
Jul 02 22:09:02 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:02Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0/rke2.linux-amd64.tar.gz"
Jul 02 22:09:02 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:02Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0/sha256sum-amd64.txt"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0/run.sh"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + check_target_mountpoint"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + mountpoint -q /usr/local"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + check_target_ro"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + touch /usr/local/.rke2-ro-test"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + test 0 -ne 0"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' /usr/local = /opt/rke2 ']'"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ cat /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + PRIOR_RESTART_STAMP=736900ca65517ef0f5e2f66f4a87f1b5c2d7bbf6230af08df12d61b4e7c6c459"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' -n '' ']'"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + RESTART=false"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0 INSTALL_RKE2_TAR_PREFIX=/usr/local installer.sh"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0/sha256sum-amd64.txt"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0/rke2.linux-amd64.tar.gz"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stdout]: [INFO]  verifying tarball"
Jul 02 22:09:03 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:03Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ awk '{print $1}'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ env"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ grep '^RKE2_'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ true"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + RKE2_ENV="
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' -n '' ']'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ env"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ true"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + PROXY_ENV_INFO="
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' -n '' ']'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: ++ awk '{print $1}'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 '!=' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ']'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' -z '' ']'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + INSTALL_RKE2_TYPE=server"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' -n '' ']'"
Jul 02 22:09:04 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:04Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + systemctl daemon-reload"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' '' = true ']'"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' server = server ']'"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + systemctl is-active --quiet rke2-agent"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + systemctl enable rke2-server"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + '[' true = true ']'"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_0:stderr]: + exit 0"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240702-220901/936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[Applyinator] Running command: rke2 [etcd-snapshot save]"
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --agent-token found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --cloud-provider-name found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --cni found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --disable-kube-proxy found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --etcd-expose-metrics found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --kube-controller-manager-arg found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --kube-controller-manager-arg found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --kube-controller-manager-extra-mount found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --kube-scheduler-arg found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --kube-scheduler-arg found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --kube-scheduler-extra-mount found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --node-label found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --node-label found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --node-taint found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --node-taint found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --private-registry found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Unknown flag --protect-kernel-defaults found in config.yaml, skipping\\n\""
Jul 02 22:09:05 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:05Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:05Z\" level=warning msg=\"Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation.\""
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:06Z\" level=info msg=\"Snapshot on-demand-new-rke2-custom-01-1719958146 saved.\""
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="[936d2528e0346852c390ba538eee8dbf09bd5e14bae1c90bde847634b35bfaee_1:stderr]: time=\"2024-07-02T22:09:06Z\" level=info msg=\"Snapshot on-demand-new-rke2-custom-01-1719958146 saved.\""
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="[Applyinator] Command rke2 [etcd-snapshot save] finished with err: <nil> and exit code: 0"
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="[K8s] updated plan secret fleet-default/custom-8e5e20291d80-machine-plan with feedback"
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (535231 vs 535294)"
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=error msg="error syncing 'fleet-default/custom-8e5e20291d80-machine-plan': handler secret-watch: secret received was too old, requeuing"
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf"
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.27.14-rke2r1 to directory /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0"
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Jul 02 22:09:06 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:06Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-rke2:v1.27.14-rke2r1"
Jul 02 22:09:07 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:07Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0/installer.sh"
Jul 02 22:09:07 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:07Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0/rke2.linux-amd64.tar.gz"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0/sha256sum-amd64.txt"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0/run.sh"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + check_target_mountpoint"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + mountpoint -q /usr/local"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + check_target_ro"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + touch /usr/local/.rke2-ro-test"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + test 0 -ne 0"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' /usr/local = /opt/rke2 ']'"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ cat /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + PRIOR_RESTART_STAMP=736900ca65517ef0f5e2f66f4a87f1b5c2d7bbf6230af08df12d61b4e7c6c459"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' -n 736900ca65517ef0f5e2f66f4a87f1b5c2d7bbf6230af08df12d61b4e7c6c459 ']'"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' 736900ca65517ef0f5e2f66f4a87f1b5c2d7bbf6230af08df12d61b4e7c6c459 '!=' 736900ca65517ef0f5e2f66f4a87f1b5c2d7bbf6230af08df12d61b4e7c6c459 ']'"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + RESTART=false"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0 INSTALL_RKE2_TAR_PREFIX=/usr/local installer.sh"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0/sha256sum-amd64.txt"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0/rke2.linux-amd64.tar.gz"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: [INFO]  verifying tarball"
Jul 02 22:09:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:08Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ awk '{print $1}'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ env"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ grep '^RKE2_'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ true"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + RKE2_ENV="
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' -n '' ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ env"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ true"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + PROXY_ENV_INFO="
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' -n '' ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: ++ awk '{print $1}'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 '!=' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' -z '' ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + INSTALL_RKE2_TYPE=server"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' -n 736900ca65517ef0f5e2f66f4a87f1b5c2d7bbf6230af08df12d61b4e7c6c459 ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + echo 736900ca65517ef0f5e2f66f4a87f1b5c2d7bbf6230af08df12d61b4e7c6c459"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + systemctl daemon-reload"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' '' = true ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' server = server ']'"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + systemctl is-active --quiet rke2-agent"
Jul 02 22:09:10 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:10Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + systemctl enable rke2-server"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' '' = true ']'"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stderr]: + '[' false = true ']'"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: Name                                        Location                                                                                     Size     Created"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: etcd-snapshot-new-rke2-custom-01-1719914404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719914404 22515744 2024-07-02T10:00:04Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: etcd-snapshot-new-rke2-custom-01-1719932403 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719932403 22515744 2024-07-02T15:00:03Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719943029     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943029     7708704  2024-07-02T17:57:09Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719943328     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943328     9875488  2024-07-02T18:02:08Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719943561     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943561     9945120  2024-07-02T18:06:01Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: etcd-snapshot-new-rke2-custom-01-1719950404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719950404 10498080 2024-07-02T20:00:04Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719955088     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719955088     10498080 2024-07-02T21:18:08Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719956537     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956537     10498080 2024-07-02T21:42:17Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719956621     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956621     10498080 2024-07-02T21:43:41Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719956826     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956826     10498080 2024-07-02T21:47:06Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719957159     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719957159     10498080 2024-07-02T21:52:39Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719958146     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719958146     10711072 2024-07-02T22:09:06Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240702-220906/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3 2>/dev/null]"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: Name                                        Location                                                                                     Size     Created"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719914404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719914404 22515744 2024-07-02T10:00:04Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719932403 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719932403 22515744 2024-07-02T15:00:03Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943029     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943029     7708704  2024-07-02T17:57:09Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943029     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943029                              7708704  2024-07-02T17:57:09Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943328     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943328                              9875488  2024-07-02T18:02:08Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943328     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943328     9875488  2024-07-02T18:02:08Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943561     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943561     9945120  2024-07-02T18:06:01Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943561     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943561                              9945120  2024-07-02T18:06:01Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719950404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719950404 10498080 2024-07-02T20:00:04Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719950404 s3://custom-rke2-bucket/etcd-snapshot-new-rke2-custom-01-1719950404                          10498080 2024-07-02T20:00:04Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719955088     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719955088                              10498080 2024-07-02T21:18:08Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719955088     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719955088     10498080 2024-07-02T21:18:08Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956537     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956537                              10498080 2024-07-02T21:42:17Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956537     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956537     10498080 2024-07-02T21:42:17Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956621     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956621                              10498080 2024-07-02T21:43:41Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956621     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956621     10498080 2024-07-02T21:43:41Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956826     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956826     10498080 2024-07-02T21:47:06Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956826     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956826                              10498080 2024-07-02T21:47:06Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719957159     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719957159     10498080 2024-07-02T21:52:39Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719957159     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719957159                              10498080 2024-07-02T21:52:39Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719958146     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719958146     10711072 2024-07-02T22:09:06Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719958146     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719958146                              10711072 2024-07-02T22:09:06Z"
Jul 02 22:09:11 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:11Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3 2>/dev/null] finished with err: <nil> and exit code: 0"
Jul 02 22:09:12 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:12Z" level=info msg="[K8s] updated plan secret fleet-default/custom-8e5e20291d80-machine-plan with feedback"
Jul 02 22:09:12 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:12Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (535305 vs 535363)"
Jul 02 22:09:12 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:09:12Z" level=error msg="error syncing 'fleet-default/custom-8e5e20291d80-machine-plan': handler secret-watch: secret received was too old, requeuing"
Jul 02 22:19:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:08Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240702-221908/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0"
Jul 02 22:19:08 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:08Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: Name                                        Location                                                                                     Size     Created"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: etcd-snapshot-new-rke2-custom-01-1719914404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719914404 22515744 2024-07-02T10:00:04Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: etcd-snapshot-new-rke2-custom-01-1719932403 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719932403 22515744 2024-07-02T15:00:03Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719943029     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943029     7708704  2024-07-02T17:57:09Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719943328     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943328     9875488  2024-07-02T18:02:08Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719943561     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943561     9945120  2024-07-02T18:06:01Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: etcd-snapshot-new-rke2-custom-01-1719950404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719950404 10498080 2024-07-02T20:00:04Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719955088     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719955088     10498080 2024-07-02T21:18:08Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719956537     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956537     10498080 2024-07-02T21:42:17Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719956621     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956621     10498080 2024-07-02T21:43:41Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719956826     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956826     10498080 2024-07-02T21:47:06Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719957159     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719957159     10498080 2024-07-02T21:52:39Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_0:stdout]: on-demand-new-rke2-custom-01-1719958146     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719958146     10711072 2024-07-02T22:09:06Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240702-221908/b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3 2>/dev/null]"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: Name                                        Location                                                                                     Size     Created"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719914404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719914404 22515744 2024-07-02T10:00:04Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719932403 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719932403 22515744 2024-07-02T15:00:03Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943029     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943029                              7708704  2024-07-02T17:57:09Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943029     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943029     7708704  2024-07-02T17:57:09Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943328     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943328                              9875488  2024-07-02T18:02:08Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943328     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943328     9875488  2024-07-02T18:02:08Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943561     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943561     9945120  2024-07-02T18:06:01Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719943561     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943561                              9945120  2024-07-02T18:06:01Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719950404 s3://custom-rke2-bucket/etcd-snapshot-new-rke2-custom-01-1719950404                          10498080 2024-07-02T20:00:04Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: etcd-snapshot-new-rke2-custom-01-1719950404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719950404 10498080 2024-07-02T20:00:04Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719955088     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719955088                              10498080 2024-07-02T21:18:08Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719955088     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719955088     10498080 2024-07-02T21:18:08Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956537     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956537                              10498080 2024-07-02T21:42:17Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956537     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956537     10498080 2024-07-02T21:42:17Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956621     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956621     10498080 2024-07-02T21:43:41Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956621     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956621                              10498080 2024-07-02T21:43:41Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956826     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719956826     10498080 2024-07-02T21:47:06Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719956826     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719956826                              10498080 2024-07-02T21:47:06Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719957159     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719957159     10498080 2024-07-02T21:52:39Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719957159     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719957159                              10498080 2024-07-02T21:52:39Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719958146     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719958146                              10711072 2024-07-02T22:09:06Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[b66698c29dbc3155386c5a4457b1aa4271533e1026e89a7de59275c8ab6a61bf_1:stdout]: on-demand-new-rke2-custom-01-1719958146     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719958146     10711072 2024-07-02T22:09:06Z"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3 2>/dev/null] finished with err: <nil> and exit code: 0"
Jul 02 22:19:09 new-rke2-custom-01 rancher-system-agent[11855]: time="2024-07-02T22:19:09Z" level=info msg="[K8s] updated plan secret fleet-default/custom-8e5e20291d80-machine-plan with feedback"

And from the v2.8.2, vCluster Addon Based Rancher v2.8.2 management instance they also seemingly see the correct etcd snapshot information:

2024/07/02 21:52:35 [INFO] [planner] rkecluster fleet-default/new: refreshing etcd create state
2024/07/02 21:52:35 [INFO] [planner] rkecluster fleet-default/new: starting etcd snapshot on node new-rke2-custom-01
2024/07/02 21:52:35 [INFO] [planner] rkecluster fleet-default/new: waiting for etcd snapshot on node new-rke2-custom-01
2024/07/02 21:52:35 [INFO] [planner] rkecluster fleet-default/new: waiting for etcd snapshot on node new-rke2-custom-01
2024/07/02 21:52:35 [INFO] [planner] rkecluster fleet-default/new: waiting for etcd snapshot on node new-rke2-custom-01
2024/07/02 21:52:37 [INFO] Error while handling cluster log: write tcp 10.52.0.94:80->10.52.0.29:53636: write: broken pipe
2024/07/02 21:52:37 [INFO] Error while handling cluster log: write tcp 10.52.0.94:80->10.52.0.29:35698: write: broken pipe
2024/07/02 21:52:37 [INFO] Error while handling cluster log: write tcp 10.52.0.94:80->10.52.0.29:55408: write: broken pipe
2024/07/02 21:52:40 [INFO] [planner] rkecluster fleet-default/new: waiting for etcd snapshot on node new-rke2-custom-01
2024/07/02 21:52:40 [INFO] [planner] rkecluster fleet-default/new: refreshing etcd create state
2024/07/02 21:52:40 [INFO] [planner] rkecluster fleet-default/new: configuring bootstrap node(s) custom-8e5e20291d80: waiting for plan to be applied
2024/07/02 21:52:40 [INFO] [planner] rkecluster fleet-default/new: configuring bootstrap node(s) custom-8e5e20291d80: waiting for plan to be applied
2024/07/02 21:52:40 [INFO] [planner] rkecluster fleet-default/new: configuring bootstrap node(s) custom-8e5e20291d80: waiting for plan to be applied
2024/07/02 21:52:40 [INFO] [planner] rkecluster fleet-default/new: configuring bootstrap node(s) custom-8e5e20291d80: waiting for plan to be applied
2024/07/02 21:52:44 [INFO] [planner] rkecluster fleet-default/new: configuring bootstrap node(s) custom-8e5e20291d80: waiting for plan to be applied
2024/07/02 21:52:44 [INFO] [planner] rkecluster fleet-default/new: configuring bootstrap node(s) custom-8e5e20291d80: waiting for plan to be applied
2024/07/02 21:52:46 [INFO] [snapshotbackpopulate] rkecluster fleet-default/new: processing configmap kube-system/rke2-etcd-snapshots
2024/07/02 21:52:46 [INFO] [planner] rkecluster fleet-default/new: configuring bootstrap node(s) custom-8e5e20291d80: waiting for plan to be applied
2024/07/02 21:52:46 [INFO] [planner] rkecluster fleet-default/new: custom-8e5e20291d80
2024/07/02 21:52:46 [INFO] [planner] rkecluster fleet-default/new: refreshing etcd create state
2024/07/02 21:52:46 [INFO] [snapshotbackpopulate] rkecluster fleet-default/new: processing configmap kube-system/rke2-etcd-snapshots

And additionally, directly on the CP + ETCD Node (15.5 Leap Based OS) they also seem to see the correct information with:

opensuse@new-rke2-custom-01:~> sudo /usr/local/bin/rke2 etcd-snapshot ls --debug
WARN[0000] Unknown flag --agent-token found in config.yaml, skipping 
WARN[0000] Unknown flag --cloud-provider-name found in config.yaml, skipping 
WARN[0000] Unknown flag --cni found in config.yaml, skipping 
WARN[0000] Unknown flag --disable-kube-proxy found in config.yaml, skipping 
WARN[0000] Unknown flag --etcd-expose-metrics found in config.yaml, skipping 
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping 
WARN[0000] Unknown flag --kube-controller-manager-arg found in config.yaml, skipping 
WARN[0000] Unknown flag --kube-controller-manager-arg found in config.yaml, skipping 
WARN[0000] Unknown flag --kube-controller-manager-extra-mount found in config.yaml, skipping 
WARN[0000] Unknown flag --kube-scheduler-arg found in config.yaml, skipping 
WARN[0000] Unknown flag --kube-scheduler-arg found in config.yaml, skipping 
WARN[0000] Unknown flag --kube-scheduler-extra-mount found in config.yaml, skipping 
WARN[0000] Unknown flag --node-label found in config.yaml, skipping 
WARN[0000] Unknown flag --node-label found in config.yaml, skipping 
WARN[0000] Unknown flag --node-taint found in config.yaml, skipping 
WARN[0000] Unknown flag --node-taint found in config.yaml, skipping 
WARN[0000] Unknown flag --private-registry found in config.yaml, skipping 
WARN[0000] Unknown flag --protect-kernel-defaults found in config.yaml, skipping 
WARN[0000] Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation. 
Name                                        Location                                                                                     Size     Created
etcd-snapshot-new-rke2-custom-01-1719914404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719914404 22515744 2024-07-02T10:00:04Z
etcd-snapshot-new-rke2-custom-01-1719932403 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719932403 22515744 2024-07-02T15:00:03Z
on-demand-new-rke2-custom-01-1719943029     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943029                              7708704  2024-07-02T17:57:09Z
on-demand-new-rke2-custom-01-1719943029     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943029     7708704  2024-07-02T17:57:09Z
on-demand-new-rke2-custom-01-1719943328     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943328                              9875488  2024-07-02T18:02:08Z
on-demand-new-rke2-custom-01-1719943328     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943328     9875488  2024-07-02T18:02:08Z
on-demand-new-rke2-custom-01-1719943561     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719943561                              9945120  2024-07-02T18:06:01Z
on-demand-new-rke2-custom-01-1719943561     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719943561     9945120  2024-07-02T18:06:01Z
etcd-snapshot-new-rke2-custom-01-1719950404 s3://custom-rke2-bucket/etcd-snapshot-new-rke2-custom-01-1719950404                          10498080 2024-07-02T20:00:04Z
etcd-snapshot-new-rke2-custom-01-1719950404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-new-rke2-custom-01-1719950404 10498080 2024-07-02T20:00:04Z
on-demand-new-rke2-custom-01-1719955088     s3://custom-rke2-bucket/on-demand-new-rke2-custom-01-1719955088                              10498080 2024-07-02T21:18:08Z
on-demand-new-rke2-custom-01-1719955088     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-new-rke2-custom-01-1719955088     10498080 2024-07-02T21:18:08Z

irishgordo · 2024-07-03T00:32:46Z

Restarting the rke2-server.service on the Control-Plane-&-ETCD Node, seems to have fixed everything 😄 👍

Once restarted, the MinIO based snapshots all the data was reconciled and seems to have balanced out 😄

FrankYang0529 · 2024-07-03T14:11:25Z

Hi @irishgordo, sounds good that you can run all scenario without error. After reading the document again, I found that we already mentioned using /etc/rancher/rke2/rke2.yaml to generate kubeconfig for guest clusters in the note block.

:::note
The generate_addon.sh script depends on kubectl and jq to operate the Harvester cluster.

The script needs access to the Harvester Cluster kubeconfig to work. You can find the kubeconfig file from one of the Harvester management nodes in the /etc/rancher/rke2/rke2.yaml path. The content is like following. Remember to change server IP to the VIP address.

jillian-maroket

Review done

docs/rancher/cloud-provider.md

Co-authored-by: Jillian <[email protected]>

jillian-maroket

LGTM. Thanks!

starbops

Generally LGTM, just some nits.

docs/rancher/cloud-provider.md

Signed-off-by: PoAn Yang <[email protected]>

FrankYang0529 · 2024-07-04T06:38:35Z

Hi @starbops, I addressed all comments. Thanks for the review.

starbops

LGTM, thank you!

bk201 · 2024-07-04T07:41:06Z

docs/rancher/cloud-provider.md

+
+    ![](/img/v1.3/rancher/custom-cluster-vm-user-data.png)
+
+1. On the **Basics** tab of the **Cluster Configuration** screen, select **Harvester** as the **Cloud Provider** and then select **Create** to spin up the cluster.


In the doc preview, the ordered list starts with 1. again. I'm not sure if the intention is to continue with 3.. And we should mention here the user should go to the Rancher's GUI, but not Harvester's GUI anymore.

Please also add the doc source to v1.3/v1.2 versions if needed (it's currently only in the dev version).

Fixed it and also migrated the document to v1.3/v1.2. Thanks.

Signed-off-by: PoAn Yang <[email protected]>

bk201

LGTM, thank you!

FrankYang0529 mentioned this pull request Mar 12, 2024

[DOC/BUG] Unable to Provision RKE2 Custom Cluster w/ Harvester Cloud Provider harvester/harvester#4077

Closed

FrankYang0529 requested review from bk201 and irishgordo March 12, 2024 08:52

jillian-maroket reviewed Mar 13, 2024

View reviewed changes

starbops reviewed Mar 14, 2024

View reviewed changes

docs/rancher/cloud-provider.md Outdated Show resolved Hide resolved

FrankYang0529 force-pushed the HARV-4077 branch from f08a606 to b1b911a Compare March 26, 2024 08:41

starbops self-requested a review July 2, 2024 00:50

FrankYang0529 and others added 8 commits July 2, 2024 10:41

feat: add steps to deploying to the RKE2 custom cluster

b89f0ab

Signed-off-by: PoAn Yang <[email protected]>

Update docs/rancher/cloud-provider.md

ee028f8

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

8ff2dee

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

834d784

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

0b14a97

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

63a318a

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

172b9be

Co-authored-by: Jillian <[email protected]>

refactor: move note close to step title

59abebf

Signed-off-by: PoAn Yang <[email protected]>

FrankYang0529 force-pushed the HARV-4077 branch from b1b911a to 59abebf Compare July 2, 2024 03:00

jillian-maroket reviewed Jul 4, 2024

View reviewed changes

docs/rancher/cloud-provider.md Outdated Show resolved Hide resolved

docs/rancher/cloud-provider.md Outdated Show resolved Hide resolved

docs/rancher/cloud-provider.md Outdated Show resolved Hide resolved

docs/rancher/cloud-provider.md Outdated Show resolved Hide resolved

FrankYang0529 and others added 4 commits July 4, 2024 10:57

Update docs/rancher/cloud-provider.md

5ff1fa5

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

f0c5440

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

19916c5

Co-authored-by: Jillian <[email protected]>

Update docs/rancher/cloud-provider.md

729e5c8

Co-authored-by: Jillian <[email protected]>

jillian-maroket approved these changes Jul 4, 2024

View reviewed changes

starbops reviewed Jul 4, 2024

View reviewed changes

docs/rancher/cloud-provider.md Outdated Show resolved Hide resolved

docs/rancher/cloud-provider.md Outdated Show resolved Hide resolved

fix: fix note block

7a26ae2

Signed-off-by: PoAn Yang <[email protected]>

starbops self-requested a review July 4, 2024 06:43

starbops approved these changes Jul 4, 2024

View reviewed changes

bk201 reviewed Jul 4, 2024

View reviewed changes

FrankYang0529 added 2 commits July 4, 2024 16:08

fix: fix list number

ccbc8e2

Signed-off-by: PoAn Yang <[email protected]>

feat: backport rke2 custom cluster to v1.2, v1.3

db49d78

Signed-off-by: PoAn Yang <[email protected]>

bk201 approved these changes Jul 4, 2024

View reviewed changes

FrankYang0529 merged commit bb87eb6 into harvester:main Jul 5, 2024
2 checks passed

FrankYang0529 deleted the HARV-4077 branch July 5, 2024 02:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add steps to deploying to the RKE2 custom cluster #525

feat: add steps to deploying to the RKE2 custom cluster #525

FrankYang0529 commented Mar 12, 2024

harvesterhci-io-github-bot commented Mar 12, 2024 •

edited

Loading

jillian-maroket left a comment

starbops left a comment

irishgordo commented Jul 1, 2024

github-actions bot commented Jul 2, 2024 •

edited

Loading

FrankYang0529 commented Jul 2, 2024

irishgordo commented Jul 2, 2024 •

edited

Loading

irishgordo commented Jul 2, 2024

irishgordo commented Jul 3, 2024

FrankYang0529 commented Jul 3, 2024

jillian-maroket left a comment

jillian-maroket left a comment

starbops left a comment

FrankYang0529 commented Jul 4, 2024

starbops left a comment

bk201 Jul 4, 2024

FrankYang0529 Jul 4, 2024

bk201 left a comment


		![](/img/v1.3/rancher/custom-cluster-vm-user-data.png)

		1. On the Basics tab of the Cluster Configuration screen, select Harvester as the Cloud Provider and then select Create to spin up the cluster.

feat: add steps to deploying to the RKE2 custom cluster #525

feat: add steps to deploying to the RKE2 custom cluster #525

Conversation

FrankYang0529 commented Mar 12, 2024

harvesterhci-io-github-bot commented Mar 12, 2024 • edited Loading

✅ Deploy Preview for harvester-preview ready!

jillian-maroket left a comment

Choose a reason for hiding this comment

starbops left a comment

Choose a reason for hiding this comment

irishgordo commented Jul 1, 2024

github-actions bot commented Jul 2, 2024 • edited Loading

FrankYang0529 commented Jul 2, 2024

irishgordo commented Jul 2, 2024 • edited Loading

irishgordo commented Jul 2, 2024

irishgordo commented Jul 3, 2024

FrankYang0529 commented Jul 3, 2024

jillian-maroket left a comment

Choose a reason for hiding this comment

jillian-maroket left a comment

Choose a reason for hiding this comment

starbops left a comment

Choose a reason for hiding this comment

FrankYang0529 commented Jul 4, 2024

starbops left a comment

Choose a reason for hiding this comment

bk201 Jul 4, 2024

Choose a reason for hiding this comment

FrankYang0529 Jul 4, 2024

Choose a reason for hiding this comment

bk201 left a comment

Choose a reason for hiding this comment

harvesterhci-io-github-bot commented Mar 12, 2024 •

edited

Loading

github-actions bot commented Jul 2, 2024 •

edited

Loading

irishgordo commented Jul 2, 2024 •

edited

Loading