Terraform apply always fails with apiServer.certSANs: Invalid value: ""

[sadlil]

Trying to create a simple minikube_cluster resource with terraform and terraform-provider-minikube failes with the following errors

│ Error: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.26.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
│ stdout:
│ 
│ stderr:
│ W0320 15:53:06.074140    6026 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
│ apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
│ To see the stack trace of this error execute with --v=5 or higher
│ 
│ 
│   with module.minikube_cluster.minikube_cluster.periscope,
│   on cluster/minikube.tf line 5, in resource "minikube_cluster" "periscope":
│    5: resource "minikube_cluster" "periscope" {
│

Looking at the rootcasue it seems with the provider the kubeadm config contains a apiServer.certSANs value with "".

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  certSANs: ["127.0.0.1", "localhost", ""]

The "" string entry is invalid.

I looked into the implementation and this implementations seems to be using minikube library instead of just using os.Exec command to start a local minikube. This way we are missing a lot of default check and runtime override done by minikube cli.

[scott-the-programmer]

Thanks for reporting this! Would you mind sharing the terraform snippet for

resource "minikube_cluster" "periscope" {

[sadlil]

Its here. https://github.com/sadlil/system-samples/blob/main/periscope/infra/cluster/minikube.tf. Please Ignore the subnet and apiserver_ips field. I tried to override with various values to see if I can ignore the error.

[scott-the-programmer]

I think I've gotten to the root of the issue (both with the tf provider and the minikube CLI directly). This is primarily due to minikube being in an inconsistent state where there is an existing node (in this case a docker container) running, but minikube wasn't able to finish bootstrapping. The host machines state would look something like

minikube profile:

❯ minikube profile list
|---------------------------------|-----------|---------|----|------|---------|---------|-------|--------|
|             Profile             | VM Driver | Runtime | IP | Port | Version | Status  | Nodes | Active |
|---------------------------------|-----------|---------|----|------|---------|---------|-------|--------|
| minikube                        | docker    | docker  |    | 8443 | v1.26.1 | Stopped |     1 | *      |
|---------------------------------|-----------|---------|----|------|---------|---------|-------|--------|

docker:

❯ docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS          PORTS                                                                                                                                  NAMES
138d6ff342c3   gcr.io/k8s-minikube/kicbase:v0.0.37   "/usr/local/bin/entr…"   4 minutes ago    Up 4 minutes    127.0.0.1:52666->22/tcp, 127.0.0.1:52667->2376/tcp, 127.0.0.1:52669->5000/tcp, 127.0.0.1:52665->8443/tcp, 127.0.0.1:52668->32443/tcp   minikube

minikube start:

❯ minikube start --alsologtostderr --v=2
...

W0322 21:37:19.366060   85748 out.go:239] ❌  Exiting due to K8S_INVALID_CERT_HOSTNAME: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.26.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
stdout:

stderr:
W0322 08:37:18.725784    4716 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
To see the stack trace of this error execute with --v=5 or higher

❌  Exiting due to K8S_INVALID_CERT_HOSTNAME: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.26.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 3
stdout:

stderr:
W0322 08:37:18.725784    4716 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
apiServer.certSANs: Invalid value: "": altname is not a valid IP address, DNS label or a DNS label with subdomain wildcards: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'); a wildcard DNS-1123 subdomain must start with '*.', followed by a valid DNS subdomain, which must consist of lower case alphanumeric characters, '-' or '.' and end with an alphanumeric character (e.g. '*.example.com', regex used for validation is '\*\.[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
To see the stack trace of this error execute with --v=5 or higher

W0322 21:37:19.366616   85748 out.go:239] 💡  Suggestion: The certificate hostname provided appears to be invalid (may be a minikube bug, try 'minikube delete')
💡  Suggestion: The certificate hostname provided appears to be invalid (may be a minikube bug, try 'minikube delete')
W0322 21:37:19.366697   85748 out.go:239] 🍿  Related issue: https://github.com/kubernetes/minikube/issues/9175
🍿  Related issue: https://github.com/kubernetes/minikube/issues/9175
W0322 21:37:19.366710   85748 out.go:239] 

W0322 21:37:19.368017   85748 out.go:239] ╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
I0322 21:37:19.462785   85748 out.go:177] 

I think in this case, running minikube delete --profile=periscope would be the best workaround as it'll clean the current configuration, allowing you to spin up fresh minikube cluster.

[scott-the-programmer]

Although one interesting idea is we could set the IP directly on the node. Minikube currently infers the IP from the driver much later in the provisioning, however, I don't see any reason this couldn't be set earlier.

I'll have to play around with it a bit to see if this is feasible. The main advantage is it would provide a better recovery path without having to delete the cluster manually or mess around with terraform state

[sadlil]

Hey @scott-the-programmer , I have actually ran minikube delete --profile=periscope multiple times. I have also done minikube delete --all --purge before running the terraform scripts. Still gave me the same issue.

[scott-the-programmer]

Hey @scott-the-programmer , I have actually ran minikube delete --profile=periscope multiple times. I have also done minikube delete --all --purge before running the terraform scripts. Still gave me the same issue.

I'm wondering if #66 fixes the issue - I noticed similar behavior due to the provider using the wrong ssh client by default. This would result in a partially configured cluster returning the apiServer.certSANs: Invalid value: "" error 🤔

[caerulescens]

I've been using mainly kvm2 and qemu2 drivers, and I notice that there's upstream quirks with minikube itself sometimes. There's a number of things I'm going to report.

[pgebert]

Hi, I am running into the same apiServer.certSANs: Invalid value error using v0.4.0 using docker driver on Windows. #66 didn't solve it for me.

[scott-the-programmer]

Hey @pgebert - apiServer.certSANs: Invalid value is usually a symptom of running minikube against an previously incomplete minikube cluster.

If you delete the cluster with minikube delete and/or delete the cluster's files in $HOME/.minikube, are you able to reproduce the issue?