fix: docker driver does not work when nodes>=2

[caerulescens]

This is a continuation of #131 for fixing docker driver multi-node usage. See logs below.

---

There is still an issue with the docker driver in v0.3.9; both of the following configurations need to work for this issue to be solved:

Configuration using cni=auto:

resource "minikube_cluster" "default" {
  cluster_name        = "dev-local-docker"
  driver              = "docker"
  nodes               = 3
  cni                 = "auto"
}

Configuration without cni=auto (logged below):

resource "minikube_cluster" "default" {
  cluster_name        = "dev-local-docker"
  driver              = "docker"
  nodes               = 3
}

---

Logs:

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of scott-the-programmer/minikube from the dependency lock file
- Using previously-installed scott-the-programmer/minikube v0.3.9

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # minikube_cluster.default will be created
  + resource "minikube_cluster" "default" {
      + addons                     = [
          + "default-storageclass",
          + "metrics-server",
          + "storage-provisioner",
        ]
      + apiserver_ips              = (known after apply)
      + apiserver_name             = "minikubeCA"
      + apiserver_names            = (known after apply)
      + apiserver_port             = 8443
      + auto_pause_interval        = 1
      + auto_update_drivers        = true
      + base_image                 = "gcr.io/k8s-minikube/kicbase:v0.0.42@sha256:d35ac07dfda971cabee05e0deca8aeac772f885a5348e1a0c0b0a36db20fcfc0"
      + cache_images               = true
      + cert_expiration            = 1576800
      + client_certificate         = (sensitive value)
      + client_key                 = (sensitive value)
      + cluster_ca_certificate     = (sensitive value)
      + cluster_name               = "dev-local-docker"
      + container_runtime          = "docker"
      + cpus                       = 8
      + delete_on_failure          = false
      + disable_driver_mounts      = false
      + disable_metrics            = false
      + disable_optimizations      = false
      + disk_size                  = "32768mb"
      + dns_domain                 = "cluster.local"
      + dns_proxy                  = false
      + download_only              = false
      + driver                     = "docker"
      + dry_run                    = false
      + embed_certs                = false
      + enable_default_cni         = false
      + extra_disks                = 0
      + force                      = false
      + force_systemd              = false
      + host                       = (known after apply)
      + host_dns_resolver          = true
      + host_only_cidr             = "192.168.59.1/24"
      + host_only_nic_type         = "virtio"
      + hyperkit_vsock_ports       = (known after apply)
      + hyperv_use_external_switch = false
      + id                         = (known after apply)
      + insecure_registry          = (known after apply)
      + install_addons             = true
      + interactive                = true
      + iso_url                    = (known after apply)
      + keep_context               = false
      + kvm_gpu                    = false
      + kvm_hidden                 = false
      + kvm_network                = "default"
      + kvm_numa_count             = 1
      + kvm_qemu_uri               = "qemu:///system"
      + memory                     = "8192mb"
      + mount                      = false
      + mount_9p_version           = "9p2000.L"
      + mount_gid                  = "docker"
      + mount_msize                = 262144
      + mount_port                 = 0
      + mount_string               = "/home:/minikube-host"
      + mount_type                 = "9p"
      + mount_uid                  = "docker"
      + namespace                  = "default"
      + nat_nic_type               = "virtio"
      + native_ssh                 = true
      + nfs_share                  = (known after apply)
      + nfs_shares_root            = "/nfsshares"
      + no_kubernetes              = false
      + no_vtx_check               = false
      + nodes                      = 3
      + ports                      = (known after apply)
      + preload                    = true
      + registry_mirror            = (known after apply)
      + service_cluster_ip_range   = "10.96.0.0/12"
      + ssh_port                   = 22
      + ssh_user                   = "root"
      + vm                         = false
      + wait_timeout               = 6
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + client_certificate     = (sensitive value)
  + client_key             = (sensitive value)
  + cluster_ca_certificate = (sensitive value)
  + host                   = (known after apply)
  + id                     = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

minikube_cluster.default: Creating...
minikube_cluster.default: Still creating... [10s elapsed]
minikube_cluster.default: Still creating... [20s elapsed]
minikube_cluster.default: Still creating... [30s elapsed]
minikube_cluster.default: Still creating... [40s elapsed]
minikube_cluster.default: Still creating... [50s elapsed]
minikube_cluster.default: Still creating... [1m0s elapsed]
minikube_cluster.default: Still creating... [1m10s elapsed]
minikube_cluster.default: Still creating... [1m20s elapsed]
minikube_cluster.default: Still creating... [1m30s elapsed]
minikube_cluster.default: Still creating... [1m40s elapsed]
minikube_cluster.default: Still creating... [1m50s elapsed]
minikube_cluster.default: Still creating... [2m0s elapsed]
minikube_cluster.default: Still creating... [2m10s elapsed]
minikube_cluster.default: Still creating... [2m20s elapsed]
minikube_cluster.default: Still creating... [2m30s elapsed]
minikube_cluster.default: Still creating... [2m40s elapsed]
minikube_cluster.default: Still creating... [2m50s elapsed]
minikube_cluster.default: Still creating... [3m0s elapsed]
minikube_cluster.default: Still creating... [3m10s elapsed]
minikube_cluster.default: Still creating... [3m20s elapsed]
minikube_cluster.default: Still creating... [3m30s elapsed]
minikube_cluster.default: Still creating... [3m40s elapsed]
minikube_cluster.default: Still creating... [3m50s elapsed]
minikube_cluster.default: Still creating... [4m0s elapsed]
minikube_cluster.default: Still creating... [4m10s elapsed]
minikube_cluster.default: Still creating... [4m20s elapsed]
minikube_cluster.default: Still creating... [4m30s elapsed]
minikube_cluster.default: Still creating... [4m40s elapsed]
minikube_cluster.default: Still creating... [4m50s elapsed]
╷
│ Error: joining cp: error joining worker node to cluster: kubeadm join: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.28.3:$PATH" kubeadm join control-plane.minikube.internal:8443 --token 917zc4.932xquy3qc0a9pet --discovery-token-ca-cert-hash sha256:3d1b312f8cca25fc2b6360a0e7a1f38804a4eb1c280a749d9107ab72d0431b63 --ignore-preflight-errors=all --cri-socket /var/run/cri-dockerd.sock --node-name=dev-local-docker-m02": Process exited with status 1
│ stdout:
│ [preflight] Running pre-flight checks
│ [preflight] The system verification failed. Printing the output from the verification:
│ KERNEL_VERSION: 6.1.0-16-amd64
│ OS: Linux
│ CGROUPS_CPU: enabled
│ CGROUPS_CPUSET: enabled
│ CGROUPS_DEVICES: enabled
│ CGROUPS_FREEZER: enabled
│ CGROUPS_MEMORY: enabled
│ CGROUPS_PIDS: enabled
│ CGROUPS_HUGETLB: enabled
│ CGROUPS_IO: enabled
│ [preflight] Reading configuration from the cluster...
│ [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
│ [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
│ [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
│ [kubelet-start] Starting the kubelet
│ [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
│ [kubelet-check] Initial timeout of 40s passed.
│ 
│ stderr:
│ W0204 00:38:30.046049    2382 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
│       [WARNING FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
│       [WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
│       [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.1.0-16-amd64\n", err: exit status 1
│       [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
│       [WARNING Port-10250]: Port 10250 is in use
│       [WARNING FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
│ error execution phase kubelet-start: error uploading crisocket: Unauthorized
│ To see the stack trace of this error execute with --v=5 or higher
│ 
│ 
│   with minikube_cluster.default,
│   on main.tf line 10, in resource "minikube_cluster" "default":
│   10: resource "minikube_cluster" "default" {
│ 
╵
...

[scott-the-programmer]

Hmm, I wonder if this is due defaulting the container runtime (to docker) when it previously was set to blank. I'll dig into this soon and see if I can get to the bottom of it!

[scott-the-programmer]

This.. is odd. I think i've found a bug with upstream. I see a similar error when I run

minikube start --driver=docker --nodes=3 -p "laaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaarge" -v=8 --alsologtostderr

it seems sensitive to the cluster name for some reason 😕

I can't seem to reproduce it when I provide it a smaller name i.e.

resource "minikube_cluster" "small" {
  cluster_name        = "small"
  driver              = "docker"
  nodes               = 3
  cni                 = "auto"
}

I'm also able to run this snippet fine

resource "minikube_cluster" "default" {
  cluster_name        = "dev-local-docker"
  driver              = "docker"
  nodes               = 3
}

[caerulescens]

Interesting; I'm going to do a cleaner test of this because I was checking v0.3.9 quickly.