New 4.7 cluster still not working correctly

[tman24]

I've finally got an OKD 4.7 cluster installed - it's been a steep learning curve, but I'm nearly there, but there are still some problems. The cluster has 3 CP nodes and 7 workers at this time.

The bootstrap finished ok, but some components (inc console) haven't come up. I've also noticed there's no password been created in /install_dir/auth/kubeadmin-password (no file). Here's some info;

oc get nodes
NAME       STATUS   ROLES    AGE   VERSION
compute1   Ready    worker   23h   v1.20.0+01994f4-1091
compute2   Ready    worker   22h   v1.20.0+01994f4-1091
compute3   Ready    worker   22h   v1.20.0+01994f4-1091
compute4   Ready    worker   21h   v1.20.0+01994f4-1091
compute5   Ready    worker   21h   v1.20.0+01994f4-1091
compute6   Ready    worker   21h   v1.20.0+01994f4-1091
compute7   Ready    worker   20h   v1.20.0+01994f4-1091
master1    Ready    master   41h   v1.20.0+01994f4-1091
master2    Ready    master   41h   v1.20.0+01994f4-1091
master3    Ready    master   41h   v1.20.0+01994f4-1091

oc get pod -n openshift-console
NAME                         READY   STATUS             RESTARTS   AGE
console-5d56f856d4-nn22h     0/1     Running            245        23h
console-5d56f856d4-s9rf2     0/1     CrashLoopBackOff   245        23h
downloads-7dbbfd7b49-2w4pc   1/1     Running            0          41h
downloads-7dbbfd7b49-68scg   1/1     Running            0          41h

oc describe pod console-5d56f856d4-s9rf2 -n openshift-console
Name:                 console-5d56f856d4-s9rf2
Namespace:            openshift-console
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 master1/192.168.30.21
Start Time:           Thu, 23 Sep 2021 10:21:07 +0100
Labels:               app=console
                      component=ui
                      pod-template-hash=5d56f856d4
Annotations:          console.openshift.io/console-config-version: 25395
                      console.openshift.io/default-ingress-cert-config-version: 21873
                      console.openshift.io/image: quay.io/openshift/okd-content@sha256:806090370bd0cd3e5d5113f0775ab582f82ff625bf0dca49a2a6a1792d5718b8
                      console.openshift.io/oauth-secret-version: 21920
                      console.openshift.io/proxy-config-version: 617
                      console.openshift.io/service-ca-config-version: 21904
                      console.openshift.io/trusted-ca-config-version: 22447
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.129.0.40"
                            ],
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.129.0.40"
                            ],
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: restricted
Status:               Running
IP:                   10.129.0.40
IPs:
  IP:           10.129.0.40
Controlled By:  ReplicaSet/console-5d56f856d4
Containers:
  console:
    Container ID:  cri-o://ae488d28dde31dd4f9b8a460ef50b12895e553b2b9ddfa9760fc8565875e104a
    Image:         quay.io/openshift/okd-content@sha256:806090370bd0cd3e5d5113f0775ab582f82ff625bf0dca49a2a6a1792d5718b8
    Image ID:      quay.io/openshift/okd-content@sha256:806090370bd0cd3e5d5113f0775ab582f82ff625bf0dca49a2a6a1792d5718b8
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /opt/bridge/bin/bridge
      --public-dir=/opt/bridge/static
      --config=/var/console-config/console-config.yaml
      --service-ca-file=/var/service-ca/service-ca.crt
      --v=2
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   d: Head https://oauth-openshift.apps.prod.k8s.lan: dial tcp: lookup oauth-openshift.apps.prod.k8s.lan on 172.30.0.10:53: no such host
E0924 09:08:45.322957       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.prod.k8s.lan/oauth/token failed: Head https://oauth-openshift.apps.prod.k8s.lan: dial tcp: lookup oauth-openshift.apps.prod.k8s.lan on 172.30.0.10:53: no such host
E0924 09:08:55.354123       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.prod.k8s.lan/oauth/token failed: Head https://oauth-openshift.apps.prod.k8s.lan: dial tcp: lookup oauth-openshift.apps.prod.k8s.lan on 172.30.0.10:53: no such host
E0924 09:09:05.383014       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.prod.k8s.lan/oauth/token failed: Head https://oauth-openshift.apps.prod.k8s.lan: dial tcp: lookup oauth-openshift.apps.prod.k8s.lan on 172.30.0.10:53: no such host
E0924 09:09:15.436426       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.prod.k8s.lan/oauth/token failed: Head https://oauth-openshift.apps.prod.k8s.lan: dial tcp: lookup oauth-openshift.apps.prod.k8s.lan on 172.30.0.10:53: no such host
E0924 09:09:25.480632       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.prod.k8s.lan/oauth/token failed: Head https://oauth-openshift.apps.prod.k8s.lan: dial tcp: lookup oauth-openshift.apps.prod.k8s.lan on 172.30.0.10:53: no such host
E0924 09:09:35.520203       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.prod.k8s.lan/oauth/token failed: Head https://oauth-openshift.apps.prod.k8s.lan: dial tcp: lookup oauth-openshift.apps.prod.k8s.lan on 172.30.0.10:53: no such host

      Exit Code:    2
      Started:      Fri, 24 Sep 2021 10:06:24 +0100
      Finished:     Fri, 24 Sep 2021 10:09:43 +0100
    Ready:          False
    Restart Count:  245
    Requests:
      cpu:        10m
      memory:     100Mi
    Liveness:     http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
      /var/console-config from console-config (ro)
      /var/default-ingress-cert from default-ingress-cert (ro)
      /var/oauth-config from console-oauth-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from console-token-pvh49 (ro)
      /var/service-ca from service-ca (ro)
      /var/serving-cert from console-serving-cert (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  console-serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-serving-cert
    Optional:    false
  console-oauth-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-oauth-config
    Optional:    false
  console-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      console-config
    Optional:  false
  service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      service-ca
    Optional:  false
  default-ingress-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      default-ingress-cert
    Optional:  false
  trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trusted-ca-bundle
    Optional:  false
  console-token-pvh49:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-token-pvh49
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-reachable:NoExecute op=Exists for 120s
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Normal   Pulled     20m (x242 over 23h)     kubelet  Container image "quay.io/openshift/okd-content@sha256:806090370bd0cd3e5d5113f0775ab582f82ff625bf0dca49a2a6a1792d5718b8" already present on machine
  Warning  Unhealthy  5m27s (x4933 over 23h)  kubelet  Readiness probe failed: Get "https://10.129.0.40:8443/health": dial tcp 10.129.0.40:8443: connect: connection refused
  Warning  BackOff    26s (x2923 over 23h)    kubelet  Back-off restarting failed container

Is there an OAuth DNS issue here? What host should be resolving that address (the services node?). All certs have been approved successfully, but there's still a problem here somewhere. The console isn't online either looking at this;

oc get clusteroperator
NAME                                       VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                                                             False       True          True       42h
baremetal                                  4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
cloud-credential                           4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
cluster-autoscaler                         4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
config-operator                            4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
console                                    4.7.0-0.okd-2021-08-22-163618   False       True          True       23h
csi-snapshot-controller                    4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
dns                                        4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
etcd                                       4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
image-registry                             4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
ingress                                    4.7.0-0.okd-2021-08-22-163618   True        False         True       23h
insights                                   4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
kube-apiserver                             4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
kube-controller-manager                    4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
kube-scheduler                             4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
kube-storage-version-migrator              4.7.0-0.okd-2021-08-22-163618   True        False         False      23h
machine-api                                4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
machine-approver                           4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
machine-config                             4.7.0-0.okd-2021-08-22-163618   True        False         False      23h
marketplace                                4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
monitoring                                 4.7.0-0.okd-2021-08-22-163618   True        False         False      23h
network                                    4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
node-tuning                                4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
openshift-apiserver                        4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
openshift-controller-manager               4.7.0-0.okd-2021-08-22-163618   True        False         False      23h
openshift-samples                          4.7.0-0.okd-2021-08-22-163618   True        False         False      41h
operator-lifecycle-manager                 4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
operator-lifecycle-manager-catalog         4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
operator-lifecycle-manager-packageserver   4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
service-ca                                 4.7.0-0.okd-2021-08-22-163618   True        False         False      42h
storage                                    4.7.0-0.okd-2021-08-22-163618   True        False         False      42h

I also noticed in the HA Proxy web stats that the ingress_be traffic for http and https on compute3-7 is showing down - I've confirmed that compute1-2 are both listening on 80/443 ok, but the others aren't.

I'm so close to getting this working now, it would be great to be able to deploy a full cluster, so any help would be greatly appreciated.

Thanks

[JaimeMagiera]

Was this UPI or IPI? What provider? (or bare metal?)

[tman24]

Sorry, been away from this for a few days. I added the *.apps.prod.k8s.lan A record to the DNS server on the services node (resolving to the service nodes itself), and that does indeed seem to have made a big difference. The output I now get is;

oc get clusteroperator
NAME                                       VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.7.0-0.okd-2021-08-22-163618   True        False         False      25m
baremetal                                  4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
cloud-credential                           4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
cluster-autoscaler                         4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
config-operator                            4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
console                                    4.7.0-0.okd-2021-08-22-163618   True        False         False      25m
csi-snapshot-controller                    4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
dns                                        4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
etcd                                       4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
image-registry                             4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
ingress                                    4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
insights                                   4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
kube-apiserver                             4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
kube-controller-manager                    4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
kube-scheduler                             4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
kube-storage-version-migrator              4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
machine-api                                4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
machine-approver                           4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
machine-config                             4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
marketplace                                4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
monitoring                                 4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
network                                    4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
node-tuning                                4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
openshift-apiserver                        4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
openshift-controller-manager               4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
openshift-samples                          4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
operator-lifecycle-manager                 4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
operator-lifecycle-manager-catalog         4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
operator-lifecycle-manager-packageserver   4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
service-ca                                 4.7.0-0.okd-2021-08-22-163618   True        False         False      11d
storage                                    4.7.0-0.okd-2021-08-22-163618   True        False         False      11d

oc describe co ingress
Name:         ingress
Namespace:    
Labels:       <none>
Annotations:  include.release.openshift.io/ibm-cloud-managed: true
              include.release.openshift.io/self-managed-high-availability: true
              include.release.openshift.io/single-node-developer: true
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2021-09-22T14:22:45Z
  Generation:          1
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:include.release.openshift.io/ibm-cloud-managed:
          f:include.release.openshift.io/self-managed-high-availability:
          f:include.release.openshift.io/single-node-developer:
      f:spec:
      f:status:
        .:
        f:extension:
    Manager:      cluster-version-operator
    Operation:    Update
    Time:         2021-09-22T14:22:45Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:relatedObjects:
        f:versions:
    Manager:         ingress-operator
    Operation:       Update
    Time:            2021-09-23T09:21:21Z
  Resource Version:  5534167
  Self Link:         /apis/config.openshift.io/v1/clusteroperators/ingress
  UID:               208c126e-300d-401e-8ba7-4d83ef9349b1
Spec:
Status:
  Conditions:
    Last Transition Time:  2021-09-23T09:21:21Z
    Message:               desired and current number of IngressControllers are equal
    Reason:                AsExpected
    Status:                True
    Type:                  Available
    Last Transition Time:  2021-09-23T09:21:21Z
    Message:               desired and current number of IngressControllers are equal
    Reason:                AsExpected
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2021-10-04T11:56:35Z
    Reason:                NoIngressControllersDegraded
    Status:                False
    Type:                  Degraded
  Extension:               <nil>
  Related Objects:
    Group:      
    Name:       openshift-ingress-operator
    Resource:   namespaces
    Group:      operator.openshift.io
    Name:       
    Namespace:  openshift-ingress-operator
    Resource:   IngressController
    Group:      ingress.operator.openshift.io
    Name:       
    Namespace:  openshift-ingress-operator
    Resource:   DNSRecord
    Group:      
    Name:       openshift-ingress
    Resource:   namespaces
    Group:      
    Name:       openshift-ingress-canary
    Resource:   namespaces
  Versions:
    Name:     operator
    Version:  4.7.0-0.okd-2021-08-22-163618
    Name:     ingress-controller
    Version:  quay.io/openshift/okd-content@sha256:5be59a445c7187bc7d9e0bf4971277b058b0786485597379969a5c7fdc136409
Events:       <none>

Strangely, 5 of the 7 worker nodes (3-7) still show with ingress_http_be and ingress_https_be problems in HAProxy. The first two I built look ok. The error is 'layer 4 connection problem: connection refused'. iptables is up and running ok (same as the two active workers).

[rvanderp3]

That is actually normal. You likely only have 2 router pods configured for ingress(its the default). You can adjust that here:
https://docs.okd.io/latest/networking/ingress-operator.html#nw-ingress-controller-configuration_configuring-ingress

[tman24]

Thanks - that would make sense then. I'll give it a try and see if I can scale up the ingress router pods (sorry, I'm new to OKD/Kube - it's been quite a steep learning curve!). Finally got a cluster up though, which is very satisfying...

[tman24]

Just for clarification, is it sensible to have all workers configured for ingress, or is 2 (the default) best practise?

[vrutkovs]

imo this should be guided by security requirements (you may not want to expose ports on all nodes to the external network) and performance (router pod is not particularly heavy but may use node's CPU/IO), so starting with 2 and scaling up is probably the safest option